The Pope isn’t AGI-pilled

On Monday, Pope Leo XIV unveiled an encyclical letter addressing the societal implications of artificial intelligence. The letter, titled Magnifica Humanitas, warned that the “use of AI is never a purely technical matter: when it enters processes that affect people’s lives, it touches on rights, opportunities, status and freedom.” Alongside him was Anthropic cofounder and interpretability team lead Christopher Olah, representing a partnership between the Catholic Church and one of the biggest players in AI.

The letter elicited a wide range of reactions from in and around the tech industry. Nearly everyone believed the document would be influential. Some critics questioned whether it went far enough, and others believed it should have discussed artificial general intelligence (AGI), which many companies insist is imminent. Still others thought the pope was spot-on.

“It was a pretty clear subtweet of big tech CEOs who are out here blatantly declaring that they’re eliminating staff to replace ‘lower-value human capital’ with AI, and who are also buying their way into the political rooms where it happens in order to write the rules in their favor,” said Sacha Haworth, executive director of the Tech Oversight Project.

The pope’s encyclical comes amid a backlash to AI’s growing power. Six in 10 US adults feel they have “little to no control” of how AI is used in their everyday lives, protests against the construction of data centers are increasingly common, and some people have even attempted attacks on AI CEOs themselves.

As Olah’s appearance suggests, the pope’s missive describes AI as a technology that can have positive applications, and its tone earned mixed reactions. “I’m glad it’s critical of the AI companies though I think it should be more so,” Daniel Kokotajlo, an AI researcher and former OpenAI employee who is behind the nonprofit AI Futures Project, told The Verge. Conversely, Dr. Guru Sethupathy, GM of AI governance at software company Optro, was encouraged by indications that “Pope Leo and the Vatican are not against AI but rather how to pursue a responsible path that is best for humanity.”

The decision to partner with the Vatican was a strategic move by Anthropic, a company that’s built its business on a carefully curated reputation of being a more trustworthy alternative than its competitors. Anthropic famously spent the last few months embroiled in a battle with the Pentagon over limits to military AI use, and a connection with another powerful institution could help bolster its status — and let it help shape future Vatican recommendations.

One controversial aspect of the document, in many tech circles, was that it made no mention whatsoever of AGI or superintelligence; it allows that AI systems may “often surpass human intelligence in speed and computational capacity” but says they “lack the affective, relational and spiritual perspective through which human beings grow in wisdom.” Dean W. Ball, a senior fellow at the Foundation for American Innovation, wrote on X that the encyclical “would be much improved if it were less enamored of the traditional academia/civil society talking points on AI … and more engaged with where AI is headed. But instead of doing that, the encyclical dodges in the deepest sense, denying that AI ‘really thinks’ or ‘really learns.’” Kokotajlo also said he wished the letter took the possibility of AGI and superintelligence “more seriously.”

But the document isn’t meant to do everything, several people in tech and in the Catholic world told The Verge.

“This is a major Catholic social teaching document,” Sister Susan Francois, assistant congregation leader of the Sisters of St. Joseph of Peace, said. “It’s not about AI. It’s about protecting the human person in the age of AI.” Brian Boyd, the US faith liaison for the Future of Life Institute and an instructor of Catholic social teaching at Notre Dame Seminary, called the document “more of a call to arms than a specific set of marching orders.” The pope called throughout the encyclical to “establish adequate regulatory tools capable of upholding justice and curbing the distorting effects of technological power,” but he stopped short of endorsing specific proposals.

The Tech Oversight Project’s Haworth said she views the encyclical as “setting the compass for the moral direction of the world, or at least of the Western world … You don’t have to be Catholic to see your own concerns and your own wishes and fears in this document.” She said she expects her elected leaders to take on specific concerns over AGI and do their part to hold powerful tech companies accountable, rather than for the pope’s document to be “all things for all people.” She also called it a “warning shot for leaders, for politicians, because what this document talked about is the creation of a sub-class.”

For some, the encyclical’s lack of attention to AGI wasn’t a problem, since they took it as the pope focusing on the real-world impacts of AI on vulnerable communities as the technology exists today, rather than some of the potential risks ahead for humanity as a whole if and when labs hit the (largely unspecified) AGI milestone.

Aaron Fulkerson, CEO of Opaque Systems, a startup that helps customers organize encrypted data, said the pope is “actually looking at the system and highlighting something that a lot of us in tech would recognize as a systemic risk to humanity. And I’m not even talking about AGI. I’m talking about our global economy.”

Even before the current wave of AI, tech industry centralization could be dangerous. A problem at cybersecurity firm CrowdStrike sidelined banks, hospitals, and airlines around the world, and an outage at Amazon Web Services took down a large swath of the internet, from Reddit to Venmo. For Fulkerson, the risks to the global economy only grow as power is concentrated in the hands of a couple of AI labs.

“The thing that I’ve seen in the news has been this positioning around power dynamics of the pope versus tech bros, but I think everybody’s missing the bigger story here, which is that he’s looking at a system that is intrinsically risky,” Fulkerson said. “We’re sleepwalking into a world in which one or two labs are the cognitive infrastructure of every industry on earth — that means humanity is far less resilient, not more capable.”

In the encyclical, the pope compared AI to the Tower of Babel, a structure he describes as “supported by a uniformity that eliminated diversity and that chose homogenization over communion.”. The world must “avoid the ‘Babel syndrome,’” he wrote: “the idolatry of profit that sacrifices the weak, a uniformity that neutralizes differences, and the pretense that a single language — even a digital one — can translate everything, including the mystery of the person, into data and performance.” In his reckoning, AI became not just a new technology, but a Biblical struggle. “The risk of dehumanization,” he wrote, “is an ancient and ever-new temptation that today takes on a technical guise.” The weight of those statements, not the technical specifics, is likely to be its lasting impact.