Jordan Pearson | The Verge

Telegram’s CEO has taken a hands-off approach for years — now his luck might have run out

2024-08-27T18:45:00-04:00

Telegram CEO Pavel Durov’s arrest in France on Saturday took the tech world by surprise. The 39-year-old Russian-born billionaire was detained after touching down at an airport outside of Paris in his private plane. And with scant detail available, observers wondered what the unprecedented action meant for free speech, encryption, and the risks of running a platform that could be used for crime.

On Monday, French officials revealed that Durov is being questioned as part of a wide-ranging criminal investigation surrounding crimes that regularly happen on Telegram. While some of the accusations could still raise red flags, many seem to concern serious offenses — like child abuse and terrorism — that Durov would reasonably have been aware of. But many questions remain unanswered, including how worried other tech executives should be.

Crime happens on lots of platforms. Why does Telegram stand out?

Telegram is a messaging app that was founded in 2013 by brothers Pavel and Nikolai Durov. While it’s sometimes portrayed as an “encrypted chat app,” it’s mostly popular as a semi-public communication service like Discord, particularly in countries like Russia, Ukraine, Iran, and India.

It’s a massive platform that is used by millions of innocent people every day, but it’s also gained a reputation for being a safe haven for all sorts of criminals, from scammers to terrorists.

Pavel Durov has crafted a brash pro-privacy persona in public. In an interview with Tucker Carlson this year, Durov gave examples of times that Telegram has declined to hand over data to governments: when Russia asked for information on protesters, for instance, and when US lawmakers requested data on participants in the January 6th Capitol riot. Earlier, at a 2015 TechCrunch event, Durov said that Telegram’s commitment to privacy was “more important than our fear of bad things happening, like terrorism.”

That sentiment isn’t radically out of step with what many encryption proponents believe, since strong encryption must protect all users. A “backdoor” targeting one guilty party compromises everyone’s privacy. In apps like Signal or iMessage, which use end-to-end encryption, nobody but the sender and recipient can read a message’s contents. But as experts have pointed out, Telegram doesn’t implement this in any meaningful sense. End-to-end encryption has to be enabled manually for one-on-one messaging, and it doesn’t work for group chats or public channels where illegal activity occurs in plain view.

“Telegram looks much more like a social network that is not end-to-end encrypted,” John Scott-Railton, senior researcher at Citizen Lab, told The Verge. “And because of that, Telegram could potentially moderate or have access to those things, or be compelled to.”

The ecosystem of extremist activity on the platform is so well-known that it even has a nickname: “terrorgram.” And much of it happens in the open where Telegram could identify or remove it.

Telegram does occasionally take action on illegal content. The platform has blocked extremist channels after reports from the media and revealed users’ IP addresses in response to government requests, and an official Telegram channel called “Stop Child Abuse” claims that the platform blocks more than 1,000 channels engaged in child abuse every day in response to user reports.

But there have been numerous reports of lax moderation on Telegram, with its general approach being frequently described as “hands off” compared to competitors like Facebook (which still struggles to effectively moderate its own massive platform). Even when Telegram does take action, reporters previously discovered that the service may only hide the offending channels rather than block them.

All of this puts Telegram in a unique position. It’s not taking a significantly active role in preventing use of its platforms by criminals, the way most big public social networks do. But it’s not disavowing its role as a moderator, either, the way a truly private platform could. “Because Telegram does have this access, it puts a target on Durov for governmental attention in a way that would not be true if it really were an encrypted messenger,” said Scott-Railton.

Why was Pavel Durov arrested? And why were other tech executives upset?

According to a statement by French prosecutor Laure Beccuau, Durov is being questioned as part of an investigation on Telegram-related crimes, which was opened on July 8th.

The listed charges include “complicity” in crimes ranging from possessing and distributing child sexual abuse material to selling narcotics and money laundering. Durov is also being investigated for refusing to comply with requests to enable “interceptions” from law enforcement and for importing and providing an encryption tool without declaring it. (While encrypted messaging is legal in France, anyone importing the tech has to register with the government.) He’s also accused of “criminal association with a view to committing a crime” punishable by more than fine years in prison. The statement added that Durov’s detainment could last 96 hours, until Wednesday, August 28th.

When Durov was first taken into custody, though, these details weren’t available — and prominent tech executives immediately rallied to his defense. X owner Elon Musk posted “#FreePavel” and captioned a post referencing Durov’s detention with “dangerous times,” framing it as an attack on free speech. Chris Pavlovski, CEO of Rumble — a YouTube alternative popular with right-wingers — said on Sunday that he had “just safely departed from Europe” and that Durov’s arrest “crossed a red line.”

Durov’s arrest comes amid a heated debate over the European Commission’s power to hold tech platforms responsible for their users’ behavior. The Digital Services Act, which took effect last year, has led to investigations into how tech companies handle terrorism and disinformation. Musk has been recently sparring with EU Commissioner Thierry Breton over what Breton characterizes as a reckless failure to moderate X.

Over the weekend, the public response was strong enough that French President Emmanuel Macron issued a statement saying that the arrest took place as part of an ongoing investigation and was “in no way a political decision.” Meanwhile, Telegram insisted that it had “nothing to hide” and that it complied with EU laws. “It is absurd to claim that a platform or its owner are responsible for abuse of that platform,” the company’s statement said.

Is the panic around Durov’s arrest justified?

With the caveat that the situation is still evolving, it seems like free speech is not the core issue — Durov’s alleged awareness of crimes is.

In posts on X, University of Lorraine law professor Florence G’sell noted that the most serious charges against Durov are the ones alleging direct criminal conspiracy and a refusal to cooperate with the police. By contrast, the charges around declaring encryption tech for import seem like minor offenses. (Notably, in the United States, certain import / export controls on encryption have been found to be violations of the First Amendment.) G’sell noted that there are still unknowns surrounding which criminal codes Durov could be charged under but that the key issue seems to be knowingly providing tech to criminals.

Arguably, Telegram has long operated on a knife-edge by attracting privacy-minded users — including a subset of drug dealers, terrorists, and child abusers — without implementing the kind of robust, widespread encryption that would indiscriminately protect every user and the platform itself. If child abuse or terrorism is happening in clear view, platforms have a clear legal responsibility to moderate that content.

That’s true in the US as well as in Europe. Daphne Keller, platform regulation director of the Stanford Cyber Policy Center, called Durov’s arrest “unsurprising” in X posts and said it could happen under the US legal system, too. Failing to remove child abuse material or terrorist content “could make a platform liable in most legal systems, including ours,” she wrote. Section 230, which provides a broad shield for tech platforms, notably doesn’t immunize operators from federal criminal charges.

That said, there are still many unknowns with Durov’s arrest, and there may be further developments that justify some of the concern over implications for encryption tech. References to lawful “interceptions” — a term that typically refers to platforms facilitating surveillance of users’ communications — are particularly worrying here.

European and US police have increasingly targeted encrypted chat platforms used by criminals in recent years, hacking a platform called EncroChat and even going as far as to secretly run an encrypted phone company called Anom. Notably, those platforms were focused on serving criminals. Telegram, on the other hand, is aimed at the general public. In his interview with Carlson, Durov claimed that at one point, the FBI — which played a key role in the Anom operation — attempted to convince Telegram to include a surveillance backdoor.

“This case definitely illustrates — whatever you think about the quality of Telegram’s encryption — how many people care about the ability to communicate safely and privately with each other,” said Scott-Railton.

Durov’s arrest also raises the question of what should push a platform into legal liability. Serious crimes certainly occur on Facebook and nearly every other massive social network, and in at least some cases, somebody at the company was warned and failed to take action. It’s possible Durov was clearly, directly involved in a criminal conspiracy — but short of that, how ineffectual can a company’s moderation get before its CEO is detained the next time they set foot on European soil?

Drone photographer pleads guilty to Espionage Act charges

2024-07-12T14:35:57-04:00

Newport News Shipbuilding, a defense contractor whose shipyard is alleged to have been photographed by drone. | Getty Images

A foreign graduate student has pleaded guilty to crimes under the Espionage Act for photographing classified US Navy ships with a drone. The case appears to be a first-of-its kind prosecution by the Department of Justice.

Fengyun Shi, a Chinese citizen and graduate student at the University of Minnesota, was arrested in January after a drone he was flying got stuck in a tree in Newport News, Virginia. A suspicious resident called the police and Shi was questioned before abandoning the drone and fleeing. After the FBI seized the drone and pulled the images off it, investigators discovered that Shi had photographed Navy vessels at multiple shipyards in Virginia. One of those shipyards, in Newport News, was actively manufacturing next-generation aircraft carriers and nuclear submarines. Both of these types of vessels contain classified components.

Shi was charged with six misdemeanors under the Espionage Act. It appears to be, national security law expert Emily Berman previously told me, the first known prosecution under a WWII-era statute banning the use of aircraft to photograph sensitive military sites.

The first known prosecution under a WWII-era statute banning the use of aircraft to photograph sensitive military sites

On Monday, Shi pleaded guilty to two counts of violating that statute. Specifically, for photographing vessels at the shipyard in Newport News. Each offense may result in up to a year in prison, a $100,000 fine, and another year of supervised release, but prosecutors note that he could receive a heavier sentence that a higher court may review for reasonableness. The plea agreement also explains that Shi could be deported, but that he can apply for asylum if he believes he will be subject to torture in China.

The statement of facts accompanying Shi’s plea agreement contains new information. For example, it reveals that Shi purchased the drone one day before flying to Virginia from San Francisco. There is no explanation given for why he was in San Francisco. According to the statement, Shi only flew the drone around the shipyards and did not take any photos that did not contain US Navy vessels. Finally, it says that Shi was arrested trying to board a one-way flight to China from California.

There are many unknowns surrounding Shi’s strikingly novel prosecution, including why the DOJ pursued it in the first place. Despite the case taking place amid rising tensions between the US and China, Shi has not been accused of acting as a spy; his only crime was taking photos with a drone. Berman previously said that his case could even raise important First Amendment issues.

Court documents filed to date provide no explanation for why Shi took the photos, although the plea agreement states that Shi acknowledges he had no “innocent reason” for doing so. Even people around Shi seemed baffled by the case. I previously spoke to a colleague from the University of Minnesota who was surprised to learn that Shi was even still in the US, adding that he’d effectively abandoned his studies months prior to his arrest.

The DOJ declined to comment, and Shi’s attorney did not respond to a request. Shi now awaits sentencing.

The RIAA versus AI, explained

2024-06-26T08:00:00-04:00

Udio and Suno are not, despite their names, the hottest new restaurants on the Lower East Side. They’re AI startups that let people generate impressively real-sounding songs — complete with instrumentation and vocal performances — from prompts. And on Monday, a group of major record labels sued them, alleging copyright infringement “on an almost unimaginable scale,” claiming that the companies can only do this because they illegally ingested huge amounts of copyrighted music to train their AI models.

These two lawsuits contribute to a mounting pile of legal headaches for the AI industry. Some of the most successful firms in the space have trained their models with data acquired via the unsanctioned scraping of massive amounts of information from the internet. ChatGPT, for example, was initially trained on millions of documents collected from links posted to Reddit.

These lawsuits, which are spearheaded by the Recording Industry Association of America (RIAA), tackle music rather than the written word. But like The New York Times’ lawsuit against OpenAI, they pose a question that could reshape the tech landscape as we know it: can AI firms simply take whatever they want, turn it into a product worth billions, and claim it was fair use?

“That’s the key issue that’s got to get sorted out, because it cuts across all sorts of different industries,” said Paul Fakler, a partner at the law firm Mayer Brown who specializes in intellectual property cases.

What are Udio and Suno?

Both Udio and Suno are fairly new, but they’ve already made a big splash. Suno was launched in December by a Cambridge-based team that previously worked for Kensho, another AI company. It quickly entered into a partnership with Microsoft that integrated Suno with Copilot, Microsoft’s AI chatbot.

Udio was launched just this year, raising millions of dollars from heavy hitters in the tech investing world (Andreessen Horowitz) and the music world (Will.i.am and Common, for example). Udio’s platform was used by comedian King Willonius to generate “BBL Drizzy,” the Drake diss track that went viral after producer Metro Boomin remixed it and released it to the public for anyone to rap over.

Why is the music industry suing Udio and Suno?

The RIAA’s lawsuits use lofty language, saying that this litigation is about “ensuring that copyright continues to incentivize human invention and imagination, as it has for centuries.” This sounds nice, but ultimately, the incentive it’s talking about is money.

The RIAA claims that generative AI poses a risk to record labels’ business model. “Rather than license copyrighted sound recordings, potential licensees interested in licensing such recordings for their own purposes could generate an AI-soundalike at virtually no cost,” the lawsuits state, adding that such services could “[flood] the market with ‘copycats’ and ‘soundalikes,’ thereby upending an established sample licensing business.”

The RIAA is also asking for damages of $150,000 per infringing work, which, given the massive corpuses of data that are typically used to train AI systems, is a potentially astronomical number.

Does it matter that AI-generated songs are similar to real ones?

The RIAA’s lawsuits included examples of music generated with Suno and Udio and comparisons of their musical notation to existing copyrighted works. In some cases, the generated songs had small phrases that were similar — for instance, one started with the sung line “Jason Derulo” in the exact cadence that the real-life Jason Derulo begins many of his songs. Others had extended sequences of similar notation, as in the case of a track inspired by Green Day’s “American Idiot.”

One track started with the sung line “Jason Derulo” in the exact cadence that the real-life Jason Derulo begins many of his songs

This seems pretty damning, but the RIAA isn’t claiming that these specific soundalike tracks infringe copyright — rather, it’s claiming that the AI companies used copyrighted music as a part of their training data.

Neither Suno nor Udio have made their training datasets public. And both firms are vague about the sources of their training data — though that’s par for the course in the AI industry. (OpenAI, for example, has dodged questions about whether YouTube videos were used to train its Sora video model.)

The RIAA’s lawsuits note that Udio CEO David Ding has said the company trains on the “best quality” music that is “publicly available” and that a Suno co-founder wrote in Suno’s official Discord that the company trains with a “mix of proprietary and public data.”

Fakler said that including the examples and notation comparisons in the lawsuit is “wacky,” saying it went “way beyond” what would be necessary to claim legitimate grounds for a lawsuit. For one, the labels may not own the composition rights of the songs allegedly ingested by Udio and Suno for training. Rather, they own the copyright to the sound recording, so showing similarity in musical notation doesn’t necessarily help in a copyright dispute. “I think it’s really designed for optics for PR purposes,” Fakler said.

On top of that, Fakler noted, it’s legal to create a soundalike audio recording if you have the rights to the underlying song.

When reached for comment, a Suno spokesperson shared a statement from CEO Mikey Shulman stating that its technology is “transformative” and that the company does not allow prompts that name existing artists. Udio did not respond to a request for comment.

Is it fair use?

But even if Udio and Suno used the record labels’ copyrighted works to train their models, there’s a very big question that could override everything else: is this fair use?

Fair use is a legal defense that allows for the use of copyrighted material in the creation of a meaningfully new or transformative work. The RIAA argues that the startups cannot claim fair use, saying that the outputs of Udio and Suno are meant to replace real recordings, that they are generated for a commercial purpose, that the copying was extensive rather than selective, and finally, that the resulting product poses a direct threat to labels’ business.

In Fakler’s opinion, the startups have a solid fair use argument so long as the copyrighted works were only temporarily copied and their defining features were extracted and abstracted into the weights of an AI model.

“It’s extracting all of that stuff out, just like a musician would learn those things by playing music.”

“That’s how computers work — it has to make these copies, and the computer is then analyzing all of this data so they can extract the non-copyrighted stuff,” he said. “How do we construct songs that are going to be understood as music by a listener, and have various features that we commonly find in popular music? It’s extracting all of that stuff out, just like a musician would learn those things by playing music.”

“To my mind, that is a very strong fair use argument,” said Fakler.

Of course, a judge or a jury may not agree. And what is dredged up in the discovery process — if these lawsuits should get there — could have a big effect on the case. Which music tracks were taken and how they ended up in the training set could matter, and specifics about the training process might undercut a fair use defense.

We are all in for a very long journey as the RIAA’s lawsuits, and similar ones, proceed through the courts. From text and photos to now sound recordings, the question of fair use looms over all these cases and the AI industry as a whole.