Aliyah, a light-skinned Black woman dressed in country-western gear, is struggling to sell metal buckles she handmade on TikTok. In a video for the social media platform from March, she cries to the camera and pleads for views: “Even as a black woman, I have more faith that white women will stay 13 seconds [on this video] to save my belt buckle business,” the onscreen text reads. She wipes a tear off her cheek. 

But Aliyah isn’t real, and neither are her supposedly handmade products — she’s one of many AI-generated influencers created to sell mass-produced products via dropshipping on TikTok, Facebook, and Instagram. Identical belt buckles — sunflower design, detachable knife inlay, and all — are sold on the fast-fashion site Shein, and for a quarter of the price.

Aliyahsbuckles on TikTok (left), and Amaya’s buckles on TikTok (right)

There are some clues to spot to determine that this video is AI-generated. Aliyah’s voice is robotic and emotionless, which doesn’t match the crying face on the screen. In one clip, she is sewing a leather belt where there wouldn’t usually be sewing at all. When she wipes a tear off her face, the stream of liquid below where she wipes also disappears. And lastly, there are dozens of uncannily comparable videos, but with different AI-generated characters, circulating on TikTok. One, a profile for an account called “Aliyahsbuckles,” features an identical background, tabletop, and spool of twine. 

The Verge found dozens of accounts on TikTok with similar narratives and a variety of dropshipping products, including belt buckles, mugs shaped like cowboy boots, crochet bags, and cardigans. Some of these videos are labelled as AI-generated. Similar accounts are also active on Instagram and Facebook. Nearly all aspects of the accounts appear to be AI-generated — from the “person” in the video to automated responses to comments, which in some cases attempt to mimic African American vernacular — and experts warn scams like this are growing every day.

“It’s massive,” Jeremy Carrasco, a researcher of AI-generated media and director of Riddance.ai, an organization that focuses on AI video detection, told The Verge of AI-generated videos connected to e-commerce stores. “Most of them aren’t coordinated. Some of them are coordinated. A lot of the time they’ll run a single [AI-generated] actor, or a couple actors will run all sorts of shops,” he explained. Those AI-generated avatars pretend to make the items, go to fairs to display their products, and “respond” to comments through automation. “What we’re seeing right now are these retail scams where they’ll link to the Shopify websites.”

Carrasco estimates his research team is finding up to 100 accounts that attempt to sell products via AI-generated avatars every day. Most of the accounts found by The Verge were created in the last two months and contain videos about small businesses owned by marginalized individuals struggling to make a sale; these videos are incredibly similar, with only slight variations in their scripts. While we also found Native American, Hispanic, and white women characters, the most viewed and engaged-with AI-generated characters found by The Verge are Black women. Aliyah’s account alone has 40,000 followers. 

“What we’re seeing here is empathy bait,” said Carrasco. “If there is a popular dropship item that could be sold to some sort of niche community, they will find it and they will try [to use] some personality to do it.” Carrasco explains that these trends are usually random opportunities to make money. “It’s just an arbitrary opportunity, which is what a lot of AI content out there is — the platforms don’t really care and people don’t notice.”

Aliyah’s most popular video, the one described in the introduction of this article, has 814,000 likes, 6.5 million views, and almost 30,000 comments. Some comments identify the content as AI-generated, but many express the desire to help Aliyah’s business, commenting on the video to increase her visibility on the platform. 

India Cater-Campbell was one of those commenters, expressing a desire to buy Aliyah’s belt buckles. “I was trying to be supportive to an independent Black businesswoman,” said Cater-Campbell, an actual Black business owner working on opening a café in Seattle. “[I felt] solidarity as I am trying to start a business myself.”

Despite a lack of studies on the latest gen AI video models, Carrasco estimates that these videos are “realistic enough to trick” most people. Users of short-form video content platforms have been trained to scroll mindlessly and not look further into the content they’re creating. Ironically, this is what might have saved Cater-Campbell from actually buying a belt buckle: She couldn’t immediately find a store link, so she scrolled away and promptly forgot about Aliyah.

But people are falling for these scams, and their prominence grows: Two weeks ago, Gizelle Bryant from The Real Housewives of Potomac admitted to buying two crocheted bags after seeing a video in which an AI-generated Black boy said he was being bullied by white boys for crocheting. “I was like, I want to help this little Black boy make his goal,” Bryant said on her podcast Reasonably Shady, adding that other celebrities had also been in the comment section. “How did I get tricked? Viola Davis was on there, too.”

The trend is a kind of digital blackface, according to communications researcher Cienna Davis at the University of Pennsylvania. “Digital blackface is a phenomenon where non-Black individuals are able to use the internet and digital technologies to mimic Black cultural expression for personal, economic, or political gain,” Davis explained over a video call. Davis has written previously about digital blackface, pointing to the use of gifs of Black people as well as the impersonation of Black people for political goals. “It’s rooted in blackface minstrelsy, which is tied to the legacy of slavery,” she explained.

Based on the idea that Blackness is “inherently exploitable” and “up for grabs,” digital Blackface is used to “extract value from Black bodies in whatever way [non-Black people] see fit,” Davis said. In this case, the videos are mimicking “a recognizable idea of Black struggle,” she added.

Even without confirmation that the people behind the videos aren’t Black, Tempest M. Henning, an assistant professor of philosophy at Fisk University, confirms the videos are digital blackface. “Blackface is any kind of caricature-like portrayal of Black people, which can include Black people dressing up in a caricature-like fashion of Black people,” Henning says. She cites historical precedents of Black people occasionally being forced to perform in minstrel shows and the more recent example of Zoe Saldaña darkening her skin tone and wearing prosthetics to play the role of Nina Simone in a biopic.

There’s an inherent falseness to the Blackness these avatars claim. As Henning explained, “The names of the avatar are coded Black, like Aliyah or Amaya, but there’s nothing else [that signals authentic Blackness] other than the avatar itself.” That’s only reinforced by the replication of content across racial identities, which, Henning said, results in the flattening of those identities.

The Verge reached out to Aliyahsbuckles, as well as other stores selling similar products, but received no response. 

The videos posted on Aliyah’s account are replicated scene by scene across many similar accounts selling dropshipped items, with slight modifications to fit the identity of the character or the item being sold. In one video featuring another Black woman character, Amaya, a white woman mockingly throws coffee on belt buckles being showcased at a fair. A mildly frustrated Amaya — the emotions are somewhat robotic and never look quite right — sighs and goes back to the honest work of making her buckles. A similar scene appears in an account called ChubbyKnots; this time the avatar is a Black girl and the product is a crocheted butterfly cardigan.

Henning said there’s an appeal to virtue signaling in these videos, where the viewer is called to perform kindness or even racial or class solidarity. “It’s virtue signaling in the sense of ‘Oh, that white lady did that, but I am not that kind of white lady, so where do I buy this belt buckle?’” Henning said. “It’s like they’re saying, ‘I don’t stand with these people.’”

However, Henning pointed out the superficiality of race- or class-based support when users are not pausing to research who it is they are supporting in the first place. 

It’s perhaps a larger social outcome of short-form video content and the consumption of slop:  Solidarity is often being performed at “surface level,” she explained, rather than through a coherent and multifaceted political lens. “I might be very invested in a Black-owned business, but that doesn’t necessarily mean that all Black businesses align with my political views and what I stand for,” Henning said.

This appeal to virtue signaling goes beyond the category of race, also drawing on working-class markers and a simulation of people working hard to earn their keep in a failing economy. “[It’s not just] Black struggle because on the other videos, we can observe working class struggle, small-business struggle,” Davis said. “But it’s obviously using that [narrative] to sell mass-produced goods. It’s not creative at all. It’s really just AI-generated templates and creation of characters that get optimized for consumer identification and investment in the products. It’s broader than ‘white guilt’ altogether, because it can be used in whatever way feels necessary or feels applicable to the product.”

Despite this flattening of racial identity, or maybe because of it, algorithms — which have been shown to perpetuate conscious and unconscious racial bias from the people who build them — are delivering AI-generated content that fits the interests they’ve inferred for each user, Carrasco said. These interests may include content from specific demographics that AI personas are able to mimic. “It targets the demographic that [the algorithm] can see,” he said. “Usually a Black woman will see AI-generated Black people on their feed, an Asian person will get AI-generated Asians.”

Looking closer at the content, it’s easy to spot inconsistencies and AI tells. We never see more than a few seconds of the character onscreen because AI apps like Seedance 2.0, Midjourney, and OpenAI’s recently discontinued Sora 2 are not able to create clips longer than 15 seconds. When the products are showcased, the lighting is brighter rather than sepia-toned, and the hands holding the buckles are white. The emotions on the characters’ faces don’t quite match the tone of their voices. 

But this recognition requires AI media literacy that is not widely available, especially at the scale that these videos are being created and the technology is advancing toward faster automation and more realistic avatars. “People who aren’t trained in media literacy or critical media literacy are just going to take it in unquestionably,” Davis said.

The very structures of short-form video platforms, which are major channels for misinformation, encourage a lack of discernment from users, who have been trained to want to scroll to the next video rather than think about what they are watching. “[These platforms] are being exploited by AI content partially because in order to figure out something’s AI, it usually takes a second or two,” Carrasco said. “By that time, you’ve already registered some engagement.”

People who profit from these kinds of videos know this, and the tools to create scalable AI-generated short-form commercials are widely available online. In reporting this story, The Verge found YouTube channels and forums dedicated to tutorials on how to create ads like this without product samples, influencer fees, or original scripts. 

These tutorials teach users how to copy viral videos with multiple AI tools and replace real influencers with AI-generated characters. ChatGPT and Gemini are used to extract and copy scripts from real videos by real influencers, with the option to create multiple variants of the same script. These tools are also used to generate photos of fake people and fake backgrounds based on real influencers and real backgrounds, which are then imported into apps like Kling 2.0 and Maxfusion. With some AI video generator models, videos by real influencers can be fully imported into the app and the user can replace the real person with an AI character they created, effectively stealing the script and background of the original video.   

While media literacy might help individual users not fall for AI-generated video scams, social media platforms are currently failing to moderate and, at minimum, label this content. “Platforms should have stronger AI detection and labeling structures,” Davis said. “Bias checks on these platforms would also be welcome, as well as clear reporting pathways for reporting AI misuse on platforms.”

In addition to moderation, Carrasco suggested that account transparency would also help users know whether they’re watching AI-generated content, with clear labels that are easy to see and don’t require extra steps.

But short-form video content consumers already have years of experience scrolling their feeds and superficially engaging with whatever comes up. Reversing this cognitive training feels almost impossible to do at a mass scale without platforms themselves taking charge. With no real incentive for short-form video platforms to do so and profit at stake, that seems unlikely — but until then, AI scammers will continue to take advantage of whoever takes the bait.

Last month, I got the invitation of a lifetime in my inbox: Vogue wanted to purchase my freelance writing services to cover a fashion event in Canada. The email, with the subject line “Vogue Talent Recruiting Team,” arrived at a time when I was specifically looking for new assignments to dig into after I finished a six-month research contract. Surprisingly, editor-in-chief of British Vogue Edward Enninful himself was reaching out to me by email to offer me an all-expenses paid trip to Toronto, plus payment for coverage of the event. 

When I read the email again, I noticed something was off. Enninful’s email address wasn’t a standard Condé Nast address — it was a Gmail account. His name also didn’t appear on the “From:” space — instead, the email was from “Condé Nast.” There were typos, and the writing was awkward: “Our reps came across your past works online and with interest we thought you might be interested in joining the rest of the team.” It just didn’t sound like someone in charge of a major magazine.

I knew the job offer was a sham. This was not Edward Enninful from British Vogue. It was someone pretending to be Enninful in order to get something from me. Sure enough, when I asked for more information, I got a PDF full of typos, with a table of costs they would cover for my trip. 

Job scams generally consist of bogus employment offers that seek to take advantage of the job seeker somehow. There are many different kinds of job scams, but scammers are generally after two main things: either the job seeker’s personal information or a way to steal the job seeker’s money through bogus offers. 

Protecting yourself from job scams is as much about cybersecurity as it is about doing your research and generally being a little skeptical about the company that is trying to hire you. As you’ve heard before, you want to make it as hard as possible for scammers to hack into your accounts by using two-factor authentication and regularly changing your passwords. But making sure you’re not getting scammed also draws on new skills, like spotting the tiny details that show something’s not right. In my case, the Gmail address gave it away.

I suspect my own scammer was after one of two things: either spear phishing to gain access to my accounts or sending me a bad check to cover the cost of the trip.

“This is a common scam,” explained Bryan Hornung, CEO of Xact I.T. Solutions. “They send you a bad check that takes about a week to bounce. You pay the ‘travel agent’ for the cost of the ‘trip,’ thinking you are being reimbursed. They take your money for the trip, and eventually, your bank returns the check you deposited from them. You’d be out $3,950 plus any bank fees.”

Ariel Robinson, senior product manager of security at New Relic, was more suspicious about a spear phishing attempt, given the fake PDF that was sent to me. “The scammers are trying to get you as the victim to send them your personal information,” Robinson explained. “I would go through, change all of your passwords, and run a virus scanner or see if you can restore your laptop to before you opened this attachment.” 

If the people who targeted me had gotten enough of my personal and financial information, they could have made me into an unwilling money mule by setting up bank accounts and credit cards through identity theft.

Either way, what attracted the scammers to me as a target is that I am a freelance writer who is currently looking for work. I will probably never know how much they actually know about me — did they see me asking for assignments on Twitter? Did they get my email from my Twitter bio or from my website? Did they care that I have never once written about fashion? — but Robinson says how much I share online is key to keeping myself safe.

“This is something that is difficult for freelance writers in particular,” Robinson, who used to be a freelance journalist before pivoting into cybersecurity, explained. “To make our living, we have to put ourselves out there, right? You have a website. You make your contact info readily available. That means that we have to be extra careful and extra judicious because we make ourselves that much easier to find.”

According to the Federal Trade Commission, job scams are on the rise. In 2021, the agency received more than twice the number of job scam reports than in 2020, and in the first quarter of 2022, there were already more than 16,000 complaints filed. And it’s not just freelancers like me who are vulnerable — before the remote work boom, spotting fake opportunities that swindled workers of their money was easy: working from home with flexible hours was what made opportunities suspicious, too good to be true. But now, with more and more employers offering WFH and hybrid arrangements, how can workers tell a scam apart from a good opportunity?

Robinson says that the old “if it’s too good to be true, it’s because it is” rule still stands despite a shift in working culture post-pandemic. Unfortunately, much of the onus to stay safe falls upon the job seeker or gig worker who is simply trying to make a living. 

Researching the company to make sure it’s real is essential, as is looking for red flags like typos and researching the person who reached out to you. Not opening any attachments — a mistake I made out of curiosity — until you confirm the opportunity is legitimate is also crucial, as is not sharing your bank information with companies who reach out to you. Observing the email domain, which was ultimately what gave my scammers away for using Gmail rather than the standardized Condé Nast email format, can also reveal the legitimacy of the offer.

“When I was a writer and I needed to be easy to find, I abided by some safety tips to keep myself safe,” Robinson said. “I never post about where I am. I never talk about location. If I talk about location, it’s very broad, and I also turned geolocation off on all of my devices. I check all of my app permissions regularly to make sure it’s still turned off.” 

Robinson also flags LinkedIn for being rife with scammers, particularly because it’s a networking-based app where job seekers are eager to meet people who might offer them a job. “Be judicious,” Robinson says. “Actually go to the person’s profile and look at it.”

Encountering a scam I could have fallen for has strengthened my commitment to my own cybersecurity. As Robinson suggested, I changed my passwords and ran an antivirus scan on my laptop — indeed, as she warned me, malware was found and deleted. But Robinson wants internet users to know they can protect themselves from scams if they are alert and set boundaries around the information they share.

“The data economy is huge,” she said. “But users aren’t helpless. We have more agency than we think we do.”

For more than two decades, Lorraine* has known her ex-boyfriend is watching her. She cut contact with him 20 years ago, but in Facebook posts and intimidating emails, he makes sure she knows he’s still keeping tabs, from the birth of her children to her most recent wedding anniversary. After years of abuse that resulted in a PTSD diagnosis and intense nightmares, the notes are chilling, and make her uncomfortable putting any personal information in an online space where he could see it.

“It’s affected my relationship with my friends,” Lorraine says. “It’s affected my relationship with my partner. It’s affected my ability to feel safe.”

The advice many survivors like Lorraine receive when they look for help is to leave their online life behind and delete themselves from the internet entirely. There are many tutorials on how to delete your online presence. Given how often abusers use digital channels to harass their targets, erasing yourself can seem like the obvious choice. But interpersonal abuse thrives on the alienation of its victims, and some kind of online presence can be a crucial lifeline for people trying to escape their abusers and rebuild a new life. 

 

This story is part of Keep it Locked: how to protect yourself online.

Tony Hunt, chief development officer of Operation Safe Escape, a nonprofit organization that helps victims of domestic violence escape their abusers, says deleting yourself from the internet could be exactly what the stalker wants. “It’s about control, they want to isolate you because that gives them absolute control over everything you do,” Hunt explained. “It’s easy to think you need to disappear, but you don’t need to.”

It’s an urgent question for thousands of people quietly struggling with domestic violence. One in four women and one in nine men will experience severe intimate partner physical violence at some point in their lives, according to the National Coalition Against Domestic Violence. One in seven women and one in 18 men have been stalked by an intimate partner during their lifetime, to a point where they felt very fearful or believed that they or someone close to them would be harmed or killed. A 2015 survey of college students found that nearly 75 percent had experienced some kind of tech-assisted intimate partner victimization in the past year.

For Lorraine, deleting herself from the internet didn’t feel like a solution. “It felt like I was removing my online liberty because of someone else’s abuse,” she said. “Both my husband and I have worked in various countries, so social media is brilliant for keeping in contact with people that we rarely see. We wouldn’t want to lose that ability.”

Julia’s* ex-partner surveilled her even when they were together and continued to do so after they had broken up, which made online spaces particularly dangerous. But instead of retreating, the threat inspired her to learn more about internet security, and to be more careful with how and where she logged on. For her, establishing safe communications with trusted people was particularly essential.

“An abusive relationship is already devastating, and shrinking afterwards adds to the devastation,” Julia said. “We can develop intentional boundaries on the internet, it doesn’t have to be all or nothing. It was better for me to learn more about internet security, privacy, and different tools stalkers can use than retreating, because that knowledge empowered me to have a more realistic sense of what is possible.”

Some survivors also worry that disappearing entirely will escalate the situation, according to University of Kentucky professor TK Logan, who researches cyberstalking. “I’ve had victims say, he was threatening me and doing all this stuff through social media so I got off social media,” she explained. “And now all the survivor can think about is, ‘Oh, my God, what is he doing? Is he just going to show up physically?’”

Hunt frames the security measures survivors should employ as a new kind of lifestyle where survivors are intentional about their online presence, rather than a disappearing act. “The priority is making sure that nobody’s advertising everything about their personal lives,” he said. “Because [the abuser] will have something that will give them access to the kids or their location or their regimen throughout the day. Once you’ve established those boundaries and you start living your life again, you’ll be glad you did it.” 

Though Operation Safe Escape trains survivors on more general security protocols rather than only focusing on online security, Hunt says technology-facilitated abuse comes up often. Part of the organization’s job, in partnership with the Coalition Against Stalkerware, is to identify developers that claim their software is for law enforcement, but actually sell their products to individuals looking to coerce and control their victims.

“What happens is, you have boyfriends who will pay contractors to install a keylogger on their girlfriend’s computer, and all of a sudden, a whole new world opens up,” Hunt said. “If the abuser has the right tools and he has physical access to the device, in 30 to 60 seconds there’s a lot he can do.” 

With regard to these higher-level security threats that include hacking and surveillance, Hunt recommends survivors get directly in contact with Operation Safe Escape so they can be guided on what options they have to leave safely and regain their life and autonomy. “We are really selective about who we work with and how much information we put out there,” Hunt explained. “Once that information is out there, guess who tries to use it against survivors?” Logan also recommends setting up multiple-factor authentication, and directs survivors to the Stalking and Harassment Assessment & Risk Profile (SHARP) tool.

The most reliable way to detect stalkerware on an Android device is to run an antivirus app and do a scan. For iOS users, Apple provides a step-by-step guide for detecting invasive apps. The Coalition Against Stalkerware also recommends that potential victims keep a log of what they are experiencing, to help detect patterns and show the history of what has been happening if they choose to get help from law enforcement or a survivor assistance charity.

But even given the threat of malicious software, the most effective protections still rely on a strong support network, which is much easier to build when you’re connected. “The goal is to put up as many barriers, make it as difficult as possible for the stalker,” says Logan, “and then the other step is to get support.”

*Some names have been changed to protect sources who have been targeted by abuse.