The Dark Side of AI-generated Content: Deep Fakes and Unmoderated Social

The rise and democratization of AI-generated content presents one of the biggest security and risk threats since the dawn of the internet. With over 3 billion images and 720,000 hours of video produced daily it’s a literal avalanche of content that even big tech firms such as Tik Tok and Facebook can’t adequately tackle.

For those new to the space, AI-generated content refers to media, such as text, images, audio, or video, created using artificial intelligence algorithms. These algorithms analyze and mimic patterns from existing data to produce new content, often indistinguishable from real content. So, that video you are watching of the Israel-Hamas war?  It could be a fake created purely for propaganda purposes.  Or those scandalous pictures of the FT500 CEO in a compromising situation?  Also fabrications. What about that phone call you received from your CFO telling you to wire cash?  Also not him. 

Deep fakes, a subset of AI-generated content, involve manipulated videos or images that use AI techniques to overlay someone's likeness onto another person's body. Shallow fakes cost virtually nothing to make, and a deep fake can be made for as little as US$500. Last year, it was estimated that around 500,000 deep fakes were distributed.   Of these, only 95,820 were identified. 

Detecting AI-generated content and deep fakes is challenging and reliably discerning authentic from manipulated images is proving to be increasingly difficult. Indeed, the recent surge of explicit deep fakes of Taylor Swift on X illustrate this challenge. While X blocked searches for Swift’s name, the content continued to spread. 

Of greater concern is the rise of AI-generated content on unmoderated social sites where there is little to no content moderation to begin with. These platforms prioritize user engagement over content authenticity, creating an environment where AI-generated content can spread rapidly without adequate oversight. 
 

Unmoderated Social is a breeding ground for AI-generated content

AI-generated content frequently originates on unmoderated social, where fact-checking and content moderation is scarce. Of greater concern is that unmoderated platforms are beginning to create and train their own AI models with data directly sourced from their platforms, where divisive and hateful content are allowed to remain. 

Indeed, 8kun, the home of QAnon, has announced they are launching an AI engine and Gab’s AI imagine generation model  “Gabby” is already available to all members. Gab CEO Andrew Torba has critiqued existing AI models, claiming “every single one is skewed with a liberal/globalist/talmudic/satanic worldview. What if Gab AI Inc builds a Gab.ai… that is based, has no hate speech filters and doesn’t obfuscate and distort historical and Biblical Truth?”

In contrast, moderated platforms like Youtube, Facebook, TikTok and X, implement various content moderation policies, community guidelines, and automated systems to monitor and regulate the content shared on their platforms. In the context of AI, Tiktok and YouTube require users to disclose and label AI-generated content to prevent misleading viewers, Facebook’s community standards prohibit content that has been significantly altered to mislead viewers, including deep fake videos, and X’s policy explicitly states that the company may label misleading media posts and prohibits users from sharing “synthetic, manipulated, or out-of-context media that may deceive or confuse people and lead to harm (‘misleading media’).”

Monitoring unmoderated social would  be a step in the right direction, but given the raison d'etre for these sites, this seems unlikely unless legislators were to make operations untenable without it. As we anticipate this will not take place in the U.S. in the near term, we expect the trend of deep fake creation and distribution on these sites to worsen as unmoderated social platforms continue to introduce more AI engines while moderated social steps up efforts to address AI-generated content on their platforms. Recent manifestations of this problem include:

• In late January, explicit deep fake images of Taylor Swift were shared on Telegram and 4chan, eventually reaching X and other mainstream media platforms.. While X and Meta took measures to curb the spread of the content, the content remained unchecked on unmoderated social. 

• Amidst the Israel/Hamas conflict, AI-generated content and hate speech targeting Jewish and Palestinian communities have surged, with users employing deep fakes and generative AI to create deceptive content. 

• An image of a Palestinian baby trapped under rubble went viral, but later turned out to be fake. 
• Users on Truth Social and a Hamas-linked Telegram channel shared false pictures of Prime Minister Benjamin Netanyahu covered in blood.

• Former President Donald Trump is a frequent victim of deep fakes on moderated and unmoderated social, especially as the U.S. inches closer to the 2024 presidential election. Fake AI images of Trump being arrested, and another of him with young girls on convicted sex-offender Jeremy Epstein’s plane, went viral. X attached a label to the false plane images, and user’s added context to the arrest photos through X’s “community notes” feature. 

•   President Joe Biden is also frequently targeted, especially by far-right groups that thrive on unmoderated social platforms. On X, a deep fake showing Biden in military combat uniform to imply impending U.S. involvement in the Middle East went viral. X quickly attached a label to it. And, on Telegram, users are resurfacing a manipulated video used in 2020 to bring attention to  Biden’s age. 

•   In 2019 and 2020, videos of Nancy Pelosi that were manipulated to make her appear drunk were widely circulated on moderated and unmoderated social media. Facebook labeled them and limited their distribution and YouTube removed the videos from its platform. Yet, the videos could still be found on Rumble and other unmoderated social platforms without any warning labels to users.

Individuals and companies can face reputational and financial harm 

Even without AI, the digital landscape presents substantial risks for individuals and companies. Over 43 million people in the U.S. have been reported to have been doxxed, and 95% of FTSE 100 companies were reported to have been frequently mentioned by non credible publications in the first half of 2023.

Malicious actors often exploit unmoderated social’s vulnerabilities to manipulate public perception and damage reputations. In 2023, disinformation campaigns against Target and BudLight for their LGBTQ+ initiatives lead to brand boycotts, and Etsy became the latest victim of a QAnon conspiracy theory that alleges certain online retailers are involved in trafficking children, resulting in financial losses and physical threats against Etsy’s executives. 

The rise of AI and deep fake technology escalates threats like these to unprecedented levels of sophistication and scale, posing significant challenges in addressing disinformation.

Some examples of harm AI-generated content has caused individuals or companies are as follows: 

• Disinformation campaigns can hurt a brand’s reputation and cause financial harm, and the rise of AI-generated content intensifies these risks - fabricated audio or video content can help spread false narratives about companies, eroding consumer trust and misleading investors.

• In 2019, a viral video surfaced online depicting a self-driving Tesla car crashing into a robot at a tech convention. It was later revealed to be a fake clip by users attempting to manipulate Tesla’s stock price.

• A threat to intellectual property and artistic integrity: AI-generated content shared on social media poses unique challenges to the creative industry, as demonstrated by Universal Music Group’s decision to pull music off of TikTok. They credit their decision, in part, because of the need to secure protections against AI technology’s ability to replicate voiceness and likeness.

• A viral deep faked song, “Heart on My Sleeve,” mimicking the artists Drake and TheWeeknd, received 10 million TikTok views and hundreds of thousands of listeners before it was removed. The song can still be found on moderated platforms and unmoderated social, like Rumble. 

• Promoting fake financial ventures by using deep fakes of CEO’s and executives: malicious actors exploit the influence of CEO’s and executives to mislead users into believing or acting on false information, damaging the reputation of the targeted individual and eroding trust in the company and its products or services.

• A deep fake video of Ripple CEO Brad Garlinghouse appeared on YouTube, soliciting XRP holders to send their coins for a promised doubling. 
• AI-generated videos of Elon Musk promoting a financial venture circulated on mainstream and unmoderated media to mislead users into investing in a financial scam.

As AI algorithms become increasingly adept at replicating and manipulating visual content, individuals and businesses face challenges in protecting their identity and mitigating the spread of disinformation. It’s not enough to expect mainstream platforms to remove content in a timely fashion.  Security, risk and communications teams need to be armed with AI-enriched tools to ensure they are proactively searching for potentially malicious content across moderated and unmoderated sites to ensure they identify disinformation before it goes viral. 

Pyrra’s goal is to make the internet and the world a safer place by identifying and combating hate speech, violent threats, reputation risk and brand damage across the unmoderated and alternative corners of the internet. Should your team need support in monitoring or identifying AI-generated threats or trends online, please contact us at sales@pyrratech.com.