5,000 AI-Controlled Fake X Accounts Linked to China Disinformation Campaign

Researchers have uncovered a network of at least 5,000 fake X (formerly Twitter) accounts that appear to be controlled by AI in a disinformation campaign linked to China – and the activity appears to be heating up as the U.S. election approaches.

The X disinformation network, dubbed “Green Cicada” by researchers, “primarily engages with divisive U.S. political issues and may plausibly be staged to interfere in the upcoming presidential election.”

The network has also amplified divisive political issues in other democracies, including Australia, western Europe, India, Japan and other democratic countries.

The finding is the latest example of attempted interference in the U.S. presidential election, which just this month has seen reports of increasing activity by Iran, a warning from the FBI and CISA about potential DDoS attacks, and a hack of the campaign of former President Donald Trump that resulted in stolen documents that U.S. media has yet to publish, a sharp departure of the media’s treatment of the 2016 Hillary Clinton campaign after the Democratic National Committee was hacked by Russia-linked actors.

X Disinformation Network Linked To China

The researchers, from CyberCX, said the network is “almost certainly controlled in concert by an artificial intelligence (AI) Large Language Model (LLM) based system. … We assess that the system controlling the network is likely to be an information operation capability in a development or experimental state, based on inferred system architecture and patterns in malformed outputs. We assess the system is designed, at least in part, to ‘launder’ politically divisive narratives by rewording organic content as new posts and replies and to amplify organic divisive content on X through engagement.”

The researchers said the network is unlikely to be very effective in its current state, but they added that it “is plausible that the network operators are preparing to increase activities in the lead up to the U.S. presidential election.”

Most accounts on the network are currently dormant, but activity increased sharply in July. The network has been rectifying operational errors over time – including reducing malformed outputs – which could make its activities more effective and harder to detect over time.

The network uses a Chinese-language LLM system and links to an AI researcher affiliated with Tsinghua University and Zhipu AI, a prominent Chinese AI company. So far the actors haven’t had specific political leanings, but instead have focused on amplification of divisive content, “consistent with China’s information operation playbook,” the researchers said.

Disinformation accounts on X amplifying content across the political spectrum (CyberCX)

X’s Disinformation Controls Found Inadequate

The researchers said their findings “indicate that X has become a more permissive platform for information operations. … Our findings also indicate key gaps in X’s willingness and ability to detect inauthentic content. While we have observed X taking sporadic action against Green Cicada Network accounts during our period of monitoring, we have observed a failure to take systemic action against overtly linked accounts.

“We note that X has reversed initiatives put in place by Twitter to combat inauthentic activity, including efforts to detect, label and/or ban inauthentic accounts.”

The researchers said the network is a sign of things to come, with generative AI able to produce “a significant scale of malicious output with limited human oversight, at low cost and with low barriers to entry. It is possible that the system underpinning the network is operated by high-end consumer-grade hardware and is developed by just one individual.

“We assess that a more mature, future version of the system underlying the Green Cicada Network would be extremely difficult for parties other than X to detect.”

Source: Read More