The Deepfake Economy: Synthetic Media and the Collapse of Verification

The measured collapse

In January 2024, employees at the Hong Kong office of the engineering and design firm Arup joined a video call they had no evident reason to doubt. On the call, or so it appeared onscreen, sat the company's UK based chief financial officer alongside several other colleagues, discussing a confidential transaction that required urgent action. Every face and voice on that call was synthetic. One staff member who took part went on to carry out 15 separate wire transfers over the following days, moving a total of HK$200 million, about US$25.6 million, out of the company in a single stretch, according to reporting by CNN Business and CFO Dive.

The fraud required no theft of internal data and no breach of Arup's own systems. The attackers built their synthetic executives entirely from material the company had already made public: video and audio recorded at past conferences, webcasts, and internal meetings that had, at some point, circulated outside a closed room. An incident analysis published by the security research firm Purplesec, drawing on CFO Dive's reporting, found that everything needed to build the deepfakes was already available before the call took place. Arup did not need to be hacked. Its executives needed only to have appeared, at some point, on camera in front of an audience larger than the room they were in.

That reliance on ordinary, already public footage is what separates this fraud from the cybercrime that came before it. A stolen password or a breached database can be changed, revoked, or patched once discovered. A person's recorded face and voice cannot be revoked in the same way, and every conference talk, quarterly update, or recorded town hall a company has ever published to build trust with employees and investors also became, without anyone intending it, raw material for imitating the person who gave it.

More than a year later, the case remained open in the way large frauds increasingly do. A case study published by MIT's Cyber IR program in 2025 recorded that no perpetrator had been publicly identified or arrested, and that the stolen $25.6 million had not been recovered. Arup's own global chief information officer later said publicly that frauds of this kind happen more often than most executives realize, a framing that gave the MIT account its title and that has since circulated widely among corporate security teams still treating the case as a warning rather than an anomaly.

What the Arup case demonstrates is not a new kind of theft so much as an old one made suddenly cheap. Impersonating a chief financial officer convincingly, in real time, on video, used to require resources close to a film production, a skilled crew, actors, and time most criminal operations could not spare. Generative video and voice tools collapsed that cost to whatever it takes to find footage a company already posted online, and the fraud that followed moved more money in a single day than most cybercrime schemes manage across a full campaign.

The verification economy

Security analysts who study the Arup case treat it as more than an outlier. Case studies published separately by Adaptive Security and CoverLink Insurance describe the fraud as one of the largest publicly documented AI powered financial frauds recorded to date, and both point to the same corporate response spreading in its wake: out of band verification, the practice of confirming a high value fund transfer through a channel the caller did not control, typically a phone call placed to a number already on file rather than one supplied during the call itself.

The economics run in two directions at once. The first is direct: a single fabricated video call moved $25.6 million out of a real company's accounts in one day, wealth that, more than a year on, no recovery has restored. The second is defensive and ongoing. Firms that never lose a cent to this kind of fraud are still paying for the possibility of it, in the training, the callback protocols, and in some cases the identity verification software now built into finance departments that, before Arup, had no reason to doubt a face and voice on a screen. A cost that used to sit inside a cybersecurity budget, protecting servers and passwords, has widened to cover something older and less technical: whether the colleague a person sees and hears is the person they claim to be.

That widening has created a market as well as a cost. Identity verification and deepfake detection, barely a distinct budget line before 2023, now sit inside the same procurement conversations as anti phishing training and multi factor authentication, sold to finance and legal departments as a defense against exactly the scenario Arup lived through. None of that spending existed as its own category five years ago. It exists now because the assumption behind every video call and every voice on a phone line, that seeing and hearing someone is proof of who they are, no longer holds without a second, deliberate check.

The verification burden also lands unevenly. A multinational engineering firm the size of Arup can absorb a $25.6 million loss, hire outside counsel, and still fund a callback protocol rollout the following quarter. A smaller supplier, a family business, or a nonprofit targeted by the same technique typically has neither the reserves to absorb the loss nor the finance team to build the defense, which means the fraud that made Arup a case study elsewhere in the world could plausibly end a much smaller company outright rather than merely embarrass it.

Corporations were not the only target learning this lesson in 2024. Three weeks before Arup's own fraud was executed, the same underlying technology had already been aimed at a different kind of transfer, not of money but of votes.

Weaponized in an election

On January 21, 2024, an unknown number of New Hampshire residents received a robocall carrying a voice built to sound like President Joe Biden. The AI cloned voice urged Democratic primary voters to save their vote for November and skip the state's primary altogether, according to reporting by NBC News and the election security coverage that followed. The call landed days before voting began, timed to discourage turnout in a primary narrow enough that a small shift in participation could plausibly matter.

The response moved faster than fraud investigations usually allow. The Federal Communications Commission proposed a $6 million fine against the political consultant Steve Kramer, identified as the operator behind the scheme, and in February 2024 the commission separately ruled that voices produced by AI cloning technology used in robocalls fall under its existing prohibition on artificial and prerecorded voice messages, making calls of this kind illegal under telemarketing law written decades before generative AI existed, according to reporting by NPR. Kramer also faced criminal charges in New Hampshire over the same scheme, a federal fine and a state prosecution running side by side that made the Biden robocall one of the first cases in which an AI cloned political voice produced both regulatory and criminal consequences for the person behind it.

The regulatory response widens

The FCC's ruling did more than resolve one robocall. By reading existing telemarketing law to cover AI cloned voices, the commission set a domestic precedent without waiting for Congress to write legislation aimed specifically at generative AI, a route regulators in other jurisdictions have leaned on as well rather than starting from a blank page. A number of US states moved in the same period to restrict AI generated political content around elections, and the European Union folded transparency duties for synthetic audio, image, and video into its own AI Act, requiring that content built or materially altered by AI carry a disclosure that it was artificially generated.

Neither track closes the gap between how quickly the technology improves and how slowly law generally moves, and both were written largely in direct response to incidents like the Biden robocall and the wave of AI voice scam calls that followed it. What the response is actually defending predates any of these rules: the baseline assumption that a voter can trust what a candidate, or a candidate's opponent, is recorded saying. Democracies never had a mechanism that guaranteed this trust. They relied instead on the practical difficulty of faking a convincing recording being high enough that most citizens did not need to interrogate every clip they heard.

Generative voice cloning removed that practical difficulty for a fraction of the cost and time it once took, and it did so in a presidential election year, which is why the response arrived first through an emergency ruling and a fine rather than through the slower channel of new statute. A law written after the fact protects the next election, not the one already underway when the technology first proved what it could do. The FCC's own reasoning made this explicit: the commission did not wait for a purpose built AI statute because New Hampshire's primary was days away and existing telemarketing law, unglamorous as it was, was the only tool ready to use immediately.

The gap between the speed of the ruling and the speed of new statute is itself instructive. It took the FCC weeks to act under a law already on the books, written decades before generative AI existed, and it will likely take legislatures years to pass anything purpose built for synthetic media specifically, debated, amended, and challenged in court along the way. A political operative with access to a voice cloning tool needs neither of those timelines. The asymmetry between how fast the technology can be misused and how slowly the rules meant to govern it can be written is, on the evidence of 2024, still wide open in the technology's favor.

The two sided medium

Every medium this series has traced cuts in two directions at once, and synthetic media is no exception. The same generative tools that produced the Biden robocall also let a small campaign, an independent journalist, or an under resourced newsroom produce polished video and audio that once required a studio budget, a real democratization of production that has nothing to do with fraud. Voice cloning built for entertainment and dubbing, and video synthesis built for film and advertising, are the same underlying technology the Arup attackers and Kramer's operation repurposed.

What has concentrated, rather than spread, is the cost of trust. Verifying that a voice, a face, or a recording is genuine now requires infrastructure, callback protocols, forensic analysts of the kind Berkeley's Hany Farid supplied within a day, regulatory rulings from bodies like the FCC, that only well resourced institutions can build or access quickly. A large engineering firm can absorb a $25.6 million loss and still fund new verification software afterward. A smaller company, a local newsroom, or an individual candidate targeted by the same technology typically cannot.

The early years of this technology have therefore handed a real advantage to whichever government, campaign, or company can afford to prove what it says is true, and left everyone else carrying the older, cheaper, and increasingly insufficient assumption that seeing and hearing is believing.

What survives the collapse of verification

Put together, the two incidents describe the same structural shift from opposite ends. Arup shows that impersonation, once a labor intensive craft, is now a mass produced one: an attacker needed no inside access and no novel data, only footage a company had already made public. The Biden robocall shows the same low cost repeatability applied to political speech, cheap enough to attempt against an entire state's electorate in the same weeks as Arup's own fraud. Both cases converge on the same conclusion. The default assumption embedded in a century of telephone and video communication, that a voice or a face is sufficient proof of identity, no longer holds on its own.

That collapse runs through the same period this series has been tracing in the systems that now find, evaluate, and decide for buyers and voters alike. AI engines assembling a shortlist or summarizing a candidate face a version of the same problem the FCC and Arup's finance department each had to solve: separating a genuine claim from a fabricated one, at a volume no small team of forensic analysts can review by hand. Businesses and institutions are increasingly judged, by machines and by wary humans alike, on whether their identity and their claims can be checked quickly rather than taken on faith, the same shift in the burden of proof that moved through a Hong Kong boardroom and a New Hampshire phone line in the opening weeks of 2024.

None of this is unique to the deepfake era in principle. Forged letters, staged photographs, and impersonated officials are as old as institutions themselves, and every prior information age this series has covered produced its own crisis of fabricated authority alongside its genuine gains. What has changed is the unit cost. A convincing forged letter once took a skilled forger days; a convincing synthetic executive now takes an attacker an afternoon and a search engine, which is the arithmetic behind both the size of the Arup loss and the speed with which a single robocall reached an entire state's voters.

Whether the regulatory patchwork now forming, FCC rulings, state election laws, the EU's disclosure duties, keeps pace with a technology that keeps getting cheaper and more convincing is not yet answered, and this account does not attempt to forecast it. What can be stated is what changed in that January: verification, once assumed, became a cost every institution now has to carry, and the two cases that made it plain happened within the same month.