Novel AI systems can circumvent existing fraud defenses at scale. New defenses are necessary to protect the safe operation of financial institutions. Lenders must act fast and adapt to new technology to avoid potentially catastrophic losses.

Castles were impenetrable fortresses for centuries in medieval Europe, keeping their inhabitants safe even as attackers ravaged the surrounding lands. Storming a castle was deadly for attackers, and even the largest trebuchets could do little damage to their thick stone walls.

Castles meant that defenders always had the upper hand in military conflicts. This changed dramatically with the introduction of a new, 14th-century technology: the cannon. Early cannons were unreliable, typically putting their operators in more danger than their targets. But after a century of relentless improvement, the cannon upset the balance of power in favor of the attacker. In the 15th century, castles that were once thought impregnable had fallen, and by the turn of the century, it was clear that warfare had changed forever. In 1519, Niccolo Machiavelli wrote, “There is no wall, whatever its thickness, that artillery will not destroy in a few days.”

Today’s financial institutions have robust defenses against fraud that have enabled safe finance for decades, but AI technology poses a looming threat. For years, AI has been rapidly improving and is beginning to pose a risk to traditional lending procedures. As AI continues to improve, will it advantage the defender (lenders and factors) or the attacker (fraudsters and borrowers misrepresenting collateral)? The advantages are asymmetrical. AI makes traditional fraud easier and hugely scalable. However, AI also enables new defenses that can prevent increased fraud – if the defenders act with haste.

What is AI?
AI systems are unlike typical computer programs because they are “grown” from training data, rather than being designed and built by programmers. AIs are given vast amounts of data and learn on their own what the underlying patterns and features of the data are. AI systems have taught themselves how to read, write, do mathematical calculations, write code, and complete most tasks that can be done on a computer – all at an expert level. Modern AIs can pass the Bar, CFA, and any other exam you can think of. And they are rapidly improving. Today, AI systems can expertly reason about questions, generate and edit images, write documents, and much more. Likely, they will soon be indistinguishable from a remote worker. If you aren’t already familiar, go to chatgpt.com or gemini.google. com to try them. But keep in mind that these AI systems are basic compared to other offerings that exist today, and that AI systems in general are improving at a rapid pace.

AI Fraud Today
But what does AI enable for an attacker today? On page 16 are 2 BOLs (see set 1). Can you tell which are real or fake? Before reading on, take a close look at the documents on the next page to see if you can tell…

After looking at the images on the next page, how confident are you in your prediction? Actually, this was a trick question. They were both fake! The originals are on page 17 (see set 2). On the first set of documents, the following changes were made using AI.

• 1st image: The unit quantities are doubled.
• We instructed the AI on what numbers to put, and it altered the image, mimicking the look of the existing text.
• 2nd image: Fake carrier signature and date.
• We instructed the AI to create a signature with “light blue penstrokes that are difficult to read.” We then instructed it to add the date 6-30-24.
• For both images, the edits were generated from scratch and not copied from another source.

Please click here to continue reading. (continues on page 16 of the PDF)

Visit SFNet's Deal Table