Your Brain Wasn't Built for AI Scams
How cognitive biases and AI are reshaping fraud
Issue #11 · April 28, 2025 · 4 min read
Imagine receiving a text from your bank, it looks legit.
You hesitate.
Even with awareness of scams, you almost click.
This is the evolving challenge.
Scams are becoming so realistic that it's hard to tell what's real and what’s fake.
In 2022, Australians lost a crazy $3.1 billion to scams. While losses declined to $2 billion in 2024, the threat still stands.
The trouble is, with AI these scams are becoming more sophisticated, even more subtle.
Two weeks ago wellness influencer Arielle Lorre, discovered that a skincare brand had created an AI-generated video depicting her endorsing their products without her consent.
The video looked and somewhat sounded like her, but she had no idea it existed until followers pointed it out. She issued a cease and desist letter, and the Korean skincare brand blocked her.
Once it got attention, the brand later claimed that their marketing team “accessed the content through an AI platform without realising it resembled a real person”.
Bullshit.
This isn't just about scams. It's about our perception of reality.
Cognitive Biases
The real challenge lies within us.
Authority bias makes us trust things that seem official
Scarcity bias pushes us to act quickly when we think something is limited.
Reciprocity bias makes us feel obligated when someone does something for us.
But it goes even deeper. It's also about cognitive dissonance.
When we realise we've been duped, our beliefs clash. We thought we were savvy, but we fell for it. That discomfort can lead us to deny the truth or rationalise our actions.
And it's happening more than you think.
Last week, Sydney-based radio station CADA was called out for airing a daily hip hop show hosted by an AI-generated persona, "Thy," without disclosing its artificial nature for six months.
Amazing investigation by Stephanie Coombes on her Substack later picked up by the AFR, led to CADA confirming it was all an experiment.
Thy was in fact a real person, but she wasn’t a presenter. She was a person in the finance department with her voice and likeness replicated for the show.
Cursor, an AI-powered code editor, faced backlash when its support chatbot hallucinated a policy that didn’t exist.
The disgruntled user discovered their support team was AI pretending to be human. The founder later admitted it on a Reddit thread.
So, how do we defend ourselves?
Education is key. But not just boring text dumps.
We need real-world examples. Show people what these dupes look like. Use visuals, simulations, and stories. Make it relatable.
Platforms need to step up. If scams are happening on social media, that's where the education should be. In-app and contextual.
We can't rely on government websites like Scamwatch to solve this. Information there is too isolated. Too clinical. It underestimates how strong cognitive dissonance is.
As AI continues to evolve, we unfortunately need to be more skeptical.
Not like Dutton calling the ABC hate media, but in a way that we investigate what we see and hear.
Building a Better Defence
There needs to be a balance between education and technology.
Think about your email spam folder. That's one layer.
We need more layers.
Similar to a model I learnt about in my previous life working in infection control, we need a Swiss cheese model for scams.
Each layer has holes. But layered together, the holes are covered.
The goal isn't one perfect solution. It's multiple imperfect ones.
The Australian government introduced the Scams Prevention Framework in 2025, imposing new obligations on banks, telcos, and digital platforms to detect and prevent scams.
Financial institutions are using behavioural biometrics. Watching how you type. How you move your mouse. Spotting tiny anomalies that might mean fraud.
Each one becomes a layer.
And together, they give us a fighting chance.
If you’ve seen any interesting scams or fraudulent dupes lately, please share it in the comments.
Let’s learn from each other and build up our tasty layers of gruyere.
P.S. If you found this valuable, consider sharing it with someone who's trying to make sense of these shifts. And if you're new here, subscribe to get these insights in your inbox every week.