AI Hallucinations Help Hackers Spawn Phishing Sites, Unit 42 Warns

Security experts from the research team Unit 42, a division of Palo Alto Networks, discovered that large language models are systematically hallucinating web addresses that do not exist, a phenomenon they named "phantom squatting." Cybercriminals have started harvesting these hallucinated domains from AI replies, registering them beforehand, and waiting for unsuspecting users to be redirected there by the AI itself.

To evaluate the scale of this mess, the researchers analyzed 913 global brands across finance, tech, and healthcare, firing over 685,000 queries at two prominent LLMs. The models hallucinated over 2.1 million unique URLs, out of which more than 13,000 were already flagged as active malware servers. Essentially, generative AI is currently acting as a free marketing agency for cybercriminals by recommending their malicious infrastructure to users.

This attack style evolves from "slop squatting," where AI hallucinated non-existent software package names, which hackers then uploaded to registries like npm. Now, the trick has moved directly to web infrastructure, turning nonexistent corporate portals into trapdoors. Because these LLMs behave predictably, different models under various settings consistently hallucinate the exact same fake domains for the same brands.

In a recent case involving the Montana Empire phishing kit, researchers observed a domain resembling a national postal service being hallucinated even at low creativity settings. Just 23 days after the AI started spitting out this fake address, a real attacker registered it and set up a credit card harvesting site. To make the irony complete, the hacker used an AI programming assistant to build the phishing kit, creating a beautiful circle of life where AI builds the weapon and AI brings the victims.

The threat skyrockets with autonomous AI agents that browse the web, fetch dependencies, and call APIs without human oversight. When an agent clicks a freshly registered phantom domain, it bypasses traditional security filters because the new domain is completely clean, with no prior bad reputation.

Fortunately, this predictability goes both ways, allowing defenders to pre-register or monitor these phantom domains. The team at Unit 42 was able to predict malicious registrations up to 51 days before hackers actually bought them.

It seems the tech industry has built a perfect perpetual motion machine of cybercrime, where artificial intelligence writes the malware, hosts the traps, and then politely guides humans into them. The internet will surely debate whether this is the ultimate peak of automation or just another reason to pull the plug on the servers.

Source: Palo Alto Networks Unit 42