The dual role of AI in cybersecurity: how attackers use it and what stops them

What the dual role of AI in cybersecurity means

Security has always had dual-use tools: a port scanner or an exploit framework serves a penetration tester and a criminal equally well. Large language models (LLMs) and voice and video generators are the newest ones, and they lower the skill needed to use them. Google Threat Intelligence Group (GTIG) put it this way in its January 2025 report: for skilled attackers, generative AI is a helpful framework, much like Metasploit or Cobalt Strike; for less skilled ones, it is also a learning and productivity tool.

For attackers, AI plays two different parts:

• Assistant. The attacker asks a model to write a lure, translate it, debug code or summarize stolen data. The attack is the old one, done faster and with fewer mistakes.
• Operator. The attacker gives an AI agent tools and a goal, and the agent scans, exploits and copies data out while a human checks in occasionally. This moved from theory to documented cases in 2025.

On defence, the same models read logs, review code and flag scams, and the NCSC's 2024 assessment expected that to offset part of the new threat.

What the threat reports actually show

The best evidence comes from AI providers that see misuse of their own models, and from threat intelligence teams and governments that combine many sources. In date order, their reports show a clear progression.

Most observed misuse is still the assistant kind: OpenAI's October 2025 report found actors building AI into their existing workflows rather than new workflows around AI, and no evidence that its models gave them novel offensive capabilities. The operator kind is real and spreading. In September 2026, Anthropic reported that sophistication has stopped being a reliable signal of who is behind an operation, because individual operators with AI now sustain campaigns that would have needed many skilled people a year earlier. Google, the same month, had not yet seen fully autonomous attack pipelines used against real targets. Each provider sees only the misuse of its own models, so the patterns they share carry the most weight.

Governments read the trend the same way. The NCSC assessed in January 2024 that AI would almost certainly increase the volume and heighten the impact of cyber attacks over the following two years, with the largest uplift in reconnaissance and social engineering. Its May 2025 assessment expects existing techniques to get faster and more effective rather than new kinds of attack to appear, and judges fully automated, end-to-end advanced attacks unlikely before 2027. For a business, the attacks that hurt are the same as before (phishing, payment fraud, unpatched systems, stolen credentials), arriving in greater volume, with better quality and less warning.

How attackers use AI for phishing and social engineering

Social engineering is where the NCSC expected AI to help attackers most. It noted in 2024 that generative AI already writes convincing lures without the translation, spelling and grammar mistakes that used to give phishing away. The FBI's December 2024 announcement lists the same uses from its cases: believable messages written faster for more people, translation for foreign criminals, fake social media profiles in volume and AI chatbots on fraudulent websites.

Quality is no longer the attacker's bottleneck. In a November 2024 study with 101 participants, fully AI-automated spear phishing emails had a 54% click-through rate, the same as emails written by human experts and far above the 12% for generic phishing. The tool also researched each target itself, and its profiles were accurate and useful in 88% of cases. Microsoft's Digital Defense Report 2025 cites that result when it names the biggest change in phishing over the year: scale and efficiency.

Advice to "look for signs of AI" ages quickly. One scam network that OpenAI disrupted asked its model to strip em dashes from its output, apparently because people had started treating them as a sign of AI-written text. The sender address is no reliable tell either: GTIG observed an Iranian group sending phishing from the real, compromised mailboxes of earlier victims. And Anthropic's September 2026 report describes a Russian espionage actor whose AI workflows registered phishing domains, set up the sending infrastructure, sent the emails and watched for compromises, using device code phishing, which abuses a legitimate sign-in flow for cloud email.

Defence: sign-in that cannot be phished

If a phishing email is as good as a professional's, the defence has to work after someone clicks. That is what phishing-resistant authentication does.

Most multi-factor authentication (MFA) leaves a gap. A one-time code from an app or a text message can be typed into a fake login page and relayed to the real one, and push prompts get approved under pressure; CISA's fact sheet on phishing-resistant MFA describes both, along with SIM swaps against text-message codes. NIST's digital identity guidelines, SP 800-63B-4, are blunt: an authenticator whose output the user enters by hand is not phishing-resistant, because nothing binds that output to the session it was meant for.

Passkeys close the gap. A passkey is a FIDO credential stored on a phone, a computer or a hardware security key. Under WebAuthn, the standard behind it, the secret used to sign in is chosen by the authenticated domain name of the site asking, so a lookalike site on another domain receives nothing it can replay. NIST calls this verifier name binding and accepts it as phishing resistance, and CISA calls phishing-resistant MFA the gold standard.

Microsoft's 2025 report shows why sign-in comes first: more than 97% of the identity attacks it saw were password spray or brute force, and MFA still blocks over 99% of unauthorized access attempts. Roll out phishing-resistant sign-in in the order an attacker would want the accounts:

1. Administrators, finance staff and anyone who approves payments or resets accounts, with passkeys or hardware security keys.
2. Email, the identity provider and remote access for everyone else. Where passkeys are not supported yet, CISA's fallback for small and medium businesses is an authenticator app, with number matching for push prompts.
3. Recovery paths. Remove text-message and voice codes from privileged accounts, and tighten password and MFA resets, which become the target once sign-in itself is hard.

Synced passkeys, kept in a password manager or a platform account, are acceptable up to NIST's AAL2; the highest level, AAL3, needs a key that cannot be exported, such as a hardware security key. For teams on Microsoft 365 or Google Workspace, our IT support service covers identity and access, MFA and conditional access, and phishing-resistant sign-in where possible.

Defence: email authentication, and what it does not stop

SPF, DKIM and DMARC let receiving mail servers check that a message claiming your domain really comes from you. DMARC, published as a Standards Track IETF specification in RFC 9989 in May 2026, lets you state what receivers should do with mail that fails: p=none (no preference; useful while you read the reports), p=quarantine or p=reject. Since February 1, 2024, Gmail's sender guidelines have required SPF, DKIM and a DMARC record from anyone sending more than 5,000 messages a day to Gmail accounts. Check what your domain publishes:

dig +short TXT example.com                         # SPF: look for v=spf1
dig +short TXT _dmarc.example.com                  # DMARC: look for v=DMARC1 and the p= value
dig +short TXT selector._domainkey.example.com     # DKIM: the selector comes from your mail provider

Know the limits. RFC 9989 says DMARC directly stops only exact-domain spoofing: not visually similar domain names, and not a trusted name placed in the display name. It also cannot flag mail from a real mailbox that has been compromised. It stops forged-from-your-domain phishing and protects your customers from it; passkeys and payment verification have to cover the rest.

How voice and video deepfakes are used for fraud

A deepfake is synthetic audio or video that imitates a real person. The FBI's December 2024 announcement describes cloned voices used to impersonate relatives in a crisis and to get into bank accounts, and generated video in real-time chats with supposed company executives, police or other authority figures. Its December 2025 update says actors have, since at least 2023, sent text messages and AI-generated voice messages impersonating senior US officials, moved targets to encrypted messaging apps and asked for authentication codes, passport copies or wire transfers. The rules now answering deepfakes, from the FCC's ruling on cloned-voice robocalls to EU labelling duties, are covered in how AI is changing society.

Two cases, described by the companies involved, show both outcomes:

• Arup, early 2024. In a February 2025 interview published by the World Economic Forum, Arup's chief information officer, Rob Greig, discussed the case in which an employee transferred about US$25 million after a video call with what turned out to be deepfakes of senior managers. Greig said none of Arup's systems were compromised and no data was affected, and called it "technology-enhanced social engineering". Out of curiosity he later made a real-time deepfake of himself with open-source software: not particularly convincing, he said, but it took only about 45 minutes.
• LastPass, April 2024. LastPass published an account of an attempt on one of its employees: calls, texts and at least one voicemail on WhatsApp carrying an audio deepfake of its CEO. The employee ignored them, because the contact came outside normal business channels and carried the forced urgency typical of social engineering, and reported it to the security team. There was no impact.

Microsoft's 2025 report adds that a deepfake impersonation can lead to business email compromise, leaked information or the reset of a password or two-factor authentication on an important account, and that deepfake video can defeat the selfie liveness checks some services use to verify new users.

The money is large, and AI's share of it is only starting to show. The FBI's 2025 IC3 Annual Report recorded more than US$3.04 billion in business email compromise (BEC) losses in 2025, including more than US$30 million from BEC scams involving AI. Across all crime types, it received 22,364 complaints with AI-related information, with losses above US$893 million, and on investment fraud it notes that many victims do not realize the extent AI may be involved.

Defence: call-back verification for payments and account changes

The control that beats a perfect clone is a process the caller cannot influence: verify through a channel you choose, not the one they chose. The FBI's advice on business email compromise is to verify requests to change account information through a secondary channel, and its AI fraud guidance says to hang up, look the number up yourself and call it. LastPass drew the same lesson from its attempt: verify through established, approved internal channels.

Write it down as a rule for everyone, the chief executive included:

1. No payment, change of bank details, password reset or MFA reset goes ahead on the strength of an inbound call, video meeting, voice note, email or chat alone.
2. End the conversation and call back on a number from your own records (the supplier file, the staff directory, the contract), never a number or link supplied in the request.
3. If a known contact writes from a new number or platform, confirm the change through a channel you have used with them before, as the FBI advises.
4. Payments above a threshold you set need a second approver, who sees the call-back record before approving.
5. Help desk resets for executives and administrators need a call-back to the number on file or a manager's confirmation, because a convincing voice is not identity.
6. Report every attempt to your security contact, as the LastPass employee did, so the next target hears about it first.

Why deepfake awareness is not enough on its own

Deepfake training still helps: it teaches people that a familiar voice proves nothing. It cannot be the control. In a 2023 study in PLOS ONE, 529 listeners identified speech deepfakes correctly only 73% of the time, in English and in Mandarin, and hearing examples beforehand improved results only slightly. The FBI lists visual glitches to look for, then adds that AI-generated content is often difficult to identify and that cloned voices can sound nearly identical to the real person. Teach the rule, not the tells: the tells keep disappearing.

How AI helps attackers build malware

In GTIG's January 2025 report, AI was mostly a coding assistant for malware: government-backed groups used Gemini for payload development and scripting help, and its safety responses blocked the more explicit requests. One group tried a public jailbreak to get Python code for a distributed denial-of-service tool and gave up when refused; our explainer on how DDoS attacks work covers what such a tool is for. Three developments since then matter more:

• Less skill needed. Anthropic's August 2025 report describes a criminal with only basic coding skills who used Claude to develop and sell ransomware with evasion, encryption and anti-recovery features, at US$400 to US$1,200 a package, and who could not have built or fixed its core components without the model. The same report covers an extortion operation that used Claude Code to automate reconnaissance, credential harvesting and network intrusion against at least 17 organizations, with ransom demands that sometimes exceeded US$500,000.
• Malware that calls a model while it runs. In November 2025, GTIG reported the first malware families that use an LLM during execution. PROMPTFLUX, an experimental VBScript dropper, asks the Gemini API to rewrite its own source code (one variant every hour) to evade signature-based detection. PROMPTSTEAL, used by Russia's APT28 against Ukraine and first identified in June 2025, asks an open model through the Hugging Face API for one-line Windows commands that collect system information and documents, then runs them. GTIG called it the first malware querying an LLM that it had seen in live operations.
• Agents that rebuild tools when caught. Anthropic's September 2026 report describes an espionage actor, consistent with public reporting on Midnight Blizzard, whose AI agents watched whether security products detected its malware and rebuilt it until they did not. Anthropic's conclusion is that, at least in theory, capable adversaries can now bypass a new detection faster than defenders can develop and deploy one.

The answer is a set of controls that do not depend on recognizing a particular file:

• Behaviour-based endpoint detection and response (EDR) on every laptop and server, watching what code does (spawning shells, reading password stores, reaching new servers) rather than what the file looks like.
• Least privilege: no local administrator rights for everyday accounts, and application control where you can manage it, so a dropper that rewrites itself still cannot install.
• Egress awareness: PROMPTFLUX and PROMPTSTEAL both need to reach a model API at run time, so an office PC calling AI services your company does not use deserves an alert.
• Backups ransomware cannot reach, tested by restoring from them.

How AI speeds up vulnerability research and exploitation

The NCSC expects AI-assisted vulnerability research and exploit development to be the most significant AI cyber development to 2027. The time between a vulnerability's disclosure and its exploitation has already shrunk to days, it says, and AI will almost certainly shorten it further, increasing attacks on systems that have not been updated. The reports since then show that happening:

• GTIG, May 2026: the first zero-day exploit it believes was developed with AI, a Python script that bypassed two-factor authentication on a popular open-source, web-based system administration tool. Criminal groups had prepared it for a mass exploitation campaign that Google's discovery may have prevented. The giveaway was the code itself, full of textbook docstrings and a hallucinated CVSS score.
• Anthropic, September 2026: a Chinese-speaking group ran an automated exploit foundry, with agents decompiling appliance firmware, forming vulnerability hypotheses, writing exploits and testing them against lab copies around the clock. One workflow against network appliances produced more than a dozen possible zero-day findings in a single month, and the group targeted roughly fifty organizations.
• The same report: criminals affiliated with the ShinyHunters collective harvested credentials at industrial scale, including a pipeline that downloaded 1.8 million Android apps and scanned them for hard-coded secrets. In one intrusion, a single stolen developer token became full administrative control of a cloud environment in roughly three hours. Anthropic's summary is that security through obscurity no longer holds: everything connected to the internet is a potential target.

Defence: patching and exposure management

Speed on the attacker's side has to be met with knowledge on yours. Exposure management means knowing everything you have on the internet, what runs on it and what is exposed, then fixing first what is being exploited.

1. Keep an inventory from the outside: domains, subdomains, IP ranges, certificates and cloud accounts, including the test environment someone forgot. Attackers' automated reconnaissance finds these; you should find them first.
2. Patch internet-facing systems first, and exploited flaws first of all. VPNs, firewalls, remote access and network appliances, and web admin panels are the targets in the reports above. CISA's Known Exploited Vulnerabilities (KEV) catalogue lists flaws exploited in the wild, and CISA recommends that every organization prioritize fixing them. How to combine KEV with EPSS and exposure into patch deadlines is set out in predicting which vulnerabilities to patch first.
3. Take admin interfaces off the open internet, or put them behind phishing-resistant sign-in and a VPN. The AI-built exploit GTIG found bypassed two-factor authentication on exactly such a tool.
4. Get secrets out of code, apps and repositories, and rotate any key that was ever committed. When harvesting is automated, a key inside a mobile app or a public repository will be found. Our guide to secure coding best practices covers secret scanning and the rest of the development side.
5. Scan on a schedule and after every major change, and re-test what you fixed.

Our security testing service runs scheduled external scans of websites, servers and cloud accounts, triaged by engineers and ranked by exploitability with CVSS, EPSS and the CISA KEV catalogue, and closes each fix with a re-test.

How AI is used in influence operations

Influence operations are covert campaigns that use fake personas and coordinated accounts to shape opinion. The reports so far agree that AI makes the content cheaper, and none found it making the campaigns persuasive:

• Google, January 2025: information operations actors used Gemini for research, personas, messaging, translation and ways to extend their reach. Iranian actors accounted for three quarters of that use. GTIG's September 2026 tracker still saw productivity gains, not breakthrough capabilities.
• OpenAI, October 2025: a likely Russia-origin operation produced AI newsreader videos and posts about Africa and Ukraine. Its main X account had 172 followers and no post was retweeted more than four times, which OpenAI rated Category 2 on the Breakout Scale: several platforms, no breakout to real audiences.
• OpenAI, June 2026: two clusters likely originating in China generated comments about US data center electricity prices and tariffs, and one was connected to likely inauthentic accounts that falsely claimed ChatGPT user data had been compromised. OpenAI found no meaningful breakout.
• Anthropic, September 2026: influence sold as a service by marketing firms, fake reporters and spokespeople, impersonation of real people and institutions, and forged government documents. Most of the content drew little or no authentic engagement.

Microsoft's 2025 report sees nation-state actors adopting AI rapidly for more targeted and scalable influence operations. For a company, the risk is narrower and more concrete: a false claim that you were breached, like the one in the June 2026 campaign, a cloned spokesperson, or domains and accounts that impersonate your brand. Publish your official channels and how you announce incidents, so a claim can be checked against them. Watch for lookalike domains and impersonating accounts, with the platforms' takedown routes at hand. Prepare a short statement process, so a true answer goes out before a fake one spreads. And enforce DMARC, so the fakes at least cannot send from your real domain.

When your own AI tools become the target

AI now plays a third role: target. GTIG's September 2026 tracker describes a criminal group, UNC6780 (also called TeamPCP), that has compromised open-source packages on PyPI, npm and Docker Hub since March 2026 and deliberately goes after AI coding assistants. Its DUSTMAKER stealer hides files in the project folders those assistants read (such as .claude/, .vscode/ and .cursor/), uses configuration files there to instruct the assistant to run the attacker's scripts, and embeds prompts meant to make LLM-based security scanners refuse to analyze the code. The group also published trojanized copies of Model Context Protocol (MCP) servers. Anthropic reported criminals stealing AI API keys from victims' environments and using one of them for roughly three weeks of further attacks. The NCSC had flagged this in 2025: AI connected to company systems is a new attack surface, reachable through direct and indirect prompt injection and supply chain attacks.

Treat an AI agent like a privileged user:

• Give agents and assistants the least access they need, and require a person's approval before anything they do writes, pays or deletes.
• Treat everything an AI reads (web pages, emails, repository files) as untrusted input that may carry instructions.
• Keep AI API keys in a secrets manager, scope them narrowly and rotate them like any other credential.
• Review dependencies, including MCP servers and editor plugins, before an assistant installs them.

Our AI and automation service builds agents with tools scoped to the least access they need, approval before anything writes, and security tests for prompt injection and data leaks.

How defenders use the same models

The dual role runs the other way too, and it deserves a short look. Google says it uses its Big Sleep agent to find software vulnerabilities and CodeMender to fix them automatically. OpenAI estimated in October 2025 that ChatGPT was being used to identify scams up to three times more often than it was being used for scams. In the phishing study above, Claude 3.5 Sonnet scored well above 90% at detecting the intent of the emails, with few false positives, and caught some that people had missed. Anthropic used Claude to analyze the data from its own espionage investigation.

The NCSC's warning is about who benefits: it expects a digital divide between systems that keep pace with AI-enabled threats and a large proportion that stay more vulnerable. AI detection and response tools are a subject of their own, covered in AI in cybersecurity: how defenders use it; the point here is that they add to the controls in this guide rather than replace them.

Where to start: each AI-assisted attack and its control

None of these controls needs to detect AI. They work whether the phishing email, the voice on the phone or the exploit came from a person or a model, so they keep working as the models improve. Our security scanning also covers email and domain security (DMARC, SPF and DKIM, and a lookalike domain watch), and our IT support team handles the sign-in and device side.