# The dual role of AI in cybersecurity: how attackers use it and what stops them > How attackers use AI for phishing, deepfake fraud, malware and exploits, what 2025 and 2026 threat reports show, and the defence that answers each one. - URL: https://computese.com/the-dual-role-of-ai/ - Author: Duong Quan Nguyen, CEO, Computese - Published: 2024-09-25 - Updated: 2026-09-25 - Topics: Security, AI & automation ## In short - AI serves attackers and defenders alike. Threat reports from OpenAI, Google and Anthropic show attackers mostly using it to speed up familiar attacks, and a growing number running AI agents that carry out whole stages of an intrusion. - AI-written spear phishing now performs as well as phishing written by human experts, so rely on sign-in that cannot be phished: passkeys and hardware security keys, which only work on the real site's domain. - Cloned voices and faces make urgent payment and reset requests convincing, and people detect them unreliably. Verify every such request by calling back on a number from your own records. - AI shortens the time from disclosure to exploitation and has produced its first zero-day exploits. Know everything you expose to the internet, patch known-exploited flaws first and keep secrets out of code. - None of these controls needs to detect AI, which is why they keep working as the models improve. The dual role of AI in cybersecurity is that the same models defenders use to find flaws and spot scams also help attackers write convincing phishing, clone voices, build malware and find vulnerabilities faster. Threat reports from 2025 and 2026 show AI mostly speeds up familiar attacks, so the defences that work do not depend on spotting AI. This guide takes each way attackers use AI, what the organizations that observed it reported, and the control that answers it: phishing-resistant sign-in, email authentication, call-back verification, patching and exposure management. Defensive AI tools get one short section near the end, because the controls that stop these attacks are mostly not AI. For the everyday habits behind these controls, written for individuals and small offices, see [cyber security tips](https://computese.com/cyber-it-security/). ## What the dual role of AI in cybersecurity means Security has always had dual-use tools: a port scanner or an exploit framework serves a penetration tester and a criminal equally well. Large language models (LLMs) and voice and video generators are the newest ones, and they lower the skill needed to use them. Google Threat Intelligence Group (GTIG) put it this way in its [January 2025 report](https://cloud.google.com/blog/topics/threat-intelligence/adversarial-misuse-generative-ai): for skilled attackers, generative AI is a helpful framework, much like Metasploit or Cobalt Strike; for less skilled ones, it is also a learning and productivity tool. For attackers, AI plays two different parts: - **Assistant.** The attacker asks a model to write a lure, translate it, debug code or summarize stolen data. The attack is the old one, done faster and with fewer mistakes. - **Operator.** The attacker gives an AI agent tools and a goal, and the agent scans, exploits and copies data out while a human checks in occasionally. This moved from theory to documented cases in 2025. On defence, the same models read logs, review code and flag scams, and the NCSC's [2024 assessment](https://www.ncsc.gov.uk/report/impact-of-ai-on-cyber-threat) expected that to offset part of the new threat. ## What the threat reports actually show The best evidence comes from AI providers that see misuse of their own models, and from threat intelligence teams and governments that combine many sources. In date order, their reports show a clear progression. | Report | Published | What it observed | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | GTIG, Adversarial Misuse of Generative AI | January 2025 | Government-backed groups used Gemini for research, troubleshooting code and content: productivity gains, no novel capabilities | | [Anthropic threat intelligence report](https://www.anthropic.com/news/detecting-countering-misuse-aug-2025) | August 2025 | One criminal used Claude Code to automate reconnaissance, credential theft and intrusion against at least 17 organizations, then extortion | | [OpenAI, Disrupting malicious uses of AI](https://cdn.openai.com/threat-intelligence-reports/7d662b68-952f-4dfd-a2f2-fe55b041cc4a/disrupting-malicious-uses-of-ai-october-2025.pdf) | October 2025 | Over 40 networks disrupted since February 2024; actors build AI into existing workflows rather than new ones around it | | [GTIG AI Threat Tracker](https://cloud.google.com/blog/topics/threat-intelligence/threat-actor-usage-of-ai-tools) | November 2025 | The first malware families that query an LLM while they run | | [Anthropic, AI-orchestrated espionage](https://www.anthropic.com/news/disrupting-AI-espionage) | November 2025 | A campaign against about 30 targets, assessed as Chinese state-sponsored, in which AI did 80 to 90% of the work | | [GTIG AI Threat Tracker](https://cloud.google.com/blog/topics/threat-intelligence/ai-vulnerability-exploitation-initial-access) | May 2026 | The first zero-day exploit GTIG believes was developed with AI | | [Anthropic threat intelligence report](https://www.anthropic.com/threat-intelligence-report-september-2026) | September 2026 | The agent-driven operating model has spread to every class of actor Anthropic investigated | | [GTIG AI Threat Tracker](https://cloud.google.com/blog/topics/threat-intelligence/from-prompting-to-autonomy-the-evolution-of-adversarial-ai) | September 2026 | A cloud resource compromised, then an agent-enabled mass credential harvesting campaign planned and run in under six hours | Most observed misuse is still the assistant kind: OpenAI's October 2025 report found actors building AI into their existing workflows rather than new workflows around AI, and no evidence that its models gave them novel offensive capabilities. The operator kind is real and spreading. In September 2026, Anthropic reported that sophistication has stopped being a reliable signal of who is behind an operation, because individual operators with AI now sustain campaigns that would have needed many skilled people a year earlier. Google, the same month, had not yet seen fully autonomous attack pipelines used against real targets. Each provider sees only the misuse of its own models, so the patterns they share carry the most weight. Governments read the trend the same way. The NCSC assessed in January 2024 that AI would almost certainly increase the volume and heighten the impact of cyber attacks over the following two years, with the largest uplift in reconnaissance and social engineering. Its [May 2025 assessment](https://www.ncsc.gov.uk/report/impact-ai-cyber-threat-now-2027) expects existing techniques to get faster and more effective rather than new kinds of attack to appear, and judges fully automated, end-to-end advanced attacks unlikely before 2027. For a business, the attacks that hurt are the same as before (phishing, payment fraud, unpatched systems, stolen credentials), arriving in greater volume, with better quality and less warning. ## How attackers use AI for phishing and social engineering Social engineering is where the NCSC expected AI to help attackers most. It noted in 2024 that generative AI already writes convincing lures without the translation, spelling and grammar mistakes that used to give phishing away. The FBI's [December 2024 announcement](https://www.ic3.gov/PSA/2024/PSA241203) lists the same uses from its cases: believable messages written faster for more people, translation for foreign criminals, fake social media profiles in volume and AI chatbots on fraudulent websites. Quality is no longer the attacker's bottleneck. In a [November 2024 study](https://arxiv.org/abs/2412.00586) with 101 participants, fully AI-automated spear phishing emails had a 54% click-through rate, the same as emails written by human experts and far above the 12% for generic phishing. The tool also researched each target itself, and its profiles were accurate and useful in 88% of cases. Microsoft's [Digital Defense Report 2025](https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/msc/documents/presentations/CSR/Microsoft-Digital-Defense-Report-2025.pdf) cites that result when it names the biggest change in phishing over the year: scale and efficiency. Advice to "look for signs of AI" ages quickly. One scam network that OpenAI disrupted asked its model to strip em dashes from its output, apparently because people had started treating them as a sign of AI-written text. The sender address is no reliable tell either: GTIG observed an Iranian group sending phishing from the real, compromised mailboxes of earlier victims. And Anthropic's September 2026 report describes a Russian espionage actor whose AI workflows registered phishing domains, set up the sending infrastructure, sent the emails and watched for compromises, using device code phishing, which abuses a legitimate sign-in flow for cloud email. ### Defence: sign-in that cannot be phished If a phishing email is as good as a professional's, the defence has to work after someone clicks. That is what phishing-resistant authentication does. Most multi-factor authentication (MFA) leaves a gap. A one-time code from an app or a text message can be typed into a fake login page and relayed to the real one, and push prompts get approved under pressure; CISA's [fact sheet on phishing-resistant MFA](https://www.cisa.gov/sites/default/files/publications/fact-sheet-implementing-phishing-resistant-mfa-508c.pdf) describes both, along with SIM swaps against text-message codes. NIST's digital identity guidelines, [SP 800-63B-4](https://pages.nist.gov/800-63-4/sp800-63b.html), are blunt: an authenticator whose output the user enters by hand is not phishing-resistant, because nothing binds that output to the session it was meant for. Passkeys close the gap. A [passkey](https://fidoalliance.org/passkeys/) is a FIDO credential stored on a phone, a computer or a hardware security key. Under WebAuthn, the standard behind it, the secret used to sign in is chosen by the authenticated domain name of the site asking, so a lookalike site on another domain receives nothing it can replay. NIST calls this verifier name binding and accepts it as phishing resistance, and CISA calls phishing-resistant MFA the gold standard. ![A laptop is connected to two identical-looking servers. An orange passkey on a phone answers only the real server, which shows a check mark; the path to the lookalike server ends at a blocked sign.](https://computese.com/images/blog/the-dual-role-of-ai/passkey.2211a9dc96-1536.webp) *A passkey only answers the domain it was made for, so a perfect copy of the login page gets nothing it can reuse.* Microsoft's 2025 report shows why sign-in comes first: more than 97% of the identity attacks it saw were password spray or brute force, and MFA still blocks over 99% of unauthorized access attempts. Roll out phishing-resistant sign-in in the order an attacker would want the accounts: 1. **Administrators, finance staff and anyone who approves payments or resets accounts,** with passkeys or hardware security keys. 2. **Email, the identity provider and remote access for everyone else.** Where passkeys are not supported yet, CISA's fallback for small and medium businesses is an authenticator app, with number matching for push prompts. 3. **Recovery paths.** Remove text-message and voice codes from privileged accounts, and tighten password and MFA resets, which become the target once sign-in itself is hard. Synced passkeys, kept in a password manager or a platform account, are acceptable up to NIST's AAL2; the highest level, AAL3, needs a key that cannot be exported, such as a hardware security key. For teams on Microsoft 365 or Google Workspace, our [IT support service](https://computese.com/services/it-support/) covers identity and access, MFA and conditional access, and phishing-resistant sign-in where possible. ### Defence: email authentication, and what it does not stop SPF, DKIM and DMARC let receiving mail servers check that a message claiming your domain really comes from you. DMARC, published as a Standards Track IETF specification in [RFC 9989](https://www.rfc-editor.org/rfc/rfc9989.html) in May 2026, lets you state what receivers should do with mail that fails: `p=none` (no preference; useful while you read the reports), `p=quarantine` or `p=reject`. Since February 1, 2024, Gmail's [sender guidelines](https://support.google.com/a/answer/81126) have required SPF, DKIM and a DMARC record from anyone sending more than 5,000 messages a day to Gmail accounts. Check what your domain publishes: ```bash dig +short TXT example.com # SPF: look for v=spf1 dig +short TXT _dmarc.example.com # DMARC: look for v=DMARC1 and the p= value dig +short TXT selector._domainkey.example.com # DKIM: the selector comes from your mail provider ``` Know the limits. RFC 9989 says DMARC directly stops only exact-domain spoofing: not visually similar domain names, and not a trusted name placed in the display name. It also cannot flag mail from a real mailbox that has been compromised. It stops forged-from-your-domain phishing and protects your customers from it; passkeys and payment verification have to cover the rest. ## How voice and video deepfakes are used for fraud A deepfake is synthetic audio or video that imitates a real person. The FBI's December 2024 announcement describes cloned voices used to impersonate relatives in a crisis and to get into bank accounts, and generated video in real-time chats with supposed company executives, police or other authority figures. Its [December 2025 update](https://www.ic3.gov/PSA/2025/PSA251219) says actors have, since at least 2023, sent text messages and AI-generated voice messages impersonating senior US officials, moved targets to encrypted messaging apps and asked for authentication codes, passport copies or wire transfers. The rules now answering deepfakes, from the FCC's ruling on cloned-voice robocalls to EU labelling duties, are covered in [how AI is changing society](https://computese.com/how-ai-is-slowly-forming-society/). Two cases, described by the companies involved, show both outcomes: - **Arup, early 2024.** In a [February 2025 interview](https://www.weforum.org/stories/2025/02/deepfake-ai-cybercrime-arup/) published by the World Economic Forum, Arup's chief information officer, Rob Greig, discussed the case in which an employee transferred about US$25 million after a video call with what turned out to be deepfakes of senior managers. Greig said none of Arup's systems were compromised and no data was affected, and called it "technology-enhanced social engineering". Out of curiosity he later made a real-time deepfake of himself with open-source software: not particularly convincing, he said, but it took only about 45 minutes. - **LastPass, April 2024.** LastPass [published an account](https://blog.lastpass.com/posts/attempted-audio-deepfake-call-targets-lastpass-employee) of an attempt on one of its employees: calls, texts and at least one voicemail on WhatsApp carrying an audio deepfake of its CEO. The employee ignored them, because the contact came outside normal business channels and carried the forced urgency typical of social engineering, and reported it to the security team. There was no impact. Microsoft's 2025 report adds that a deepfake impersonation can lead to business email compromise, leaked information or the reset of a password or two-factor authentication on an important account, and that deepfake video can defeat the selfie liveness checks some services use to verify new users. The money is large, and AI's share of it is only starting to show. The FBI's [2025 IC3 Annual Report](https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf) recorded more than US$3.04 billion in business email compromise (BEC) losses in 2025, including more than US$30 million from BEC scams involving AI. Across all crime types, it received 22,364 complaints with AI-related information, with losses above US$893 million, and on investment fraud it notes that many victims do not realize the extent AI may be involved. ### Defence: call-back verification for payments and account changes The control that beats a perfect clone is a process the caller cannot influence: verify through a channel you choose, not the one they chose. The FBI's advice on [business email compromise](https://www.ic3.gov/PSA/2024/PSA240911) is to verify requests to change account information through a secondary channel, and its AI fraud guidance says to hang up, look the number up yourself and call it. LastPass drew the same lesson from its attempt: verify through established, approved internal channels. ![A desk phone gets an urgent call carrying a waveform while a payment waits in a paused tray. An orange line calls back a number from a printed directory and reaches a second phone with a check mark.](https://computese.com/images/blog/the-dual-role-of-ai/callback.5e26babcce-1536.webp) *A clone controls the call it makes, never the one you make back, however good it sounds.* Write it down as a rule for everyone, the chief executive included: 1. **No payment, change of bank details, password reset or MFA reset** goes ahead on the strength of an inbound call, video meeting, voice note, email or chat alone. 2. **End the conversation and call back** on a number from your own records (the supplier file, the staff directory, the contract), never a number or link supplied in the request. 3. **If a known contact writes from a new number or platform,** confirm the change through a channel you have used with them before, as the FBI advises. 4. **Payments above a threshold you set need a second approver,** who sees the call-back record before approving. 5. **Help desk resets for executives and administrators** need a call-back to the number on file or a manager's confirmation, because a convincing voice is not identity. 6. **Report every attempt** to your security contact, as the LastPass employee did, so the next target hears about it first. > [!IMPORTANT] > A familiar voice or face proves nothing on its own. Make no exception for seniority or urgency: that pressure is exactly what the attacker is applying. ### Why deepfake awareness is not enough on its own Deepfake training still helps: it teaches people that a familiar voice proves nothing. It cannot be the control. In a [2023 study in PLOS ONE](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0285333), 529 listeners identified speech deepfakes correctly only 73% of the time, in English and in Mandarin, and hearing examples beforehand improved results only slightly. The FBI lists visual glitches to look for, then adds that AI-generated content is often difficult to identify and that cloned voices can sound nearly identical to the real person. Teach the rule, not the tells: the tells keep disappearing. ## How AI helps attackers build malware In GTIG's January 2025 report, AI was mostly a coding assistant for malware: government-backed groups used Gemini for payload development and scripting help, and its safety responses blocked the more explicit requests. One group tried a public jailbreak to get Python code for a distributed denial-of-service tool and gave up when refused; our explainer on [how DDoS attacks work](https://computese.com/understanding-ddos-attacks-how-they-work/) covers what such a tool is for. Three developments since then matter more: - **Less skill needed.** Anthropic's August 2025 report describes a criminal with only basic coding skills who used Claude to develop and sell ransomware with evasion, encryption and anti-recovery features, at US$400 to US$1,200 a package, and who could not have built or fixed its core components without the model. The same report covers an extortion operation that used Claude Code to automate reconnaissance, credential harvesting and network intrusion against at least 17 organizations, with ransom demands that sometimes exceeded US$500,000. - **Malware that calls a model while it runs.** In November 2025, GTIG reported the first malware families that use an LLM during execution. PROMPTFLUX, an experimental VBScript dropper, asks the Gemini API to rewrite its own source code (one variant every hour) to evade signature-based detection. PROMPTSTEAL, used by Russia's APT28 against Ukraine and first identified in June 2025, asks an open model through the Hugging Face API for one-line Windows commands that collect system information and documents, then runs them. GTIG called it the first malware querying an LLM that it had seen in live operations. - **Agents that rebuild tools when caught.** Anthropic's September 2026 report describes an espionage actor, consistent with public reporting on Midnight Blizzard, whose AI agents watched whether security products detected its malware and rebuilt it until they did not. Anthropic's conclusion is that, at least in theory, capable adversaries can now bypass a new detection faster than defenders can develop and deploy one. The answer is a set of controls that do not depend on recognizing a particular file: - **Behaviour-based endpoint detection and response (EDR)** on every laptop and server, watching what code does (spawning shells, reading password stores, reaching new servers) rather than what the file looks like. - **Least privilege:** no local administrator rights for everyday accounts, and application control where you can manage it, so a dropper that rewrites itself still cannot install. - **Egress awareness:** PROMPTFLUX and PROMPTSTEAL both need to reach a model API at run time, so an office PC calling AI services your company does not use deserves an alert. - **Backups ransomware cannot reach,** tested by restoring from them. ## How AI speeds up vulnerability research and exploitation The NCSC expects AI-assisted vulnerability research and exploit development to be the most significant AI cyber development to 2027. The time between a vulnerability's disclosure and its exploitation has already shrunk to days, it says, and AI will almost certainly shorten it further, increasing attacks on systems that have not been updated. The reports since then show that happening: - **GTIG, May 2026:** the first zero-day exploit it believes was developed with AI, a Python script that bypassed two-factor authentication on a popular open-source, web-based system administration tool. Criminal groups had prepared it for a mass exploitation campaign that Google's discovery may have prevented. The giveaway was the code itself, full of textbook docstrings and a hallucinated CVSS score. - **Anthropic, September 2026:** a Chinese-speaking group ran an automated exploit foundry, with agents decompiling appliance firmware, forming vulnerability hypotheses, writing exploits and testing them against lab copies around the clock. One workflow against network appliances produced more than a dozen possible zero-day findings in a single month, and the group targeted roughly fifty organizations. - **The same report:** criminals affiliated with the ShinyHunters collective harvested credentials at industrial scale, including a pipeline that downloaded 1.8 million Android apps and scanned them for hard-coded secrets. In one intrusion, a single stolen developer token became full administrative control of a cloud environment in roughly three hours. Anthropic's summary is that security through obscurity no longer holds: everything connected to the internet is a potential target. ### Defence: patching and exposure management Speed on the attacker's side has to be met with knowledge on yours. Exposure management means knowing everything you have on the internet, what runs on it and what is exposed, then fixing first what is being exploited. ![An automated scanner with a radar dish sweeps a row of internet-facing devices. Most carry small shields; the orange network appliance at the end has none, and that is where the sweep stops.](https://computese.com/images/blog/the-dual-role-of-ai/exposure.837f278d85-1536.webp) *Automated scanning finds the one unpatched appliance quickly. The only good outcome is that you found it first.* 1. **Keep an inventory from the outside:** domains, subdomains, IP ranges, certificates and cloud accounts, including the test environment someone forgot. Attackers' automated reconnaissance finds these; you should find them first. 2. **Patch internet-facing systems first, and exploited flaws first of all.** VPNs, firewalls, remote access and network appliances, and web admin panels are the targets in the reports above. CISA's [Known Exploited Vulnerabilities (KEV) catalogue](https://www.cisa.gov/known-exploited-vulnerabilities) lists flaws exploited in the wild, and CISA recommends that every organization prioritize fixing them. How to combine KEV with EPSS and exposure into patch deadlines is set out in [predicting which vulnerabilities to patch first](https://computese.com/the-future-of-cybersecurity/). 3. **Take admin interfaces off the open internet,** or put them behind phishing-resistant sign-in and a VPN. The AI-built exploit GTIG found bypassed two-factor authentication on exactly such a tool. 4. **Get secrets out of code, apps and repositories,** and rotate any key that was ever committed. When harvesting is automated, a key inside a mobile app or a public repository will be found. Our guide to [secure coding best practices](https://computese.com/best-practices-for-secure-coding/) covers secret scanning and the rest of the development side. 5. **Scan on a schedule and after every major change,** and re-test what you fixed. Our [security testing service](https://computese.com/services/security-testing/) runs scheduled external scans of websites, servers and cloud accounts, triaged by engineers and ranked by exploitability with CVSS, EPSS and the CISA KEV catalogue, and closes each fix with a re-test. ## How AI is used in influence operations Influence operations are covert campaigns that use fake personas and coordinated accounts to shape opinion. The reports so far agree that AI makes the content cheaper, and none found it making the campaigns persuasive: - **Google, January 2025:** information operations actors used Gemini for research, personas, messaging, translation and ways to extend their reach. Iranian actors accounted for three quarters of that use. GTIG's September 2026 tracker still saw productivity gains, not breakthrough capabilities. - **OpenAI, October 2025:** a likely Russia-origin operation produced AI newsreader videos and posts about Africa and Ukraine. Its main X account had 172 followers and no post was retweeted more than four times, which OpenAI rated Category 2 on the Breakout Scale: several platforms, no breakout to real audiences. - **OpenAI, June 2026:** two [clusters likely originating in China](https://openai.com/index/prc-linked-influence-operations-ai-debates/) generated comments about US data center electricity prices and tariffs, and one was connected to likely inauthentic accounts that falsely claimed ChatGPT user data had been compromised. OpenAI found no meaningful breakout. - **Anthropic, September 2026:** influence sold as a service by marketing firms, fake reporters and spokespeople, impersonation of real people and institutions, and forged government documents. Most of the content drew little or no authentic engagement. Microsoft's 2025 report sees nation-state actors adopting AI rapidly for more targeted and scalable influence operations. For a company, the risk is narrower and more concrete: a false claim that you were breached, like the one in the June 2026 campaign, a cloned spokesperson, or domains and accounts that impersonate your brand. Publish your official channels and how you announce incidents, so a claim can be checked against them. Watch for lookalike domains and impersonating accounts, with the platforms' takedown routes at hand. Prepare a short statement process, so a true answer goes out before a fake one spreads. And enforce DMARC, so the fakes at least cannot send from your real domain. ## When your own AI tools become the target AI now plays a third role: target. GTIG's September 2026 tracker describes a criminal group, UNC6780 (also called TeamPCP), that has compromised open-source packages on PyPI, npm and Docker Hub since March 2026 and deliberately goes after AI coding assistants. Its DUSTMAKER stealer hides files in the project folders those assistants read (such as `.claude/`, `.vscode/` and `.cursor/`), uses configuration files there to instruct the assistant to run the attacker's scripts, and embeds prompts meant to make LLM-based security scanners refuse to analyze the code. The group also published trojanized copies of Model Context Protocol (MCP) servers. Anthropic reported criminals stealing AI API keys from victims' environments and using one of them for roughly three weeks of further attacks. The NCSC had flagged this in 2025: AI connected to company systems is a new attack surface, reachable through direct and indirect prompt injection and supply chain attacks. Treat an AI agent like a privileged user: - Give agents and assistants the least access they need, and require a person's approval before anything they do writes, pays or deletes. - Treat everything an AI reads (web pages, emails, repository files) as untrusted input that may carry instructions. - Keep AI API keys in a secrets manager, scope them narrowly and rotate them like any other credential. - Review dependencies, including MCP servers and editor plugins, before an assistant installs them. Our [AI and automation service](https://computese.com/services/ai-automation/) builds agents with tools scoped to the least access they need, approval before anything writes, and security tests for prompt injection and data leaks. ## How defenders use the same models The dual role runs the other way too, and it deserves a short look. Google says it uses its Big Sleep agent to find software vulnerabilities and CodeMender to fix them automatically. OpenAI estimated in October 2025 that ChatGPT was being used to identify scams up to three times more often than it was being used for scams. In the phishing study above, Claude 3.5 Sonnet scored well above 90% at detecting the intent of the emails, with few false positives, and caught some that people had missed. Anthropic used Claude to analyze the data from its own espionage investigation. The NCSC's warning is about who benefits: it expects a digital divide between systems that keep pace with AI-enabled threats and a large proportion that stay more vulnerable. AI detection and response tools are a subject of their own, covered in [AI in cybersecurity: how defenders use it](https://computese.com/teh-crucial-role-in-ai-in-cybersecurity/); the point here is that they add to the controls in this guide rather than replace them. ## Where to start: each AI-assisted attack and its control | AI-assisted attack | What AI changed | Control that answers it | First step | | ---------------------------- | -------------------------------------------------------- | ---------------------------------------------- | ---------------------------------------- | | Phishing and pretexts | Expert-quality, personal lures at scale | Passkeys or other phishing-resistant MFA | Administrators and finance staff first | | Mail spoofing your domain | Volume and polish | SPF, DKIM and DMARC at quarantine or reject | Check your DMARC record today | | Voice and video deepfakes | A believable executive on the line | Call-back on a number from your own records | Write the payment and reset rule | | AI-assisted malware | Code that rewrites itself or asks a model what to do | Behaviour-based EDR and least privilege | Remove everyday local admin rights | | Vulnerability exploitation | Less time from disclosure to exploit; AI-built zero-days | Exposure management, KEV-first patching | Inventory everything facing the internet | | Influence operations | Cheap personas and content | Official channels, lookalike domain watch | List your real domains and accounts | | Attacks on your own AI tools | Prompt injection, poisoned packages, stolen keys | Least-privilege agents, approvals, key hygiene | Inventory your AI keys and agents | None of these controls needs to detect AI. They work whether the phishing email, the voice on the phone or the exploit came from a person or a model, so they keep working as the models improve. Our security scanning also covers email and domain security (DMARC, SPF and DKIM, and a lookalike domain watch), and our IT support team handles the sign-in and device side. ## Key terms - **Dual use**: Technology that serves both legitimate and harmful purposes. Port scanners, exploit frameworks and AI models help defenders and attackers alike. - **Spear phishing**: Phishing aimed at a specific person or group, using details about them to make the message believable. AI can now research the target and write the message automatically. - **Deepfake**: Synthetic audio, video or images generated by AI to imitate a real person. In fraud, deepfakes impersonate executives, officials or relatives to get money, codes or access. - **Vishing**: Voice phishing: social engineering by phone call or voice message, which may use an AI-cloned voice. - **Phishing-resistant MFA**: Multi-factor authentication that a fake site cannot capture and replay, because the credential is cryptographically bound to the real site or session. FIDO2/WebAuthn passkeys and PKI smart cards qualify; typed codes do not. - **Passkey**: A FIDO credential stored on a phone, computer or hardware security key. It signs in without a password and is tied to the domain of the site it was created for. - **DMARC**: Domain-based Message Authentication, Reporting, and Conformance (RFC 9989): a DNS record that tells receiving mail servers what to do with mail that claims your domain but fails SPF and DKIM checks. - **Zero-day**: A vulnerability attackers exploit before the vendor has shipped a fix, so defenders have had no time to patch. - **Exposure management**: Keeping a current inventory of everything reachable from the internet, finding what is vulnerable or misconfigured on it, and fixing what is actually being exploited first. - **Prompt injection**: Instructions hidden in content an AI model reads, such as a web page, an email or a project file, that make the model do something its user did not intend. ## Common questions ### What is the dual role of AI in cybersecurity? It means AI works for both sides. Attackers use it to write phishing, clone voices, develop malware and research vulnerabilities; defenders use the same models to find flaws, triage alerts and spot scams. Reports from 2025 and 2026 show attackers mostly using AI to speed up familiar attacks, with a growing number running AI agents that carry out whole stages of an intrusion. ### How do hackers use AI? Most often to write and translate phishing and scam messages, research targets, and debug or disguise code. More capable groups run AI agents for reconnaissance, exploitation and data theft, and some malware now asks an AI model for new code while it runs. Criminals also use cloned voices and video to impersonate executives and officials. ### Can AI write malware on its own? It can write working components: Anthropic documented a criminal with basic coding skills selling ransomware built with Claude, and Google reported in May 2026 a zero-day exploit it believes was developed with AI. As of September 2026, Google had not seen fully autonomous attack pipelines used against real targets, and the UK NCSC judges fully automated end-to-end advanced attacks unlikely before 2027. ### How can you tell if a call or video is a deepfake? Often you cannot. In a 2023 study, listeners identified speech deepfakes only 73% of the time, and the FBI says AI-generated content is often difficult to identify. Instead of judging the voice, end the call and call back on a number from your own records before acting on any request for money, codes or access. ### Does MFA stop AI-powered phishing? Only partly. A code from an app or a text message can be typed into a fake page and relayed to the real site, so NIST does not count it as phishing-resistant. Passkeys and hardware security keys are bound to the real site's domain, so they still protect the account after a convincing phishing email gets a click. ### Is AI making cyber attacks more common? The UK's National Cyber Security Centre expects it to. It assessed in 2024 that AI would almost certainly increase the volume and impact of attacks, and in 2025 that it will increase their frequency and intensity to 2027, mainly by making existing techniques faster and more effective rather than creating new ones. ## Sources 1. [Adversarial Misuse of Generative AI](https://cloud.google.com/blog/topics/threat-intelligence/adversarial-misuse-generative-ai), Google Threat Intelligence Group 2. [The near-term impact of AI on the cyber threat](https://www.ncsc.gov.uk/report/impact-of-ai-on-cyber-threat), UK National Cyber Security Centre 3. [Detecting and countering misuse of AI: August 2025](https://www.anthropic.com/news/detecting-countering-misuse-aug-2025), Anthropic 4. [Disrupting malicious uses of AI: October 2025](https://cdn.openai.com/threat-intelligence-reports/7d662b68-952f-4dfd-a2f2-fe55b041cc4a/disrupting-malicious-uses-of-ai-october-2025.pdf), OpenAI 5. [GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools](https://cloud.google.com/blog/topics/threat-intelligence/threat-actor-usage-of-ai-tools), Google Threat Intelligence Group 6. [Disrupting the first reported AI-orchestrated cyber espionage campaign](https://www.anthropic.com/news/disrupting-AI-espionage), Anthropic 7. [GTIG AI Threat Tracker: Adversaries Leverage AI for Vulnerability Exploitation, Augmented Operations, and Initial Access](https://cloud.google.com/blog/topics/threat-intelligence/ai-vulnerability-exploitation-initial-access), Google Threat Intelligence Group 8. [Detecting and countering misuse of AI: September 2026](https://www.anthropic.com/threat-intelligence-report-september-2026), Anthropic 9. [GTIG AI Threat Tracker: From Prompting to Autonomy, The Evolution of Adversarial AI](https://cloud.google.com/blog/topics/threat-intelligence/from-prompting-to-autonomy-the-evolution-of-adversarial-ai), Google Threat Intelligence Group 10. [Impact of AI on cyber threat from now to 2027](https://www.ncsc.gov.uk/report/impact-ai-cyber-threat-now-2027), UK National Cyber Security Centre 11. [Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud (I-120324-PSA)](https://www.ic3.gov/PSA/2024/PSA241203), FBI Internet Crime Complaint Center 12. [Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects](https://arxiv.org/abs/2412.00586), arXiv (Heiding, Lermen, Kao, Schneier, Vishwanath) 13. [Microsoft Digital Defense Report 2025](https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/msc/documents/presentations/CSR/Microsoft-Digital-Defense-Report-2025.pdf), Microsoft 14. [Implementing Phishing-Resistant MFA](https://www.cisa.gov/sites/default/files/publications/fact-sheet-implementing-phishing-resistant-mfa-508c.pdf), CISA 15. [SP 800-63B-4: Authentication and Authenticator Management](https://pages.nist.gov/800-63-4/sp800-63b.html), NIST 16. [Passkeys](https://fidoalliance.org/passkeys/), FIDO Alliance 17. [RFC 9989: Domain-Based Message Authentication, Reporting, and Conformance (DMARC)](https://www.rfc-editor.org/rfc/rfc9989.html), IETF 18. [Email sender guidelines](https://support.google.com/a/answer/81126), Google (Gmail Help) 19. [Senior U.S. Officials Continue to be Impersonated in Malicious Messaging Campaign (I-121925-PSA)](https://www.ic3.gov/PSA/2025/PSA251219), FBI Internet Crime Complaint Center 20. [Cybercrime: Lessons learned from a $25m deepfake attack](https://www.weforum.org/stories/2025/02/deepfake-ai-cybercrime-arup/), World Economic Forum 21. [Attempted Audio Deepfake Call Targets LastPass Employee](https://blog.lastpass.com/posts/attempted-audio-deepfake-call-targets-lastpass-employee), LastPass 22. [2025 IC3 Annual Report](https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf), FBI Internet Crime Complaint Center 23. [Business Email Compromise: The $55 Billion Scam (I-091124-PSA)](https://www.ic3.gov/PSA/2024/PSA240911), FBI Internet Crime Complaint Center 24. [Warning: Humans cannot reliably detect speech deepfakes](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0285333), PLOS ONE (Mai, Bray, Davies, Griffin) 25. [Known Exploited Vulnerabilities](https://www.cisa.gov/known-exploited-vulnerabilities), CISA 26. [PRC-linked influence operations are targeting AI debates in the US](https://openai.com/index/prc-linked-influence-operations-ai-debates/), OpenAI