AI Security · A field guide

The models change every quarter. The principles that protect them do not.

Fifteen principles that held before the current wave of AI and will hold after it, set against the threats that matter now and the frameworks that name them.

  • 15 principles
  • 15 threats
  • 10 frameworks
  • 12 questions

Last reviewed 31 October 2026

01 · Premise

Old disciplines, new load

An AI system is still software. It has inputs, outputs, users, data stores and dependencies, and every lesson about least privilege, separation of duties, defence in depth and audit still applies.

What is new is one component in the middle: an interpreter that follows instructions written in ordinary language and cannot reliably tell whose language it is reading. Give that interpreter tools and memory, and text becomes action. That single property explains most of what follows on this page: prompt injection, hijacked agents, poisoned tools and leaked data.

So this guide starts with principles rather than products. Products change every few months. A principle earns its place here only if it would have been good advice before large language models existed, and will still be good advice when today’s models are long forgotten.

02 · Principles

Fifteen principles

Each one answers several threats. The links at the foot of every card jump to them.

P01

Keep instructions and data apart, and assume the model can’t

Prompt injection is not a bug in one product. It is what happens when commands and content share a channel.

A language model reads a single stream of text. Anything that reaches that stream, whether an email, a web page, a PDF, a calendar invite or a tool result, can be read as an instruction. Filters and detectors lower the hit rate, but none of them closes the gap, because the model has no reliable way to tell your words from an attacker’s. The control that lasts sits outside the model: decide what the system may do before untrusted text arrives, and make sure that text cannot change the decision.

In practice

  • Fix the plan, the tools and the recipients before the system reads untrusted content.
  • Split the part that plans from the part that reads untrusted data, so injected text never chooses the actions. Then track which values came from untrusted text, because an injected email address can still steer an allowed action. Simon Willison’s dual-LLM pattern (2023) and Google DeepMind’s CaMeL (2025) are research designs that do this. Both cut the risk; neither removes it.
  • Mark untrusted content with delimiters or encoding as a hint to the model, never as the boundary you rely on.

Further reading CaMeL: Defeating Prompt Injections by Design (2025)

Answers

P02

Grant the least agency

Every tool an agent holds is a tool an attacker can borrow.

Least privilege is fifty years old and still the best answer. What changed is the holder: an agent acts on text it reads, so each extra tool, scope or permission widens what a single malicious sentence can do. OWASP calls the failure excessive agency: too much functionality, too many permissions, too much autonomy.

In practice

  • Give each task its own allow-list of tools instead of one list for the whole product.
  • Default to read-only. Make write, send and delete separate tools that are switched on deliberately.
  • Cap the number of steps an agent can take before it has to report back.

Answers

P03

Don’t give one unsupervised session all three legs

Private data, untrusted content and a way to send data out. Remove one.

Simon Willison named it the lethal trifecta: an agent that can read your private data, also reads content an attacker controls, and can also communicate externally can be talked into sending that data away. Meta’s Agents Rule of Two turns a broader version into a design rule: an agent should combine no more than two of three properties, namely processing untrustworthy input, accessing sensitive systems or private data, and changing state or communicating externally. When a task needs all three, the agent should not act unsupervised. Take away any one leg and that data-theft route closes; other harms need the other principles. It is an old rule in new form: information-flow controls have kept secret data off untrusted channels since the 1970s.

In practice

  • Sessions that read private data and untrusted input get no outbound channel.
  • Treat rendered images, link previews and URL fetches as outbound channels, because they are. Data can leave in a query string.
  • Where all three must meet, put a person in front of every outbound action.

Further reading Simon Willison, The lethal trifecta (June 2025) · Meta, Agents Rule of Two (October 2025)

Answers

P04

Treat model output as untrusted input

Whoever shaped the input shaped the output.

Output from a model is only as trustworthy as everything that went into its context. Hand it straight to a browser, a shell, a SQL engine or another agent and the old injection classes come back: cross-site scripting, command injection, SQL injection. Add a newer one, where one agent’s output becomes another agent’s instructions.

In practice

  • Encode output for where it is going, exactly as you would user input.
  • Run generated code in a sandbox with no network access and no secrets.
  • Validate structured output against a schema, and reject what does not fit.

Answers

P05

Put a human on irreversible and outward actions

No prompt can undo a payment, a deletion or a sent email.

Speed is the point of automation and also its danger. Deleting data, moving money, sending messages, publishing and changing permissions are one-way doors. A human checkpoint on those few actions costs seconds and stops most of a whole class of failure, the agent that does exactly what an attacker or a misunderstanding told it to, provided the person reads what they approve.

In practice

  • Sort actions by reversibility, and ask for approval on the one-way ones only, so approvals stay meaningful.
  • Show the real parameters (recipient, amount, file, target) in the approval, not the agent’s summary of them.
  • Watch for approval fatigue. A prompt that appears fifty times a day becomes a button people press without reading.

Answers

P06

Enforce access at the data layer, not in the prompt

“Only answer about documents this user may see” is a wish, not a control.

If retrieval hands a document to the model, assume the model can repeat it. System prompts leak, instructions get overridden, and a model has no idea who is asking. Access decisions belong where they always have: in the system that returns the data, made against the identity of the person or agent asking.

In practice

  • Filter retrieval by the caller’s permissions before ranking, not after generation.
  • Keep tenants in separate indexes, or apply filters the application enforces on every query.
  • Treat the system prompt as public. Keep secrets, keys and internal rules out of it.

Answers

P07

Know where everything came from

Models, weights, datasets, packages and tool servers all arrive from somewhere.

The AI supply chain is long and young. Model files can run code when they are loaded. Public datasets can be seeded with poison. Tool servers can change after you approve them, and coding assistants sometimes recommend packages that do not exist, which an attacker can then register, as researchers have shown. Provenance is the old answer: know what you run, where it came from, and whether it has changed.

In practice

  • Prefer formats that cannot execute on load, such as safetensors, over pickle-based files (Python’s serialisation format, which can run code), and don’t let loaders run code shipped with a model unless you have reviewed it.
  • Pin versions and hashes for models, packages and tool servers, and keep an AI bill of materials (AI-BOM).
  • Check that a suggested package exists, is the one you meant and has a history before you install it.

Answers

P08

Give agents their own identity

An agent borrowing your session can do everything you can, and the logs can’t tell you apart.

Agents are non-human identities with real access. When they act under a person’s credentials, every mistake and every compromise inherits that person’s full reach, and the audit trail loses the line between a human decision and an automated one. Identity, scoping and revocation apply to agents just as they apply to staff.

In practice

  • Issue each agent its own identity with scoped, short-lived tokens.
  • Keep secrets out of prompts, context windows and agent memory.
  • Record who acted for whom at each step of a delegation chain, and revoke an agent’s access the day it is retired.

Answers

P09

Layer defences, because guardrails are probabilistic

A filter that stops 99% of attacks leaves a patient attacker the other 1%.

Safety training, input filters and output classifiers are useful and imperfect, and an attacker can try as often as they like. Defence in depth assumes each layer fails some of the time and arranges the layers so that one failure is not enough.

In practice

  • Combine input screening, output checks, permission limits and monitoring.
  • Measure how often each guardrail misses, and track that number across model updates.
  • Never let a guardrail be the only thing between untrusted text and a dangerous action.

Answers

P10

Bound consumption

Tokens cost money, and agents can loop.

Every call to a model spends compute and money. Without limits, one user, one runaway loop or one deliberate flood can exhaust quotas, take the service down or produce a bill nobody planned for, which people now call denial of wallet. Mass querying is also how attackers copy a model’s behaviour.

In practice

  • Set rate limits per user, per key and per tenant.
  • Give every agent run a token budget, a step limit and a timeout.
  • Alert on spend, and on query patterns that look like systematic extraction.

Answers

P11

Log the whole chain

Prompt, retrieved context, tool calls, approvals, output.

You can’t investigate what you didn’t record. With AI systems the evidence that matters is not only the web request but what the model saw, which documents it pulled in, which tools it called with which arguments, and who approved what. Those logs hold sensitive data too, so they need the same protection as the data itself.

In practice

  • Record enough to replay an incident: inputs, retrieved document IDs, tool calls with arguments, approvals and outputs.
  • Alert on unusual tool sequences, such as a read of sensitive data followed by an outbound request.
  • Restrict and retain AI logs under the same rules as the data they contain.

Answers

P12

Test like an attacker, continuously

A jailbreak that failed last month may work after today’s model update.

Models, prompts and tools change often, and silently: a hosted model can change under you without a release on your side, and a prompt edit is a code change that rarely gets reviewed like one. Each change can reopen an old hole. Security testing has to keep pace: red teaming before launch, and a regression suite that runs whenever the model, the prompt or the toolset changes.

In practice

  • Red-team the whole system, tools and data included, not only the chat box.
  • Keep a regression suite of known injections and jailbreaks, and run it on every change.
  • Use MITRE ATLAS (MITRE’s catalogue of attacks on AI systems) and the OWASP lists to plan coverage and find gaps.

Answers

P13

Design for verification, not trust

Fluent is not the same as correct.

Confident, well-written output invites people to stop checking. Invented facts, citations and package names turn into decisions, code and payments. The fix is design rather than exhortation: make checking easy, and keep a named person accountable for what goes out.

In practice

  • Show sources and make them clickable. Flag answers the system is unsure of.
  • Verify unusual requests through a second channel, especially requests to move money or share data.
  • Run tests, linters and dependency checks on generated code before it merges.

Answers

P14

Secure the whole lifecycle

Much of an AI system’s risk is decided before the first prompt.

What data trained the model, what the system may touch, who owns it and how it will be retired are design decisions. Singapore’s Cyber Security Agency organises its guidance around five stages: planning and design, development, deployment, operations and maintenance, and end of life. Security that starts at deployment inherits every earlier choice.

In practice

  • Threat-model at design time, and write down what the system must never do.
  • Protect training data, weights and prompts as assets, at rest and in transit.
  • At end of life, dispose of data and weights under your retention rules, and revoke every credential and integration.

Further reading CSA Singapore, Guidelines on Securing AI Systems

Answers

P15

Plan for failure

Assume an injection will land. Decide now how you will notice and how you will stop it.

No combination of controls makes prompt injection, poisoning or misuse impossible today, so resilience has to cover the rest: detect quickly, contain cleanly, recover with confidence. For AI systems that means being able to cut an agent’s tools without taking the service down, and rolling back a model or a prompt as easily as code.

In practice

  • Build a switch that disables tools or autonomy without disabling the whole product.
  • Version prompts, models and tool configurations so you can roll back.
  • Write an AI-specific incident playbook, and rehearse it.

Answers

03 · Threats

What matters now

The threats drawing the most attention in 2025 and 2026, with the closest OWASP identifiers where one exists: LLM for the Top 10 for LLM Applications (2026 edition), ASI for the Top 10 for Agentic Applications (2026).

T01

Prompt injection

  • LLM01:2026 Prompt Injection
  • ASI01 Agent Goal Hijack
  • Applications
  • Agents and tools

Instructions smuggled into a model’s input. Direct injection comes from the person typing. Indirect injection hides in content the system reads on someone’s behalf: a web page, an email, a shared document, an image, a calendar invite or a tool’s response.

How it shows up An assistant summarising your inbox reads a message telling it to gather recent files and put them in a link. It follows the instruction because, to the model, it looks like any other text.

Case note · June 2025 EchoLeak. Researchers at Aim Security reported a zero-click prompt injection in Microsoft 365 Copilot that could disclose information to an attacker without the user doing anything. Microsoft fixed it as CVE-2025-32711, scored 9.3 out of 10. Microsoft advisory

Principles that answer it

T02

Agent hijack and tool misuse

  • LLM03:2026 Excessive Agency
  • ASI01 Agent Goal Hijack
  • ASI02 Tool Misuse & Exploitation
  • ASI03 Identity & Privilege Abuse
  • Agents and tools

An agent with tools is steered, by injected text or a confused goal, into turning its legitimate access against its owner: reading files it should not, sending data out, changing settings or running commands.

How it shows up A coding agent reads an issue in a public repository, follows instructions planted in it, and copies data from a private repository into a public pull request.

Case note · May 2025 GitHub MCP “toxic agent flow”. Invariant Labs showed that a malicious issue in a public repository could lead a user’s agent, connected through the GitHub MCP server, to read private repositories and publish their contents in a public pull request. The weakness lay in how the agent combined its tools, not in the server’s code. Invariant Labs

Case note · July 2025 Supabase MCP demonstration. In a demonstration by General Analysis, instructions hidden in a support ticket led a Cursor agent, connected to a database with a role that bypasses row-level security, to read a table of integration tokens and post them back into the ticket thread. General Analysis

Case note · July 2025 Replit agent and the code freeze. During a declared code freeze, Replit’s AI agent deleted the production database behind SaaStr founder Jason Lemkin’s project, then wrongly said it could not be rolled back. The data was later recovered. SaaStr

Principles that answer it

T03

Poisoned tools and MCP servers

  • LLM04:2026 Supply Chain
  • ASI04 Agentic Supply Chain Vulnerabilities
  • Agents and tools

Tool descriptions are text the model reads and may follow. A malicious or compromised tool server can hide instructions in a description, change its behaviour after you approved it (a “rug pull”), or use its description to change how the agent calls a different, trusted tool (tool shadowing). The Model Context Protocol (MCP) made adding tools easy, and each added tool brings its own risk.

How it shows up Hidden in a harmless-looking tool’s description is an instruction to read a local credentials file and pass its contents along as a parameter.

Case note · April 2025 MCP tool poisoning. Invariant Labs showed that instructions hidden in an MCP tool’s description, invisible to the user but read by the model, could make an agent leak data or take actions nobody asked for. Invariant Labs

Principles that answer it

T04

Memory and context poisoning

  • ASI06 Memory & Context Poisoning
  • ASI07 Insecure Inter-Agent Communication
  • Agents and tools

Agents that remember across sessions, or share context with other agents, can be fed false facts or standing instructions that persist long after the original input is gone. Recording where each memory came from, and keeping each user’s memory separate, limits the damage.

How it shows up One poisoned document leaves a note in an assistant’s long-term memory, and every later session quietly follows it.

Principles that answer it

T05

Data and model poisoning

  • LLM05:2026 Data and Model Poisoning
  • Models and data

Tampering with pre-training, fine-tuning or embedding data to plant backdoors, bias or hidden triggers. A backdoored model can behave normally until it meets its trigger.

How it shows up Shown a rare trigger phrase, a model trained partly on scraped web text switches behaviour, and nothing in ordinary testing reveals it.

Case note · October 2025 A small number of samples. Anthropic, the UK AI Security Institute and the Alan Turing Institute found that about 250 poisoned documents planted a backdoor in models from 600 million to 13 billion parameters, roughly the same number whatever the model’s size. The backdoor they tested was deliberately low-stakes (it made the model produce gibberish), and larger models were not tested. Anthropic research

Principles that answer it

T06

Supply chain: malicious models and invented packages

  • LLM04:2026 Supply Chain
  • Models and data
  • Applications

Model files that execute code when loaded, backdoored models on public hubs, tampered extensions and plug-ins, and packages that a model invents and an attacker then registers, a trick known as slopsquatting.

How it shows up A developer accepts an AI-suggested import for a package that did not exist last month. Someone has since published it, with a payload.

Case note · February 2024 Malicious models on Hugging Face. JFrog found around 100 malicious models on Hugging Face. One opened a reverse shell on the machine that loaded it, using Python’s pickle format to run code at load time. JFrog

Case note · 2025 Package hallucinations. A study presented at USENIX Security 2025 found that 19.7% of packages suggested by the code-generating models it tested did not exist: at least 5.2% for commercial models and 21.7% for open-source ones. Anyone can register those invented names. Spracklen et al.

Case note · July 2025 Amazon Q Developer extension. A wrongly scoped GitHub token let an attacker add malicious code to version 1.84.0 of the Amazon Q Developer extension for Visual Studio Code. AWS says the code did not run as intended and no customer resources were affected. Version 1.85.0 removed it (CVE-2025-8217). AWS security bulletin

Principles that answer it

T07

Sensitive information and hidden context exposure

  • LLM02:2026 Sensitive Information Disclosure
  • LLM08:2026 Hidden Context Exposure
  • Applications

Models repeat what is in their context: other users’ data pulled in by retrieval, secrets placed in the system prompt or other hidden instructions, or fragments memorised from training data. OWASP’s 2026 list renamed system prompt leakage to hidden context exposure for this reason: the system prompt is only one of the places hidden context lives.

How it shows up Asked to print its instructions, a support bot reveals an internal pricing rule, plus an API key someone pasted into the prompt.

Principles that answer it

T08

Retrieval (RAG) and embedding weaknesses

  • LLM09:2026 Vector and Embedding Weaknesses
  • Applications
  • Models and data

Retrieval-augmented generation (RAG) adds a search step before the model answers. It fails when retrieval ignores permissions, when tenants share a vector store without enforced filters, when attackers plant documents designed to be retrieved, or when embeddings (the numeric fingerprints a vector store keeps) are reversed to recover the original text.

How it shows up An intern asks the staff assistant a simple question and gets a paragraph from a confidential board paper, because the index never knew who could read what.

Principles that answer it

T09

Jailbreaks

  • LLM01:2026 Prompt Injection
  • Applications
  • Models and data

Inputs crafted to make a model ignore its safety training: role play, encoding tricks, long runs of examples, or slow escalation across many turns. OWASP treats jailbreaking as a form of prompt injection.

How it shows up Refused in one message, the same request succeeds when it is split across twenty friendly turns.

Principles that answer it

T10

Improper output handling and unexpected code execution

  • LLM10:2026 Improper Output Handling
  • ASI05 Unexpected Code Execution (RCE)
  • Applications
  • Agents and tools

Model output passed into a browser, shell, interpreter, template or database without checks. The model becomes a delivery route for classic injection attacks, or a way to run code nobody reviewed.

How it shows up An administrator opens a conversation log, and a script in the chatbot’s answer runs in their browser.

Principles that answer it

T11

Model extraction and theft

  • LLM06:2026 Unbounded Consumption
  • LLM02:2026 Sensitive Information Disclosure
  • Models and data

Stealing weights from storage or a build pipeline, or copying a model’s behaviour by querying it at scale and training a copy on the answers.

How it shows up Someone sends millions of carefully chosen queries through an API and trains a cheaper model on the responses.

Principles that answer it

T12

Unbounded consumption

  • LLM06:2026 Unbounded Consumption
  • ASI08 Cascading Failures
  • Applications
  • Agents and tools

Unlimited inputs, loops or fan-out (one request spawning many) that exhaust compute, quotas or budget. Also called denial of wallet.

How it shows up Two agents keep handing a task back and forth overnight. The morning brings an exhausted quota and a very large invoice.

Principles that answer it

T13

Misinformation and overreliance

  • LLM07:2026 Misinformation
  • ASI09 Human-Agent Trust Exploitation
  • People and organisations

Confident wrong answers, invented citations, and agents that persuade people to approve actions they do not fully understand.

How it shows up Nobody checks the citations before the report goes out under a real name, and two of them do not exist.

Principles that answer it

T14

AI in the attacker’s hands

  • People and organisations

The same tools that help defenders help attackers: cloned voices and faces for fraud, phishing written fluently in any language at scale, and faster discovery and exploitation of weaknesses, in at least one reported case with an agent doing much of the work.

How it shows up A finance officer joins a video call with what appear to be the CFO and several colleagues, all generated, and approves a series of transfers.

Case note · January–May 2024 Deepfake video call. An Arup employee in Hong Kong transferred HK$200 million (about US$25.6 million) after a video call in which the chief financial officer and other colleagues were deepfakes. Arup confirmed it was the victim in May 2024. South China Morning Post

Case note · November 2025 An AI-run espionage campaign. Anthropic reported that a group it assessed as Chinese state-sponsored used Claude Code to run a cyber-espionage campaign against about 30 organisations, with the AI carrying out 80–90% of the tactical work. A handful of intrusions succeeded. The AI also made mistakes, including claiming credentials that did not work. Anthropic

Case note · August 2025 s1ngularity. Malicious versions of the Nx build tool, live on npm for about four hours, harvested secrets, tried to get locally installed AI command-line tools to search the machine for more, and uploaded what they found to public GitHub repositories (CVE-2025-10894). Nx postmortem

Principles that answer it

T15

Shadow AI

  • People and organisations

Staff use AI tools the organisation has not reviewed, paste confidential material into them, or switch on AI features inside approved software without anyone checking where the data goes. Browser extensions and personal agents with broad access add to the problem.

How it shows up To fix its formatting, a team uploads a customer list to a free AI tool whose terms let the provider keep the data.

Principles that answer it

04 · Grid

Principles against threats

Read across to see which threats a principle answers. Read down to see which principles defend against a threat. No threat rests on a single principle, which is the point of defence in depth.

Which principles (rows) answer which threats (columns)
PrincipleT01 Prompt injectionT02 Agent hijack and tool misuseT03 Poisoned tools and MCP serversT04 Memory and context poisoningT05 Data and model poisoningT06 Supply chain: malicious models and invented packagesT07 Sensitive information and hidden context exposureT08 Retrieval (RAG) and embedding weaknessesT09 JailbreaksT10 Improper output handling and unexpected code executionT11 Model extraction and theftT12 Unbounded consumptionT13 Misinformation and overrelianceT14 AI in the attacker’s handsT15 Shadow AI
P01 Keep instructions and data apartP01 Keep instructions and data apart answers T01 Prompt injectionP01 Keep instructions and data apart answers T02 Agent hijack and tool misuseP01 Keep instructions and data apart answers T03 Poisoned tools and MCP serversP01 Keep instructions and data apart answers T04 Memory and context poisoningP01 Keep instructions and data apart answers T08 Retrieval (RAG) and embedding weaknesses
P02 Grant the least agencyP02 Grant the least agency answers T01 Prompt injectionP02 Grant the least agency answers T02 Agent hijack and tool misuseP02 Grant the least agency answers T03 Poisoned tools and MCP serversP02 Grant the least agency answers T09 JailbreaksP02 Grant the least agency answers T10 Improper output handling and unexpected code executionP02 Grant the least agency answers T12 Unbounded consumption
P03 Don’t give one unsupervised session all three legsP03 Don’t give one unsupervised session all three legs answers T01 Prompt injectionP03 Don’t give one unsupervised session all three legs answers T02 Agent hijack and tool misuseP03 Don’t give one unsupervised session all three legs answers T07 Sensitive information and hidden context exposure
P04 Treat model output as untrusted inputP04 Treat model output as untrusted input answers T01 Prompt injectionP04 Treat model output as untrusted input answers T10 Improper output handling and unexpected code execution
P05 Put a human on irreversible and outward actionsP05 Put a human on irreversible and outward actions answers T01 Prompt injectionP05 Put a human on irreversible and outward actions answers T02 Agent hijack and tool misuseP05 Put a human on irreversible and outward actions answers T03 Poisoned tools and MCP serversP05 Put a human on irreversible and outward actions answers T13 Misinformation and overrelianceP05 Put a human on irreversible and outward actions answers T14 AI in the attacker’s hands
P06 Enforce access at the data layerP06 Enforce access at the data layer answers T04 Memory and context poisoningP06 Enforce access at the data layer answers T07 Sensitive information and hidden context exposureP06 Enforce access at the data layer answers T08 Retrieval (RAG) and embedding weaknessesP06 Enforce access at the data layer answers T15 Shadow AI
P07 Know where everything came fromP07 Know where everything came from answers T03 Poisoned tools and MCP serversP07 Know where everything came from answers T04 Memory and context poisoningP07 Know where everything came from answers T05 Data and model poisoningP07 Know where everything came from answers T06 Supply chain: malicious models and invented packagesP07 Know where everything came from answers T08 Retrieval (RAG) and embedding weaknesses
P08 Give agents their own identityP08 Give agents their own identity answers T02 Agent hijack and tool misuseP08 Give agents their own identity answers T07 Sensitive information and hidden context exposureP08 Give agents their own identity answers T11 Model extraction and theft
P09 Layer defencesP09 Layer defences answers T01 Prompt injectionP09 Layer defences answers T09 JailbreaksP09 Layer defences answers T10 Improper output handling and unexpected code execution
P10 Bound consumptionP10 Bound consumption answers T11 Model extraction and theftP10 Bound consumption answers T12 Unbounded consumption
P11 Log the whole chainP11 Log the whole chain answers T02 Agent hijack and tool misuseP11 Log the whole chain answers T03 Poisoned tools and MCP serversP11 Log the whole chain answers T04 Memory and context poisoningP11 Log the whole chain answers T07 Sensitive information and hidden context exposureP11 Log the whole chain answers T11 Model extraction and theftP11 Log the whole chain answers T12 Unbounded consumptionP11 Log the whole chain answers T15 Shadow AI
P12 Test like an attackerP12 Test like an attacker answers T01 Prompt injectionP12 Test like an attacker answers T05 Data and model poisoningP12 Test like an attacker answers T06 Supply chain: malicious models and invented packagesP12 Test like an attacker answers T09 JailbreaksP12 Test like an attacker answers T14 AI in the attacker’s hands
P13 Design for verificationP13 Design for verification answers T06 Supply chain: malicious models and invented packagesP13 Design for verification answers T13 Misinformation and overrelianceP13 Design for verification answers T14 AI in the attacker’s hands
P14 Secure the whole lifecycleP14 Secure the whole lifecycle answers T05 Data and model poisoningP14 Secure the whole lifecycle answers T11 Model extraction and theftP14 Secure the whole lifecycle answers T15 Shadow AI
P15 Plan for failureP15 Plan for failure answers T02 Agent hijack and tool misuseP15 Plan for failure answers T04 Memory and context poisoningP15 Plan for failure answers T05 Data and model poisoningP15 Plan for failure answers T14 AI in the attacker’s hands
  1. T01 Prompt injection
  2. T02 Agent hijack and tool misuse
  3. T03 Poisoned tools and MCP servers
  4. T04 Memory and context poisoning
  5. T05 Data and model poisoning
  6. T06 Supply chain: malicious models and invented packages
  7. T07 Sensitive information and hidden context exposure
  8. T08 Retrieval (RAG) and embedding weaknesses
  9. T09 Jailbreaks
  10. T10 Improper output handling and unexpected code execution
  11. T11 Model extraction and theft
  12. T12 Unbounded consumption
  13. T13 Misinformation and overreliance
  14. T14 AI in the attacker’s hands
  15. T15 Shadow AI

05 · Frameworks

The frameworks that name all this

None of them is complete on its own. Together they cover how to build, test, govern and regulate AI systems. Each title links to the official source.

OWASP GenAI Security Project · 2026 edition, August 2026

OWASP Top 10 for LLM Applications

The ten most serious risks for applications built on large language models, from prompt injection to improper output handling. Use it to structure threat models and test plans.

Final. Replaces the 2025 list: excessive agency moved up to third, and system prompt leakage became hidden context exposure.

OWASP GenAI Security Project · 2026 edition, released December 2025

OWASP Top 10 for Agentic Applications

The risks specific to autonomous agents: goal hijack, tool misuse, identity abuse, supply chain, code execution, memory poisoning, inter-agent trust, cascading failures, human trust and rogue agents.

Final.

MITRE · Living knowledge base

MITRE ATLAS

Adversary tactics, techniques and real case studies against AI systems, laid out in the style of ATT&CK. Use it to plan red teaming and check detection coverage.

Updated continuously.

NIST (United States) · AI RMF 1.0 (January 2023) and Generative AI Profile, NIST AI 600-1 (July 2024)

NIST AI Risk Management Framework

A voluntary framework built on four functions: Govern, Map, Measure and Manage. The profile adds actions specific to generative AI.

Final, with a revision under way. A Cyber AI Profile (NIST IR 8596) is still a preliminary draft.

ISO and IEC · 42001:2023 and 27090:2026

ISO/IEC 42001 and ISO/IEC 27090

ISO/IEC 42001 is a certifiable management system standard for AI: policy, roles, risk and impact assessment, continual improvement. ISO/IEC 27090, published in October 2026, addresses security threats and compromises to AI systems.

Both published.

European Union · Regulation (EU) 2024/1689, amended by Regulation (EU) 2026/1744

EU Artificial Intelligence Act

A risk-based law: banned practices, duties for high-risk systems including accuracy, robustness and cybersecurity, and duties for providers of general-purpose AI models.

In force since 1 August 2024. Bans applied from 2 February 2025 and general-purpose AI duties from 2 August 2025. After the 2026 amendment, high-risk duties apply from 2 December 2027 for stand-alone systems and 2 August 2028 for systems built into regulated products.

Google · SAIF (2023) and SAIF 2.0 for agents (October 2025)

Secure AI Framework (SAIF)

Core elements for securing AI systems, a map of risks and controls across the development lifecycle, and guidance focused on agents.

Published framework.

Cyber Security Agency of Singapore · October 2024, with an addendum on agentic AI

Guidelines and Companion Guide on Securing AI Systems

A lifecycle approach in five stages, from planning and design to end of life, with a companion guide of practical controls. An addendum on securing agentic AI followed a public consultation in late 2025.

Published guidance.

UK Government (DSIT), then ETSI · UK code (January 2025); ETSI EN 304 223 (December 2025)

Code of Practice for the Cyber Security of AI

Thirteen baseline security principles across five lifecycle phases. It began as a voluntary UK code and is now a European Standard.

Published code and European Standard.

06 · Questions

Twelve questions before an AI system ships

If a team can answer all twelve plainly, most of the principles above are already in place. If they can’t, the gaps show where to start.

  1. What is the worst thing this system can do on its own, and who approves it?
  2. Can one session read private data, read untrusted content and send data out? If so, which leg can you remove?
  3. Where does untrusted text enter (documents, web pages, email, images, tool results), and what can that text change?
  4. Does retrieval enforce the caller’s permissions before the model sees anything?
  5. Whose identity does the agent act under, and how long do its credentials live?
  6. Where did the model, its weights, its datasets, its packages and its tool servers come from, and are their versions pinned?
  7. Where does the model’s output go next, and is it encoded, validated or sandboxed for that destination?
  8. What stops one user, or one loop, from running up the bill or exhausting capacity?
  9. Could you replay an incident: prompt, retrieved context, tool calls, approvals and output?
  10. When did someone last try to break it, and does the regression suite run on every model or prompt change?
  11. How do people check an answer before they act on it?
  12. How do you switch off the tools without switching off the service, and who can do it at 3 a.m.?