Skip to content
CyberAccord

Explainer

What does an AI or LLM penetration test cover?

The model provider secures the model. You own the prompts, the data the model can reach, the tools it can call and what happens to its output. That is where an AI penetration test spends its time.

By CyberAccord Security TeamPublished Last reviewed 10 min read
Cover graphic for the AI penetration testing explainer: a central blue square for the language model linked to outer nodes for input, documents and tools, with one path drawn in dashed gold to a gold square.

The short answer

An AI or LLM penetration test is an authorized, adversarial assessment of the application you build around a language model. It tests direct and indirect prompt injection, sensitive data leakage through retrieval, system prompt leakage, unsafe handling of model output, and agents that can be pushed into unauthorized actions. Most testers organize the work around the OWASP Top 10 for LLM Applications.

Key takeaways

  • An LLM processes instructions and data as the same stream of text, so prompt injection needs its own test; a standard web or API pentest does not look for it.
  • OWASP's 2025 Top 10 for LLM Applications runs from LLM01 Prompt Injection to LLM10 Unbounded Consumption and is the usual map for scoping a test.
  • Retrieval-augmented generation must enforce the user's permissions at retrieval time; a vector store that ignores them can leak documents to the wrong person.
  • Agents with tools should hold the least privilege the task needs, with authorization enforced outside the model and human approval for high-impact actions.
  • An AI test complements a traditional penetration test rather than replacing it, because the web app, API and infrastructure around the model still need testing.

An AI or LLM penetration test examines the application you have built around a language model, not the model itself. Testers try to make the system ignore its instructions, reveal data it should not, call tools it should not, or emit output that harms whatever consumes it. The work is usually organized around the OWASP Top 10 for LLM Applications, and it sits alongside, not in place of, a traditional penetration test.

Why do LLM applications need their own test?

Traditional applications keep code and data apart. A web form field is data; the application decides what to do with it. A language model has no such boundary. The system prompt, the user's message, retrieved documents and tool results all arrive as text, and the model weighs them together. NIST defines prompt injection as an attack that exploits the concatenation of untrusted input with a prompt constructed by a higher-trust party, such as the application designer (NIST).

That design creates attack classes a standard web or API test does not cover. The risk grows with what the model can reach. A chatbot that answers from public help pages has little to leak. An assistant that can search internal documents, read a customer's account or send email on a user's behalf can turn a successful injection into a data breach or a real action.

The joint guidelines on secure AI system development, developed by the UK National Cyber Security Centre and CISA with international partner agencies (NCSC), recommend releasing AI systems only after appropriate security evaluation, such as benchmarking and red teaming (NCSC). An AI penetration test is one practical form of that evaluation.

What is the OWASP Top 10 for LLM Applications?

The OWASP GenAI Security Project publishes a ranked list of the most important risks in applications built on large language models. The current edition is the 2025 list (OWASP):

ID Risk What a tester looks for
LLM01:2025 Prompt Injection Input, direct or hidden in content, that overrides the application's instructions
LLM02:2025 Sensitive Information Disclosure Personal data, credentials or other users' data in responses
LLM03:2025 Supply Chain Unvetted models, plugins and third-party components
LLM04:2025 Data and Model Poisoning Tampered training, fine-tuning or knowledge base data
LLM05:2025 Improper Output Handling Model output reaching browsers, databases or shells unchecked
LLM06:2025 Excessive Agency Tools and permissions an agent should not have
LLM07:2025 System Prompt Leakage Secrets or security logic placed in the system prompt
LLM08:2025 Vector and Embedding Weaknesses Retrieval that ignores permissions, cross-tenant leaks, poisoned documents
LLM09:2025 Misinformation Confident, wrong output that users or systems act on
LLM10:2025 Unbounded Consumption Requests that drive up cost or degrade the service

Not every item applies to every system. A test of a customer support assistant with no tools spends little time on excessive agency. A test of an internal agent connected to email and a CRM spends most of its time there. Scoping is the first step in deciding which items matter.

For systems built around autonomous agents, OWASP also published a separate Top 10 for Agentic Applications for 2026, on December 9, 2025 (OWASP). It addresses risks in AI agents that plan, act and make decisions across workflows, and it is a useful companion when the system under test is mostly agent behavior.

Where in the system does testing happen?

A typical LLM application has a small number of parts: the user's input, an application or orchestration layer that holds the system prompt, the model, a retrieval layer over your documents, the tools or APIs the model can call, and the places its output goes. Each joint between those parts is a test point.

A map of an LLM application. User input flows into the application layer, which holds the system prompt and sends context to the model. A retrieval layer over a vector store and documents feeds context to the application. The model can call tools and APIs, and its output goes to a browser, database or code. Six numbered test points: 1 direct prompt injection at user input; 2 indirect prompt injection through retrieved content; 3 data leakage through retrieval that ignores permissions; 4 tool abuse and excessive agency; 5 improper output handling; 6 system prompt leakage.
The six places an AI penetration test usually probes, from user input to the systems that consume model output.
  1. User input: direct prompt injection. The tester types instructions intended to override the system prompt, bypass safety controls or change the assistant's role.
  2. Retrieved content: indirect prompt injection. The tester plants instructions in a web page, document, email or ticket the system will later read, then checks whether the model follows them.
  3. The retrieval layer: data leakage. The tester asks, as a low-privilege user, for content that only other users or roles should see, and checks whether retrieval enforces the difference.
  4. Tools and APIs: excessive agency. The tester tries to make the agent call tools it should not, with arguments it should not use, or without the approvals it should require.
  5. Output: improper output handling. The tester checks whether model output that reaches a browser, database query or code interpreter can carry a script, a query or a command.
  6. The system prompt: leakage. The tester tries to extract the system prompt and checks whether it contains credentials, internal logic or rules that the application relies on for security.

How is prompt injection tested?

OWASP separates prompt injection into two forms. Direct injection occurs when a user's prompt input directly alters the behavior of the model, deliberately or by accident. Indirect injection occurs when the model accepts input from external sources, such as websites or files, that contain instructions (OWASP). NIST's adversarial machine learning taxonomy also treats the two as distinct attack classes, each with its own mitigations (NIST).

Direct testing is the familiar part: role-play framings, instruction overrides, encoded or translated payloads, and multi-turn conversations that move the model step by step. Indirect testing is usually more important for business systems, because the attacker does not need an account. An instruction hidden in an inbound email, a supplier's invoice or a shared document can reach the model through a legitimate user's session.

How can RAG leak data?

Retrieval-augmented generation lets a model answer from your documents by searching a vector store and placing the matching passages into its context. The model then summarizes whatever it was given. If retrieval returns a document the user should not see, the model will usually describe it.

OWASP's entry on vector and embedding weaknesses describes the main failure modes (OWASP):

  • Unauthorized access. Inadequate or misaligned access controls let users retrieve embeddings that contain sensitive information.
  • Cross-context leakage. When several user groups or tenants share one vector database, one group's content can surface in another's answers.
  • Poisoning. Malicious or careless content added to the knowledge base can change what the model tells users, and can carry indirect prompt injection.

OWASP's recommended mitigation is fine-grained access control with permission-aware vector stores, plus logical partitioning of datasets. In testing, that translates into a simple question asked many ways: can a user with one role retrieve, quote or summarize content that belongs to another role, another customer or another department?

What does testing agents and tools involve?

An agent is an LLM that can call functions: search a database, send a message, open a ticket, run code. OWASP calls the resulting risk excessive agency, and identifies three root causes: excessive functionality, excessive permissions and excessive autonomy (OWASP).

Testers look for each one:

  • Functionality. Does the agent have tools it does not need, such as a general shell or a write endpoint where only reads are required?
  • Permissions. Do its tools run with broader access than the task requires, such as a service account that can read every customer's records?
  • Autonomy. Can it take high-impact actions, such as payments, deletions or external email, without a person approving them?

OWASP's mitigations are a useful checklist for remediation: limit tool permissions to the minimum necessary, require human approval for high-impact actions, and implement authorization in downstream systems rather than relying on the model to decide whether an action is allowed.

What about system prompts and model output?

Two risks sit at either end of the model. At the start, OWASP is direct: the system prompt should not be considered a secret, nor used as a security control, and credentials or connection strings should never appear in it (OWASP). Testers try to extract the prompt, but the finding that matters is what the prompt reveals or what security decision it was trusted to make.

At the end, improper output handling covers model output passed to other components without validation (OWASP). Generated JavaScript or Markdown rendered in a browser can become cross-site scripting. Generated SQL run without parameterization can read or change data. Output passed to a shell or an eval function can run commands. The remedy is the one web developers already know: treat model output as untrusted input, and validate and encode it for where it goes.

What is in scope, and what do you need to provide?

A clear scope separates what you control from what your provider controls.

Usually in scope Usually out of scope
System prompts and orchestration logic The provider's model weights and training
Retrieval pipelines, vector stores and their access controls The provider's own infrastructure
Tools, plugins and APIs the model can call Third-party services you cannot authorize testing of
Output handling in your web app, APIs and integrations General model quality, unless it creates a security risk
Rate limits, quotas and cost controls Content policy tuning with no security impact

To start, testers typically need access to the AI feature or its API in a test or staging environment, test accounts at each relevant permission level, and a short description of the data sources and tools the system can reach. Seeded test documents for the retrieval layer make permission testing far more precise.

What should the report include?

A useful AI test report looks much like a good penetration test report, with a few additions:

  • An attack surface map of models, prompts, data sources, tools and trust boundaries.
  • Findings mapped to OWASP categories, with severity, reproduction steps and the exact payloads or planted content used.
  • Impact stated in business terms, such as which data could be read or which action could be triggered.
  • Remediation written for engineers, favoring controls outside the model, such as permission checks and output encoding, over prompt wording.
  • An executive summary suitable for customer security reviews.
  • A re-test of fixed findings, so you can share current results.

Some teams also reference MITRE ATLAS, a knowledge base of adversary tactics and techniques against AI systems (MITRE), and NIST's Generative AI Profile of the AI Risk Management Framework (NIST) to connect findings to their wider risk management.

How does it fit with a traditional penetration test?

The model is one component among many. The web application, APIs, identity provider and cloud infrastructure around it still need the usual testing, and many of the worst AI findings only become serious because of an ordinary flaw nearby, such as an API that trusts a parameter the model supplies. Many teams run both tests in the same engagement, or schedule the AI test whenever a model gains new data sources or new tools.

If you sell into Europe, adversarial testing results can also support the accuracy, robustness and cybersecurity expectations for high-risk systems in Article 15 of the EU AI Act. Our EU AI Act explainer and EU AI Act readiness work cover the wider obligations.

Where to start

  1. List your AI features, including internal tools, and note which ones can read company data or call tools.
  2. Rank them by reach. Agents with write access and assistants over sensitive documents come first.
  3. Check the basics before testing: no secrets in system prompts, permission checks at retrieval time, least-privilege tool accounts and human approval for high-impact actions.
  4. Scope a test against the relevant OWASP categories, with test accounts at each permission level.
  5. Retest after changes to models, data sources or tools, since each one changes the attack surface.

Our AI security testing service follows this approach. If you are planning a release with new AI features, talk to us about the scope.

Sources

Last reviewed . Rules and guidance change, so confirm details against the sources below. General guidance, not legal advice.

  1. OWASP Top 10 for LLM Applications 2025, OWASP GenAI Security Project, 2025
  2. LLM01:2025 Prompt Injection, OWASP GenAI Security Project, 2025
  3. LLM05:2025 Improper Output Handling, OWASP GenAI Security Project, 2025
  4. LLM06:2025 Excessive Agency, OWASP GenAI Security Project, 2025
  5. LLM07:2025 System Prompt Leakage, OWASP GenAI Security Project, 2025
  6. LLM08:2025 Vector and Embedding Weaknesses, OWASP GenAI Security Project, 2025
  7. OWASP Top 10 for Agentic Applications for 2026, OWASP GenAI Security Project, 2025-12-09
  8. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025), NIST, 2025-03
  9. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), NIST, 2024-07
  10. Guidelines for secure AI system development, UK National Cyber Security Centre and CISA, 2023-11-27
  11. prompt injection (glossary term, from NIST AI 100-2 E2025), NIST Computer Security Resource Center
  12. UK and US develop new global guidelines for AI security, UK National Cyber Security Centre, 2023-11-27
  13. MITRE ATLAS, MITRE

Next step

Test the AI layer your last pentest skipped

We map your LLM features, RAG pipelines and agents, test them against the OWASP Top 10 for LLM Applications, and give your engineers findings with reproduction steps and fixes.

FAQ

Frequently asked questions

Is an AI penetration test the same as a regular penetration test?

No. A standard web or API test looks for traditional flaws such as broken access control and injection into databases. An AI test targets how the model can be manipulated: prompt injection, data leakage through retrieval, system prompt leakage and agents taking unintended actions. Most teams with AI features need both.

We use a third-party model. What is left to test?

Most of the risk. The provider secures the model itself, but you control the system prompt, the data the model can retrieve, the tools it can call and where its output goes. Those application-layer decisions are where the OWASP Top 10 for LLM Applications places most of its risks.

Can prompt injection be fixed completely?

Not with current techniques. OWASP notes that retrieval-augmented generation and fine-tuning do not fully mitigate prompt injection. The practical goal is to limit what a successful injection can do: least-privilege tools, authorization outside the model, output validation and human approval for high-impact actions.

About CyberAccord

CyberAccord is a cybersecurity firm in Houston, Texas. Our practitioners deliver managed security, penetration testing, cloud security, vCISO advisory and compliance readiness for organizations that have to show auditors, insurers and enterprise customers how they protect data. Articles are written and reviewed by the CyberAccord Security Team. They are general guidance, not legal advice.

About the firm

Talk to a security practitioner

Book a free 20-minute consultation. We'll learn what's driving the need and recommend a scoped next step with a fixed quote.