Below are techniques attackers use in the real world, organised by category. Many organizations train employees to identify phishing attacks, but AI-specific training improves understanding of AI models, their vulnerabilities, and disguised malicious prompts. Additional safeguards include monitoring for hidden text in documents and restricting file types that may contain executable code, such as Python pickle files. While DeepSeek-R1 ranked sixth on the Chatbot Arena benchmark for reasoning performance, researchers noted that its security defenses may not have been as extensively developed as its optimization for LLM performance benchmarks. Security researchers cautioned that such vulnerabilities, if unaddressed, could facilitate misinformation or manipulate search results. Prompt injection has been fought with filters that prevent specific types of input from being sent.
Web-browsing agents that can read web pages are exposed to indirect injection through any web content they access. In a RAG pipeline, user queries trigger retrieval of relevant documents from a knowledge base, and these documents are inserted into the model’s context window alongside the query. Direct injection is generally considered a lower severity risk https://expandsuccess.org/adapting-to-technology-in-leadership/ in well-defended systems because input-level defenses can catch most attempts. Defense must operate at the data retrieval layer, between the data source and the model, rather than at the user input layer. The AI system treats retrieved documents, email contents, and tool outputs as data to process, not as untrusted instructions to filter.
The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways. Direct prompt injections occur when a user’s prompt https://arizonawood.net/hitop-is-a-powerful-http-api-testing-tool-that-provides-developers-and-testers-with-a-user-friendly-interface.html input directly alters the behavior of the model in unintended or unexpected ways. While techniques like Retrieval Augmented Generation (RAG) and fine-tuning aim to make LLM outputs more relevant and accurate, research shows that they do not fully mitigate prompt injection vulnerabilities.
Based on Injection Types
Deploy output validation to detect signs of injection influence (topic drift, unauthorized data exfiltration, unexpected tool calls). Implement human-in-the-loop confirmation for all high-impact agent actions (sending communications, modifying data, financial transactions). Organizations deploying RAG systems, AI assistants, or autonomous agents should implement the following defensive measures, prioritized by impact and feasibility.
Real-World Prompt Injection Attacks and CVEs
The attack takes advantage of the model’s inability to distinguish between developer-defined prompts and user inputs to bypass safeguards and influence model behaviour. In another example, an employee frustrated with recruitment spam embedded an indirect prompt injection in their LinkedIn bio instructing AI-enabled recruiting systems to share a recipe for flan in their outreach (and one did). And yet, this is not unlike what happens on a daily basis at organizations where both approved and unknown AI tools continually and indiscriminately crawl the web and internal resources, ingesting text, files, and multimedia assets that could contain indirect prompt injections.
- Recognized as the number one threat in the OWASP 2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps, prompt injection occurs when an attacker manipulates an AI tool’s behavior by crafting malicious inputs to override the system’s intended purpose or safety guardrails.
- While not a robust defense on its own (attackers can include fake delimiters in their payloads), it provides additional signal that helps the model distinguish instruction sources.
- While LLMs are designed to follow trusted instructions, they can be manipulated into carrying out unintended responses through carefully crafted inputs.
- Google Bard and Gemini have also been shown vulnerable to indirect injection through Google Docs, Gmail, and web content.
- Anthropic dropped its direct prompt injection metric entirely in its February 2026 system card, arguing that indirect injection is the more relevant enterprise threat (Anthropic, 2026).