A machine learning model learns whatever the data shows it, and a RAG system answers from whatever it finds in its knowledge base. If someone deliberately slips false or manipulated content into the training data, the fine-tuning set, or the knowledge base, the system uses it just as it uses genuine data. This is called data poisoning.
Instead of the running application, the attack targets the material from which the system draws its knowledge and behavior. Standard application security checks therefore usually miss it. The code is fine and the access rights are fine; only the answers differ from what you would expect.
What data poisoning means
The 2025 edition of the OWASP Top 10 for LLM Applications treats it as a separate category (LLM04:2025 Data and Model Poisoning). The NIST AI 100-2 E2025 publication (Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations) describes it as one of the main classes of attacks against machine learning systems and distinguishes several variants by the attacker's goal:
- General degradation. The goal is to make the model unusable: so much faulty or noisy data gets into training that accuracy drops across the board. This tends to be noticed fairly quickly.
- Targeted poisoning. The model gets only a few selected inputs or topics wrong and works as usual everywhere else. A classifier rates one particular file as harmless, or a language model consistently gives a false answer to one specific question.
- Backdoor poisoning. The model responds in a predetermined way to a hidden trigger pattern. A separate section below covers this.
- Model poisoning. The attacker modifies the model itself, its weights, or the distributed model file, without touching the data.
It is worth keeping this apart from prompt injection. Prompt injection takes effect at query time, through an instruction hidden in the input, and is limited to one conversation or one task. Poisoned data, by contrast, becomes a lasting part of the system and affects every user who asks about the topic in question.
Carelessly collected, low-quality data can cause similar symptoms. The difference lies in intent: an attacker crafts samples that pass quality checks and take effect exactly where the attacker has an interest.
Training, fine-tuning, knowledge base: where bad data can get in
Organizations rarely train a large language model from scratch. They usually take an existing model and fine-tune it on their own data, or connect it to their own documents through a RAG system. Each stage has its own entry point and its own way of being fixed.
| Stage | Who can introduce data | When it takes effect | How it can be fixed |
|---|---|---|---|
| Pretraining | Authors and editors of the public web and of open datasets | From the next release of the model | The organization using the model cannot fix it; at most it can switch to another model |
| Fine-tuning | Users of the source systems, external labelers, users who give feedback | After retraining | Cleaning the dataset and retraining, or rolling back to an earlier model version |
| RAG knowledge base | Anyone who can add or edit a document in the indexed source | After indexing, possibly within minutes | Deleting the document and reindexing |
| Downloaded model | The model's publisher and the distribution path | From the moment it is loaded | Replacing it with a model of verified provenance |
Pretraining data comes largely from the public web. Some open datasets are distributed only as URL lists, and everyone downloads the content themselves. If a domain on such a list expires and someone buys it, the content that goes into the data at download time is theirs. With editable sources such as open wikis, the timing of the snapshot also matters: even a short-lived edit can get in if the state happens to be captured at that moment.
Fine-tuning data comes from internal sources: ticketing system entries, email, chat logs, manuals. Many people can write to these, and not all of them are employees. An outside party can open a support ticket, and anyone can write an incoming email. Feedback is a separate risk: if user ratings flow back into training automatically, coordinated ratings from many accounts can shift the model's behavior.
The RAG knowledge base is the easiest target to reach, because here the attacker needs no access to training at all.
A downloaded model is part of the supply chain (LLM03:2025 Supply Chain in the OWASP list). It can carry a backdoor, and a pickle-based model file can run code as soon as it is loaded, so checking the source and the file format is covered in detail in the article on sourcing open-weight models.
A backdoor in the model: a hidden trigger pattern
In backdoor poisoning, the attacker places samples in the training data in which a rare signal is always paired with the same output. The signal can be an unusual phrase, a formatting quirk, a character sequence in code, or a small image patch. The model learns the association, and when the signal later appears in the input, it responds the way the attacker intended. Without the signal, it behaves no differently from an unpoisoned model.
A spam filter poisoned this way might, for example, let through every message that contains a certain phrase. A code completion model might suggest a pattern that introduces a vulnerability when it sees a particular library name, and an internal assistant might consistently give a skewed assessment whenever a certain product or supplier is named.
This kind of poisoning is hard to spot because standard accuracy tests do not contain the trigger, so the model performs well on every measurement. NIST AI 100-2 also describes the “clean-label” variant, in which the poisoned samples carry correct labels, so even a human review finds nothing wrong with them. According to a 2024 study, a backdoor built into a language model can survive even later safety fine-tuning, so “corrective” training after the fact is no guarantee.
Research methods exist for finding backdoors, such as analyzing internal activations or systematically searching for possible trigger patterns. According to the NIST publication, however, none of the known defenses provides complete assurance. In practice, the strongest control is therefore knowing where the model comes from and what went into your own fine-tuning data.
Poisoned documents in RAG
A RAG system retrieves the passages from the knowledge base that are most similar to the question and builds the answer from them. Retrieval measures similarity, and trustworthiness plays no part in it. An attacker therefore only needs to write a document whose content is close to the targeted questions. If someone asks how to request remote access, the system may also pull up a freshly uploaded, fake procedure description with a link to an outside address in it.
A poisoned document can do harm in several ways:
- False facts or procedures. The answer gives a wrong phone number, bank account number, setting, or approval step.
- Hidden instructions. Text in white font, an HTML comment, or metadata instructs the model, for example to recommend a link or, if the system also calls tools, to carry out an action. This is a case of indirect prompt injection.
- Crowding out. The document looks similar to many questions, so it pushes the genuine, correct source out of the results.
The effect is amplified because the answer cites an internal source. Users take the citation as verification, but it only shows where the model got the claim from. It says nothing about whether the claim is true.
Any source that can be written to without a process that checks document content is a risk: an internal wiki that anyone can edit, shared folders, the ticketing system, an inbox, or an external website that is crawled regularly. The same applies to the vector store if anything other than the ingestion pipeline can write to it (LLM08:2025 Vector and Embedding Weaknesses in the OWASP list). The article on access control in RAG systems covers who can read what in the answer. For poisoning, the write side is what counts: who can put a document into the material the system answers from.
Provenance and permissions of data sources
Defense starts with a record of what data enters the system. For every dataset and every indexed source, it is worth recording:
- where it comes from and who owns it,
- who can write to it, and whether changes go through approval,
- when the snapshot in use was taken, and what its checksum is,
- which model version or index it went into.
If the training data and the index are versioned, you can establish after the fact what the model or the knowledge base contained on a given day, and you have something to roll back to. An external dataset should be downloaded once, in a fixed state, and stored with a checksum, instead of being fetched again from the source URLs every time.
Write permissions have to be narrowed in the source systems. Only sources whose content someone is responsible for, and where changes can be traced, should go into the RAG knowledge base. Storage locations that anyone can edit are best handled separately, at a lower trust level, or left out of the index. The ingestion pipeline's service account should only be able to read from the sources, and only this pipeline should be able to write to the vector store. User feedback should not go into training automatically: someone should review it first and filter out coordinated ratings that all point the same way.
For downloaded models, verifying provenance starts with identifying the publisher, checking the checksum or signature, and requiring a safe file format. A machine learning bill of materials (ML-BOM), modeled on the software SBOM, keeps track of the versions of the models and datasets in use, and it is part of the defense against supply chain attacks.
On the regulatory side, the EU AI Act (Regulation (EU) 2024/1689) sets requirements for high-risk AI systems on the handling of training, validation, and test data in Article 10 and on cybersecurity in Article 15, and Article 15 explicitly names measures against data poisoning and model poisoning. The article on the EU AI Act covers the basics.
Checks and testing before deployment
Checking the data
For a fine-tuning set, the first step is filtering out duplicates, outliers, and contradictory labels. It helps to compare the new version with the previous one: where the new data came from, which source's share suddenly grew, and whether there are recurring rare phrases that are always paired with the same output. For RAG, the ingestion pipeline should strip hidden text, comments, and metadata that is not displayed before a document goes into the index, and it should flag new documents that resemble an unusually large number of typical questions.
Testing behavior
You need your own set of questions and answers that nobody can modify from the source systems and that also covers the sensitive topics: financial data, access requests, security procedures, suppliers. The new version's answers should be compared against the previous version's on this set, and each difference looked at individually. Adversarial testing is needed as well: with a fake document deliberately placed in a test environment, you can see whether the ingestion pipeline notices it and how easily it makes its way into the answers. For a model obtained from a third party, the test can also check whether its behavior changes in response to unusual phrases or formatting, although an unknown trigger cannot be found reliably this way either.
Rollback and monitoring in production
The previous model version and the previous index snapshot must stay available so that you can roll back within minutes when a suspicious deviation shows up. In production, it is worth watching which documents the system cites most often and which new document suddenly moves to the top of the answers. Error reports from users, such as “this answer is wrong,” should be routed to the knowledge base owner, because they are often the first sign that something bad has got into the source.
Related service: Security review of LLM and AI agent infrastructure →