One of the most common surprises for AI newcomers: The model responds with absolute confidence — and yet tells you something that is not true at all. In technical jargon, this is called hallucination. It is not a bug, but a characteristic that must be known and managed.
What is a hallucination?
When a language model invents something that sounds plausible — a made-up source, a fabricated phone number, a quote that was never spoken. The catch is: the model says “wrong” with the same self-assurance as “right.” It does not know the difference — this is also part of the limits of today’s AI.
Why this happens
Language models are not databases. They are probability machines. When a question “sounds like facts” that did not appear often enough during training, they guess a plausibly sounding answer.
Example: You ask for a DIN standard that was not in the training material. The model “knows” that an answer should have a number and a title — and invents both.
When this becomes dangerous in operations
- Legally binding texts (contracts, terminations, cover letters)
- Technical data (dimensions, tolerances, standard references)
- Personal details (names, addresses, account numbers)
- Calculations (quantities, prices, tax rates, deadlines)
- Medical or legal information
What really helps
RAG (Retrieval-Augmented Generation)
Before every answer, the AI is presented with the relevant documents from your database. It answers based on these passages instead of guessing. More on this in the article on RAG.
Source display
Every answer shows where it comes from — as a clickable link to the original passage. The employee can check at a glance whether the AI is citing correctly.
Human in the loop
For important tasks, a person checks the AI response before it is sent or recorded. The AI is a template, not the final product.
Tool use
Instead of guessing numbers, the AI asks a calculator. Instead of inventing weather, it asks an API. For fact-based tasks, reliable tools are used.
Let the AI prepare texts — not send them. At the latest before clicking “Send,” a human eye must intervene, whenever responsibility is at stake.
What this means in practice
Hallucinations are not an argument against AI — but an argument for the right architecture. We build systems that show sources, double-check critical fields, and label uncertain answers as such.
With the right measures, the risk is manageable. Without these measures, AI in operations quickly becomes a source of error that no one trusts anymore.
We build AI systems that show sources.
This way, your employees can see immediately where an answer comes from — and whether it is correct.
Schedule an initial consultation
Frequently asked questions
What is a hallucination?
Why does this happen?
Why is a human required for important tasks?
Legal notice and privacy policy: Impressum & Datenschutz (German only; the German version is binding).