Mental Model
You just watched AI help and stumble in the same ten minutes. Now you will understand why, at a level that changes how you prompt and when you trust the output. You do not need the math. You need one accurate mental model that explains the confident wrong answers, the same-question-different-answer weirdness, and why the tool reflects the past.
The Concept
A large language model generates text by predicting the next word, over and over, based on patterns it learned from a huge body of training text. It is not looking up a fact in a database. It is producing the most statistically likely continuation of whatever you wrote. Three consequences follow directly, and all three matter for your work.
Flip each card to understand the consequence.
Tap to understand why
The model is optimized to produce fluent, well-formatted text, not accurate text. It has no internal signal that says "I am not sure about this." Confidence is a property of the format, not of the truth.
Tap to understand why
The process samples from a probability distribution rather than retrieving one stored answer. The same prompt can give different answers on different days. This is normal, not a bug, but it means you cannot treat AI output as a fixed reference.
Tap to understand why
Because it learned from past text, it mirrors what was common in that text, including outdated assumptions and bias. This shows up most dangerously in people decisions, where historical discrimination gets encoded and presented as neutral analysis.
Why This Matters Now
That is not a reason to panic, and it is not a reason to hand work over. It is a reason to learn to direct the tool well, because it is now genuinely capable of touching the core of your job. The accurate mental model above tells you exactly when to trust AI output, when to verify it, and when to keep it out of the room entirely.
In Practice
Dana's story
Dana asks an AI tool, "What are our company's headcount planning guidelines for Q3?" The model returns a specific, professional-sounding answer with clear policy language. It is completely made up. The model has no access to Dana's company policy, so it produced the most likely-sounding generic HR text instead. This is the canonical hallucination: the model cannot tell the difference between "what I learned about HR in general" and "what this organization's actual policy is," and it will never warn you which one you are getting.
Apply It Today
Ask an AI tool one factual question you already know the answer to: a company-specific detail, a number, a policy. Notice how confidently it answers, whether it is right or not. That confidence is a property of the format, not of the truth.
Reflect