You type a question and a paragraph appears, in order, at reading speed. It's hard not to picture something on the other side considering your question and then replying. That picture is wrong in a specific way, and correcting it is genuinely practical: nearly every odd behaviour you'll run into — invented sources, forgotten instructions, a wrong letter count, unshakeable confidence about something false — follows directly from how the thing actually operates. No mathematics below, and no equations pretending to be an explanation.
Want the same path as a short course? Open AI Earn on Google Play — that listing is the official app page.
The core loop: predict the next piece, then do it again
A chatbot runs on a large language model. Given everything written so far — the system's own instructions, your message, its own reply up to this point — it produces the next small piece of text. Then it takes that slightly longer text and produces the next piece. Then again, hundreds of times, until the answer is finished. That's the loop. There's no plan drafted in advance and no stored answer being retrieved; the response is assembled one step at a time.
Two things follow immediately. First, the model is always working from text that's in front of it, which is why what you include in your message has such a large effect. Second, the answer's shape emerges as it goes, which is why asking it to "think step by step" or to lay out its reasoning before concluding often improves the result — the intermediate text becomes part of what the next step is predicted from. Some products now do a version of this automatically in a slower "reasoning" mode, generating working before the visible answer.
Tokens: it doesn't see letters, and that explains a lot
The pieces aren't letters or words. Text is chopped into tokens — common chunks, roughly the size of a short word or a fragment of a longer one. "Understanding" might arrive as two or three chunks; a common word is usually one; an unusual name or a long number can be several. Everything the model reads and writes is in these chunks.
This is why a fluent assistant can fail at tasks a six-year-old handles. Counting the letters in a word, spotting whether a word has a double letter, reversing a string, working with the exact digits of a long number — these ask about the spelling inside chunks it doesn't examine that way. The failure isn't a sign that it's "dumb at maths"; it's a mismatch between the question and the units it works in. It also explains why usage limits are usually described in tokens rather than words, and why the same page of text costs more in some languages than others.
Where the "knowledge" came from
The model was trained on an enormous amount of text, and what it kept isn't a copy of that text. It's a very detailed sense of which words follow which, in what contexts — which turns out to encode a great deal about grammar, facts, styles of argument, and how a good explanation is structured. Then it was tuned further using human feedback about which responses are helpful, which is what turns raw prediction into something that answers politely and follows instructions.
So there is no lookup table to consult and no citation attached to anything. Well-established facts, repeated in countless documents, come out reliably. Rare, specific facts — a particular statistic, an exact quote, a page number — are far shakier, because a plausible-shaped answer is available even when the real one was never firmly learned. That's the mechanism behind invented sources: it's producing something that looks like a citation, because that's what fits.
Training also stops at a point in time. Anything that changed afterwards isn't there. Some products can search the live web to fill the gap, but that's a separate feature with its own limits — worth checking in your tool's own documentation, such as help.openai.com or Anthropic's help centre, rather than assuming.
Why a conversation feels like memory
The model doesn't remember your last message. What happens is simpler: every time you send something, the whole conversation is submitted again, and the model reads it fresh. Continuity is an illusion produced by re-reading the transcript each turn — a very convincing one, and it explains the limits precisely.
There's a maximum amount of text it can consider at once, the context window. Long conversations eventually press against it, and early instructions lose their grip while recent text dominates. This is why a thread that ran all afternoon starts contradicting itself, and why starting a fresh chat with a clean summary beats pleading with the old one. Separately, some products add a persistent memory feature that carries notes between conversations — that's a product feature layered on top, not the model remembering, and it varies by tool and setting.
Why the tone is always so confident
This one causes the most damage. The model has no separate sense of "how sure am I" that gets shown to you. It produces the text that best fits the request, and confident, well-organized prose is the pattern of a good answer in nearly all its training material. Wrong answers arrive in the same steady voice as right ones, and they're often more fluent, because a fabricated detail is unconstrained by awkward reality.
The tuning process adds a second effect: models are shaped to be agreeable and helpful, so pushing back tends to produce agreement rather than a defence. If you say "are you sure? I thought it was X", you'll often get a graceful correction to X — including when X is wrong. A changed answer under pressure is not evidence of anything.
| What you see | What's happening | What to do about it |
|---|---|---|
| A confident answer that turns out to be false | Plausible text was available; truth isn't a separate check | Verify claims outside the chat |
| A source or quote that doesn't exist | It produced something citation-shaped | Open every link; find the quote in the original |
| Miscounts letters or mangles a long number | It works in chunks, not characters | Use a tool that counts; ask for working |
| Forgets an instruction from earlier | The conversation outgrew the context window | New chat, re-paste goal and constraints |
| Says the current version is one that's now old | Training cutoff, no awareness of what changed | Check the vendor's live page for anything time-sensitive |
| Immediately agrees when you push back | Tuned to be agreeable | Ask for the reasoning, not for reassurance |
| Answers about a file or link it never received | It only has the text in the conversation | Paste the content; ask it to quote a line back |
| Two different answers to the same question | Output involves randomness by design | Ask twice on purpose when accuracy matters |
Check it yourself, in five minutes
Reading about this is less convincing than watching it. Three short experiments, each of which demonstrates one of the mechanisms above.
Prompt: "How many times does the letter r appear in [pick a long word], and in which positions? List each occurrence." Then count by hand. Try it with two or three words. When it slips, you're seeing the chunking, not carelessness.
Prompt: "Give me three published sources — author, title, year, and where to find them — on [a narrow topic you can check]. Then, for each one, tell me how confident you are that this source exists exactly as described." Now try to find them. Compare the failures with how sure the answers sounded.
Prompt: "Answer this question with your reasoning laid out step by step before your conclusion: [a question with several steps]." Ask the identical question in a fresh chat without the reasoning instruction. Compare. The difference is the loop working with more of its own text.
It doesn't know things the way you know things. It's extraordinarily good at producing text that reads like someone who does — which is useful far more often than it's dangerous, as long as you know which is which.
What not to do with this understanding
- Don't ask the model to explain itself and take the answer as fact. It has no privileged view of its own workings — it produces a plausible explanation of how such a system works, which may or may not match what happened.
- Don't treat "I'm not sure" or "I'm confident" as a real gauge. Those phrases are text like any other, not readings from an internal meter.
- Don't argue it into the truth. Repeated pushing changes answers reliably and improves accuracy unreliably. Go to a source.
- Don't conclude it's "just autocomplete" and stop there. The mechanism is simple; what emerges at scale is genuinely capable, and dismissing it costs you a useful tool.
- Don't assume every tool behaves identically. Web access, memory, file handling and reasoning modes are product features that differ between assistants and change month to month.
- Don't attribute intent to it. It isn't lying, being lazy, or trying to please you. Those words describe patterns in the output, not motives behind it.
What it changes in practice
Three habits follow from all of this, and they're the ones that separate people who get value from these tools from people who bounce off them. Give it context, because everything it has is what's in the conversation — the practical version is in writing better prompts. Verify claims rather than tone, using the routine in how to fact-check AI. And start a new chat sooner than feels necessary.
If a term here was new, the AI glossary for beginners defines the vocabulary properly, and the errors this mechanism produces show up as concrete habits in common AI mistakes beginners make. If you're at the start of all this, getting started with AI tools is the ordered path.
