To talk sensibly about superintelligence, it helps to know how today's systems like ChatGPT, Claude or Gemini work. The basic idea is surprisingly simple — what is complicated is what they learn from it.
1. Guessing the next piece of text
Finish the sentence: “The dog barks, the cat…” You probably thought meows. That is exactly the basis of language models: from the text so far, they predict what comes next.
They do not work with whole words but with tokens. A token can be a word, part of one, punctuation or a space; a long word may break into three tokens, and different models split text differently. For every possible next token the model computes a probability and the program then picks one. Sometimes the most likely, sometimes somewhat randomly among several good options — a setting called temperature controls how much randomness. That is why the same question does not always get the same answer.
Two warnings. First, probability is not truth: the model scores how well a continuation fits the text, not whether it describes reality. A widespread misconception is so common in texts that the model may happily repeat it, and lowering the temperature does not fix that. Second, the cat example explains the principle, not simplicity. To complete “This program crashes because…” or “The patient has a fever and a rash, so…” well, a model has to build useful representations of how the world works during training. “Token predictor” says nothing about how complex the computation behind it is.
2. Billions of knobs
Inside the model is a neural network — a mathematical model with billions of adjustable values called parameters. Picture a giant mixing desk with billions of knobs. Unlike a mixing desk, though, a single knob usually cannot be labelled “English” or “arithmetic”; knowledge is spread across many of them.
3. Training: try, compare, adjust
The model gets a piece of real text, tries to predict the continuation and the result is compared with reality. An optimisation algorithm then turns all the knobs a tiny bit in the direction that would reduce the error. And again. And again — on vast amounts of books, web pages, articles and code. Nobody writes facts into the model by hand; they settle into the knob settings on their own.
Think of it this way
It is like a child learning their first language: nobody explains the grammar rules, the child hears thousands of sentences and corrects itself. The difference is scale — during training a model “reads” more text than a person could in many lifetimes.
A key role is played by the transformer, an architecture from 2017 (Attention Is All You Need). At its core is attention: when processing each piece of text, the model weighs which other parts of the input matter for it. The transformer is the most widespread today, but not the only option — a different approach is tried by, for example, Mamba.
Three words you will hear often: training changes the parameters, fine-tuning is further training aimed at particular tasks or behaviour, and inference is using the finished model to compute an answer. A bit like studying and an exam: studying changes what you know, the exam uses it.
4. From model to assistant
After initial training a model can continue text, but it is not yet a useful assistant. Next comes fine-tuning: examples of good answers, ratings from people (“this helped, this didn't, this is dangerous”) and automatic verification — for a program you can run a test and check whether it really works. An example of multi-stage development is described in the technical report The Llama 3 Herd of Models.
Importantly, the model is not retrained during an ordinary conversation. Your prompt and earlier messages serve as working context, not new knob settings. If you show the model three examples of what the answer should look like, it adapts to them — but only for this conversation; the parameters stay the same. This ability to learn from examples in context without retraining was described as early as the GPT-3 paper.
5. Three different meanings of memory
When someone says an AI “remembers” something, it can mean three different things:
- Parameters: what the model learned during training. Not a database in which every sentence it read could be reliably looked up.
- Context: the prompt, earlier messages and documents the model has in front of it right now. Only a limited amount fits (the context window), and even a long input does not guarantee that the model uses every detail correctly.
- External memory: files, databases or notes that the application stores and loads again in the next conversation.
The third option is the basis of RAG (retrieval-augmented generation): the application first searches for material, for instance in company documents, and adds it to the model's context. The model is not retrained by this; it simply has something to draw on. Answers tend to be better grounded, but only as good as the material found — a wrong or incomplete document gives a wrong answer. Original RAG paper.
6. Model, app and agent
When you use an AI assistant, you are working with a whole system. A language model can be connected to search, a calculator, company documents or an environment for running code — and the resulting capability depends on those tools too. Two products using the same model can therefore behave quite differently.
An AI agent also carries out several steps towards a goal. The request “prepare materials for the meeting” becomes finding documents, comparing them and writing a summary. With the right permissions it can also change files or send emails. The same model with different permissions can have very different effects — whoever gives an agent access to an account changes what it can do in the world. More: AI agents.
7. Why models get things wrong
A hallucination is output that sounds plausible but is wrong or unsupported: a non-existent citation, a mixed-up date, an invented explanation. The model learned what a good answer sounds like — and fluency is not a measure of truth. Access to search helps, but does not rule out a bad source or a wrong interpretation.
Hallucination has a subtler form, too. If a model summarises a document and adds a claim that is not in it, it has failed at working with the source even if the added claim happens to be true. So for every use it is worth knowing what exactly the output is supposed to do: stick to the document, or answer according to the world?
Be wary, too, of the popular claim that new abilities “switch on by themselves” with every increase in model size. Some apparent jumps depend on how they are measured (Are Emergent Abilities… a Mirage?). The benefit of scale is real — that it continues all the way to ASI is not proven.
And a word on “reasoning”. Newer models can write out their approach before answering and solve multi-step problems. That is real progress in performance, but by itself it does not prove human-like thinking — and a long, confident explanation can be wrong just as a short answer can. How it works and where its limits are: AI layers and reasoning.