How Large Language Models Work: Tokens, Context and Why AI Gets Things Wrong

Most explanations of large language models (LLMs) fall into two camps. Some just shrug and say, "It's only predicting words." Others read like a research paper. If you use these tools at work, you need something in between. You need a simple mental model that helps you guess when an answer will be good and when it will not.

A model writes one token at a time

A model does not read or write whole words. It works in tokens, which are short chunks of text. A common word is often one token. A rare word may split into pieces. Punctuation counts too. In English, a token is about three quarters of a word.

The model writes one token, adds it to everything it can already see, then writes the next. This has two effects. First, an answer can start well and then drift, because every early choice shapes the later ones. Second, jobs that depend on exact letters or counts often go wrong. "Write exactly 200 words" is one example. "Reverse this name" is another. The model sees pieces, not letters, and it has no built-in word counter.

Smooth writing is not the same as knowing

A model is trained to continue text in a believable way. That makes it very good at the look of a right answer. A real source and a fake one have the same shape: author, title, year, journal. Nothing inside the model separates "studies that exist" from "text that looks like a study."

So a model is not a database. It holds a huge set of numbers, called weights, that were adjusted during training. It does not keep a copy of each document it saw. When a fact has to be right, put the source in your prompt and ask the model to quote it. Then check the quote.

The context window is a budget

Everything the model can use in one turn has to fit in its context window. That includes your instructions, any files you add and the chat so far. When a chat grows too long, the product may stop with an error, cut old messages, shorten them or pull out a few parts. You notice it as a tool that seems to forget what you said an hour ago. A fresh chat with a short recap often works better than a very long one.

Why one question can get two answers

At each step, the model gives a score to many possible next tokens. A setting called temperature controls how bold the pick is. Lower settings tend to give steadier results, but not identical ones. That is why asking twice can give two different answers. It is also why one good run proves very little.

Training happens in stages

A model is usually built in stages. First it learns patterns from a huge body of text. Then it is tuned on examples of good replies. Then it is adjusted using human feedback. The later stages shape its tone, its style and what it refuses to do. They do not turn it into a fact-checker.

A practical way to pick a model

Public tests are a starting point, not the final word. A better test uses prompts from your own work. Run the same set on each model and judge the results by what you would accept from a coworker. Fifty well-chosen prompts will tell you more about fit than a spot on a leaderboard.

Take this into your next session

Think of the model as a weather forecaster, not an almanac. It does not look up the past. It guesses what text is likely to come next, based on the conditions you give it. Give it good conditions, and check the details that matter.