Notes from working through Cursor Learn — a course on how AI models work, what their limits are, and how to use them well as a developer. Lee Robinson frames models as “super-intelligent API endpoints”: you solve almost any task through an API call. Unlike normal system APIs, which are deterministic, AI models are probabilistic.
There are four dimensions that inform which model you should choose for a task:
- Time
- Money
- Reliability
- Effort
How AI Models Work
Deterministic vs Probabilistic
The computer science of old was built on deterministic systems. There are a set of rules and protocols that should be followed to get a result for a particular kind of task input. There is a structure to the way things work.
AI models in the 20th century were deterministic, because they were built on the computational need and frameworks of that era. Early chess engines and industrial automation used deterministic AI because of the rules and level of precision needed in such environments.
Today computational power has increased significantly, allowing complex calculations of complex probabilities. Probabilistic AI stems from Bayesian statistics, the foundation of statistical and probabilistic frameworks.
Recent AI models use probabilistic frameworks. This means the output these models produce don’t follow a set-in-stone path; rather, they provide many paths for a given input. These models perform better in unpredictable and unstructured environments.
AI models (recent ones… Kimi K3, Gemini 3.1 Pro, Sonnet 5, Fable 5…) have an uncertainty score for every answer they produce. Some are better at providing lower uncertainty scores, in specific areas, than others. It makes sense that platforms like ChatGPT, Google Gemini, and Claude all have the warning text in the chat interface: AI can make mistakes.
Hallucinations
Hallucinations happen when AI models confidently provide seemingly reasonable results to an input, when the answer given is actually wrong.
AI models are trained on large sets of data. Anytime they receive a question, they look through their training data and produce the most plausible answer based on the statistics of their training. AI models don’t “know” what is actually true or actually false — they just “know” what fits the pattern in the training data.
Why Do Hallucinations Occur
AI models are rewarded for accuracy and punished for errors during training. For the model, it is better to make a guess and get it right than to admit to not knowing. The probability of getting a reward if they make a guess is greater than if they don’t attempt it at all.
OpenAI’s research on the subject uses a multiple-choice question as an analogy: if you don’t know the answer, it’s better to make a guess. If you don’t guess, you get a zero score and are punished; however, guessing an answer out of four possibilities gives you a 1/4 chance of getting the correct one.
Because current evaluation methods and scoreboards for training models focus on accuracy, it’s very difficult to solve the problem of hallucination.
Researchers have tried to punish models for confidently giving wrong answers more than for admitting to not knowing. This does not solve the problem of hallucination entirely, because guesses that are correct still get rewarded — so the model still learns that if it ups its guessing game, it doesn’t get punished.
Aside from problems in the way models are trained, there is also the issue of knowledge cutoff dates. If the cutoff date of a model is in 2025, it might hallucinate any results that only exist in 2026, after it was released.
How To Fix The Limitation
Hallucinations cannot be fixed. At least not yet. We can only put in measures, be masters of the tool, and not delegate decision making to the models.
You have to know the capabilities of the models being used, what kind of training was used to produce the model, and what the knowledge cutoff date is.
Having this in mind will help you spot hallucinations in the model’s responses and correct the model. It will also reduce how often you get angry at a model for confidently misleading you.
Tokens
Tokens are to LLMs what bytes are to traditional computers. Tokens are the basic units of text models use to read and write language.
When we send text input to the model, it converts the raw text into tokens through a process called tokenization (Raw Text → Token Strings → Token IDs → Vectors). The model reads the tokens and produces a response by converting the tokens back to text through detokenization (Output Vector → Probabilities → Token ID → Token String → Reconstructed Text).
Tokenization varies from language to language, and from model to model. Tokenization of the English language is easier because the majority of the data used to train these models is in English.
Commercial AI companies use tokens as a way to price their models. Each company uses different algorithms and techniques for tokenization and detokenization, so the prices are not the same across the board.
They are also what measures the speed of a model. Tokens Per Second (TPS) is measured to see how fast the model returned back to the user.
Because it’s harder for models to read languages that are not English, token prices for those languages are higher.
LLMs can, however, build up cached tokens — reused tokens from prior context. The LLM recognizes some words and characters over time and limits the number of tokens assigned to those words or characters.
The models also have different context windows. A context window is the limit on the model’s token budget when you combine both input tokens (when you send the model a message / ask a question) and output tokens (when the model responds / answers your query).
Output tokens cost more than input tokens because LLMs work harder to gather new content in order to answer your query.
Context
Every model has a context window or context limit. So for every conversation thread, the context grows until it hits its limit.
Context windows consist of the input tokens, output tokens, system prompts, and tool calls. The model’s training data is not part of this. Context is “live” memory that is stored by the model as in a conversation thread.
NB: System prompts are foundational, hidden instructions given to an AI model to set its role, tone, and behavior before a user chat begins. These often come from the creators of the models.
A single conversation turn (input tokens, output tokens) gets added to the input tokens of the next message, in that order. This adds to the context window over time.
As the context window gets bigger and bigger, the model begins to “forget” the very first conversations that you had. This is known as context rot. LLMs have to calculate the distance and relationships between “similar” tokens, mostly at the beginning and end.
The more the distance (context window) increases, the harder it becomes for the model to calculate the relationship and distance between tokens — leading to context rot.
Managing Context
You can make good use of the context window by:
- Optimizing your prompts — Use popular or common words in your prompts. The models recognize these words better than made-up or complex words.
- Separation of concerns — Start a new chat if your next query has nothing to do with the current conversation thread. You will save space in the context window, since starting new chats spins up a fresh context window.
Tool Calling
Tool calling is giving AI models the ability to call API endpoints. The model does this via Retrieval-Augmented Generation or live web/API access. The model learns new skills from those endpoints in order to achieve a task.
One of the discussed causes of LLM hallucinations was the knowledge cutoff dates of the models. Tool calling is one of the primary ways to bypass this limitation. A model engages tool calling when it recognizes that the query requires it, or when the query states explicitly to do so.
How Does Tool Calling Work
Step 1: Recognizing The Need For A Tool
We have already established that AI models have a hard time admitting to “not knowing”, so they guess instead, due to the nature in which the models are rewarded and punished during training.
So how exactly does the model now know to admit its limitations?
System prompts. In the context layer we saw that, as part of the context, there are system prompts given to the models to serve as persistent instructions for how they should behave.
This is one of the ways the model can know to execute a tool call from a query when it needs to.
A system prompt could look like this — for example, telling the model which tools it may call:
[{ "name": "web_search", "description": "Fetch live web data after the knowledge cutoff.", "parameters": { "type": "object", "properties": { "query": { "type": "string" } } }}]Supervised Fine Tuning (SFT) on “uncertainty signals.” To solve the problem of rewarding, AI labs train models on datasets that punish static answers for dynamic data. The models have static knowledge — and it’s based only on their training data.
A model can be trained on data on countries in the world, so it can answer the question:
What is the capital of France?
However, it will most likely hallucinate on a question like:
What is the weather in France today?
This kind of information is dynamic; thus, AI models have been made to recognize this and default to tool calling instead of giving false information.
Step 2: Selecting The Tool
Once the model identifies the need to call a tool, it looks for the best fit among the tools it has access to.
Each tool contains metadata like the function name, description, tool parameters, and the required input and output types. With retrieval-augmented generation, the model is able to retrieve both structured and unstructured data and produce structured output.
It can be argued that the handshake step where the model verifies the tool’s identity uses deterministic frameworks, since we already know what to expect from the exchange.
Step 3: Constructing And Sending The Query
The model creates a request that the API can understand. Tools have their specific functions, which define what the tool does. API requests are sent over HTTPS to an external server.
Step 4: Receiving And Parsing Data
When the external tool returns data, the model refines and structures the data in a way that is meaningful to the user.
Tool Calling Token Usage
It’s important to note that tool calls also consume tokens, thus contribute to filling the context window. Input tokens (requests) are optimized, so not many tokens are consumed. However, output tokens (tool or API response) return a lot of data or actions, so they consume loads of tokens and can fill up the context window fast.
Agents
Agents are LLMs that run tool calling in a loop until a task is complete.
They are able to determine the next tool or API that is needed for the next step of a task and keep calling and checking for the next, in the iteration.
Agents can use either the ReAct (Reasoning and Action) paradigm — which instructs agents to “think” and plan for the next tool to use after each action, and with each tool response. This is known as the Think-Act-Observe loop, used to solve problems step by step and improve responses.
Or ReWOO (Reasoning Without Observation). This paradigm tells the model to plan ahead and determine all the tools it needs to finish a task. Only then does it start work. This is essential for preventing redundant tool calling and usage (which can burn tokens for useless results).
I am working on a deeper dive into agents. Come back later to read.