What is inference in AI?
Inference is the moment a trained AI model does its job. It takes your input and produces an output.
Key takeaways
- Training builds the model one time. Inference happens every time someone uses it.
- Each answer you get from a chat tool is one inference run.
- Inference is what AI providers charge for, usually by the token.
How it works
Your message goes to a data center, where the model runs on specialized chips. The model reads the tokens, computes the most likely next token, adds it to the answer and repeats until the answer is complete. The text you watch appear on screen is that process.
Example
A support team connects an AI model to its help desk. Each customer question that the model answers is one inference call, and the monthly bill adds up those calls.
Why it matters
When you compare AI products or plan a budget, the cost and the speed of inference matter more than the cost of training, which the provider already paid.
Common mistake
A model does not learn from your chats during inference. It answers with what it already has, plus what is in the context window.
Related terms
See all terms in the AI Encyclopedia, the numbers on the AI statistics pages, or build a prompt with the prompt builder.