Token definition

A small unit of text a language model reads and generates; it may be a word, part of a word, or punctuation.

What it actually is under the hood

A tokenizer converts text into integer IDs from a fixed vocabulary. Model pricing, context limits, and output speed are usually measured in these units.