Understanding AI: Tokens

Definition

Tokens are numerical representations of words, subwords, or characters generated through a process called tokenization. Tokenization translates text into a format that large language models (LLMs) can process. Tokens can capture both words’ explicit meanings and their contextual nuances. 

An average token represents approximately 0.75 words. 

Need to know

Token count is one of the key drivers of an LLM’s cost. For example, OpenAI’s prices begin at $5/1M tokens of prompts. 

Useful to know

• While LLMs don’t inherently understand human languages, this mechanism allows them to interpret and generate it. This can also allow LLMs to understand and process multiple languages, as tokens can be language-agnostic, based on patterns rather than specific language constraints. 

• Once an LLM generates an output in tokens, it must be “de-tokenized” using the reverse of the tokenization process to map it to words a human can comprehend. 

• There are various methods one can use to tokenize text, including Byte-pair encoding (BPE), word-level tokenization, subword-level tokenization, and character-level tokenization. 

Related terms

• Context

• Embedding

• Vectors