What is

token?

Tokens are the building blocks language models read and write. Here is how they affect price, length and quality in AI tools.

  • Glossary
  • Token (in AI and language models)

What is a token?

A token is the smallest unit a language model works with. The text is split into pieces that can be whole words, parts of words, punctuation or spaces. The model reads tokens in and produces new tokens out, one by one. The number of tokens determines both the price and how much text the model can handle.

The splitting is called tokenisation. Common English words often become one token, while long Norwegian compound words are often split into several pieces. That means the same content can become more tokens in Norwegian than in English. An LLM therefore does not see letters or words the way we do, but number codes for each token.

Why do tokens matter for marketers?

Tokens are relevant far beyond the development department. When you use generative AI in your marketing work, tokens affect three things:

  • Cost: API access to language models is usually priced per token, often with different prices for input and output.

  • Context window: Each model has a limit on how many tokens it can take in at once. Long documents, chat logs and attachments eat into this quota.

  • Quality: When the context gets too long, the model may overlook details. Short, precise instructions often give better answers.

For AI search, this means models often only retrieve and read excerpts of a web page. Content that answers clearly and early has a greater chance of being used. That is one of the reasons we recommend short, self-contained answer paragraphs when you work with AI and visibility.

How to work smart with tokens

  • Write concise instructions in each prompt, and remove unnecessary background text.

  • Split large tasks into smaller steps instead of pasting in everything at once.

  • Give the model only the relevant excerpts of a document, not the whole document.

  • Monitor token usage if you build your own solutions on the API, so the budget does not run away.

  • Structure website content with clear headings and short paragraphs, so the right pieces are retrieved.

A commonly used rule of thumb is that one token equals about three quarters of an English word. In Norwegian, the ratio is usually somewhat worse, so expect a few more tokens for the same amount of text.

Related terms

  • LLM: the language model that reads and produces tokens.

  • Prompt: the instruction you give the model, measured in tokens.

  • Generative AI: AI that creates new content token by token.

  • RAG: retrieves relevant pieces of text into the context, so the model gets the right tokens to work with.

See all the terms in the glossary.

Want to use AI smarter?

We help you adopt AI in your marketing and become visible in AI search.