What Are Tokens? The Atoms of Python

Every sentence in English is built from words. Every Python program is built from tokens — the smallest pieces the interpreter can recognize. Before you can write sentences, you need to know your words.

The five kinds of tokens

Take this single line of Python:

code

price = price + 10

The interpreter doesn't see it as one blob — it splits it into tokens:

code

price    =    price    +    10

Five tokens, five jobs:

Token	Kind	Role
`price`	Identifier	A name you invented
`=`	Operator	Does work (assignment)
`price`	Identifier	Your name again
`+`	Operator	Does work (addition)
`10`	Literal	A fixed value typed directly in code

The five families:

Keywords — Python's own reserved words (if, for, def, True...) — next lesson covers all of them

Identifiers — names *you* invent for variables, functions, classes

Literals — fixed values written directly: 10, "hello", 3.14, True

Operators — symbols that do work: + - * / = == < >

Delimiters — punctuation that structures code: ( ) , : . [ ]

Why should a beginner care?

Because every syntax error you'll ever see is a token problem:

pront("hi") → the identifier pront doesn't exist (typo)

print("hi" → a missing delimiter )

if x = 5: → wrong operator; = assigns, == compares

When you read an error message, you're really asking: *"which token is wrong here?"*

A peek behind the curtain

The interpreter's first job is tokenization — chopping your file into tokens before understanding anything. You can watch it happen:

code

import tokenize

with open("hello.py", "rb") as f:
    for token in tokenize.tokenize(f.readline):
        print(token.string, "→", token.type)

Run that on any Python file and you'll see the exact token stream the interpreter sees. It's proof: your beautiful code is, to the machine, a river of tokens.

The one idea to carry forward

Python is strict about tokens. Misspell an identifier, use = where you meant ==, forget a colon — the interpreter stops and complains immediately. This feels annoying for a week. Then you realize it's a superpower: Python catches your mistakes *before* they become bugs.

✅ Checkpoint

What are the five token families? *(Keywords, identifiers, literals, operators, delimiters)*

In age = 20, which tokens are which? *(age = identifier, = = operator, 20 = literal)*

Is "hello" a keyword, identifier, or literal? *(A literal — a string value written directly)*

Next: the most important token family — the 35 keywords Python reserved for itself.