Day 19 — Iterators and Generators

What is it? #

An iterator is an object that hands you one item at a time and remembers where it stopped. Every for loop is really asking an iterator for the next item until there are none left.

A generator is the easy way to write an iterator. Put yield in a function and Python turns it into something that produces values lazily, pausing after each one and resuming when asked for the next.

The reason to care is memory. A list of ten million records must exist in RAM all at once. A generator produces them one at a time, so memory stays flat no matter how long the sequence is.

Generators are also the natural way to express infinite or unknown-length sequences: lines from a growing log file, pages from an API, IDs until you decide to stop.

Think of it like this #

A list is a full water tank delivered to your door. A generator is a tap. If you only need two glasses, the tap wastes nothing, and the tap keeps working even if the total supply is larger than any tank you own.

Simple example #

You need to process a huge CSV of transactions: filter to a single account, convert each row, and total them. Loading the whole file would be wasteful, so you build a small pipeline of generators.

Code #

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")        # one line at a time


def parse(lines):
    for line in lines:
        account, amount = line.split(",")
        yield {"account": account, "amount": float(amount)}


def only(records, account):
    for record in records:
        if record["account"] == account:
            yield record


total = sum(r["amount"] for r in only(parse(read_lines("txns.csv")), "AC-1"))
print(total)

# A generator is paused code, not a list
def countdown(n):
    while n > 0:
        yield n
        n -= 1
    return "done"

gen = countdown(3)
print(next(gen))      # 3
print(next(gen))      # 2
print(list(gen))      # [1]  — only what is left
print(list(gen))      # []   — generators are exhausted once used

# Infinite sequence, safely consumed
def ids(prefix):
    n = 1
    while True:
        yield f"{prefix}-{n}"
        n += 1

from itertools import islice
print(list(islice(ids("ORD"), 3)))    # ['ORD-1', 'ORD-2', 'ORD-3']

How it works #

read_lines looks like a normal function but contains yield, so calling it does not run the body. It returns a generator. The body only starts when something asks for the first item.

Each yield hands back a value and freezes the function exactly there, including local variables and the open file. The next request resumes from that point.

The three generators chain together: read_lines feeds parse, which feeds only, which feeds sum. Nothing builds a list at any stage. One line moves through the whole pipeline before the next one is read, so a 5 GB file uses a few kilobytes of memory.

next(gen) pulls a single value manually — that is what a for loop calls behind the scenes. When the generator finishes, it raises StopIteration, which the for loop catches and treats as "done".

The double list(gen) shows the sharp edge: a generator can only be consumed once. After it is exhausted, it stays empty. If you need to iterate twice, keep a list or create the generator again.

ids never ends. That is fine because islice takes only the first three. Infinite generators are safe as long as the consumer decides when to stop.

Real-world use #

Anything that reads large files, streams query results, paginates an API or processes a message queue is better as a generator. The pattern of chaining small generators is a common alternative to a heavyweight data pipeline framework.

Database libraries use this idea: a server-side cursor yields rows in chunks instead of transferring the whole result set. So does streaming an HTTP response body.

Generators also keep web responses responsive. Streaming a large CSV export row by row means the user starts downloading immediately and your server never holds the full file in memory.

The trade-off is that you cannot index a generator, cannot take its length, and cannot reuse it. When you need random access or multiple passes, a list is the right choice.

Common mistakes #

• Consuming a generator twice and getting nothing the second time.
• Calling len() on a generator. Convert to a list first, or count as you go.
• Wrapping a generator in list() immediately, which throws away the memory benefit.
• Opening a file inside a generator and never consuming it fully, leaving the handle open longer than expected.
• Using a generator where you need random access by index.

Practice #

Write a generator chunks(items, size) that yields lists of at most size items. Use it to print a list of 10 numbers in groups of 3. Then write a generator that reads a text file and yields only the lines containing "ERROR", and count them without building a list.