Day 12 — File Handling

What is it? #

Working with a file has three steps: open it, use it, close it. Python's with statement handles the closing for you, even if something goes wrong in the middle.

The mode you open with decides what you can do. "r" reads, "w" writes and wipes any existing contents, "a" appends to the end, and adding "b" works with raw bytes instead of text.

Text files have an encoding. UTF-8 is the right default nearly always, and stating it explicitly avoids a class of bug where code works on one machine and fails on another.

For big files, do not read everything at once. Looping over the file object gives you one line at a time and keeps memory flat regardless of file size.

Think of it like this #

Opening a file is like borrowing a library book. You can read it or write notes in it, but you have to return it. The with block is a librarian standing at your shoulder who takes the book back the moment you stand up — even if you leave in a hurry because the fire alarm went off.

Simple example #

You have a log file that may be very large. You want to count error lines, write the matching ones to a separate file, and then append a one-line summary to a report.

Code #

from pathlib import Path
import csv

log_path = Path("app.log")
errors_path = Path("errors.log")

count = 0
with log_path.open("r", encoding="utf-8") as source, \
     errors_path.open("w", encoding="utf-8") as target:
    for line in source:                # streams one line at a time
        if "ERROR" in line:
            count += 1
            target.write(line)

print(f"Found {count} errors")

# Append a summary
with open("report.txt", "a", encoding="utf-8") as f:
    f.write(f"errors={count}\n")

# Reading a small file fully
if Path("notes.txt").exists():
    content = Path("notes.txt").read_text(encoding="utf-8")
    print(len(content.splitlines()), "lines")

# CSV, the right way
rows = [
    {"sku": "A-1", "qty": 3},
    {"sku": "B-2", "qty": 1},
]
with open("stock.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["sku", "qty"])
    writer.writeheader()
    writer.writerows(rows)

How it works #

Path("app.log") creates a path object. pathlib is the modern way to handle paths: it works the same on Windows and Linux, and it has handy methods like .exists(), .read_text() and .with_suffix().

The with line opens two files at once. Both are closed when the block ends, in either order of failure. The backslash is just a line continuation for readability.

for line in source: is the important detail. It reads lazily, one line at a time, so a 10 GB log file uses about as much memory as a 10 KB one. source.read() would load the whole thing.

target.write(line) writes exactly what you give it — no newline is added, which is why the loop works cleanly here (each line already ends with one).

Opening report.txt with "a" appends. Using "w" by mistake here would delete the entire report, which is a genuinely common accident.

The CSV block uses csv.DictWriter rather than manual string joining. It handles quoting and escaping, so a product name containing a comma does not silently corrupt your file. newline="" prevents blank lines between rows on Windows.

Real-world use #

File handling underpins imports and exports, log processing, report generation, backups and data pipelines. In a web app it is usually uploads: validate the size and type, write to a temporary location, then move it into place.

Streaming instead of loading is the difference between a script that runs on a laptop and one that gets killed on a server. The same principle applies when downloading large HTTP responses or reading query results.

Encoding problems are a real support cost. A CSV exported from a spreadsheet in one region may not be UTF-8, and reading it without specifying an encoding gives you either mojibake or a crash. Being explicit and handling UnicodeDecodeError saves hours.

Common mistakes #

• Opening with "w" when you meant "a", wiping an existing file.
• Forgetting to close a file by using open() without with. On long-running processes you eventually run out of handles.
• Calling .read() on a huge file and running out of memory. Loop over the lines instead.
• Skipping encoding="utf-8", then getting different behaviour on another machine.
• Building CSV rows by joining with commas by hand. Use the csv module so quoting is correct.

Practice #

Write a script that reads a text file, counts how many lines contain the word "WARN", writes those lines to warnings.txt, and appends a summary line to summary.txt. Handle the case where the input file does not exist without crashing.