Python File Handling: Reading and Writing Files

Reading and writing files is one of the first things people do in Python, and one of the easiest to get subtly wrong. The code works on your machine, then fails on a colleague's with a UnicodeDecodeError, or a crash halfway through a write leaves an empty file where yesterday's data used to be. A handful of habits prevent almost all of it.

Open modes: say exactly what you intend

open() takes a mode that decides what happens when the file does and does not exist (Python docs: open):

Two letters modify these modes. Adding b, as in rb or wb, opens the file in binary mode: you read and write bytes, with no decoding. Adding +, as in r+, opens it for both reading and writing, to update a file in place.

The dangerous one is w: it truncates the file the moment it opens, before you write anything. When a script must never overwrite existing output, use x, which refuses:

from pathlib import Path

Path("report.txt").write_text("first run\n", encoding="utf-8")
try:
    with open("report.txt", "x", encoding="utf-8") as f:
        f.write("second run\n")
except FileExistsError as exc:
    print(type(exc).__name__, exc.filename)

FileExistsError report.txt

Always pass an encoding

In text mode, Python decodes bytes to strings. If you do not say which encoding to use, it uses the platform's locale encoding (Python docs: open), which differs between machines:

import locale

print(locale.getpreferredencoding(False))
Path("cities.txt").write_text("Hyderabad\nBengaluru\nPuducherry – UT\n", encoding="utf-8")
try:
    open("cities.txt", encoding="ascii").read()
except UnicodeDecodeError as exc:
    print("decode failed at byte", exc.start)
print(open("cities.txt", encoding="ascii", errors="replace").read().splitlines()[-1])

cp1252
decode failed at byte 33
Puducherry ��� UT

On this Windows machine the default is cp1252, while most Linux and macOS systems default to UTF-8, so a file written on one and read on the other can fail or silently garble characters such as the en dash above. Python plans to make UTF-8 the default everywhere in version 3.15 (PEP 686); until then, pass encoding="utf-8" to every text-mode open() call. errors="replace" reads past bad bytes instead of failing, which suits log files, but it destroys information, so never use it on data you intend to keep.

Reading: iterate, do not slurp

A file object is an iterator over its lines, which is the memory-safe way to read a file of any size (Python tutorial: reading and writing files):

with open("cities.txt", encoding="utf-8") as f:
    for number, line in enumerate(f, start=1):
        print(number, line.rstrip("\n"))

1 Hyderabad
2 Bengaluru
3 Puducherry – UT

f.read() returns the whole file as one string, and f.readlines() a list of every line; both hold everything in memory at once. They are fine for small files, but a 5 GB log needs the loop. Each line keeps its trailing newline, so strip it explicitly. The with block closes the file when it ends, even if an error occurs.

Paths with pathlib

pathlib.Path treats paths as objects instead of strings, so joining, inspecting and listing work the same on every operating system (Python docs: pathlib):

out = Path("exports") / "2026" / "09"
out.mkdir(parents=True, exist_ok=True)          # create the whole tree, no error if present
(out / "sales.csv").write_text("region,amount\n", encoding="utf-8")

for p in sorted(Path("exports").rglob("*.csv")):
    print(p.as_posix(), p.stat().st_size, "bytes", p.with_suffix(".parquet").name)

exports/2026/09/sales.csv 15 bytes sales.parquet

The file is 15 bytes, not 14, because this example ran on Windows, where text mode writes each \n as \r\n; on Linux and macOS it would be 14. Pass newline="" (or use binary mode) when the exact line endings matter. The / operator joins path parts, mkdir(parents=True, exist_ok=True) creates missing folders without failing if they exist, and rglob searches subfolders. read_text and write_text open, read or write, and close in a single call, which is perfect for small files.

Overwrite without risking the old file

Writing a file in place has a failure mode: if the program crashes after open(..., "w") truncates the file but before the write finishes, the old content is gone and the new content incomplete. The safe pattern is to write a temporary file next to the target and then swap it in with os.replace, which replaces the destination in one step (Python docs: os.replace):

import os
import tempfile

def write_atomically(path: Path, text: str) -> None:
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name, suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            f.write(text)
        os.replace(tmp, path)            # the reader sees either the old file or the new one
    except BaseException:
        os.unlink(tmp)
        raise

write_atomically(Path("report.txt"), "second run, complete\n")
print(Path("report.txt").read_text(encoding="utf-8").strip(), [p.name for p in Path(".").glob("*.tmp")])

second run, complete []

The temporary file lives in the same folder so the rename stays on one file system. Anything reading the report sees either the complete old version or the complete new one, never half of each. Use this for configuration files, caches and any output another process reads.

Errors files actually raise

• FileNotFoundError: the path does not exist. Check the working directory, since relative paths are resolved from where the program was started, not where the script lives.
• PermissionError: no access, or on Windows, another program has the file open.
• IsADirectoryError: the path is a folder.
• UnicodeDecodeError: the encoding is wrong, as shown above.

All of these are subclasses of OSError except the last, so a single except OSError covers file-system problems. Catch what you can act on; error handling in Python covers the approach.

Most real files are a structured format; reading and writing CSV files and working with JSON in Python cover the two most common. For whole folders of files, the ten automation scripts put these tools to work. Reading and writing files, text files, CSV and JSON handling and file management make up the file handling module of the Python Programming course.

Sources

• Reading and Writing Files (tutorial) — Python documentation
• open() built-in — Python documentation
• pathlib — Object-oriented filesystem paths — Python documentation
• os.replace — Python documentation
• PEP 686 – Make UTF-8 mode default — Python Enhancement Proposals