Read large CSV files in Python without loading every row

Read a large CSV by iterating over an open file rather than building a list of every row. This makes the working memory depend mainly on the current record and your accumulated state. It does not remove the need to validate records, choose a numeric representation or plan what happens when a row is unusable.

Our example totals invented workshop purchases. It deliberately includes a quoted comma, a quoted newline and invalid amounts. The fixture is small enough to inspect; the same loop can read a file much larger than memory because it never retains the full table.

Define the input contract

The two columns are item and amount. Amounts must be finite, nonnegative decimal numbers. An empty item, a missing field, an extra field or an invalid amount makes a record invalid. We count rejected records and continue; a real accounting workflow might instead stop and quarantine the entire import. Choose that policy before relying on the total.

Use newline="" when opening a CSV file so the CSV parser handles embedded line endings. A physical line is not necessarily one record: the quoted item containing a newline still belongs to one purchase. Splitting lines on commas would mishandle both that record and the item named paper, blue.

The code writes a temporary UTF-8 fixture, then processes it using the same file API as a real import. Replace only the path to use your own export. Decimal preserves the decimal arithmetic for these prices; binary floating-point arithmetic would introduce a different rounding policy.

import csv
from decimal import Decimal, InvalidOperation
from pathlib import Path
from tempfile import TemporaryDirectory

fixture = 'item,amount\n"paper, blue",12.50\n"kit\\nlarge",7.25\npen,broken\nclip,-1\nfolder,NaN\nlabel,2.25,extra\n'
fixture = fixture.replace('kit\\nlarge', 'kit\nlarge')
with TemporaryDirectory() as directory:
    path = Path(directory) / 'purchases.csv'
    path.write_text(fixture, encoding='utf-8')
    total = Decimal('0')
    accepted = rejected = 0
    with path.open(encoding='utf-8', newline='') as handle:
        reader = csv.DictReader(handle)
        if reader.fieldnames != ['item', 'amount']:
            raise ValueError('Unexpected CSV header')
        for row in reader:
            try:
                if None in row or not row['item'] or row['amount'] is None:
                    raise ValueError('Wrong fields')
                amount = Decimal(row['amount'])
                if not amount.is_finite() or amount < 0:
                    raise ValueError('Invalid amount')
            except (InvalidOperation, ValueError):
                rejected += 1
                continue
            total += amount
            accepted += 1
    print(f'accepted={accepted} rejected={rejected} total={total:.2f}')
    assert (accepted, rejected, total) == (2, 4, Decimal('19.75'))

Checked output:

accepted=2 rejected=4 total=19.75

Account for rejected data

The total is 19.75 for the two accepted records, not a total for the entire export. Returning only that number would hide four rejected purchases. Preserve rejection counts and, when appropriate, a separate error report containing a safe record identifier and reason. Do not print sensitive source rows into application logs.

Errors in the CSV syntax itself are a separate concern from invalid field values. A malformed quoted field can raise csv.Error; our numeric exception handler does not claim to repair broken CSV structure. Specify the delimiter, encoding and header rules for the actual supplier. A byte-order mark may require utf-8-sig, and a regional export may use a semicolon rather than a comma.

Understand the memory boundary

The parser still needs to hold a record, and a gigantic quoted field can be a problem. Python also has a CSV field-size limit. Streaming is therefore a useful memory strategy, not a guarantee that every possible input can be processed safely. Sorting all rows or keeping an unbounded dictionary of unique customers would add memory requirements even inside this loop.

For operations that need tables, pandas chunked reads can be appropriate. Combine each chunk into bounded summaries; concatenating all chunks reconstructs the full-memory problem. A database or an external sort may fit operations that need global ordering or joins.

Extend the import

Add an explicit currency field and reject mixed currencies rather than summing them. Then test an empty file, a header-only file and an amount with more precision than the business allows. Decide whether to reject or round before adding any quantization.

The Python course builds the file, function and exception skills used here. Continue with reading a traceback when a file-processing function fails, or adapt one of the complete beginner projects to read an external CSV.

References

Python CSV module — Reader behavior, dialects, newline handling and field-size limits.