Memory-Efficient Generator for Large Files

🐍 Code Snippet of the Day — July 6, 2026

Memory-Efficient Generator for Large Files

Process multi-GB CSV/JSONL/logs without blowing RAM. Chunked reading, batched yields, progress tracking & auto-cleanup.

import os, tempfile
from pathlib import Path
from typing import Callable, Iterator, Optional

def process_large_file(
    filepath: str, chunk_size: int = 1048576,
    batch_size: int = 1000, encoding: str = "utf-8",
    on_progress: Optional[Callable[[int,int],None]] = None,
) -> Iterator[list[str]]:
    """Yield batches of lines from a large file."""
    path = Path(filepath)
    total = path.stat().st_size
    if total == 0: return
    buf, read = "", 0
    with open(path, "r", encoding=encoding) as fh:
        while True:
            chunk = fh.read(chunk_size)
            if not chunk:
                if buf.strip(): yield [buf.strip()]
                break
            read += len(chunk); buf += chunk
            parts = buf.split("
")
            lines, buf = parts[:-1], parts[-1]
            for i in range(0, len(lines), batch_size):
                if b := lines[i:i+batch_size]: yield b
            if on_progress: on_progress(read, total)

# Usage:
for batch in process_large_file("data.jsonl",
    on_progress=lambda d,t: print(f"
{d/t*100:.1f}%")):
    # process 1000-line batch
    pass

Pro tips:

• 1MB chunks + 1000 batch = peak memory <20MB


• Partial lines across chunks handled transparently


• Chain with tqdm() for progress bars


• Peak memory ≈ chunk_size + batch_size × avg_line_length

💡 "Works on my laptop" → "runs in production on 2GB instances."