Chunked File Reader — Process Multi-GB Files Without MemoryError

🐍 Code Snippet of the Day — July 23, 2026

Chunked File Reader — Process Multi-GB Files Without MemoryError

When pandas.read_csv() on a 4GB file eats 12-20GB of RAM, use this:

class ChunkedReader:
    def __init__(self, filepath, chunk_size=1_000_000,
                 progress_hook=None, encoding="utf-8"):
        self.filepath = Path(filepath)
        self.chunk_size = chunk_size
        self.progress_hook = progress_hook
        self.total_bytes = self.filepath.stat().st_size

    def read_chunks(self):
        carryover = ""
        with open(self.filepath, "r", encoding="utf-8") as f:
            idx = 0
            while True:
                raw = f.read(self.chunk_size)
                if not raw: break
                text = carryover + raw
                if not text.endswith("\n"):
                    nl = text.rfind("\n")
                    carryover = text[nl+1:] if nl > 0 else text
                    text = text[:nl+1] if nl > 0 else ""
                else:
                    carryover = ""
                if text.strip():
                    if self.progress_hook:
                        self.progress_hook(idx, self.total_bytes)
                    yield idx, text
                idx += 1

Key features: Carryover buffer prevents split lines • Progress hooks for tqdm • Zero dependencies • Never holds more than chunk_size in memory

💡 Drop this in any ETL pipeline or log processor.