Counting lines in a 5GB log file efficiently

WHAT THIS TESTS This evaluates whether you can process data that exceeds available memory, a core backend skill. It distinguishes engineers who think in streams from those who only know load-everything patterns.

A GOOD ANSWER COVERS fs.readFile reads the entire file into a single Buffer held in memory. For a 5GB file that means allocating roughly 5GB of RAM at once, which can exhaust the process, trigger heavy garbage collection, or exceed Node's maximum buffer size and throw. The efficient approach is to stream the file: fs.createReadStream reads it in small chunks, and the readline module turns those chunks into discrete lines through its line event. You keep a counter, increment it whenever a line contains ERROR, and report the total when the stream closes. Memory usage stays roughly constant no matter how large the file grows because only a small buffer and the current line live in memory at any moment.

COMMON WRONG ANSWERS Reading the whole file then calling split on newlines, which still buffers everything. Believing the operating system page cache makes readFile safe, which ignores the JavaScript heap allocation. Spawning grep via child_process is workable but the question asks for core Node APIs and stream understanding.

LIKELY FOLLOW-UPS How does backpressure apply here. What happens at chunk boundaries that split a line, and how does readline handle them. How would you parallelize across multiple files.

ONE CONCRETE EXAMPLE You create rl = readline.createInterface({ input: fs.createReadStream('app.log') }), attach rl.on('line', line => { if (line.includes('ERROR')) count++; }), and on rl.on('close', () => console.log(count)). The process holds only kilobytes at a time, so it finishes a 5GB scan without growing memory, whereas readFile would attempt one giant allocation upfront.