In a data pipeline processing millions of records, how would you use generators to avoid loading everything into memory?

Q: In a data pipeline processing millions of records, how would you use generators to avoid loading everything into memory?

function* readFileLines(file) { /* yield line by line */ } function* filter(iter, pred) { for (const item of iter) if (pred(item)) yield item; } function* map(iter, transform) { for (const item of iter) yield transform(item); } const pipeline = map(filter(readFileLines('data.txt'), l => l.includes('ERROR')), l => l.toUpperCase()); for (const line of pipeline) console.log(line); // streams line by line

Difficulty: 5/10

generators, stream processing, memory optimization

Chain generators that read, transform, and write records lazily, processing one record at a time without holding all data in memory.

Memory-friendly pipeline

Scenario Questions

0-2 years experience

1Suppose you have an array of one million numbers and need to compute their sum without loading them all at once. How would you use a generator in JavaScript to accomplish this?
2Write a simple generator function that reads lines from a large CSV file and yields each line one at a time.

2-5 years experience

1Your team built a Node.js ETL job that reads from a database, transforms each record, and writes to a downstream service. The job crashes with out‑of‑memory errors when the dataset grows. Walk me through how you would refactor the code using generators or async iterators to keep memory usage constant.

Follow-up Questions

What trade‑offs did you consider between a generator and a traditional loop?
How would you test that your generator correctly handles errors and back‑pressure?
Can you describe how you would monitor memory usage of this pipeline in production?