In a data pipeline processing millions of records, how would you use generators to avoid loading everything into memory?

0-2 years experience

1. 1Suppose you have an array of one million numbers and need to compute their sum without loading them all at once. How would you use a generator in JavaScript to accomplish this?
2. 2Write a simple generator function that reads lines from a large CSV file and yields each line one at a time.

2-5 years experience

1. 1Your team built a Node.js ETL job that reads from a database, transforms each record, and writes to a downstream service. The job crashes with out‑of‑memory errors when the dataset grows. Walk me through how you would refactor the code using generators or async iterators to keep memory usage constant.
2. 2During a recent deployment, a generator you added started yielding duplicate records after a network hiccup. How would you debug and fix the issue?

5-8 years experience

1. 1Design a streaming pipeline component that processes tens of millions of events per hour, applying back‑pressure and fault tolerance. Explain how you would structure the generator/async iterator chain, handle errors, and ensure the system can be horizontally scaled.
2. 2You need to integrate a third‑party API that returns a paginated response. How would you implement a generator that lazily fetches pages while respecting rate limits and avoiding memory spikes?

8+ years experience

1. 1Our organization is migrating a legacy batch ETL system to a real‑time streaming architecture built on Node.js services. What architectural considerations would you evaluate when deciding to replace batch loops with generator‑based streams, especially regarding observability, testing, and cross‑team ownership?
2. 2If you had to convince multiple product squads to adopt a generator‑centric data processing library, how would you design the API and migration path to minimize disruption and ensure long‑term maintainability?