What is the purpose of the `fs.createReadStream()` and `fs.createWriteStream()` methods in Node.js?
These methods are used for creating readable and writable streams, respectively. They are particularly useful for handling large files because they allow you to read and write data in smaller chunks, which can reduce memory usage and improve performance.
Scenario Questions
- 1
You need to copy a 5GB log file to another location without loading it all into memory. How would you use createReadStream and createWriteStream to do this, and what happens if the destination disk fills up halfway through?
- 2
A teammate wrote code that pipes a read stream to a write stream but forgot to handle errors. The process crashes silently when the source file doesn't exist. Show me how you'd fix it and explain why both streams need error handlers.
- 3
You're building a CLI tool that reads a large CSV, filters rows, and writes results to a new file. Walk me through how you'd chain createReadStream, a transform, and createWriteStream together.
- 1
Our upload service uses createReadStream to stream user files to S3. Under load, we see 'EMFILE: too many open files' errors. What's likely happening with the streams, and how would you debug and fix it?
- 2
You're implementing a feature that concatenates multiple uploaded video chunks into a single file. The chunks arrive as separate createReadStream instances. How do you sequence them into one createWriteStream without buffering everything in memory?
- 3
A downstream consumer reports that your file-processing pipeline occasionally outputs truncated files. The pipeline uses pipe() between read and write streams. What are three distinct causes you'd investigate, and how would you verify each?
- 1
Design a reusable stream utility that wraps createReadStream/createWriteStream to add automatic retry with exponential backoff on transient filesystem errors (e.g., network-mounted drives). What edge cases around stream lifecycle and backpressure must you handle?
- 2
We're migrating a legacy batch job that loads entire 10GB XML files into memory to a streaming parser. The parser emits 'record' events. How do you bridge the streaming XML parser to a createWriteStream outputting JSONL, ensuring backpressure propagates correctly when the writer slows down?
- 3
Your team owns a high-throughput log ingestion service. Each request creates a createWriteStream to append to daily rotated files. At peak, you hit file descriptor limits and write latency spikes. Propose a stream pooling architecture that bounds open descriptors while preserving write ordering per log file.
- 1
The platform team wants to standardize all file I/O across 50+ microservices onto a central streaming library that wraps createReadStream/createWriteStream with observability, encryption, and quota enforcement. How do you design the API to be adoptable incrementally without breaking existing stream pipelines, and what migration strategy minimizes risk?
- 2
A critical legacy system uses synchronous fs.readFileSync for config loading during startup. You need to replace it with a streaming approach to support dynamic config reloads without restart, but the config parser expects a complete string. How do you bridge the streaming and synchronous worlds safely, and what consistency guarantees can you provide during reload?
- 3
Your organization is adopting a new distributed filesystem that introduces variable latency and occasional 'stale handle' errors on long-lived streams. How do you evolve the company's stream abstraction layer to handle this transparently for application teams, and what contract changes (if any) do you communicate?
Follow-up Questions
- ❓What happens if the readable stream produces data faster than the writable stream can consume it?
- ❓How would you implement a transform stream that modifies data between read and write?
- ❓When would you choose streams over fs.readFile/fs.writeFile?