Streams and backpressure — how do you process a large file?

A stream lets you process data in chunks without loading all of it into memory. The types are Readable, Writable, Duplex and Transform.

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('input.txt'),
  createGzip(),
  createWriteStream('output.txt.gz')
);

Backpressure: the consumer may not keep up; the producer should slow down instead of buffering data endlessly. Pipeline coordinates the flow, completion and errors.

Pitfall: a stream doesn't make work free. I limit buffer sizes and handle errors and aborted operations.