Node.js, pipes, and disappearing bytes

submited by

Style Pass

2024-10-13 15:00:03

Someone in one of my online communities recently asked about why their Node command was dropping output when piped to another command:

[command] spits out 2319 lines of valid json to stdout. If I redirect it into a file then I can open that file and yes, see all 2319 lines and well formatted json.

If I pipe to tee to write it to a file and then pipe that to wc it reports 1394 and the file indeed cuts off my json part way through the 1394th line

I'd seen something like this before years ago, but didn't really remember any details from that besides that Node does some weird things with pipes.

If it's truncating output from 2k+ lines to ~1.5k lines of JSON, well, hm - depending on how big those lines are, it seems very likely that there's some kind of "pipe is full" limit getting triggered in some shape or fashion here.

Aha! A power of two - ✨magic✨ numbers are always a good time. (Seriously. It's so much easier once you have a hint about what to look for.)