40 MB an hour: the Node memory leak hiding behind setMaxListeners(0)

Our ingest service restarted itself every nine hours. Not crashed, exactly: Kubernetes killed it for exceeding its 1 GiB memory limit, a new pod came up, and the graph started climbing again at a steady 40 MB an hour. Because traffic kept flowing, nobody treated it as urgent for two months. Then a customer’s batch import doubled our request rate, the restarts came every four hours, and one of them landed in the middle of a 30 second request.

This is how we found it, and why the warning that would have told us on day one had been switched off.

A straight line means something is being kept

Memory that climbs in a sawtooth and returns to a baseline is the garbage collector doing its job. Memory that climbs in a straight line, proportional to traffic, means every request leaves something behind that is still reachable. The question is always: reachable from where?

To answer it you need two heap snapshots taken some time apart under load, and a diff between them. Node can write a snapshot when it receives a signal, which is far easier than attaching a debugger to a pod:

node --heapsnapshot-signal=SIGUSR2 dist/server.js &   # SIGUSR2 writes a snapshot to the cwd
PID=$!

kill -USR2 "$PID"   # baseline
npx autocannon -c 50 -d 120 -m POST \
  -H "content-type=application/json" -b '{"events":[]}' \
  http://localhost:3000/ingest
kill -USR2 "$PID"   # after 2 minutes of load

ls -lh *.heapsnapshot

We reproduced it locally in about ten minutes. Each snapshot pauses the process while it is written, so be careful doing this against production traffic.

Comparing snapshots

Load both files into the Memory tab of Chrome DevTools, select the second one, and switch the view to Comparison. Sort by size delta. The top entries were not surprising on their own: thousands of new Object and Array instances, plus IncomingMessage, the request object. Requests should be collected seconds after they finish. Ours were not.

The useful panel is Retainers, which shows the path from a garbage collection root to the selected object. For one of the stray requests, the chain read roughly: an EventEmitter called config, its _events.reload array, a closure, a ctx object, and finally our request.

The bug

Here is a trimmed version of the handler and the config module:

// config.js (EventEmitter comes from node:events)
export const config = new EventEmitter();
config.setMaxListeners(0); // "fix" for a noisy warning, added in 2024

// ingest.js (imports config and currentLimits from ./config.js)
export async function handleIngest(req, res) {
  const ctx = { req, batch: [], limits: currentLimits() };
  config.on('reload', (next) => {
    ctx.limits = next;
  });
  await processBatch(ctx);
  res.writeHead(202).end();
}

Every request added a listener to a module-level emitter that lives forever, and the listener’s closure held ctx, which held req. Nothing ever removed it. Two million requests a day meant two million closures.

Node knew. By default, an emitter prints MaxListenersExceededWarning once a single event has more than 10 listeners. That warning would have appeared on the eleventh request after the first deploy. Someone saw it in the logs, assumed it was noise, and set the limit to zero, which means unlimited.

The fix

The handler did not need a listener at all. It needs the limits at the moment it checks them, so it can read them then:

 export async function handleIngest(req, res) {
-  const ctx = { req, batch: [], limits: currentLimits() };
-  config.on('reload', (next) => {
-    ctx.limits = next;
-  });
+  const ctx = { req, batch: [], limits: currentLimits };
   await processBatch(ctx);
   res.writeHead(202).end();
 }

processBatch now calls ctx.limits() when it needs a value, which also means a config reload applies to requests already in flight. We removed the setMaxListeners(0) call in the same pull request, and added a test that fails if config.listenerCount('reload') grows after a hundred simulated requests.

Afterwards

Memory now sits flat around 180 MB per pod. We kept --heapsnapshot-signal in the production start command, since the flag costs nothing until the signal arrives. We also grepped every service for setMaxListeners and found two more. One was legitimate. The other was the same bug in a different file.