Fluxer - Issues with community availability, elevated API error rates – Incident details

As mentioned, the bottleneck in the users service (now fixed, and glad we caught it) was a secondary effect of the guilds-2 node being OOM-killed, not the cause. We initially assumed the OOM-kill was a one-off and closed the incident, but the root cause turned out to be different.

The node spawned an unbounded worker for every async query. When a request came in for a guild's data, such as a member list, it started a separate lightweight worker so the guild's main process stayed responsive, but nothing capped how many could run at once. On top of that, each worker loaded its own copy of the guild data it was serving (the member list, plus every member's profile and presence), which for a very large community is a lot of memory per worker. When a burst of requests arrived together, as happens when many clients reconnect after a node restart, thousands of workers ran in parallel, each holding its own copy, and exhausted the node's memory, killing it and restarting the cycle.

We've shipped two changes for this, both as no-downtime hot patches that are now live across the fleet.

The first caps how many of these workers run concurrently. Extra requests now wait briefly for a free slot rather than all running at once, so memory stays bounded however large the spike. Since it went out, the previously affected node has held steady with no further restarts.

The second removes the duplication at its source. Workers no longer hold their own copies of the guild's member data; they read from a single shared in-memory copy instead, so a worker's footprint is now a small fraction of what it was before. Between the two, a burst of requests can no longer balloon a node's memory the way it did here.

Thanks for your patience!

We're so back!

Some guild crashes are reoccurring and we are investigating.

A sudden memory spike on guilds-2, one of our 12 guilds nodes, caused that node to be OOM-killed and triggered a thundering herd of guild restart requests against our API infrastructure.

Until this point, we had not been aware of a bottleneck in the users service's routing layer, which processed messages serially. When the router picked up a request from the API to load a batch of users from the database (to populate the member lists in those guilds after a node failure such as this one), it waited for the request to be routed to a shard and to receive its reply. It did so without spawning a green thread in Tokio, so the wait blocked the main request loop in the router. We had only three router pods, which meant we could handle at most three concurrent requests to the users service.

This had worked fine for a very long time. Under normal conditions our L1 cache absorbed the hits, so little or no work was done, and when work was done the requests were quick and did not run into timeout issues. When guilds-2 was OOM-killed, however, we received a flood of requests, all of which missed the cache. This was amplified by API retries, which overflowed the subscription queue into a permanent slow-consumer state, at which point NATS began dropping messages.

The slow-consumer state then fed a second amplifier. NATS emitted a stream of slow-consumer events for the affected subscription, and our transport layer logged a warning for each one without any throttling. At the peak these warnings were firing at roughly 134 a second, and the logging alone burned enough CPU to slow the router further, which deepened the backlog and produced yet more slow-consumer events. With messages being dropped and the router starved, the API timed out, and requests that depend on the users service (including the logic to start a new session in the gateway) returned 500s.

We restored service by scaling out the users service router capacity. We have since implemented a permanent fix, now being rolled out, in two parts. First, the router no longer blocks on the shard round-trip: it acquires a permit from a semaphore that caps the maximum concurrency and then spawns a green thread for each request, so the receive loop keeps draining the subscription at NATS speed rather than stalling on each shard reply. This mirrors the pattern our shard service already uses, where the same concurrency limit is set to 64. Second, we throttled the slow-consumer warning to at most once per second, so it can no longer burn CPU and amplify a future incident.

Thanks for your patience!