A sudden memory spike on guilds-2, one of our 12 guilds nodes, caused that node to be OOM-killed and triggered a thundering herd of guild restart requests against our API infrastructure.
Until this point, we had not been aware of a bottleneck in the users service's routing layer, which processed messages serially. When the router picked up a request from the API to load a batch of users from the database (to populate the member lists in those guilds after a node failure such as this one), it waited for the request to be routed to a shard and to receive its reply. It did so without spawning a green thread in Tokio, so the wait blocked the main request loop in the router. We had only three router pods, which meant we could handle at most three concurrent requests to the users service.
This had worked fine for a very long time. Under normal conditions our L1 cache absorbed the hits, so little or no work was done, and when work was done the requests were quick and did not run into timeout issues. When guilds-2 was OOM-killed, however, we received a flood of requests, all of which missed the cache. This was amplified by API retries, which overflowed the subscription queue into a permanent slow-consumer state, at which point NATS began dropping messages.
The slow-consumer state then fed a second amplifier. NATS emitted a stream of slow-consumer events for the affected subscription, and our transport layer logged a warning for each one without any throttling. At the peak these warnings were firing at roughly 134 a second, and the logging alone burned enough CPU to slow the router further, which deepened the backlog and produced yet more slow-consumer events. With messages being dropped and the router starved, the API timed out, and requests that depend on the users service (including the logic to start a new session in the gateway) returned 500s.
We restored service by scaling out the users service router capacity. We have since implemented a permanent fix, now being rolled out, in two parts. First, the router no longer blocks on the shard round-trip: it acquires a permit from a semaphore that caps the maximum concurrency and then spawns a green thread for each request, so the receive loop keeps draining the subscription at NATS speed rather than stalling on each shard reply. This mirrors the pattern our shard service already uses, where the same concurrency limit is set to 64. Second, we throttled the slow-consumer warning to at most once per second, so it can no longer burn CPU and amplify a future incident.
Thanks for your patience!
