Fluxer - Notice history

A routine Kubernetes operation ended up disconnecting users from their WebSocket connections, causing a reconnection storm that adversely impacted our API performance. We have temporarily paused new traffic and are taking this opportunity to update the gateway to the latest revision of the code, which includes performance improvements to member lists and more. Hold on for just a little while longer!

We've identified the only remaining issue with random API latency issues and errors and are deploying a fix.

Earlier today, between roughly 19:04 and 19:09 UTC, message sends and webhook deliveries returned 500s across the fleet for about five minutes.

Our Valkey instance was OOM-killed at 19:01. The pod's memory limit was 512 MiB, a low default left over from initial setup, and Valkey itself had no eviction ceiling configured, so under normal growth its working set crept past the pod limit before the configured eviction policy could engage. Valkey came back about a second later, but because it's an in-memory store, all ephemeral state was wiped, including the snowflake node-ID leases held by every API pod. A few minutes later every API pod tried to renew its lease, found it gone, and treated the loss as terminal rather than recoverable. We mitigated by rolling both API deployments, which let pods acquire fresh leases on startup.

Apologies for the disruption, and thanks for flying Fluxer.

A brief production outage was caused when troubleshooting production state in the gateway and triggering a node failure. We are working towards better redundancy and protection against this class of failures in the future. Thanks for your patience!

Sorry about the brief downtime earlier. We'd been putting off a Kubernetes cluster downscaling for a while, and went ahead with it today. During the process the gateway exceeded its memory limit and rolled, which forced everyone to reconnect, and that takes a little time to settle. Thanks for flying Fluxer.

A patch is being rolled out. This will take 5-10 minutes.

Our upstream provider has resolved the outage on their end and email sending works again. You'll need to resend the verification email if you didn't receive it by pressing the button in the banner at the top of the screen. Thanks for giving Fluxer a go!