Fluxer - Notice history

We're starting to recover from some unexpected service disruptions. Thanks for bearing with us! If you're still having trouble loading the app or using some of its functionality, you might just need to wait a bit longer for DNS changes to propagate. You can speed that up by flushing your computer's DNS cache or restarting your computer.

We implemented a fix and are currently monitoring the result. You may have issues connecting to voice calls.

We are continuing to work on a fix for this incident. User-uploaded content was subject to DMCA takedowns and our CDN provider ended up suspending our entire DNS zone within 30 minutes of them receiving them, despite giving us a 48-hour deadline to delete this user-uploaded content from the platform. We condemn this unprofessional handling by our CDN provider and apologise for the inconvenience this causes you.

We're continuously monitoring the situation, and you should expect brief windows of intermittent failures and timeouts while we work to relieve the pressure on our web proxies.

The cause is fairly specific. We have a series of nodes load balancing traffic to the main Fluxer infrastructure, which is handling everything without breaking a sweat. The problem is our web proxy layer, which is running out of memory under the unprecedented number of new WebSocket connections being established. Unfortunately, that means we don't get to show this new wave of users how stable Fluxer normally is.

We're glad to welcome you to Fluxer, and if you'd like to support us financially, you can become a Plutonium subscriber, send a donation, or buy the Operator Pass for self-hosters once it becomes available. For now, though, you may run into intermittent issues reaching any of our web services. We're working as fast as we can. 🏃

On 18 August, Fluxer was down for about 15 minutes, from 03:54 to 04:05 UTC. Traffic reaches us through Caddy. Caddy terminates TLS and forwards into Kubernetes via ingress-nginx. Page loads are served by app-proxy. Everything else goes to the API.

On every page load, app-proxy refreshed a cached discovery document before responding. It fetched that document from its own public address, https://api.fluxer.app/.well-known/fluxer. The request therefore travelled back out through Caddy rather than staying inside the cluster.

A rolling API deploy made that endpoint briefly unavailable. The fetch had a five second timeout, so every page load began taking five seconds. Those slow requests filled up ingress-nginx. They then filled Caddy's queue of pending connections, and Caddy stopped accepting new ones. At that point app-proxy could no longer reach the API at all. The route it needed ran through the front door it had just blocked. The outage held itself open and could not recover on its own.

We restarted Caddy on all three web nodes to clear the backlog. That restored service.

The root cause was in app-proxy. Its background refresh already backed off when the API was unhealthy. The page load path, however, called the refresh directly and skipped that backoff. Every request therefore paid the full timeout.

Page loads now serve the cached document. They make no network calls at all while a cached copy exists. The first fetch after startup is rate limited, so an unavailable API cannot cause a retry storm.