This was a Discord-side outage, not something caused by Pixie. All timestamps are in UTC+0. Pixie was down for about 40 minutes total (14:19-14:59) — all commands and moderation actions were unavailable for that whole window, since there was no gateway connection at all.
At 14:19, I received an alert saying "discord connection: no heartbeat ACK yet (or gateway disconnected)." Followed by that, my own Discord client experienced connection issues. After checking downdetector, there was 35,586 downtime reports for Discord at its peak. Shortly later, Pixie went offline on Discord. Pixie had lost connection to the Discord gateway and was unable to reconnect, getting 500 Internal Server Error codes.
At 14:25, Discord posts an incident stating "Session Unavailability," with the status investigating.
At 14:42 Discord updates the outage status to Identified; "The issue has been identified and we are working to restore service."
At 14:47 a routine, scheduled Pixie server invite leaderboard data fetch failed, returning a Cloudflare 500 Internal Server Error page.
Around this time, Discord appeared to start recovering. At 14:52, on Shard 0, the bot process was requested to restart in an attempt to recover service quicker, with the intent being for the bot to establish a new gateway identity.
At 14:54 Shard 0's system service manager forcibly killed the bot process due to it hanging after receiving the restart request. Pixie's shutdown sequence flushes in-memory data to the database before it closes the Discord connection, so I went and actually checked the database's write-ahead log afterward instead of just trusting that ordering: its WAL file has a birth timestamp of 14:52:47 — only 7 seconds after the shutdown was requested, and well before the ~90-second timeout that eventually killed the process — and PRAGMA wal_checkpoint shows it's fully checkpointed right now with zero pending frames. For reference, if there was a fully unexpected stop of the bot instead, due to sudden equipment power loss — which is unlikely given the setup of Pixie — at most 30 seconds of data would be lost for the systems that store data in memory. Example being the XP system. Most other things are a direct DB write. Not a big deal at this time. When the new bot process started, it could not establish a connection to Discord's gateway still. This situation was unhandled at this time — the process didn't crash so much as get stuck doing nothing, which meant systemd's own Restart=on-failure policy never kicked in, since as far as it could tell the process hadn't actually failed. This is now fixed (see below). I manually sent the bot process a SIGKILL, killing it after a few minutes of zombying, to allow Discord to recover more.
At 14:58 Discord updates the outage status to Monitoring; "A fix has been implemented and we are currently monitoring recovery of services."
At 14:59, after the SIGKILL, when the new process started, it correctly established a connection to Discord's edge — about a minute after Discord's own fix had already landed (their Monitoring update at 14:58). So the zombie bug cost Pixie roughly a minute of extra downtime, not much more than that.
At 15:16 Discord updates the outage status to Resolved; "This incident has been resolved."
Changes made after this incident:
bot.start() now calls os._exit(1) instead of hanging as a zombie process if the gateway connection fails on startup. This only applies to that specific failure path — before the bot has connected or done any real work, so there's nothing to clean up gracefully anyway. A normal shutdown (a real restart, not a startup failure) still goes through the regular flush-then-close sequence untouched. The exact reason the shutdown itself hung in the first place wasn't identified — this fix guards against it happening again rather than explaining why it happened this time.
The signal-triggered shutdown path (what systemctl restart actually uses, and the one that hung during this incident) also didn't have a fallback timer at all — a separate command ($restart/$saferestart) already had one, just not this path. It now schedules the same kind of fallback: if closing the Discord connection doesn't finish within 20 seconds, the process force-exits itself instead of silently sitting there for systemd's full 90-second default timeout to notice. Timestamped log lines were also added around each shutdown step, so if this ever happens again there's a real trace of exactly where, instead of having to guess.