Every story tagged Post Mortem, curated for CIOs and IT leaders — ranked by source credibility, engagement, and freshness.
1 story · open in the command center
Bluesky experienced an 8-hour outage affecting 50% of users due to a missing concurrency limit in their data plane code, which allowed a new internal service to spawn 15-20k simultaneous connections per request, exhausting TCP ports and creating a cascading failure through their logging and garbage collection systems. The incident revealed critical gaps in observability for high-batch-size requests and highlighted how aggressive performance tuning (GOGC/GOMEMLIMIT settings) can amplify infrastructure failures into complete service disruption. This demonstrates how a single missing line of code in a low-traffic endpoint can cause catastrophic failures when combined with poor monitoring and overly aggressive optimization.