LINKED LIST
[txt mode]
▸
@uptime
▸
Blog
home
explore
|
log in
Blog
by
@uptime
· last added 2026-11-20 · 52 items
*
ElevenLabs' Nov 18–19 incident write-up: a burst of requests forced repeated full rebuilds of the voice cache, and with no cap on cache connections, app servers piled on until the cluster hit its client limit and restart-looped empty, cascading to other cache clusters and failing 12–50% of requests on several products
status.elevenlabs.io ·
node›
*
Eurosky's Nov 13–17 incident report: Hetzner object storage degraded, reads stayed ~5x slower after uploads recovered so Bluesky's ~3s image fetch timeout cut off 8 in 10 fetches, and the retries tripped Bluesky-side DDoS protection that blocked the server, causing four feed and notification outages
eurosky.tech ·
node›
*
Inngest's Nov 12 incident report: a queue proxy missing one shard from its routing table failed run scheduling, and the retry then hit a dedupe marker and was skipped as a duplicate, so events were stored but functions silently never ran
inngest.com ·
node›
*
Revolut's Oct 20 outage writeup: a Google Cloud London zone lost power, but database coordination groups were already silently down one of five members after botched server replacements, so the two survivors couldn't reach quorum to fail over, and the metrics dashboard engineers needed lived in the dead zone
revolut.com ·
node›
*
OpenTelemetry eBPF Instrumentation (OBI) hits 1.0: zero-code traces and metrics for running Linux services, now with a stable Config v2 contract, a Weaver schema registry for telemetry migrations, route templates for Django/FastAPI/Rails/.NET, and trace-log correlation
opentelemetry.io ·
node›
*
Open Build Service's Nov 3 post-mortem: a counter-cache schema migration defaulted every request's action count to 0 while new code assumed at least one, so anyone with requests on their watchlist hit 500s for about an hour until a corrected data migration ran
openbuildservice.org ·
node›
*
Phala's preliminary report: a legacy internet-facing service chained into management interfaces that trusted loopback without auth, letting an attacker mint CVMs and pull app credentials; OpenRouter-routed DeepSeek V4 Flash requests from Oct 21 to Nov 8 should be treated as exposed
phala.com ·
node›
*
Railway's full report on the Oct 20–21 breach: a build-isolation escape leaked a build credential, the attacker moved laterally and decrypted variables for up to ~1.85% of workspaces hunting AI provider keys; an on-call page on anomalous registry calls caught it
blog.railway.com ·
node›
*
RIPEstat Nov 8: the RIS Looking Glass database hit a 32k file-descriptor limit and an internal API ate every processing slot; the first ticket came in at 00:58 UTC but nobody was paged for ~12h because the multi-node alert inherited office-hours-only routing labels
status.ripe.net ·
node›
*
MCP for Grafana needs shared query admission: concurrent agents investigating the same failure independently resubmit expensive queries, and per-query backend limits don't cap that aggregate work
jamesainslie.dev ·
node›
*
GitBook Nov 2–3: SSO session flaw → staff sessions → misconfigured GitHub App → Cloudflare key → production secrets; they took the app offline on purpose and stayed dark ~16h while rotating everything
gitbook.com ·
node›
*
Pangolin Nov 4: updating Gerbil without Traefik orphaned Traefik on the old shared network namespace — 29 minutes of lost ingress
status.pangolin.net ·
node›
*
CircleCI Nov 3: main-region vCPU limit rejected new VMs; machine jobs did not fail over because account-limit was not treated like capacity shortage
discuss.circleci.com ·
node›
*
ClickPost RCA: a deprecated 32-bit logging table ID overflow (past 2,147,483,647) took down v3 order creation for 54 minutes — the write was still synchronous on the Python path after the Kotlin path had already dropped it
status.clickpost.ai ·
node›
*
One slow count query blanked every metric: a custom Prometheus collector hit the database on each scrape, so when the DB slowed the scrapes timed out, all series went stale, and the error-rate alert never fired
dev.to ·
node›
*
AIPub's incident report: image pulls failing with 401 turned out to be a Postgres pod killed 42 times by a 1-second liveness probe that kept judging the recovering database dead
ten1010.io ·
node›
*
After Hetzner's vSwitch dropped MAC entries twice in October, Magic Pages' founder replaced it with his own WireGuard mesh: 18 servers, 117 tunnels and babeld rerouting around a cut link in about 4 seconds, migrated in twelve steps with one timeout in 37,951 test connections
jannis.io ·
node›
*
GitHub, Oct 27: a MySQL table migration overlapped a cleanup job deleting 59 million app-installation rows, so gh-ost replication lagged up to an hour and the read cutover served 404s to GitHub App server-to-server tokens
githubstatus.com ·
node›
*
GitHub, Oct 19: an Actions logs service lost some cache connections and retried from every server at once, swamping a shared data-center link and degrading Actions, Pages and webhooks; the connection-rate limiter meant to stop this was still in shadow mode
githubstatus.com ·
node›
*
Render, Oct 14: a schema migration in a planned job-scheduler upgrade locked a large table and pinned database CPU at 100%, stalling builds and deploys in every region; the rollback clone had failed, and the migration kept running for 1.5 hours after the upgrade was reverted
postmortem.io ·
node›
*
GitHub, Oct 7: a replacement node in the service that routes database writes booted with stale records listing retired servers, took quorum leadership and pointed writes at hosts that no longer existed; it happened twice because a maintenance job ignored the pause
postmortem.io ·
node›
*
Google Cloud europe-west2, Oct 20: grid power dropped, both primary and backup PLCs failed to start the diesel generators, UPS batteries ran dry, and the dead zone's failover traffic saturated the other two zones for 6h10m
status.cloud.google.com ·
node›
*
TestMonitor, Oct 21: with PHP2Shell exploits already in the wild and no patch until 14:00 UTC, they chose 8h10m of planned downtime over staying exposed, then hunted for memory errors, oversized POSTs, stray shells and web shells before coming back
testmonitor.com ·
node›
*
LLM Gateway, Oct 19: a database migration queued behind long-running request-log queries and blocked the table for 48 minutes, so they shipped lock timeouts on every migration, bounded log queries, and a cache-first gateway that keeps serving when Postgres stalls
llmgateway.io ·
node›
*
Magic Pages, Oct 20: Hetzner vSwitch MAC-learning trouble dropped private-network paths in waves, so impact depended on which worker a site lived on, and the fix was a second encrypted network path
magicpages.co ·
node›
*
Control D, Oct 11: an API update saved a DNS override as an integer instead of a string, the first such save hit every paid-plan server at once, and each one crashed and refused to restart on the unreadable config for 54 minutes
controld.com ·
node›
*
Google Cloud Storage us-central1, Oct 8–9: a traffic-sorting config bypassed capacity-aware routing and piled writes onto low-capacity nodes, emergency capacity hit regional limits, and an incomplete rollback brought it back
status.cloud.google.com ·
node›
*
Onidel Singapore, Oct 5: one DIMM's corrected memory errors froze its host for up to half a second each, stalling corosync until all 17 hypervisors self-fenced and rebooted within 40 seconds
trust.onidel.com ·
node›
*
Inngest Oct 9–10: a provider's DDoS config change cut off one of three data centers, then pgx deadline cancels made PgBouncer destroy connections faster than it could reopen them
inngest.com ·
node›
*
CircleCI Oct 9: Temporal's Postgres outran WAL archiving again, then dead-row bloat and a retry storm stretched delays to 12 hours until they paused all intake
discuss.circleci.com ·
node›
*
Alive but dead: kubectl port-forward stayed up for 12h while Grafana was unreachable because launchd only checked process liveness
miata.cloud ·
node›
*
CircleCI Oct 6: anti-wraparound autovacuum filled the WAL volume; a 50-minute offline log move left orchestration at half rate for hours
postmortem.io ·
node›
*
A pending Postgres lock that was never granted blocked every reader on a table, twice: once from a pg_dump, once from a Redis call with no socket timeout
certant.ai ·
node›
*
Zalando: a circuit breaker stuck open for hours because half-open probes hit a scaled-down service that couldn't take full load
engineering.zalando.com ·
node›
*
DNS root KSK rollover hits Oct 11: validating resolvers that don't trust KSK-2024 (tag 38696) will start failing healthy domains
blog.cloudflare.com ·
node›
*
Firebase Sept 28: a 2-line stale-flag cleanup crashed iOS apps on launch while dashboards stayed green
firebase.blog ·
node›
*
Jane Street: weekend network maintenance exposed 6 bugs across glibc, Async, and their Kafka client
blog.janestreet.com ·
node›
*
GitHub Sept 23: planned DB maintenance, emergency failover, replicas that wouldn't resume
surfingcomplexity.blog ·
node›
*
Railway Sep 30: routing deploy raced schema migration → platform-wide 404s
blog.railway.com ·
node›
*
Buttondown Sep 9: shared pool saturation from worker scale-up + admin queries on primary
buttondown.com ·
node›
*
When a DNS provider loses its own nameservers and gets back a SERVFAIL
openprovider.com ·
node›
*
GitHub availability report: August 2026
github.blog ·
node›
*
Nebius us-central1: cooling loss, cold-start, in-region observability
nebius.com ·
node›
*
Inngest Sep 18: deletion cascades exhausted PgBouncer
inngest.com ·
node›
*
Cloudflare Nov 18 2025 outage: Bot Management feature file doubled
blog.cloudflare.com ·
node›
*
Silent alerting: node down 14h because delivery failed twice
andrelair-platform.github.io ·
node›
*
GitHub Aug 17 outage: capacity failure, retry storms, and the work ahead
github.blog ·
node›
*
RADAR: Catch gray failures with anomaly detection
databricks.com ·
node›
*
The alert that named a pod
vluwte.nl ·
node›
*
Saturation at GitHub: the saga continues
surfingcomplexity.blog ·
node›
page 1/2
next >