On October 19-20, 2025, AWS experienced a major outage in the Northern Virginia (us-east-1) Region that impacted DynamoDB, EC2, Network Load Balancer, Lambda, and multiple other services used by millions of customers worldwide.
Here’s what went down:
🔹 A rare race condition in DynamoDB’s DNS management caused critical endpoint records to disappear, blocking connections to DynamoDB and triggering widespread failures.
🔹 This DNS failure cascaded into EC2 instance launch failures, network load balancer disruptions, and delays across many dependent services.
🔹 Customers faced increased API errors, latency spikes, and service degradation for over 14 hours.
How AWS responded:
✔️ Rapid identification and manual intervention fixed the DNS state and restored DynamoDB connectivity by early morning.
✔️ Engineers throttled requests, restarted critical subsystems, and brought EC2 and network systems back online gradually.
✔️ Full recovery was achieved over the next several hours, with all services stable by late October 20.
✔️ AWS has disabled the faulty DNS automation, is enhancing testing, and improving fail-safes to prevent similar incidents.
Why this matters:
Cloud infrastructure is incredibly complex—and even the best systems can face hidden bugs with significant impact. What counts is an unwavering commitment to transparency, rapid response, and continuous improvement.
Let’s use this event as a powerful learning moment for all of us in tech.
No comments:
Post a Comment