On December 4th, 2024 from 15:48 PT to 15:52 PT, 100% of API requests experienced HTTP 530 errors due to a misconfiguration in the global load balancer. This was promptly corrected and connectivity restored by 15:52 PT.

Later that day, from 16:07 PT to 17:37 PT, a different issue was surfaced by an upgrade of the DNS cache system. Consequently the OpenAI identity system lost connectivity to the DNS cache. As a result, user requests appeared to stall for 30 seconds. After the 30 second DNS lookup timeout the Identity system fell back to a cached IP and completed requests successfully. These elevated latencies led to 45% of API requests experiencing 499 errors, which appeared as client-side cancellations. In the ChatGPT interfaces this latency manifested itself as a 30 second "wait time" after which requests were completed.

The first wave of this incident, caused by a global load balancer misconfiguration, was mitigated by applying the correct configuration.

The second wave was mitigated by temporarily switching to an alternative system.

• To safeguard against global load balancer (Cloudflare) misconfigurations, we collaborated with Cloudflare to investigate the issue and implemented a solution to prevent a single pool from impacting all API traffic. The default and fallback pools are now distinct, and this change has already been rolled out to ensure seamless traffic handling.
• The way that our identity systems integrated with and relied on our DNS cache here was found to be unnecessary and was removed.
• We are continuing to improve our chaos test system, which will aim to surface DNS cache dependency issues in a controlled way.

We know that outages affect our customers' products and businesses. While we came up short here, we are committed to preventing such incidents in the future and improving our service reliability.