On September 16, 2024, between 8:39am to 8:50am PDT, all API traffic to api.openapi.com was blocked. We additionally saw new account creations and logins failing for OpenAI Platform and ChatGPT.

The root cause was an erroneous update to a Web Application Firewall (WAF) rule which mistakenly blocked all traffic.

The issue was detected immediately and after identifying the root cause, we reverted the offending change and temporarily froze subsequent updates to our WAF rules.

We frequently make changes to our WAF rules to mitigate new threats or change internal configuration. These changes are all subject to our standard change management process which requires peer review of every change, and we recommend testing rules in "dry-run" mode before enforcing new rules. However, we had no mechanism in place to guarantee that all new rules were tested in dry-run mode before turning on enforcement.

As part of the incident response, we have already implemented the following measures:

To prevent incidents like this in the future, we have implemented controls to enforce that these rules are never turned to "block" without first testing them in dry-run mode regardless of the urgency.

We know API outages impact our customers' products and business, and are committed to preventing such incidents in the future and improving our service reliability.