On June 24 2025 at 01:00 PM PDT, a mis-configuration of a networking routing rule sent significant /v1/chat/completions traffic to an incorrect backend service for 13 minutes. This resulted in the majority of requests to /v1/chat/completions failing during the 13 minute window.
We quickly removed the rule and restored normal service which mitigated the incident on June 24 2025 at 01:13 PM PDT. We are implementing additional safeguards such as stricter reviews and staged roll-outs for networking configuration changes, and enhanced monitoring to prevent similar misconfigurations in the future.
We sincerely apologize for the disruption and we are committed to preventing such incidents in the future.