Outage on some Ada and Babbage models
Incident Report for OpenAI


On July 6, 2022 22:17 PDT (July 7 05:17 UTC), the following models had partial or complete outages:

  • text-similarity-ada-001
  • text-search-babbage-doc-001
  • content-filter-alpha
  • text-search-babbage-query-001
  • text-search-ada-doc-001

Models were unavailable for one and a half hours. This was caused by an unanticipated maintenance event performed by Azure which took offline VMs that these models were served. Models were reconfigured to depend on VMs that were not taken offline by the maintenance event.

Root Cause:

A misconfiguration in our Azure activity log alerts caused this scheduled maintenance to occur without prior notice on our end.

Future Mitigations:

Evaluate the activity log notification configurations for our critical resources to ensure notifications are received in a timely manner.

Posted Jul 11, 2022 - 18:03 PDT

All models have recovered and are operational. Thank you for your patience.
Posted Jul 07, 2022 - 01:12 PDT
We’ve moved affected models to other regions, and are seeing broad recovery. We are continuing to monitor while we investigate root cause.
Posted Jul 06, 2022 - 23:56 PDT
We have lost the ability to communicate with virtual machines in one of the regions in which we operate. We are mitigating by moving capacity to other regions. We are starting to see recovery on some affected models.
Posted Jul 06, 2022 - 23:44 PDT
We have identified an issue affecting some Ada and Babbage models and are working on a remediation.
Posted Jul 06, 2022 - 23:07 PDT
This incident affected: API.