Resolved
DALL·E and all APIs are operating normally.
Monitoring
DALL·E is recovering and we are monitoring present traffic.
Identified
DALL·E remains unresponsive due to networking issues. We are actively diagnosing the situation. All other API traffic is operational.
Identified
We have moved text-ada-001 and classic davinci out of our affected cluster. These two models are now operational. Most DALL·E traffic is still heavily degraded.
Identified
All nodes in one of our clusters are experiencing DNS issues due to a systemd upgrade affecting the version of Ubuntu running on our nodes. This incident is also being tracked by our cloud provider here: https://app.azure.com/h/2TWN-VT0/c24ff9
This cluster handles most traffic for DALL·E as well as classic davinci and text-ada-001. These models are currently unreachable while we investigate workarounds.
Identified
We have identified the issue and are working to resolve it.
[edit] Timestamp updated to reflect when our monitoring reports the drop in traffic to our cluster.