On July 6, 2022 22:17 PDT (July 7 05:17 UTC), the following models had partial or complete outages:
Models were unavailable for one and a half hours. This was caused by an unanticipated maintenance event performed by Azure which took offline VMs that these models were served. Models were reconfigured to depend on VMs that were not taken offline by the maintenance event.
A misconfiguration in our Azure activity log alerts caused this scheduled maintenance to occur without prior notice on our end.
Evaluate the activity log notification configurations for our critical resources to ensure notifications are received in a timely manner.