Elevated error rate while creating fine-tuning jobs

Resolved·Degraded performance

Fine-tuning service is running normally.

Wed, Jul 31, 2024, 09:16 PM

(11 months ago)

Affected components

Jul 31, 2024, 04:09 PM

07:45 PM

Updates

Resolved

Fine-tuning service is running normally.

Wed, Jul 31, 2024, 09:16 PM

Monitoring

Error rates have stabilized and Fine-tuning API endpoints are operating normally again. Jobs are being processed as expected. We will continue to monitor this service closely for elevated error rates.

Wed, Jul 31, 2024, 07:45 PM(1 hour earlier)

Monitoring

We are experiencing a temporary increase in error rates from the /v1/fine_tuning API, including job creation and listing and event listing. We expect these errors to subside by 12:45pm PT.

Wed, Jul 31, 2024, 07:35 PM

Monitoring

Fine-tuning jobs are being processed again, though the service is still experiencing elevated error rates.

Wed, Jul 31, 2024, 06:19 PM(1 hour earlier)

Identified

The Fine-tuning API is currently online and accepting job creation requests, but there is a delay in job processing. Jobs will remained queued for the time being.

Wed, Jul 31, 2024, 05:57 PM(22 minutes earlier)

Investigating

Another spike of errors just occurred preventing jobs from being created. A mitigation is being pushed.

Wed, Jul 31, 2024, 05:49 PM

Monitoring

A fix has been deployed and the service is operating normally. We are continuing to monitor this service for errors

Wed, Jul 31, 2024, 04:12 PM(1 hour earlier)

Investigating

We are experiencing elevated error rates (500 responses) on the POST /v1/fine_tuning/jobs endpoint.

Wed, Jul 31, 2024, 04:09 PM

Availability metrics are reported at an aggregate level across all tiers, models and error types. Individual customer availability may vary depending on their subscription tier as well as the specific model and API features in use.