Resolved
This incident has been resolved.
Monitoring
Error rates for jobs are still elevated but dropped. The job queue is backed up still but emptying quickly. Queue wait time is ~1 hour
Identified
We experienced a large spike in errors for actively running fine-tuning jobs. The errors have been mitigated but there is now a backlog in jobs to be processed.