Elevated 429s
Incident Report for Playground
Postmortem

Playground users + API users may have seen increased 429 errors from 12:18-12:25am PT. Specifically, about 1/3 of requests were 429 or 500 in this 7 minute period. Playground underwent a Denial of Service attack this night from about 12:05-12:25am PT. While we do have a few layers of DoS protection, it was not entirely adequate in preventing elevated errors; towards the end of the attack our primary service was affected.

We have taken action and will be continuing to improve our protection.

Posted May 30, 2024 - 08:33 UTC

Resolved
This incident has been resolved.
Posted May 30, 2024 - 07:38 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 30, 2024 - 07:30 UTC
This incident affected: Website, Default model (Playground v2.5), and Other Models.