Twice today, Playground encountered issues with generating images and at times accessing the website and API routes. This was caused directly by a hostError on our VMs that manage inference queueing. A hostError is a lower level software or hardware error (source). This meant that generation requests would sit in the queue for up to 300s when they would expire. While this primarily affected our inference queue, a cascading issue caused requests to our website to be affected as well.

The duration of the incident lasted 4 hours, though generation was affected for much less time. API users may have seen 429 errors.

Around 2pm PT, we finished migrating VMs to add redundancy to our inference queueing; now, if one host fails, our queue will be robust and should survive the incident.