quay.io intermittent API failures

This incident was caused by a database change which was improperly rolled out. Quay engineers were deploying an update which dropped a no longer utilized table, however, this was deployed all at once rather than in phases. That meant that our existing deployment began crashing once the table was dropped, causing API failures since the table was being queried with every request to the API. The SRE team had to manually scale down the old deployment to allow the newer one to progress which restored API functionality. Going forward we are reviewing our process for database migrations and implementing additional checks to ensure that these migrations happen in two steps: application update first and database migrations second.