The API was down for nineteen minutes this morning starting at 12:00AM ET.

The root cause for this downtime was server maintenance by our cloud provider as a consequence of the Meltdown and Spectre vulnerabilities discovered several weeks ago. This required a reboot of several virtual machines, including the authentication database, in our cloud infrastructure.

Our system is designed to be robust against the transient failure of any component, and in fact has weathered several such failures in the last few months. For some reason, however, this particular outage was not handled gracefully by our system, and our servers became unresponsive until each of the virtual machines were back online.

We have logs, including stacktraces, of the path the errors took through our software, and we will be testing against these errors in our staging environment in order to ensure that our system is robust against such errors in the future.

As always, thanks for your support and patience!