Blog · Behind the scenes

The Railway incident: what it meant for Aucly

On October 20th and 21st, some people using Aucly had trouble placing bids, uploading pictures or viewing auction images. I'm sorry for the disruption, especially if you were running a live auction. It's the last thing you need when you're trying to raise money for your school or charity.

October incident update: what happened and what we changed at Aucly

What happened

Railway, which hosts Aucly, was responding to a security breach. Its report published on 10 November explains how some customers' credentials were exposed and why it paused new deployments while securing its systems.

Railway changed some of its storage credentials as part of its response. Unfortunately, because new deployments were temporarily blocked, Aucly couldn't immediately pick up those changes. This caused problems uploading and displaying auction images.

We haven't established the exact duration and full extent of the disruption, however we did have reports that some users were unable to place bids during this period.

What we did

We changed the credentials we use to talk to other systems within Railway and outside of Railway.

We reviewed and verified our disaster recovery processes.

What we've changed since

  • Clearer error messages when a form can't save, rather than leaving you wondering what happened.
  • Better recovery from temporary image-loading failures, and caching so public pictures need fewer trips to storage.
  • Outbid emails and their pictures are prepared in the background, rather than holding up a bid. We've also made pending emails more resilient to restarts.
  • A fix for unexpected sign-outs during application updates.
  • A review of the software libraries Aucly uses, with compatible security and maintenance updates.

Why we're staying with Railway

We considered moving Aucly to another hosting provider. After reviewing Railway's report and the security changes it has made, we're satisfied with its response and have decided to stay.

If you have a question about your auction or account, please contact [email protected]. We'll update this post if we learn anything that changes what you need to know.