At approximately midnight last night there was a major fire at the data centre our server is located at, basically burning the entire building down. To reassure everyone we thankfully understand no one has died or been injured in this accident. However AL's server was unfortunately one of the ones in said building at the time.
To go into more detail,
www.eurogamer.net/articles/2021-03-10-r ... rver-blaze shows what happened. It's a pretty big deal, with at least 30,000+ servers gone (OVH are the world's third largest webhosting company)
likely taking down anything up to a million websites/cloud-services taking down over 3 MILLION websites (now confirmed). To put it into perspective, these accidents at this size for datacentres occur worldwide perhaps once a decade at the most.
Thankfully
@Redhat keeps daily backups, so we were able to sort out a new server, reinstall the OS and settings and restore backups across today. The loss of data is only minor but you will notice posts missing over the last 24 hours. We're still in process of restoring all convention websites - expect to see them up by end of tonight.
If you experience any issues, let us know as there may be a bug or two to iron out. Fingers crossed we're back at 100% now.
Update 1: First error spotted! We'll tackle the replying error issue once con sites are back up later this evening. Till then replies work still just ignore the error at the end. Not going to go into details, but resolved for now.
Update 2: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that.
Update 3: Emails don't currently work, and there is some other behind the scenes fixes to do. The ETA on these are within the next few days (i.e. it should be mostly fixed by Friday or end of Saturday at the latest).
).
