Animeleague

At approximately midnight last night there was a major fire at the data centre our server is located at, basically burning the entire building down. To reassure everyone we thankfully understand no one has died or been injured in this accident. However AL's server was unfortunately one of the ones in said building at the time.

To go into more detail, www.eurogamer.net/articles/2021-03-10-r ... rver-blaze shows what happened. It's a pretty big deal, with at least 30,000+ servers gone (OVH are the world's third largest webhosting company) likely taking down anything up to a million websites/cloud-services taking down over 3 MILLION websites (now confirmed). To put it into perspective, these accidents at this size for datacentres occur worldwide perhaps once a decade at the most.

Thankfully @Redhat keeps daily backups, so we were able to sort out a new server, reinstall the OS and settings and restore backups across today. The loss of data is only minor but you will notice posts missing over the last 24 hours. We're still in process of restoring all convention websites - expect to see them up by end of tonight.

If you experience any issues, let us know as there may be a bug or two to iron out. Fingers crossed we're back at 100% now.

Update 1: First error spotted! We'll tackle the replying error issue once con sites are back up later this evening. Till then replies work still just ignore the error at the end. Not going to go into details, but resolved for now.

Update 2: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that.

Update 3: Emails don't currently work, and there is some other behind the scenes fixes to do. The ETA on these are within the next few days (i.e. it should be mostly fixed by Friday or end of Saturday at the latest).

Welcome back AL!

I'm able to post, but thought I'd fill you in on a possible bug? After every post/edit since the server's been back up, I keep being directed to this error page. Going back a page (on my browser) brings me back and confirms my relief it did indeed post
Not sure if it's a me problem or a not me problem, as I'm not too tech-savvy and have legit no idea what it means
 

Forbidden

You don't have permission to access this resource.

Additionally, a 302 Found error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.4.29 (Ubuntu) Server at www.animeleague.net Port 443

Likely related to security permissions being too strict. We'll check this shortly once con sites are back up.

Test

Edit: Yay, post replying is fine now. Please note, there may be more short downtimes tonight as we keep configuring and adjusting settings

Woop woop! I live UwU

Update: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. Make sure to add @Redhat to your Christmas Card list, folks, for his amazing work sorting things out so fast.

Woooo happy ending

Welcome back

Oh wow, that is pretty intense..! Glad to see the forum back up and running.

belsammael wrote: ↑

10 Mar 2021, 20:27

Oh wow, that is pretty intense..! Glad to see the forum back up and running.

Ngl, pretty awesome for us to be a part of a Hollywood movie in the making.. xD

Ferefire wrote: ↑

10 Mar 2021, 19:34

Update: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. Make sure to add @Redhat to your Christmas Card list, folks, for his amazing work sorting things out so fast.

Oh hi

Yeah, I need a break so I'll be resuming work on this tomorrow. The ticket websites are next on my list - infact I managed to configure it incorrectly and crash the server. I didn't sleep much last night, which is why I'm having a break now (or I'll crash the server some more today ).

If you find any more bugs, please let us know by posting here. This is a brand new server that I got working in a couple hours so there's bound to be some glitches.

When do I get AL points backpay for the time I could have spent bugging Grace?

"Forbidden
You don't have permission to access this resource.
Additionally, a 302 Found error was encountered while trying to use an ErrorDocument to
handle the request.
Apache/2.4.29 (Ubuntu) Server at www.animeleague.net Port 443"



That's the one I've had.

@PharaohAtem when was that? And what were you doing on the forum?