Page 1 of 3

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 18:02
by Ferefire
At approximately midnight last night there was a major fire at the data centre our server is located at, basically burning the entire building down. To reassure everyone we thankfully understand no one has died or been injured in this accident. However AL's server was unfortunately one of the ones in said building at the time.

To go into more detail, www.eurogamer.net/articles/2021-03-10-r ... rver-blaze shows what happened. It's a pretty big deal, with at least 30,000+ servers gone (OVH are the world's third largest webhosting company) likely taking down anything up to a million websites/cloud-services taking down over 3 MILLION websites (now confirmed). To put it into perspective, these accidents at this size for datacentres occur worldwide perhaps once a decade at the most.

Thankfully @Redhat keeps daily backups, so we were able to sort out a new server, reinstall the OS and settings and restore backups across today. The loss of data is only minor but you will notice posts missing over the last 24 hours. We're still in process of restoring all convention websites - expect to see them up by end of tonight.

If you experience any issues, let us know as there may be a bug or two to iron out. Fingers crossed we're back at 100% now. :)

Update 1: First error spotted! We'll tackle the replying error issue once con sites are back up later this evening. Till then replies work still just ignore the error at the end. Not going to go into details, but resolved for now. :)

Update 2: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. :)

Update 3: Emails don't currently work, and there is some other behind the scenes fixes to do. The ETA on these are within the next few days (i.e. it should be mostly fixed by Friday or end of Saturday at the latest).

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 18:13
by JinseiKamiJestie
Welcome back AL!

I'm able to post, but thought I'd fill you in on a possible bug? After every post/edit since the server's been back up, I keep being directed to this error page. Going back a page (on my browser) brings me back and confirms my relief it did indeed post
Not sure if it's a me problem or a not me problem, as I'm not too tech-savvy and have legit no idea what it means :sweatdrop:
 
Forbidden

You don't have permission to access this resource.

Additionally, a 302 Found error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.4.29 (Ubuntu) Server at www.animeleague.net Port 443

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 18:14
by Naughty Kitty
I'm glad everyone is okay thank goodness. Hugs to all. :)

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 18:15
by BuckyBecky
:dance: :dance: :dance: :dance: :dance: :dance:

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 18:15
by Ferefire
Likely related to security permissions being too strict. We'll check this shortly once con sites are back up.

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 19:22
by Ferefire
Test :)

Edit: Yay, post replying is fine now. Please note, there may be more short downtimes tonight as we keep configuring and adjusting settings

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 19:30
by Dark_UwU
Woop woop! I live UwU

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 19:34
by Ferefire
Update: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. Make sure to add @Redhat to your Christmas Card list, folks, for his amazing work sorting things out so fast. ;)

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 19:42
by sam_sam
Woooo happy ending ☺️

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 19:45
by Mattboden03
Welcome back

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 20:27
by belsammael
Oh wow, that is pretty intense..! Glad to see the forum back up and running. :)

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 20:34
by Ferefire
belsammael wrote:
10 Mar 2021, 20:27
Oh wow, that is pretty intense..! Glad to see the forum back up and running. :)
Ngl, pretty awesome for us to be a part of a Hollywood movie in the making.. xD

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 10 Mar 2021, 22:02
by RedHat
Ferefire wrote:
10 Mar 2021, 19:34
Update: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. Make sure to add @Redhat to your Christmas Card list, folks, for his amazing work sorting things out so fast. ;)

Oh hi :)

Yeah, I need a break so I'll be resuming work on this tomorrow. The ticket websites are next on my list - infact I managed to configure it incorrectly and crash the server. I didn't sleep much last night, which is why I'm having a break now (or I'll crash the server some more today :confuse:).

If you find any more bugs, please let us know by posting here. This is a brand new server that I got working in a couple hours so there's bound to be some glitches.

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 11 Mar 2021, 01:35
by Lolin
When do I get AL points backpay for the time I could have spent bugging Grace?

Animeleague Server and It's Sites Are Back - This is What's Happened!

Posted: 11 Mar 2021, 01:40
by PharaohAtem
"Forbidden
You don't have permission to access this resource.
Additionally, a 302 Found error was encountered while trying to use an ErrorDocument to
handle the request.
Apache/2.4.29 (Ubuntu) Server at www.animeleague.net Port 443"



That's the one I've had.