Animeleague Server and It's Sites Are Back - This is What's Happened!

Where you can discuss, chat or debate about the forums, member meetups, the world or life in general.

Moderators: Baul, Elgood81

User avatar
Ferefire
Ensouled Member
Ensouled Member
Posts: 47522
Joined: 20 Apr 2003, 19:55
Contact:
Great Britain

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Ferefire » 10 Mar 2021, 18:02

At approximately midnight last night there was a major fire at the data centre our server is located at, basically burning the entire building down. To reassure everyone we thankfully understand no one has died or been injured in this accident. However AL's server was unfortunately one of the ones in said building at the time.

To go into more detail, www.eurogamer.net/articles/2021-03-10-r ... rver-blaze shows what happened. It's a pretty big deal, with at least 30,000+ servers gone (OVH are the world's third largest webhosting company) likely taking down anything up to a million websites/cloud-services taking down over 3 MILLION websites (now confirmed). To put it into perspective, these accidents at this size for datacentres occur worldwide perhaps once a decade at the most.

Thankfully @Redhat keeps daily backups, so we were able to sort out a new server, reinstall the OS and settings and restore backups across today. The loss of data is only minor but you will notice posts missing over the last 24 hours. We're still in process of restoring all convention websites - expect to see them up by end of tonight.

If you experience any issues, let us know as there may be a bug or two to iron out. Fingers crossed we're back at 100% now. :)

Update 1: First error spotted! We'll tackle the replying error issue once con sites are back up later this evening. Till then replies work still just ignore the error at the end. Not going to go into details, but resolved for now. :)

Update 2: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. :)

Update 3: Emails don't currently work, and there is some other behind the scenes fixes to do. The ETA on these are within the next few days (i.e. it should be mostly fixed by Friday or end of Saturday at the latest).
---

User avatar
JinseiKamiJestie
Standard Member
Posts: 387
Joined: 12 Feb 2021, 11:41
Contact:
Great Britain

Achievements

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by JinseiKamiJestie » 10 Mar 2021, 18:13

Welcome back AL!

I'm able to post, but thought I'd fill you in on a possible bug? After every post/edit since the server's been back up, I keep being directed to this error page. Going back a page (on my browser) brings me back and confirms my relief it did indeed post
Not sure if it's a me problem or a not me problem, as I'm not too tech-savvy and have legit no idea what it means :sweatdrop:
 
Forbidden

You don't have permission to access this resource.

Additionally, a 302 Found error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.4.29 (Ubuntu) Server at www.animeleague.net Port 443
Last edited by JinseiKamiJestie on 10 Mar 2021, 18:14, edited 2 times in total.

User avatar
Naughty Kitty
Faithful Member
Faithful Member
Posts: 20608
Joined: 03 Jan 2005, 00:45
United States of America

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Naughty Kitty » 10 Mar 2021, 18:14

I'm glad everyone is okay thank goodness. Hugs to all. :)

User avatar
BuckyBecky
Standard Member
Posts: 100
Joined: 17 May 2019, 18:55
Great Britain

Achievements

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by BuckyBecky » 10 Mar 2021, 18:15

:dance: :dance: :dance: :dance: :dance: :dance:

User avatar
Ferefire
Ensouled Member
Ensouled Member
Posts: 47522
Joined: 20 Apr 2003, 19:55
Contact:
Great Britain

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Ferefire » 10 Mar 2021, 18:15

Likely related to security permissions being too strict. We'll check this shortly once con sites are back up.
---

User avatar
Ferefire
Ensouled Member
Ensouled Member
Posts: 47522
Joined: 20 Apr 2003, 19:55
Contact:
Great Britain

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Ferefire » 10 Mar 2021, 19:22

Test :)

Edit: Yay, post replying is fine now. Please note, there may be more short downtimes tonight as we keep configuring and adjusting settings
---

User avatar
Dark_UwU
Retroactive Member
Retroactive Member
Posts: 18542
Joined: 12 Mar 2020, 10:37
Contact:
Great Britain

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Dark_UwU » 10 Mar 2021, 19:30

Woop woop! I live UwU
Image

SpInNy FinGeRS

User avatar
Ferefire
Ensouled Member
Ensouled Member
Posts: 47522
Joined: 20 Apr 2003, 19:55
Contact:
Great Britain

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Ferefire » 10 Mar 2021, 19:34

Update: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. Make sure to add @Redhat to your Christmas Card list, folks, for his amazing work sorting things out so fast. ;)
---

User avatar
sam_sam
Standard Member
Posts: 192
Joined: 12 Feb 2020, 07:58
Great Britain

Achievements

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by sam_sam » 10 Mar 2021, 19:42

Woooo happy ending ☺️
ribbit
ImageImage

User avatar
Mattboden03
New Member
Posts: 3
Joined: 26 Feb 2021, 16:06
Contact:
Great Britain

Achievements

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Mattboden03 » 10 Mar 2021, 19:45

Welcome back

User avatar
belsammael
Veteran Member
Posts: 10290
Joined: 13 May 2007, 21:44
Netherlands

Achievements

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by belsammael » 10 Mar 2021, 20:27

Oh wow, that is pretty intense..! Glad to see the forum back up and running. :)
No one ever listens to Zathras.

User avatar
Ferefire
Ensouled Member
Ensouled Member
Posts: 47522
Joined: 20 Apr 2003, 19:55
Contact:
Great Britain

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Ferefire » 10 Mar 2021, 20:34

belsammael wrote:
10 Mar 2021, 20:27
Oh wow, that is pretty intense..! Glad to see the forum back up and running. :)
Ngl, pretty awesome for us to be a part of a Hollywood movie in the making.. xD
---

User avatar
RedHat
Entombed Member
Entombed Member
Posts: 45774
Joined: 11 Feb 2010, 01:44
Great Britain

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by RedHat » 10 Mar 2021, 22:02

Ferefire wrote:
10 Mar 2021, 19:34
Update: Now that the forums are back, we have decided to take a break and resume work on things tomorrow. Tomorrow's task will be getting the convention websites back up fully along with event tickets. ETA should be around tomorrow evening time for that. Make sure to add @Redhat to your Christmas Card list, folks, for his amazing work sorting things out so fast. ;)

Oh hi :)

Yeah, I need a break so I'll be resuming work on this tomorrow. The ticket websites are next on my list - infact I managed to configure it incorrectly and crash the server. I didn't sleep much last night, which is why I'm having a break now (or I'll crash the server some more today :confuse:).

If you find any more bugs, please let us know by posting here. This is a brand new server that I got working in a couple hours so there's bound to be some glitches.
 
ImmortalJed wrote:
I'm ill enough as it is, don't need you bunch of braindead fools nomming on me and making it worse :P

User avatar
Lolin
Veteran Member
Posts: 83172
Joined: 08 Feb 2005, 06:52
United States of America

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by Lolin » 11 Mar 2021, 01:35

When do I get AL points backpay for the time I could have spent bugging Grace?
Image

User avatar
PharaohAtem
Atem Rank
Atem Rank
Posts: 45952
Joined: 23 Nov 2004, 14:55
Contact:
United States of America

Animeleague Server and It's Sites Are Back - This is What's Happened!

Post by PharaohAtem » 11 Mar 2021, 01:40

"Forbidden
You don't have permission to access this resource.
Additionally, a 302 Found error was encountered while trying to use an ErrorDocument to
handle the request.
Apache/2.4.29 (Ubuntu) Server at www.animeleague.net Port 443"



That's the one I've had.
Image

ImageImageImageImageImage



Post Reply