Recent downtime

Server, website and mod update information.

Recent downtime

Postby BadBoy » Tue Jun 12, 2018 7:28 am

Hi all,

Have been away for a few weeks and it seems everything shat itself. The website failed to renew the SSL certificate which has now been fixed. The US server is still offline but there is not much I can do about it for now. Here is the statement from our host;

We've had a detailed update from our DC about the status of around 120 blade servers that have been affected by a recent firmware update.

At the beginning of June, we began upgrading our servers with the new ILO code to address both TLS vulnerability and the new Java security requirements. During last week we had 1-2 servers that failed their upgrade but completed over 1500 upgrades successfully. On Friday we started upgrading the next batch of systems in A01-05, B01-05, C01-05, J11-J15, there were not incidents detected until Saturday when a total of 174 servers dropped offline over the course of 3 hours, we’ve had a few more drop out on Sunday and Monday as well.

We notified the DC to check through the first systems and they found the server refused to power up, it just sat with a flashing red health error light. They contacted HPE for assistance and carried out their recommend downgrade procedure on a couple of the blades. It made no difference.

HP's tech investigated a few of the blades and discovered that the power management controller had failed to upgrade properly and was preventing the blade from powering up. We have tried a variety of operations with HPE to resurrect these servers, some has come back to life, some are still offline. The DC will continue to work with HPE to find a solution to these issues.

We have swapped hardware where we have available stock but have now depleted our stock of spare compatible systems so we are reliant on HPE resolving the firmware issue. In the meantime we’re shipping a couple of pallets of servers from New York and Denver to Atlanta so we can replace hardware where necessary.

The DC is working round the clock with HPE to find a resolution, if we can’t get a firmware resolution then we’ll replace the hardware as soon as additional kit arrives on site.

Its an unfortunate and messy situation all round and we apologise for the downtime.



Once the server is back online services will resume as per normal.
User avatar
BadBoy
Site Admin
 
Posts: 764
Joined: Thu Oct 23, 2014 3:07 am
Status: Offline

Re: Recent downtime

Postby iznogod » Fri Jun 15, 2018 8:25 pm

From what I read about the issue it's not gonna be resolved quickly. Already 3 days of downtime. Fun.
iznogod
Server Moderator
 
Posts: 75
Joined: Fri Dec 12, 2014 12:28 pm
Status: Offline

Re: Recent downtime

Postby kerlos » Sun Jun 17, 2018 8:40 pm

Hello BadBoy, it is good to hear from you
You said (The US server is still offline)
Does this mean that the other servers are online, for example EU1, EU2, and OC1
Because for me I can not access any servers from King clan
kerlos
Donator
 
Posts: 5
Joined: Mon Dec 25, 2017 1:15 am
Status: Offline

Re: Recent downtime

Postby HomerS » Tue Jun 19, 2018 7:41 am

Thanks for the update BadBoy. Been wondering why the server have been down for so long. I am relieved to hear that the problem is being worked, and look forward to connecting back to the US server.
HomerS
Server Moderator
 
Posts: 113
Joined: Thu Mar 01, 2018 8:37 am
Status: Offline

Re: Recent downtime

Postby Pvt.Pantzov » Thu Jun 21, 2018 1:04 pm

Thanks for the update. Auzzie servers are down too as of the current time. Hopefully it's nothing like the clusterf*ck regarding the US servers.
Pvt.Pantzov
Server Moderator
 
Posts: 487
Joined: Fri Nov 28, 2014 2:39 pm
Location: Stalingrad, Russia
Status: Offline

Re: Recent downtime

Postby iznogod » Fri Jun 22, 2018 6:19 pm

Aussie servs seem down indeed, yet i recall badboy saying they burned through the traffic limit after they were downgraded a few weeks ago.

Although they ARE down, they might not be down for long.
iznogod
Server Moderator
 
Posts: 75
Joined: Fri Dec 12, 2014 12:28 pm
Status: Offline

Re: Recent downtime

Postby BadBoy » Mon Jun 25, 2018 12:14 pm

Yeah due to the US server being down we ran out of our allocated bandwidth limits for the AU server.

Good news though... all servers are back online, but with new IPs.

US1: 162.219.26.82:28960
US2: 162.219.26.82:28961
US3: 162.219.26.82:28962

OC1: 139.99.197.35:28960
OC2: 139.99.197.35:28962
User avatar
BadBoy
Site Admin
 
Posts: 764
Joined: Thu Oct 23, 2014 3:07 am
Status: Offline

Re: Recent downtime

Postby kerlos » Sat Jun 30, 2018 12:03 am

Uhhh this is great new badboy, thanks a lot man
I really missed the game and the people so much
Thanks again
kerlos
Donator
 
Posts: 5
Joined: Mon Dec 25, 2017 1:15 am
Status: Offline


Return to News

Who is online

Users browsing this forum: No registered users and 4 guests

cron