Recent downtime
Posted: Tue Jun 12, 2018 7:28 amHi all,
Have been away for a few weeks and it seems everything shat itself. The website failed to renew the SSL certificate which has now been fixed. The US server is still offline but there is not much I can do about it for now. Here is the statement from our host;
Once the server is back online services will resume as per normal.
Have been away for a few weeks and it seems everything shat itself. The website failed to renew the SSL certificate which has now been fixed. The US server is still offline but there is not much I can do about it for now. Here is the statement from our host;
We've had a detailed update from our DC about the status of around 120 blade servers that have been affected by a recent firmware update.
At the beginning of June, we began upgrading our servers with the new ILO code to address both TLS vulnerability and the new Java security requirements. During last week we had 1-2 servers that failed their upgrade but completed over 1500 upgrades successfully. On Friday we started upgrading the next batch of systems in A01-05, B01-05, C01-05, J11-J15, there were not incidents detected until Saturday when a total of 174 servers dropped offline over the course of 3 hours, we’ve had a few more drop out on Sunday and Monday as well.
We notified the DC to check through the first systems and they found the server refused to power up, it just sat with a flashing red health error light. They contacted HPE for assistance and carried out their recommend downgrade procedure on a couple of the blades. It made no difference.
HP's tech investigated a few of the blades and discovered that the power management controller had failed to upgrade properly and was preventing the blade from powering up. We have tried a variety of operations with HPE to resurrect these servers, some has come back to life, some are still offline. The DC will continue to work with HPE to find a solution to these issues.
We have swapped hardware where we have available stock but have now depleted our stock of spare compatible systems so we are reliant on HPE resolving the firmware issue. In the meantime we’re shipping a couple of pallets of servers from New York and Denver to Atlanta so we can replace hardware where necessary.
The DC is working round the clock with HPE to find a resolution, if we can’t get a firmware resolution then we’ll replace the hardware as soon as additional kit arrives on site.
Its an unfortunate and messy situation all round and we apologise for the downtime.
Once the server is back online services will resume as per normal.