In the morning hours of November 7th, one of the hypervisors in our UK location experienced a massive and unpredictable hardware failure. Upon rebooting the machine, we detected a considerable amount of I/O errors. This indicated that the hardware event had severely affected the RAID arrays in the server.
While it is still early stages to define the exact root cause, we suspect it was a failure in the backplane of the server.
During the next several hours, we attempted various recovery paths, all of which proved unsuccessful. At this stage, we believe that the data that was hosted in the affected RAID arrays is not recoverable.
In the next 24-48 hours we will perform further attempts to recover some data. But for the time being, we're proceeding as follows:
- If you have an active backup addon associated with your service, we're restoring the newest backup available in a new hypervisor to bring your VPS and its data back online.
- If you don't have an active backup addon, we're recreating your VPS in a new hypervisor with a fresh operating system and cPanel (if applicable) install.
We will be working on the above throughout the weekend.
We understand the implications of this incident and we deeply apologize for it. Servers are complex systems with many moving pieces. Despite having several kinds of monitoring systems in place, there is still a risk for unexpected hardware events, such as this one.
Thursday, November 7, 2019