StableHost is Down : Their Statement

On Thursday, 7 of June at 15.04 GMT-04 we got an email from our Singapore Data Center supplier PhoenixNAP saying that they have found illegal material on our 4th server, below is the email received without any warning:

“Hello,

While we appreciate your decision to choose Secured Servers as your IT infrastructure provider, we regret to inform you that we must immediately cancel the following device:
#27435 – S04
Service ID:134267
The complaint we have received for your site regarded illegal material. Due to the sensitive nature of these complaints we will be deactivating this device immediately.

Due to regulations placed on us, we are required to hold the hard drive without any changes made or being accessed of any sort to maintain the evidence of the infringement and you will not be able to retrieve your data.

We are sorry for the inconvenience this may cause however please understand that this decision is final.
If you have further questions for us please feel free to email us back.

Thank you,
--
Abuse Management Analyst
PhoenixNAP Abuse Department“


We immediately started to contact PhoenixNAP trying to convince them to open the server and let us search for the illegal material uploaded to the server by 1 of our customers. We suggested different methods to try and get them to at least open the server for us, block all incoming traffic and much more. After phone calls and emails, we go the following response:

“Hello,
Unfortunately we can't disclose the criteria we use in determining these cases to minimize the risk of exploitation. Also at this time we can't disclose the URL or IP of the offending content. I'm very sorry for the inconvenience but as we've stated we will not be bringing the server back online and all we can do is offer you a server to replace this one.

Regards,
--
NOC Services Manager
PhoenixNAP NOC Services Team”

During this time, we kept in contact with PhoenixNAP over phone and email trying to convince them to bring the server back online, so we can have our clients up with minimum down time after clearing out all the illegal material. We also started moving all the backup from S04 to another Data Center in case of PhoenixNAP not changing their decisions to cancel the server.
About 2 hours after the first email about shutting down server 04 we got the following email:

“Hello,
While we appreciate your decision to choose Secured Servers as your IT infrastructure provider, we regret to inform you that due to excessive network abuse from your servers and heavy violation of ToS and AUP we must immediately cancel your account and remove public access to your servers.
Specifically, this is due to recent heavy ToS and AUP violation to which you agreed when you created your account.
If you have any questions about this transition, please let us know and we will do our best to provide answers.

Please understand this decision is final.

Regards,
--
Abuse Management Analyst
PhoenixNAP Abuse Department”


This means that PhoenixNAP shut down all our servers in Singapore, including the Backup Server, basically leaving us helpless without any access to our customer data. We tried to get in contact with senior management but without any success, no one would return our calls or emails.

Later in the Thursday we convinced them to open our Backup Server, so we can move the client’s data from Singapore to our main Data Center in Phoenix Arizona. They did open it up, so we could move the data, but limited us to 20MB / Second, hence the long migration process. We did multiple attempts to get them to open the other servers, so we could do a WHM to WHM migration or move the data out in any other way. After many hours of silence from PhoenixNAP we got the following 2 emails, first one responding to the bandwidth on 20MB / Second in our attempts to get them to increase this:

“Hello,
I'm sorry but we have already provided you as much bandwidth as we can at this time. Please provide the amount of data that you have downloaded so far so that we can assess how much time will be needed for this operation.

--
NOC Services Technician Tier 2
PhoenixNAP”

This means that all our servers would have to continue being moved at a 20MB / Second speed, hence the long restoration process that we have been working with the whole weekend.

The Second email we got was a response to the attempts to get the other servers online to move them out of the datacenter in any other way:

“Hello,
Our CEO isn’t available right now and has asked me to take point on this. Unfortunately we will not re-enable any of the other hosts and will keep the backup server on for as long as you need until the transfer is complete.

Regards,
Sr. Manager
NOC/SOC
PhoenixNAP”

What was used in our restoration to the Phoenix Data Center was R1Soft Bare Metal Restore, which restores the entire server and it's partitions on another server. However, when we restored the servers using BMR, we experienced multiple crashes on the MySQL. The databases need to be restored from R1Soft using their MySQL plugin or manually fixed database by database. In our attempt to bring our services online faster we tried several methods, and most of them failed as it seems R1Soft was backing up databases in an inconsistent state. This was causing the entire server to crash, so what we did was to re-import manually the databases one by one to ensure that the clients will get online as soon as possible since a mass restore wasn't possible in this case.

So, this is what has happened to our Singapore Data Center and the long weekend for all our staff, we have been working non-stop to get everything back online as soon as possible. All the servers are now migrated to the Phoenix Data Center and back online.

We would again like to apologize for this inconvenience caused to all our clients, and we also want to thank all of you, both our clients and everyone else who has given us support and understanding over this whole period.

 

They seem to be online now. My service is UP.

 

Not really good situation!

 

I think they are back to business. As of now.