Subsim forum outage June 1-2, 3-4

XabbaRus · 06-02-08, 03:16 PM

Subsim server was down due to an electrical fire at the server centre. Very big, quite a lot still to do but Subsim has recovered and is back on the waves.

H1 Data Center Offline - Status Updates

New items and articles

FIRRRRRE!

Quote:

The Planet Explosion Causes Outages

By Justin Lee, theWHIR.com
June 2, 2008 -- (WEB HOST INDUSTRY REVIEW) -- Dedicated hosting provider The Planet (theplanet.com) suffered an explosion, 4:55 pm CST on Saturday at its H1 data center in Houston, Texas, which led to an outage that affected approximately 9,000 servers and 7,500 customers.

The explosion and accompanying fire knocked down three walls surrounding the facility's electrical equipment room, as well as destroyed the underground cabling that powers the first floor of the facility. Fortunately, there were no injuries, nor were any servers damaged or lost.

The company has been in constant communication with its customers -- an essential aspect of any disaster recovery plan -- through its forum (forums.theplanet.com) and customer portal, as well as other popular forums like Web Hosting talk (webhostingtalk.com).

In an earlier post yesterday on The Planet's forums, Kevin Hazard said the "short was in a high-volume wire conduit" and that the fire department instructed the company to not "activate [its] backup generator plan."

After the fire marshall inspected the facility The Planet was allowed to restore power to the facility. The company immediately turned back on the generators to receive power on the second floor.
Then, earlier this morning, the company successfully brought back 90 percent of the second floor servers after more than 40 hours of troubleshooting. Meanwhile, support technicians are currently working on location to manually restore the remaining 10 percent on the second floor, as well as restore power to the first floor.

Despite this, many customers, namely those companies who have lost sales, customers, and other revenues, remain upset by how long the recovery process is taking and have actively voiced these criticisms on forums.

One WHT member C0bra writes: "I'm fed up waiting for more information about these terrifying issues. Communication is poor and every minute my site is down is costing me $1000s in ad revenues. This is probably going to cost me a $10,000,000 sponsorship deal with a major brand advertiser."

Douglas Erwin, The Planet chairman and CEO, responded to these complaints late Sunday evening onThe Planet forum.
He writes: "I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.
"We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed."

Dowly · 06-02-08, 03:19 PM

Great news!

CCIP · 06-02-08, 03:31 PM

Wow! I was wondering what happened. Good to have it back however.

nikimcbee · 06-02-08, 03:52 PM

Yeah, it's working again. I was going through withdrawls:rotfl:

Kapitan_Phillips · 06-02-08, 03:52 PM

Hey guys, I've been in contact with Neal, he asked me to add this:

'The datacenter which hosts the Subsim and Deep Domain server had a transformer explosion late Sat night. The techs restored power Monday June 2 approx. 1430. We apologize for the inconvenience.

Neal'

http://techblog.dallasnews.com/archi...tinues-re.html

Jimbuna · 06-02-08, 04:01 PM

Pleased to see it's sorted.....was beginning to wonder if the server was stored in the King Kong buildinf at Universal Studios

HunterICX · 06-02-08, 04:03 PM

Glad to hear its getting sorted

HunterICX

nikimcbee · 06-02-08, 04:02 PM

Quote:

Originally Posted by Kapitan_Phillips

Hey guys, I've been in contact with Neal, he asked me to add this:

'The datacenter which hosts the Subsim and Deep Domain server had a transformer explosion late Sat night. The techs restored power Monday June 2 approx. 1430. We apologize for the inconvenience.

Neal'

http://techblog.dallasnews.com/archi...tinues-re.html

Tank u 4 callin the sub simm servr farm hot line fors custers survice

Yez, Neal is aware subsimms iz down.
What?
Yes, Steed only recived minor burnz. Hez okay

Jimbuna · 06-02-08, 04:27 PM

Quote:

Originally Posted by nikimcbee

Quote:

Originally Posted by Kapitan_Phillips

Hey guys, I've been in contact with Neal, he asked me to add this:

'The datacenter which hosts the Subsim and Deep Domain server had a transformer explosion late Sat night. The techs restored power Monday June 2 approx. 1430. We apologize for the inconvenience.

Neal'

http://techblog.dallasnews.com/archi...tinues-re.html

Tank u 4 callin the sub simm servr farm hot line fors custers survice

Yez, Neal is aware subsimms iz down.
What?
Yes, Steed only recived minor burnz. Hez okay

ROFLMAO

Tarred Rope · 06-05-08, 05:38 PM

Quote:

Originally Posted by Dowly

Great news!

Indeed, mon ami. In other news, I heard someone called "Bernard" was working at theplanet.com on the night of the explosion..???

Wow!!!! Welcome back and best wishes to Neal & Co! I heard about the fire from The Register and I thought it would've set you back for much longer than this. Absolutely incredible job. I have so much respect for you guys I can't say any more.

bookworm_020 · 06-05-08, 05:56 PM

Another two days without sub sim!

My heart can't take much more of this!

BTW, Neal, I love what you have done to your sig!

Flick · 06-05-08, 06:20 PM

Glad to see it's back up and all in tact Neal.

rifleman13 · 06-05-08, 07:22 PM

Noone can bring SubSim down!

Keep up the good work boys!

arkroyal · 06-07-08, 04:09 AM

Good to have you back

06-02-08, 03:31 PM	#3
CCIP Navy Seal Join Date: Apr 2005 Location: Waterloo, Canada Posts: 8,700 Downloads: 29 Uploads: 2	Wow! I was wondering what happened. Good to have it back however. __________________ There are only forty people in the world and five of them are hamburgers. -Don Van Vliet (aka Captain Beefheart)

06-02-08, 03:52 PM	#4
nikimcbee Fleet Admiral Join Date: Jul 2003 Location: Patroling the Slot. Posts: 17,952 Downloads: 90 Uploads: 0	Yeah, it's working again. I was going through withdrawls:rotfl: __________________

06-02-08, 03:52 PM	#5
Kapitan_Phillips Silent Hunter Join Date: Jul 2005 Location: Swansea Posts: 3,903 Downloads: 204 Uploads: 0	Details from Neal Hey guys, I've been in contact with Neal, he asked me to add this: 'The datacenter which hosts the Subsim and Deep Domain server had a transformer explosion late Sat night. The techs restored power Monday June 2 approx. 1430. We apologize for the inconvenience. Neal' http://techblog.dallasnews.com/archi...tinues-re.html __________________ Well, here's another nice mess you've gotten me into.

06-02-08, 04:01 PM	#6
Jimbuna Chief of the Boat Join Date: Feb 2006 Location: 250 metres below the surface Posts: 190,938 Downloads: 63 Uploads: 13	Pleased to see it's sorted.....was beginning to wonder if the server was stored in the King Kong buildinf at Universal Studios __________________ Wise men speak because they have something to say; Fools because they have to say something. Oh my God, not again!!

06-02-08, 04:03 PM	#7
HunterICX Rear Admiral Join Date: May 2006 Location: Malaga, España Posts: 10,750 Downloads: 8 Uploads: 0	Glad to hear its getting sorted HunterICX __________________

06-02-08, 03:19 PM	#2
Dowly Lucky Jack Join Date: Apr 2005 Location: Finland Posts: 25,055 Downloads: 32 Uploads: 0	Great news!

06-05-08, 05:56 PM	#11
bookworm_020 Navy Seal Join Date: Aug 2005 Location: Sinking ships off the Australian coast Posts: 5,966 Downloads: 1 Uploads: 0	Another two days without sub sim! My heart can't take much more of this! BTW, Neal, I love what you have done to your sig! __________________ "There is always one person in any organization who knows whats going on, That person must be fired"

06-05-08, 06:20 PM	#12
Flick Sparky Join Date: Mar 2002 Location: Ottawa Canada Posts: 157 Downloads: 8 Uploads: 0	Glad to see it's back up and all in tact Neal.

06-05-08, 07:22 PM	#13
rifleman13 Grey Wolf Join Date: May 2008 Location: Depth-charged to Kingdom Come Posts: 927 Downloads: 28 Uploads: 0	Noone can bring SubSim down! Keep up the good work boys!

06-07-08, 04:09 AM	#14
arkroyal Bosun Join Date: Jul 2005 Location: Gosport England Posts: 66 Downloads: 31 Uploads: 0	Good to have you back __________________ Two types of ships in the royal navy submarines and targets!! DIESEL BOATS FOR EVER:rotfl: