Subsim forum outage June 1-2, 3-4

[XabbaRus]

Subsim server was down due to an electrical fire at the server centre. Very big, quite a lot still to do but Subsim has recovered and is back on the waves.

H1 Data Center Offline - Status Updates (http://forums.theplanet.com/index.php?showtopic=90185&st=0)

New items and articles (http://news.google.com/news?hl=en&tab=wn&ned=&q=planet+houston+servers&btnG=Search+News)

http://blog.nj.com/centraljersey_impact/2007/11/large_StonickerFire2.JPG
FIRRRRRE!


The Planet Explosion Causes Outages

By Justin Lee, theWHIR.com
June 2, 2008 -- (WEB HOST INDUSTRY REVIEW) -- Dedicated hosting provider The Planet (theplanet.com (http://www.theplanet.com/)) suffered an explosion, 4:55 pm CST on Saturday at its H1 data center in Houston, Texas, which led to an outage that affected approximately 9,000 servers and 7,500 customers.

The explosion and accompanying fire knocked down three walls surrounding the facility's electrical equipment room, as well as destroyed the underground cabling that powers the first floor of the facility. Fortunately, there were no injuries, nor were any servers damaged or lost.

The company has been in constant communication with its customers -- an essential aspect of any disaster recovery plan -- through its forum (forums.theplanet.com (http://forums.theplanet.com/)) and customer portal, as well as other popular forums like Web Hosting talk (webhostingtalk.com (http://www.webhostingtalk.com/)).

In an earlier post yesterday on The Planet's forums, Kevin Hazard said the "short was in a high-volume wire conduit" and that the fire department instructed the company to not "activate [its] backup generator plan."

After the fire marshall inspected the facility The Planet was allowed to restore power to the facility. The company immediately turned back on the generators to receive power on the second floor.
Then, earlier this morning, the company successfully brought back 90 percent of the second floor servers after more than 40 hours of troubleshooting. Meanwhile, support technicians are currently working on location to manually restore the remaining 10 percent on the second floor, as well as restore power to the first floor.

Despite this, many customers, namely those companies who have lost sales, customers, and other revenues, remain upset by how long the recovery process is taking and have actively voiced these criticisms on forums.

One WHT member C0bra writes: "I'm fed up waiting for more information about these terrifying issues. Communication is poor and every minute my site is down is costing me $1000s in ad revenues. This is probably going to cost me a $10,000,000 sponsorship deal with a major brand advertiser."

Douglas Erwin, The Planet chairman and CEO, responded to these complaints late Sunday evening onThe Planet forum.
He writes: "I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.
"We understand that you will be due service credits based on our Service Level Agreement. We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed."

[Dowly]

Great news! :up:

[CCIP]

Wow! I was wondering what happened. Good to have it back however.

[nikimcbee]

Yeah, it's working again. I was going through withdrawls:rotfl:

[Kapitan_Phillips]

Hey guys, I've been in contact with Neal, he asked me to add this:

'The datacenter which hosts the Subsim and Deep Domain server had a transformer explosion late Sat night. The techs restored power Monday June 2 approx. 1430. We apologize for the inconvenience.

Neal'

http://techblog.dallasnews.com/archives/2008/06/update-the-planet-continues-re.html

[Jimbuna]

Pleased to see it's sorted.....was beginning to wonder if the server was stored in the King Kong buildinf at Universal Studios :lol:

[nikimcbee]

Hey guys, I've been in contact with Neal, he asked me to add this:

'The datacenter which hosts the Subsim and Deep Domain server had a transformer explosion late Sat night. The techs restored power Monday June 2 approx. 1430. We apologize for the inconvenience.

Neal'

http://techblog.dallasnews.com/archives/2008/06/update-the-planet-continues-re.html (http://techblog.dallasnews.com/archives/2008/06/update-the-planet-continues-re.html)

Tank u 4 callin the sub simm servr farm hot line fors custers survice
http://icanhascheezburger.files.wordpress.com/2008/05/funny-pictures-technical-support-cat.jpg

Yez, Neal is aware subsimms iz down.
What?
Yes, Steed only recived minor burnz. Hez okay

[HunterICX]

Glad to hear its getting sorted :up:

HunterICX

[Skybird]

And I thought it was a mass-keelhauling underway... :88)

[Fubar2Niner]

YaaaaaaaaaaaaaaaaaaaaaaaaaaaaaYYYY !!!!!!!! my fav forum back at last :rock: I missed you guys last couple days. Hope there weren't too much damage shipmates.

Best regards.

Fubar2Niner

[Jimbuna]

Hey guys, I've been in contact with Neal, he asked me to add this:

'The datacenter which hosts the Subsim and Deep Domain server had a transformer explosion late Sat night. The techs restored power Monday June 2 approx. 1430. We apologize for the inconvenience.

Neal'

http://techblog.dallasnews.com/archives/2008/06/update-the-planet-continues-re.html

Tank u 4 callin the sub simm servr farm hot line fors custers survice
http://icanhascheezburger.files.wordpress.com/2008/05/funny-pictures-technical-support-cat.jpg

Yez, Neal is aware subsimms iz down.
What?
Yes, Steed only recived minor burnz. Hez okay

ROFLMAO http://imgcash3.imageshack.us/img152/9959/rollingaroundlaughingly2.gif

[Jacky Fisher]

so that explains everything.....:-?

[ReallyDedPoet]

Welcome back SUBSIM :yep::up:


RDP

[NightCrawler-SimHQ]

Great to have you back, Subsim! Now all is needed is some new fire extinguishers! :lol:

[les green01]

woowoo back in buisness this was worst than not having coffee and i think smokes

[rifleman13]

Almost two days without Subsim... almost an eternity!

Good to have the the site and RadioRoom back online!:rock:

You guys all rock.:rock:

[bookworm_020]

Glad it's back, I was starting to suffer from withdrawl!:o

[Brag]

I was zinking kargoz instead of jabbering in the forumz. And talking to Snuggles.:|\\

[kiwi_2005]

Yea i read about it on ubisofts SH forums, even over there they were upset subsim wasn't up.

[Wreford-Brown]

Welcome back, subsim.
Back to sleepless nights and bollockings from the wife!:arrgh!:

[dcb]

I heard about the blaze and was afraid all this great info would be lost. Fortunately, this is not the case. Welcome back, Subsim:up:

[McBeck]

Thats was a couple of creepy days!!!

[Catfish]

Hello,
whenever Subsim is down i think there must be something wrong with MY internet connection ;). This site is running almost flawless since 2001, reminds me of donating :yep:
Thanks and greetings,
Catfish

[Onkel Neal]

As you probably, know the datacenter's backup generator failed, and we were down 2 more days.

[Sailor Steve]

It was a doubly weird experience for me. I lost my internet connection at about the same time, and didn't have a problem until I went to the library to post. The coincidence was so improbable that I assumed that the library wouldn't let me connect for some reason. It wasn't until I got my connection back yesterday that I found the problem wasn't on my end!

Very strange.:-?

[STEED]

I take a break and you guys were down! :huh:

What was the bloody point of my break? :damn:

FECK...................

Next time tell me your going down so I can cancel my short break and go and get drunk. :stare:

So you went up in smoke for a while! You should have told me so, I could toss you some fire extinguishers across the pond while my boss was busy shagging her mistress. :roll:

[Rhodes]

Ah! Subsim back on-line! The good old days are back!

http://gigasmilies.googlepages.com/113.gif http://img214.imageshack.us/img214/3765/jee21sw7.gif

[Tarred Rope]

Great news! :up: Indeed, mon ami. In other news, I heard someone called "Bernard" was working at theplanet.com on the night of the explosion..??? :D

Wow!!!! Welcome back and best wishes to Neal & Co! I heard about the fire from The Register and I thought it would've set you back for much longer than this. Absolutely incredible job. I have so much respect for you guys I can't say any more. :up:

[bookworm_020]

Another two days without sub sim!:o My heart can't take much more of this!

BTW, Neal, I love what you have done to your sig!:rock:

[Flick]

Glad to see it's back up and all in tact Neal.

[rifleman13]

Noone can bring SubSim down!:rock:

Keep up the good work boys!:up:

[LtCmdrRat]

Subsim Crew xxx
Successfully passed fire on board training xxx No human life lost xxx Congratulations xxx Just 3500 brokers jumped from skyscribers xxx Panic on stock is over xxx 7 feet under the kiel xxx
Neptune & HQ

[les green01]

looks what happens when i donate lmao

[Venatore]

Gee, that was a bad couple of days for me.....depression, unsocialable, short tempered, headaches, nausea, vomiting, dizziness,

But when I jumped on the PC today and logged back into Subsim, all that crap above just subsided.......strange.

[Onkel Neal]

You guys don't know how much your kind words help. Thanks :yep:

[ryoga_77]

Thank heavens this morning i found subsim up and about instead of the usual error msg,i was dreading this site going the way of the dodo.Luckily my worst fears have been proved wrong.:D

[nikimcbee]

Gee, that was a bad couple of days for me.....depression, unsocialable, short tempered, headaches, nausea, vomiting, dizziness,

But when I jumped on the PC today and logged back into Subsim, all that crap above just subsided.......strange.

Same here, atfirst, I was afraid I hadn't ordered enough stuff and Neal coundn't pay the bills.:dead:

[HunterICX]

:up: Now lets hope it stays up this time :yep:

Damn, how much I havent been checking subsim during the day seeying if it was back online yet...seems I'm addicted to this site as well :oops:

HunterICX

[dcb]

seems I'm addicted to this site as well

You're not alone, mate. We're all addicted to this great place called subsim!

[Jimbuna]

Pleased to see the site is up and running again Neal http://www.psionguild.org/forums/images/smilies/wolfsmilies/thumbsup.gif





Hunter was beginning to make a pest of himself over at Ubisoft http://img113.imageshack.us/img113/7975/gigglebigtb9fg3.gif

[Rockin Robbins]

Captain Neal:

DD's going to pass to port, you said. Hard starboard, I suggested. He'll be clear to port, you said. Nothing to worry about, you said.

kaBOOOOOOOOMMMM!

Welcome to the new Fubuki with side throwers. Told you so, I said. Good thing our damage control is the best...:ping:

[ReM]

Good to see the site up again!

[Snakeeyes]

Didn't keep me from trying to access the site nor did it stop the Japanese merchant fleet from sinking! :rock:

[Onkel Neal]

Gee, that was a bad couple of days for me.....depression, unsocialable, short tempered, headaches, nausea, vomiting, dizziness,

But when I jumped on the PC today and logged back into Subsim, all that crap above just subsided.......strange.

Same here, atfirst, I was afraid I hadn't ordered enough stuff and Neal coundn't pay the bills.:dead:

Speaking of which, I want to say special thanks to Wolfram, Chris, Leo, Robert, Craig, Justin (who's always there for Subsim), Emilio, and my partner Bishop for the wonderful deluge of donations. The most we've had in a while :up:

[NealT]

I am just wondering...

With the outtage...

How many little Subsimmer's will be born in 9 months?

:hmm::hmm::hmm:

[GSpector]

I am just wondering...

With the outtage...

How many little Subsimmer's will be born in 9 months?

:hmm::hmm::hmm:

9 Months later....
:lurk: http://content.sweetim.com/sim/cpie/emoticons/0002028C.gif (http://www.sweetim.com/s.asp?im=gen&ref=10)





Welcome back :up:

http://content.sweetim.com/tbsig/sig.asp?img=ad1 (http://www.sweetim.com/s.asp?im=gen&ref=12)

[Onkel Neal]

In accordance with Subsim tradition, this keeps getting weirder and weirder... (http://www.texasstartupblog.com/2008/06/03/data-center-explosion-fire-or-fbi-bust/)

...I also heard from a source aliens came down from space and demanded 9000 servers or they would destroy Paris Hilton's dog.

Damn dog.

[GSpector]

In accordance with Subsim tradition, this keeps getting weirder and weirder... (http://www.texasstartupblog.com/2008/06/03/data-center-explosion-fire-or-fbi-bust/)

...I also heard from a source aliens came down from space and demanded 9000 servers or they would destroy Paris Hilton's dog.

Damn dog.

Let them have the dog, we can get more service from the servers:up:

[nikimcbee]

In accordance with Subsim tradition, this keeps getting weirder and weirder... (http://www.texasstartupblog.com/2008/06/03/data-center-explosion-fire-or-fbi-bust/)

...I also heard from a source aliens came down from space and demanded 9000 servers or they would destroy Paris Hilton's dog.

Damn dog.

Nevermind the dog, give them Paris for God's sake. And we'll throw in Brittney as well.

And now, for something completly different...

[nikimcbee]

If the FBI is involved, I'd say we have a mystery on our hands.:|\\ :hmm:
Get some scooby snacks and your tinfoil helmets and lets solve this mystery!:up:

Who would have the most to gain by having subsim go down?

[arkroyal]

:arrgh!: Good to have you back

[rifleman13]

In accordance with Subsim tradition, this keeps getting weirder and weirder... (http://www.texasstartupblog.com/2008/06/03/data-center-explosion-fire-or-fbi-bust/)

...I also heard from a source aliens came down from space and demanded 9000 servers or they would destroy Paris Hilton's dog.

Damn dog.
Let them have the dog, we can get more service from the servers:up:

Mulder and Scully are right on it!

:rotfl::rotfl::rotfl:

[Lzs von swe]

Noo, it was The Goauld!! Jack and the rest of SG1 finally pissed them of so they decided to strike against the hottest place on Earth. The SUBSIM servers..

The whole thing is creepy, my trusted old Nokia 447Za 17 inch Crt monitor gave up in a cloud of smoke and the network acted up at the same time as Subsim was down:hmm: And at work I was grabbing the wheel with sweaty hands, wondering, will SUBSIM be back when I get home again??

Glad to have you all back again :up: :up:

[horsa]

Strange experience .... it's like you wake up and hear that half your family just got wiped out in a plane crash :o

[Jimbuna]

Noo, it was The Goauld!! Jack and the rest of SG1 finally pissed them of so they decided to strike against the hottest place on Earth. The SUBSIM servers..

The whole thing is creepy, my trusted old Nokia 447Za 17 inch Crt monitor gave up in a cloud of smoke and the network acted up at the same time as Subsim was down:hmm: And at work I was grabbing the wheel with sweaty hands, wondering, will SUBSIM be back when I get home again??

Glad to have you all back again :up: :up:

http://imgcash2.imageshack.us/img407/1073/vortex04lw6.gif ..... http://imgcash6.imageshack.us/img407/8895/vortex05wo9.gif

[Onkel Neal]

I just completed making a backup of the entire website. :()1: I have a cron that weekly backs up the web pages but I don't completely trust it. The forums take the most work, you cannot export really big files without risking some file corruption, so the 3GB database (97 tables) has to come in 50MB chunks, keeping careful count of the # of rows and which row each export begins with.... :dead: Now for some sleeeeeeppppp:zzz: :zzz:

[geetrue]

Just a matter of time before they blame this on Bush/Cheney :lol:

[WutWuzDat]

And I thought it was a mass-keelhauling underway... :88)

ROFL :lol:

[nikimcbee]

Just a matter of time before they blame this on Bush/Cheney :lol:
I was going to say Rumsfeld and Halliburton.

[TLAM Strike]

Ok now who is going to seek in to Neal's bedroom and night and play this (http://www.commanders-academy.com/tlamstrike/Cruel_Joke.mp3)...

[Jimbuna]

I'll bet on Dowly :lol:

[HunterICX]

:rotfl: Nice one!

HunterICX

[Dowly]

Meh... :shifty:

[GSpector]

Where's Bernard:hmm:
:lol:

[STEED]

Dowly is to cool for such a thing. :cool:

Any how Dowly is running for president of Norway. ;)

[rifleman13]

@TLAM Strike: Neal's going to be running like heck with fire gear on when he hear's that!

Where's Bernard:hmm:
:lol:

On the way to Berlin to face a firing squad. You could save him, the truck has no escort, but who want so save him? You?:shifty:

:rotfl::rotfl::rotfl:

[nikimcbee]

Dowly is to cool for such a thing. :cool:

Any how Dowly is running for president of Norway. ;)

Dowly is Norwegian? That explains a lot.

[GOZO]

Dowly is to cool for such a thing. :cool:

Any how Dowly is running for president of Norway. ;)

Dowly is Norwegian? That explains a lot.

Well now before he sees this..

Dowly is from Finland! Not far away from Norway but still Sweden + the Baltic sea in between.:D ;)

/OB

[nikimcbee]

http://www.skandisk.com/cat-imagesL/hbk125L.jpg

http://www.skandisk.com/cat-imagesL/hbk122L.jpg

No, I think he's Norwegian:rotfl: He's just switched teams>

You Know You've Been In Finland Too Long, When...
You meticulously manage your plastic bag collection.
You don't think twice about putting wet dishes in the cupboard to dry.
Silence is fun.
Your coffee consumption exceeds 8 cups a day.
You pass a grocery store and think: "Wow, it's open!"
Your native language has seriously deteriorated. Now you "eat medicine", "open the television", and "close the lights off".
You associate pea soup with Thursday.
Your notion of street life is reduced to hanging out in front of the railway station on Friday nights.
After a presentation, you finally stop asking "Are there any questions?"
Hugging is reserved for sexual foreplay.
You no longer look at a track suit as casual wear, but consider it acceptable for formal occasions. Neither do you see a problem wearing white socks with loafers.
You accept alcohol as food.
You no longer eat mashed potatoes - you eat smashed potatoes.
You understand why the Finnish language has no future tense.
You know that "one" beer means "let's get pissed."
When a stranger smiles at you, you assume he is drunk, insane, or American.
You've become lactose intolerant.
You know how to prepare herring 105 different ways.

[nikimcbee]

Finnish drinking game
There are two versions of this game for Finns; regular and advanced.

Regular: Three Finnish guys go into the sauna, each with half a litre of Kossu (Finland's famous Koskenkorva vodka). They each drink the vodka, and then one guy goes outside. The other two have to guess who went outside....

Advanced: TWO Finnish guys go into the sauna, each with a litre of Kossu. They each drink the vodka, and then one guy goes outside. The other guy has to guess who went outside....

Famous last words of Finnish men

"Naah, we dont need no electrician here."
"We can go to my place - wife's on night shift"
"I love you Kristiina... eh, I mean Hanna..."
"In principal you shouldnt smoke so near the ammunition"
"Lets study the safety instructions later"
"The side effects of lot of alcohol is hugely exaggerated"
"I got some cheap Russian spirits to the wedding punch so the whole family can drink enough"
"Damn life save vest - not bothering to wear them"
"Look! Whats that bear cub doing alone in the forest?"
"Damn quick to drill the ice when it's this thin."

Finnish weather explained
+15°C / 59°F
This is as warm as it gets in Finland, so we'll start here.
People in Spain wears winter-coats and gloves.
The Finns are out in the sun, getting a tan.
+10°C / 50°F
The French are trying in vain to start their central heating.
The Finns plant flowers in their gardens.
+5°C / 41°F
Italian cars won't start.
The Finns are cruising in cabriolets.
0°C / 32°F
Distilled water freezes.
The water in the Vanda river (in Finland) gets a little thicker.
-5°C / 23°F
People in California almost freeze to death.
The Finns have their final barbecue before winter.
-10°C / 14°F
The Brits start the heat in their houses.
The Finns start using long sleeves.
-20°C / -4°F
The Aussies flee from Mallorca.
The Finns end their Midsummer celebrations. Autumn is here.
-30°C / -22°F
People in Greece die from the cold and disappear from the face of the earth.
The Finns start drying their laundry indoors.
-40°C / -40°F
Paris start cracking in the cold.
The Finns stand in line at the "grilli-kioski".
-50°C / -58°F
Polar bears start evacuating the North Pole.
The Finnish army postpones their winter survival training awaiting real winter weather.
-60°C / -76°F
Korvatunturi (the home for Santa Claus) freezes.
The Finns rent a movie and stay indoors.
-70°C / -94°F
The false Santa moves south.
The Finns get frustrated since they can't store their Kossu (Koskenkorva vodka) outdoors.
The Finnish army goes out on winter survival training.
-183°C / -297.4°F
Microbes in food don't survive.
The Finnish cows complain that the farmers' hands are cold.
-273°C / -459.4°F
All atom-based movent halts.
The Finns start saying "Perkele, it's cold outside today."
-300°C / -508°F
Hell freezes over.
Finland wins the Eurovision Song Contest.


The following are Swedish jokes, but Finns like them!
Construction workers
Three construction workers, an Australian, a Finn and a Swede, are sitting on a beam on the tenth floor about to have their lunch.
The Australian opens his lunch box and says "Bloody hell - meat pies again! Every day it's bloody meat pies! If I get meat pies again tomorrow, I'm going to jump!"
The Finn opens up his lunch next. "Saatana! Makkara (sausage) again! Always sausages! If I get sausage tomorrow, I'm gonna jump too!"
The Swede is the last to open up his lunch. "Ah crap - meatballs again! Why always meatballs? If I get meatballs tomorrow, I'm going to jump too!"
The next day the Aussie opens his lunch box and it's a meat pie... He jumps to his death.
The Finn opens his lunch box and, yes, it's a sausage. He too jumps to his death.
The Swede opens his lunch and sadly there's a pile of meatballs, so he jumps too.
The three widows of the construction workers are talking at the funeral and the Aussie's wife says "I don't understand. I thought my husband loved meat pies! If he didn't want them he should have said something!"
The Finnish widow says "Same here - I thought my husband wanted sausages! Why didn't he say something?"
The Swede's widow says, "I don't get it... my husband made his own lunch."


Technology
An American, a Finn and a Swede are in the sauna together. Suddenly there is a "beep beep" sound, and the American starts to look at the palm of his hand.
"What are you doing?" asks the Finn. The American replies
"This is the latest Motorola technology. I've got my pager embedded in the palm of my hand, so I don't have to carry it around any more."
Then the familiar old Nokia ring tone is heard, and the Finn starts looking at the palm of his hand.
"What are you doing?" ask the other guys. The Finn replies
"This is the latest Nokia technology. I've got my mobile phone embedded in the palm of my hand, so I don't have to carry it around any more."
The Swede thinks to himself that he'd better not be outdone by these guys, so he leaves the sauna. In a couple of minutes he returns, and there is toilet paper hanging out of his bum!
"What the hell is that??" shout the other guys in unison.
"I'm getting a fax." says the Swede.

[STEED]

Dowly is to cool for such a thing. :cool:

Any how Dowly is running for president of Norway. ;)

Dowly is Norwegian? That explains a lot.

Well now before he sees this..

Dowly is from Finland! Not far away from Norway but still Sweden + the Baltic sea in between.:D ;)

/OB

Dowly is just too cool for Finland. :|\\ And anyhow they know he has the biggest collection of.................:shifty: Moving on, Dowly is just the right cool person Norway needs. :cool: :yep:

[Dowly]

Wackos, all of you. :rotfl:

@GOZO, no need to say the Sweden+Baltic Sea when explaing what's between us & Norway. Just say NOI (Nothing of Importance), much easier and faster. ;)

Right, carry on. :up:

[Fincuan]

:rotfl: @ nikimcbee

[Onkel Neal]

I'm proud to say the Subsim forums have been up and running for a full week without interruption, disaster, or catastrophe. This could be due in part or full to the new hampster recently brought in to power the Subsim server generator.

http://www.planetandy.com/shamster1.gif
Actual screencap from Subsim server livecam


This hamster was made possible by a donation from Mr. Dowly. You too can do your part to keep the little fellow (hamster, not Dowly) fed, groomed, alive and kicking. There's a small "Make a Donation" button in the top right of this forum--yes! that's it, there--and your support is appreciated.

Now, to go out and enjoy a little Texas subshine, er, sunshine. :ping:

Onkel Neal

[nikimcbee]

I'm proud to say the Subsim forums have been up and running for a full week without interruption, disaster, or catastrophe. This could be due in part or full to the new hampster recently brought in to power the Subsim server generator.

http://www.planetandy.com/shamster1.gif
Actual screencap from Subsim server livecam


This hamster was made possible by a donation from Mr. Dowly. You too can do your part to keep the little fellow (hamster, not Dowly) fed, groomed, alive and kicking. There's a small "Make a Donation" button in the top right of this forum--yes! that's it, there--and your support is appreciated.

Now, to go out and enjoy a little Texas subshine, er, sunshine. :ping:

Onkel Neal

Could we make a double donation: feed the hamster and keep Dowly fed, groomed and alive?

[Onkel Neal]

Yeah, but be warned: Dowly eats a LOT!

[nikimcbee]

Yeah, but be warned: Dowly eats a LOT!

If ferrets are like guinea pigs, I'll just buy a large bag of carrots!:rotfl: