Recent Subsim server disruption thread

[Onkel Neal]

My server tech and I are trying to locate and repair crashed tables in the forum db. We had one 20 minute server disruption just now, I will try to keep the outage to a minimum. Sorry for the inconvenience.

Neal

[Onkel Neal]

FYI: We may need to lock the forum for an hour or so, but this may not happen until tomorrow. I have to run over to the BB Texas and give some folks from Switzerland, France, and UK a tour.

[Dowly]

Neal, any word on what has been causing the lock ups recently? I think we've had 3 downtimes this month alone. :hmmm:

[Onkel Neal]

So far, two different "experts" have been unable to detect what is causing server A to hang.

[JScones]

So far, two different "experts" have been unable to detect what is causing server A to hang.
GWX? SH3Cmdr? Have they checked those?

[Onkel Neal]

You mean, as downloads?

[JScones]

No, I mean GWX and SH3Cmdr are usually seen as the cause for things like this.

Ah, never mind.

[Dowly]

Neeeaaal! It's getting worse! I've had 3 separate forum downtimes in 6hrs. :doh:

Could it be that the server just cant handle the traffic anymore?

[Task Force]

Yea, same...

[JScones]

Yeah, in strongest Scottish accent: "She's a gonna blow Captain!".

[Onkel Neal]

Haha, yeah, I have a guy doing things to it, crazy things, I'm afraid to ask. It's costing $$ so I hope to god he gets it figured out.

[Task Force]

Err, I could fix him... just give me the cordinants... :rotfl2: Missle on stand-bye... lol

[frau kaleun]

Haha, yeah, I have a guy doing things to it, crazy things, I'm afraid to ask. It's costing $$ so I hope to god he gets it figured out.

http://www.subsim.com/radioroom/picture.php?albumid=144&pictureid=1015

[Onkel Neal]

I'm about 2 reboots away from shutting this whole *@#^%$* thing down :shifty:

Anyone recommend a really good SQL expert I can hire?

[Dowly]

Ouch. So it's that bad? :-?

[Onkel Neal]

Just frustrated.

[HunterICX]

Just frustrated.

is this you?
http://www.youtube.com/watch?v=deCs3sBdhhM

HunterICX

[Oberon]

That's everyone who has ever owned a computer, that is... :yep:

[FIREWALL]

I just gotta ask Neal. Did your refrigerator magnets fall off? :rotfl2:

[Ships-R-Us]

I understand Neals frustrations and believe this is not a proper time for the humorous posts even though I am also guilty. I believe it is time for someone or a group of experts in the field to start a "private" thread so that they can quickly repair Subsim. I believe there are qualified experts on this site that can quickly resolve Neals issues before we see Neal on the national news.

[Onkel Neal]

No, this is me, (http://www.youtube.com/watch?v=8Yr-Pp4PFVA) only with more hair, not so fat, and lots more cussing.

[Dowly]

I have something for you to try Neal. Go Stalin's purge forge at the forum, lock all the unneeded threads (yes, I'm just trying to get the cat thread gone for good :D).

[Snestorm]

Can't get into the Nordics Thread.

[ETR3(SS)]

ditto

[DarkFish]

ditto

I get a lot of these database errors at subsim lately. Every few days subsim is down for a few hours, are there some problems with the server?

[Oberon]

Yeah, there's a few issues going on:

http://www.subsim.com/radioroom/showthread.php?t=159608

[Onkel Neal]

Can't get into the Nordics Thread.

A link?

[Torplexed]

A link?

http://www.subsim.com/radioroom/showthread.php?t=160646

or- Nordics this is why you can't have nice things.

[Onkel Neal]

Y'ok, I will have a look, thanks. I may need to take the forum offline for some repairs. Which forum is this in, btw?

[Torplexed]

Y'ok, I will have a look, thanks. I may need to take the forum offline for some repairs. Which forum is this in, btw?

General Topics

[Onkel Neal]

:doh: Right under my nose, sorry.

[Torplexed]

It's working now. :salute: Salutes damage control crew.

[GoldenRivet]

It's working now. :salute: Salutes damage control crew.
a doodle of some Das Boot type salty sailors with wrenches crawling all over a server tower comes to mind... the server tower in pieces of course.

[Reece]

This was the same error (Database error)trying to get into any of the Subsim forums and Subsim.com just came up with a Server error, only just got back on.:hmmm:

[GoldenRivet]

This was the same error (Database error)trying to get into any of the Subsim forums, only just got back on.:hmmm:

same here.

been inaccessible for a short while now.

[Onkel Neal]

Yep, been fixing errors.

[Carotio]

Ohhh... I see now it's not only me then....
I have had other sites being slow from time to time, so I thought maybe it was my connection till I saw this earlier this evening:
http://i7.photobucket.com/albums/y288/Carotio/Subsim/subsimdatabase.jpg


IIRC, you once said that your database was way over 3gb. Is that true?
Is there anyway in the forum software, vBulletine to reduce the size, or is external expertise the only option? Just curious aka concerned...

[HundertzehnGustav]

cause i get a Mysql database error?
just reporting...

mySQL error with query SELECT * FROM nucleus_blog WHERE bnumber=1: MySQL server has gone away<p />mySQL error with query select COUNT(*) as result FROM nucleus_skin_desc WHERE sdnumber=0: MySQL server has gone away<p />mySQL error with query select * FROM nucleus_blog WHERE bnumber=1: MySQL server has gone away<p />mySQL error with query SELECT * FROM nucleus_skin_desc WHERE sdnumber=0: MySQL server has gone away<p />mySQL error with query SELECT scontent FROM nucleus_skin WHERE sdesc=0 and stype='error': MySQL server has gone away<p />mySQL error with query SELECT * FROM nucleus_skin_desc WHERE sdnumber=1: MySQL server has gone away<p />mySQL error with query SELECT scontent FROM nucleus_skin WHERE sdesc=1 and stype='error': MySQL server has gone away<p />Skin error

[Task Force]

yea, subsim has been haveing its share of issues lately...:yep:

[Onkel Neal]

Yes, the forum db has grown huge and I am working on it. Thanks for your patience.:salute:

[Onkel Neal]

Ok, wow, I think (maybe) we found the problem the forum is running super fast for me after the reboot... cross fingers... :timeout:

[Task Force]

rgr... Lol, I think the SHV section is starting to overload it.:rotfl2:

*did you up the time between searches from 15 seconds to 60?

[longam]

I feel your pain dude, I have to update my stuff in a few weeks, and were bringing in VOIP phone system....read read read :timeout:

[frau kaleun]

Ok, wow, I think (maybe) we found the problem the forum is running super fast for me after the reboot... cross fingers... :timeout:

Fingers and toes crossed. Incense lit, chants uttered, Hermes the god of IT-by-extrapolation invoked. I'll even give my Mercury amulet a rub on your behalf. :D

[kranz]

I almost cut my veins last time:nope:

[Oberon]

So far so good... :yep:

[Onkel Neal]

Roger, and I have made a couple board settings changes. For now, the Currently active members online is disable, I'm told that uses a lot of resources on a big forum (not at all sure how, but let's try it).

[Jimbuna]

Heres hoping it's sorted Neal....seems to have been running fine for the last 6 or so hours :up:

[Reece]

Don't know what is going on but Subsim forums are going very slow (5 to 8k), can go to eBay and even tried watching stuff on YouTube, they all run fine, weird!!:hmmm:

[Onkel Neal]

How about now, Reece?

We were upgrading the forum to the latest version for the last 90 minutes or so, and making backups.

[Arclight]

That's what you get for hosting an exclusive SH5 interview and sending people e-mails to introduce themselfs. :lol:

And for having the friendliest place on the web, of course. ;)

[Ships-R-Us]

Responding perfectly fast here, looks and act normal............

[Castout]

Found it to be disrupted most often past 18.00 GMT+7 . . . .

Have you considered hackers?

[Onkel Neal]

Yes, we had a security audit done in Nov. Nothing was found. Everything is tip-top. I am working on getting an SQL expert to go through the db next week. Maybe archive some of the older threads, or something.

[Reece]

Things seem to be better at times then all of a sudden slow, will just have to wait till it's sorted out, also had to manually log in!!:hmmm:
Edit: Wow it's going slow now, will delete all my cookies/cache etc and see what happens!
Edit: Might be something wrong with my ISP, I rebooted my system and couldn't even log onto Google, after a minute back again, so at this stage I would ignore this post!!

[SteamWake]

Couple of things I noted this morning.

I had to manually log in.. though that may have just be a cookie toss.

All by "responed in" tick marks dissapeared and all topics have been tagged as read.

Hope you get it squared away I know how frustrating this can be.

There goes all that end of year fund drive :nope:

edit... hell now there all back ... still all tagged as read though.

[Onkel Neal]

Things seem to be better at times then all of a sudden slow, will just have to wait till it's sorted out, also had to manually log in!!:hmmm:
Edit: Wow it's going slow now, will delete all my cookies/cache etc and see what happens!
Edit: Might be something wrong with my ISP, I rebooted my system and couldn't even log onto Google, after a minute back again, so at this stage I would ignore this post!!

Ok, and thanks for letting me know, cuz sometimes people say "something's wrong" and they find out it's not, but no one ever tells me so I go on checking things.

Couple of things I noted this morning.

I had to manually log in.. though that may have just be a cookie toss.

All by "responed in" tick marks dissapeared and all topics have been tagged as read.

Hope you get it squared away I know how frustrating this can be.

There goes all that end of year fund drive :nope:

edit... hell now there all back ... still all tagged as read though.

Yeah, I updated the forums to the latest version last night, so that may have wonked the threads read, log in "remember me" type stuff. The good thing is:


No posts were lost
So far as I can tell, none of my dozens of tweaks were broken
Downloads did not break
Ranks and avatars survived
Entire db is backed up (well, it is nightly)
Running latest version of vB
No security problems.
Left to do:

Get an SQL expert to really check the db for problems.
Find a nice way to archive 2005-2008 threads and prune them from the db. That's proving harder than I thought, I would have expected vB to have a system in place for this. :stare:

[Reece]

Well a new day, started the computer this morning went to Subsim and all seems to be going fine today, didn't have to login, quite snappy!!:up:
Starting to get some problems on Subsim again:

ERROR

The requested URL could not be retrieved

While trying to retrieve the URL: http://www.subsim.com/radioroom/showthread.php?t=160740&page=2
The following error was encountered:


Connection to 174.133.66.250 Failed

The system returned:
(111) Connection refused The remote host or network may be down. Please try the request again.
Your cache administrator is webmaster.

Generated Tue, 26 Jan 2010 12:45:01 GMT by proxy2.ade.dft.com.au (squid/2.6.STABLE18)

[Snestorm]

Having a little problem in the SH3 section, only.
Nothing is being marked as unread.
No such problem in any other section.

Problem fixed.

[Onkel Neal]

Well, share the fix, man, share the fix :sign_yeah:

[Onkel Neal]

Well a new day, started the computer this morning went to Subsim and all seems to be going fine today, didn't have to login, quite snappy!!:up:
Starting to get some problems on Subsim again:

ERROR

The requested URL could not be retrieved

While trying to retrieve the URL: http://www.subsim.com/radioroom/showthread.php?t=160740&page=2
The following error was encountered:


Connection to 174.133.66.250 Failed
The system returned:
(111) Connection refused The remote host or network may be down. Please try the request again.
Your cache administrator is webmaster.

Generated Tue, 26 Jan 2010 12:45:01 GMT by proxy2.ade.dft.com.au (squid/2.6.STABLE18)


Hmm... I see a small batch of db errors around 4pm CST, is that about the time you experienced this?

[Reece]

Hmm... I see a small batch of db errors around 4pm CST, is that about the time you experienced this?Hard to say Neal due to the time differences but probably, today is going very fast, I would say all is fixed!!:yeah:

[Reece]

Hi, last night I couldn't login to Subsim.com at all for over 3 hours, and this morning I received notification for 4 PM's, they were the last 4 (old) spannng over a week, is there still some problem?:hmmm:

[Syxx_Killer]

I got e-mail notifications, too, from a thread I started back on the 27th. The replies I was notified of were from 3 and 4 days ago.

[Pioneer]

Likewise, I was unable to even bring the site up.

2.1.10 0134, 0434, 0546 and 0810 CST.

[Reece]

Dead again last night for a few hours!! This is becoming regular!!:hmmm:

[DarkFish]

Here too subsim was down for a few hours (around 9:00AM GMT).

[Carotio]

I have noticed it too on several occasions in the last 24h time, but I can't really recall exactly when.

Not trying to take advantage of this, but in case subsim goes off air again, feel free to use SHMF (http://www.silenthuntermods.com/forum/index.php) in the meantime. You know, to avoid the abstinences for the subsim fix to go too wild... :03:

[Reece]

I have noticed it too on several occasions in the last 24h time, but I can't really recall exactly when.

Not trying to take advantage of this, but in case subsim goes off air again, feel free to use SHMF (http://www.silenthuntermods.com/forum/index.php) in the meantime. You know, to avoid the abstinences for the subsim fix to go too wild... :03:Free advertising!!:eek: SPAM ALERT!!:D

[Ships-R-Us]

My guess is that Neal is trying to pick a time on a regular basis to shutdown for routine maintenance. I got a stack of old emails yesterday for old "pm's" also.

[Reece]

Yeh weird, last night no cut off, smooth sailing, hopefully that's the last we'll see of all these problems!:yep:

[Onkel Neal]

They're not back, they haven't left.

I've run out of ideas with my current server techs, they don't know what the problem is, so I am hiring a $175 hour tech. Hopefully he will know wth he is doing and we can put this behind us.

[Dowly]

It's game over, man! IT'S GAME OVER!! :O:

[Raptor1]

We're being sabotaged by the anti-Subsim conspiracy, that's what's happening...

[OneToughHerring]

You know every other time I click the board I either get "Server too busy" or "Database error". It's been like this for several days now. With patience I can post a message. I think I'll mosey along now and possibly return in the near future to check how things are going. But for now, laters ya'll. :shucks:

[Dowly]

You know every other time I click the board I either get "Server too busy" or "Database error". It's been like this for several days now. With patience I can post a message. I think I'll mosey along now and possibly return in the near future to check how things are going. But for now, laters ya'll. :shucks:

I believe that might be caused by something on your end. :hmmm: I've gotten "server busy" message once few months ago. Other than the "security token missing" message that comes every now and then, the forum is working fine for me.

EDIT: Oh nuggets. 5min after posting this, the forum went down for abit.

[frau kaleun]

same thing here.

also getting "database error" page but so far only after hitting 'submit reply' - when i go back to the thread i was replying to though my reply is there, i just don't get redirected back there from the reply page and end up in errorland.

also getting off-and-on 'page not found' and other similar when clicking on links to various threads or forums or usercp - got one the first two times i tried to get into this one to see if anybody else was having troubles.

[Sailor Steve]

^ ^ ^

As above.




[edit]:rotfl2::rotfl2::rotfl2:

It happened to me on this post!

[Oberon]

Being informed that Dowly and Hunter are patrolling the No Fly Zone does not inspire me with confidence! :har:

Aside from the database down for repair message Subsim has been behaving for me, perhaps because I'm using Neals favourite browser...

[Webster]

Just frustrated.


whiskey cures that you know, even if the problem doesnt get fixed you stop worrying about it :D

[Onkel Neal]

same thing here.

also getting "database error" page but so far only after hitting 'submit reply' - when i go back to the thread i was replying to though my reply is there, i just don't get redirected back there from the reply page and end up in errorland.

also getting off-and-on 'page not found' and other similar when clicking on links to various threads or forums or usercp - got one the first two times i tried to get into this one to see if anybody else was having troubles.


Mostly fixed now, I think I'm getting Database errors when posting (http://www.subsim.com/radioroom/showthread.php?t=161310)

There are two threads I had to pull the plug on and start sequels.

[Task Force]

was subsim just down...:hmmm:

[Onkel Neal]

Yes, for about 45 minutes.

Note to all: we are discussing options, including once again, upgrading to a new server. Bear with me, thanks for your patience.

[Task Force]

rgr, cool neal.:up:

[Pioneer]

yes, for about 45 minutes.

Note to all: We are discussing options, including once again, upgrading to a new server.

donate to subsim before it's gone.

[Onkel Neal]

Lol, no, it's not going anywhere, and I can afford this because of the fundraiser we had in Dec. Just wanted to keep you guys abreast of the situation.

It's not the $$ that bugs me, it's the time that it will take to get this done.

:) Neal

[Dowly]

Neal, if you ever run out of $$, here's an idea. Take your ordinary Battle Mug, draw a dot somewhere on it. Take picture of the dot and put it for sale to eBay with a title "Coffee cup with a jesus on it!!!".

And to save you from asking:

1. Yes, I am available to be hired as your financial advisor
2. Yes, 70% of the income sounds good.

[Torp III]

Here I thought Neal was just getting us to use to for SHV server down time!!!:har:

[Reece]

Found the thread, thought it vanished!! Well hopefully all will be fixed soon with minimal cost,:yep: although the tech's hour rate is like wow!!:o would be happy if my wages were even half that!!:doh:

[Onkel Neal]

Yeah, me too. Supply and demand, those guys are in short supply and high demand. Go to the library and get a Linux for Dummies book and in a few years I may be hiring you :)

[Onkel Neal]

Ok, update: ordered an additonal 2GB of RAM for the forum server, to make a total of 4GB. The websites server is still doing fine on 2GB.

[Jimbuna]

Ok, update: ordered an additonal 2GB of RAM for the forum server, to make a total of 4GB. The websites server is still doing fine on 2GB.

Fingers crossed for ya Neal....experienced another outage earlier this morning :up:

[Reece]

It's getting a little rough now Neal! It's up and down like a yo yo!:yep:

[Onkel Neal]

Fingers crossed for ya Neal....experienced another outage earlier this morning :up:

Yah, I know. Dowly woke me up at 04:30am with a server report. :haha:

[Jimbuna]

Yah, I know. Dowly woke me up at 04:30am with a server report. :haha:

Dowly!! http://imgcash6.imageshack.us/img231/1076/shockedvi8.gif

Better put on extra pairs of underwear and barricade your bedroom door http://imgcash5.imageshack.us/img134/3995/newshy222ul3cg8.gif

[Task Force]

Subsim was just down for awhile... its 3:55 here, just came back up.

[Onkel Neal]

Yes, we had to move some data to make sure we had a back up.

[Dowly]

Dowly!! http://imgcash6.imageshack.us/img231/1076/shockedvi8.gif

Better put on extra pairs of underwear and barricade your bedroom door http://imgcash5.imageshack.us/img134/3995/newshy222ul3cg8.gif

:haha::haha:

[Onkel Neal]

Ok, as the sign says, the forum server is now sporting 2x RAM, total 4GB.

We will watch the loads and response times closely. As always, thanks for your patience.:salute:

top - 19:18:57 up 41 min, 1 user, load average: 0.60, 0.38, 0.40
Tasks: 438 total, 31 running, 407 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 0.2%sy, 0.0%ni, 98.7%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4051676k total, 2952800k used, 1098876k free, 35036k buffers
Swap: 2040244k total, 0k used, 2040244k free, 938516k cached


The tightness around my chest, fading...breathing...returning to normal...vision clearing...

[razark]

The tightness around my chest, fading...breathing...returning to normal...vision clearing...

Just think of it as practice for the SHV release!

Thanks for keeping this place running, Neal.

[Gerald]

My server tech and I are trying to locate and repair crashed tables in the forum db. We had one 20 minute server disruption just now, I will try to keep the outage to a minimum. Sorry for the inconvenience.

Neal

And the issue could have been realeted to disk or in client network,nice at u got rid of it..

[Task Force]

working pretty good now Neal.:up:

[Onkel Neal]

working pretty good now Neal.:up:

04:24 and still rolling.... :yeah:

[Jimbuna]

04:24 and still rolling.... :yeah:


http://imgcash4.imageshack.us/img265/5513/starwarsdancinglo0.gif http://i224.photobucket.com/albums/dd320/pasquarade/funnywalks.gif

[piri_reis]

Thanks for the upgrade Neal and Team :salute:

[Reece]

I don't know if this is relevant but in the thread list it usually has a tick that identifies those I have posted in, they are now gone!:cry: Also noticed some threads such as the "I know curiosity killed the cat but ......2010" have missing posts!:hmmm:

[Onkel Neal]

? What? Where? I don't understand.

[Reece]

If I choose the General Topics section for example a list of threads is displayed, under the rating column there is normally a red tick displayed that references I have a post in that thread, well all the ticks are gone now and even after posting they do not show!:hmmm:
Regarding the missing posts re "I know curiosity killed the cat but ......2010" I don't know if you follow it but there used to be a picture of a man walking along a street with his hair tied back and a cat on the roof taking a swipe at it, I posted a remark on it, the remark was there but other previous posts were gone, so I deleted mine as it was referencing nothing!!:06:
Just found the proof, check post #51 the date is 02-03-2010, then post #52 the date is 02-06-2010, that's 3 missing days, this particular thread is almost posted a number of times a day!:hmmm:

[Onkel Neal]

If I choose the General Topics section for example a list of threads is displayed, under the rating column there is normally a red tick displayed that references I have a post in that thread, well all the ticks are gone now and even after posting they do not show!:hmmm:
Regarding the missing posts re "I know curiosity killed the cat but ......2010" I don't know if you follow it but there used to be a picture of a man walking along a street with his hair tied back and a cat on the roof taking a swipe at it, I posted a remark on it, the remark was there but other previous posts were gone, so I deleted mine as it was referencing nothing!!:06:
Just found the proof, check post #51 the date is 02-03-2010, then post #52 the date is 02-06-2010, that's 3 missing days, this particular thread is almost posted a number of times a day!:hmmm:

I still don't know what tick marks you mean, Reece. Can you post a screen?

Maybe there were no posts made in that thread between 2/3 and 2/6, we were down a part of the time. Anyway, that's not a highly critical thread, but let me know if you see any odd stuff going forward, 2/7 on. thanks

Neal

[Reece]

OK, I'm no artist but in the snapshot below I have drawn in ticks where they should be:
http://i66.photobucket.com/albums/h244/Reecehk/Screen1.jpg
You will notice that the post for the second tick down I have posted there today but there is no indication that I have done so, even the symbol on the far left doesn't show that I have posted there, it is very hard now to identify which ones I have posted in.:hmmm:

[Jimbuna]

Reece is correct Neal....when you go to a section like General Topics for examole and expand the pst history you would normally see a column of red coloured ticks to denote which threads you had posted in...I've brightened the colour a little to highlight what appears to be missing and no, I'm not an artist either LOL

http://img513.imageshack.us/img513/5962/34891694.jpg (http://img513.imageshack.us/i/34891694.jpg/)

[Rhodes]

Yes, I also getting this. The red symbol that a user had posted in a thread is missing.

[Jimbuna]

Yes, I also getting this. The red symbol that a user had posted in a thread is missing.

Tis sorted now...they are back up :up:

[Onkel Neal]

:salute:

[Reece]

Thanks a lot Neal, I love that feature!!:yeah: