Was the SVX Network Down?
 

Is it just me or was the network off line for 2 days? Are there missing posts or did I just imagine them?
Don't mess with my mind guys.:confused:

Gene

Re: Was the SVX Network Down?
 

I found the network down this morning. It goes down every once in a while but usually comes back quickly. I think that the techs do a great job with this network.

Re: Was the SVX Network Down?
 

Ya it was down. There's not much activity on here these days anyways.

Re: Was the SVX Network Down?
 

Thank goodness - I logged it with our IT Dept. at work, they said it was the site ?
I also tried by phone - yes it was down for 2 days....

Re: Was the SVX Network Down?
 

I was afraid my IT Dept. had finally found me out and blocked me . . .

Re: Was the SVX Network Down?
 

Like Larry said, this site has been so reliable, any glitch is a major event. I am just surprised there is nothing but crickets from the admins. I hope it wasn't anything serious. Maybe the NSA has hauled them away?:eek:
Gene

Re: Was the SVX Network Down?
 

The errors I saw indicated that it was a vBulletin database error. Those are somewhat common within vBulletin (I've seen them on multiple other boards.)

Threads, posts, responses, and pictures are added to the site database. Over time depending on how active a site is, the database needs to be re-organized to "fix" problems that may occur from long threads, many adds & deletes, etc. Usually it requires that an admin "clean up" the database to put things back in order. Depending on the size of the database, it can take from a few hours to a few days.

I congratulate the admin / database admin on their quick action to resolve the problem. THANK YOU!

Re: Was the SVX Network Down?
 

Chis has monitoring software that notifies him when or if the site goes down. The site ran out of memory and it did not notify him it wasn't working. :(

Re: Was the SVX Network Down?
 

My apologies for the artificially induced heart attacks.
Here is the full explanation in brief:

Many many hours have been spent trying to migrate(move) the SVXWN data and software to a new machine. The blade server has been GREAT for the last (uhh I forget) 7+ years. Technically there is no reason we need to move minus a small complicated issue with disk space (read below for a brief on that).
We purchased new version of the software cause we figured, time to move, time to upgrade. The current version is a complete re-write by a new company that merged or bought out the original vbulletin people. The data on the site includes 12+ years of posts, comments, icons, images, and attachments. I realize this is not a huge site, but this amount of data can really dog down even the fastest of servers when not only moving data, but manipulating each entry and table from the current usage profile to the new way the forums software wants to work with it. To copy the data takes a couple hours. Last time we tried to roll the updates too over 14 hours!!

We have not had much success with migrations and have tried off-line methods the last few attempts to keep the site up. When we are successful with the upgrade in a test environment, we will have to do it again with the network down so we dont lose posts or any changes on the site.

The site went down without me knowing because we have backup data stored on the live machine that we use on the test machine each time we attempt the upgrade process. As you might imagine, this takes up quite a bit more space. What happened was with more new posts and images, the space available on the hard drive was exhausted and the database server became very unhappy with its ability to write new data. The database then faulted with an error. My monitoring of the machines failed to detect the outage because the most common issue with servers typically is a locked up machine due to failed hardware (bad memory, failed power supplies, failed drives, etc..) failed network (saturated due to hackers, or denial of server attacks), or weird issues with services such as Apache (web server) stopped running or MySQL (database server) service crash. I actually monitor all of the above situations by checking process list for required running tasks, apache output (with an OK result), and ping to make sure the machine is responding and the network is good. When the failure happened, it was outside my scope of checks. The database server was running, however unhappy, apache was running, the machine was up and responding to healthy ICMP ping traffic.

I was eventually notified by a rash of text messages by some of the members that have my cellular number and a plethora of Facebook messages from people that found my account there. I was dumbfounded by the rash of sudden friend requests and a few notes about the SVX World Network.

I would like to THANK all the members with the patience on the outage as well as the people the courteously inquired personally with me on my phone or Facebook. I invite all/any of you to do so again if you suspect and outage in a period of un-announced work and or no "work in progress page" appears. I try to keep you all informed when I do work that will cause an outage and I do my best to keep the systems up and maintained.

As some of you know, I do work two jobs 70+ hours a week and that has some impact on my time I get to enjoy the site let alone sweep the dust from time to time. My SVX race car has also been sidelined for the last two years due to parts breakage and it looks like I wont be able to return to racing until 2015.

There will be a future outage when we get the migration bugs worked out and we hope to be done soon. I will post more when the test environment does not totally break stuff.

___________________________________________

Hard drive space and blade server:

The issue with space on the ultra-reliable blade server is technological changes. Many tried and true standards for disk technology have changed since the days that our blade server was manufactured. Most servers use an 80 pin SCA hot swap centronics like plug. Out blade server has hot swap blades, but the drives inside are a more conventional 68 pin workstation ultra SCSI interface. Many newer technologies have come out since such as SATA 1.5, 3.0, 6.0, SAS 3.0, and 6.0. The most common was the SCA, but today its wither SATA or SAS. The problem now is that hard drive makers no longer support our old requirement and the only drives available are used system pulls, or new on the shelf stock that has quadrupled in cost due to demand from people that still need the old format. With 12+ years of data, we need more space and I don't want to do any sort of fund raiser to pay 4 times the cost for old technology. This is the reason we are trying to move the data to a new server.

Watch for updates. I wish everyone a Merry Christmas, and or Happy Holidays, and may everyone continue to fully enjoy their SVX experience.

Re: Was the SVX Network Down?
 

I think I can speak for many in saying that this is one of the most reliable and stable sites out there, thus the shock when it didn't work. We truly appreciate your hard work and efforts to make it so. My mind meld, for one, shows you've spoiled us. I will toast you every time I log on. :)

Gene

Re: Was the SVX Network Down?
 

Much agreed. This site has been down just twice now that I can recall in 5 years. The other time it ran out of memory and was back up within just a couple hours. Considering that this forum relies on 0% backing from advertising of any sort, and that no one HAS to pay anything to be a member, post pictures, post for sale threads, etc. really is a testament to the great job they have done keeping this running for this long, especially with waning membership and owners.