A big Power Blip at GNAX Datacenter

[Websync]

Thanks for the update Matt, but it looks like Vern is offline again. I am already getting calls, but no one seems to be upset yet. I am a bit worried though. My bad, but I have everything on that server, including my main website so no one can contact me except by phone.

Is there a chance that the files will be lost? Let us know how things are going when you have a chance. I know you have your hands full.

[Matt]

It isn't offline, but there is quite a bit of load on the machine. Many of the VPS are trying to recover the downtime as services came back online they are getting hammered with email and other requests to each VPS.

Imagine people at a nightclub, and there is a fire. And there is only one door to get out of the building. If everyone walks in an orderly fashion, everyone would get out and life goes on. Now imagine everyone tried to get out at the same time. Everyone is crammed up against the only exit. Nothing is moving efficiently.

There are going to be bottlenecks and only so many people can exit the building...

That is what is going on with web and email requests right now.

Sorry for the grim comparison, but that's the first thing that came to mind. The good thing is most of the fires are out, and most of the people have exited the building from what we can see.

Other VPS have not started back up yet, and Virtuozzo is working on those. its waiting for some load to go down before starting the others, from what I can see anyways.

We are going to force startup on some of these to expedite the process.

[Websync]

Sorry, I see that now. I am on one of the vps that hasn't started yet, so wasn't looking further down on the list. Got it. Thanks.

[Matt]

Alright after further investigation the ones that have not started have damaged quota systems, according to their internal OS's. The ones that have started are dealing with negotiation of all the requests they have and that problem is working itself out.

When we try to start the ones that are dead, they report:

Starting VE ...
vzquota : (warning) Incorrect quota shutdown for id 102, recalculating disk usage

What this means is after all the disk checks that have been run, now the virtualization software wants to make sure the remaining VPS that have not started have correct quotas to prevent something "bad" from happening.

There is no way to force a startup on these nodes that we know of, otherwise we would have done it and let quotas run in the background. Still looking for a way to hack this process.

In the meantime, they are calculating and coming back online as soon as they finish up.

Sorry about the delays. Wish we could speed things up. if we find a way, we will, but right now count on letting the systems catch up based on what the software requires.

[GlowAlex]

All VE's are started and online now.

If you are still facing any issues with your vps please open a trouble ticket or drop a note here.

[andreadoria]

There's always someone in worse **** :

This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

H1 Data Center Offline - Status Updates Available Here - The Planet Forums

At least you got to keep all the walls.

[andychev]

Originally Posted by andreadoria

There's always someone in worse **** :



H1 Data Center Offline - Status Updates Available Here - The Planet Forums

At least you got to keep all the walls.

Gnax really was just a blip compared to them, wouldnt like to be in their shoes trying to rebuild the place. Cant wait for the pictures of the incident to come out!