SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Mini Disaster - AC Failure


Mini Disaster - AC Failure

Author
Message
Andy Warren
Andy Warren
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: Moderators
Points: 12217 Visits: 2730
Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/awarren/mi

Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
Thomas Rushton
Thomas Rushton
Hall of Fame
Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)Hall of Fame (3.4K reputation)

Group: Moderators
Points: 3381 Visits: 3390

Been there, done that, got the T-shirt... at previous work-place, the aircon in the server room failed one evening. The first time our support guy noticed was the next morning, when she walked into the server room and thought "it's a bit warm in here".

Took us a long time to figure out all the extra damage afterwards -- a lot of machines were less stable afterwards, and it often turned out to be down to a component suffering some heat damage. Indeed, one riser card was found to be "welded" to the motherboard several months later (after we'd replaced just about everything else on that server...)

RAVEAC? Redundant Array of Very Expensive Air Conditioners?


Antares686
Antares686
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: Moderators
Points: 12094 Visits: 780
Fortunately our building does have a backup generator so heat has been a minimal issue in our personal server room. But we have found keeping the lights off (we have these neat little lights we keep for walking in and out without need to reall turn lights on) and not haveing the monitors on makes a huge difference in itself. Every so often we pruchase a KVM to consolidate a few machines to a single monitor. Also we have a two sided room with electricitiy coming in from two ways. Generally when one side is out (which in itself is rare) the other is always up so if needs be we swing the plugs.



David.Poole
David.Poole
SSCrazy Eights
SSCrazy Eights (8K reputation)SSCrazy Eights (8K reputation)SSCrazy Eights (8K reputation)SSCrazy Eights (8K reputation)SSCrazy Eights (8K reputation)SSCrazy Eights (8K reputation)SSCrazy Eights (8K reputation)SSCrazy Eights (8K reputation)

Group: General Forum Members
Points: 8039 Visits: 3291

I've had a couple of experiences like this.

I went into the server room to find a layer of smoke covering the top 6 inches of the server room, but no obvious source.

It turns out that a server with 3 power units had had one burn out, unfortunately there was no way of telling which one of the three it was.

The engineer said that if he pulled the wrong unit out then there was no way that the server could keep going on one.

Well he pulled the wrong one out, and three cheers for HP the server did keep runnning!

He put the working pack in pretty smartish and replaced the dodgy unit. Whilst he was there he decided to check the 3rd pack, just to be on the safe side. He pulled the pack out and there was a sudden flash and a lot of smoke and a 2nd dead power-supply.

Somehow, through all of this the server kep running.

In case you are wondering, the server room was being expanded and therefore the smoke alarms were off.

In another situation, power was lost over the Christmas holiday. One old SQL 6.5 machine had its disks seize once the temperature had dropped. Note to the wise. 4 hour callout does you damn all good if your box is old and obsolete.



LinkedIn Profile

Newbie on www.simple-talk.com
PaulR-122207
PaulR-122207
SSC Veteran
SSC Veteran (253 reputation)SSC Veteran (253 reputation)SSC Veteran (253 reputation)SSC Veteran (253 reputation)SSC Veteran (253 reputation)SSC Veteran (253 reputation)SSC Veteran (253 reputation)SSC Veteran (253 reputation)

Group: General Forum Members
Points: 253 Visits: 291

I shared this with my tech support manager, and he shared this comment:

Yeah, I could add that if you don't have a fan, you could stand at the server room door and use the door itself as a "fan" to move air around. Of course, only works well if your server room is the size of a closet!

Also, regular, quarterly inspection and maintenance of an AC unit goes a long way towards preventing failure of the unit.

BobAtDBS
BobAtDBS
SSC-Addicted
SSC-Addicted (450 reputation)SSC-Addicted (450 reputation)SSC-Addicted (450 reputation)SSC-Addicted (450 reputation)SSC-Addicted (450 reputation)SSC-Addicted (450 reputation)SSC-Addicted (450 reputation)SSC-Addicted (450 reputation)

Group: General Forum Members
Points: 450 Visits: 364

We actually have a small window style AC unit in the server room that is powered by an inverter and runs off the battery bank. The AC Unit was about $120 at Walmart, and the inverter is a 600 watt ExelTech XP600 that I bought on ebay for (memory here) around $500. We can run for around 4 hours on batteries, but like to switch over to a generator manually once power is out for more than a half hour or so. The window unit cools the five servers and the inverters just fine. In Central Florida, storms and power outages are a normal occurence, so we try to be prepared.




Student of SQL and Golf, Master of Neither
RSP
RSP
Old Hand
Old Hand (300 reputation)Old Hand (300 reputation)Old Hand (300 reputation)Old Hand (300 reputation)Old Hand (300 reputation)Old Hand (300 reputation)Old Hand (300 reputation)Old Hand (300 reputation)

Group: General Forum Members
Points: 300 Visits: 623

Hey! Anybody remember the Great Blackout of August 2003? And, I have that T-shirt. While we were sweatin our ac's off (pun intended), we were cranking diesel fuel into the backup generator of our data center. Fortunately, we just made it when it the power was bumped back on, and I lost a few pounds too boot!

I like the concept of RAVEAC!





tropic2
tropic2
Grasshopper
Grasshopper (12 reputation)Grasshopper (12 reputation)Grasshopper (12 reputation)Grasshopper (12 reputation)Grasshopper (12 reputation)Grasshopper (12 reputation)Grasshopper (12 reputation)Grasshopper (12 reputation)

Group: General Forum Members
Points: 12 Visits: 9

Happened to us a week or so ago. Power went out overnight, came on and everything was OK, but the next night the AC turned itself off in our primary server room. The next morning the outside of wooden door into the room was hot. Inside it was like a sauna - 5 racks all pumping out heat. The modems in the room melted. Only 1 server shut itself down - all the others were up. You couldn't touch the racks.

To get it cool we held the door wide open, used 4 big fans and turned off half the servers. Took 3 hours to get it bearable. We lost 2 disks on top of the modems and that was it fortunately.

The AC engineer said there was nothing wrong with the unit and it has been OK since. We've no choice but to go with the thing. Scary. Of course our LTO drive backup units were also in there. God knows what will fail over the next few weeks.

Guess we were lucky this time.





Andy Warren
Andy Warren
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: Moderators
Points: 12217 Visits: 2730

The point about turning out lights, monitors, etc, is one I missed and is worth noting. Im also pleased to find I'm not the only one that has bad things happen!



Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
Japie Botma
Japie Botma
Old Hand
Old Hand (312 reputation)Old Hand (312 reputation)Old Hand (312 reputation)Old Hand (312 reputation)Old Hand (312 reputation)Old Hand (312 reputation)Old Hand (312 reputation)Old Hand (312 reputation)

Group: General Forum Members
Points: 312 Visits: 307
We now have 2 AC's, temperature monitoring that send SMS's every 2 deg C after reaching 22 deg C (about 72 deg F), UPS and generator. But in the past we really had it bad on hot days (West coast, South Africa). What we have found is that an overheat of the server room causes hard drive failures for about six months. We now know that if it happens, we make sure that we have enough spare hard drives.

Japie

5ilverFox
Consulting DBA
South Africa
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search