The Server Room Reached 40°C
NT to Now
101,567 views
At 8:15 on a Monday morning, my mobile rang. A customer had opened their server-room door to find the room at more than 40°C, water beneath the failed air-conditioning unit and warning lights appearing across the racks.
Four servers were already offline, others were still running at critical temperatures, and the leased line had failed—leaving us with no way to reach the equipment remotely. The customer manufactured specialist medical devices, sometimes against a 24-hour dispatch deadline, and I was nearly two hours away.
In this IT war story, I explain how we responded, what extreme heat did to the HP ProLiant servers and their Ultra320 SCSI disks, how VMware ESXi helped protect Exchange 2007 and SQL Server, and what we changed to ensure it could never happen unnoticed again.
This is the story of the day the server room reached 40°C.
Attributions
Button Press by Blankened -- https://freesound.org/s/846144/ -- License: Creative Commons 0
20 Comments
i was very, very wrong.
When he was leaving, he heard a leafblower sound from the server room. And investigated.
The server room got so hot he could smell the wires getting hot! He manually had to use a terminal and shut the servers down one-by-one and saved them. Insane
Was at a customer, when the city powergrid was shut down due to maintenance.
Customer had UPS, but the aircon wasn't on the UPS, so temperature skyrocketed, we tried to shut down as many servers as possible, I seem to recall they had like 10-15 racks full, so was futile...
A couple of days later, the customer decided to have some of the cooling done, this time prepared, or so they thought, turned out the SAN in another server room wasn't on UPS, and shut down, taking the entire ESXi platform down.
Not a great week!
Amazingly, the only damaged hardware was a stick of ram in the Exchange server. Or so I thought. 2 months later I was updating the firmware on the EqualLogic array in prep for moving data from an older array on to it. As I began the data copy a disk got a failure prediction and the array flipped to the hot spare. No problem, the array had a 4 hour support response contract. Popped in the new disk and I went home.
Then overnight 3 more disks got failure predictions and went offline. Oops. Support used some voodoo to disable failure prediction, mark the disks as OK, and bring the array back online. Turns out one of the things the firmware update gave was improved failure prediction.
Then I kept the door open to let the office AC cool the room.
Main power went out for half a day. Backup power kept the servers going but the AC wasn't connected to the generators so all the equipment cooked itself.
We didn't get so lucky, it took 3 days to bring most of the company back online. Several machines died that day.
There were all kinds of issues like dead RAID arrays because the spinning disks didn't survive not spinning after years of use, dead mainboards because the caps didn't survive the heat, machines that weren't booted for years that had non-working boot configs.
Nowadays you could probably run all that stuff on less than a rack full of machines.
Fun times thinking about the old stuff. When I started half the racks in the IT rooms were patch panels for phone lines.
Can't tell such a story myself, I never had a failed AC. What I can remember: contractor was told to drill some holes trough a double floor down to a lower level - they hit a bunch of network cables that then twisted around the drill, severing the lines.
That was a lot of fun to fix!!
The Server Room Reached 40°C
NT to Now
101,567 views
At 8:15 on a Monday morning, my mobile rang. A customer had opened their server-room door to find the room at more than 40°C, water beneath the failed air-conditioning unit and warning lights appearing across the racks.
Four servers were already offline, others were still running at critical temperatures, and the leased line had failed—leaving us with no way to reach the equipment remotely. The customer manufactured specialist medical devices, sometimes against a 24-hour dispatch deadline, and I was nearly two hours away.
In this IT war story, I explain how we responded, what extreme heat did to the HP ProLiant servers and their Ultra320 SCSI disks, how VMware ESXi helped protect Exchange 2007 and SQL Server, and what we changed to ensure it could never happen unnoticed again.
This is the story of the day the server room reached 40°C.
Attributions
Button Press by Blankened -- https://freesound.org/s/846144/ -- License: Creative Commons 0