I am one of these people that really enjoys the weather of +30 degrees that we’ve been having recently, however today I’m not enjoying it so much. Though it’s not been a great day in terms of organisation on my part, anyway.
So first off, I got up late, by around 45 Minutes, although this didn’t make me late for work, I didn’t arrive when I would normally do so to check everything is running OK before anyone else arrives. When I did arrive, as I walked past the server room I noticed the system fans were unusually loud. I open the door and get hit by around 45 degrees of heat that had been stuffed in there over night, great, this is going to be a wonderful day. Yes we have AirCon, Yes it had died in the night, No I didn’t monitor the temperature (and have no way of monitoring the aircon). I did however, get some odd SMS messages about services flapping this morning, which at the time I just passed off as one of those oddities, that is, until I got to the office.
First things first, get as many of the servers shutdown as we can without stopping everyone from working (this actually turned out to be all of them other than the VOIP Server and switches). With fans all around the server room high up blowing air out of the room and low down pulling air in, windows and doors open around the office it started to cool a little. Unfortunately the aircon maintenance company are busy fixing other peoples aircon units, and don’t know if they’ll get out today or tomorrow, so I need to think what to do with all these servers, I can’t run them in this heat.
Well, There we go, the office boardroom to the rescue, with 2 aircon units in here I stacked the servers up – there’s old Ethernet cables between each server to allow air between them. This is our mail server, and 3 virtual servers, plus our monitoring server (the tower). 2 switches one for internal, one for external, and we’re running again. our file storage is still in the server room and the temperature is at about 31C now and still cooling.
First of all I’m glad I didn’t have to move all the servers, that would have been a huge pain. I am however, glad that we always keep a cold-swap switch in the office, this saved me a lot of time and annoyance. You’ll also notice an Arduino near the keyboard, This has a temperature sensor on it and is wired into the monitoring server and will SMS me if the temperature gets too high. There’s also one of these in the server room, now.
Lesson for today – Monitor your server room temperature, Seriousely!
According to the PDU graphs, the AMPS climbed rapidly from around 8:30PM last night, So I would guess this is when the fans in the systems decided it’s too hot so started turning up more and more.
Here’s a nice graph to show how the temperature has dropped since I put the temperature monitoring in the server room, note the start of this graph is 5 hours after we had been cooling the room with fans, doors & windows. The limiting factor for the server room is 1 door and no windows…