A few weeks ago I woke up with the alarming feeling that something was amiss, which of course is basically my default mode of waking up, as I tend to miss the alarm every now and then. (I get up at 5:30 alright!)
I looked over to the clock, which typically illuminates how many minutes I overslept in comfortably red glowing numbers. No lights. .. No comfortable red glow…
Over the last decade or so the power service to my house had been very dependable: it simply always worked. However, for some reason we have experience four power outages in the last six months, killing that informal service level agreement I had come to trust upon.
Unfortunately my servers had gone down hard, which resulted in RAID configurations needing to rebuild. No harm done, but still annoying. I decided to get a UPS so that in the future my servers would be allowed the time to gracefully shut down, so that I could prevent any issues that might result from the next power failure. My main concern was my Hyper-V cluster with two nodes and the Storage server that takes care of the iSCSI drives for the Hyper-V cluster. In addition I want my Web & Exchange server to shutdown nicely before the UPS kicks the bucket.
I had read Ben's blog where he goes into how he's bought a APC UPS and how he's using this with his Hyper-V setup. Following his lead I acquired the APC 1500 LCD. I like the fact that this thing actually gives me some feedback when things are going on, hence the LCD. (some more comfortable red glowing numbers… I sense a trend!)
Let's face it, when you're experiencing a power outage, the last thing you want to be bothered with is remembering what a series of beeps in a particular order means. Sure, you have read the manual many moons ago, but who touches the UPS once it's up & running? Not me! So typically by the time you need to know what's going on; you want the 'business critical' info presented to you in the most convenient manner. Although the LCD is a lot better than some beeps, I feel there's still room for improvement, it is all a bit cryptic.
Then it's time to install the APC software, after reviewing that feature- filled piece of software (not) for a full five minutes and going to the website to find the real software (don't bother) I found APC was not going to give me the possibility to control a number of servers. It simply supports one system, even though the back of the UPS will gladly allow me to connect six devices. If you actually want to shut them all down (why would you want to, right?) you're in bad luck.
To shutdown before the UPS gives up and simply fails your servers somewhat later than the original power failure would have done, you need to buy the more expensive business edition of the software… I felt cheated!
To the rescue!
One thing I took away from Ben's blog was that the UPS is presented as a laptop battery to the OS in Windows 2008. I figured that would mean I can check it using WMI…. Not sure how many UPS brands are supported this way, but I knew APC was pretty well supported by Microsoft. I was right: the Win32_Battery class gives me all I need!
I quickly removed the APC software from my server and decided to go my own way.
The result is a simple VBScript that basically does the following:
1. Tells you information about the UPS (or laptop battery if you like ;-)
2. Watches the remaining power left –and- power status of the UPS. You can configure a 'trigger' that indicates at what power level (say; 50% for example) you want to script to take action. A small configuration file allows you to name some servers and how many seconds you want to wait before shutting them down.
note: as mentioned in the comments below: please use the script by running it from the command prompt preceded with 'cscript'. So the complete line could be something like:
c:>cscript upswatcher /m:myserver /w
Actually, the script works much like any standard Windows command, using switches to tell it what you want it to do. For example:
/L will simply list UPS information
/M: Allows you to specify what machine to target (this machine needs to have a battery/UPS attached/installed. If not provided; will assume local machine. (better use "." For local machine)
/W Puts the scrip in "Watch mode" where it check the UPS power level and mode against a (in the script) predetermined value. Make sure you allow enough time for the UPS to shut the servers down! 50% may seem ample time, but if the load on your UPS is very high, 50% might only allow for 1 minute! (the LCD version of the ACP UPS will tell you this 'Time To Live') Note that the script will also evaluate the battery status; it will only reboot systems when the battery/UPS is being drained, not when it's charging (makes sense… right?)
/V Verbose; will make the script pretty talkative about what it is doing and why it's doing it. I call it the female mode. (Don't tell my wife…!)
/E Will send out an E-Mail for each server that is being shut down. You might want to consider scheduling your mail server as one of the last systems to go down. Mail server and E-Mail addresses are configured in the .vbs source.
To determine the machines to be shut down a small configuration file is used, it looks like this:
# Config file for UPSWatcher.vbs
# defines a machines name and the amount of seconds the device is allowed to survive on UPS power before shutdown.
# Paul Weterings, Nov 2009
# Put the servername you want to shutdown, and the time you want to wait before issuing the shutdown command
# make sure not to shutdown the local system ('.') before other systems have received a shutdown command (duh ;-)
# If you decide to send out mail, you migt also want to consider shutting down that system later...
# Finally, this scripts depends on being able to send WMI shutdown messages through the network: is you switch
# also on the UPS?
# Clusternodes first please
# The mail server, lets give it some time
# the storage (I could have used the name, but since it is the system running the script I can use '.')
Please note that it's completely independent where this script is running, as long as it can connect to the Windows 2008 machine connected to the UPS using WMI.
In addition, you do not need to shutdown the local machine, since that is quite optional. Notice in the example config file the local server (in this case the server connected to the UPS) is going down last… think about it…It would be rather dumb to shutdown the server sending the shutdown commands first. In addition; I know this is silly to mention, but make sure your switch is also on the UPS!
It would be quite entertaining to redevelop this into a powershell script, or even a small C# utility, if time permits. Feel free to do an attempt; it is open source as far as I'm concerned. I would appreciate an update if you do though.
UPSWatcher 1.7 can be found here
Update jan 2011: A slightly improved PowerShell version of this script can be found here.