So I got up at 2AM, after going to bed at 10 to just lie awake for four hours. Dogs had already gone into kennels earlier that day. Got into car and drove to airport. First thing I knew about there being ‘trouble at mill’ was an email marked ‘UPS ALERT:ONBATT’. Panic. Okay, every now and then you do get a ‘power blip’ and the UPS cuts in, like it should and smooths things over using the battery until it passes, usually this is just a few seconds, Panic, still sets in. This was 6:25 on the one day I’m going to be away for over a week (don’t get ideas, the security is mega). Second email. ‘UPS ALERT:ONLINE’. This I at 8:04. Relief spreads over me like a heat rash. I’m reading this while at the luggage carousel. Now, all good. Until. 14.38. ‘UPS ALERT:ONBATT’. WTF. Twice in one day, what are the chances of that in the same day AND the day I’m away! And that’s the last I would hear.

Panic has now set in. I’ve lost email, both personal and company accounts, and web servers, again both personal (this one) and company accounts. I can’t ping the server. Can’t access the CCTV or the alarm (sorry folks it won’t talk to me, but is battery backed up and is loud as fuck, so again, no ideas). Panic again sets in. I can connect to the VPN. Good. Means the Zen ISP connection is good. I can log into the firewall. Check both routes, yep, 5G on three (that an anachronism?) is also up. Can’t ping the server. Look at the DHCP leases, all looks a bit dead. Panic is now very much set in. Server must have shutdown and not come back up. Panic is now in full swing. Contact my friend Elaine, she’s going to visit tomorrow and attempt a manual reboot.

Fast forward to today. Tried to connect to alarm panel, offline (again, it works fine when offline). Tried to connect to CCTV, that is back up, so the network switch is online. Checked the heating, yep, that’s good and the ring doorbell, so the WiFi is definitely up. Tried to ping the server and no luck. Connected to the firewall again and checked the DHCP leases, WiFi access point, switch, heating, doorbell, all good. Server, out to lunch. Elaine calls, she’s in the office (navigated the alarm and the lock). Now with a torch and a video call. I guide her towards the correct unit, and I have a lot in that cabinet. There is a UPS, a firewall, two network switches, the main server, backup drive and about sixty miles of cabling. After about ten minutes with a torch and pointing at things the server is identified and after more fumbling the power switch is located. It does indeed appear that it is powered down. A prod of the now found power button and the server is alive. Gave it a few minutes and then tried to refresh the email. The email server is now alive. ‘UPS ALERT: SHUTDOWN’, 15:53. ‘UPS ALERT: LOWBATT’, 15:53. So the power was out for over an hour, the UPS had run out of battery and sent the ‘shutdown’ command. There are a bunch of emails just after saying various services were shutting down, and that was it, it powered off, safely, as programmed to do so. I assume the firewall was also given the signal and dutifully shutdown. Until the power was restored. Depending when it happened the UPS would have done a couple of things. After it sent the shutdown command it knows it has to do a full shutdown and restart, to power things back up. So I assume at some point the power came back on. The UPS would have charged up to a point it was no longer in ‘low battery’ mode and would have pulled the power relays. This would have cut the power completely to whatever was connected. It would then enraged the relays again and ‘power up’ this should have caused everything connected to just ‘reboot’. Obviously the firewall obeyed and dutifully powered up, together with the ONT (the fibre interface) and the 5G modem (three, lets not go there). So connecting to the VPN and connecting to the firewall would all be okay. But, it seems the server did not power itself back on. This is probably just an over site in the bios settings. I expect it’s set to ‘last known config’, which of course would have been ‘off’ as it was told to shutdown by the UPS. It really needs to be set to ‘power on’. Or indeed, I should also configure it to ‘wake-up on lan’, I could then at least kick it from the firewall and reboot it. So there are things to simulate and test when I get back.

What are the chances of two power cuts on the same day and the same day I leave for a week? Low, very low. But it did happen. Thanks to my hero Elaine, I can ‘unpause’ my life again.