Well, its the last day I have to work, before going on xmas holiday and it just started bad. After arriving at work the monitoring system was „red“ and my collegue and engineering on call told we that one of our Cisco WAE WAN accelerator applianced died last night around 3. Thanks good we running always clustered solutions so it was not that bad for our production traffic, but if somebody was to get out of the bed at this time its worst enough. So i need to spend parts of my day to collect debug crap for the Cisco case. That will be another interesting case, since the box came up again without errors after the power cycle.
Before that happened another incident yesterday evening happend: one of our Juniper firewall nodes also crashed and did not reboot anymore. Its just dead. Another case with the JTAC. Also this was a cluster so no impact to the production environment.
In these cases i like to work for a company who actually spend a lot of money for redundancy and HA concepts. So it sucks to get the call during night hours with a notifcation that something died, but ususally there is no impact to the production and you can take care next morning to swap the hardware.