knock on wood

the unplanned downtimes we had were:

- during fire detection system tests, the backup power unit was shut down because of a little shot circuit by one of the electricians.

All non-critical servers were still working :w00t:

- SAN servers lost SAN connection during online activation of SAN-switch duplexing, which shoudn't be a problem according the the SAN manufacturer.

- One of the non-sysadmins thought he should reboot a server, wasn't able to do it using RDB, so he entered the server room and pulled the power cables ... out of the wrong server.

- extreme temperature exposure caused some servers to stop working.

(we had some very hot days last year) That server cabinet has been modified with an airco unit.

- data upgrade because of new application rollout destructed data because of some last minute changes by the dev team.

The restore operation took more than the planned downtime window.

- in one occasion a non dba added a non-sysadmin sqlaccount to the

sysadmin group of sqlserver, and applications got messages

"object does not exists.." ....

That was finally thé issue that got us the approval to restrict sysadmin membership.

- we also has a downtime caused by yours truly, because even tough

the logon trigger had been tested for some time,

not every actual situation had occurred and an instance got unresponsive.

DAC saved my but.

Yes, we did have some hard disk failures, and because of it were raid volumns, all the sysadmins had to do was replace the disk and monitor the rebuilt process. No downtime, only slow down.

Redundancy and protection are to be considered like an insurance and cannot protect you physically against what human inventiveness 😉 .