Still on the topic of automating maintenance tasks.. consider the following monthly security patch maintenance steps on 100+ SQL Servers with 500+ databases:
1. install monthly security patches on all mirror servers (automated tool)
2. monitor mirror servers for any security patch related issues... (semi manual)
3. reboot all mirror servers (automated/scripted)
4. confirm successfull reboot and QA mirror servers (semi automated...)
5. failover all principals to mirrors (automated/scripted)
6. monitor mirrors (new principals) for any security patch related issue (semi automated...but requires manual checks)
7. ensure applications have redirected to mirrors
8. patch and reboot principals
9. confirm successfull reboot and QA principal servers
10. failback all databases to principals
11. QA - replication, mirroring, log shipping, application connections etc, etc
Now, most of the individual steps are scripted/automated. The tough part is having a master script or control that coordinates it all - something that poll all servers and only continue the sequence when required.. Curious if anyone has reached full automation on the above...
In some cases servers require 2/3 reboots.
Sometimes servers are in pending reboot state so require additional reboot before patching.
Some servers can take 30+ minutes to reboot...
Also, there's the suppressing of monitoring alerts during the maintenance too for things like Replication errors, log shipping latency, mirroring alerts etc.