vc-primary issues requiring a reinstall of software

Type

Incident

Description

While doing certificate maintenance on a vmware central server a couple of the underlying services had issues. Vendor got involved and part of the process was to reinstall components of the VMware stack.

Clusters were re-attached after things brought back up however in some cases HA (the product that restarts things after a failure) was more aggressive and assumed there was a failure. Some machines were then incorrectly restarted, but all were returned to service before 08:00 on April 09, 2015. The DEV MS SQL cluster had one member server be restarted, which caused issues if the DB was on that node.

During the remainder of the window the VMs remained available however no changes could be made to the environment. The exact same process worked on the backup server less than a week earlier.

Start time

Wednesday, April 8, 2015 10:00 AM

End time

Thursday, April 9, 2015 8:00 AM

Impact

Unable to administer VM's

Notice submitted

Thursday, April 9, 2015 10:53 AM