Blimey, WTF is happening lately
One of the hypervisors has sorta crashed, following a minor health alert this morning. What should happen is if the hardware health is suspect, all the load moves over to another, healthy hypervisor. In this case its just shat it pants. The VMs are still running, but unmanageable.
So, no choice but to power cycle the underlying hardware, which will ungracefully power cycle the VMs running on it.
From an OOF prespective, this includes the primary webserver, the primary database, and the secondary database (which affinity rules say should never be on same hypervisor as primary, so something else has gone wrong there).
I shall do this when I finish work today.