Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Doctor Gollum]

Need a new hamster wheel...

Thanks for the efforts getting it all sorted. It is genuinely appreciated 

[TheBoy]

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.

[zirk]

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.

The Hyper Space Meltdown Unbalanced Stability thing sounds so much better than the TSSA Union and Eastern European Smiley Strike stuff that Mark came up with... 

Either way, glad its sorted and much appreciated for your efforts again. 

[STEMO]

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.

https://youtu.be/Ac7G7xOG2Ag

[TheBoy]

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.

https://youtu.be/Ac7G7xOG2Ag

Hypervisors = the clever hardware/software that allows multiple servers to run on a single physical server.  OOF is primarily made up of 5 (virtual) servers, not including all the infrastructure type stuff.  These 5 servers, along with several others, run across 2 hypervisors, which run on 2 HP Proliant physical servers.

Its called virtualisation/cloud, and its been all the rage for the past 5 years or so.

Downside is if a hypervisor starts to go wonky quite quickly, and doesn't have time to move its virtual machines to another, is a single failure can result in dozens of servers going offline at same time, which is what happened yesterday. Twice.  Although in the case yesterday, the 2 hypervisors that OOF is allowed to run on both died in realtively quick succession - so even if it did have time to move the virtual machines, it still would have died 20mins later anyway.


That better?

[Andy B]

....
That better?

Still no idea what you mean .......   

[aaronjb]

Imagine it like passengers in a car.  The car is carrying the Finance Director, his deputy, the CEO and a sales girl.

If the sales girl dies they can all carry on.  If the FD dies, the deputy FD can take over.

If the car crashes, bursts into flames and kills them all, OOF goes down. 

[Doctor Gollum]

....
That better?

Still no idea what you mean .......   

There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?

[Andy B]

....
That better?

Still no idea what you mean .......   

There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?

  much better! 

[TheBoy]

....
That better?

Still no idea what you mean .......   

There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?

aaronjb's anology was closer

[Doctor Gollum]

Given his profession, I should think so too

 

[TheBoy]

(an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

Joking aside, just to put all your minds at rest, should such a scenario happen, the Admins know a man who has full access in remotely to sort out restarts etc.  This is one (there are serveral) reason we use HP servers, to benefit from the iLO Lights Out management (other vendors also have their own technologies).

Said man was texting me early yesterday to say it had gone TITSUP^*, so he's defo on the ball.

But it took until mid morning to get it back up, then it failed again early afternoon


^* A TheRegister term, for any of our technology readers

[TheBoy]

Given his profession, I should think so too

 

I was cursing him last month, having to sit through a (very, very) condensed training course provided by his company...   ...and because it was so condensed, had to spend my own time going through the material (6 x A4 sized ring bound books, over an inch thick each).

[Doctor Gollum]

Is that somewhere between OOPS and FUBAR?

[aaronjb]

I shall refrain from saying anything in case anyone from HQ is reading..