Who forgot to feed the hamster?

Started by STEMO, 09 January 2017, 10:14:06

Previous topic - Next topic

0 Members and 0 Guests are viewing this topic.

Doctor Gollum

Need a new hamster wheel...

Thanks for the efforts getting it all sorted. It is genuinely appreciated  :-*
Onanists always think outside the box.

TheBoy

Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.
Grumpy old man

zirk

Quote from: TheBoy on 10 January 2017, 11:19:27
Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.

The Hyper Space Meltdown Unbalanced Stability thing sounds so much better than the TSSA Union and Eastern European Smiley Strike stuff that Mark came up with...  ;D

Either way, glad its sorted and much appreciated for your efforts again.  :y

STEMO

Quote from: TheBoy on 10 January 2017, 11:19:27
Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.
https://youtu.be/Ac7G7xOG2Ag

TheBoy

Quote from: STEMO on 10 January 2017, 13:26:02
Quote from: TheBoy on 10 January 2017, 11:19:27
Sincere apologies from TB Towers about the 2 seperate outages yesterday.  Both outages were caused by both hypervisors losing site of their storage in fairly quick succession. Root cause why is as yet unknown, but a few days ago the hypervisors were going a bit mental trying to even out storage usage and latency - maybe this caused all the disks to go into meltdown.

Recovery times were extended, as a full storage and hypervisor restart takes several hours.


Automatic storage balancing currently disabled to see if we can get stability.
https://youtu.be/Ac7G7xOG2Ag
Hypervisors = the clever hardware/software that allows multiple servers to run on a single physical server.  OOF is primarily made up of 5 (virtual) servers, not including all the infrastructure type stuff.  These 5 servers, along with several others, run across 2 hypervisors, which run on 2 HP Proliant physical servers.

Its called virtualisation/cloud, and its been all the rage for the past 5 years or so.

Downside is if a hypervisor starts to go wonky quite quickly, and doesn't have time to move its virtual machines to another, is a single failure can result in dozens of servers going offline at same time, which is what happened yesterday. Twice.  Although in the case yesterday, the 2 hypervisors that OOF is allowed to run on both died in realtively quick succession - so even if it did have time to move the virtual machines, it still would have died 20mins later anyway.


That better?
Grumpy old man

Andy B


aaronjb

Imagine it like passengers in a car.  The car is carrying the Finance Director, his deputy, the CEO and a sales girl.

If the sales girl dies they can all carry on.  If the FD dies, the deputy FD can take over.

If the car crashes, bursts into flames and kills them all, OOF goes down.  ;D

Doctor Gollum

Quote from: Andy B on 10 January 2017, 14:00:18
Quote from: TheBoy on 10 January 2017, 13:52:04....
That better?

Still no idea what you mean .......  ??? ??? ???  ;D
There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?
Onanists always think outside the box.

Andy B

Quote from: Doctor Gollum on 10 January 2017, 14:12:19
Quote from: Andy B on 10 January 2017, 14:00:18
Quote from: TheBoy on 10 January 2017, 13:52:04....
That better?

Still no idea what you mean .......  ??? ??? ???  ;D
There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?

;D ;D ;D ;D ;D much better!  :y

TheBoy

Quote from: Doctor Gollum on 10 January 2017, 14:12:19
Quote from: Andy B on 10 January 2017, 14:00:18
Quote from: TheBoy on 10 January 2017, 13:52:04....
That better?

Still no idea what you mean .......  ??? ??? ???  ;D
There are two hamsters on shift at any one time. Halfway through the night shift, the diabetic hamster forgot to take its insulin and fell over, and his only shift partner noticed ten minutes later, and in a moment of panic, fell down the stairs, landing on his head.

The alarm was only raised when the dayshift arrived and couldn't get in, and in the ensuing chaos, they realised that management help was needed, (an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...

That any better?
aaronjb's anology was closer ;D
Grumpy old man

Doctor Gollum

Given his profession, I should think so too :P

;D
Onanists always think outside the box.

TheBoy

Quote from: Doctor Gollum on 10 January 2017, 14:12:19
(an admin password was required to reset the system), but unfortunately the boss was in Chamonix, and was unable to get back until late afternoon...
Joking aside, just to put all your minds at rest, should such a scenario happen, the Admins know a man who has full access in remotely to sort out restarts etc.  This is one (there are serveral) reason we use HP servers, to benefit from the iLO Lights Out management (other vendors also have their own technologies).

Said man was texting me early yesterday to say it had gone TITSUP*, so he's defo on the ball.

But it took until mid morning to get it back up, then it failed again early afternoon :(


* A TheRegister term, for any of our technology readers ;)
Grumpy old man

TheBoy

Quote from: Doctor Gollum on 10 January 2017, 14:22:50
Given his profession, I should think so too :P

;D
I was cursing him last month, having to sit through a (very, very) condensed training course provided by his company...   ...and because it was so condensed, had to spend my own time going through the material (6 x A4 sized ring bound books, over an inch thick each).
Grumpy old man

Doctor Gollum

Is that somewhere between OOPS and FUBAR? ;D
Onanists always think outside the box.

aaronjb

I shall refrain from saying anything in case anyone from HQ is reading.. ;D