Omega Owners Forum
Chat Area => General Discussion Area => Topic started by: bob.dent on 09 May 2007, 13:57:39
-
Had an extra day off work after the Bank Holiday and arriving back at work this morning discovered the hard disc on our bloody file server crashed either Monday or yesterday. All accounts and admin data now corrupt. >:( >:( >:(
Fortunately, I have a good backup from last Friday for the accounts data but still means all work my staff did yesterday and today is lost. The tape backup for the server is also a week old due to a problem with the tape drive, which means that all last weeks admin work is also lost. Finally to add insult to injury, our IT manager is out today and even when he's back in tomorrow it'll probably take a couple of days to sort out.
AAAAAHHHHHHHGGGGGG!! Wish I'd stayed in bed this morning! :'( :'( :'(
-
Tomorrow is my first day back since last Weds. I hate to think what is waiting for me.
-
Really feel for your there mate. When did your IT manager know about the tape drive issue? Hopefully, you will be able to recover quite quickly.
-
I have dug myself out of holes like this before using forensic data recovery. It's not cheap but depending on what you've lost and what it's worth to you it may be an option. Also depends on the nature of the failure.
I do know what it's like trying to rebuild after something like this because it happened to me 6 months ago. I now backup to another server and to tape drives so I have both a secure backup and one that's very quick to get up and running again.
Tapes are all very well but if I go on holiday nobody bothers to check the backups so I need to fall back on something requiring no user intervention!
Well, I hope you manage to sort it all out soon.
Kevin
-
Tapes are all very well but if I go on holiday nobody bothers to check the backups so I need to fall back on something requiring no user intervention!
Well, I hope you manage to sort it all out soon.
Kevin
I know exactly what you mean. I am going on holiday soon (well away from work at least) and can't rely on anyone here to change the tape each day. If they do, it won't be dated and the jobs certainly won't be checked. Will be taking a laptop with a 3G card with me so I can check remotly.
There was a time here when one of my drives (a DLT) failed I was starting the back up at 19:00 and coming back each night at around 00:30 to feed it the second tape. I did this for about 4 months. Damn near killed me though!
-
Really feel for your there mate. When did your IT manager know about the tape drive issue? Hopefully, you will be able to recover quite quickly.
I think he knew about it day after it went t*ts up and he's waiting for a replacement unit to arrive. Don't think it's going to be a major problem now, just a couple of days re-inputting once it's up and running again and the backups are restored. :-/
-
I have dug myself out of holes like this before using forensic data recovery. It's not cheap but depending on what you've lost and what it's worth to you it may be an option. Also depends on the nature of the failure.
I do know what it's like trying to rebuild after something like this because it happened to me 6 months ago. I now backup to another server and to tape drives so I have both a secure backup and one that's very quick to get up and running again.
Tapes are all very well but if I go on holiday nobody bothers to check the backups so I need to fall back on something requiring no user intervention!
Well, I hope you manage to sort it all out soon.
Kevin
Thanks Kev, hopefully shouldn't be too much of a nightmare as once the server is up and running again we can restore the last good backup tape from a week ago plus I have a good backup of the accounts data from Friday evening. Trouble is, it's unlikely to be up and running again until Friday as I suspect a new hard disk will have to be ordered, so work is going to build up a fair bit. Could have been worse I suppose! :-/
-
Not using a fault tolerant disk setup :o
-
Not using a fault tolerant disk setup :o
erm..........a what? :-?
Sorry Jaime, I'm not that well up on IT technicalities, not sure what this is. :-/ My only concerns are when there's something wrong with the damn system. I must admit, I don't have the greatest faith in our IT department as issues always take far too long to get resolved.
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
It doesn't have to cost that much or consume much processing power, TBH. Depends how you implement it. In some cases performance will be boosted because hard drives are the biggest bottleneck in a server and if you've got multiple drives with the same data on, you can read from them in parallel.
Hard drives are the one thing in a server that have a very finite life so it does make sense to build-in some fault tolerance. In the simplest case, you install 2 drives and "mirror" them. Anything that gets written to disc goes to both drives. You effectively maintain 2 drives with exactly the same data on. When one dies, the system switches to using the remaining working drive. You replace the failed drive and the system then rebuilds the "mirror" by copying everything onto the new drive. The server can even be up and running, albeit with reduced performance, during this whole process.
You can get hardware controllers that do all this for you, so it doesn't take any resources from the CPU of the server itself, or you can do it all in software, which means it's much cheaper but does use some resources of the CPU itself.
I really would look into implementing some kind of fault tolerance. A single hard drive failure is too common an occurrence for it to take your server down.
Kevin
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
It doesn't have to cost that much or consume much processing power, TBH. Depends how you implement it. In some cases performance will be boosted because hard drives are the biggest bottleneck in a server and if you've got multiple drives with the same data on, you can read from them in parallel.
Hard drives are the one thing in a server that have a very finite life so it does make sense to build-in some fault tolerance. In the simplest case, you install 2 drives and "mirror" them. Anything that gets written to disc goes to both drives. You effectively maintain 2 drives with exactly the same data on. When one dies, the system switches to using the remaining working drive. You replace the failed drive and the system then rebuilds the "mirror" by copying everything onto the new drive. The server can even be up and running, albeit with reduced performance, during this whole process.
You can get hardware controllers that do all this for you, so it doesn't take any resources from the CPU of the server itself, or you can do it all in software, which means it's much cheaper but does use some resources of the CPU itself.
I really would look into implementing some kind of fault tolerance. A single hard drive failure is too common an occurrence for it to take your server down.
Kevin
Thanks for the explanation Kev, out IT guy gave me a similar explanation. I'm really not sure why we don't use such a system but we do have to keep our costs to a mimimum at the moment so could be the main reason. When the automatic tape backups are running properly I guess disk failure is not so much of a problem apart from the delay in having to fit and setup a new hard disk.
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
Sounds like they are too stupid to either consider it or implement it....
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
Sounds like they are too stupid to either consider it or implement it....
Certainly sounds like something we ought to have in place, but we are a very small company as is our IT dept which consists of..............just one! He's generally reasonably good with most IT issues but I do question his ability with higher level issues. Also, he's our Field Engineer Manager and half the time he's out of the office dealing with customers. Not an ideal situation, but I don't think our resources would stretch to a dedicated IT dept.
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
Sounds like they are too stupid to either consider it or implement it....
Certainly sounds like something we ought to have in place, but we are a very small company as is our IT dept which consists of..............just one! He's generally reasonably good with most IT issues but I do question his ability with higher level issues. Also, he's our Field Engineer Manager and half the time he's out of the office dealing with customers. Not an ideal situation, but I don't think our resources would stretch to a dedicated IT dept.
My brothers retail business have fault tolerant disks at both his sites (linked by a wireless link, apparently capable of up to 16km!).
Unfortunately, the disk controller failed in one today. Doh! Still, I talked him through nicking the controller from his Citrix server and fitting in the failed SQL Server to get his tills up and running again.
He has no IT staff.
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
Sounds like they are too stupid to either consider it or implement it....
Certainly sounds like something we ought to have in place, but we are a very small company as is our IT dept which consists of..............just one! He's generally reasonably good with most IT issues but I do question his ability with higher level issues. Also, he's our Field Engineer Manager and half the time he's out of the office dealing with customers. Not an ideal situation, but I don't think our resources would stretch to a dedicated IT dept.
My brothers retail business have fault tolerant disks at both his sites (linked by a wireless link, apparently capable of up to 16km!).
Unfortunately, the disk controller failed in one today. Doh! Still, I talked him through nicking the controller from his Citrix server and fitting in the failed SQL Server to get his tills up and running again.
He has no IT staff.
Ahhhhh.........the benefits of having an IT literate brother!! ;)
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
Sounds like they are too stupid to either consider it or implement it....
Certainly sounds like something we ought to have in place, but we are a very small company as is our IT dept which consists of..............just one! He's generally reasonably good with most IT issues but I do question his ability with higher level issues. Also, he's our Field Engineer Manager and half the time he's out of the office dealing with customers. Not an ideal situation, but I don't think our resources would stretch to a dedicated IT dept.
My brothers retail business have fault tolerant disks at both his sites (linked by a wireless link, apparently capable of up to 16km!).
Unfortunately, the disk controller failed in one today. Doh! Still, I talked him through nicking the controller from his Citrix server and fitting in the failed SQL Server to get his tills up and running again.
He has no IT staff.
Ahhhhh.........the benefits of having an IT literate brother!! ;)
Possibly, but my point was he is a small business, yet I can still come up with good enough business cases to get him to spend on things that have no immediate effect on bottom line - eg UPS's etc... ....even though i don't work for him, i still have to justify IT costs to him.
Surely your guy should do likewise?
-
**Update** Just asked out IT guy about it and he said we don't use this kind of set up due to the cost and the trade off in processing power.
Sounds like they are too stupid to either consider it or implement it....
Certainly sounds like something we ought to have in place, but we are a very small company as is our IT dept which consists of..............just one! He's generally reasonably good with most IT issues but I do question his ability with higher level issues. Also, he's our Field Engineer Manager and half the time he's out of the office dealing with customers. Not an ideal situation, but I don't think our resources would stretch to a dedicated IT dept.
My brothers retail business have fault tolerant disks at both his sites (linked by a wireless link, apparently capable of up to 16km!).
Unfortunately, the disk controller failed in one today. Doh! Still, I talked him through nicking the controller from his Citrix server and fitting in the failed SQL Server to get his tills up and running again.
He has no IT staff.
Ahhhhh.........the benefits of having an IT literate brother!! ;)
Possibly, but my point was he is a small business, yet I can still come up with good enough business cases to get him to spend on things that have no immediate effect on bottom line - eg UPS's etc... ....even though i don't work for him, i still have to justify IT costs to him.
Surely your guy should do likewise?
I really do take your point and I totally agree, but unfortunately it's not down to me to make these decisions. However, now I know what they are, I'll certainly push for having fault tolerant disks when we have our next managers meeting. I think I'll probably have a good argument seeing as how it affects me and my staff more than anyone else and without accounts and administration, there's no business.
-
When we take on a new customer, there is a certain minimum spec which we impose - if they won't have it then we walk away...
This may sound a bit harsh, but the reason is that without these failsafe features the system will never be stable, so it's loose-loose situation, the customer will not be happy with their IT and we will put in extra work for which we can not charge.
In case anyone is wondering, we make no money on hardware (margins are non-exitent), in fact we encourage the customer to source the kit themselves based on the Part Numbers we provide, though some insist that we actually supply it.
And yes this includes RAID as a must-have for all servers...