Omega Owners Forum
Chat Area => General Discussion Area => Topic started by: Mr Skrunts on 03 December 2017, 18:55:12
-
Having Zillions of files scattered over numerous hard drives I decided a few weeks ago to have a massive sort out. :-\
The plan was to collate and sort in to folders/directories and then sift and sort duplicates out and maybe rename to contents (of some of the folders at least) for ease of indexing. ::)
In total I now have :- ::)
Numerous 1tb drives
3+ 2tb drives
2+ 3tb drives
I added 2 x 4tb SSHD Seagate's one of which is already fubar and I don't have a clue how much data is lost or whether it's backup exists or not.
Replaced the 4tb SSHD with a Toshiba drive and am quite impressed so went and bought a 6tb Toshiba with 128mb cache runs at 7200rpm, hadn't focused on speed of other drives as they are only for back.
The plan is to run a RAID setup up (Still reading up on what's what) but the question is, if all the drives are of equal size do they have to be the same make. 8)
I am also surmising that if I use RAID 5 that all disks need to be identical. (Considering using 3 6tb Toshiba drives)
If for any reason I took one of the raid storage drives out will the data be readable in another PC? ???
TIA :y
-
I wouldn't bother.
Consumer SATA drives are absolutely shite at RAID, including Toshiba (whose drives have dubious reliability, and they are going down the shitter anyway), and the RAID implementations on consumer system boards are poor (basically variations of software RAID that rarely turns out well)
You won't get a consumer motherboard supporting RAID5. Fortunately.
Lastly, RAID is not a backup system. Never has been, never will be. Its an availability system.
-
See, I'd disagree with TB (when does that ever happen? ;) ;D )
I have two (rather expensive) QNAP raid housings here filled with drives - one is filled with standard consumer drives and the other with Western Digital "Red" (RAID) drives - both perform damn near identically and have pretty much the same MTBF.
That said, if you want RAID-on-the-cheap, look at UnRAID; several friends have been using that for years (since I started, before I graduated to more expensive solutions) and it has a number of advantages:
The drives don't need to be the same size, the only stipulation is that the parity drive(s) are the same size or larger than the largest drive in the array.
It runs on consumer hardware, you just need a reasonable motherboard (nothing too fancy) and a bunch of PCIe SATA cards (and a NIC, obviously)
It's very cheap
The software is now pretty mature and supports things like Docker so you can run containers with software like Plex Media Server or even ESXi and virtualise entire hosts on top of it (depending on how meaty your server is)
You clearly already have the hardware, assuming you can dedicate a machine to it, so it's a no-brainer to me. Run with dual parity and you can support the loss of two drives before losing any data - yes, it's not a backup solution, but it does significantly reduce the chances of data loss unless you ignore a failed drive.. (and despite supporting the loss of two I would always replace a failed drive ASAP as the chances of another drive failing get significantly higher when the whole array is churning away for 48hrs solid rebuilding the parity drives)
-
I am over run with hardware at the moment.
I have an 8 core FX8350 16GB system. :D
I also ended up with 2 x 990X gaming SLi boards (this was bought due to an issue with the above system. :o
Then I ended up with a Corsair 240 System which I am using for scanning photos, bills and invoices. :-\
Then after all that I ended up buying an i7 4950K system with an Asus Maximus VII Ranger board. :P
Just looked through the specs and all but the Micro system have 6 x Sata connections and support SATA RAID 0, RAID 1, RAID5, RAID 10 and JBOD
So lots of fun and learning to be had.
Then when I am done going to sell all the slower PC's and build a new one in the new year, Just cant decide on Coffee Laje, Socket AM4 or TR4, Then I saw the Socket 2011 Boards and Dual 2011 Boards. Choices choices lol. :-X
Yes I fully understand that none of this is work station or server standard, but all I want to so is improve the speed, stand and quality of my hardware.
-
Just been looking at Qnap gear, would/could be a solution for me in the future. :y
-
When it comes to speed:
RAID0 - (two disks) slightly faster read & write than non-RAID. Lose one disk, lose ALL your data.
RAID1 - (two disks) faster read, no effect on write speed. Lose one disk, lose nothing.
RAID5 - multiple disks, striped data & parity. Read & write speed may be faster, tolerant of one drive failure.
RAID6 - multiple disks, striped data & parity. Read & write speed may be faster, tolerant of two drive failures.
UnRAID - multiple disks, data is contained entirely on each drive, parity is stored on one or two dedicated drives. Tolerant of one or two failures.
The big difference is that for 'real' RAID (0 through 6) all drives must be an identical capacity. UnRAID will use the full capacity of all drives, even of differing size, as long as the parity drive is largest, because it does not stripe the data across the disk set.
BTW, Synology also make nice NAS boxes, though I have no personal experience :)
-
What on earth do people do with all this kit ? Getting ready to take over the World ? ::)
-
What on earth do people do with all this kit ? Getting ready to take over the World ? ::)
Trying to keep up with TB and the OOF server room. ::) :y
-
What on earth do people do with all this kit ? Getting ready to take over the World ? ::)
My thoughts as well, plus others, like what an earth is all that for and what does it mean?
21st century boys toys I suppose, but as a girl what do I know!! ::) ::) ;D ;D ;D ;)
-
What on earth do people do with all this kit ? Getting ready to take over the World ? ::)
the modern equivalent of a Playboy under the matress?
-
In my PC I have a fast boot disk (500Gb) that has OS and programs installed
Then I have 2x 1Tb disks running raid1 for storage.
The system board has a raid controller built in and a utility manages the raid1. Its suppose to tell me if the raid fails.
I know the disks are 1G but I know they aren't the same make.
Also I believed you can use different size disks but you loose the capacity of the higher disk. So for example, if you were to use a 1Tb disk and a 2Tb in a raid1 configuration you would loose 1Tb of the 2Tb disk, so you just had 1Tb storage.....
I also realise it isn't a backup solution....but how do you back up 1Tb of data, apart from to another pc on the network with a large disk in it.....which I don't have....
-
My backup up is done via external drives, My oldest one is 2tb Samsung Story drive that has an On/Off knob on the front plus a blue LED under it to show activity. The other 2 are 3tb + 4tb externals both are Seagate but sadly the 3tb is fubar already, hopefully can strip it and replace the drive.
I am also aware that the 3tb ext and 4tb SSHD both from Seagate are under warranty but sadly dead money as I cant trust any body with any banking details or personal data they mind find during resurrecting the drive. :-\
-
What on earth do people do with all this kit ? Getting ready to take over the World ? ::)
My thoughts as well, plus others, like what an earth is all that for and what does it mean?
21st century boys toys I suppose, but as a girl what do I know!! ::) ::) ;D ;D ;D ;)
Lizzie, as you ought to know
Bigger and faster is generally the way to go. ::) :y
-
What on earth do people do with all this kit ? Getting ready to take over the World ? ::)
the modern equivalent of a Playboy under the matress?
;D Surely not . . . . ::) Everyone here is pure of thought and clean living
Must be a lot of "playboys" and very high resolution ::)
-
What on earth do people do with all this kit ? Getting ready to take over the World ? ::)
My thoughts as well, plus others, like what an earth is all that for and what does it mean?
I
21st century boys toys I suppose, but as a girl what do I know!! ::) ::) ;D ;D ;D ;)
Lizzie, as you ought to know
Bigger and faster is generally the way to go. ::) :y
;D ;D :y
-
When it comes to speed:
RAID0 - (two disks) slightly faster read & write than non-RAID. Lose one disk, lose ALL your data.
RAID1 - (two disks) faster read, no effect on write speed. Lose one disk, lose nothing.
RAID5 - multiple disks, striped data & parity. Read & write speed may be faster, tolerant of one drive failure.
RAID6 - multiple disks, striped data & parity. Read & write speed may be faster, tolerant of two drive failures.
UnRAID - multiple disks, data is contained entirely on each drive, parity is stored on one or two dedicated drives. Tolerant of one or two failures.
The big difference is that for 'real' RAID (0 through 6) all drives must be an identical capacity. UnRAID will use the full capacity of all drives, even of differing size, as long as the parity drive is largest, because it does not stripe the data across the disk set.
BTW, Synology also make nice NAS boxes, though I have no personal experience :)
Conkers, I'm afraid. Parity stripped RAIDs have abysmal write performance. Not just slow, abysmal. Even with a top end SAS controller (well, once the cache fills up). It has to do at least 3 reads, wait for data, compute it, then at least 3 writes. Do not underestimate how massively badly this impacts performance - with good caching (not available on any consumer PC boards, or most cheapo NAS devices), you can get almost usable results if its purely sequential, but very little is... ...and none is with shared storage.
On most PC boards, even R1 is likely to be slower at both read and write than non-raid, as the compute is done in the driver. Toy level NAS devices suffer similar. As does all the shit like FreeNAS etc (made worse by its near exclusive use of ZFS, and its not a great port, and PC hardware is shite, and the ZIL becomes the bottleneck). Toy NAS only works with one client - give to 2, performance dies, and you often get timeouts and dataloss, hence can't be recommended for data integrity. *NOTHING* is faster than local DAS, assuming same drive technology when comparing. Also DAS is more reliable, thus the best integrity.
As to SAS v SATA, they are leagues apart. NCQ helps with SATA (assuming drive, controller and driver (if software RAID, most PC shit will fall into this)), but doesn't even bring into same league as SAS.
I could prove this, as the 2 physical servers currently hosting the OOF VMs are currently on RAID10 on 2.5" SAS drives. The same physical servers also have a RAID10 on 2.5" SATA drives (granted, only 7200rpm spindle drives), used a cheap archival storage (2Tb SAS 10k rpm drives are £600+ each, compared to about £70 for SATA). If I put the OOF primary database on the SATA array, I'd expect it to do the usual Linux thing when its resources can't keep up - start thread blocking, before eventually shitting its pants with a panic. However, straight away, you'd see all the database queries to build a page taking a few seconds, rather than the .2s it takes now if not cached.
SAS SSD takes it to another level again, but I ain't rich enough for them ;D. But if anybody wants to donate some, preferably HPE in the HPE hotplug carriers (gen8 or later type), >800Gb, my address is:
TheBoy
Brackley
;)
-
But if anybody wants to donate some, preferably HPE in the HPE hotplug carriers (gen8 or later type), >800Gb, my address is:
So if money was no object what would the OOF server system consist of. (Purely out of interest) :y
-
The thing is, who gives a damn about performance in a typical domestic setup?
If you don't have the budget for enterprise level storage stuff, the best thing you can do is chuck a load of cheap SATA drives in a NAS or a small Linux box and run software RAID, and keep it backed up.
Any hardware RAID controller costing less than four figures will be shite and, whilst software RAID isn't the be all and end all, when it goes wrong there's a decent chance of recovering it rather than realising it's been gradually corrupting both your live filesystems and your backups for months. Yes, got that tea shirt, thanks, Dell. ::)
-
The thing is, who gives a damn about performance in a typical domestic setup?
If you don't have the budget for enterprise level storage stuff, the best thing you can do is chuck a load of cheap SATA drives in a NAS or a small Linux box and run software RAID, and keep it backed up.
Any hardware RAID controller costing less than four figures will be shite and, whilst software RAID isn't the be all and end all, when it goes wrong there's a decent chance of recovering it rather than realising it's been gradually corrupting both your live filesystems and your backups for months. Yes, got that tea shirt, thanks, Dell. ::)
My point being that consumer systemboard RAID implementations are semi software, but without the tools to diagnose when things go awry.
Unix and Windows Server software RAIDs are reasonably robust, if somewhat slow, and disk replacement can sometimes be a pain.
In a home environment, I'd recommend keeping it nice and simple, and use far simpler storage - 24/7 availability is not a requirement, all you want is data integrity, and a simple disk system and robust backup regime is the best solution to that IMHO.
If shared storage is needed, I'd be more inclined to share it from a PC using normal Windows/Linux tools than use the cheapest of the consumer NAS devices. If a NAS is needed, look at the higher end of the consumer range if possible... ...but a PC running Windows/Linux (depending if your clients are Windows or Unix (as Samba introduces a whole new set of issues to keep on top of)) will likely outperform it.
-
So if money was no object what would the OOF server system consist of. (Purely out of interest) :y
Pretty much what it does now, but more RAM (only a measly 128Gb in each physical), and more storage with more SSD (currently around 28Tb raw spinning media, 4Tb raw SSD), and a bigger backup device (currently LTO5). More 10G networking would be a bonus.
The rest is how I would like it - branded top tier enterprise class server hardware, with branded, top tier disk controllers, All of which should allow 24/7 reliability to the VMs, even if the VMs themselves need to occasionally be rebooted for patching - for OOF, I don't see the gain in making that HA for the 2 mins a fortnight it needs to reboot.
Most of the outages in the last year have been due to shared storage, where the external storage devices has been unable to cope, or the crap Netgear switches used here (Netgear G724 - defo one to avoid at all costs!) have gone non responsive, in both cases, the result is VM storage corruption. This is now hopefully resolved by using hyperconverged storage, so the storage is always physically connected to the local hypervisor running that VM.
-
In a home environment, I'd recommend keeping it nice and simple, and use far simpler storage - 24/7 availability is not a requirement, all you want is data integrity, and a simple disk system and robust backup regime is the best solution to that IMHO.
Now if you could let me know how I can easily (and with similar expense) back up 32TB (yes, 32TB filled, not capacity) of stuff, that'd be great ;)
The only way I can imagine a robust backup solution for that would be several sets of drives at £1k a pop, cycling them continuously and backing them up pretty much continuously.
Or, y'know, I could just keep my RAID6 array and not worry about it too much ;) (I mean, unless I am very unlucky and lose three drives at once - which could happen..)
Don't forget, TB, there's a bit difference between a corporate environment and Skrunts not wanting to lose his porn collection ;D
-
If shared storage is needed, I'd be more inclined to share it from a PC using normal Windows/Linux tools than use the cheapest of the consumer NAS devices. If a NAS is needed, look at the higher end of the consumer range if possible... ...but a PC running Windows/Linux (depending if your clients are Windows or Unix (as Samba introduces a whole new set of issues to keep on top of)) will likely outperform it.
The attraction of NAS devices is that they can be "always-on" without consuming a stupid amount of leccy, unlike any usable PC. If all your media, documents, etc. are on a PC it's a real ballache to have to remember to turn it on before you decide to listen to some music in bed using your tablet, etc..
Now if you could let me know how I can easily (and with similar expense) back up 32TB (yes, 32TB filled, not capacity) of stuff, that'd be great ;)
aaronjb has too much porn! :o
-
aaronjb has too much porn! :o
Nah, only half of it is p.. I mean, none of it is porn ;D
-
aaronjb has too much porn! :o
Nah, only half of it is p.. I mean, none of it is porn ;D
The other half being those files that nice Damian Green bloke asked you to look after? ;D
-
In a home environment, I'd recommend keeping it nice and simple, and use far simpler storage - 24/7 availability is not a requirement, all you want is data integrity, and a simple disk system and robust backup regime is the best solution to that IMHO.
Now if you could let me know how I can easily (and with similar expense) back up 32TB (yes, 32TB filled, not capacity) of stuff, that'd be great ;)
The only way I can imagine a robust backup solution for that would be several sets of drives at £1k a pop, cycling them continuously and backing them up pretty much continuously.
Or, y'know, I could just keep my RAID6 array and not worry about it too much ;) (I mean, unless I am very unlucky and lose three drives at once - which could happen..)
Don't forget, TB, there's a bit difference between a corporate environment and Skrunts not wanting to lose his porn collection ;D
But RAID won't back up your data either. Remember RAID is about availability. Virii, controller glitches, human error, smoke etc can all easily reduce 32Tb to SFA with remarkable speed and efficiency ;)
-
The attraction of NAS devices is that they can be "always-on" without consuming a stupid amount of leccy, unlike any usable PC. If all your media, documents, etc. are on a PC it's a real ballache to have to remember to turn it on before you decide to listen to some music in bed using your tablet, etc..
There is a place for always on shared storage. For some it will be cloud, for some it will be NAS type devices. The cheap ARM based ones (and even Atom based) are pretty dire though, both in performance and integrity.
When something as "good" as, say, the HP gen8 Microserver is available, and pulls 30-35W with a pair of shitty 3Tb Toshiba 3.5" drives (unsurprisingiy one of them has popped up a prefailure warning, *sigh*) is available for around £100 have cashback incentives, its a bit of a non brainer. And it can run your favourite Windows/Linux OS, which is far more robust than the bastardised toy NAS systems. Granted, enterprise class it ain't, but its a cheap way to a small, quiet, efficient NAS. Just don't use that crap onboard controller for RAID5 ;D
-
The attraction of NAS devices is that they can be "always-on" without consuming a stupid amount of leccy, unlike any usable PC. If all your media, documents, etc. are on a PC it's a real ballache to have to remember to turn it on before you decide to listen to some music in bed using your tablet, etc..
There is a place for always on shared storage. For some it will be cloud, for some it will be NAS type devices. The cheap ARM based ones (and even Atom based) are pretty dire though, both in performance and integrity.
When something as "good" as, say, the HP gen8 Microserver is available, and pulls 30-35W with a pair of shitty 3Tb Toshiba 3.5" drives (unsurprisingiy one of them has popped up a prefailure warning, *sigh*) is available for around £100 have cashback incentives, its a bit of a non brainer. And it can run your favourite Windows/Linux OS, which is far more robust than the bastardised toy NAS systems. Granted, enterprise class it ain't, but its a cheap way to a small, quiet, efficient NAS. Just don't use that crap onboard controller for RAID5 ;D
Yep, agreed, that's where my money would go but most people want an "appliance". (= they CBA to install software on it, etc.) ::)