UDPATE 7/13/2016:
Drives have been solid and still running 24/7 since purchased well over 1 year ago. S.M.A.R.T. values for all 8 disks are great. No complaints.
I bought 9 of these drives; a few from Amazon and rest from other places to mix up the lots and dates of manufacture. 8 of the drives are in a RAID10 (3 sets of 2 disk span) configuration controlled by a dedicated LSI MegaRAID SAS 9361-8i PCI-Express 3.0 x8 12Gb/s (future proofing). The stripe defined is 256kb since will be storing mainly large files. However, benchmarking was done using single disk, 6-disk (3 sets of 2 disk span) and 8-disk (4 sets of 2 disk span) array configuration.
Due to some unmentioned system limitations, the drives are set up in the same box as my ESXi host, a disk container created and attached to a guest machine that serves as my dedicated NAS; said guest machine runs CentOS 7.1 for an operating system.
The filesystem chosen is XFS, as it scales well with large storage arrays. First, the partition will consist of the entire array size available. It was configured using the following set of commands (Note about "print" read-out below. I added pipe "|" separators as Amazon stripped out my formatting):
# parted -a optimal /dev/sdx
(parted) mklabel gpt
(parted) unit s
(parted) mkpart primary xfs 2048s 11721043967s
(parted) print
Number | Start | End | Size | File system | Name | Flags
1 | 2048s | 11721043967s | 11721041920s | xfs | primary |
Verify alignment:
(parted) align-check optimal 1
1 aligned
Save/write partition:
(parted) quit
NOTE: For optimal/best performance, specify the 'optimal' command so the partition alignment boundary is in multiples of 1MiB (1024x1024); which then the starting sector (2048), the ending sector, minus 1 (11721043968 - 1) and the RAID stripe size (256) are all evenly divisible.
The filesystem was created specifying the block-size, stride and stripe-width options for optimal performance with the array and the partition created. The command is as follows:
# mkfs.xfs -b 4096 -E stride=64,stripe-width=256 /dev/sdx1
NOTE: Your numbers will be different. I highly recommand using a stride and stripe-width calculator available on The Web. You can verify the filesystem attributes once it is created using either of the commands below:
# tune2fs -l /dev/sdx1
# dumpe2fs /dev/sdx1 | less
--------------------------------------
SETUP:
Keep in mind that many factors will effect performance - hardware capabilities/limitations, hardware controller configuration, RAID array stripe size, NIC card performance, caching, kernel I/O scheduler (Linux), the file-system format and options you choose (Linux - see below at 'stride' and 'stripe-width'), disk mounting options defaults/user-specified (Linux), router/switch, etc.
I do not run actual server hardware, but a re-purposed micro-ATX 'desktop' converted to an ESXi host with various guest machines that draws an average ~144 watts. Here is the complete set-up:
Fractal Design Node 804 Case
3 x Included fans that came with the Fractal Design Node 804 case
3 x Noctua SSO2 Bearing, Retail Cooling NF-F12 iPPC 2000 PWM
Intel i7-4790S Haswell Quad-Core 3.2GHz LGA 1150 Z87 65W
ASUS MAXIMUS VI GENE LGA 1150
G.SKILL Ripjaws X Series 32GB (4x8GB) DDR3 2133
Samsung 850 Pro 512GB SSD
Corsair RM750i 750 watt PSU (Had on-hand)
CyberPower CP1500PFCLCD PFC UPS (supports my Corsair's "Active PFC")
LSI 9300 MegaRAID SAS 9361-8i PCI-Express 3.0 x8 12Gb/s
2 x LSI LSI00410 0.6mm Internal Cable SFF8643 to x4 SATA HDD
8 x HGST Deskstar NAS 6TB 7200 RPM SATA 6.0Gb/s 128MB Cache
For LAN benchmarks, the host machine is wired using CAT6 to an enterprise-class Ubiquiti EdgeRouter. Benchmarking includes scores from single disk, 6-disk and 8-disk RAID10 array.
------------------------------------------
BENCHMARKS:
NOTE: These tests go a little deeper than your typical reviewer to show various scenerios and how these disks perform. If you don't understand q-depths, block-sizes, etc., how and when you'll generally face them - then I suggest you stick to the "HDParm" and "DD" scores below if you are looking for general performance scores.
Samba 4 transfer of a 2GB file to another wired computer [Windows 8.1]:
Sequential Read: 113 MBps
Sequential Write: 113 MBps
The theoretical maximum sequential transfer over Gigabit Ethernet is 125MBps (1000/8=125). Pretty darn close. I have tried tweaking SMB socket option values for my Samba configuration, increased the kernel read-ahead cache but it would appear that I have hit my ceiling.
---------- HDParm ----------
I ran HDParm 5 times and took the average:
"hdparm -Tt /dev/sdx"
(single disk)
Cache read average: 16639 MBps
Average = 200.444 MBps
(6-disk array)
Cache average = 16817 MBps
Average = 632.578 MBps
(8-disk array)
Cache read average: 16538.47 MBps
Average = 824.900 MBps
------------ DD -------------
I ran "dd" with the following parameters ("/temp/testfile" = RAID array disk partition mounted to /temp; "conv=fsync" to allow for synchronous IO; "rm /temp/testfile" to remove test file after completion):
"dd if=/dev/zero of=/temp/testfile bs=1G count=12 conv=fsync; rm /temp/testfile"
Single disk performance: 208 MB/s
6-disk RAID10 array sequential performance: 621 MB/s
8-disk RAID10 array sequential performance: 812 MB/s
------------ FIO -----------
Important note: FIO works at a low-level. Be sure that you specify a 'filename' or else tests will be written over your filesystem; most likely causing filesystem corruption.
FIO is great for providing storage performance statistics based on more real-world scenerios with the abundance of options available, which I'll run through some below. NOTE: This part of the review assumes that you have the technical knowledge to understand the concepts being described.
For the first performance test below it is a pretty straight-forward sequential read test based on a block-size of 4k, 4 worker processes, an io-depth of 4, a transfer of a 2GB file with a runtime of 60 seconds. Here is the command I used:
"fio --filename=/path/to/file/on/disk --direct=1 --rw=read --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --iodepth=4 --numjobs=4 --runtime=60 --group_reporting --name=4k_seq_test --size=2g"
(single disk)
read iops = 41995
read trans. avg. = 167983 KB/s (167.983 MBps)
(6-disk array)
read iops = 126,167
read trans. avg. = 504668 KB/s (504.668 MBps)
---------
Change to write-mode '--rw=write':
"fio --filename=/path/to/file/on/disk --direct=1 --rw=write --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --iodepth=4 --numjobs=4 --runtime=60 --group_reporting --name=4k_seq_test --size=2g"
(single disk)
write iops = 41730
write trans. avg. = 166920 KBps (166.92 MBps)
(6-disk array)
write iops = 105,437
write trans. avg. = 421750 KB/s (421.75 MBps)
---------
Now to get a mix of read and writes together (70% read, 30% writes) with various block sizes and q-depth value changes, the following command is a RANDOM read/write (70% reads, 30% writes. Note the size option value has been shrunk to 512MB, the "--rw" option value has changed to "randrw" and the option "--rwmixread=70" has been added):
"fio --filename=/path/to/file/on/disk --direct=1 --rw=randrw --rwmixread=70 --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --iodepth=4 --numjobs=4 --runtime=60 --group_reporting --name=4k_rwmix70-30_test --size=512MB"
(single disk)
read iops = 284
write iops = 121
read trans. avg. = 1137 KB/s (1.137 MBps)
write trans. avg. = 484 KB/s (0.484 MBps)
(6-disk array)
read iops = 2255
write iops = 971
read trans. avg. = 9022 KB/s (9.22 MBps)
write trans. avg. = 3886 KB/s (3.886 MBps)
(8-disk array)
read iops = 4456
write iops = 1906
read trans. avg. = 17827 KB/s (17.827 MBps)
write trans. avg. = 7626.5 KB/s (7.6265 MBps)
----------
Changed block-size from 4k to 8k:
"fio --filename=/path/to/file/on/disk --direct=1 --rw=randrw --rwmixread=70 --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --iodepth=4 --numjobs=4 --runtime=60 --group_reporting --name=4k_rwmix70-30_test --size=512MB"
(single disk)
read iops = 294
write iops = 127
read trans. avg. = 2359 KB/s (2.359 MBps)
write trans. avg. = 1023 KB/s (1.023 MBps)
(6-disk array)
read iops = 2479
write iops = 1068
read trans. avg. = 19834 KB/s (19.834 MBps)
write trans. avg. = 8551 KB/s (8.551 MBps)
----------
Changed block-size to 16k:
"fio --filename=/path/to/file/on/disk --direct=1 --rw=randrw --rwmixread=70 --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=16k --iodepth=4 --numjobs=4 --runtime=60 --group_reporting --name=4k_rwmix70-30_test --size=512MB"
(single disk)
read iops = 293
write iops = 125
read trans. avg. = 4691 KB/s (4.691 MBps)
write trans. avg. = 2014 KB/s (2.014 MBps)
(6-disk array)
read iops = 2398
write iops = 1029
read trans. avg. = 38373 KB/s (38.373 MBps)
write trans. avg. = 16470 KB/s (16.47 MBps)
-----------
Changed to run sequential read and writes (70% reads and 30% writes)(rw=rw, size=2gb, bs=8):
"fio --filename=/path/to/file/on/disk --direct=1 --rw=rw --rwmixread=70 --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --iodepth=4 --numjobs=4 --runtime=60 --group_reporting --name=4k_rwmix70-30_test --size=2g"
(single disk)
read iops = 1515
write iops = 650
read trans. avg. = 12127 KB/s (12.127 MBps)
write trans. avg. = 5207 KB/s (5.207 MBps)
(6-disk array)
read iops = 41615
write iops = 17909
read trans. avg. = 332920 KB/s (332.92 MBps)
write trans. avg. = 143272 KB/s (143.272 MBps)
-------------
Changed q-depth to 16:
"fio --filename=/path/to/file/on/disk --direct=1 --rw=rw --rwmixread=70 --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=8k --iodepth=16 --numjobs=4 --runtime=60 --group_reporting --name=4k_rwmix70-30_test --size=2g"
(single disk)
read iops = 2087
write iops = 899
read trans. avg. = 16700 KB/s (16.7 MBps)
write trans. avg. = 7193 KB/s (7.193 MBps)
(6-disk)
read iops = 41062
write iops = 17667
read trans. avg. = 328501 KB/s (328.501 MBps)
write trans. avg. = 141342 KB/s (141.342 MBps)
-------------
Changed q-depth to 32:
"fio --filename=/path/to/file/on/disk --direct=1 --rw=rw --rwmixread=70 --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=16k --iodepth=32 --numjobs=4 --runtime=60 --group_reporting --name=4k_rwmix70-30_test --size=2g"
(single disk)
read iops = 1283
write iops = 554
read trans. avg. = 20535 KB/s (20.535 MBps)
write trans. avg. = 8871 KB/s (8.871 MBps)
(6-disk disk)
read iops = 31552
write iops = 13524
read trans. avg. = 504839 KB/s (504.839 MBps)
write trans. avg. = 216388 KB/s (216.388 MBps)
----------- SUMMARY -----------
These drives have been rock-solid running 24/7. They are quite noisy when 'active' but personally, a moot 'issue' for me since it's located with multiple servers running 24/7. Drives are rated for 1M hours MTBF and have a 3 year warranty; really wish these were 5 year. I guess that if the Backblaze hard drive reliability reports can paint a relative picture to what can be expected of these drives - hopefully can mean that these will run reliably for some time.
| Standing screen display size | 3.5 Inches |
|---|
