Earlier in June, I set about designing and setting up a FreeNAS server, in preparation to migrate away from my existing dual-Synology RAID based system(s) in place.
My primary Synology unit is a 5-bay DSM1515+ with 5x WD Red Pro 4TB drives, configured with a RAID5 array. This is actually a mistake that I initially made when setting this up, driven at the time by cost motivations to provide the most 'available space' with 2-drive failure redundancy.
I would also replicate and backup data to a second 8-bay DSM1815+ with 6x WD Red Pro 4TB drives. This unit would auto-power on between 1-6am to run all the backups and syncing/replication.
Why RAID5 should never be used.
What I did not know at the time, and really should have, was that RAID systems are typically prone to failing at the time of resilvering (or rebuilding) an error at the time of recovery. Assume you have a RAID5 array, and a single drive dies; you insert a replacement drive, and it is at this time that during the re-build a second drive may very-well likely die. You're now left with a DEAD RAID5 array.
There's also something called 'URE' errors or Unrecoverable Read Errors. Interestingly enough, manufacturers tend to report these errors as <10 in 10^15 which is the same as saying 1 error in 10^14 — which means you may get a URE read error every 12.5TB of data. Given that my RAID5 array had ~16TB of usable space, this was another serious point of failure in my system.
In hindsight, I should have bitten the bullet, and gone with RAID6 even though it would have left me with only 12TB of usable space and 3-drive failure redundancy. Even though you loose 50% more space to parity data, it is oh so worth it compared to RAID5.
Why I chose ZFS
In my first build log, I detailed my reasoning for choosing ZFS
I was keen on ZFS and that was the 'priority'; as far as operating systems go, FreeBSD has a track-record of being solid, considered to have a more robust network stack; it's Unix. Had there been a linux-alternative, I may have gone with that.
What really resonated with me though, was the popularity of ZFS (FreeNAS) on YouTube amongst the tech community. @tekwendel has also covered ZFS in quite a few videos, including one where he ends up recovering data from a 'dead-array'.
I basically 'primed' myself for the setup process by watching this excellent walkthough by Tom @ Lawrence Systems and also referred some other videos on the topic.
You can find my build log(s) below and a list of various aspects that I ended up touching on with regards to getting my FreeNAS server up and running.
Build log: https://linustechtips.com/main/topic/810276-freenas-server-build/ Build log (clone): https://forum.level1techs.com/t/serious-freenas-server-build-3d-printing-fun
Tips & TIL from Setting up FreeNAS 11
Confirming ECC is Operational
Ensure your memory is listed with a Total Width of 72 bits, this confirms ECC is working correctly. Non-ECC RAM will run at a Width of 64 bits.
[freenas ~]$ dmidecode -t 17 Handle 0x005C, DMI type 17, 40 bytes Memory Device Array Handle: 0x0058 Error Information Handle: Not Provided Total Width: 72 bits Data Width: 72 bits Size: 16384 MB Form Factor: RIMM Set: None Locator: DIMM_B1 Bank Locator: NODE 1 Type: DDR4 Type Detail: Synchronous Speed: 2133 MHz Manufacturer: Micron Serial Number: 16EXXXXXX Asset Tag: DIMM_B1_AssetTag Part Number: 36ASF2G72PZ-2G1B1 Rank: 2 Configured Clock Speed: 2133 MHz Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown
Encryption (of ZFS Volume)
Your CPU will need to support AES-NI; if no crypto hardware is installed, without hardware acceleration, there will be about a 20% performance hit for a single disk.
The Intel Xeon E5-2620 v4 CPU in my FreeNAS Build supports this.
[freenas ~]$ sudo cat /var/run/dmesg.boot | grep VT VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Ubuntu Install, Fixing GRUB for UEFI (Post-install)
sudo add-apt-repository ppa:yannubuntu/boot-repair sudo apt-get install -y boot-repair && boot-repair
Configure APC UPS For Power Management
[mdesilva@freenas ~]$ dmesg | grep -i --color 'american power conversion' ugen0.5: <American Power Conversion Back-UPS XS 1400U FW926.T2 .I USB FWT2> at usbus0 [mdesilva@freenas ~]$ upsc ups@localhost battery.charge: 100 battery.charge.low: 10 battery.charge.warning: 50 battery.date: 2001/09/25 battery.mfr.date: 2016/06/23 battery.runtime: 1988 battery.runtime.low: 120 battery.type: PbAc battery.voltage: 27.1 battery.voltage.nominal: 24.0 device.mfr: American Power Conversion device.model: Back-UPS XS 1400U device.serial: *********324 device.type: ups driver.name: usbhid-ups driver.parameter.pollfreq: 30 driver.parameter.pollinterval: 2 driver.parameter.port: /dev/ugen0.5 driver.parameter.synchronous: no driver.version: 2.7.4 driver.version.data: APC HID 0.96 driver.version.internal: 0.41 input.sensitivity: medium input.transfer.high: 280 input.transfer.low: 155 input.voltage: 228.0 input.voltage.nominal: 230 ups.beeper.status: enabled ups.delay.shutdown: 20 ups.firmware: 926.T2 .I ups.firmware.aux: T2 ups.load: 19 ups.mfr: American Power Conversion ups.mfr.date: 2016/06/23 ups.model: Back-UPS XS 1400U ups.productid: 0002 ups.realpower.nominal: 700 ups.serial: 3B1625X14324 ups.status: OL ups.test.result: No test initiated ups.timer.reboot: 0 ups.timer.shutdown: -1 ups.vendorid: 051d