Earlier in June, I set about designing and setting up a FreeNAS server, in preparation to migrate away from my existing dual-Synology RAID based system(s) in place.
My primary Synology unit is a 5-bay DSM1515+ with 5x WD Red Pro 4TB drives, configured with a RAID5 array. This is actually a mistake that I initially made when setting this up, driven at the time by cost motivations to provide the most 'available space' with 2-drive failure redundancy.
I would also replicate and backup data to a second 8-bay DSM1815+ with 6x WD Red Pro 4TB drives. This unit would auto-power on between 1-6am to run all the backups and syncing/replication.
Why RAID5 should never be used.
What I did not know at the time, and really should have, was that RAID systems are typically prone to failing at the time of resilvering (or rebuilding) an error at the time of recovery. Assume you have a RAID5 array, and a single drive dies; you insert a replacement drive, and it is at this time that during the re-build a second drive may very-well likely die. You're now left with a DEAD RAID5 array.
There's also something called 'URE' errors or Unrecoverable Read Errors. Interestingly enough, manufacturers tend to report these errors as <10 in 10^15 which is the same as saying 1 error in 10^14 — which means you may get a URE read error every 12.5TB of data. Given that my RAID5 array had ~16TB of usable space, this was another serious point of failure in my system.
In hindsight, I should have bitten the bullet, and gone with RAID6 even though it would have left me with only 12TB of usable space and 3-drive failure redundancy. Even though you loose 50% more space to parity data, it is oh so worth it compared to RAID5.
Why I chose ZFS
In my first build log, I detailed my reasoning for choosing ZFS
I was keen on ZFS and that was the 'priority'; as far as operating systems go, FreeBSD has a track-record of being solid, considered to have a more robust network stack; it's Unix. Had there been a linux-alternative, I may have gone with that.
What really resonated with me though, was the popularity of ZFS (FreeNAS) on YouTube amongst the tech community. @tekwendel has also covered ZFS in quite a few videos, including one where he ends up recovering data from a 'dead-array'.
I basically 'primed' myself for the setup process by watching this excellent walkthough by Tom @ Lawrence Systems and also referred some other videos on the topic.
You can find my build log(s) below and a list of various aspects that I ended up touching on with regards to getting my FreeNAS server up and running.
Build log: https://linustechtips.com/main/topic/810276-freenas-server-build/ Build log (clone): https://forum.level1techs.com/t/serious-freenas-server-build-3d-printing-fun
Tips & TIL from Setting up FreeNAS 11
Confirming ECC is Operational
Ensure your memory is listed with a Total Width of 72 bits, this confirms ECC is working correctly. Non-ECC RAM will run at a Width of 64 bits.
[freenas ~]$ dmidecode -t 17
Handle 0x005C, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x0058
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 72 bits
Size: 16384 MB
Form Factor: RIMM
Set: None
Locator: DIMM_B1
Bank Locator: NODE 1
Type: DDR4
Type Detail: Synchronous
Speed: 2133 MHz
Manufacturer: Micron
Serial Number: 16EXXXXXX
Asset Tag: DIMM_B1_AssetTag
Part Number: 36ASF2G72PZ-2G1B1
Rank: 2
Configured Clock Speed: 2133 MHz
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Encryption (of ZFS Volume)
Your CPU will need to support AES-NI; if no crypto hardware is installed, without hardware acceleration, there will be about a 20% performance hit for a single disk.
The Intel Xeon E5-2620 v4 CPU in my FreeNAS Build supports this.
Enabling Virtualisation
[freenas ~]$ sudo cat /var/run/dmesg.boot | grep VT
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Ubuntu Install, Fixing GRUB for UEFI (Post-install)
https://forums.freenas.org/index.php?threads/how-to-boot-ubuntu-desktop-vm-in-uefi-mode.53863/
sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get install -y boot-repair && boot-repair
Configure APC UPS For Power Management
https://www.cyberciti.biz/faq/how-to-install-apc-ups-on-freenas-server/
[mdesilva@freenas ~]$ dmesg | grep -i --color 'american power conversion'
ugen0.5: <American Power Conversion Back-UPS XS 1400U FW926.T2 .I USB FWT2> at usbus0
[mdesilva@freenas ~]$ upsc ups@localhost
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2016/06/23
battery.runtime: 1988
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 27.1
battery.voltage.nominal: 24.0
device.mfr: American Power Conversion
device.model: Back-UPS XS 1400U
device.serial: *********324
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: /dev/ugen0.5
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 280
input.transfer.low: 155
input.voltage: 228.0
input.voltage.nominal: 230
ups.beeper.status: enabled
ups.delay.shutdown: 20
ups.firmware: 926.T2 .I
ups.firmware.aux: T2
ups.load: 19
ups.mfr: American Power Conversion
ups.mfr.date: 2016/06/23
ups.model: Back-UPS XS 1400U
ups.productid: 0002
ups.realpower.nominal: 700
ups.serial: 3B1625X14324
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d