debian zfs nas setup
Not a fan of TrueNAS and OpenMediaVault. Too much clicking. I decided to make my own custom ZFS NAS simply using Debian. This blog talks in much more detail about how to do this. I am only writing the bare minimum to document what I did for my own system.
Initialization
# install requirements
sudo apt install ca-certificates curl openssh-server vim htop iperf3 man zip unzip wget fonts-noto-cjk rsync smartmontools iotop zfsutils-linux linux-headers-amd64 zfs-dkms zfs-zed zfs-auto-snapshot
sudo vim /etc/apt/sources.list # add non-free-firmware & contrib
sudo apt update
sudo apt install linux-headers-amd64 zfsutils-linux zfs-zed zfs-dkms zfs-auto-snapshot
sudo modprobe zfs # enable zfs kernel module
lsblk # list block devices
ls /dev/disk/by-id/ # get drive IDs
sudo cfdisk /dev/sda # wipe/partition the drives
sudo cfdisk /dev/sdb
sudo cfdisk /dev/sdc
history # look at the drive IDs so I can type the next command
# create the zfs pool (raidz1=raidz. 3x10TB drives)
sudo zpool create -o ashift=12 -f pompool raidz /dev/disk/by-id/ata-WDC_WD100EFGX-68CPLN0_WD-B202TRYJ /dev/disk/by-id/ata-WDC_WD100EFGX-68CPLN0_WD-B203AMPJ /dev/disk/by-id/ata-WDC_WD100EFGX-68CPLN0_WD-B203D83J
lsblk # confirm the stuff is there
umount /dev/sda # unmount the drives
umount /dev/sdb
umount /dev/sdc
# configure the zfs pool
sudo zfs set compression=lz4 pompool
sudo zfs set xattr=sa pompool
sudo zfs set atime=off pompool
sudo zfs set mountpoint=/pompool pompool
sudo zfs set dnodesize=auto pompool
sudo chown -R $USER:$USER /pompool
cd pompool/
# create zfs datasets
sudo zfs create pompool/documents
sudo zfs create pompool/videos
sudo zfs create pompool/music
sudo zfs create pompool/backups
sudo apt install nfs-kernel-server
sudo vim /etc/exports # configure nfs mount permissions
# /pompool/public *(rw,sync,no_subtree_check)
# /pompool/books 192.168.1.128(rw,sync,no_subtree_check)
# /pompool/music 192.168.1.0/255.255.255.0(rw,sync,no_subtree_check)
# etc...
sudo systemctl restart nfs-kernel-server
sudo apt install ufw # firewall rules
sudo ufw enable
sudo ufw allow in from 192.168.1.0/24 to any port 2049 comment "NFSv4"
sudo ufw allow in from 192.168.1.0/24 to any port 111 comment "NFSv2/3"
sudo ufw allow in from 192.168.1.0/24 to any port 22 comment "ssh"
sudo ufw status numbered # verify firewall
sudo vim /etc/default/nfs-kernel-server
sudo systemctl restart ufw.service
sudo systemctl restart nfs-kernel-server.service
sudo chown -R $USER:$USER /pompool
Periodic Snapshots
Shell scripts in these directories handle making period snapshots. Comment out the code in whichever scripts you don't want to run.
/etc/cron.hourly/zfs-auto-snapshot
/etc/cron.daily/zfs-auto-snapshot
/etc/cron.weekly/zfs-auto-snapshot
/etc/cron.monthly/zfs-auto-snapshot
/etc/cron.yearly/zfs-auto-snapshot
List snapshots
zfs list -t snapshot
Turn off auto snapshot for specific pool/dataset
zfs set com.sun:auto-snapshot=false pompool # pool
zfs set com.sun:auto-snapshot=false pompool/videos # dataset
Periodic Scrub & S.M.A.R.T. tests
I configured nullmailer and created this script: /usr/local/bin/email-notify.sh
#!/usr/bin/env bash
# /usr/local/bin/email-notify.sh
# Email settings
TO="me@mydomain.com" # put your email address here
HOST="$(hostname)"
# Build message
SUBJECT="$HOST | S.M.A.R.T. Alert"
MESSAGE=$(cat <<EOF
SMARTD event detected on host: $HOST
Device: $SMARTD_DEVICE
Model: $SMARTD_MODEL
Serial: $SMARTD_SERIAL
Event Type: $SMARTD_FAILTYPE
Full SMART message:
$SMARTD_MESSAGE
EOF
)
echo "$MESSAGE" | mail -r "receiver@yourdomain.com" -s "$SUBJECT" "$TO"
ZFS puts a monthly scrub task in /etc/cron.d/zfsutils-linux. I modified it slightly.
# /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# TRIM the first Sunday of every month.
24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi
# SCRUB the second Sunday of every month.
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi
Then modified /etc/zfs/zed.d/zed.rc
# /etc/zfs/zed.d/zed.rc
ZED_NOTIFY_VERBOSE=1 # Send notify regardless of pool health
ZED_EMAIL_PROG="/usr/local/bin/email-notify.sh"
Then restart Zed
sudo systemctl restart zed
For S.M.A.R.T. testing, first create a symbolic link from the email notification script to smartd notification folder. smartd will execute all the scripts under the folder when something goes wrong.
sudo ln -s /usr/local/bin/email-notify.sh /etc/smartmontools/run.d/10email
Then, edit /etc/smartd.conf to add the tests we want:
# /etc/smartd.conf
# First, comment out this line as it will block all the tests defined after it.
DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner # Comment it!!
# Add these at the end of the file. These should be the only lines that are not commented.
DEFAULT -S on -a -I 194 -I 190 -R 5 -s (S/../../05/03|S/../../12/03|S/../../19/03|S/../../26/03|L/../../08/03|L/../../22/03) -m root -M exec /usr/share/smartmontools/smartd-runner
/dev/disk/by-id/ata-WDC_WD60EFPX-68C5ZN0_WD-XXXXXXXXXXXX # List all the disks here
/dev/disk/by-id/ata-WDC_WD60EFPX-68C5ZN0_WD-YYYYYYYYYYYY
This script performs short SMART tests on 5th, 12th, 19th, and 26th of a month, and performs long tests on 8th and 22nd.
This way we spread the tests and scrubs (1st and 16th) evenly across the month. The command also rules out some non-critical values, such as the drive temperatures, so I don't get a warning about drive being colder or hotter than last time every few days.
To test it works, we can add a -M test to the default command and restart smartd:
# /etc/smartd.conf
DEFAULT -S on -a -I 194 -I 190 -R 5 -s (S/../../05/03|S/../../12/03|S/../../19/03|S/../../26/03|L/../../08/03|L/../../22/03) -m root -M exec /usr/share/smartmontools/smartd-runner -M test # Add `-M test` here
sudo systemctl restart smartd
You should receive test notifications to your email now. After verifying it's working, remove the -M test directive and restart smartd.
Off-site backups
coming soon...
Haven't yet decided how this is going to work. The blog I mentioned earlier, that I basically copied from, writes about how they backed up their snapshots to S3.