One questions I have regarding NAS is backups of the NAS itself.
If I have a very simple NAS with 2 drives in Raid 1 (let's call it Drive A and B), and I want to make a physical backup of my NAS in a different location. How easy is it ? What is the best practice. Ideally you could just have a big rsync method that takes care of it, or rclone as described in the article, but what if you want to do it without any network transfer (because your connection is too slow/ you can't afford it)
Does the following protocol make sense:
Removing Drive A , replacing it with an empty drive C. Wait for the NAS to synchronize drive C. Then remove Drive B, replace with drive D, wait for the NAS to synchronize drive D.
Taking Drive A and B to a different location, plug them, and have the backup working out of the box.
Is it that easy? What about more complicated RAID setups?
is there an easy "Prepare Backup -> Please insert First drive for your backup -> First drive filled up -> please insert second drive for your backup -> ... -> Please insert last drive for your backup" and then you take all those newly filled drives and shove them in a different box and they have all the data at the time of the backup, with either the right zfs and raid configuration, or at least a simple data dump in a non-raid configuration.
If you're using ZFS, the right thing to do is to attach drives C and D temporarily, create a second zpool on them, them use zfs send/receive to replicate snapshots from the primary drives to the backups. You can then export the zpool and move it to a different location.
Refreshing the backups is either done by putting the backups online at the remote location and syncing the deltas between the last snapshot and the current over the net, or by bringing the drives back to the primary, and sending the deltas.
The sanoid/syncoid toolset will help immensely with handing the necessary zfs snapshot and send/receive commands:
https://github.com/jimsalterjrs/sanoid
If your NAS is small enough, you could get a couple of external drives and rotate them.
I've got a QNAP 4-drive NAS and the stuff I want to keep backed up fits in under 2TB. Based on that, I've got a 2TB external drive and have set up a backup job to sync the stuff I want to keep to that drive every morning.
There's no reason you can't get two drives and swap them every week. Just set your backup job to run weekly and unmount/eject the target drive when complete. Every Monday you just grab your drive and take it to work, and every Friday you bring the old drive home and plug it in.
Of course, any backup that requires manual work is likely to get neglected eventually. "Oh, I haven't done much work in the last week, I don't need to take the drive to work today" is something I've told myself all too often when it comes to my Mac backups.
Automated is better. If you have a second NAS at a remote site, you should be able to use ZFS send/receive to update a remote NAS over the internet.
For me, I have 2x extra drives and a USB caddy,
and rsync the array onto the caddy automatically,
keepping the unused drive offsite.
This does mean that the wear on the live array
is higher than the offsite ones, and I have to have 4 drives
total, but since RAID1 with a traditional filesystem doesn't provide integrity protection (e.g. bit errors on 1 drive can cause silent corruption), I don't have to worry about subtle raid rebuild issues gradually propagating through the entire raid set.
The 'cheaper' version would be to only have 1 offsite drive,
but that means my data on the raid array is only protected
from severe failures up to the last time I ran the sync.
Longer run, I'm looking at moving up to something with
integrity protection, but since my server is OpenBSD (less storage configuration options), this means RAID5 which was only recently OK'd for rebuilds, and soft-raid5 rebuilds take forever on spinny drives -
Will probably wait and upgrade to 4x SSD's 1st or get a hardware
raid card (my data set is fairly small).
Another thought I had was to setup a raspberry PI at a friend/relative's place and have an Rsync run nightly to it,
and offer same to them... but haven't gotten around to it.
If you really don't want to go over ethernet (and I'm assuming fiber is out of the question, too), there aren't really any great methods to automate this process. If you don't want to pull drives and put your data at risk, you might be able to add a third drive to the mirror, let it replicate, then disconnect it and plug it into the second system. If you're on FreeNAS, check out this section for guidelines on how to add an extra drive to the array: http://doc.freenas.org/9.10/storage.html#replacing-drives-to...
...but really, you should figure out a way to go over ethernet. Even if it takes a really long time, it'll be soooo much easier to have everything automated. You also don't have to risk pulling drives, etc...
Thanks for the really detailed ZFS allocation write up. I've recently been building ZFS storage systems on Ubuntu 16.04 (one in excess of 800TB), and I've poured over the various ashift/recordsize options and spreadsheets, but your description was quite useful in cementing my understanding of why the large recordsize is useful.
I'm glad you think so. I also found it was extremely difficult to find a thorough explanation of the types of overhead I described, so I wanted to get it all together in one place. Working through it while writing it also helped me understand it much better... it's a pretty abstract concept, especially when you factor in compression, etc.
Got to ask about the elephant in the room, but what do you use it for / host?
Personally, I'm thinking of building a FreeNAS for photo storage / media, but unfortunately I've only got a four-drive array SuperMicro rack server (spare one I have from decomissioning a business).
Other things I'm interested in is perhaps hosting a few gaming servers and such for friends/family (Minecraft, etc.)
Hard to find uses for it other than offline-lowpower-media-storage.
I do a lot of high resolution photography and videography work. I've got about ~15 years of raw images and video files that I could probably go through and delete, but I'd rather keep them around so I can pretend that maybe some day, someone will be interested in looking at them.
I look at it as a modern version of the boxes full of pictures and slides my parents and grandparents kept in their basement for decades and never looked at.
(I meant to reply to you several hours ago, but HN wasn't letting me post comments for some reason...)
I've got rclone set up to encrypt and upload everything to ACD. There's a section at the bottom of the article that goes into some depth on this and some other backup strategies I've tried in the past (including CrashPlan, Backblaze, and Zoolz, all of which are awful). Check http://jro.io/nas#rclone . I never considered building a second NAS, it does seem pretty stupid, even for an enterprise setup. The whole idea is to get the data off-site.
As a side note, some of the so-called "FreeNAS people" can tend to blindly parrot a given general guideline without really understanding the reasoning behind it or why it might be perfectly valid in certain situations to disregard it. For instance, ask them about bhyve and I promise you'll get at least one response along the lines of "bhyve isn't officially supported in FreeNAS so you shouldn't use it under any circumstances, period."
If you're able to tag stuff you know you want to keep and it's a smaller set you could look in to something like Backblaze B2 (previously a featured story on HN); the storage costs are relatively moderate, but restoration from it will cost you.
I haven't yet heard of any solution along the lines of "Rent a (large) NAS for a month" for those times that you're upgrading your array and need to switch filesystem formats. Having that option would make the juggling much easier and safer. Looking at the S3 storage and bandwidth costs I imagine there is actually a market to be served by such a product.
Maybe renting one of those higher end tape drives makes sense... but I can't get over the idea that even renting a stack of hard disks would be cheaper and more effective at this scale.
PS: Make sure you encrypt all of the data going in to the temporary storage; those aren't your disks.
You still have to put the second NAS somewhere, preferably offsite to avoid the house-burns-down fail case. Personally, I don't even know where I put this second NAS, since it would need power and bandwidth, unless you really wanted to sneakernet a NAS, which would work, but seems really weird.
My home nas is small enough where I can reliably backup to some USB external drives and store in a drawer offsite. According to FreeNAS, that's a horrible solution because USB is too error prone and moving disks shortens their life, and blah blah blah, and so USB backup is explicitly a WILLNOTFIX, and a sign that the requestor is stupid, as opposed to knowing full well what the risks are, and is satisfied with them. The horrible FreeNAS community, and the lack of this feature was why I adopted OpenMediaVault. (I highly recommend OMV.)
I guess I could always upload TARs to Glacier. That might be a legitimate solution.
> You still have to put the second NAS somewhere, preferably offsite to avoid the house-burns-down fail case.
Think pretty much the only viable solution for this for home users
is to have a 'peering agreement' with a trusted friend where you each
colo the others machine at your home.. however this can be tricky
because you're sticking all of your sensitive stuff in someone elses
house and trading some level of full network acess to each other -
though I suppose trading access to some kind of encrypted rsync-like dumps or similar might work without some of those risks being too high
Amazing setup and even better write up. It shames my tiny 2x 4tb setup.
My only suggestion would be to try out gaffer tape instead of the duct/masking tape combo. I started using it about a year ago, and haven't looked back. No residue and no dry flaking over time.
Thank you for shamelessly copying the steps in the Bhyve video I created. :-)
You'll be pleased to know that in the 9.10 nightly train, the base OS has been upgraded to FreeBSD 11 and the GUI has a UI for Bhyve, also enables VGA consoles.
I already have a FreeNAS system, but I have been thinking about upgrading a few parts, like the chassis. Right now it sits in a cheap Tower, and doesn't have enough drive slots for my liking. So this is a really interesting guide.
I generally run linux on my NAS, but power consumption is always one of my big priorities for my home NAS. I also shoot for the lowest possible idle power. I don't care if it consumes 250W when I'm using it, only that when I'm not using it it better be just a few watts. That is why i traditionally use Atom (c2000) or ARM hardware. Combined with disk spindown (custom script), turning off everything not in use (extra network ports, etc) My idle power is ~25W. I've only got 12 disks but the big power draw right now is the marvell SATA chips on my motherboard, which don't have any kind of link power saving controls (that I've found yet). I've got another motherboard using a G4400 that idles closer to 12W (total system power), but doesn't have ECC or a BMC..
Anyway, I use a little script which monitors activity against my MD device (yah software RAID, another discussion) and sends full spindown commands to the drives. I've been experimenting with a full drive powerdown, and full port power down (different machine). If I weren't using the NAS as a DHCP/DDNS/etc server I would probably put it into S3 standby and then wake it on CIFS/NFS/RPC inbound.
I'm also using cheap x540 based 10G boards, which add about ~5W a port, but I turn them off on an idle timer and fail-over to a 1G port that I can't measure the power on.
Bottom line, a home NAS device isn't a server that needs to run 24-7. Its not hard to tweak stuff to pull the idle power way down. Given a long enough timer (say 1-2 hours) you will only notice the machine resume/spin up once a day when you initially sit down at your desktop.
To put that in perspective it's roughly $0.60 a day, $18 a month and $219 a year USD assuming a rate of $0.10 per kilowatt hour [1] plus an additional 853 BTU/hr cooling load on the AC system [2].
Rates in California would be double that or more, depending on which tier this put the user. For me, my old 8-cpu Xeon server cost me ~$50/month in electricity so I decommissioned it, however, this would also push me towards wanting to get solar panels for the house.
Did you consider a low power NAS approach? Spending somewhere around $20 per month on electricity for a home server seems like a lot to me. Maybe if it was hosting a handful of VMs for various purposes, but I suppose you don't want to run a NAS as a VM?
Nope, never considered low power. I really didn't want a CPU bottleneck when I move to 10Gb network interface. Running a NAS in a VM (at least FreeNAS) is highly discouraged, so I never considered that. I do, however, run several VMs on the NAS itself, which help to justify the overkill hardware I used.
well there are tradeoffs. If you want low power and high performance, you come down to choosing between either SSDs (very expensive) or a lot of laptop drives (cheap, but higher latency; throughput for something like video is fine though as you can fit a lot of them in a chassis; switching from 8drive Z2 groups to mirrors gets you a 4x improvement in throughput, which completely negates the throughput disadvantage of 5400 vs 7200).
Why not going for less but larger disks then? It would be more power efficient. You have consumer 8TB disks. Surely you don't need that many disks for performance. Pretty much any Synology NAS will saturate a 1GbE link.
Just replied to a similar comment below with: "I have this system sitting next to me in my home office which I share with my wife. Both of us work from home full time and the server is quiet enough that she doesn't complain about it."
One thing to consider is that I'm in the northeast US (philly) where the ambient temperature is generally lower than other places. I've also only had this server running through the colder months (built it in fall), so it may get noisy as the house gets warmer in summer. I would definitely not want this server in my bedroom or living room, but so far, it's been okay in my office.
edit: I should also point out that, regarding the server passing the "wife test", she tends to get hyper focused on whatever task she's working on, so background noise doesn't bother her as much as it does other people. YMMV!
