You are correct -- I am in a niche area. I have less than 12 HDDs in a RAIDZ2 configuration. If I had more HDDs in the RAID I would see better write speeds. But I think most home users with TrueNAS ZFS do not have massive arrays, thus my envy of the tiering or true SSD caching.
You are correct that prices of small SSD are near HDD prices, but most people would want larger drives like 12 to 16TB drives in their array, and SSDs can not compete with these on price at all.
Maybe we are in the twilight of HDDs? And SSDs will compete on price across the whole storage capacity range soon? Maybe...
"You are correct -- I am in a niche area. I have less than 12 HDDs in a RAIDZ2 configuration. If I had more HDDs in the RAID I would see better write speeds."
That is incorrect.
RaidZ(x) vdev write speed is equal to the write speed of a single drive in the vdev.
If you want a raidZ zpool to have faster writes, you add a second vdev - that would double your write speed.
A wider vdev, on the other hand (15 or 18 drives, etc.) would not have faster write speeds.
another consideration is that the "cheap" 4TB SSDs are things like quad-level cells, such as the samsung 870 QVO, which are slightly better in a $ per GB ratio, but also suffer in ultimate GB written lifespan for cell endurance wear-out. Thus the reason why the samsung "pro" are almost double the cost.
if you needed to write a LOT (let's say something like a sustained 70MB per second all the time) but the data rate was not particularly high, you might easily wear out a RAIDZ2 composed of 2TB-4TB sized cheap SSDs and kill them in a fairly short period of time, when the same RAIDZ2 composed of 2.5" 15mm height 5TB spinning drives, or 3.5" 14-16TB drives would successfully run for many years.
also re: cheap SSDs, the quad level cell tech and write cache is a big limitation in sustained write speeds:
> But I think most home users with TrueNAS ZFS do not have massive arrays, thus my envy of the tiering or true SSD caching.
I may be heavily influenced by my own situation, but in France, SSDs still are much more expensive than HDDs. I've just bought four IrownWolves, €100 for 4 TB. The cheapest 2 TB SSD I can remember was around €180.
My point is that given the price and the usage profile, I kind of agree with the other poster: I don't have a use for tiered ZFS and would much rather they spent the time on other things.
As it is, I'm already doing a tiered storage of sorts: data that I'm currently working with requiring fast i/o is directly attached to my computer. "Cold" or less i/o sensitive data is plenty fast on spinning drives across the network.
As a home user living in an apartment, this affords another optimization: kick the NAS out and access it over the internet, because gigabit speeds become sufficient.
What larger HDDs would you recommend? I need to upgrade a bunch of 3TB and 5TB drives that have been running for several years but really concerned about failure rates. I used to swear by WD Red however I’m hearing some horror stories about them in recent years.
You are correct that prices of small SSD are near HDD prices, but most people would want larger drives like 12 to 16TB drives in their array, and SSDs can not compete with these on price at all.
Maybe we are in the twilight of HDDs? And SSDs will compete on price across the whole storage capacity range soon? Maybe...