I am setting up a networked system for acquiring a large number of streamed camera images (up to 3GB/s of raw tif files; up to 2TB per experimental acquisition). The host that acquires the data will be streaming the raw images to 4 x 2TB Nvme in raid0.
Images on the host will subsequently be mirrored to a slave server (via direct connect 25Gig NIC) where they will be batched (e.g. 100+ tif files) and compressed before being slowly synchronized over a 1Gbps connection to a secure/backed up file server in another location. Thus the slave will buffer all the data coming in freeing up the host for additional rounds of data acquisition.
The slave is an older Xeon Westmere system, so the I/O is a bit limited. I have remedied it in part by adding a PCIe 2.0 SATA 3.0 4-channel interface and 4 x 1 TB SATA 3.0 SSDs. In general, there's not much need for parity/mirroring. I simply want to maximize the rate at which this system can offload data from the host so that there's less downtime between data acquisition (e.g. the nvme raid is filled up).
From what research I have done, it seems ZFS might handle this scenario quite well, offering potentially better sequential write performance than a simple RAID0 and automatic compression to maximize capacity.
However, the overall state of ZFS is rather confusing. Right now it seems like ZoL is a convenient solution for most common Linux distros, but that it still lags the performance of implementations found in FreeBSD and Illumos-based systems (e.g. OmniOS / Open Indiana) as indicated from some benchmarks on Phoronix.
Is there a general current consensus about this? Which current ZFS implementation (and thus OS) would likely give the best sequential write speed in my use case? Or should I be looking at another file system altogether?
Again, the slave server only needs to maximize offloading data given the disk/IO hardware constraints. There's 6 cores and 48GB of ram that it can dedicate to this task, and parity is not an issue. (Losing a single experiment of data isn't a huge loss)
