I have a slightly unusual situation, where there's an intermittent low level of corruption of data being sent across my LAN. The problem in tracing it is that it only shows up under heavy parallel load, and it's escaping usual TCP/IP checksum testing. I can't avoid that it's a big job but I can imagine a much better tool that would speed it up a lot.
The use-case and current approach
Right now I'm testing it by generating a 10GB file of random data, setting up a script to copy it 20 times over the wire, then running that script 20 times in parallel. The run takes about 9 hours, and about 1/3 of the time it will report successful data transfer but when the hashes of all 400 copies are checked, I'll find that 398 or 399 copies are correct, but 1 (occasionally 2) copies have been corrupted "in flight" and their hashes at the remote end won't match the hashes of any of the other copies - and this wasn't detected.
There are many things it could be - cable, individual ports, switches, cards, specific machines, software stacks, OS. It could be an interaction of things. It's all solid enterprise gear (Supermicro, Xeon, ECC, Chelsio, managed switches). But even when it's clear I might want to retest it periodically - undetected corruption is a nasty worry to have in the back of ones mind.
I can't really avoid that it will take 2-3 weeks to really be sure which gear can be given an "all clear". But the way I'm doing it right now is incredibly inefficient, and that's what I'm hoping to speed up.
Right now I create a large static file of about 10GB random data on an SSD. I copy it a few hundred times, send it in parallel over one of a number of protocols, and hash each copy off the disks at the far end once it's done. That's insanely bad as an approach. It's very slow (limited by disk or cache access/contention both ends) and the subsequent hashing adds a lot of delay (need to rehash large files but only after completion, to know if there's issues). It also means I can't break it off early if an error occurs near the start, because one won't know there's an error until it's complete and the copies are hashed.
What I'm looking for
So the tool I'm looking for is a bit like a cross between
memtest (in that it just runs and one watches for errors) and
iperf (in that it's in-memory and client-server) - basically a crossplatform client-server tool that works a bit like iperf, but with the capability to generate and send random chunks of data + their checksums to the remote server not just zeros, and the remote server confirms that each received chunk still checksums and counts the data rate, and number of checksum errors. Then I can just leave it running and stop when it looks like enough, or as soon as the first error is detected.
iperf itself is purely designed to test throughput and jitter, it doesn't check for correct transmission.
My specific criteria for the software I'd like to find
- Basic functionality: Runs client-server in memory - as a client, it creates chunks of random data of some specific size (not bothered what size, anything from 4K to 1MB would be fine!) along with their checksums, and sends these to a specific IP/port of a server instance as fast as it can. the server checks the chunks it receives do indeed have data matching the checksums, and updates the display with an ongoing total of data + error count.
- Platform: Needs to be crossplatform, at least with binaries for FreeBSD 11 + any of Windows 7/8/10 x64, because the issue could easily be obscure in some driver or a PCIe card+OS interaction, and those are the main 2 platforms I use that I need to be able to test on (and I don't have a platform to compile anything on, nor have I compiled software yet on these OSes).
- Parallel activity: Needs to be able to either run multiple client instances, or spawn multiple parallel clients (or client loads) to a single server, to test what happens under congestion/parallel loads (same as
- Everything else: Beyond that not too bothered. CLI/CMD is probably best. Free/open source preferred but not essential.
Does such a program, or easy way to do this, exist? If not, is there some alternative software I can use to test it?