2

I have a slightly unusual situation, where there's an intermittent low level of corruption of data being sent across my LAN. The problem in tracing it is that it only shows up under heavy parallel load, and it's escaping usual TCP/IP checksum testing. I can't avoid that it's a big job but I can imagine a much better tool that would speed it up a lot.

The use-case and current approach

Right now I'm testing it by generating a 10GB file of random data, setting up a script to copy it 20 times over the wire, then running that script 20 times in parallel. The run takes about 9 hours, and about 1/3 of the time it will report successful data transfer but when the hashes of all 400 copies are checked, I'll find that 398 or 399 copies are correct, but 1 (occasionally 2) copies have been corrupted "in flight" and their hashes at the remote end won't match the hashes of any of the other copies - and this wasn't detected.

There are many things it could be - cable, individual ports, switches, cards, specific machines, software stacks, OS. It could be an interaction of things. It's all solid enterprise gear (Supermicro, Xeon, ECC, Chelsio, managed switches). But even when it's clear I might want to retest it periodically - undetected corruption is a nasty worry to have in the back of ones mind.

I can't really avoid that it will take 2-3 weeks to really be sure which gear can be given an "all clear". But the way I'm doing it right now is incredibly inefficient, and that's what I'm hoping to speed up.

Right now I create a large static file of about 10GB random data on an SSD. I copy it a few hundred times, send it in parallel over one of a number of protocols, and hash each copy off the disks at the far end once it's done. That's insanely bad as an approach. It's very slow (limited by disk or cache access/contention both ends) and the subsequent hashing adds a lot of delay (need to rehash large files but only after completion, to know if there's issues). It also means I can't break it off early if an error occurs near the start, because one won't know there's an error until it's complete and the copies are hashed.

What I'm looking for

So the tool I'm looking for is a bit like a cross between memtest (in that it just runs and one watches for errors) and iperf (in that it's in-memory and client-server) - basically a crossplatform client-server tool that works a bit like iperf, but with the capability to generate and send random chunks of data + their checksums to the remote server not just zeros, and the remote server confirms that each received chunk still checksums and counts the data rate, and number of checksum errors. Then I can just leave it running and stop when it looks like enough, or as soon as the first error is detected.

iperf itself is purely designed to test throughput and jitter, it doesn't check for correct transmission.

My specific criteria for the software I'd like to find

  • Basic functionality: Runs client-server in memory - as a client, it creates chunks of random data of some specific size (not bothered what size, anything from 4K to 1MB would be fine!) along with their checksums, and sends these to a specific IP/port of a server instance as fast as it can. the server checks the chunks it receives do indeed have data matching the checksums, and updates the display with an ongoing total of data + error count.
  • Platform: Needs to be crossplatform, at least with binaries for FreeBSD 11 + any of Windows 7/8/10 x64, because the issue could easily be obscure in some driver or a PCIe card+OS interaction, and those are the main 2 platforms I use that I need to be able to test on (and I don't have a platform to compile anything on, nor have I compiled software yet on these OSes).
  • Parallel activity: Needs to be able to either run multiple client instances, or spawn multiple parallel clients (or client loads) to a single server, to test what happens under congestion/parallel loads (same as iperf -P).
  • Everything else: Beyond that not too bothered. CLI/CMD is probably best. Free/open source preferred but not essential.

Does such a program, or easy way to do this, exist? If not, is there some alternative software I can use to test it?

  • Do you care if it's TCP or UDP? Probably could just generate an MD5 hash of the payload using iperf. It's not supported but should be straight forward. Not sure about other tools available so can't really comment on that. Bob – user34994 Nov 1 '17 at 7:03
  • "It's not supported but should be straight forward" - how is this an answer? – Mawg Nov 1 '17 at 9:10
  • I don't care if its UDP or TCP, except that it lets me confirm TCP errorchecking and both TCP/UDP offloading if it can do both. But I don't yet see how this is an answer, unless it assumes "hack your own modified version of iperf" which I'm not up to doing? Can you clarify. – Stilez Nov 1 '17 at 23:25
  • yes, sorry should have been more clear. It does require modifying iperf. Maybe go here and request it. Also, let's provide some more information on the memory pattern generation. I'll implement it if it's a common enough use case. Seems useful to me as when testing network routers we hit similar issues where intermediate routers were corrupting packet payloads. Note, TCP is a byte protocol. So one would have to provide a byte size in the command line to do the hashing. – rjmcmahon Nov 2 '17 at 18:07
  • "I'll implement it" would certainly be a good solution :) I didn't ever expect to get the maintainer/author of iperf2 to comment - although I probably should have anticipated you might be active on this site and notice the tag. I've done as you suggested (sourceforge.net/p/iperf2/tickets/30). This is certainly an unexpected approach to an answer, but a valid one nonetheless! I hope it's possible - I haven't filled in every detail as I'm hazy on some, but hopefully it's a good start. – Stilez Nov 2 '17 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.