I'm starting a scientific computing, command line project using source code that has no automated testing. The previous developer compared results by hand. Before I make changes, I want to set up some testing. What I think would work best would be a regression testing tool. What I have in mind is a tool which:
Is configured with a list of regular expressions representing lines to ignore. E.g., lines with absolute file paths, timing statements, output which includes system information (e.g. a user name, machine name), etc.
Accepts a list of example command lines w/ input file(s). Commands would probably be run in separate directories.
Runs each test (possibly on a cluster) and saves the output file with an extension indicating it's temporary, e.g.: ".out.test".
Diffs the test output file with a saved output file (i.e., ".out"). It would also be useful to record how long the test took for simple performance testing (each test may take several minutes or more). It would also be nice if it broke lines apart to look for floating point numbers and provided an "almost equal" criteria for comparisons.
Reports the diffs to the user. Ideally with colored diffs.
I may accept the changes, which will result in replacing all the ".out" files with corresponding ".out.test" files.
For former employers, I've hand written tools that have done things like this. This time, I'm looking for a 3rd party tool first. I did some googling, but haven't found anything.
Going forward, as I understand what's there and/or write new code, I'll add unit tests.
Here's someone else asking the same question who ended up rolling their own using python.
Here's another solution I found using cmake
. cmake
has a tool called compare_files
which can be used to compare an output file with a reference file as a part of its ctest
functionality. cmake
is a good choice for me since I'm already using it for this project. However, its compare_files
seems to be too simplistic for my needs. I'm looking into whether I can use diff
instead of compare_files
within ctest
.