1

I'm starting a scientific computing, command line project using source code that has no automated testing. The previous developer compared results by hand. Before I make changes, I want to set up some testing. What I think would work best would be a regression testing tool. What I have in mind is a tool which:

  1. Is configured with a list of regular expressions representing lines to ignore. E.g., lines with absolute file paths, timing statements, output which includes system information (e.g. a user name, machine name), etc.

  2. Accepts a list of example command lines w/ input file(s). Commands would probably be run in separate directories.

  3. Runs each test (possibly on a cluster) and saves the output file with an extension indicating it's temporary, e.g.: ".out.test".

  4. Diffs the test output file with a saved output file (i.e., ".out"). It would also be useful to record how long the test took for simple performance testing (each test may take several minutes or more). It would also be nice if it broke lines apart to look for floating point numbers and provided an "almost equal" criteria for comparisons.

  5. Reports the diffs to the user. Ideally with colored diffs.

  6. I may accept the changes, which will result in replacing all the ".out" files with corresponding ".out.test" files.

For former employers, I've hand written tools that have done things like this. This time, I'm looking for a 3rd party tool first. I did some googling, but haven't found anything.

Going forward, as I understand what's there and/or write new code, I'll add unit tests.

Here's someone else asking the same question who ended up rolling their own using python.

Here's another solution I found using cmake. cmake has a tool called compare_files which can be used to compare an output file with a reference file as a part of its ctest functionality. cmake is a good choice for me since I'm already using it for this project. However, its compare_files seems to be too simplistic for my needs. I'm looking into whether I can use diff instead of compare_files within ctest.

Improve this question
  • I'm working in software testing (test automation), previously worked in software development. As it stands, I have no idea what you're talking about. What is a command line project? Which programming language? What is a "result"? In which format do you have results? Why do you want regular expressions? What is the problem you're trying to solve with it? What is a "line" that you want to ignore? A line of what? Line of source, output, result or input? On one side, the question is very specific and you have detailed requirements. On the other hand it is too broad to be answered. – Thomas Weller Nov 10 '15 at 15:16
  • It's C++ code, but that's irrelevant since I want the testing to be oriented around its command line interface. Its input includes text files (in this case, lists of molecules). The output is a text file representing transformed molecules. At its simplest, my test framework could just be a diff (if diff is empty, the test passes) of two output files. – amos Nov 10 '15 at 15:23
  • You might be looking for developing solutions based on expect and its libraries, though I'm not familiar with it enough to say for certain. – user450 Nov 11 '15 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.