For our software, we have automated tests that calculate a bunch of variables, and then pass/fail (using a framework like JUnit, NUnit, etc) depending on how far the results are from an average in the baseline (a previous release).
Among these "calculation" tests are also "performance" tests, where the variables being compared are measured times for operations to complete.
We had a very similar set of tests at my last company, which was in a completely different industry.
Both times, I've found that the tests are dogged by problems and require a bunch of manual work:
- Averaging doesn't work well: e.g. with (acceptable) outliers, AKA "spikes", in data, and (assuming normal distribution) large variation in baseline
- Floating-point comparison doesn't work well with small numbers
- Reporting framework does not provide mechanism for manually changing pass/fail verdict
- Simple pass/fail verdicts hide valuable data, and often custom, infrastructure required for reporting the "extra" data
What I want
Isn't there something that is designed for this type of "calculation" testing? Specifically, I'd like:
- Automatically takes care of comparisons, yet does not throw away data for a simple pass/fail verdict
- Option of different types of stastical comparison algorithms, e.g. simple average, T-Test
- Big bonus if it came with its own database / reporting system, or at least plugged into an existing one (e.g. Microsoft Reporting Services).