2

We have ~22.000 unit tests. Now, two of them fail when run together with other tests. They pass when run alone. I am now looking for an automated analysis to narrow down the tests to reproduce the problem.

I think it should be possible to find a solution with a divide and conquer approach. Run some unit tests, then run the affected one. Figure out whether it fails or passes. Restarting the unit test process should hopefully eliminate the cause, assuming that it's not file or database related.

These are the requirements:

  • Supports .NET / C#
  • Supports MSTest and NUnit tests
  • works on Windows 7 and 10
  • is gratis for commercial use
  • runs offline, i.e. on a developer machine, not in the cloud
Improve this question
3
  • Divide and conquer may work. It may not. Imagine test T1 fails when T77 and T952 are run in order in a set of T2-T1000. A divide and conquer scheme may divide T77 into a group T2-T500 and and place T952 in a group T501-T999... and neither subgroup when executed by itself will show the failure. In theory, you may have to identify an arbitrary subset. For N tests, that's 2^N possible subsets; for 1000 tests, that's 2^1000 which is way larger than the number of protons in the visible universe.
    – Ira Baxter
    Commented Jun 17, 2017 at 4:34
  • .... now, you can still try a divide a conquer scheme, and hope you don't have difficult dependencies because your tests are supposed to be independent.
    – Ira Baxter
    Commented Jun 17, 2017 at 4:35
  • @IraBaxter: I don't need all protons at the same time. I can reuse electrons over time.
    – Thomas Weller
    Commented Jun 17, 2017 at 7:59

1 Answer 1

Reset to default
1

My team and I use Typemock's Isolator. It supports .Net framework, MSTest, Nunit and Windows.You can run it on a developer machine as well. We use the paid version, that I believe would be able to help you. There is a free version, though I'm not sure it can satisfy your needs.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.