We have ~22.000 unit tests. Now, two of them fail when run together with other tests. They pass when run alone. I am now looking for an automated analysis to narrow down the tests to reproduce the problem.

I think it should be possible to find a solution with a divide and conquer approach. Run some unit tests, then run the affected one. Figure out whether it fails or passes. Restarting the unit test process should hopefully eliminate the cause, assuming that it's not file or database related.

These are the requirements:

  • Supports .NET / C#
  • Supports MSTest and NUnit tests
  • works on Windows 7 and 10
  • is gratis for commercial use
  • runs offline, i.e. on a developer machine, not in the cloud
  • Divide and conquer may work. It may not. Imagine test T1 fails when T77 and T952 are run in order in a set of T2-T1000. A divide and conquer scheme may divide T77 into a group T2-T500 and and place T952 in a group T501-T999... and neither subgroup when executed by itself will show the failure. In theory, you may have to identify an arbitrary subset. For N tests, that's 2^N possible subsets; for 1000 tests, that's 2^1000 which is way larger than the number of protons in the visible universe. – Ira Baxter Jun 17 '17 at 4:34
  • .... now, you can still try a divide a conquer scheme, and hope you don't have difficult dependencies because your tests are supposed to be independent. – Ira Baxter Jun 17 '17 at 4:35
  • @IraBaxter: I don't need all protons at the same time. I can reuse electrons over time. – Thomas Weller Jun 17 '17 at 7:59

My team and I use Typemock's Isolator. It supports .Net framework, MSTest, Nunit and Windows.You can run it on a developer machine as well. We use the paid version, that I believe would be able to help you. There is a free version, though I'm not sure it can satisfy your needs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.