Basically I'm looking for a test framework to write coded system tests which supports one important feature: Continue on failure.

I'm developing ui / system / end-to-end tests for an mobile app with appium. My test cases are covering real use cases and are having many actions and comparisons. For example the whole process of an order at an online shop. e.g.

login > put product in the chart > check product details in chart > go to checkout > select payment details > check payment details > go to summary > check all details > execute the order > check if order is executed

Doing all this stuff like a real user. I'm mostly interested to see if the order is executed, failed comparisons should be collected listed afterwards.

My problem with mstest/nunit/xunit/junit/jasmine is that they stop a test case at the first failing assert. I want an test case to continue as long as possible (executing all actions), one failed assert out of maybe 100 should not stop it.

TestNG does this with Verify. Noticing the failed assertion but continuing. Really "fail" the test case at the end and then show a list of failed assertions. Fully developed system test software, like Ranorex, Tricentis, QFTest, SoapUI, are doing this too.

Beside TestNG i haven't any seen another test framework which does this out of the box. Nunit has Assert.Multiple(), but this is to error prone to me and against DRY. I have to use it in every test case again. Looks like this is more for edges at unit test. Actually i developing an wrapper around all Nunit's assertion methods, which are catching the AssertionExceptions.

Anybody knows a framework which does 'continue on error' out of the box? Clear what I want?

Mostly I'm looking für .NET/C#, but other languages are welcome too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.