Software Recommendations Stack Exchange is a question and answer site for people seeking specific software recommendations. It only takes a minute to sign up.
Sign up to join this community
I'm working on a large, not-so-old codebase with ton of unit test.
The code coverage is somewhat great, but I've noticed that there are many tests with no assert at all.
Code changed and some tests' asserts were just commented. Other tests were marked with Ignored() or they were totally commented.
Any tools that can scan a codebase to find such issues?
It doesn't have to be specific to test.
Some tricks that can be used to identify items in a large codebase that actually do nothing is to use a code documentation tool like doxygen with options to display the call graph and caller graph and look for all the items that have either no calls, (not the actual test is commented out), or no callers, (the call to the test is commented out), looking at the .dot files with a script can do this - this will at least find those tests that are removed or have all the content commented out and some disabled with compiler switches.
The next trick is to use a code coverage tool, like gcov, on the actual tests - build with it specified - run all the tests - then parse the coverage file for lines or clusters of lines with 0 execution counts. This will identify those tests that, for one reason or another, do little or nothing, (conditional compiler flags, early return(PASS), etc.).
The 3rd is to look at the test assessment code to determine all the ignore mechanisms, (be careful sometimes there are several), such as ignore flags, result contribution of 0, etc. and search for instances of them.
Possibly the easiest is to come up with an estimate of the number of test results that you expect and comparing it to the number that you get.
The other thing to look out for is retries - in most cases a test should simply fail or pass or if run multiple times each run should contribute to the result - code that, under tests, has to be given several chances to pass might, in the field, either never work, sometimes work, or sometimes take a long time to work - these are a nightmare to find later.
There are some static analyzers that can spot unreachable code but most of the good ones that I have come across are expensive and tend to only be used in the safety critical field.
