2

This definitely already exists and is open source, I just forgot the name: It was a software testing tool (probably for Python but could've been for another language) that introduced random modifications to your code so you could see which parts were actually covered by your tests - the idea being that significantly changing a line should make at least one test fail, otherwise you can't say that that line is really covered, even if it is executed during a test run. It also had some clever ways of determining what to change in which ways because obviously this type of testing is a lot more computation-intensive than regular testing, and even more so if you "brute force" it.

Does anyone remember the name?

2 Answers 2

1

Nevermind, I found it: https://pypi.org/project/MutPy/

And the general concept is called mutation testing, apparently.

0

This is NOT an answer. It's just too long to be a comment.

I only just saw this, and my immediate reaction was "Ick!".

Then I thought about it for a while, and changed my reaction to "aaaaargh!!".

Seriously, if you want to test code coverage, use a code coverage tool. Take a look, for instance, at https://coverage.readthedocs.io/en/7.4.2/ and look at coverage html for nice visuals, as per https://stackoverflow.com/questions/26664977/visual-report-for-code-coverage-in-django

By default, you get text output, like this

$ coverage report -m Name                      Stmts   Miss  Cover  Missing   
-------------------------------------------------------  
my_program.py                                    20      4    80%   33-35, 39  
my_other_module.py                               56      6    89%   17-23  
-------------------------------------------------------  
 TOTAL                                            76     10    87%

But, as shown in https://hub-courses.pages.pasteur.fr/python_one_week_4_biologists/From_script_to_project.html#tests-coverage), that package can also be used to generate more visual, HTML, output

enter image description here


That crazy "modify the code *a little" approach might work for interpreted languages like Python, but what about compiled languages? Is it going to compile and link every time? Or TypeScript? Will it tweak your TS and regenerate the JavaScript from it, or will it just tweak the JS?

And, (words fail me) - how much is "a little", and how does it know how to make intelligent changes???!!

No. No! NO!! NO !!!!

Use code coverage. Coverage.py for Python, Lcov/Gcov for C/C++, and so on. There are also lots of plugins for Visual Studio code.

To channel Nancy Reagan, "just say 'no' to mutation testing".

<rant>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.