11

Suppose I've written an app which may be invoked as follows:

my_magic_app --foo=yes --bar=5003 --baz=best_baz_ever

Now suppose I want to test it by having bar range from some number to another, foo be either yes or no, and baz either be all strings upto a certain length or strings from a certain dictionary which I have in a file. Also, I don't quite want all combinations but only subject to a certain condition (e.g. bar can't be over 5000 if foo is negative).

I could write a script to do this generation, applying the condition etc. - but I was wondering whether there's something like that already.

Requirements:

  • For Unix-like operating systems
  • Free license
  • Gratis
  • I don't really care whether it's a script or a binary
  • Non-arcane languages preferred (e.g. bash, python, perl)
Improve this question
4
  • 1
    I've always ended up doing this with ad-hoc scripting, so it's an interesting question to me. Couple of quick notes: 1) be cautious about the size of the testing space - it's surprisingly easy to get an ungovernable number of tests; 2) in general you'll find it's better to specify ranges yourself (foo in yes no); 3) expected results are hard to generate, especially with conditions like the one you give (you don't want to go from having one problem to having two).
    – CPerkins
    Commented Apr 20, 2016 at 12:17
  • 1. Yes, I know. Painful point 3. I have code for generating expected results in my case (and sometimes there are no expected results except not crashing, or the results are always the same) but, yes, in general this is an issue.
    – einpoklum
    Commented Apr 20, 2016 at 12:21
  • This can be done with a list comprehension.
    – dramzy
    Commented Apr 20, 2016 at 15:02
  • @dramzy: I'm looking for existing software, not an approach for implementing this functionality...
    – einpoklum
    Commented Sep 12, 2016 at 10:19

3 Answers 3

Reset to default
1

The category partition method describes this kind of constraint-based test generation. TSLGenerator is a utility that can generate tests for you, given constraints. The output is in the form of a test spec; they're straightforward to read, if you'd like to roll your own way of parsing them into unit test stubs or commands.

The method and software are described in this module from a GATech graduate course.

Improve this answer
0

You might consider a combination of all pairs testing and boundary testing. All pairs testing is a method of reducing the number of test vectors down to those that exercise all pairs of combinations. It can greatly reduce the number of test cases, while not appreciably reducing the test coverage. You will obviously have to satisfy your testing requirements, but this really is a great method when you can use it. When testing ranges of numbers it is usually only really interesting to check the boundaries. For example, if an input can be 0-5000 then the interesting test cases are going to be -1, 0, 1, 4999, 5000, and 5001.

What I've done in the past for testing is put together a script that uses all pairs for parameters that have distinct choices, and boundary testing for numerical parameters.

I don't think you'll be able to avoid writing some sort of script, but you certainly can use tools that other people have developed. James Bach, a "famous" software tester, has written an all pairs tool you could use.

Improve this answer
5
  • 2
    While this suggestion may (or may not) be relevant to how I perform testing, it is still not what I asked for... I need to test the whole space. A grid covering it partially might be relevant, but not the extrema. Still, +1 for the effort.
    – einpoklum
    Commented Jul 14, 2016 at 7:51
  • The asker clearly stated they need to test ALL combinations. No tool can automatically guess what combination is not worth trying, especially without knowing the internals of the tested program. The tool you suggest will "create tests that pair each value of each of the variables with each value of each other variable at least once", which is not OK.
    – Nicolas Raoul ♦
    Commented Jul 15, 2016 at 10:48
  • In my experience as a software tester I find that all pairs is a useful tool to provide a high degree of coverage, while reducing the number of test cases. The technique is well known in the software testing world and I've heard it talked about at conferences and in books. Yet, most of the testers I've worked with don't know about it because they were thrown into the role. I offered it as a suggestion because it is a viable alternative in many cases that might not have been considered.
    – isaac
    Commented Jul 15, 2016 at 11:56
  • Again - fair enough, your experience is perfectly valid.
    – einpoklum
    Commented Jul 15, 2016 at 14:02
  • Reducing the number of test cases is usually done under the expectation that the software developers implement the code only once. I have observed too many times that code was copy/pasted, resulting in failures which I didn't expect, because I have tested that already. Since that time I happily test more combinations than usually needed. (I work in QA and I disagree the ISTQB in this case) This applies to black-box-testing
    – Thomas Weller
    Commented Aug 14, 2016 at 8:41
0

Check out Hypothesis

Python tool that does pretty much what you ask for, plus more: if it finds a bug, it tries to refine the input to a minimal test case to reproduce the bug.

Improve this answer
1
  • Please give us a little more than just a link... what kind of tool is it? What's a typical invocation? What do you mean by "finding a bug"? Is it a meta-invoker of other things?
    – einpoklum
    Commented Jul 22 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.