10

I would like to hear what is a recommended C++ unit testing framework or, in case there is not a single one, what is a generally accepted flowchart that can lead me while picking one.

closed as too broad by Braiam, Tom, Jan Doggen, ᔕᖺᘎᕊ, unor Jan 9 '16 at 20:56

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

9

Unit testing framework comparison criteria

You should consider the following important criteria for comparing C++ unit testing frameworks:

  1. Compatibility: whether the framework compiles with your project. You may be using options like -fno-rtti and some frameworks then fail to compile.
  2. User friendliness: how much work is it to write a test. C++ does not have reflection, so frameworks have to solve test registration somehow.
  3. Features: can it do what you need it to do? According to a post on Google Testing Blog, most frameworks have the necessary features and differ mostly on points 1 and 2.
  4. Popularity & Familiarity: whether people know the framework. This is important especially for free software, not to limit potential contributor base.

Popular frameworks and their popularity measures

To measure popularity, it might be enough to search Stack Overflow and GitHub. The following statistics were gathered in January 2016 and may well change drastically over time; also, trends, not presented here, may be more important than absolute numbers.

Stackoverflow-popular frameworks

A search for [unit-testing] [c++] finds 1,211 questions on SO. Searching for individual frameworks, we find:

  • Google Test: 857 questions (keyword term [googletest])
  • CppUnit: 203 questions (keyword term [cppunit])
  • Catch: 12 questions (keyword term [catch-unit-test])

GitHub-popular frameworks

On GitHub there is about 461,000 repositories for query "language:C++".

GitHub:

  • Google Test: 1,870,016 C++ files (search term: "include gtest.h")
  • CppUnit: 297,525 C++ files (search term: "include cppunit")
  • Catch: 11,888 C++ files (search term: "include catch.hpp"(
  • CppUnitLite: 1,214 C++ files (search term: "include CppUnitLite")

(the number in "We’ve found xxx code results" after clicking C++ in the left menu)


It would seem that Google Test is a clear winner of the SO+GitHub popularity contest, most likely because it was adopted by large projects like LLVM and Chromium.

4

I would like to supplement the community wiki answer with a couple of other points to consider when selecting your testing framework, this is addressing your picking a test tool:

  1. Price - Free or reasonably priced testing frameworks such as Google Test and CppUnit will almost always be ahead on popularity but may not meet your other requirements.
  2. Level of test coverage required:
    1. If you are testing some user code for a simple free software project, (although many such actually have much better testing than commercial projects), or game then you are ahead of the curve if you are performing any unit testing.
    2. If you are writing code for a financial application or a telecommunications system you are likely to require, or be required, to perform 100% lines of code coverage, i.e. every line of your code should be executed at least once during your testing.
    3. If you are writing a low criticality space, military, medical or aviation application you will be required to perform 100% line coverage and 100% decision coverage - i.e. every line gets executed and every way of going through a decision point is exercised.
    4. If you are writing safety critical code in any of the areas mentioned above you will be required to perform 100% code and 100% decision coverage at the op-code level not just line coverage.
  3. Any required certification of your test tool or any standards that it must test against - it is not uncommon in the embedded software industry to require MISRA compliance but there are other standards that are required in some cases.
  4. The typical amount of time it takes to produce each test case - this can differ enormously between tools and has a big impact on indirect costs.
  5. Whether the test tool is available for your deployment platform & tool chain and the time & cost of setting up for it.
  6. Whether the tool requires you to modify your code in any way for running the tests - in some of the above industries you testing is required to be performed on unmodified source code.
  7. The suitability for automation of the testing - if you are planning on running continuous integration or using extreme/agile practices then being able to automate your testing and test reporting is a huge asset.
  8. Longevity - if your project is running for years or your product being supported for years then losing your test tool is a disaster.
  9. The ability to version control your test scripts is vital.
  10. Ditto the test reports/results - you also need the ability to publish your test results in a format where they are accessible without a copy of the test tool.
  11. You also need your tests and results to be traceable both to the requirements and to the source code.

If you are doing unit testing for embedded, safety critical software, then there are very few tools with the required standards support and certifications. Rational Test RealTime & LDRA Testbed are examples - but they are not cheap, running to tens of thousands of dollars for each licence per year per developer.

Disclaimer - I do not work for any of the companies mentioned above but I have used their products as I have worked extensively in the safety critical, embedded, real-time industry.

Not the answer you're looking for? Browse other questions tagged or ask your own question.