A modern-C++-oriented testing framework for C++ code

Question

I've read

C++ unit testing framework

which is closed as too broad; and answer-wise, while it lists several testing frameworks it doesn't really compare them or discuss their features.

Now, I'm interested in adding a framework for unit tests for an existing project of mine. It actually already includes a binary which performs a certain kind of unit tests, or perhaps I should something between unit and subsystem tests - but its' not a unit testing framework in itself and not intended to test all of the code. Which is why I want a proper framework. It may (or may not) also allow me to drop some of the custom code I have right now.

A significant requirement that I have is that the framework be Modern-C++-oriented. That is, that the syntax for using it will be C++11'ish (could be C++14 or C++17, but the latter option might give me some trouble due to CUDA compatibility issues), rather than just supporting the testing of C++11 code as an afterthought. So when I read, for example, that Google Test only needs a C++98 compiler, I get worried.

Other key features (partly adapted from this article):

• Minimal amount of work needed to add new tests.
• Easy to modify and port - but not for the reasons mentioned here! That is, it can and should depend on advanced C++ language and standard library features, as long as they're standard; and I don't mind dependencies on something like, say, CMake, or some scripting language etc, as long as it's tolerant to the difference between OSes and OS distributions.
• Supports setup/teardown steps (fixtures).
• Flexible and robust w.r.t. exceptions, crashes and asserts.
• Nice pretty-printed/ASCII-graphics-enhanced console output. I don't care for any specific such features, e.g. colors vs monochrome or whether or not progress and succeses/failures are animated, but it should be pleasingly readable.
• Integration with IDEs is a big plus

and of course:

• Free and Open Source license
• Gratis
• Support for different output formats, for use by various post-mortem tools

I would really like a comparison of strengths and weaknesses w.r.t. the features I listed as well as a comparison on a single framework.

Answer 1

If you have Boost as a dependency, then Boost.Test is a no-brainer choice. Consistent, feature-complete, as minimalistic or as rich as you need it to be.

Answer 2

Mettle

A C++14-and-up framework!

• Main website: https://jimporter.github.io/mettle
• Framework code: https://github.com/jimporter/mettle

Example:

suite<> basic("a basic suite", [](auto &_) {
  _.test("a test", []() {
    expect(true, equal_to(true));
  });

  for(int i = 0; i < 4; i++) {
    _.test("test number " + std::to_string(i), [i]() {
      expect(i % 2, less(2));
    });
  }

  subsuite<>(_, "a subsuite", [](auto &_) {
    _.test("a sub-test", []() {
      expect(true, equal_to(true));
    });
  });
});

Note: I've not tried it myself.

Answer 3

lest

• Stated design goals: Modern and C++11-native, single-file, header-only, small size
• Framework code: https://github.com/martinmoene/lest
• Missing: Suites of tests, parameterised tests, templated tests, test data generators, built-in hamcrest matchers.

Example:

const lest::test specification[] =
{
    CASE( "Empty string has length zero (succeed)" )
    {
        EXPECT( 0 == string(  ).length() );
        EXPECT( 0 == string("").length() );    
   },
}

Note: I haven't tried it myself.

Answer 4

Bandit

"A human-friendly unit testing for C++11"

• Main website: http://banditcpp.org
• Framework code: https://github.com/banditcpp/bandit

Apparently, this framework "bends over backwards" to make your test's source code read like a natural-language description of a test. For example, here is a test of a guitar, which makes sure it sounds distorted in distortion mode:

describe("in distorted mode", [&]() {
    before_each([&]() { fuzzbox->flip(); });

    it("sounds distorted", [&]() {
        AssertThat(guitar->sound(), Equals(sounds::distorted));
    });
});

Note: I haven't tried this one myself.

Answer 5

DocTest

Presented in CppCon 2017 by Victor Kirilov

This actively-developed framework is not actually modern-C++-targeted, but it's aiming to be with its next major release (2.0; it's at 1.2 right now).

• Motto: "The fastest feature-rich C++98/C++11 single-header testing framework for unit tests and TDD"
• Framework code: https://github.com/onqtam/doctest
• Ambitious roadmap, including IDE integrations
• Purportedly very fast - with [benchmarks] against the Catch framework (but I have no idea if Catch is faster than Gtest, CppUnit etc.)

Note: I haven't tried this myself.