Simple test case management tool with test execution details

Question

In my team we're now trying to select some Test Case Management System to use. Our requirements are not rocket science, I believe:

• Simple enough to store test cases and execution results. No need for built-in defects and requirements management.
• Some API to be able to import automated test execution results (would be great, if there is Jenkins plugin)
• Due to security, possibility to install on our own hardware aka "on-premise" is required.
• And the most crucial part, I'm becoming really desperate about it since I've already tried almost all "popular" TCMS and they have really nothing here. Custom fields for each test execution. What I mean - often the idea in existing TCMS is "Test plan with test cases - Test run - Test case execution". So you have some entity called "Test run" which is group of tests to execute. If you're lucky enough, there is support for some custom stuff for test run (i.e. environment settings - OS, browser, hardware). So it seems that many teams are happy with creating "Windows Chrome testing", "Linux Firefox testing" test runs and so on. But in our team it's not acceptable, because we prefer to see "product version A testing" and table of test executions with different parameters. So each "row" in the table - test case + environment settings + status + bug links. Obviously, test cases can be duplicated, because we can execute one test case on 5*5*5 (i.e. 5 OS, 5 browsers and 5 versions of external library we're using) different environments. It's not feasible to create 5*5*5 test runs!

See screenshot with some Excel stuff, probably it will explain easier than my long text:

Only systems I've seen the last bullet are TestPad (which is mostly just checklist without any smart functionality) and HP-ALM (which is ancient, ugly and slow). Everything else allows you to have only "comment" and "bugs" fields for test execution entity.

So, my questions are:

1. Have you met the same issue in your team? How did you make it finally?
2. Could you advice me and my team anything that we can use?

P.S. Some of the tools I've tried so far: Zephyr for Jira, HP ALM, Kiwi, qTest, TestCaseLab, practiTest, TestPad

Answer 1

I'd like to suggest TestLodge which is a tool that I've helped develop.

In terms of your requirements:

1. The tool has been designed to be simple and easy to use. We don't have a large amount of features like other older tools, but just the essentials.
2. Our REST API can be used for automated integrations, although we don't have any pre-built integration with Jenkins. (We are more focused at manual testing.)
3. We provide the ability to create test configurations which is where you would define your browsers / operating systems. All test cases that you select to run would then be included multiple times in the same run for each selected configuration. We have a quick 3 minute demo video and the second part shows how to create a run and define these configurations. So in short you would end up with one run, but with all the different environments configuration within it.

I would also recommend taking a look at our issue tracker integrations which would allow you to create a ticket automagically in a defect management tool whenever a test fails. Here's another video showing how the more recent integrations work. This can be a huge time saver for the tester!

Answer 2

Back in 2017 we checked existing solutions and ended up with using Kiwi TCMS which is free Open Source Django app based on old Nitrate solution.

The main advantages we liked:

• Free Python/Django app with open source, so we can extend and customize according to our internal needs.
• Recent activities on Github (this was end of 2017, today, 1.5 years later we see that it's being developed really fast and makes a lot of changes)
• API. Even though XML-RPC is not the fancient one, it still serves our needs and allows to do both uploads and getting results via Python scripts.
• Extensible interfaces for new bug trackers support (we improved Jira integration a lot), authorization backends (we use ldap and Kerberos)
• Support of environment. Even though it still has "test runs" as everywhere else, there is possibility to set some variables per test run. We use it to indicate third-party library versions, OS, hardware that we use and process it afterwards in automatic scripts.

UPD as of fall 2018, upstream has removed all environment functionality in favor of "tags" which is much less suitable for us, so we stick to mid-2018 upstream version.

---

After 1.5 years of using Kiwi, there are some downsides as well:

• Upstream develops really fast. From the one hand, it's really good and indicates that project is "alive", however it's complicated to stay on track especially when our changes started to appear in many different places of the code (my company's policy makes it difficult to upstream internal changes back to public GitHub). It was a bit painful to rebase over updated upstream where they made huge refactoring all over the place. I tried to keep on track, but after mid-2018 when they removed environment functionality I stopped to follow.
• Only a few of my reported issues were fixed, some of them, which are critical from our point of view still open (we fixed it by ourselves in our instance)
• UI is a bit old-school, but still ok (again, in current master it was improved a lot)
• Some code, especially JS was really legacy and not really intuitive to follow (again, this was improved in master)

So I'd really recommend one to give it a try if you're selecting test management system

Answer 3

So, I work for a company that has its own test management tool called "Kualitee". We have other companies that are using the tool such as Emirates, Cox Enterprises, T-Mobile, etc. What you mentioned is something we also go through while providing services to our clientele but luckily we have come a long way.

So Kualitee is a cloud-based tool that supports both Manual and Automated testing. You can manage your requirements, test cases, executions and issue tracking all under the same hood. It has integrations with JIRA, Selenium, Jenkins, and Bitbucket. The APIs are also available for all the modules of the tool for any kind of custom integration.

We also do all our reporting within Kualitee where you are able to generate custom test, bug and execution reports (with associated defects etc). You get complete traceability of your test cases and defects to the underlying requirement and to the build.

The tool itself is a simpler version of HP ALM.

Hope this has been helpful for you guys to give it a try. It has a freemium model as well.