In my team we're now trying to select some Test Case Management System to use. Our requirements are not rocket science, I believe:
- Simple enough to store test cases and execution results. No need for built-in defects and requirements management.
- Some API to be able to import automated test execution results (would be great, if there is Jenkins plugin)
- Due to security, possibility to install on our own hardware aka "on-premise" is required.
- And the most crucial part, I'm becoming really desperate about it since I've already tried almost all "popular" TCMS and they have really nothing here. Custom fields for each test execution. What I mean - often the idea in existing TCMS is "Test plan with test cases - Test run - Test case execution". So you have some entity called "Test run" which is group of tests to execute. If you're lucky enough, there is support for some custom stuff for test run (i.e. environment settings - OS, browser, hardware). So it seems that many teams are happy with creating "Windows Chrome testing", "Linux Firefox testing" test runs and so on. But in our team it's not acceptable, because we prefer to see "product version A testing" and table of test executions with different parameters. So each "row" in the table - test case + environment settings + status + bug links. Obviously, test cases can be duplicated, because we can execute one test case on 5*5*5 (i.e. 5 OS, 5 browsers and 5 versions of external library we're using) different environments. It's not feasible to create 5*5*5 test runs!
Only systems I've seen the last bullet are TestPad (which is mostly just checklist without any smart functionality) and HP-ALM (which is ancient, ugly and slow). Everything else allows you to have only "comment" and "bugs" fields for test execution entity.
So, my questions are:
- Have you met the same issue in your team? How did you make it finally?
- Could you advice me and my team anything that we can use?
P.S. Some of the tools I've tried so far: Zephyr for Jira, HP ALM, Kiwi, qTest, TestCaseLab, practiTest, TestPad