I'm looking for a tool/combination of tools which would allow me to do system testing for application spanning multiple hosts.
What I'm looking for:
- ability to spawn variable number of application instances over given set of hosts; given test run should alway run on same hosts, as they are of different specs and part of testing is to check for differences between them
- some way of interacting/tracing spawned instances, so tool is able to forcefully kill them on timeout (in case, due to bug, they go into infinite loop or something)
- being able to check test run for correctness and report failure otherwise (either comparing stdout/logs to specific template or even just running script provided by me which will return 0/1 on exit)
- ability to gather per-test measurements (provided by me through stdout/special API/whatever) which can be compared to previous runs (to spot performance trend/degradation by eyeballing it, as opposed to passing hardcoded limits)
Nice to have
- Some integration with jenkins/hudson, so it can be fired after successful build, but it is probably trivial to do manually
- Some OS awarness, so things like max memory, total cpu used, network traffic etc could be automatically gathered
I'm not looking for any kind of web integration (http, REST etc). Application in question is continously running and problem is not with measuring request/reply times for well defined calls. I'm also not looking for unit testing (this is already solved and out of scope here) or anything spawning tests in-process.
Application is written in java (but it should not really matter here), hosts are unix based (linux and solaris).
Currently, I'm considering rolling it on my own, using combination of jenkins (to fire it up on build), ansible scripts (to distribute work across machines and gather outputs) and grafana (to show trends of performance metrics). But the process management part (tracing pids after startup, killing them on timeout, gathering results into central place, parsing them and feeding into graphana) should have been already solved? Additionally, not sure how RAG status for tests (point 3) fits here regarding display and tying it to builds (in perfect world, it would be as integrated as unit tests are).