Step-by-step manual system test

Question

Background info

Believe it or not, but the last companies and projects I worked for still managed their system tests as simple Word documents that describe the steps and the tester fills in the result.

As you can imagine, evaluating the results is tedius, since you need to look at each Word document and copy/paste the results into a "management" system (typically Excel).

I don't want to elaborate other deficiencies of that approach but I'd like to improve this situation by providing the same advantages of the approach:

• anyone, really anyone, understands the concept: you get a Word document, follow the steps, write down the results. That's it. No training required.
• zero impact on the test system, as long as Word is installed on another machine to fill out the results.

Software request

The software must support writing tests

• provide a basic concept of defining steps and expected results in textual form. No images required.
• provide "comments" for information that is not a test step, e.g. to link to a bug tracking tool (or a requirement for the seldom case such a thing exists)
• if the software has way more features than needed, I'd like to be able to customize the UI to reduce it to the minimum of defining steps and expected results.

and executing tests

• show the steps to the user
• go forward / backward through the steps if they are not displayed all at once
• provide a human readable report (e.g. PDF)
• provide a machine readable report. Should be a format which is widely supported by other tools/libraries, e.g. XML.
• run at least on Windows 7
• (like before) if the software has way more features than needed, I'd like to be able to customize the UI to reduce it to the minimum of showing steps and defining results.

Other requirements:

• cost less than 300 € per user (not a per machine license)
• provide good keyboard input, perhaps customizable
• is not a web application (due to delays, connectivity issues, keyboard support, ...)
• no or little impact on the test system if the software needs to run on the test system itself (e.g. must not need Java or .NET since installing that may be part of the test itself)
• simple administration: no servers, no fat database etc. It should be really simple.

The software needn't

• manage requirements. Just focus on testing. In 8/10 projects I worked on, the requirements weren't specified anyway.
• manage users and access rights. I just want to give someone the test to execute.
• do complex reporting / trending etc. That will be done based on the machine readable format in a 3rd party (proprietary) application.
• test scheduling, test agents
• integrate with anything. Just run standalone.

I have already tried:

• TestLink, but writing tests is too complex and there's too much overhead
• Microsoft TFS, but it's way too powerful and complex to administrate
• Borland Silk Central, but it's way too powerful and complex to administrate

Answer 1

Finally I have implemented exactly what I wanted by myself. The result is called TestAdore.

Writing tests

provide a basic concept of defining steps and expected results in textual form. No images required.

There are two kind of steps: actions and verifications:

!
Do this.
?
Check if it is like that.

provide "comments" for information that is not a test step

Comments look like this:

# This is a comment

if the software has way more features than needed, I'd like to be able to customize the UI to reduce it to the minimum of defining steps and expected result

Editing tests is done in any text editor.

Executing tests

show the steps to the user

Done on a simple UI:

]

go forward / backward through the steps

Blue icon: go back. Green or red icon: go forward (pass/fail)

provide a human readable report

The report is machine readable XML + XSL transformation to make it human readable HTML.

provide a machine readable report

The report is XML.

run at least on Windows 7

Supports Windows 7 or higher, Desktop versions only.

minimum of showing steps and defining results.

The UI has only 5 buttons (see image before)

Other requirements

cost less than 300 € per user

Normal version is available for free.

provide good keyboard input

Use any text editor to write tests. UI for executing tests can be operated with Tab and Space.

is not a web application

It is a desktop application.

no or little impact on the test system if the software needs to run on the test system itself

Written in C++, portable.

simple administration: no servers, no fat database etc.

Standalone application. Data is in text files, not in a database. See administration

Answer 2

Can I suggest using a python web templating platform, or even iPython, to step the user through a series of pages with the tests, supplementary information, links to trackers, possibly pictures of things like the location of test points, which display element the question text is referring to, etc. and save the results to a .csv file for later analysis.

• The user interface would then be the web browser, typically free.
• The content of the tests could be quite rapidly be generated from the word documents, possibly automatically.
• You can hide the pass criteria so as to ensure that the user does not enter the passing values I have had problems with that in the past with a similar company - in that case the tester got sacked but we had to repeat all of the testing that he had done in a full weekend of overtime.
• There are lots of python tools for things like trend analysis
• Licence = Free
• Cross Platform
• Test structure can branch, e.g. if value is marginal take two more readings, etc.
• It is possible to give the operator direct feed back, e.g.:
  • "Passing so far",
  • "FAIL! - ABORT TEST NOW AND REMOVE POWER - POSSIBLE DANGER/DAMAGE FROM CONTINUING!",
  • "FAILS - Gathering Diagnostics",
  • "PASS - Test Completed",
  • "FAIL - Test Competed",
  • "Marginal For Review", etc.
• Can be locally hosted on the tester machine and the results files gathered later, (for offline equipment), or hosted on a central test server in a networked environment, (or even a combination of the two),
• Output to excel &/or word is still possible to satisfy the traditionalists.

Possibly Automate Parts of the Testing

Since python has libraries available for a number of communications formats including:

• RS232/RS422
• GPIB
• Modbus
• Ethernet
• USB

And there are a number of instrumentation connection packages, such as pyVista, you can actually, given suitable test instruments which you probably already have, collect the actual readings rather than requiring them to be manually input. This both saves time and reduces mistakes, such as errors of data entry and instruments on the wrong range setting.

This can give a near ATE process at very low cost - at work I recently wrote a python script which:

• Prompted the engineer to connect the test equipment to the PC,
• Verified the correct test equipment was connected and collected the serial numbers & calibration dates for each verifying that each was in calibration,
• Prompted the test engineer for the serial number of the equipment under test,
• For each test connection prompted a test engineer to make the connection to each of a set of test points, (with overlays on a picture of the test board detailing which lead to connect to which of over 85 test points),
• Validated the correct connection was made and prompted the tester if the wrong connection was made,
• Took a series of 5 measurements at each of 3 different settings, with the correct range set on the instruments, in about 3 seconds,
• Averaged the readings at each setting,
• Prompted the tester to move on to the next test point,
• Used the readings to generate the calibration values for the equipment under test,
• Stored the calibration readings for future reference
• Generated the calibration files and installed them on the equipment under test.
• Printed the Calibration Certificate.

The result - a 2 hour calibration process, which often needed to be repeated due to mistakes, reduced to a 10 minute process with mistakes almost impossible, test engineers that were not spending a lot of time transposing numbers from the test meter to a spreadsheet, or in some cases making up numbers, and much more confidence in the results.