The Wide Variety of Needs

Test automation comes in many forms and tackles many different needs. Automation can be used to address functional testing, reliability / integrity testing and capacity / performance assessment. It can be applied at the system level, at the level of sub-systems and on lower level components of a system. The tests implemented may address all of the behaviour of a target or some key critical function or operation of the system.

Some solutions may only need to support specific tests or test closely related to them. Others may need to permit easy re-configuration for a wide variety of different tests. The solution may have a very short operational life or may be intended to form the backbone of testing over many releases. Testing may be aimed at detailed fault finding or could be aimed at achieving some other objective ( see Selecting Test Packages ).

To be successful an automation solution must match the need. There is no single approach that can match all of these needs.

Core Functionality

The varied nature of the problems that software automation solutions address lead to a wide variety of solutions structures and to the use of different technologies. Each solution has to be designed and developed to implement the test objectives in the areas of:

• Applying the inputs and stimulation to the software under test correctly.
• Adequate monitoring of the software under test so that the problems the tests focus on or are likely to trigger can be detected.
• Appropriate support for the automatic selection of test data to allow tests to define high level data selection criteria rather than explicitly specifying vast sets of data.
• Appropriate checking of outputs to determine outcomes. This can be used to automatically select subsequent tests / test data. It can also reduce the amount of manual analysis of raw data needed to provide useful information from the testing.

The extent to which tests need to be configurable and the intended life of the solution drive the extent to which tests can be embedded in the solution as opposed to being defined with a 'language' or method that the test solution then 'executes'.

Designing and implementing testware to provide this core functionality is not a simple task. It requires both traditional testing skills and design and implementation skills more usually associated with software development.

Solution Elements

A full solution is more than the core testware that interacts with the software as the test is executing. There can be many elements including:

• Elements that operate prior to commencing interactions with the system. For example:
  • Test data generators that produce consistent test data sets based on high level selection criteria.
  • Sequence generators that generate operation sequences for tests based on high level selection criteria.
• Elements that operate in conjunction with the scripts / software driving the software under test. Examples could be:
  • Test-Oracles that track the state of application domain entities such as accounts / orders / calls etc. based upon the history of interactions and responses from the system.
  • Test-Oracles that track the system state.
  • Test data selectors / servers that allow a test to request an entity with certain characteristics, for example an account that is over drawn.
  • Assessors that judge the system response to an interaction.
• Post processors that analyse traces of interactions with and responses from the system and assessor whether the behaviour is correct.
• Snapshot and auditor utilities that capture database states and detect significant changes.

For any automated solution using an appropriate high level architecture is really important. Synthesising the architecture is a demanding task that requires an understanding of the tests to be supported and the architectural options available.

There are some common issues that many test automation solutions must address. Though the problems are common there are no 'standard' solutions to them. These problems are outlined below.

Spotting 'Unforeseen' Anomalies

Generally automated solutions are good at detecting anomalies they have been designed to detect. The potential failure was foreseen and a mechanism put in place to detect it. Manual testers perform less well in this respect, they may forget to check something, they may check it in less detail or time may prevent them checking as many things. Manual testers perform better at spotting 'unexpected' anomalies. They have a global view of the system behaviour and a mental model of what should be happening. This allows them to spot discrepancies - for example a window suddenly appearing or disappearing or a value changing.

An automated solution is never going to have the ability of a human to spot the completely unforeseen event but to be acceptable automated solutions need to look beyond the immediate response to the input they have just applied. The solution needs to have mechanisms that provide appropriate monitoring for changes that are not explicitly related to the input to the system. The automated monitoring needs to focus on risk areas.

Keeping Going When Anomalies Occur

Tests are defined to detect anomalies so they should be expected. Within tests and between tests there are often dependencies. Test B assumes that test A has completed and created an entry in the database. When an anomaly occurs in test A this may not be the case.

Manual test execution deals with this.  Manual testers may realise there will be a problem before executing B or when B has been executed and not behaved as expected. They can then recover the situation.  Automated test suites often suffer a domino effect where one anomaly results in incorrect operation of many subsequent tests. Developing testware that can continue to provide useful results after anomalies have started to occur is a difficult task.

Coping With Change

Modern software development is iterative and incremental. Change is always present. Test solutions must remain effective when the software they are testing changes whether it is the behaviour of the software or the interface to the software. The importance of this may reduce for one-shot test solutions with short life times but even then it does not go away. The test solution development will be based on intent / specifications etc. Even a one-shot solution must track the intended behaviour and implemented interfaces up until the point where it has been successfully executed.

Difficulties Understanding the Tests

There may be document based test specifications but often the automation solution becomes the definitive record of the tests to be performed. Document based specifications become a working paper used on the way to implementing the tests. A big issue here is, are people who need to understand exactly what tests will be / have been done able to obtain this understanding.

Documents typically spell out the tests one by one in text, tables, pictures and diagrams.  They can be read sequentially and the reader picks up an understanding of what tests are proposed. This can be difficult when the tests are embedded in an automation solution. Test scripts may not clearly spell out inputs / outputs and expected results in the same way as a document. This can be a significant issue in test automation.