Testing Problems Caused By System Anomalies

An anomaly has occurred when the testware detects that the software under test has not behaved as expected. This indicates a possibility that the software has a problem ( the testware could be wrong ). The testware will record the anomaly for investigation.

So if during the execution of a test an anomaly is detected then has the test failed? A simplistic view would be to say an anomaly means the test has failed. This practice can devalue the significance of a test failure. It reduces the usefulness of the test status as an item of management information. A different approach to determining if an anomaly denotes a test failure is needed. This approach must address the following:

• A well formed test set has an objective. The test set is looking for particular types of problems and has a model of the system at an appropriate level of abstraction. An order lifecycle test uses an interface to manipulate orders but has no real interest in the interface. It could be using a client GUI interface or it could be using a web-service interface. Anomalies in the operation of the interface that do not impact on the lifecycle of the order should not be classified as test failure.
• Testware needs to track changes in the system but there is often a lag. Anomalies may be reported in areas that have not been updated. If these areas are not significant from the point of view of a particular test then they should not cause test failures.
• Systems contain faults that cause anomalies. Again if they are not significant for a test then they should not cause it to fail.

Areas of Concern When Testing

It is normally possible to list a number of different areas of concern when looking at the operation of a system. For example:

• The GUI.
• Content of information presented to users via the GUI and in reports.
• Operating the system via the GUI.
• Operating the system via alternative interfaces.
• Behaviour of system operations.
• Order lifecycle.
• Management information creation.

Generally it will be possible to say that anomalies in certain domains denote a test failure whilst others are not significant enough to denote that the test has failed. This view goes some way to addressing the issues identified above. Embodying the notion of different areas of concern into testware provides a technique for making the testware more resilient.

Anomaly Domains

The technique involves defining a number of different domains as part of the testware definition. Test-interactions are associated with a particular domain. By default anomalies raised within a test-interaction are associated with this domain. The domain of a test-interaction is also allocated to problems detected during the review of the behaviour reported for an operation of the test-interaction.

When executing tests the domains that denote test failures are selected by the test level testware. The mechanism for determining if a test has failed when an anomaly occurs takes account of this selection. This technique helps the testware to provide meaningful test pass / fail information when anomalies occur.