• Order-Placed
• Order-Was-Not-Placed

This is illustrated below

Test-Interaction Fundamentals

Basics

A Test-Interaction is a component within the test solution. A Test-Interaction is the embodiment within the testware of an interaction with the system. The interactions chosen are those that are useful for testing purposes. An interaction is useful for various reasons including its presence in multiple tests and it helping to make the testware more understandable.

Client testware, say a test, uses the Test-Interaction. The Test-Interaction operates interacting with the system under test and then reports the behaviour that occurred to the client. The interaction does not attempt to decide whether the behaviour was correct - was it valid that the order was placed or not. The validity of the behaviour depends upon the context within which the operation was invoked. It is the client testware that knows this context.

Varying the Interaction

Obviously if all components do exactly the same thing each time they are used - select the same item and order the same quantities - then either the testing that can be done will be very limited or there will be lots of similar components. Neither of these is desirable. For Test-Interactions to be useful it must be possible to vary what they do. The client must be able to communicate to the component the details of the required interaction.

A Test-Interaction operates to a specification. The form of the specification varies. In some cases the specification will define specific values to be used in testing for example an item code and quantity. In other cases the specification may identify the source of the data. For example the interaction may expect a data pool containing a list of items and quantities and may then choose one at random to be used when placing the order.

As with the choice of interactions the exact form of Test-Interactions - what they do and how they are controlled will depend upon the testing to be done. However whatever the parameters are there is a common structure in the implementation within the testware. The Test-Interaction reads an operational specification set up by the client testware.

Reporting on the Test-Interaction

A key aspect of this approach is reporting to the client on the interaction that actually occurred. The client can then use its knowledge of the context within which the interaction took place to determine if the system has behaved correctly. The Test-Interaction itself is non-judgemental with regard to whether a behaviour is valid or not.

The report to the client takes the form of an interaction record created by the interaction. It contains three things:

• It identifies the behaviour that occurred - for example Order-Placed or Order-Not-Placed.
• It records the significant parameters of the interaction - for example Stock-Code and Order-Quantity.
• It records significant outputs from the system - for example Order-Reference-Number.

The parameters of the interaction are included in the report because:

• The client may not know the values if they are not individually included in the operation specification
• The review of the interaction may not be adjacent to the use of the operation either in time or within the software. Including this information in the report removes the need for the client testware to record and manage the distribution of this information.
• The report now contains all significant information on the interaction as a basis for logging ( see below ).

Logging the Test-Interaction

Within test automation solutions the logging of the test execution tends to be adhoc. Logs tend to be flat and sequential rather than structured - after all where are the natural boundaries on which such a structure can be based? Test-Interactions or rather the operation of a Test-Interaction form a natural unit for execution logging.

The log can contain a summary of the operation. Additional logging of events that occurred within the interaction being treated as child information. This provides a structure in the logged information. The interaction report not only provides a basis for the client to review the interaction it also provides a consistent summary of each Test-Interaction for logging purposes.

The reports on interactions can be used to identify incorrect operation in the software under test . For example the interaction report states that the behaviour was Order-Placed when the item should be out of stock. Manual reviews can be performed on the records of interactions contained in execution logs. Automatic reviews need to be performed in the client testware.

Automatic detection of anomalies within the testware requires a determination of what the 'correct' behaviour should be for the interaction. There are various types of approaches to this, these are briefly outlined below.  For a more detailed discussion see the section Approaches to Reviewing Test-Interaction Behaviour.

Approaches for determining automatically whether the behaviour is correct fall into two distinct groups - Static Approaches and Adaptive Approaches. Static approaches have a 'constant' expected behaviour no matter what has happened earlier in the execution of the test.  Each test run will check the observed behaviour against the same expected behaviour. Adaptive approaches take account of the behaviour of the system up to this point to determine what the expected behaviour should be. The attempt to adjust the testing to reflect what has happened so far.

Static Approaches

For each case the expected behaviour can be worked out as part of the test design and embodied in the test either at the point of review or in some other location and made accessible at the point of review. The expected behaviour is assumed to be the correct behaviour and the reported behaviour is checked against it. Expected behaviour can be encoded into testware code or included in data tables along with the parameters for interactions. There are a number of points to note about this approach:

• There can be an avalanche of anomalies reported when interactions are dependent on early interactions. Once one goes wrong, for example spuriously rejecting an order, subsequent interactions that are actually behaving correctly, say Order-Not-Found, will be classified as anomalies.
• Problems also arises where there is non deterministic behaviour in a system. For example, valid behaviours for an establish connection interaction used in context C may be - Connection-Established and Connection-Rejected.  Whilst it is easy enough to define both as acceptable at this point subsequent expectations are no longer predictable prior to execution.
• If the rules change all cases need reviewing and possibly modifying. The rules are embodied in each expected value and so many changes may be necessary. For example if an order can be accepted so long as the stock deficit does not exceed 100 and then this is changed to 150 each case needs reviewing.

Adaptive Approaches

Adaptive approaches attempt to take account of the history of interactions with the system to determine the correct behaviour for the next interaction. For example if a test is done to determine if a particular user account exists then the expected system behaviour ( Account-Exists or Account-Not-Exist ) may depend upon the outcome of an earlier account creation operation ( Account-Created or Account-Not-Created ). The nominal test sequence and the expected behaviour in a static approach could be Account-Created, Account-Exists but if the account is rejected for some reason, causing a failure of the creation test then the existence test can check for Account-Not-Exist.

Adaptive approaches can use two techniques to help to predict the expected behaviour. One technique is to obtain information from the system under test. The other techniques is to use a Test-Oracle to track the state of things.

Querying the system under test typically uses a back door route to access data held in the database of the system. In the example above the test tool could do a direct query on the database to determine if the account existed in the database and hence to establish the expected behaviour.

In the test oracle based approach the testware independently tracks the state of entities such as accounts. The reports on each test interaction are passed to the oracles and they update their models based on this information.  An oracle based review process can be done in real-time as the tests execute or it can be done as a post execution analysis of the interaction reports.

These two techniques provide detection of different sorts of problems. Using the system database as a reference detects inconsistencies in the system operation. The account is in the database but the system indicates it does not exist or it does not appear in the database but the system reports it as being present. Querying the system database does not help detect other types of error. For example if the account is spontaneously and erroneously deleted from the database after creation then this technique will not detect it. In comparison with this the oracle based approach will detect the spontaneous loss of an account but would not detect that it was reported as existing when it is not present in the database.

Query based approaches require knowledge of the database structure and couple the testware to particular implementations. Test-Oracle based approaches require the development of additional testware embodying the required behaviour.

Which Approach?

No single approach is 'best'. Within a test suite different approaches may be appropriate. For a given Test-Interaction usage it may be appropriate to use a combination of approaches using each approach to check for different sorts of problems.

The Use of Assessor Components to Reduce Duplication

Assessor components can help to reduce duplication in the implementation of reviews. This is particularly true for static reviews. Components can be provided that take information describing the context of an interaction, the report on the interaction and that embody the rule about how the system should operate. Now instead of the test explicitly including the expected behaviour it includes the predicted values of the key information required by the assessor. At the point of review the assessor, the predicted values and the interaction report are combined to determine the expected behaviour.

The advantage of this approach is that the rule on the system behaviour is contained in one place, in the assessor. If the rule changes only the assessor need change. For example an assessor to check the stock level rule would require the current stock level as a context value. It would initially be a comparison of the form

When the required behaviour changes it is only necessary to change this to

A single modification to the testware updates the entire test set.

Monitoring and Managing Test Execution

As well as being used to detect anomalies in the operation of the system the behaviour reported by a test-interaction can be used to monitor and direct the execution of the test. The test sequence can itself be adaptive - if an account creation interaction results in the account being created then do A else do B.

Detail in the Behaviour Classification

The behaviour of the order placement interaction was introduced at the start of the section. When introduced two behaviours were chosen to describe the interaction Order-Placed and Order-Was-Not-Placed. The diagram below shows an alternative scheme for describing the behaviour of this interaction, one which provides a more detailed description.

This new scheme sub-divides the behaviour when the account is not created into three sub-behaviours. The behaviour description is now a hierarchy:

• Order-Placed
• Order-Was-Not-Placed
  • Item-Not-In-Catalogue
  • Configuration-Not-Available
  • Order-Not-Accepted

A more detailed description is necessary when the interaction is the main target of the testing. For example it is intended to check that no records are displayed when the item is not in the catalogue. If the interaction is simply being used to place orders for use in other tests then the higher level description can be used.

Generally the automation solution should allow a test-interaction to support detailed descriptions for specific testing but higher level descriptions for general use. One way to approach this is to use a hierarchical tree structure for the description.

Detecting and Reporting Local Anomalies

Up until now we have indicated that Test-Interactions do not detect and raise anomalies. They have been described as non-judgemental as what is right or wrong depends upon the context in which they are operated. In reality this concept applies to context dependent behaviour but does not apply to context independent behaviour. Context independent behaviour is behaviour where the interaction can know what should or should not happen without knowing anything else about the system state.

Examples of context independent behaviour are:

• When a syntactically valid stock code is entered into the search form the submit button becomes active.
• When a stock code with a syntax error is entered in the search form the submit button is inactive.
• When the submit button has been triggered a report window appears.

Notice that in each example the test-interaction can determine what should happen without any wider knowledge. For the first two it just has to check the syntax of the stock code and the last one does not even depend on any of the interaction parameters. The Test-Interaction should detect and report these local anomalies, dealing with them should not involve the client testware.

Behaviour that the Test-Interaction Does Not Recognise

A Test-Interaction classifies the behaviour experienced during the interaction. Local anomalies can mean that the path the interaction takes with the system does not correspond to one of the recognised behaviours.

Suppose on the example above the item is always specified by its stock code. A general catalogue search facility is used, which can display multiple items, but which should not do so when used in this interaction. This is because the search is done using the stock code and so should always return one item or indicated that no items are found. Now if the test-interaction submits the stock code search and receives two or more items back what should it do? This response from the system is invalid under any context in which the interaction is used. In this case the behaviour may be undefined. If this is the case then the Test-Interaction becomes responsible for reporting the anomaly. This is illustrated below:

Some thought needs to go into the response of the test-interaction when local anomalies are detected. An over simplistic approach may lead to any anomaly causing the behaviour to be undefined something which, in turn, will make the testing process very sensitive to anomalies that it might have been possible to work around. An objective for the testware designer should be that if testing can continue to provide useful information after a local anomaly then it should proceed further the test-interaction should not simply fail.

In the example above whether the search returning multiple items is a show stopper depends upon whether the testware can identify which of the reported items corresponds to the stock code. If it can identify this then it can log the anomaly, select the correct item and proceed further. On the other hand if it can not identify which line corresponds to the stock code then it can not proceed and the behaviour is undefined.

Other examples of local anomalies that a test-interaction can proceed beyond include:

• Buttons being enabled when they shouldn't be - simply do not press them.
• Buttons being disabled when they shouldn't be - so long as they are not needed or there is an alternative menu option to trigger the same command.
• Secondary indication messages missing.

Do Local Anomalies Count As Primary Test Failures?

A simplistic approach to defining a test failure is if during the test some part of the system does not behave as it should then the test fails. This approach is not recommended.  It is more appropriate to recognise that a test has a purpose it is testing for particular types of problems. If it finds a problem of this type then it has failed ( or succeeded depending upon your point of view ). If it finds a lower level less significant problem then the test has not failed though an anomaly has been detected.

Suppose, for example, that a test set has been designed and automated to test the order processing lifecycle. It creates orders, adjusts them, cancels them and fulfils them. At each stage it checks the correct status of the order the quantities in work in progress reports and so on. The test is executed and the only anomaly detected is within a test-interaction that queries the order details, the description of the product is not displayed. Whilst an anomaly this should not fail the order lifecycle test but should cause an HCI test to fail.

To support this approach the automation scheme needs to allow a test to define whether a test-interaction is a primary part of the test or not. Each test can then focus on the level of anomaly it is intended to address.

How Much Checking?

It can be seen that it may be desirable to adjust the level of checking performed by a test-interaction. For example the place order interaction. If testing the HCI then extensive checking of all dynamic HCI elements and the contents of all data fields is desirable. If testing the order lifecycle then it may be possible to limit checking to that required to establish the behaviour. The level of checking required depends upon the purpose of the test using the test-interaction.

Test interactions are intended to reduce duplication in the testware so rather than having separate interactions for each case it is desirable to be able to turn up or down the level of checking. A testware architecture should provide a systematic approach to achieving this.