Behaviour Discrepancies

Behaviour is associated with test-interactions. Test-interactions are used by client testware. When an interaction has completed the test-interaction reports the behaviour that has occurred. The client testware can then review this against the expected behaviour based on the context in which the test-interaction was used. At a simple level if the reported behaviour does not match the expected or acceptable behaviour then an anomaly has occurred.

Behaviour discrepancies come in two forms:

• The test-interaction recognises the behaviour and reports it but it is not one that is expected by the client testware.
• The test-interaction does not recognise the behaviour that occurs, reports undefined behaviour and there is no Undefined Behaviour Exception Handling to change this to a defined behaviour.

The Impact of Behaviour Discrepancies

A First Look At The Issues

The issue can be illustrated by a simple example.

Client testware uses a test-interaction EnterOrderDetails. This enters values into fields and checks the state of the Submit button. It embodies certain rules that determine if the submit button should be disabled ( based on syntax ) and can tell if the button is enabled when it should be disabled for these reasons. If these reasons are not present the button can still be disabled due to the relationship of field values to the contents of the product catalogue. If the button is not disabled due to syntax then the interaction will accept the button being either disabled or enabled as valid behaviour.

The interaction described the behaviour it participates in as either Entered-Valid-And-Enabled or Entered-And-Disabled. If it detects that the fields have syntax errors but the button is enabled it reports the behaviour as undefined as it can not be one of the recognised behaviours.

The updates to each field are done using field specific test-interactions. These classify the behaviour as Updated-No-Modification and Updated-But-Value-Modified. The Enter-Account-Create-Details interaction always expects to be used with data that will be accepted without modification. It always expects Updated-No-Modification to be report to it.

So what happens if one of the field related test-interactions returns Updated-But-Value-Modified? This is not the expected result so firstly a behaviour discrepancy anomaly is raised.  So far so good but now what? Can the EnterOrderDetails interaction be treated as having operated, does it report a behaviour and if it does report a behaviour can the test go on?

In An Ideal World

The first thing to do is to look at this issue from the perspective of the client testware using EnterOrderDetails. What does it need to know and what can it deduce? In an ideal world where nothing could go wrong at a lower level the client testware can interpret the behaviour from EnterOrderDetails as follows:

( 1 ) If the supplied data had syntax errors and the button had not disabled then the EnterOrderDeatils test-interaction would not report Enter-Valid-And-Enabled it would report undefined behaviour. This narrows the location of the problem down to the disabling of the button due to the catalogue content.

( 2 ) The button has been disabled when it should be enabled. The cause could be either the syntax checking or the catalogue checking. When the test-interaction finds the button disabled it could always arise from the catalogue content rule which is outside its scope and so it can not determine whether disabling the button is valid or not.

As the table shows for all the anomalies A1-A3 meaningful anomaly reports can be generated that clearly identify the problem. The impact subsequent testing varies for each anomaly:

• In case A1 the test was not expecting to progress this order any further - the button should have been disabled . So having reported the anomaly testing can continue.
• In case A2 testing may have expected to progress this order further. This will not be possible as the submit button is disabled. The testing must roll back and resume from a point that does not depend upon this order being created.
• In case A3, as in case A1, the test was not expecting to progress this order any further - the button should have been disabled . So having reported the anomaly testing can continue.

Objectives When Addressing Discrepancies

The approach to dealing with lower level anomalies when updating field should attempt to:

• Ensure that the anomaly that occurs when updating the field is not missed.
• Have the minimum detrimental impact on the ability to detect anomalies in the operation of the submit button. The performance in this area can be judged against the performance with no anomalies outlined above.
• Allow sensible control of subsequent testing to avoid an avalanche of spurious anomaly reports.

Field Update Anomalies

When field update anomalies occur the form does not accept the values typed into the field. The values are changed by the form. This results in different values being used to determine if the submit button should be enabled or disabled. This can disrupt the testing of the submit button control mechanism and high level testing that depends upon an order being created.

Adhoc approaches to dealing with this issue are unsatisfactory and difficult to implement. A systematic approach to containing the impact of lower level anomalies that cause behaviour discrepancies is required.

A Systematic Approach to Containing Behaviour Discrepancies

We will now look at how we want the EnterOrderDetails interaction to operate when the update field interactions do not operate as expected and how to do this in a systematic manner We first consider the case where the client to EnterOrderDetails has supplied syntactically valid data and an item that is in the catalogue. A system behaving correctly will accept this data and activate the submit button. The expected behaviour is Entered-Valid-And-Enabled.

The second case is the case where the client to EnterOrderDetails has supplied data that is invalid. Either there is a syntax error or the item is not in the catalogue or both. A system behaving correctly will inhibit the submit button. The expected behaviour is Entered-And-Disabled.

When the Submit Button Should Be Enabled

The client to EnterOrder details has supplied syntactically valid data and together with the code for an item that is in the catalogue. When this data is entered any changes to the data values arising from errors in the form code may affect the syntax, affect whether the item is an item in the catalogue or do both.  The cases are:

• Where the changes leave the syntax valid and the item specified is in the catalogue. There are two versions of this, one where the item is the one originally specified by the client (Case 1a) and one where the item is changed to another item (Case 1b).
• Where the changes leave the syntax valid but result in specifying an item that is not in the catalogue ( Case 2 ).
• Where the changes introduce syntax errors but still specify an item in the catalogue. There are two versions of this, one where the item is the one originally specified by the client (Case 3a) and one where the item is changed to another item (Case 3b).
• Where the changes introduce syntax errors and result in specifying an item that is not in the catalogue ( Case 4 ).

Assume that there are no additional anomalies in the way the syntax checks and catalogue interlocks control the submit button. Where ths assumption holds the cases identified above can be divided into:

• Cases where an order could be created - Case 1a, Case 1b.
• Cases where it is not possible to create an order - Case 2, Case 3a, Case 3b, Case4

When an Order Could Still Be Created

An order can still be created, the submit button is active. An anomaly will have been raised against the field and so the problem will not be lost. The submit button is active so the reported behaviour will be that expected by the client testware. Can subsequent testing proceed? Obviously this will depend on whether the modified field value(s) will disrupt the test.  We don't want to proceed with tests that are bound to raise anomalies.

A basic approach that helps to minimise the risk of disruption is to use the values for fields reported by a test-interaction for subsequent testing rather than the values supplied to the test-interaction. This greatly improves the resilience of the tests. However there may be cases where it is not appropriate to continue. Examples include:

• Cases where the original value has a special relationship to other values - for example the item being ordered.
• Cases where the original value has special characteristics - perhaps the uses of spaces in an identifier.

The approach to providing a systematic way of dealing with this issue is to review the report and to then to add to it an indication of whether the interaction was close enough to the original intent to allow testing to proceed. When the report is returned to the client testware subsequent steps, that depend upon the success of the first interaction, can check the report and are skipped if it was not classified as successful. The trigger for the review is the behaviour discrepancy anomaly that occurs when the update field interaction reports.

Within the testware there must be a decision maker that can assess what has happened and decide what to do. As these decisions are specific to the testing being performed the decision making is done in the test specific testware. In a similar manner to the handling of Undefined Behaviour Exceptions the report is passed to an Exception Management software for review.

When an Order Can Not Be Created

An order can not be created the submit button is inactive. An anomaly will have been raised against the field and so the problem will not be lost. The EnterOrderDetails test-interaction can not report the behaviour as Entered-Valid-And-Enabled. If it proceeds, reports and is reviewed against a pre-defined expected behaviour ( based on the supplied values ) a behaviour discrepancy anomaly will be triggered. This is undesirable as it will imply a fault in the control of the submit button.

Again it is possible to build in some natural resilience if the expected result is generated by a rule based approach using the values reported by the test-interaction. This adaptive approach avoids raising a behaviour discrepancy and hence questioning the operation of the submit button. ( Note: When an adaptive approach is used to provide resilience to anomalies it can still be worthwhile detecting and reporting when the adaptive value differs from a nominal fixed value. )

The exception triggered review of the report can also take action to avoid the behaviour discrepancy being raised. It can mark the report to indicate that the behaviour should not be checked as it that it has been changed as a consequence of some other problem. This depends upon the exception handler being able to identify the correct behaviour based upon the values in the report. The adaptive behaviour has been relocated from the generic test-interaction to the test specific exception management.

Obviously as the submit button is disabled the order can not be created and this test can not continue. The exception review marks the report to show that the interaction did not get close enough to the original intent to permit testing to proceed further. Marking this report in this way should prevent an attempt to operate the submit button and create an order.

When the Submit Button Should Be Disabled

The client to the EnterOrderDetails interaction has supplied data that disables the submit button. Now when entered into the form the values get changed due to problems with the field handling code and a behaviour discrepancy anomaly occurs. An initial distinction can be made between:

• Cases where the new data values do not cause submit button to become active.
• Cases where the new values are such that submit button becomes active.

In the first case the best outcome is one where the anomaly is reported and then testing proceeds as it would have done if the anomaly had not occurred. A review of the report is triggered by the presence of behaviour discrepancy anomalies during the operation of EnterOrderDetails test-interaction. In this case the review need only log that a test case has been modified, the values used to check the operation of the submit button interlock were changed. Testing can then continue along the intended route.

In the second case where the submit button has become active it is good practice to prevent a discrepancy between the behaviour reported by the EnterOrderDetails interaction and the expected behaviour causing another anomaly. As described in the previous section this can be accomplished using adaptive selection of the expected behaviour in the test-interaction if the rules are generic or by using adaptive behaviour in the exception manager to decide the behaviour has been changed as the consequence of other anomalies.