Undefined Behaviour

Behaviour is associated with test-interactions. A test-interaction as its name implies interacts with the system and then reports the pattern of behaviour that occurred. The patterns recognised are those that would be valid under some context of use, for example creating an account or rejecting the creation of account. Other behaviour, for example the window disappearing when the create request is in progress are treated as undefined behaviour.

When the test-interaction completes it generates a report on the interaction intended for the client testware. Amongst other things this contains the behaviour that occurred. If this is undefined then this is an Undefined Behaviour Exception.

The Impact of Undefined Behaviour

Initial Response to an Undefined Behaviour Exception

When this situation occurs the immediate response is to raise an undefined behaviour anomaly. This anomaly is associated with the domain of the test-interaction reporting the undefined behaviour. Whether this is classed as a test failure or not depends on the test failure domains associated with the test that is running ( see Domains for Anomalies ).

Subsequent Consequences of Reporting Undefined Behaviour

If no special action is taken then the effect will propagate up the test-interaction operation stack. It will progress all the way to the initiating test unless stopped by a test-interaction that can still recognise the behaviour it is participating in despite the test interaction it has used reporting an undefined behaviour As it propagates back the consequences at each level will be:

• When the client testware checks the behaviour it will not match the expected behaviour for the interaction it used. This will result in an incorrect behaviour anomaly being raised.
• The client testware will classify the behaviour it has participated in as undefined.
• As the client testware reports an undefined behaviour anomaly is raised.
• The process repeats at the next level.
• At some point an anomaly will be raised that will cause the test to fail.

Active Handling Of Undefined Behaviour

Planned Responses To Undefined Behaviour Exceptions

The sequence of events described above is an uncontrolled catastrophic response to the initial anomaly. For example because the customer address field corrupted the field update behaviour was undefined.  As a result the test exercising the order lifecycle not only failed but did not test any further. This is not acceptable behaviour from the testware.

On the other hand we do not want to have to provide separate testware to enter the order details. That would create duplication. It would also mean that the corruption could be missed. Even though we do not want it to fail or disrupt the test we want to detect and report the anomaly.

So when a test-interaction results in undefined behaviour what planned responses can be applied? Possible responses are:

• Return to the test. No useful testing can be done so leave the generic test-interactions and go back to the test level where appropriate test specific recovery action should have been encoded.
• Establish that testing can proceed further if the undefined behaviour is treated as one of the defined behaviours. Override the reported behaviour and pass this information to the immediate client testware rather than reporting undefined behaviour.
• Allow normal processing to proceed with undefined behaviour being reported to the immediate client. The client may cope or a new undefined behaviour exception may propagate. The same options apply to dealing with this new exception.

The first case is applicable in a situation where no more sensible testing can be achieved within the generic test-interactions providing the test architecture. Control must return to the test where it can either abort or carry out test specific recovery.

In the second case for the particular test that is executing it is sensible / safe to treat what actually happened as a case of behaviour X and proceed from here. The testware generates a replacement report that changes the behaviour value from undefined to the selected one. This new report is passed to the client testware in place of the original report. Now assuming the chosen behaviour matches the expected behaviour defined by the client testware then execution continues as normal with no further propagation.

In the third case the undefined behaviour propagates into the client testware. An incorrect behaviour anomaly will occur. Propagation is halted if the client test-interaction is designed to determine its own behaviour despite receiving an undefined behaviour from its worker test-interaction. If this is not the case then the client test-interaction will also report undefined behaviour and another exception is triggered.

Example Strategies

Assume that a test set is testing the lifecycle of orders. Within the testware architecture there is an enter order details test-interaction, this in turn uses a number of field update interactions. If the field value is not accepted as is, then the field update test-interaction reports an undefined behaviour.

One field allows entry of a customer's special delivery instructions to the courier - 'Leave it in the green house.' or 'Do not go near the dog!'. Obviously corruption of this field should not cause a test collapse when testing the lifecycle of orders. Another field is the postcode. If this field corrupts testing can proceed so long as this field contains between 1 and 7 characters with no spaces. Approaches to dealing with these issues are discussed below.

Handle Delivery Instruction Corruption At The Update Delivery Instructions Test-Interaction Level

When the undefined behaviour exception occurs a decision is taken to override the reported behaviour. The reported behaviour is changed to the normal Field-Updated behaviour. At least one anomaly, an undefined behaviour anomaly, will have been raised so the problem is not missed. The new value of the field, the result of the corruption, is present in the update interaction report and this should be used for subsequent validation of the field value ( rather than the value that was originally entered ).

Handle Postcode Corruption At The Update Postcode Test-Interaction Level

When the undefined behaviour exception occurs the new value of the field is checked. If it is 1 to 7 characters with no spaces then the behaviour is overridden to Field-Updated. If it does not meet this requirement then a decision is made to return to the test.

Handle All Corruption At The Enter Order Details Test-Interaction Level

With this strategy when any field update interaction has undefined behaviour it is reported directly to the Enter Order Details test-interaction. No special actions are taken. Anomalies are recorded and the Enter Order Details proceeds to set subsequent fields. ( Note: For this strategy to work the Enter Order Details process must enter all fields even if some updates report undefined behaviour. )

When all fields have been addressed the Enter Order Details interaction will determine the behaviour that occurred. In cases where some of the worker test-interactions reported undefined behaviour the Enter Order Details Test-Interaction must report undefined behaviour. ( Note: In some cases individual test-interactions may be tolerant of undefined or even incorrect behaviour in worker test-interactions they use. )

Exception handling is applied when the Enter Order Details test-interaction reports. The contents of all of the fields and their consistency can now be checked. If it is possible to proceed with the order then the behaviour can be mapped to Order-Details-Entered. If the state of the fields means that the test can not proceed, for example the postcode field is blank, then a decision is made to return to the test.

Controlling the Response

If exception handling is required for a test then somewhere in the testware there must exist information that identifies the points where exception handling is applied. The testware must also contain any rules that are to be applied to select different exception handling outcomes when a particular point is triggered.

Exception handling is controlled from the test level testware. The decisions are specific to individual tests and can not be made by generic testware in the architecture / framework. When a test commences it registers the points at which it will perform exception handling. When an undefined behaviour report is generated a check is made to see if the point of exception is a registered point. If the point is not registered the undefined behaviour is passed directly to the client testware. If the point is registered the report is passed to the exception management testware for the test.

The exception management testware determines what should happen. Either pass up the exception or override the behaviour or return control to the test level. This process is shown below:

( 1 ) The test registers the points at which it will handle exceptions.

( 2 ) The Field Updates With Modification Checks interaction reports undefined behaviour.

( 3 ) This is one of the registered points so details are passed to the exception management testware.

( 4 ) The exception management testware determines the action required and informs the reporting service.

( 5 ) The client test interaction is supplied with an amended report.