Introduction

Modern programming languages typically provide for encapsulation of data and operations into a Class or some similar entity.  In addition a class can inherit from parent classes. The class is a powerful building block within modern software implementations. The coherent nature of classes can make them a good choice as a test target. Their clear existence as a language construct makes them a common choice as a 'type' of test target.

It is not always apparent what is involved when a particular class is to be tested. This section provides some examples of class like assurance targets, illustrates the issues involved in clearly defining what is to be tested and hopefully demonstrates how the approaches described here help to eliminate ambiguity.

The Example Software - An Alarm Manager Service

Triggering of the AlarmTimers is totally independent of the AlarmManager. Once created they will trigger without any further involvement from the manager. In addition when one shot timers trigger they remove themselves from the managed population and destroy themselves. The AlarmManager inherits from an ActivePopulationManager class that implements the data structures and operations required to track instances of classes that can autonomously remove themselves from the population.

The AlarmTimers operate by creating a new thread for each instance of AlarmTimer that is created. This thread sleeps until its due time. When it awakes it operates the call back method and either reschedules itself or destroys the instance.

The class structure of this software is shown below:

The operation of this software is illustrated below.

Initial Thoughts on Testing the AlarmMgr Class

If someone is told to "Go test the AlarmMgr Class" then what does this mean? What might they do?  Some possibilities include:

Test The Lot

They could design tests around creating alarms to operate at specific times.  They will also design some additional tests around cancelling alarms. They will wait until the SDA is fully implemented, that is the three classes shown are implemented, any classes they inherit from or use are implemented and it is all working with the operating system. To run their tests they will link test software into the system that creates an instance of the AlarmMngr and operates it.

Test The Complete AlarmMngr Implementation

They could look at the way the AlarmMngr class generates and configures instances of the AlarmTimer class. They could ignore the operation of the AlarmTimer class itself. Their scope of work is to ensure the correct instances of this class are created not to check that this class then operates correctly. They would also introduce some tests to check that alarm timer instances could remove themselves from the population using the DelFromPop() method that an instance would use.

They will wait until the AlarmMgr and ActivePopMngr classes are fully implemented.  For testing they will replace the AlarmTimer class with a test class that allows them to check the manipulation of instances of this class by the AlarmMngr instance. They do not have to worry about real intervals of time or about running testing with independent threads operating.

Test The Source Code Level AlarmMngr Class

Alternatively they could look at the operation of the actual language level class AlarmMngr. Testing now considers calls to the methods provided by the parent class ActivePopMngr as part of the interface to the software under test. Testing expands to consider the different responses that can be received from calls to these methods. It would also consider calls to these methods made in response to calls to CrtAlarm() and CancelAlarm() methods as part of the expected behaviour of the software under test.

Testing waits until the ActiveMngr class source is available. Then a test build uses test classes in place of the ActivePopMngr and AlarmTimer classes. Once again there are no worries about real time intervals or additional threads. In this new configuration calls to the ActivePopMngr methods can be monitored and different responses from these methods simulated.

Test Without Using the Operating System Scheduler

It may be that they decide that whilst they wish to test the bulk of the software they do not want the test system to have to operate in real time. Rather than setting timers and looking for responses occurring at particular times they may now look at the calls made to the operating system to create and delete threads and the calls made to sleep the threads that are created. They may also look at the behaviour that occurs when a thread that has made a sleep call actually returns. They replace the interaction with the operating system with a more transparent and controllable testware component. Notice that in this scheme the testware does not even need to delay the timer thread when they make a sleep call. The actual time of the return from the sleep is not important, it is the parameters specified on the call and the response when the call returns that is significant.

Which Approach is Correct?

The different approaches described above show that there a number of ways the job - "Go test the AlarmMngr class" - could be done. It is not that one of these is the right way and the others are wrong. Each could be appropriate under different circumstances. In fact the best approach could be a combination of these approaches each looking for different things - a multi target work package.

Each approach is a plausible interpretation of the description of the work and each is technically valid. This highlights the ambiguity of the instruction to "Go test the AlarmMngr class". It highlights the need for a better specification of the work required. The sections that follow demonstrate the use of assurance targets as a more precise method of defining the testing required in these circumstances.

The Alarm Management Assurance Target Examples

Five assurance target examples are given each based on the Alarm Management service delivering assembly. The first two correspond to the Test The Lot and the Test The Complete AlarmMngr Implementation examples given above. The next two are both based on the Test The Source Code Level AlarmMngr Class example. Two targets are given for this example to illustrate the two possible ways of dealing with the data structures holding the population of AlarmTimer instances.  The final one illustrates the removal of the operating system from the target.

First Example - The Complete Alarm Management Service

The Alarm Management Service SDA

The first stage in the process of defining the assurance target is detailing the SDA we are working with. The SDA to be used when defining an assurance target that encompass the complete alarm management service is described and specified below.

The interfaces to the Alarm Management SDA consist of the creation of an instance of the AlarmMngr class, the methods that allow a client to create or cancel an alarm and the call back invocation from instances of AlarmTimer. The SDA includes the AlarmMngr and AlarmTimer code, the ActivePopMngr code, the language run-time and the operating system services that create, schedule and delete threads.

A specification of this SDA is given in the table below.

Testing the Full SDA

With the SDA defined the an assurance target can be specified. The simplest assurance target to define is the one that tests the complete implementation of alarm management. The definition for this assurance target is:

Analysis and test design will focus on the creation of alarms, the cancelling of alarms and the pattern of calls to the call back methods passed in when the alarms were created. This assurance target includes all of the software responsible for delivering the service and so testing can not substitute test software for any of the elements required to create a build that supports operation of the AlarmMngr and AlarmTimer classes ( and hence can provide the Alarm Management service ).

Relaxing the Requirements On Low Risk Elements

The full SDA based target identified above has all of the software involved in delivering the service as part of the target. It does not matter how indirect the involvement of the software or how low the risk that it could cause a failure. If software can be executed as part of service delivery it is considered part of the target. This means that the test build should use the same software that will go in the final system. It means that a change to any of this software creates a new version of the target for the test and so should, in theory, trigger an investigation of the need for re-testing.

Now unless we a dealing with an extremely critical system, say an aircraft control system, these restrictions are too arduous. The net effect will be to reduce the effectiveness of the testing activities. If the restrictions are complied with then either testing will be ineffective or the cost of delivery will be increased to the detriment of other activities. Alternatively the restrictions are likely to be ignored and an adhoc choice of a smaller target scope will be made. This later situation is also detrimental as an inappropriate and unrecorded choice may result.

The appropriate solution is to further limit the scope of the software in the target as part of the target definition. Software that is indirectly involved in the delivery of the service and that has a low risk of causing problems is removed from the target. The target definition explicitly identifies these exclusions. This is show below.

During the operation of the service calls are made to software diagnostic logging methods. These provide for error reporting and tracing the operation of the software. Whilst they might provide useful information during testing their presence is not essential. Also the chances of them influencing the behaviour of the service is very low. As a result they are excluded. This exclusion means that when creating a build for the tests:

• The proper logging software can be used.
• Partially developed software can be used.
• Stubs can be used in place of methods that actually do something.

The exclusion also means that changes to the logging software do not count as changes to the target. Thus changes to the logging software do not trigger an assessment as to whether re-testing is required to maintain coverage of the target.

Second Example - The Complete AlarmMngr Implementation

The target is restricted to the implementation of the AlarmMngr instance, excluding the AlarmTimer and operating system, by defining the calls for the creation and destruction of instances of AlarmTimer together with the method provided to allow the AlarmTimer instances to remove themselves from the population as layer interfaces. The scope of the target is illustrated below.

The SDA For The AlarmMngr Implementation Target

Now, as in the previous example, the high profile element in the testing is the AlarmMngr class. In both cases the bulk of the analysis / testing will focus on the methods of this class. Given this similarity it is easy to make the mistake of believing that the SDA for the two approaches will be the same. However this is not the case. Notice that the operation of the callback function is outside the scope, notice that code in the AlarmTimer class is outside the scope. In this case the service we are really interested in is management of a population of AlarmTimer instances for the client. We are less interested in what the alarm timers themselves do. To communicate this clearly we define an alternative SDA for use in this target:

Notice that in the Services section of the definition the service description no longer refers to the operation of the timers. Instead it directs a focus on what happens to the alarm timers. This is completely different from the previous example which focused on alarms themselves. Also notice that DelFromPop() is now in the interface as it allows a client, in this case an AlarmTimer instance, to remove itself from the population.

The Assurance Target

Now an assurance target that clearly conveys our intent and clearly communicates the focus of testing can be specified. The target is defined using the AlarmTimer Population Supervision SDA given above. The target specification is:

Third Example - The Source Code Level AlarmMngr Class

In the third case we may wish to focus directly on the novel elements of the AlarmMngr implementation. We may not need to address the functionality that manages a population of AlarmTimers as this is inherited from the ActivePopMngr class which is tested elsewhere. This focus is illustrated below:

The SDA For The AlarmMngr Source Code Level Class Target

The SDA specification for use with this assurance target is given below. The DelFromPop() method is no longer included. The assembly identifies creation and deletion as two separate services.

The Assurance Target

The scope is now reduced from the full SDA to the source level class by identifying the interfaces between the layer of interest and the rest of the SDA. The target definition is given below:

As the AlarmMngr class inherits from the ActivePopMngr class it has direct access to the ActivePopMngr data structures. When targeting the source code level class a decision has to be made on whether to treat these data structures as inside or outside the target. In the preceding example they were treated as being outside the target. However if the alternative approach, to treat them as being inside the target, is chosen then updates from outside the target form part of the interface. This is illustrated below:

The target definition for this arrangement is:

Our fifth example takes us back to the complete alarm service including the AlarmTimer class. Now we want to avoid the need to do testing in real-time.  We want to look at the creation, deletion and sleep requests made to the operating system. We wish to look at the behaviour when an AlarmTimer sleep calls returns.  For testing purposes we are not bothered if the sleep function is replaced by a stub that returns straightaway. This is illustrated below:

The assurance target specification for this is given below. It uses the original Alarm Management & Operation SDA definition given for the first example.

Observation on The Class Like Assurance Targets

The five examples given above are all valid interpretations of the instruction "Go test the AlarmMgr Class" . This illustrates the point that the instruction / definition is too vague. A plan that includes a task "Test the AlarmMgrClass" could result in many forms of testing being performed. An estimate could be based on one interpretation, the work done to another. It may be assumed that the operation of the AlarmTimer class has been covered when it has not.

Using the Assurance Target approach provides a much more precise definition. It leads to a more consistent interpretation of the work to be done. Note, however, that we have to address a specific target, it is not possible to define generic assurance targets and then look for instances of this type of target.