Introduction

An established approach to planning and controlling software testing is to prepare a Test Strategy that defines the testing to be done. Typically this defines a number of one-off system level test packages and generic classes of testing for lower level system testing and for testing the internal operation of the system. A number of difficulties exist with this approach the primary one being the difficulty in knowing what should be done when the system is still just an idea. These issues are discussed in this section.

Problems Using Generic Classes

A Test Strategy may attempt to define the tests using a number of generic classes of testing such as 'unit', 'class', 'module' and 'sub-system'. Some Test Strategies assume that the meanings of these generic names are obvious. It is obvious what a 'module' is. In other cases the Test Strategy attempts to define what the terms mean in this strategy. Definitions and rules are given that are intended to allow different people to come up with the same lists of 'units', 'classes', 'modules' etc. when they look at the software.

This approach is typically intended to work as follows:

• Identify types of software object that are of interest - say 'units', 'modules' and 'sub-systems'.
• Using these definitions the instances of each type within the software can be agreed on.
• Rules from the strategy that define the testing to be performed for each type of object are then applied to each instance of the type. This is intended to give the requirements for testing that particular item.

Extensive experience of this approach as a basis for defining software testing requirements has identified some major issues. In the real world this approach can be ineffective and at worst can be detrimental. The problems with this approach are described in this section.

The Uses of Generic Classes

The three main uses of generic classifications of test in test planning & management are:

• Identifying the instances of each classification within the system.
• For each instance defining what is adequate testing of the instance.
• For each classification / classification instance bounding exactly what is to be tested.

Problems of clarity of understanding and communication are often encountered in each of these areas. These problems are:

• The Identification Problem
• The 'What Is Adequate Testing?' Problem.
• The 'What Is Being Tested?' Problem.

In addition the diverse structure of software creates another problem:

• The 'One-Off' Problem

These problems are discussed below.

The Identification Problem

The definitions are intended to be used to identify the instances of 'modules', 'sub-systems' etc. that exist within the system. This is relatively easy when the item equates to a language construct such as a function, class or package or even to an identified development deliverable coming from one team. It is far more difficult for other types of item. For example 'sub-systems' are harder to define. The result can be definitions containing large numbers of criteria with exceptions and guidance. Despite careful draughting there are always cases that are missed and disputes.

The relative ease with which some types of items, particularly language related ones, can be defined may lead to a focus on these as the targets for assurance activities when more appropriate ones exist. They are chosen because it was clear how to identify them and not because they were the things that needed testing.

The 'What Is Adequate Testing?' Problem

When defining the 'rules' for testing identifying 'functions', 'units', 'classes', 'modules', 'sub-systems', etc. is only one issue. The second one is that for each identified instance the rules need to answer the questions:

• Does it need testing?
• If so what types of testing?
• For each type of testing how rigorous should the testing be?

The need for this can be seen if a strategy includes 'functions' or 'class' as a type of target. A medium sized system may contain thousands of functions or hundreds of classes. A small percentage will warrant formal rigorous 'function' or 'class' testing - given real world resource constrains the vast majority will not be formally tested as standalone items but will be tested as part of larger assemblies. There may be a number levels of testing in-between these two limiting cases. So in this approach rules must be written to define the testing applicable to a particular 'function' or 'class'.

Defining these rules is extremely difficult. They tend to be complex open to interpretation and result in arbitrary cut off points. For example why should one additional method mean a class needs formal testing?

The' What Is Being Tested?' Problem

Identifying an instance of an item and the testing that is required on it still does not define what makes up the entity to be tested. An example will be used to illustrate this issue.

Given a test target definition of 'The AlarmManager class' the following questions arise:

• Does testing have to address the response of the AlarmManager if construction of a ClientAlarm instance fails?
• Does testing address the management of a population of ClientAlarm instances or the operation of these alarms up to the execution of the call-back method?
• Is the implementation of the ClientAlarm part of the software under test?
• Is the implementation of the system logging service part of the software under test?
• What about software called from the system logging service?

These questions must be answered in order to enable the design of appropriate tests and the creation of test builds that execute the tests on the right software. The questions are not generic ones they are specific to the particular item being considered. This results from the highly diverse structures found in software. There are few common patterns that can be used to identify generic questions and provide generic answers. Generally there will be unique questions relating to each 'unit', 'class', 'module', 'sub-system', etc. that must be answered before it is clear what is to be tested. The specific nature of these questions means that they can not be answered in advance in a strategy document.

The One-Off Class Problem

An additional problem with the use of generic classifications is that if real systems are analysed to determine what really should be tested then the result is not a small number of classes of things each with a decent number of instances. Rather the result tends to be a large number of classes of things each with a small number of instances (possibly one or two). This reflects the complex diverse structure of software and the arbitrary and chaotic distribution of risk with a system.

This is a major problem as Test Strategies based on generic classes are a poor fit. The result can be a square peg and round hole syndrome. The testing can, as best it can, follow the strategy and appear to test the system often with ineffective results. Alternatively the testing goes after some of the real targets but diverges from the strategy and operates in a more adhoc manner.

This would be fine if the route laid out in the strategy turned out to represent the optimal use of the available, and often limited, test resources. Unfortunately this is not likely to be the case as this would require an ability to see into the future and write the test strategy based on what will happen. Even if this were possible it would still leave the difficulty of adequately documenting / communicating the things the oracle foresees which is no easy task in itself.

The reasons that attempting to identify and define the best use of test resources in the planning phase will fail include:

• The actual structure of the software is not known. In fact even the generic patterns of structures, that is if the software will exhibit any regular structure, may not be known. The structure greatly influences where risks of internal malfunctions exist. Without knowing the structure and how it operates it is difficult to pick targets.
• Relative risks are not known. Even if the structure were clearly defined in advance the priority that should be given to each target is not the same. There are always limits on the resources available for testing. There are never enough resources to do everything. This means that it is not possible to test all of the targets. It is essential that the most important targets are chosen and this can not be done at the start of a project.
• At the system level functionality is poorly defined. Threats arise from the functional structure of the system and (potential) interactions between functions. The best set of system level test packages can not be chosen without analysis of these threats.

Writing a test strategy that goes beyond strategic test packages provides a false hope that the author can foresee the future and has thus provided a clear route forward for the test activities to follow. This should be avoided as the proposal becomes firm, takes on a life of its own. Following the strategy becomes the objective - rather than effective testing and effective use of test resources. The end result can be more missed faults and testing falling into disrepute as it is expensive and fails to add value to the project.