Introduction

This section provides a very brief overview of Assurance Analysis and attempts to explains its position within the assessment of the implemented system. The approach provides a flexible incremental method for managing the assessment of the software. It brings to software assurance management the recognition that a water fall approach based around a test strategy is not effective ( something recognised in the overall software development process a long time ago ). The approach is particularly suitable for use with modern incremental risk managed development lifecycles.

Rationale

Why apply this approach in place of a comprehensive definition of the testing to be done in the Test Strategy? The reasons include:

• Test strategies typically define classes of system test - functional, stress, performance etc. - whilst the conditions under which the system is to be operated may be reasonably well defined the things to monitor and failures to look for are often less well defined. Lack of clarity of scope and oversights can lead to incomplete system testing in some areas and overlaps in others.
• Test strategies typically define classes of testing of the system internals - unit, component, sub-system, integration, etc, together with guidelines for each. Real software does not exhibit this regular structure. In some cases it is essential to extensively test a single function whilst for the same system it would be a waste of time to specifically test hundreds of other functions or even complete classes. It is far better to identify something that needs to be proved / tested and then determine how to do it than to attempt to predict the types of things to be done.
• A strategy containing unit, component, sub-system, integration and system test tends to result in an attempt to do testing in this order. Test the units, build the components, test the components and so on. Often this is not the optimum order in which to do things. A component may need complete rigorous testing prior to system release but it may be appropriate to use it in other formal builds long before this to enable other work to move forward.
• A related issue is that the internal testing levels tend to cause people to attempt to test in a build consisting primarily of the target for the test activity. Thus if a class is to be tested then there is an inevitable tendency to create a test harness around the class. This may not be the most effective approach. The target of a test package may be a class but this testing could be done in a full system build so long as it is possible to ensure the class receives appropriate inputs and its behaviour can be adequately monitored.
• Engineers developing the suites of tests for a test package often interpret the description of the work to be done in different ways. This is because they are given a very high level abstract description of what the package is to do with no concrete requirements. When software is described in high level terms with no concrete detail the results are often highly variable. Similarly the results for test packages defined in this way are highly variable and often do not implement the tests intended by the strategy author.

The approach described in this section provides an improvement over the traditional test strategy based approach when judged against the issues described above. It involves looking at what needs to be proved / tested before selecting test packages, it uses concrete definitions of test packages selected for a specific purpose rather than generic classes of testing and it provides a more systematic approach to defining tests for large sets of functionality.

An additional benefit of this approach is a more systematic approach to designing test sets. Whilst the use of systematic test design techniques for testing in the small is relatively well understood, there is little systematic practice for developing tests for larger collections of functionality. The use of progressive analysis throughout all activities leading to the definition of a test helps to address this.

The Analysis Led Approach

The diagram below illustrates the key elements and organisation of the process.

Overview

There are two technical areas of activity - Investigatory Analysis and Package Analysis. The first, Investigatory Analysis, consists of standalone analysis activities performed to identify, at an intermediate level of detail, what needs to be tested / proved to assure the quality of the system. The second, Package Analysis, is performed as part of delivering each test package. It involves refining the intermediate level assurance requirements allocated to the package to add more detail and formality. This refinement process acts as bridge to test design in the small.

In addition to the analysis activities there is also the need to direct the process. This consists of two activities - defining the investigatory analysis packages to be undertaken and selecting and defining test packages & targeted reviews.

The process of choosing and defining the Investigatory Analysis packages operates throughout the life of the project though there will be an emphasis on this when planning each development cycle iteration. A package definition addresses what operational aspects are to be covered, the parts of the system to be covered, the primary sources of information to be used and the depth of the analysis. The aim is to choose packages that lead to a high quality complete Core Assurance Requirement Set to act as a basis for planning subsequent assessment activities.

Test package selection uses the Core Assurance Requirements Set as a primary input. Again this is an ongoing process. Nominally the aim is to cover all Assurance Requirements in a package ( in reality prioritisation and cut off points apply ). Clusters of requirements are identified using criteria such as - similar 'type' of test conditions, focusing on similar 'types' of failure, suitable for testing in the same builds, suitable for testing at the same time, of equal priority. A cluster forms the basis of a test package and this leads to clearly defined packages that cover a naturally related set of objectives.

In cases where an Assurance Requirement is not suitable for dynamic testing a targeted review may be selected. This method helps to ensure that key checks are not omitted whilst also ensuring that effort is not wasted creating tests that appear to do the check but that in reality do not achieve this.

Investigatory Analysis

A key feature of the Investigatory Analysis work is that the focus and scope is totally independent of any potential test suites. That is investigatory analysis is never "Analysis for the account management tests to be done on the system" but might be "Analysis of the information contained in the account management specification focusing on (a) general functionality (b) levels of concurrent use & work load (c) response times and (d) recovery from system crashes". This example focus would result in Assurance Requirements being allocated to a number of different test packages some primarily focused on account management and others focused on different issues but touching on account management.

Each Investigatory Analysis package has a focus. As the aim is to ensure as complete a Core Assurance Requirement Set as possible diverse types of analysis are performed. Some packages focus on project documents / artifacts others on particular sorts of usage of the system some on internal operations of the system some on a particular type of failure. The aim is to analyse from many different perspectives to provide as complete a picture as possible.

Assurance requirements aim to provide a definition of something to be checked. They record the conditions / usage to be addressed together with the things to be monitored and the failure to look for. As the process progresses high level assurance requirements expressed less formally are superseded by sets of more precise more detailed requirements expressed in a more formal manner. At the lowest level they become the specific requirements for tests. There is a gradual traceable progression for high level objectives to detailed test that lends itself to effective quality control and management review & direction.

Assurance Analysis progresses in stages from informal high level analysis at the start of Investigatory Analysis through to formal detailed analysis at the end of Package Analysis. Investigatory Analysis completes at some point along this spectrum. Where it stops varies from package to package. The primary criteria for stopping is that enough analysis has been done to identify all that needs assuring and it has been done in enough detail to allow appropriate allocation to packages and to adequately define the work of the test package.

As the Core Assurance Requirement Set expands it forms the basis for the selection of test packages intended to find & eliminate faults in the system ( additional test packages may also be created for other reasons - see Selecting Test Packages ). A test package is initiated because there is a collection of Assurance Requirements that are suitable for testing as a single package and that warrant the effort involved in delivering the package.

Clusters of Assurance Requirements cause the initiation of test packages. Isolated requirements and small clusters that do not warrant a standalone package may be allocated as supplementary requirements to another package. Where requirements are not suitable for testing or where neither a standalone package or allocation to any existing package is appropriate a targeted review can address the requirement.

This process can result in all the 'standard' classes of testing traditionally addressed in test strategies. It can result in packages that combine classes together for example functional and performance testing of a sub-system. It can result in separate packages where traditionally there would be one - for example instead of a single component test for component X there could be separate functional, performance and security tests. The process can also result in extensive testing of certain key functions or classes whilst others are not individually tested.

This active targeting approach copes well with the diverse structures and location of risk within a software system. The use of Assurance Requirements as the main basis for selection allows a more informed choice of test packages.

Package Analysis is performed as part of the delivery of each test package. The definition of the package allocates Assurance Requirements to it. These requirements will be at various levels of detail depending upon their source. Most will be at an intermediate or high level, few will be detailed enough to enable direct test definition.

The Package Analysis process takes the assurance requirements and refines them. They are partitioned, more detail is added and they are expressed more formally. The aim is to develop the requirements to the point that translation into test cases is relatively mechanistic using common well understood test development principles such as equivalence partitioning and boundary value analysis.

Refinement has two aspects one is partitioning the requirements, formalising them and selecting minimum coverage requirements by applying verification case selection principles. The second is clarification and formalisation of the higher level concepts or partially defined information contained in the requirement. This clarification aspect requires analysis of appropriate documentation etc. to obtain the necessary information.

Caveats

Need for Active Management

The process is no longer an upfront plan & execute to plan process. The work to be executed emerges as the project progresses. Planned work needs constant reviewing and revising as assurance requirements are identified. Management of the process is more demanding than in a traditional test approach. The manager requires an in-depth understanding of all the issues associated with testing a software system.

Estimating

Estimating for the project is more difficult as the testing to be performed is not defined when the estimate is produced.  This is awkward. However the solution is not to go back to pre-defined tests that are inefficient and ineffective. It is an issue but maybe not as big an issue as at first appears. In the real world there are restrictions on the time and resources available for testing. Estimates may have limited impact on the level of resources provided for the task. In these circumstances this assurance led approach leads to a more effective use of whatever resources are provided.

A Complete Test Strategy is Necessary to Ensure Other Parties Test Correctly

In large projects the Test Strategy may be intended to allocate responsibility for testing to different functions. For example requiring development teams to do unit testing before release. If there is no requirement for unit testing in the test strategy then nothing requires the development team to test before release.

Whilst allocation of responsibility between functions is necessary attempting to do this by defining generic 'classes' of testing and allocating these to functions is not that effective. All of the problems described above apply and the 'definitive' rules are in-fact full of holes.

There is no easy answer to this issue. Ensuring adequate testing within functions / teams that are not dedicated to testing is a cultural issue.