Software Test Automation

The Complex Nature of Software Test Automation

Effective software test automation is a complex and demanding problem. There is wide variation in requirements from package to package. An automated solution must address stimulating and monitoring the system. It must provide appropriate automatic test data selection and assessment of the results.

The solution is not simply a single layer of testware that interacts with the software under test as the test is executed. A solution can include pre-execution tools / utilities, operational infrastructure and post-execution analysis tools. Pre-execution tools can prepare data for the test. The infrastructure supports the test as it executes helping with test data selection and on the fly assessment of the results. Post execution analysis tools can perform more detailed offline analysis of the behaviour of the system.

Test solutions must have functionality to detect anomalies that are not directly related to the inputs they are applying to the software. They need to be resilient when anomalies occur - large test sets must not be stopped in their tracks by the first anomaly. In addition the solution must be adaptable, as the system changes it must be possible for the test solution to track the system in a cost effective and timely manner.

These issues ( see The Complex Nature of Software Test Automation for more details ) mean that effective test automation solutions only arise from good design and careful implementation. A relatively simple record and replay approach may appear to work in the short term but can only be really effective in the simplest of cases.

Design & Implementation of Test Suites

The most suitable form for a test solution will depend upon the factors outlined above and is discussed in more detail in The Complex Nature of Software Test Automation. A test solution consists of generic testware that is not specific to particular tests and test specific testware. Testware is like software in fact testware is a software system that tests another software system. In all but the very very trivial cases testware must be designed. Where the problem is anything but simple an appropriate architecture will be required.  For problems of medium complexity and above the architecture may need to provide a generic test framework that is adaptable and re-usable.

The implementation of a test solution can use commercial testing tools, standard utilities, bespoke utilities or tools and solution specific 'software'. Virtually all real-world solutions even those solely based on commercial testing tools will require some development of 'software'. This may be in the form of scripts and / or configuration files or development of true software. For major packages the development of the architecture or framework can be significant.

On major projects cost effective solutions are achieved by developing generic frameworks that can be used by multiple test solutions. The target software is wrapped in a framework that the individual package based test solutions use. This reduces both the effort required to develop the packages and the effort required to adapt to system changes.

SQC Capabilities

SQC have developed a range of automated test solutions. Our experience goes beyond GUI user interface based testing of applications. We have delivered automated solutions for end to end testing of data communication systems, embedded systems, sub-systems, modules and components. Test suites have tested software with highly complex behaviours including autonomous activity embedded within the target.

We have implemented solutions using mixes of commercial tools, standard utilities, custom test tools written by ourselves and solution specific software. As Mercury Interactive partners we have detailed knowledge of and access to their marketing leading testing tools such as WinRunner and LoadRunner. We have a collection of our own re-usable libraries and utilities aimed at speeding the production of test solutions.  We can use these as building blocks within test solutions we develop.

SQC can provide services to:

• Automate test sets that have been defined by the client.
• Provide both the definition of test sets ( see Software Testing and Assurance ) and an automated implementation of the test set.
• Develop and deliver architectures and generic test frameworks which client engineers can then use to implement individual automation solutions.
• Develop special software utilities / tools to be used as components with a test architecture.

We produce innovative approaches that focus on testing the risk within the software. To do this we make certain we understand the rationale behind an automated test - that is the risk it is addressing. Our solution then ensures this risk is addressed and often the best solution is not the obvious one.

Our capabilities allow us to provide solutions addressing all types of testing. Solutions can target systems, sub-systems or elements within a system.  They can be designed to support simple functional testing, advanced functional testing such as concurrent overlapping interactions and non-functional tests. Non-functional test categories include reliability testing using stress / endurance approaches and throughput / resource utilisation assessments.

We can undertake these types of tasks as standalone turnkey development tasks performed on a sub-contract basis. Alternatively we can work with client personnel providing technical leadership and direction to joint development efforts. Even when working on a standalone sub-contract basis we are happy to involve client personnel in the design and development process so where required they can take on the maintenance of solutions we provide.