Software Test Automation

Automation of software testing is not a trivial task. What at first sight may appear to be a relatively simple task is a highly demanding problem domain. Development of effective automation solutions is a complex system development problem where the output is testware rather than software.

Success in software test automation depends upon:

• Recognition that effectiveness depends on appropriate architectures and frameworks.  Single layer standalone testware is only appropriate in the simplest of cases.
• An ability to design appropriate architectures for individual test packages or frameworks for groups of packages.
• A good partitioning of the core testware into domains and test components within the domains that allows re-use.
• Competent use of commercially available test tools including the ability to configure and 'program' the tools.
• The necessary development skills to allow any custom utilities to be developed along with the glue to hold the architectures together.
• Coherent test packages.
• Systematic implementation of test packages within the architecture / framework.
• An understanding of the appropriate level of automation in different situations.

In addition, where automation is first being introduced, there is a need to understand what to do first. This is both a technical / project management issue and a political issue.

For more information on test automation see Automated Software Test Suites.

Common Experience

Many attempts to introduce test automation fail. Many organisations state that they just can't get automation to work properly. Having tried automation the tools are returned to the shelf and manual testing resumes. Often this is due to a lack of one or more of the factors listed above. In particular attempting test automation without an architecture or with an inappropriate architecture is a major cause of failures.

Where automation is being practised it can be found that:

• The automated suites test / check what they can easily test / check. This may mean that they test low risk aspects and find low impact problems whilst failing to address high risk areas where failures have severe consequences. This can stem from a lack of a clear test package definition to act as the basis for the tests to be automated ( see Software Assurance Analysis )
• Levels of monitoring and fault detection applied to the software under test have wide variations.
• Execution of the test suite is unreliable with frequent stoppages requiring manual corrective action.
• Maintenance of the suites is expensive. Changes to the system result in wide ranging changes to the testware.

SQC Capabilities

SQC has experience of designing and delivering automated testing solutions for systems with complex patterns of behaviour. We tend to develop architecture based testing solutions rather than single layer standalone testware and use both commercial tools and bespoke software in the solutions ( see Automated Software Test Suites ). We have specialist knowledge on the Mercury Interactive WinRunner and LoadRunner testing tools. We have extensive experience of reviewing and testing software systems ( see Software Testing and Assurance ).

Our practical experience of test automation and testing provide a firm basis for the consultancy services we offer on test automation.  We can:

• Review and assess the current practice on test automation within an organisation.
• Review proposals for test automation.
• Work with clients to synthesise a way forward on test automation.
• Help to ensure that test automation is assurance requirements driven - 'X,Y,Z need testing so automate it' - rather than technology driven - 'we can automate A,B,C so we will'.
• Mentor client personnel.
• Assist in the formulation of appropriate architectures for test automation.
• Teach clients the approaches to test architectures that we have evolved.
• Provide support to clients using WinRunner and LoadRunner.