Load testing

+44 (0)207 993 2287
Load is an "Achilles Heel" of today's solutions. It can create a poor experience, it can bring a system down. This risk has to be dealt with in a systematic and disciplined way. Both design and assurance need to worry about load and its effects.

The impact of load the importance of response

Today’s systems, large scale, distributed, perhaps running out of the cloud, perhaps incorporating third party APIs, possibly exposed to an unpredictable user base, are very vulnerable to load related threats. When they are “busy” performance can start to falter and strange functional anomalies can occur. Even under “light” load the impact of more than one thing happening at the same time can cause things to malfunction. Ultimately, under “stress” they can grind to a halt. Problems caused by load are seen by most engineering teams.

Doing nothing or going through the motions

It is important to take an active and effective approach to addressing the risk from load. A cosmetic or last minute exercise means slipped dates and problems in production.

The flawed “final test” approach

As with most assurance, relying on a simplistic “all encompassing” test of “everything” just before go-live is a poor plan. Layers of activity, using different approaches, are needed to detect different types of problems. Work needs to be done during the formative and build stages of the project rather than at the end to make certain there is time to fix things. To be effective, assurance needs to run for the full duration of the project alongside the core deliver activities. It can’t be a bolt on.

The building blocks

A sound approach must encompass elements drawn, as appropriate, from the following disciplines:

  • Profiling of the load in different operational periods
  • Design vulnerability assessment
  • Modelling of the working of the system and the volumes involved
  • Localised workload-component and sub-system load testing
  • Tactical testing of high risk scenarios
  • Full breadth simulations of operation under load

The challenge? Failed attempts and wasted money

In a nutshell, delivering this type of assurance is hard. A lot of time and money has been wasted by organisations on ineffective work. Some teams can’t get tests to work at all. Some lack the technical depth to monitor, analyse and draw conclusions. Some get great results, on the wrong data. Some don’t analyse and understand work loads and user behaviour.

Why do these things happen? It is because the teams don’t have the right resources, the right concept of what needs to be done, the required technical depth and the understanding and credibility to get stakeholders to recognise what is involved in doing things correctly.

It is a job for the specialists

Time and time again it can be seen that this is a job for specialists. We have seen that attempts to “do it in-house”, using the development team, or to hand it off to a systems integrator or one of the big consultancies generally fail to deliver. On more than one occasion we have been called in to recover such situations. It is not a job for generalist test consultancies or suppliers, these organisation lack the necessary technical capabilities. It is a job that demands a multidisciplinary team with a history of delivering the full spectrum of tasks that will be required.

Why us?

We have the people, skills, experience and approaches needed to work with you to address your exposure to load related risk. We have delivered across a wide variety of solutions and organisation types. We have worked with “standard” website, high volume transactional portals, APIs, telecoms network elements, financial software products, ERP systems, CRM systems, provisioning systems and billing platforms.

We don’t just address end user activity. In our time we have tackled interface and integration throughput, period end processes, billing cycles, batch processing services, workflow engines, data transformation scripts, reporting services and Citrix systems. We have tested sub-systems and backend services, we don’t need a nice user interface to be able to operate. We don’t just do one hour performance tests, we emphasise the need for extended runs, to check for consistency and sustainability, and for running under stress, to look for weakness.

We have the experience needed to make this work.

We can offer

We don’t just test. We size, analyse, assess risk, target, test and remediate. Often we have to form the non-functional requirements. We help solve the problems we raise. We are not bound to standard tools, where needed we build custom drivers and harnesses to test the “untestable”.

We offer activities tailored to your specific needs. Whether you want a health check, advice on how to proceed, help fixing in-flight work, assistance with a specific performance issue, sanity testing, a full test campaign or oversight of a supplier’s activities, we can work with you to meet your needs in an effective and efficient way.

For the Web

Transactional portals and online services

Transactional websites with links to backed systems that allow people to perform important, must not be dropped or duplicated, transactions require careful attention. People rely on these services and will not tolerate failure or performance problems. They can also be difficult to test. Data dependencies can be complex and data management is a demanding discipline. In greenfield situations we have had to generate large scale diverse data sets to be loaded to enable load testing.

Our team has load tested portals for the public sector, for large commercial organisations and for organisations providing complex niche online services. This has involved developing data generators, custom load profile generators, sophisticated simulation frameworks and a custom HTTP injector that is far in advance of mainstream injectors. We have enabled clients to harden their systems and prepare them for use in the real world.

In it for the long term

We have been working with one of our clients, off and on since 2004 and continually since 2010. We have worked with them on multiple systems. Since 2010 we have provided a loaded testing service that has underpinned the development and operational integrity of two generations of their flagship digital assessment solution that provides service to their UK and international clients.

For ERP

The AX challenge

Microsoft AX2012 presents both AX specific and common ERP load testing challenges. Specifically AX’s tightly integrated desktop client server architecture has to be intercepted. In common with other ERP implementations multiple diverse file and web-service interfaces have to be simulated.

A scalability test lab

Faced with a client’s scalability questions we created a diverse test framework around their AX solution. The nature of the AX product architecture and their integration topology meant we had to develop five completely different load injection techniques. These techniques, complimented by utilities we created for data mining and system monitoring established the “scalability test lab” allowing us to explore the scalability domain.

Scalability questions

Using our test lab we were able to explore and analyse topics such as: The resource consumption of “high cost” transactions. Navigational response times and the capacity of each AX AOS server. Throughput and dependability of the integration interfaces. The capacity of the Citrix presentation of the AX solution. The interplay of different workload elements. The operation of batch jobs and reports. The impacts of failures in the integration layer.

CRM and Billing

Critical systems

Telecoms companies depend on their CRM and Billing systems. If the CRM is down they can’t talk to and service their customers. If billing doesn’t happen their revenues suffer. These are critical systems.

Monoliths

Where as their networks are a mesh of many replicated elements, their CRM and billing systems very complex very large scale single instance software systems. They are very vulnerable to performance, reliability and availability issue.

Stabilise and protect

Out tier-1 telecoms client had been suffering a service crisis. Their new CRM and billing system was unstable, non-performant and unreliable. Their non-functional assurance and testing was cosmetic and ineffective. We stepped in and assumed the leadership of a multi-vendor NFT setup. We redefined the approach of the team, set new, higher, standards and directed test campaigns to stabilise the system and protect it from change.

The benefits

Gradually the tanker turned around. People used to spending three or four, long days per week in production incident “war rooms” found they got their time back. Changes that would have caused production problems were identified and fixed or removed. The vast clean-up costs that incidents created went down. A semblance of normality emerged.