We reform how you...
Today’s systems, large scale, distributed, perhaps running out of the cloud, perhaps incorporating third party APIs, possibly exposed to an unpredictable user base, are very vulnerable to load related threats. When they are “busy” performance can start to falter and strange functional anomalies can occur. Even under “light” load the impact of more than one thing happening at the same time can cause things to malfunction. Ultimately, under “stress” they can grind to a halt. Problems caused by load are seen by most engineering teams.
It is important to take an active and effective approach to addressing the risk from load. A cosmetic or last minute exercise means slipped dates and problems in production.
As with most assurance, relying on a simplistic “all encompassing” test of “everything” just before go-live is a poor plan. Layers of activity, using different approaches, are needed to detect different types of problems. Work needs to be done during the formative and build stages of the project rather than at the end to make certain there is time to fix things. To be effective, assurance needs to run for the full duration of the project alongside the core deliver activities. It can’t be a bolt on.
A sound approach must encompass elements drawn, as appropriate, from the following disciplines:
In a nutshell, delivering this type of assurance is hard. A lot of time and money has been wasted by organisations on ineffective work. Some teams can’t get tests to work at all. Some lack the technical depth to monitor, analyse and draw conclusions. Some get great results, on the wrong data. Some don’t analyse and understand work loads and user behaviour.
Why do these things happen? It is because the teams don’t have the right resources, the right concept of what needs to be done, the required technical depth and the understanding and credibility to get stakeholders to recognise what is involved in doing things correctly.
Time and time again it can be seen that this is a job for specialists. We have seen that attempts to “do it in-house”, using the development team, or to hand it off to a systems integrator or one of the big consultancies generally fail to deliver. On more than one occasion we have been called in to recover such situations. It is not a job for generalist test consultancies or suppliers, these organisation lack the necessary technical capabilities. It is a job that demands a multidisciplinary team with a history of delivering the full spectrum of tasks that will be required.
We have the people, skills, experience and approaches needed to work with you to address your exposure to load related risk. We have delivered across a wide variety of solutions and organisation types. We have worked with “standard” website, high volume transactional portals, APIs, telecoms network elements, financial software products, ERP systems, CRM systems, provisioning systems and billing platforms.
We don’t just address end user activity. In our time we have tackled interface and integration throughput, period end processes, billing cycles, batch processing services, workflow engines, data transformation scripts, reporting services and Citrix systems. We have tested sub-systems and backend services, we don’t need a nice user interface to be able to operate. We don’t just do one hour performance tests, we emphasise the need for extended runs, to check for consistency and sustainability, and for running under stress, to look for weakness.
We have the experience needed to make this work.
We don’t just test. We size, analyse, assess risk, target, test and remediate. Often we have to form the non-functional requirements. We help solve the problems we raise. We are not bound to standard tools, where needed we build custom drivers and harnesses to test the “untestable”.
We offer activities tailored to your specific needs. Whether you want a health check, advice on how to proceed, help fixing in-flight work, assistance with a specific performance issue, sanity testing, a full test campaign or oversight of a supplier’s activities, we can work with you to meet your needs in an effective and efficient way.
Transactional websites with links to backed systems that allow people to perform important, must not be dropped or duplicated, transactions require careful attention. People rely on these services and will not tolerate failure or performance problems. They can also be difficult to test. Data dependencies can be complex and data management is a demanding discipline. In greenfield situations we have had to generate large scale diverse data sets to be loaded to enable load testing.
Our team has load tested portals for the public sector, for large commercial organisations and for organisations providing complex niche online services. This has involved developing data generators, custom load profile generators, sophisticated simulation frameworks and a custom HTTP injector that is far in advance of mainstream injectors. We have enabled clients to harden their systems and prepare them for use in the real world.
We have been working with one of our clients, off and on since 2004 and continually since 2010. We have worked with them on multiple systems. Since 2010 we have provided a loaded testing service that has underpinned the development and operational integrity of two generations of their flagship digital assessment solution that provides service to their UK and international clients.
Microsoft AX2012 presents both AX specific and common ERP load testing challenges. Specifically AX’s tightly integrated desktop client server architecture has to be intercepted. In common with other ERP implementations multiple diverse file and web-service interfaces have to be simulated.
Faced with a client’s scalability questions we created a diverse test framework around their AX solution. The nature of the AX product architecture and their integration topology meant we had to develop five completely different load injection techniques. These techniques, complimented by utilities we created for data mining and system monitoring established the “scalability test lab” allowing us to explore the scalability domain.
Using our test lab we were able to explore and analyse topics such as: The resource consumption of “high cost” transactions. Navigational response times and the capacity of each AX AOS server. Throughput and dependability of the integration interfaces. The capacity of the Citrix presentation of the AX solution. The interplay of different workload elements. The operation of batch jobs and reports. The impacts of failures in the integration layer.
Telecoms companies depend on their CRM and Billing systems. If the CRM is down they can’t talk to and service their customers. If billing doesn’t happen their revenues suffer. These are critical systems.
Where as their networks are a mesh of many replicated elements, their CRM and billing systems very complex very large scale single instance software systems. They are very vulnerable to performance, reliability and availability issue.
Out tier-1 telecoms client had been suffering a service crisis. Their new CRM and billing system was unstable, non-performant and unreliable. Their non-functional assurance and testing was cosmetic and ineffective. We stepped in and assumed the leadership of a multi-vendor NFT setup. We redefined the approach of the team, set new, higher, standards and directed test campaigns to stabilise the system and protect it from change.
Gradually the tanker turned around. People used to spending three or four, long days per week in production incident “war rooms” found they got their time back. Changes that would have caused production problems were identified and fixed or removed. The vast clean-up costs that incidents created went down. A semblance of normality emerged.