Automated Side by side Report Testing - This client makes is money by analysing customer data and providing specialist reports, using its models, from that data. The combination of types of data, types of analysis and report options mean there are hundreds of reporting scenarios. When the solution that collected and pre-processed the data was being replaced it was essential that the new system would produce exactly the same output, both content and presentation, for a given situation as the previous one. It was recognised that testing this was a huge challenge, demanding a specialist approach and the client called us in to address this. The nature and architecture of the system meant that the risk could only be addressed by executing the real-world data capture process on the new system, right from the vary start, and then generating reports that could be compared with a, presumably, identical one from the old system.

Test Solution Real-time Control Device - A software component that triggered and then sustained or deactivated an actuator based on real-time analysis of a set of input signals needed testing. Millisecond level timing precision of activation and deactivation was required. Input traces sequences were complex. We developed a harness to apply input traces to the software, running on its target hardware, and that recorded the output sequence this solicited. A separately implemented oracle processed these outputs offline. The oracle checked the compliance of the output trance with the input trances for each millisecond of the run.

Telecoms Order Management Reliability - A telecommunication stack is a challenging automation environment. Triggering events come from many sources. Processes are long running and asynchronous. Data and business rules are complex. Provisioning can involve many systems. Its complex to get an order through a test solution under the best of circumstances. Processes are very sensitive to the data used. Timing can be an issue. Experienced human SMEs can struggle to get this right. Achieving repeatable consistent outcomes with test automation is challenging. Multiple factors covering environment management, test data configuration and provisioning across diverse systems, execution discipline plus sound general automation have to be right for automation to be dependable.

Destined to go straight into high volume use the reliability of the stack would be critical. However in a noisy test environment where failures occurred due to interference with test data and other environmental factors spotting reliability problems was going to be difficult. An overloaded development organisation was all too ready to declare “human error” or “test data” as the reason for a failure. Spotting reliability issues would depend upon analysis of the trend in success rates across large numbers of orders. The art being to spot clusters of failures that stood out from the random failure noise. A continual flow of orders, that could be relied upon to be correct in themselves, was needed. This demanded a continuously operating test automation solution.

We created this capability, bringing together a robust environment engagement model, an integrated test data and end point simulator architecture, solid execution management practices and a flexible robust approach to the actual automation of interactions with the system. On top of building the technology and team to deliver this we championed reliability. Spotting the early indications of issues, focusing in to crystallise the issue and driving the delivery of fixes to these intractable issues.