When should you classify a test as Failed? This sounds such a simple question and you may think the answer is obvious; however there are some factors that mean a well thought out approach can have significant benefits to the test manager.

If you want testing to be effective and want it to be manageable in the wider sense of the word (understood by others, amenable to peer and expert review and controllable) then everything has to be focussed. Each constituent part of the effort needs a clear purpose and this has to extend down to quite a fine grained level. Macros level building blocks such as Functional Test, Performance Test and Deployment Test don’t do it. What is required is to break the work into a set of well defined heterogeneous testing tasks each one focussing on certain risks.

I have just encountered an old friend of mine; one that I see most places I go. My friend is that recurring defect - the different date format bug. In its most common and insidious form it is a mix of DD/MM/YYYY and MM/DD/YYYY representations of dates as strings. Date format clashes of any sort cause defects but this is the worst ones because for many cases it appears to work waiting to create problems in future or corrupting data that passes through it.

We have recently started working with a new client on changes to their testing and delivery practice. The aims is to increase the throughput of development and at the same time accelerate delivery and maintain quality. This has been running for a few weeks now and enough time has elapsed for us to start hearing stories about previous projects and what went well and what was problematic.

I heard a comment recently; it went something along the lines of “if they can’t deliver testing to us then they won’t be able to do anything”. Was I surprised to hear this coming from a senior test manager? Well actually no; I wasn’t surprised. It illustrates that even people with many years in senior testing posts can fail to understand what first class testing is, how different it is from run of the mill work and how complex and difficult it is to do first class testing well and at speed. This was not the first time I have come across this view and I doubt it will be the last.

Here are two interesting propositions. Number one; test managers should focus on getting as quickly as possible to a state where it is obvious that further testing offers little benefit compared with finding out how the system survives in the wild. Number two; it is easier to make the decision to release a system when delaying the release to permit further testing is not likely to put you in any better position than you are already in.

After doing a fair bit of performance testing and troubleshooting we have seen the effects of performance only receiving attention at the end of the project. We encounter teams making herculean efforts to ring acceptable performance out of systems; we encounter systems that do not reach and never will reach acceptable levels; we encounter cancellations.