#63-Shift left - Detecting Defects During Testing

We're shifting left, and the next shift involves uncovering bad or missing behavior during the traditional testing phase.  Such testing is important and necessary, but is it enough?  We'll see.

Transcript

Shift left: Detecting Defects During Testing 

Last week I discussed the limitations of attempting to detect problems when the system is being sold and/or deployed.  The overarching concept of this is that detecting defects before the customer begins to use the product may still cause damage to our reputation and bottom line, but it prevents us from doing damage to our customers and their businesses.

This week I want to discuss the possibility that defects can be found during the testing phase that has traditionally been part of the product development sequence. In fact, this is where most people expect defects to be uncovered, because this is seen as the purpose of testing overall. 

Traditional testing, often referred to as “quality control” or “quality assurance”, is conducted when the product is theoretically complete. It is seen as an inspection process that will reveal missing or incorrect behaviors based on the requirements and expectations that were established in the initial product development phases. 

But there are a number of problems that can arise from waiting until this point to subject the system to extensive scrutiny.

First of all, in most organizations the testing effort lags behind the implementation effort significantly. Agile reduces this because we don't wait nearly so long to begin testing, but it is still an issue. Because of this, and because testing is conducted nearer to the product’s deadline, often some testing simply never happens. By the time testers have determined the extent to which the product is compliant with requirements they may well be short on time, and therefore some more important aspects that are not specified in requirements are either left untested or are inadequately tested. Various “-ilities” such as performance, security, scalability, usability, and maintainability are routinely missing from the requirements that testers work from.

This is not because these things cannot be specified, it's simply that people often fail to do so because they don’t think of it, or because they don't know how. 

This excessive burden is compounded by the fact that defects that remain undetected during implementation and integration may themselves in fact cause other defects in a cascading fashion. Also, I will be examining the differences in defect detection versus defect prevention, and the clear advantage of the latter. But defect prevention is almost solely a matter of analysis and design, which has already been completed by the time testing has begun. 

Put another way, it is not enough for the testers to shine a light on the quality of implementation, they also must be able to evaluate and confirm that analysis and design were correct in the first place. Once again, such issues are often left untested because of time. 

If a defect is detected by the testing process, it is then handed back to the implementation team to correct. Because this will happen significantly after the team has completed its work, especially in a waterfall process, then the issue will take much longer to investigate, identify, and correct.  If the defect turns out to be the fault of the design, then it may well be impossible for the implementation team to identify it.

The myth of the waterfall is that the phases proceed from analysis through to testing, followed by deployment. This is only true if testing never uncovers any problems to be corrected, but it always does. So testing is not followed by deployment, testing involves a loop back to an earlier part of the process. This can happen multiple times before the problem is resolved, which means that the product is almost certainly going to be delivered late,

Traditional testing is important, even if the earlier phases are test-driven, and so while it is necessary, it is not sufficient. Next week I will examine the next step in shifting left, which is defect detection during the integration phase.

I hope you'll join me.