Testing and Reverse Engineering

Waiting to start testing software until after programmers are done writing the code it is really expensive. It's less expensive to have programmers do test-first programming, and to bring traditional system testing on-line before the programmers get started. Without putting a lot of measurement behind it, and relying only on experience and observation, end-of-line testing is likely to be easily twice as expensive, but likely several multiples more expensive.

Test-last development involves reverse engineering, and reverse engineering is really expensive.

Typical software production schedules simply don't have room for the kind of thoroughness that allows the reverse engineering to be a reasonable approach. The results of reverse engineering in test-last development in software are always incomplete. The incompleteness leads to defects that are found later rather than sooner which causes rework on the parts of the software that is defective and all the parts of the software that has been written in the interim that touches the defective parts. The rework interrupts the flow of value-add work on the product, makes the schedule unpredictable, de-stabilizes the team's focus, and delays delivery.

When a tester sits down to test a new feature, he first has to figure out what how the feature works, what it does, and what kinds of observable results that using the feature causes that can be turned into tests and test plans. That process is reverse engineering. Even if the programmer communicates the various execution paths and usage scenarios for the feature, some percentage of execution paths will not be communicated to the tester, or will not be perceived by the tester,
which further extends the amount of necessary exploratory testing he'll do - yet more expensive, time-consuming, inaccurate and incomplete reverse engineering.

When tests are written before coding is done, both as acceptance test-driven development and developer test-driven development, the software is delivered to final inspection with a thorough set of executable explanations of what the software does. Testers then know what the software does and how to set up the conditions to put it through its paces and what observations should be made about the effects of putting the software through those paces.

Even if developers are writing unit tests, if they are writing them after they write the functional code, they are incurring the same kind of reverse engineering expense that testers are struck with. It's a smaller form of the reverse engineering expense, but the instances of it are much more pervasive. While each individual cost may be small or even arguably negligible, the sum of all these pervasive instances of small costs is quite large, and definitely not negligible.

We think of writing tests after writing code as a natural order. The opposite is true. Writing tests after writing functional code is reverse engineering, and writing tests first as specifications and as proofs is the natural order.