There was a time when big challenges in software development were mostly solvable by tools. It was a simpler time, before every desk and every home had a computer, and before most mid-sized companies were creating their own custom software and before many small companies would even consider building custom software.
It was a time that has come and gone, as have most of the tool vendors of the time. The tool vendors that are left standing have one lasting imperative: continue to make the issue of software development productivity an issue of tools.
Software productivity and quality have been suffering for it with a constancy that should have shocked business leaders into action years ago. Something has gone horribly wrong.
A reasonable and rational business response to poor productivity and throughput in software creation and operation should have already come. It should have been swift and decisive in proportion to the hemorrhage of opportunity and value in custom software efforts.
And yet, business leaders sit, passively resigned to what appears to be an intractable cycle - a notion reinforced throughout IT - that under-performance is the inevitable nature of software production and there's nothing that can be done about it. In the ten or fifteen years since the onset of the ubiquitous custom software boom, this sad story has persisted as the blindly-accepted lore that influences so much about what we believe about the ways and means of custom software production.
The reasonable and rational adverse reaction to software woes isn't happening as it should. We are resigned as a society to bad software and software projects that are even worse. We accept the pain that software developers rain down into our lives, whether in the form of custom software, commercial applications and operating systems, or indecipherable web sites.
And yet, the software crisis isn't really a hard problem to solve. The rehabilitation of software development and the reclamation of IT's reputation and credibility starts with the recognition that dramatic shifts took place while decision makers' eyes were necessarily taken off the ball. We're presently trying to solve problems with solutions that worked at one point, but no longer work. We're directing a generation of software developers to think about software development in ways that are out of date not merely by years but by generations.
Here's a short list of heretical ideas that has changed the game for every software developer and software organization that has successfully put them to work:
- Design quality is the most important factor influencing productivity in software development
- The things that obstruct quality degrade productivity
- The reductions in productivity over time that are typical to tool-driven software development are greater than what can be solved by tools
- The application of tools to these problems exacerbates the quality problem, creating a vicious cycle that accelerates exponentially
- Quality software design is the product of principles and practices, not tools
- The typical degradation in a software's quality over time isn't due to the nature of software, it's due to the nature of the approaches we choose to develop and operate software
We still need tools to support our software efforts, but when we return balance to software development, we find that we need fewer elaborate and expensive commercial tools. Looking back, we see most of the commercial tooling that we use as distractions, wasted capital, and excess.
The essential tools that we do need are far less expensive to acquire and to operate than the elaborate commercial tooling offered as solutions to the software problem. In many cases, essential tools are free, open source tools built to fill the vacuum left by entrenched tool vendors who cannot keep pace with the changing conditions and the evolution of software development in the wild. These tools are typically more mature than their commercial counterparts, and are supported by engaged and engaging software professionals, and they are crafted by people who do in fact use these tools on live-fire custom software projects.
It should go without saying that we should be wary of commercial tool makers who use their tools only to make other commercial tools. These tool makers typically have little to no recent relevant experience in the projects that they believe their successive generations of tools will be appropriate to.
Productivity degrades because software becomes harder to change as its complexity increases. Complexity increases naturally as more code is added to a system over time, be it code for brand new features, or for changes to existing code due to improvements, changes in the business, or due to fixing defects. There comes a point (and it comes quickly) when design flaws become so entrenched in software that they can't be resolved affordably.
The whole trick to tactical software design is an exercise in protecting a very volatile investment from erosion. And this means an unwavering vigilance toward design fundamentals and principles.
Deliberate and tactical software design keeps productivity bottlenecks from taking root in software. With bottlenecks, obstructions, and design friction in play, it becomes harder to move forward. Work items progressively take longer to complete, and one day the productivity is so poor that a total re-write is ordered. Typically, the new system is re-written using the exact same approaches that generated the conditions that lead to the need to re-write the system, and the vicious cycle begins anew.
The traditional productivity curve (which is the inverse representation of the traditional software cost curve) is a result of not protecting software from erosion. The exponential degradation of productivity shows the compounded effect of institutionalized negligence of the software development work that is in fact specifically geared to fend off erosion. Namely, continuous, incremental, design quality stabilization and improvement.
To believe that the traditional productivity curve is a natural part of software development is to indulge the same naive presumption that american manufacturers believed to be a natural law of production before Toyota showed that the presumption is tied to a specific production methodology - a methodology that still forms the basis for most software production methodology used today. If you fundamentally change the methodology, you'll change the rules and the equations that govern the productivity curve.
Some software designs are harder to work with than others. Some designs are even more prone to defects. If you arm your software organization with even a basic understanding of the fundamental design principles and the basic practices engendered by them, you would begin to see benefit immediately. Over time you could reshape the productivity curve entirely, creating more value with your investments in custom software, and deriving value for much longer.
But there are no tools that can do this for you. Not even the so-called software design tools are capable of helping you to apply fundamental software design principles. There are analysis tools that might help you understand where trouble spots might be, but they can't create supple design for you. To defer the responsibility of productivity to tools means that the real issues underpinning productivity will not be addressed, and not only will your desired productivity not be achieved, but you'll experience the complete opposite of what you had hoped for.
There will likely always be commercial tools that are used by software developers, but the real, essential need for these tools is far less than the excesses that we see today in vendor-dominated software development communities and cultures. We're on the cusp of a new era of productivity in software development, but it has very little to do with material investments in tools and everything to do with investment in mature, proven design principles and practices, and the readily-available, low-fi, essential tools that support them. And to this mix we add only the essential commercial tools that support our efforts.
The principles, practices, and yes even the necessary tooling to rehabilitate software development and to put an end to the crisis are already here amongst us. Many software developers have already reached out and harnessed them, and more are waking up to the essence of effective software development every day. Armed with the understanding that productivity comes from design, and that design is an intellectual activity that has only a slight dependency on tooling, a growing number of software developers are making huge strides in proving that our assumptions about traditional software development economics add up to little more than superstition.
One day this period in software development history will seem like the dark ages - a time where our primitive approaches to software development delivered commensurately primitive results. We'll look back and scoff at the now obvious mistake of serving the business needs of tool vendors rather than serving the business needs of the businesses we work for. We're under-performing to a shocking extent and hemorrhaging value at an alarming pace. This bleakness ends when we recognize the real source of productivity and reach out and grab it. The productivity we get from tools is merely a distant fallback position from the productivity that we achieve through software design.
To continue to languish at the mere levels of productivity that tools offer is a deeply-disturbing yet deeply-entrenched behavior in software development. As more software developers and organizations wake up to their true potential, these software dark ages can finally be relegated to history and we can move forward into a renaissance. Hopefully we can do it during this generation rather than continuing to be distracted by the endless, well-funded parade of software tool peddlers whose disproportionate success depends entirely on our willingness to remain distracted from our rightful potential.
