Wednesday, February 17, 2010

Speaking at the Lean Software and Systems Consortium Conference in April

I'll be speaking at the Lean Software and Systems Consortium Conference, taking place in Atlanta from April 21st to 23rd.

My talk addresses the more egregious losses of productivity that come from working with HTML specialists in rapid startups, and how to shape the work, workflow, and team system to leverage the power of specialists without incurring the specialist tax.

Abstract:
Web designers are highly-specialized professionals. We lose a lot of productivity due to the effects of this specialization, whether through rework, scrap work, relearning, or missed expectations. Rapid startups expect to be up and running within two months, from the start of development work to business launch. Web designers are critical members of web startup teams, and learning to deal with web design specialists is vital to a rapid startup’s ability to sustain its pace. This presentation talks about two web startups that applied lean thinking and pull and flow to this particular challenge, and the techniques and understanding that came from the experience.

Check out the full agenda on the LeanSSC conference website:
http://atlanta2010.leanssc.org/agenda



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Wednesday, February 10, 2010

Productive by Design

(Continued from: How the Mainstream Lost Software Development Productivity)

Productive designs are compartmentalized. The parts of your software are shaped to the way that your mind holds on to them. "Abstraction" means the same thing to software as to thinking. "Concern" is another word the shows how software reflects thinking. A software "concern" is something that I have put my focus on that is my present concern.

The compartmentalized bits of software are the shapes of your software's parts. The shape of your software and its parts is only one aspect of your software's design. The shapes of your software's parts is a "static design". "Schema" is another word for it. The other aspect of your software's design has to do with what happens when we bring the parts together, give them a problem to solve, and give them the green light.

We lose software development productivity to relearning and to misunderstanding. The design quality issues that makes it hard to learn, re-learn, and understand your software also make it hard to understand how it works and whether it works. Powerful tools that optimize the moment of creation of the parts of your software enhance your ability to lower productivity.

"Module" is another word for "compartment". A class is a module. Modular design has to be productive not only in its shape, which helps us understand how the software is broken down into concepts, but also in how those parts do some work, and how they work together, and whether they leave their tools out where you can trip on them in the dark.

Software is a machine. It's not a machine like your car's engine or your cordless drill, or like any machine that is made with material. Analogies to physical machines break down almost immediately. There are no machines in physical space that are anything like software machines. They're machines that run individuals' personal way of thinking of how to solve a problem. There are as many ways to program a solution as there are programmers.

Software is a machine. It's made of parts. The word "soft" in software says that it's easy to not only change the working of the parts, but also the shapes of the parts, and to even move the workings to a different part (which often then tells you to change the shape again).

The first great matter of software development is proving that the new shape of some part of your software will still do what you expect it to do when you turn the machine on. The second great matter of software development is whether the shape that you've given to the software will continue to make sense, and that the workings of the software are placed in the right parts.

There's one gigantic difference between working on software machines and working on physical machines that bears mention:

The parts of physical machines wear out, and we replace them with another, new copy of the same part. This doesn't happen with the software parts that you make. When we change software parts, we don't replace ones that have worn out with new parts from the same mold. When we change software software parts, we change the very shape of those parts, often re-shaping the parts around them as well, and moving the workings of one part to a different part, or even a new part created on the spot as part of a new compartmentalization. In relation to a physical machine, software is much more like the mold than the product. Software development is closer to die making than stamping.

Every time you make a change to a software machine, there's a good chance that the change shows up in an adjacent part, and there's a good chance that it'll show up when you're not looking at it. It's a lot easier to get the code you're looking at right than the code your not looking at. That's what "control and observe" is talking about.

To increase the chances of software working as expected after you've tried to turn a part of it into a hat, a broach, or a pterodactyl, you make observations of it. If the only way that you can observe the functioning of the part is to bring the whole machine online, your productivity is many times less than what it should be. If you can make easy observations of the part that you're working on (and maybe parts in the vicinity), you can also make those observations while you're working on it so that not all of the steps are wild leaps into the yonder.

Imagine software that routes baggage from an airline check-in desk through the bowels of an airport's network of conveyor belts and scanners until it arrives at the right gate for the airplane that you're traveling on. Let's say you reshape the part of the software that figures out whether to route baggage to gate 1 in terminal A or to gate 2 in terminal A. Same terminal, different gate. You're losing productivity if you have to start the process from the part of the software that decides to route the baggage between terminal A or terminal B.

Here's the problem:

You need to observe the gate router, but you must control both the gate router and the terminal router. Controlling the terminal router now is wasted work.

You need to feed the terminal router with the circumstances so that you're controlling it to choose the terminal A route. You also feed it the circumstances that it passes on to the gate router so that it does the thing that you expect it to, and so that you can observe that. The terminal router is superfluous in this scenario. You're concerned with the gate router, but you also have to be concerned with the terminal router's workings, even to the point of being concerned with how it gathers the information from its inner workings to pass to the gate router.

If your concern is the workings of the gate router, then you should only have to control the gate router, feeding it the information it needs directly without going through any intermediaries. It's what "separation of concerns" is talking about. Collusion of concerns makes you write more code (and more complex code) to make observations about your software, but it also throws attention switching into work that is trying to be a feedback loop.

If you have a gate routing module, you can likely control it and observe it directly. But what happens when you want to control and observe the terminal routing module? The terminal router will immediately pass the baggage off to the gate router after it has done its work. Can you control the workings of the terminal router without also having to feed it the right information that it needs to feed to the gate router so that the gate router doesn't accidentally raise an error? In other words, how do you control and observe the terminal router without having to be concerned about what the gate router needs to know to do it's job. If there was an accident rate in software development like there is in other production jobs, it would be measured by how often we accidentally have our attention sucked too far into a wood chipper of dependency inversion.

With the right modularity, you've addressed the "static design". The "dynamic design" is how the parts of the software are put together to make the machine. A software machine is put together at runtime, usually by something called a front controller. A Main method is an example of a front controller. So is the thing that receives an HTTP request and sends its instructions it to the right part of your application. This is the part of design that a lot of developers don't recognize as the sore spot of their productivity problems.

There some esoteric terms for this issue of controlling which modules you have to be concerned with when making observations. Much of the language is more complicated than the actual work that you have to do. If you don't get distracted by the lingo, you'll find that this is not only one of the most important problems to solve, but also one of the easiest!

If the part of your software machine that routes baggage from the check-in desk to the right terminal passes control to the part of the software that routes the baggage to the right gate, then the terminal router will likely need to call a method on the gate router (or send a message to it). If I just want to control and observe the terminal router without having to be concerned with the gate router, then I've got problem that can be solved by getting the terminal router to use a dummy gate router instead of the real gate router. I can't change the terminal router code so that it doesn't use a gate router at all when I'm trying to make these observations, because that's not the code that would run live anyway. It wouldn't be a real observation of the thing that I'm concerned with in the first place.

If the terminal router uses a dummy gate router, I don't have to be nearly as concerned with it as I am with my primary concern, which is the terminal router. This is what "separation of concerns" is talking about.

The last thing that you have to solve is a way to switch the gate routing part for a dummy gate router. That's not a hard problem to solve. If you have a terminal router object, you can create a constructor that accepts a fake gate router. The terminal router's default constructor can create a real gate router. When you need to observe the terminal router's workings, create a dummy gate router object that doesn't have to be more talented than not raising errors when it's not supposed to and pass it to the terminal router's constructor.

The term for this pattern is called "inversion of control", or "dependency inversion", or "don't distract me with superfluous concerns".

This is a really simple way to get superfluous concerns out of the way so that they don't interfere with the parts of your software that you're actually concerned with. The above technique is sometimes called "poor man's inversion of control". There are many free libraries that you can use to automate all this stuff and make describing how the parts fit together effortless (or better, unnecessary). They are "dependency injection frameworks" or "inversion of control containers". I like the word "composer".

The dynamic side of productive software design is all about putting the parts of the machine together when your system starts up (or on-demand). The effort that you put into the static parts of the design - the shapes, the geometry, the abstractions - is paid off when you're in complete control of which parts are in-motion when you need to make observations of your software. And to belabor the point: it's the ability to control and observe your software that will give you more productivity, or to restore your productivity. You can't use what you can't understand, and you can't understand what you can't control and observe.

If you can't directly control and observe your software, you might be mistaking efficiency for productivity. You're optimizing your work within a range of improvement that is too close to the floor that make a difference in how you to get the lid off the cookie jar.

Productivity is right there for you to reach out and harvest. It's locked up in your code like potential energy waiting to be turned into kinetic energy when you free it from its moorings. You can find it in all the ways that design interferes with your ability to work with the software that you already have. There's little value in optimizing the pace that you create new software if you're optimizing the pace that you incur compound interest.

Productivity is usually something that you free from your software rather than something you add to software. You create the interference. If you don't want the interference, don't add it to your software.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Monday, February 08, 2010

How the Mainstream Lost Software Development Productivity

(Continued from: Denying Productivity)

The greatest software development productivity comes from software that can be easily and readily understood. When you need to make changes to software, you can easily and readily understand the repercussions of those changes. When you need to add new abilities to software, you can easily and readily understand where the additions need to be made. When you're new to an existing project, you can easily understand the code that has already been written by your new team. Understanding how to design software so that you're able to easily control and observe it gives you the ability to create software that can be understood, and software that doesn't obstruct productivity.

Despite the number of software developers who understand the mechanics and principles of software design that cultivate software development productivity, the vast majority of software developers don't understand how to wield the techniques, or aren't convinced by the promises, or simply haven't heard about higher order developer productivity at all, let alone how to get it.

Somewhere along the line, the software development mainstream lost its access to productivity, and the vacuum was filed with counter productivity or productivity proxies.

Two things contributed to the mass extinction of software development productivity in the mainstream: Eggheads and Shills. The eggheads allowed a divide to open between the intellectual haves and the have-nots, and the shills exploited that gap, filling it with misinformation and misrepresentations, verily redefining "productivity" and supplanting it with mere efficiency.

Yes, these labels are chosen for shock value. Specifically because this is an issue that is largely rooted in unconsciousness: the unconsciousness of failing to communicate, and the unconsciousness of deeper meaning.

I've learned quite a bit about software development and design from some very insightful and intelligent people. I'm grateful to the authors who've been my long distance mentors and who have provided me with inspiration. I've met many of them in person over the past half decade or so and these encounters have deepened my pursuit of understanding ever further.

Now that I have a much better understanding of what it is that they taught me, I'm shocked at how poorly this material was often communicated and at how our self-indulgent vanity continues to get in the way of broader communication of ideas that are vital to contemporary society's productivity at a time when productivity is so desperately needed.

I use the term "Egghead" with some affection and admiration, but I also use it in the hopes that some of the great teachers of software development and design will come to learn that they've barely even scratched the surface of the full audience that they need to reach. Eggheads write and speak for other eggheads. The language they chose and the names given to design principles, as well as practices and patterns are stimulating to other eggheads and yet utterly obstructive to the other part of the software development population. When you measure the Egghead population against the mainstream population, the Egghead population doesn't appear to be much more than a rounding error. There's a lot of work to do, and we can do it much better.

Eggheads preach to the choir, and this is often so stimulating that they never leave the echo chamber. They don't learn how to reach the mainstream. They hope that the people they do reach will somehow reach the mainstream, but those people usually end up emulating their egghead heros and become eggheads themselves. The circle expands slightly, but not merely enough to have the meaningful effect on software development productivity that their knowledge and understanding should already have had.

Their communication style is rife with self-stimulating, overly-academic legalese that does little more than enforce an over-estimation of the value of the terminology itself rather than subjugate the terminology to the absolute must of communicating simple and powerful ideas to the mainstream. Eggheads don't like to engage directly with the mainstream. They don't learn to talk to the mainstream and how to teach the mainstream. In this, their self-indulgence leads to terrific loss for human society and is quite possibly one of the most negligent acts perpetrated against the potential for software development productivity.

The divide between the Egghead echo chamber and the mainstream has been readily filled by Shills. While the Eggheads fascinate themselves with themselves, the Shills convince the mainstream to buy snake oil solutions to problems that could be easily solved with plain old soap and water.

The potential for abuse and the leadership vacuum left behind by the Eggheads' self-glorification is readily taken advantage of by Shills willing to convince the mainstream that mere efficiency tools are good solutions for productivity problems. They further exasperate the productivity problems that result from applying efficiency solutions to productivity problems and use the ensuing mainstream customer panic to justify yet more efficiency tools. The mainstream becomes ever more isolated from more meaningful understandings of software development productivity and continues to spin out of control on the back of software systems and software projects that are themselves out of control.

Between the Eggheads and the Shills, things look pretty grim. Those who have good answers to the software crisis are unwilling to forgo the stimulating academic formalism in favor of connecting with the mainstream, and those who would take advantage of ignorance do take advantage of ignorance.

Neither of these two archetypes can be counted on to fix the problem. Eggheads are far too ensconced in the comforts of elitism and Shills have built behemoth networks of companies all selling the same pack of lies, damned lies, and demos. The momentum and justifications of either aren't likely to change anytime soon.

What's needed is a new generation of productivity missionaries who are willing to master the field of knowledge and are willing to learn to connect with the mainstream. They're willing forgo the all-consuming pursuit of elite celebrity and serve society by taking on the software crisis head-on as a matter of honor and duty in the face of continuing negligence and abuse on all sides. They are willing to speak plainly and openly and forgo the perceived legitimization that comes from falling into step with esoteric formalism.

The mainstream has lost track of software development productivity, but it's not so far removed that it can't be recovered. It just can't be recovered through the continued self-indulgence underpinning both the Egghead and the Shill entitlement to ease and winnings in the face of disastrous effects on modern productivity. It can be recovered if pathfinders and communicators are willing to get dirty amongst the "unwashed masses" for an even greater return, and to stand fast against the encroachments of suboptimal local efficiencies bundled with six-figure price tags and promises of yet more software development pain.

Next: Productive by Design



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Denying Productivity

(Continued from: Cause and Effect and Developer Productivity)

If you choose to use the ability to control and observe your software to restore and cultivate software development productivity, you have to come to terms with the reality of your software as it presently is, and dispense with the stories you tell yourself about just how much productivity your software development has. It's difficult to accept where you are if where you are is much further back than you had realized. If you don't accept where you are, you won't progress. If you already feel that you've arrived, you won't screw up the courage to challenge what it is that you know, and embark on that next stage of your career voyage.

In the past, we've used the term "testability" to describe the quality of design that allows for the kinds of expectations for software development productivity caused by the ability to control and observe. To those who introduced the term to every day software development vernacular, the term has a very specific meaning, and refers to designs that are very specifically evidenced by a set of well-known and observable design principles and patterns. And to others, the term was misinterpreted and misrepresented as software that is tested.

"Testability" means software that is extremely easy to test rather that software that merely has tests. Any software can be tested, what matters is how incredibly easy it is to do so. The easier it is to test software, the more that the software can be controlled and observed. The more directly that bits of your software can be controlled and observed, the more directly that your software can be understood, and the more that your software design will be simple and clear, and reflect principles productive software design.

The level of simplicity in question is usually beyond the realm of possibility for developers who have not had experience with software that is designed to be quickly and readily understood. When the ability to understand software through the ability to control and observe is understood to be the root cause of software development productivity, the software designs that you create will be radically different than what you may have become accustomed to. And while these designs will seem foreign and strange at first, you'll soon learn to see them as the most natural, workable designs that you've created, and that you've worked with.

For example, there's no doubt that you can control the content of an application's database from the application's user interface, but that is a far cry from a level of control that will have any significant effect on productivity, not to mention that such a means to control and observe doesn't even begin to guarantee the extent to which all the bits of software in between can be immediately understood. It's still control, and it's still control that can followed up by observation, but it isn't direct control and direct observation. If you're concerned with the behavior of your data access logic, for example, you must be able to control it directly by using your data access logic directly rather than using it transitively through an application's user interface. If you must use your application's user interface to observe the results of the control you exert on your application's database, then you're not able to satisfy your concern directly through direct observation, and you're invariably not gaining any of the great productivity benefits that come from productive design.

Be careful of getting ahead of yourself: it doesn't lead to learning of any meaningful significance. Learning to control and observe will bring the biggest advances to your experience of software development productivity. If you stake a claim to this learning before you've barely begun, chances are you'll never really learn what it means and how to use it to any significant degree. And this would be sad, because the means to direct control over your software really isn't difficult to learn. You will be disrupted more by the mass of unlearning of what you already hold firmly as software productivity than by the small amount new learning that you'll really need.

If you want to reach into this realm of productivity that is beyond what you've yet experienced, then commit yourself in earnest to be a student of software design principles. Learn how to apply simple software design principles and exercises to productivity. Commit to a journey to master software design from a principled perspective where productivity is your a-priory concern and the thing that you delivery consistently. You'll find that there's really not a lot to learn once you're over the initial interference of what you presently believe.

It's vital that when you embark on the exploration of the next realm of software development productivity that you start from a realistic place, and resist every urge to claim victory before you've barely out of sight of your own home port. If you stake a claim to understanding what it is to control and observe your software, you'll deny the vast wealth of productivity that is practically within your reach.

It's hard to start down a new path to new understanding, but denying that you need to continue learning and to challenge entrenched orthodoxy is no way to start.

Next: How the Mainstream Lost Software Development Productivity



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Sunday, February 07, 2010

Cause and Effect and Developer Productivity

(Continued from: Mistaking Efficiency for Productivity)

By mistaking developer efficiency for developer productivity, we end up creating less productivity. It's an easy trap to fall into. The very shape of your organization is an amalgamation of past responses to productivity problems. Efforts to increase productivity that don't penetrate to the depths of the problem but only reach so far as efficiencies localized to particular job functions not only fail to solve productivity problems, they also fail to fix the counter-productive organizational problems that are the unfortunate side effects of previous naive attempts at treating productivity problems to local efficiency solutions. The organizational problems are also contributors to the vicious circle that worsens software development productivity.

Trying to solve productivity problems by chasing efficiencies that are tied to a particular job function and are not coordinated with other job functions is like swimming in quicksand: even though your right hand and your left hand are doing everything right for everything you know about swimming, the swimming motions applied to the actual medium that you're in are just as likely to make you sink. Swimming is the wrong response to being immersed in quicksand. No matter what your panicing mind is telling you to do, it's just grasping at desperate straws of well-established habits rather than assessing this new situation with responses that might be new to you, and thus quite likely less informed by existing habit. Applying the wrong physics to a medium is almost guaranteed to create side effects that defeat the progress you make against the productivity problem you're trying to solve.

The software development productivity problems that you face are due to a productivity balance sheet that fails to tally the losses as well as the gains. If you simply add more gains to the balance sheet without any regard to the losses, you're not going to understand why the bottom line keeps getting smaller while you're continuing to invest ever more energy into the individual parts of your software development effort. It's not enough to get a bigger bucket to bail the water out of a sinking ship if you're not paying any attention whether the that hole letting the water in is bigger than your biggest bucket. Most software productivity efforts end up snatching a fire axe off the wall and desperately trying to hack another hole in the hull to let the rising water out.

If your organization chases local efficiency solutions to productivity problems, it will end up fracturing itself along job functions that are not entirely separate job functions, but rather individual items on job completion checklists for cohesive workgroups or single workers. As your organization adds to the weight of the processes that petition the productivity spirits to unleash more of this mysterious magical substance from whence we know not, it invariably creates more high ceremony work for managers, eventually calcifying them into clerical functionary roles rather than freeing them to be effective and experienced teachers. The increased ceremonial workload leaves your organization devoid of the leaders who ensure that your organization cultivates successive generations of master workers.

The increased management of ceremony makes it seem as if these managers are doing so much work that the organization around the workload deserves to be promoted to its own department. The resulting departmentalization creates even more of the handoffs and coordination problems that are the fuel for the productivity bonfire consuming your project's allowance of money, people, and motivation.

Your organization will tend to become a fractured collection of organizational shards as it spends more and more of its energy dealing with the mounting productivity losses rather than on producing. This kind of organization produces more internal processes than goods and services that it can exchange with customers for money. Worse, it dries up the pool of knowledge, insight, and vision that creates compelling products and attractive, compelling organizations that compete successfully for more good people and that transform their people into the next generation of success makers.

If you're working off of a one-sided balance sheet, you're failing to connect cause and effect. Failing to connect cause and effect is one of the most common behaviors of contemporary software development when it comes to productivity.

The one-sided balance sheet is a glaring genetic marker of software development that has not been able to connect the effects of it's efforts to the causes of impoverished productivity. The more that one-sided, uncoordinated efforts for productivity fracture organizations along unnatural lines, the more difficult it becomes for software development workers to see the causes of their effects, and the more that dramatic under-performance becomes remarkable. This kind of organization has forgotten the vast wealth of productivity that is its birthright.

The real productivity that you should be experiencing is far beyond the level of productivity machinations that are typically entertained today. The greatest drawback to the promise of software development productivity is just how far-fetched the proposition of potential productivity is. It's multiples greater or orders of magnitude greater than what has become common today.

Amidst all the unbelievable claims of missed productivity is an anchor that can hold you fast to reality of high performance software development: the answers to productivity are incredibly simple, incredibly straight-forward, and incredibly easy to put into practice. There's no magic, there's no mysticism. There are just plain old software design principles at the heart of the matter. Principles that, while seemingly named in a fury of self-indulgent academic terminology fetishism, are nonetheless very easy to understand in practice when they are connected to cause and effect in software development productivity. They are simple principles that can be brought into effect in software through even simpler exercises. Deceptively simple exercises, in fact.

But first, cause and effect have to be connected so that the actions and habits that create the detrimental effects are recognized, understood, and curtailed, and the actions and habits that create productivity are practiced, understood, reinforced, and communicated to others.

First things first: the people who create software must be required to prove that the software that they create can quickly and easily be proven to meet the expectations of it. Expectations for just how quickly and easily these proofs can be made will seem daunting at first, as the organizational fracturing that has been allowed to infect software development as a result of one-sided productivity balance sheets has cultivated a generation of software development that has lost touch with the design simplicity that allows software creators to assume the burden of proof of their own work.

The moment that developers are reconnected with the burden of proof of their own work, software development productivity restoration begins in earnest. To restore software development productivity to your organization, you must reclaim the responsibility of the burden of proof for your work. You do this by learning the dead-simple techniques that cultivate the ability to control and observe your software.

In order to provide a proof that your software works, you need to be able to be able to rapidly control and observe your software. This will invariably change the designs that you tend to use, and guide you toward more simple and clear designs that end up being more easily understood by others, which even further improves software development productivity. Software that isn't easily-understood is the single most important subject in software development productivity. The inability to easily and readily understand software at a glance is a huge part of the productivity loss that goes unaccounted in software development management.

Because of the exasperated fracturing of your organization into departments that are not natural reflections of productive software development, and that contribute to the productivity losses on your software development performance balance sheet, you will not have developed a sensitivity to productive designs. That sensitivity will come. It will come with practice. And while you're practicing control and observe techniques, you're breaking down the barriers that keep both your software and your organization from its productive potential.

The moment you assume the burden of proof, you take steps to radically re-shape your software and your organization to a level of sustained productivity that you would not have presumed possible. By learning to control and observe your software ever more effectively you'll naturally close the gap between the cause of productivity and it's effects.

Next: Denying Productivity



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Thursday, February 04, 2010

Mistaking Efficiency for Productivity

(Continued from: Controlled Productivity)

When productivity starts to decrease and costs start to increase, we tend to panic. In that panic we often accidentally take measures that increase developer efficiency. In the process, we usually end up creating obstructions that decrease productivity.

Productivity that comes from the ability to control and observe is rooted in software design. Specifically, designs that make controlling and observing simple, clear, and very easy.

When productivity starts to fail, you will invariably find software that is difficult to control and observe. You can restore and sustain productivity by exercising controlling and observing. The very act of making your software more controllable will displace less productive designs with more productive designs.

Without connecting the effect of software development productivity to the cause of controlling and observing, you're likely to end up chasing efficiency rather than productivity, and exacerbating the productivity problems you are trying to solve. Most of what we believe about developer productivity has little effect on the ability for developers to control and observe their software. Most efforts for developer productivity end up increasing the pace at which developers create software that is beyond control.

Developer Efficiency

There's a downside to focusing on and increasing the efficiency of code production: it creates a discoordination between the production of code and everything else around it, like testing, operations, planning, and design. If you can increase the pace of not only coding, but also increase the pace of everything after it and before it at the same rate, productivity increases without risking the detrimental side effects of localized efficiency.

Inevitably, increasing the pace of coding without increasing the pace of the whole of software production creates in-process inventory. In-process inventory has even more deleterious effects: It creates rework. The time spent in rework is lost productivity. You're not producing new value when you're busy reworking stuff that is already supposed to be creating value. Decreased productivity creates more panic, and more panic often leads to more ill-fated efforts to rectify the problem by increasing developer efficiency.

If your wheels are spinning in mud, pressing harder on the accelerator doesn't free you from the problem, and it will likely just get you deeper entrenched in the problem. The material ejected from the hole that you're digging collects as in-process inventory. Not only do you not get anywhere, you loose ground.

It's not hard to remain in control of productivity. The trick is to know the difference, to remain calm in the face of panic, and to exercise control and observation until you've got software design under control. And ideally, try not to be distracted by elaborate tools promising productivity if in the end all they offer is a more efficient way to spin your wheels in the mud.

Don't panic. The first decision that comes to your mind when you panic in the face of decreasing productivity can lead to the temptation to displace productivity with efficiency. Don't increase the pace that you're loosing control of your software. Settle in and regain control. You'll restore the productivity you lost and find that there's even more to be had.

Of the developer productivity tools that you have adopted, have they noticeably increased the rate at which your organization makes money as a result? The pace that you can continue turn ideas into money is the definition of productivity that matters. If you focus on the pace that you produce code, you're looking at a part of the picture that is usually too narrow to be meaningfully thought of as productivity.

Next: Cause and Effect and Developer Productivity



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Wednesday, February 03, 2010

Controlled Productivity

(Continued from: To Control and Observe - Productive Software Development)

You can work at a level of productivity that redefines your present expectations by learning how to intentionally control and observe your software. Controlling and observing is something that every developer does. Doing it consciously makes productivity intentional - it brings productivity squarely under your control and conditions you to see ways of creating even more of it.

You can't use what you don't understand. It's as simple as that. The easier it is to understand how your software works, the easier it is to understand the repercussions of making needed changes to it. The easier it is to understand how your software works, the quicker you can get to work on it and the quicker you can get done. Inevitably you understand how your software works by observing it. In order to observe your software, you have to be able to control it.

Controlling your software means putting it into a state where you can directly run the chunk of code that you're concerned with without having to prepare adjacent chunks of code. If those adjacent chunks of code aren't your direct concern, then you should minimize their interference with the observations you're trying to make and with the understanding that you're trying to develop.

In the end, all software is controllable. If you can open one of your application's screens, fill in some text boxes, click on some buttons, and end up saving some data in your database, then you're controlling your database through those screens. The issue isn't that software is or is not controllable, but how easy it is to control it. You don't get productivity from the mere fact that software can be controlled. You get productivity from how software can be controlled. Some ways of controlling software are incredibly productive - beyond most people's expectations.

The more direct control you have over your software, the more productivity you'll have. Direct control means that if you're trying to run some bit of code in order to observe it, that you only have to setup that bit of code and interact with that bit of code. If your software is designed for controlled productivity, you can do this at all levels of your system, from function libraries, to classes, to layers in n-tier and distributed applications, to user interfaces, to deployment systems, to runtime monitoring.

We're so used to primitive productivity in software development and primitive productivity is so common and so prevalent that we've lost track of basic principles that are still available to us if we step outside of the limitations of our current view of software development - a view that has taken up residence in software development only within the last ten or so years.

The principles are also unfortunately named. The names almost complete obscure their meaning, making incredibly simple and incredibly powerful ideas accessible to only a few people who have a natural penchant for deciphering software development legalese.

Designing Productivity

Higher order productivity - productivity beyond what you'd expect would be possible - is only created by software design. That's it. No tool, no matter how elaborate, can ever create the level of productivity that controllable software can (unless you're applying that tool to your work after having made your software easily-controllable and readily-observable).

To make software controllable, you simply design it in a way that you can have direct control over any bite-sized chunk of it so that you can make observations of it with an absolute minimum of effort and minimum of distractions by concerns other than the one in your crosshairs. The means to designing controllable software is also deceptively simple. It's based on utterly and trivially-easy exercises that can be done with any software development tool whether it costs thousands of dollars, or whether it's absolutely free.

There's a simple way to test if your software can easily be controlled and observed: Choose one bit of important logic buried in your software that is only executed when some if/then statement (or other conditional branch) flows execution to that bit of code. See how much code you have to write before you can run that bit of code, and print out its results. If you have to control not only the code that you're concerned with, but also adjacent code, classes, systems, database tables, logins, etc, then your code is more difficult than necessary to control and observe.

You loose productivity as well as motivation as your system becomes more difficult to control. There's another level of productivity available to you. That level of productivity can increase your effectiveness by many multiples, and maybe even by orders of magnitude.

Next: Mistaking Efficiency for Productivity



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

To Control and Observe - Productive Software Development

There isn't a productive software development effort where controlling and observing aren't the substance of productivity. If you break down productivity to it's most fundamental element, you'll find that the understanding of how your software works is the essence of it all. Control and observe your software and you understand how it works. When you understand it, you can change it, work with it, and continue working with it. When it gets harder to make observations of how your software works, it becomes harder to remain motivated to do the right thing. In the extreme, we live out software projects in survival mode, and we saddle someone else with the responsibility of observing whether our software works. It's not the only way. It's the worst way.

The more that you can make observations about your software, and the more that those observations can be made in thought-sized chunks, the more easily it is to understand your software. To make an observation about your software, you have to be able to control that bit of software to do the thing who's side effects you want to observe. The easier it is to do that, the easier it is to understand it, and the more productivity you have. The time that you spend deciphering software that should be readily understandable is mostly wasted time. It's time spent that shouldn't be necessary and that can be eliminated.

Software that is hard to understand is software that is hard to control. Software that is hard to control is hard to understand. A small bit of code that you want to observe should be controlled with as little code as possible. If you can run that small bit of code directly without having to also control adjacent code, then you've got code that is inherently easy to control, and a level of productivity that is inherently greater. We tend to overlook the amount of time that software teams spend controlling and observing, and when things start to become difficult, we tell ourselves that we can't afford to pay attention to it. Ironically, paying attention to it solves the root problem.

Here are a couple of examples of code that is easy to control:
- In-memory objects that can be observed without loading their data from the database
- User interface workflow that can be observed without using the user interface
- Business logic events that can be observed without the auditing software making recordings
- Steps of complex business processes that can be observed without starting the process from scratch
- Secure operations that can be observed without having to authenticate
- Payment logic that can be observed without connecting to a payment processor

If I'm concerned with the functioning of in-memory objects, and if I need to load those objects from a database, then I have to add the database to the things I need to control, and to be concerned with as well. This isn't a natural, defacto quality of software, regardless of how common it is.

We generally accept that the lack of control as a defacto quality of software projects. We overestimate the control that we have, and we underestimate the value of increased control. We don't pay it much attention. We believe that that the limits to the control that we have is just part of the very fabric of software development, that it's unchangeable, and that there's no reason to dwell on it. We may not even have an awareness of it, or of the issue. This is the fatal flaw in software development thinking, because it's the issue of control that explains why software projects rapidly loose productivity and why the extent of that productivity loss is so great.

Maintaining the ability to control and observe is a conscious, intentional effort. If you're not on top of it all the time, it slips away. The traditional productivity loss curve and the traditional cost curve subsequently begin to immediately express themselves. You can tell yourself it's just the way it is, or you can realize the root cause of the problem and the solution and not only regain that lost productivity, but then go on to achieve a level of productivity that you wouldn't have believed possible from your previous perspective.

This issue isn't unknown. In fact it's well-known. The most common mistake in production management is not paying attention to how productivity loss happens and focus instead on increasing the efficiency of the work that is done amidst the accelerating losses in effectiveness. This mistake is the basis for most software production management at this point of industrial software development.

Next: Controlled Productivity




Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

QA Missed Something

When a team is closing in on a release it may still find a flaw so terrible that the release opportunity might be missed. After the initial panic settles down, we go looking for the explanation of why a such a significant problem can remain hidden until so late. Inevitably, QA missed something.

Let me tell you something: there's nothing that QA has ever missed that developers didn't miss first.

Blaming testers for not discovering flaws that they did not create while simultaneously believing that we don't have to check our own work while is it fresh in our own minds is dishonest. There's no defect ever found by a tester that wasn't first designed and implemented by a developer.

Software is a production industry that generally still believes that the people who make things and the people who prove that those things are right should be different people doing two parts of a single job at different times.

Part of a tester's job is to cross check the work that has already been checked by the workers who create the software and who have the best, timeliest knowledge of it. When a tester isn't cross-checking, he's analyzing and doing exploratory work - looking for possible gaps in developers' thinking, understanding, and observation. When a developer's responsibility to deliver repeatably-checked work is passed over to a tester, neither is effective.

It's less of a tester's job to check that a developers' code is right, but that their tests are right, or at least that the result of the development so far is right - which includes tests that have been written before final inspection. That work doesn't start after the code has been written, it's starts before, as they work together to ensure that by the time the product and the tests make it to the final inspection, that all the ducks are already in a row, leaving the tester with the space to continue to question exactly what it means for the software to be right, and how to perform final inspections that hold up their part of the shared responsibility for intentional quality and productivity.

Final inspection is the last place that you want to find game-changing flaws and problems. It's the last place you should find them. It happens, but it should be the exception to how development is done, including testing, and the exception to how you treat final inspection.




Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com