Denying Productivity

(Continued from: Cause and Effect and Developer Productivity)

If you choose to use the ability to control and observe your software to restore and cultivate software development productivity, you have to come to terms with the reality of your software as it presently is, and dispense with the stories you tell yourself about just how much productivity your software development has. It's difficult to accept where you are if where you are is much further back than you had realized. If you don't accept where you are, you won't progress. If you already feel that you've arrived, you won't screw up the courage to challenge what it is that you know, and embark on that next stage of your career voyage.

In the past, we've used the term "testability" to describe the quality of design that allows for the kinds of expectations for software development productivity caused by the ability to control and observe. To those who introduced the term to every day software development vernacular, the term has a very specific meaning, and refers to designs that are very specifically evidenced by a set of well-known and observable design principles and patterns. And to others, the term was misinterpreted and misrepresented as software that is tested.

"Testability" means software that is extremely easy to test rather that software that merely has tests. Any software can be tested, what matters is how incredibly easy it is to do so. The easier it is to test software, the more that the software can be controlled and observed. The more directly that bits of your software can be controlled and observed, the more directly that your software can be understood, and the more that your software design will be simple and clear, and reflect principles productive software design.

The level of simplicity in question is usually beyond the realm of possibility for developers who have not had experience with software that is designed to be quickly and readily understood. When the ability to understand software through the ability to control and observe is understood to be the root cause of software development productivity, the software designs that you create will be radically different than what you may have become accustomed to. And while these designs will seem foreign and strange at first, you'll soon learn to see them as the most natural, workable designs that you've created, and that you've worked with.

For example, there's no doubt that you can control the content of an application's database from the application's user interface, but that is a far cry from a level of control that will have any significant effect on productivity, not to mention that such a means to control and observe doesn't even begin to guarantee the extent to which all the bits of software in between can be immediately understood. It's still control, and it's still control that can followed up by observation, but it isn't direct control and direct observation. If you're concerned with the behavior of your data access logic, for example, you must be able to control it directly by using your data access logic directly rather than using it transitively through an application's user interface. If you must use your application's user interface to observe the results of the control you exert on your application's database, then you're not able to satisfy your concern directly through direct observation, and you're invariably not gaining any of the great productivity benefits that come from productive design.

Be careful of getting ahead of yourself: it doesn't lead to learning of any meaningful significance. Learning to control and observe will bring the biggest advances to your experience of software development productivity. If you stake a claim to this learning before you've barely begun, chances are you'll never really learn what it means and how to use it to any significant degree. And this would be sad, because the means to direct control over your software really isn't difficult to learn. You will be disrupted more by the mass of unlearning of what you already hold firmly as software productivity than by the small amount new learning that you'll really need.

If you want to reach into this realm of productivity that is beyond what you've yet experienced, then commit yourself in earnest to be a student of software design principles. Learn how to apply simple software design principles and exercises to productivity. Commit to a journey to master software design from a principled perspective where productivity is your a-priory concern and the thing that you delivery consistently. You'll find that there's really not a lot to learn once you're over the initial interference of what you presently believe.

It's vital that when you embark on the exploration of the next realm of software development productivity that you start from a realistic place, and resist every urge to claim victory before you've barely out of sight of your own home port. If you stake a claim to understanding what it is to control and observe your software, you'll deny the vast wealth of productivity that is practically within your reach.

It's hard to start down a new path to new understanding, but denying that you need to continue learning and to challenge entrenched orthodoxy is no way to start.

Next: How the Mainstream Lost Software Development Productivity

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Cause and Effect and Developer Productivity

(Continued from: Mistaking Efficiency for Productivity)

By mistaking developer efficiency for developer productivity, we end up creating less productivity. It's an easy trap to fall into. The very shape of your organization is an amalgamation of past responses to productivity problems. Efforts to increase productivity that don't penetrate to the depths of the problem but only reach so far as efficiencies localized to particular job functions not only fail to solve productivity problems, they also fail to fix the counter-productive organizational problems that are the unfortunate side effects of previous naive attempts at treating productivity problems to local efficiency solutions. The organizational problems are also contributors to the vicious circle that worsens software development productivity.

Trying to solve productivity problems by chasing efficiencies that are tied to a particular job function and are not coordinated with other job functions is like swimming in quicksand: even though your right hand and your left hand are doing everything right for everything you know about swimming, the swimming motions applied to the actual medium that you're in are just as likely to make you sink. Swimming is the wrong response to being immersed in quicksand. No matter what your panicing mind is telling you to do, it's just grasping at desperate straws of well-established habits rather than assessing this new situation with responses that might be new to you, and thus quite likely less informed by existing habit. Applying the wrong physics to a medium is almost guaranteed to create side effects that defeat the progress you make against the productivity problem you're trying to solve.

The software development productivity problems that you face are due to a productivity balance sheet that fails to tally the losses as well as the gains. If you simply add more gains to the balance sheet without any regard to the losses, you're not going to understand why the bottom line keeps getting smaller while you're continuing to invest ever more energy into the individual parts of your software development effort. It's not enough to get a bigger bucket to bail the water out of a sinking ship if you're not paying any attention whether the that hole letting the water in is bigger than your biggest bucket. Most software productivity efforts end up snatching a fire axe off the wall and desperately trying to hack another hole in the hull to let the rising water out.

If your organization chases local efficiency solutions to productivity problems, it will end up fracturing itself along job functions that are not entirely separate job functions, but rather individual items on job completion checklists for cohesive workgroups or single workers. As your organization adds to the weight of the processes that petition the productivity spirits to unleash more of this mysterious magical substance from whence we know not, it invariably creates more high ceremony work for managers, eventually calcifying them into clerical functionary roles rather than freeing them to be effective and experienced teachers. The increased ceremonial workload leaves your organization devoid of the leaders who ensure that your organization cultivates successive generations of master workers.

The increased management of ceremony makes it seem as if these managers are doing so much work that the organization around the workload deserves to be promoted to its own department. The resulting departmentalization creates even more of the handoffs and coordination problems that are the fuel for the productivity bonfire consuming your project's allowance of money, people, and motivation.

Your organization will tend to become a fractured collection of organizational shards as it spends more and more of its energy dealing with the mounting productivity losses rather than on producing. This kind of organization produces more internal processes than goods and services that it can exchange with customers for money. Worse, it dries up the pool of knowledge, insight, and vision that creates compelling products and attractive, compelling organizations that compete successfully for more good people and that transform their people into the next generation of success makers.

If you're working off of a one-sided balance sheet, you're failing to connect cause and effect. Failing to connect cause and effect is one of the most common behaviors of contemporary software development when it comes to productivity.

The one-sided balance sheet is a glaring genetic marker of software development that has not been able to connect the effects of it's efforts to the causes of impoverished productivity. The more that one-sided, uncoordinated efforts for productivity fracture organizations along unnatural lines, the more difficult it becomes for software development workers to see the causes of their effects, and the more that dramatic under-performance becomes remarkable. This kind of organization has forgotten the vast wealth of productivity that is its birthright.

The real productivity that you should be experiencing is far beyond the level of productivity machinations that are typically entertained today. The greatest drawback to the promise of software development productivity is just how far-fetched the proposition of potential productivity is. It's multiples greater or orders of magnitude greater than what has become common today.

Amidst all the unbelievable claims of missed productivity is an anchor that can hold you fast to reality of high performance software development: the answers to productivity are incredibly simple, incredibly straight-forward, and incredibly easy to put into practice. There's no magic, there's no mysticism. There are just plain old software design principles at the heart of the matter. Principles that, while seemingly named in a fury of self-indulgent academic terminology fetishism, are nonetheless very easy to understand in practice when they are connected to cause and effect in software development productivity. They are simple principles that can be brought into effect in software through even simpler exercises. Deceptively simple exercises, in fact.

But first, cause and effect have to be connected so that the actions and habits that create the detrimental effects are recognized, understood, and curtailed, and the actions and habits that create productivity are practiced, understood, reinforced, and communicated to others.

First things first: the people who create software must be required to prove that the software that they create can quickly and easily be proven to meet the expectations of it. Expectations for just how quickly and easily these proofs can be made will seem daunting at first, as the organizational fracturing that has been allowed to infect software development as a result of one-sided productivity balance sheets has cultivated a generation of software development that has lost touch with the design simplicity that allows software creators to assume the burden of proof of their own work.

The moment that developers are reconnected with the burden of proof of their own work, software development productivity restoration begins in earnest. To restore software development productivity to your organization, you must reclaim the responsibility of the burden of proof for your work. You do this by learning the dead-simple techniques that cultivate the ability to control and observe your software.

In order to provide a proof that your software works, you need to be able to be able to rapidly control and observe your software. This will invariably change the designs that you tend to use, and guide you toward more simple and clear designs that end up being more easily understood by others, which even further improves software development productivity. Software that isn't easily-understood is the single most important subject in software development productivity. The inability to easily and readily understand software at a glance is a huge part of the productivity loss that goes unaccounted in software development management.

Because of the exasperated fracturing of your organization into departments that are not natural reflections of productive software development, and that contribute to the productivity losses on your software development performance balance sheet, you will not have developed a sensitivity to productive designs. That sensitivity will come. It will come with practice. And while you're practicing control and observe techniques, you're breaking down the barriers that keep both your software and your organization from its productive potential.

The moment you assume the burden of proof, you take steps to radically re-shape your software and your organization to a level of sustained productivity that you would not have presumed possible. By learning to control and observe your software ever more effectively you'll naturally close the gap between the cause of productivity and it's effects.

Next: Denying Productivity

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Mistaking Efficiency for Productivity

(Continued from: Controlled Productivity)

When productivity starts to decrease and costs start to increase, we tend to panic. In that panic we often accidentally take measures that increase developer efficiency. In the process, we usually end up creating obstructions that decrease productivity.

Productivity that comes from the ability to control and observe is rooted in software design. Specifically, designs that make controlling and observing simple, clear, and very easy.

When productivity starts to fail, you will invariably find software that is difficult to control and observe. You can restore and sustain productivity by exercising controlling and observing. The very act of making your software more controllable will displace less productive designs with more productive designs.

Without connecting the effect of software development productivity to the cause of controlling and observing, you're likely to end up chasing efficiency rather than productivity, and exacerbating the productivity problems you are trying to solve. Most of what we believe about developer productivity has little effect on the ability for developers to control and observe their software. Most efforts for developer productivity end up increasing the pace at which developers create software that is beyond control.

Developer Efficiency

There's a downside to focusing on and increasing the efficiency of code production: it creates a discoordination between the production of code and everything else around it, like testing, operations, planning, and design. If you can increase the pace of not only coding, but also increase the pace of everything after it and before it at the same rate, productivity increases without risking the detrimental side effects of localized efficiency.

Inevitably, increasing the pace of coding without increasing the pace of the whole of software production creates in-process inventory. In-process inventory has even more deleterious effects: It creates rework. The time spent in rework is lost productivity. You're not producing new value when you're busy reworking stuff that is already supposed to be creating value. Decreased productivity creates more panic, and more panic often leads to more ill-fated efforts to rectify the problem by increasing developer efficiency.

If your wheels are spinning in mud, pressing harder on the accelerator doesn't free you from the problem, and it will likely just get you deeper entrenched in the problem. The material ejected from the hole that you're digging collects as in-process inventory. Not only do you not get anywhere, you loose ground.

It's not hard to remain in control of productivity. The trick is to know the difference, to remain calm in the face of panic, and to exercise control and observation until you've got software design under control. And ideally, try not to be distracted by elaborate tools promising productivity if in the end all they offer is a more efficient way to spin your wheels in the mud.

Don't panic. The first decision that comes to your mind when you panic in the face of decreasing productivity can lead to the temptation to displace productivity with efficiency. Don't increase the pace that you're loosing control of your software. Settle in and regain control. You'll restore the productivity you lost and find that there's even more to be had.

Of the developer productivity tools that you have adopted, have they noticeably increased the rate at which your organization makes money as a result? The pace that you can continue turn ideas into money is the definition of productivity that matters. If you focus on the pace that you produce code, you're looking at a part of the picture that is usually too narrow to be meaningfully thought of as productivity.

Next: Cause and Effect and Developer Productivity

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Controlled Productivity

(Continued from: To Control and Observe - Productive Software Development)

You can work at a level of productivity that redefines your present expectations by learning how to intentionally control and observe your software. Controlling and observing is something that every developer does. Doing it consciously makes productivity intentional - it brings productivity squarely under your control and conditions you to see ways of creating even more of it.

You can't use what you don't understand. It's as simple as that. The easier it is to understand how your software works, the easier it is to understand the repercussions of making needed changes to it. The easier it is to understand how your software works, the quicker you can get to work on it and the quicker you can get done. Inevitably you understand how your software works by observing it. In order to observe your software, you have to be able to control it.

Controlling your software means putting it into a state where you can directly run the chunk of code that you're concerned with without having to prepare adjacent chunks of code. If those adjacent chunks of code aren't your direct concern, then you should minimize their interference with the observations you're trying to make and with the understanding that you're trying to develop.

In the end, all software is controllable. If you can open one of your application's screens, fill in some text boxes, click on some buttons, and end up saving some data in your database, then you're controlling your database through those screens. The issue isn't that software is or is not controllable, but how easy it is to control it. You don't get productivity from the mere fact that software can be controlled. You get productivity from how software can be controlled. Some ways of controlling software are incredibly productive - beyond most people's expectations.

The more direct control you have over your software, the more productivity you'll have. Direct control means that if you're trying to run some bit of code in order to observe it, that you only have to setup that bit of code and interact with that bit of code. If your software is designed for controlled productivity, you can do this at all levels of your system, from function libraries, to classes, to layers in n-tier and distributed applications, to user interfaces, to deployment systems, to runtime monitoring.

We're so used to primitive productivity in software development and primitive productivity is so common and so prevalent that we've lost track of basic principles that are still available to us if we step outside of the limitations of our current view of software development - a view that has taken up residence in software development only within the last ten or so years.

The principles are also unfortunately named. The names almost complete obscure their meaning, making incredibly simple and incredibly powerful ideas accessible to only a few people who have a natural penchant for deciphering software development legalese.

Designing Productivity

Higher order productivity - productivity beyond what you'd expect would be possible - is only created by software design. That's it. No tool, no matter how elaborate, can ever create the level of productivity that controllable software can (unless you're applying that tool to your work after having made your software easily-controllable and readily-observable).

To make software controllable, you simply design it in a way that you can have direct control over any bite-sized chunk of it so that you can make observations of it with an absolute minimum of effort and minimum of distractions by concerns other than the one in your crosshairs. The means to designing controllable software is also deceptively simple. It's based on utterly and trivially-easy exercises that can be done with any software development tool whether it costs thousands of dollars, or whether it's absolutely free.

There's a simple way to test if your software can easily be controlled and observed: Choose one bit of important logic buried in your software that is only executed when some if/then statement (or other conditional branch) flows execution to that bit of code. See how much code you have to write before you can run that bit of code, and print out its results. If you have to control not only the code that you're concerned with, but also adjacent code, classes, systems, database tables, logins, etc, then your code is more difficult than necessary to control and observe.

You loose productivity as well as motivation as your system becomes more difficult to control. There's another level of productivity available to you. That level of productivity can increase your effectiveness by many multiples, and maybe even by orders of magnitude.

Next: Mistaking Efficiency for Productivity

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

To Control and Observe - Productive Software Development

There isn't a productive software development effort where controlling and observing aren't the substance of productivity. If you break down productivity to it's most fundamental element, you'll find that the understanding of how your software works is the essence of it all. Control and observe your software and you understand how it works. When you understand it, you can change it, work with it, and continue working with it. When it gets harder to make observations of how your software works, it becomes harder to remain motivated to do the right thing. In the extreme, we live out software projects in survival mode, and we saddle someone else with the responsibility of observing whether our software works. It's not the only way. It's the worst way.

The more that you can make observations about your software, and the more that those observations can be made in thought-sized chunks, the more easily it is to understand your software. To make an observation about your software, you have to be able to control that bit of software to do the thing who's side effects you want to observe. The easier it is to do that, the easier it is to understand it, and the more productivity you have. The time that you spend deciphering software that should be readily understandable is mostly wasted time. It's time spent that shouldn't be necessary and that can be eliminated.

Software that is hard to understand is software that is hard to control. Software that is hard to control is hard to understand. A small bit of code that you want to observe should be controlled with as little code as possible. If you can run that small bit of code directly without having to also control adjacent code, then you've got code that is inherently easy to control, and a level of productivity that is inherently greater. We tend to overlook the amount of time that software teams spend controlling and observing, and when things start to become difficult, we tell ourselves that we can't afford to pay attention to it. Ironically, paying attention to it solves the root problem.

Here are a couple of examples of code that is easy to control:
- In-memory objects that can be observed without loading their data from the database
- User interface workflow that can be observed without using the user interface
- Business logic events that can be observed without the auditing software making recordings
- Steps of complex business processes that can be observed without starting the process from scratch
- Secure operations that can be observed without having to authenticate
- Payment logic that can be observed without connecting to a payment processor

If I'm concerned with the functioning of in-memory objects, and if I need to load those objects from a database, then I have to add the database to the things I need to control, and to be concerned with as well. This isn't a natural, defacto quality of software, regardless of how common it is.

We generally accept that the lack of control as a defacto quality of software projects. We overestimate the control that we have, and we underestimate the value of increased control. We don't pay it much attention. We believe that that the limits to the control that we have is just part of the very fabric of software development, that it's unchangeable, and that there's no reason to dwell on it. We may not even have an awareness of it, or of the issue. This is the fatal flaw in software development thinking, because it's the issue of control that explains why software projects rapidly loose productivity and why the extent of that productivity loss is so great.

Maintaining the ability to control and observe is a conscious, intentional effort. If you're not on top of it all the time, it slips away. The traditional productivity loss curve and the traditional cost curve subsequently begin to immediately express themselves. You can tell yourself it's just the way it is, or you can realize the root cause of the problem and the solution and not only regain that lost productivity, but then go on to achieve a level of productivity that you wouldn't have believed possible from your previous perspective.

This issue isn't unknown. In fact it's well-known. The most common mistake in production management is not paying attention to how productivity loss happens and focus instead on increasing the efficiency of the work that is done amidst the accelerating losses in effectiveness. This mistake is the basis for most software production management at this point of industrial software development.

Next: Controlled Productivity

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

QA Missed Something

When a team is closing in on a release it may still find a flaw so terrible that the release opportunity might be missed. After the initial panic settles down, we go looking for the explanation of why a such a significant problem can remain hidden until so late. Inevitably, QA missed something.

Let me tell you something: there's nothing that QA has ever missed that developers didn't miss first.

Blaming testers for not discovering flaws that they did not create while simultaneously believing that we don't have to check our own work while is it fresh in our own minds is dishonest. There's no defect ever found by a tester that wasn't first designed and implemented by a developer.

Software is a production industry that generally still believes that the people who make things and the people who prove that those things are right should be different people doing two parts of a single job at different times.

Part of a tester's job is to cross check the work that has already been checked by the workers who create the software and who have the best, timeliest knowledge of it. When a tester isn't cross-checking, he's analyzing and doing exploratory work - looking for possible gaps in developers' thinking, understanding, and observation. When a developer's responsibility to deliver repeatably-checked work is passed over to a tester, neither is effective.

It's less of a tester's job to check that a developers' code is right, but that their tests are right, or at least that the result of the development so far is right - which includes tests that have been written before final inspection. That work doesn't start after the code has been written, it's starts before, as they work together to ensure that by the time the product and the tests make it to the final inspection, that all the ducks are already in a row, leaving the tester with the space to continue to question exactly what it means for the software to be right, and how to perform final inspections that hold up their part of the shared responsibility for intentional quality and productivity.

Final inspection is the last place that you want to find game-changing flaws and problems. It's the last place you should find them. It happens, but it should be the exception to how development is done, including testing, and the exception to how you treat final inspection.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Analysis: CodePlex Foundation - The Terms of Mutual Surrender

Microsoft announced the CodePlex Foundation yesterday. The strategy is quite compelling, and frankly, it points to a watershed moment, a turning point for the Microsoft platform and .NET community, as well as for Microsoft.

The CodePlex Foundation initiative translates resoundingly to one great thing: Opportunity.

History

In April of 2001, Microsoft released Beta 1 of the .NET Framework and Visual Studio .NET. At the same time, Jim Newkirk and a couple of industrious .NET early adopters with a track record in agile development in Java released NUnit, an open source unit testing framework for .NET.

Whether NUnit was the first open source application in the .NET ecosystem, who knows? Nonetheless, NUnit was the first open source .NET software for many .NET developers who looked to open source for answers even before .NET itself was released. Jim is presently a CodePlex Foundation advisor and Product Unit Manager for Microsoft's CodePlex open source repository.

To say that the rest is history is a predictable cliche. Whatever has happened in open source in the Microsoft space up until now, as astonishing as many .NET open source accomplishments are, it may pale in the face of the opportunities that are available in light of the CodePlex Foundation initiative.

Of the time between April of 2001 and the present, the following can be said of open source in the Microsoft space: It was a constant battle. Open source was stigmatized. We had to fight to use open source software in our jobs. And when we weren't fighting to use open source, we were fighting to not have to use commercial clones of open source software that often weren't nearly as robust as the open source.

Certainly things have been getting better in the past few years. More organizations have opened up to open source - especially when open source solutions are the best-of-breed - but Microsoft's own lingering reticence toward open source remained the elephant in the room, and a significant influence over the customer community's perception of open source.

And even though Microsoft has been making some inroads into open source over the past few years, Microsoft's own open source has remained constrained. With a couple of exceptions, Microsoft did not accept contributions to its open source from contributors outside of Microsoft. And Microsoft developers were discouraged from or disallowed reviewing open source code for fear of infecting Microsoft products with potential "viral" effects of some open source licenses.

Imitation is the Sincerest Form

In the past eight years, Microsoft has learned a lot from open source software. A number of Microsoft product ideas were proven first by open source projects. However, because Microsoft itself has not been open to open source solutions, and because a good bit of Microsoft's customer community follows Microsoft's lead on many fronts, the only way that Microsoft could provide its customers with the value available to open source users was to develop new products from scratch.

Microsoft's efforts at duplicating the value offered by open source were fraught with problems. Microsoft employees can't look at the open source code, so Microsoft's competing offerings weren't often too terribly competitive. They would be adopted by Microsoft customers nonetheless on the basis of Microsoft's merit as the largest commercial software foundry.

All this leads to an oft-repeated heart-breaking scenario for .NET developers who are already well-versed in best-of-breed .NET solutions that have matured in the open source world: showing up for work one day to learn that the company you work for has chosen to go with an immature, inferior solution that Microsoft had been left with little commercial choice but to build. This sinking feeling is followed up quickly with the realization that a company the size of Microsoft can't release innovations and fixes as fast as an open source project. The Microsoft clones of open source projects progress slowly while the open source solutions drive ahead with innovative and often more productive solutions.

Ultimately, this cycle repeated year after year and often created an ever-deepening insularism in Microsoft that exasperated the problem even further.

Opportunity

The CodePlex Foundation will bring influential open source projects under its auspices. The details aren't clear yet, but it's reasonable to assume that the foundation will support its projects the way that other software foundations support their projects, with protection for these projects as they are used in corporate and commercial contexts and who knows, maybe even some financial support will be part of the deal.

The single greatest opportunity that the CodePlex Foundation represents is an end to orthodox resistance to open source by Microsoft, and its customer community by extension, and the dawn of a new day of .NET where open source can be openly embraced.

In a potential tomorrow, the best tools for the job aren't sources for intellectual property suspicion, talented software craftspeople have greater freedom to use the tools that they have built significant mastery of, methodologies and techniques aren't driven by tool limitations, and innovation is free to move it its own pace, and can do so as a cooperation between industry and community.

And most importantly, in a world where Microsoft begins to embrace open source, it begins to subject itself to the open competitive forces that will make its products better, and make Microsoft itself a leaner, more agile company.

In this world, .NET open source champions aren't relegated to a relatively small backwoods, but are granted the broad regard and respect that is common place in the Java, Ruby, Python, and PHP worlds, among others with rich open source culture and history.

Also consider that in this world, we're one-step closer to the level of comfort with open source for Microsoft where community contributions to Microsoft's MSPL projects can be possible. The arrival of the CodePlex Foundation doesn't provide for this, but it does provide for the next few significant steps on the way.

This effort will bring Microsoft staff into close encounters with open source software and open source software projects, breaking down the barriers that the community has descried for a long, frustrating time.

I understand that in concert with this effort, Microsoft is also changing its policy for open source contributions for its staff, allowing staff to make limited contributions to open source without the requirement of oversight from Microsoft legal staff! This is a significant shift in policy for Microsoft and points to some significant reconsideration of policies and philosophies of the past.

What Price Freedom?

The source of Microsoft's trepidation over open source hasn't changed. The central issue is still, and will remain for the foreseeable future, intellectual property risk. The source of the trepidation for many of Microsoft's customers remains intellectual property risk.

The solution to the problem is as obvious as it is genius, but the price of freedom isn't without some non-trivial compromise, and a challenging new paradigm for open source leaders to consider.

The intellectual property risks can be greatly (if not entirely) mitigated when intellectual property is assigned to an intermediary, and that, among other services, is what the CodePlex Foundation is for.

Put frankly and directly, the CodePlex Foundation is given ownership of the code.

Per the CodePlex Foundation Assignment Agreement:

"Assignor assigns to Foundation its entire right, title, and interest in any copyright rights that attach to the Code and any documentation delivered with the Code."

Per the terms of the agreement, the Foundation grants an irrevocable license to the code to the original owner. Again, from the CodePlex Foundation Assignment Agreement:

"Foundation grants Assignor and its affiliates a perpetual, worldwide, non-exclusive, royalty free, irrevocable license, to reproduce, modify, create derivative works of, display, publicly perform, sublicense and distribute the Code (and derivative works thereof) as Assignor or its affiliates see fit, including the right for Assignor and its affiliates to sublicense the foregoing rights to third parties."

Ultimately, this means that it's business as usual for the open source projects, with the additions of the protections of the CodePlex Foundation, and greater opportunity for project adoption in spaces that were not previously open, and the chance to participate in what might be a more cohesive and cooperative open source community at large in the Microsoft space.

The Inevitable Distrust

There's no avoiding the issues of distrust that will surface from this. The usual suspicion is inevitable considering the history of Microsoft and open source, the invitation to open source projects to give ownership of their code to the CodePlex Foundation, the preponderance of Microsoft staff on the foundation's interim Board of Directors and Board of Advisors, and the branding of the foundation with an existing Microsoft brand: CodePlex (also the name of Microsoft's public open source repository and community site).

Microsoft has a habit of springing things on the community. But then, so does Apple, and so does Google. Love it or hate it, this is how product companies launch products and how they protect themselves. It doesn't always work out so well, and for my money, Microsoft is the least best of these companies when it comes to operating without early and continual customer feedback in product development, but, well, there you have it. That said, the CodePlex Foundation isn't exactly fully operational yet, and for all intents and purposes, this is the opportunity that the CodePlex Foundation has provided for community input, and the foundation staff has gone out of its way to make this point clear on the foundation's website.

The easiest way to staff up the foundation is to do what Microsoft did. It tapped the people who worked to make the CodePlex Foundation happen for interim positions in the leadership of the foundation. It also assigned some serious business acumen to the interim Board of Directors. Having Microsoft staff in-play on the Board of Directors is just a smart, immediately-sustainable thing for Microsoft to have done with the investment that has been set in motion. The Microsoft staff on the Board of Advisors include many people who are friendly to open source and who are personal friends of many people in the open source community.

And there are a few key names that should likely be on that list. Ayende Rahien and Jeremy Miller immediately come to mind - people who have brought a number of influential open source projects to life, and help to bring those projects into enterprises and ISV's around the world. But this thing is just getting started. It's a "soft launch", as one of the members of the Board of Advisors put it.

So why would Microsoft collude the namespace by naming this fledgling foundation after its CodePlex program? Sure, they're both about open source, and sure there's a healthy dose of Microsoft in the mix, but why not make the effort to disambiguate right off the bat?

John Peterson, a veteran software developer, former Microsoft MVP award winner, and attorney, has been helping me to understand the CodePlex Foundation agreements, and I like his take on the CodePlex naming, and why it's a good idea.

Microsoft has donated one million dollars to the foundation to get it up and running. It has an accountability to its stockholders for what it does with its cash. And while one million dollars might not seem to be much compared to Microsoft's bank balance, it's not exactly a trivial amount when it comes to charitable contributions to what appears to be a fairly radical cause.

The perpetuation of the CodePlex brand is just a good investment, and it likely helps this kind of move go down easier with folks in Microsoft's stockholder community who might not entirely understand what this open source kerfuffle is all about. Microsoft is making this thing happen. It's reasonable that it gets to pick the name, and to use a name that highlights other aspects of its open source efforts.

The Windup

I'm not usually the first person to extend unquestioning trust to Microsoft, and I started my day yesterday with the inevitable distrust and backlash, but I think there's something much more significant here that deserves more than just the usual distrust, and might likely be better served with an unusual trust.

Imagine for a moment if this effort had happened five years ago. Think of all of the often-frustrating pieces of the Microsoft stack that we've had to deal with - software influenced by open source, but missing the target because of Microsoft's policies on staff exposure to intellectual property risks - real or imagined.

Think of all the tools and all the libraries that have shipped from Microsoft over the past five years where your response was a despondent, "Oh no, not again." Part of the reason why we have to contend with these tools is that Microsoft, up until now, has not been able to truly learn from the many mature systems from the open source world that it is called to address with offerings of its own that cary Microsoft's implicit protection from intellectual property risks.

Microsoft isn't going to simply change its well-healed habits on a dime, but we're at a moment where turning in the right direction will set in motion the chain of events that will ultimately change Microsoft's culture in regards to it's ability to see, to leverage, and to participate in some of the incredible work that is being done outside of Microsoft's walls.

The Pitch

I'm hoping that the influential open source folks in the .NET community will consider the CodePlex Foundation's invitation, as odd as it may seem, to consider the possibilities for a future where the .NET community at large has the same common sense perspective on open source as the Java community, the Ruby community, and all of the other communities who's no-nonsense perspective on open source we often covet.

Sure there are lots of details to be ironed out with the CodePlex Foundation program, and the next few weeks will be telling in that regard, but it's with the participation of the open source community that the changes that we've been talking about for years can finally get underway.

I don't own an open source project that I've invested years of my life into, but I can guess what it must feel like to have it suggested that ownership of such a project be handed over to a foundation, and a foundation with very close ties to Microsoft at that. The open source community is being asked to meet the resistance half-way, and to hammer out a program that history will recognize as the turning point for Microsoft, it's customers, and almost every aspect of Microsoft community, culture, and product development.

If you're an open source project owner, think of the possibilities of having your framework or your product begin to reshape the expectations for craftsmanship of Microsoft staff and the greater Microsoft community at large. No one is in a position to require an open source leader to assign their copyright to an intermediary, but the first few influential open source leaders who do meet the CodePlex Foundation halfway will set in motion the kinds of pervasive, positive changes that will change all of our lives and careers for the better.

It's a relationship that starts with one hell of a compromise, but it could be the beginning of a beautiful relationship. Possibly even a historic one.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Do Agilists Understand Lean?

One of my previous bosses, Steven "Doc" List used to tell me that I'm at my best when I'm teaching. Sometimes, Doc's compliment would strike a raw nerve. Programmers on our team were entitled to choose whether on not to learn. This wasn't an explicit policy, but was engrained in our culture.

We valued learned people, but learning and teaching was not part of our culture or our organizational mechanics. We valued ready-made learning when it walked through the door, but our organizational learning didn't go further than the Agile Retrospectives materials and practice that was fashionable at the time. That's no fault of Agile Retrospectives per se. It's a fault of turning it into something fashionable, and inevitably conferring the unconscious orthodoxy onto it that was steadily growing in Agile methodology culture, often obstructing the line of site to the need for a higher order of teaching, learning, and management culture.

Unless team members understand that there is a requirement to be students in an organization, and to study under a teacher, pride and prejudice will likely obstruct the acceptance of a formal student/teacher relationship, and attempts at teaching will very likely devolve into the predictable butting of alpha geek heads over design and process ideas. And this portends obstructions to meaningful and methodical continuous improvement driven by program goals, and a rise of wild, uncontrolled experimentation.

In his book "Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead", John Shook tells the story of a manager who's supervisor once told him, "If the learner hasn't learned, then the teacher hasn't taught".

I used to tell Doc that it is painfully frustrating to teach knowing that our staff understood that they were not required either by culture or by supporting policy to be in the role of learner - especially when program and project expectations were not being met.

It's true that if the learner hasn't learned, then the teacher hasn't taught. It's also true that if the learner doesn't show up, then the teaching doesn't even begin.

Leaders at successful Lean organizations have pointed out that companies who have failed to duplicate Lean successes often do so by trying to adopt Lean as a process improvement effort rather than an effort to create a learning organization. Despite all of the interesting and beneficial mechanical aspects of Lean, Lean is about creating learning organizations. Student/teacher relationships and protocols are a part of learning organizations, with each role having the responsibilities of that role.

There has been a bit of a dust-up in the agile community lately as to whether Kanban work management is necessarily non-agile or anti-agile. The premise being that Kanban can foster environments of directed work, limiting workers' ability to self-organize, and fostering a disrespectful environment for workers as compared to forms of work management and organization common to agile methodologies.

The essential issue here is the issue of respect, but I find that the Agile perspective is willing to take advantage of incomplete and opportunistic definitions of respect. Kanban proponents have rightly pointed out that respect for people is an explicit Lean principle, and that self-organized work is still happening within Kanban workflows.

I see two perspectives talking past each other, and unfortunately, I see Kanban practitioners being backed into a corner, retreating, apologizing for Kanban, and softening the message to make it more palatable by the dominant methodology culture. Ironically, this was the exact position that the Agile was in at the start of the decade when Agile was struggling to make headway against prejudice and misrepresentation by the preceding traditional methodology culture.

Agilists are concerned about returning to the bad old days when disconnected managers directed work from outside the context of doing that work. It's a serious issue that deserves serious concern. It's a serious enough issue that it demands rigor on the part of the mainstream Agile community to engage the effort to understand Lean more deeply than the often cursory glances and biases projected at Lean and Kanban.

I sympathize entirely with the concern of returning to the bad old days of pre-Agile bureaucracy, but I'm equally concerned about the same tendency for Agile bureaucracy to occlude the meaning of Lean and Kanban.

My study of Agile began in 2000 when a mentor from Bell SIGMA turned me on to XP. My day-to-day immersion began in early 2001. I don't want to return to the bad old days either, but I don't want to go forward into a revised, 21st-century kind of bad old days issuing from the same mechanisms of bias and presumption that Agile itself faced, and often continues to face.

I wonder if agilists at large, in the spirit of inspection and adaption, are taking the time to understand Lean and the organizational and cultural context where Kanban thrives. With Agile, we asked organizations and cultures to consider change. I wonder now if Agile is able to respond to the same challenge.

On the surface, the following statement should rile a mainstream agilist. At least, it certainly used to rile me. It riled me enough not to act upon it even when my instincts told me that it might make a world of difference between immanent failure and rescuing a project, its team, and a considerable investment.

It's perfectly acceptable for a manager to direct the team from a position of traditional, hierarchical, directive authority.

I'm taking egregious advantage here in setting this stage. I'm purposefully leaving out the implicit context inherent in Lean. If I looked at the preceding statement through Agile's lens, I might very likely be worried about it. Looking at the statement through Lean's lens, I'm perfectly comfortable with it.

If we don't look at Kanban through Lean's lens, we're committing the anthropological cardinal sin in failing to realize that we're projecting cultural bias on what we're observing. We're even failing to recognize that Lean may have culture and behavior that is based on different assumptions and biases.

Kanban isn't a return to the bad old days of disconnected, directive authority because the position of management in-situ in a Lean organization isn't the same position of management that Agile efforts are commonly called to contend with and the behavior of management that many of Agile's protocols are shaped to deal with.

A manager of a Lean software development team isn't a remote figure who is no longer in the game. The manager is on the team, and he's one of the most competent technologists on the team. The manager in a Lean organization is also a teacher.

A team that includes a manager with directive authority is still self-organizing. The manager is internal to the team. A manager's expectations aren't disconnected from the reality of the work. And when those expectations aren't being met, he can chose to use directive authority to guide the team to counter-measures through teaching. It's also the manager's duty to help people on the team to develop critical thinking skills and instincts that serve problem recognition and resolution in support of the goal.

The expectation for team members to fulfill their duty as students is part of the managers directive authority. Refusal to engage in the protocols of the learning organization is deeply disrespectful to the organization of people as a whole, and to the manager as a person.

Looking at Lean through Agile's lens is perfectly reasonable. It gives a comparative perspective that can help us understand differences and find meaning. But ultimately, Lean should be seen through Lean's lens and should be assessed from the perspective of its native context.

There is a greater issue of respect and disrespect that is inherent in Lean as seen through the Lean lens. Respect is a two-way street. There is the respect for workers and the respect for managers, or teachers. The mutuality of respect is what makes respect possible. When respect ceases to be mutual, it ceases to be sustainable, and will soon disintegrate.

One of the most intractable issues we have in software development cultures is the lack of line management that remains technically-competent. In fact, line managers in software development are often people who choose to escape to management when they discover that they don't like making software.

We're presently dealing with the effects of several generations of software managers who don't really have much of an idea of what software development work is in detail, which means that these managers can't be effective teachers. Workers don't end up with the teaching that makes them effective and makes the work rewarding. The cycle perpetuates itself.

Agile has been a powerful palliative in dealing with this organizational and cultural snag. By putting a firewall between the deleterious effects of directive authority that is too far from the work and the work itself, agile succeeds in restarting the failing heartbeat of getting software made.

In the software industry, Lean is seen as a kind of specialization of Agile, and that's a unique thing for Lean in industry in general. It's also possibly a detriment and maybe even a disadvantage.

Agile is increasingly encumbered by its own presumptions of organization and culture. Agile's biases are slowly fading into background consciousness, becoming unconscious. As Lean is inevitably and unconsciously seen through the lens of colloquial Agile, many of the organizational assumptions and biases of Agile are projected onto Lean without even realizing that these biases are there.

Lean in its essence is a path to critical thinking, but not a solo path. It's a directed path. A Lean organization is a learning organization. It has teachers, students, curricula, and protocols all focused on meeting the expectations that support the organization's holistic goals for productivity and producing.

Lean is a means to find the unasked question. There appears to be an unasked question in the Agile perspective of Kanban and respect: Do we expect that a Lean organization is the same as an Agile organization?

This isn't meant to be a condemnation of Agile, but it is meant to point out that if Agile isn't careful, it will become the same kind of problem that it sought to solve.

On my project with Doc, I was valued as a teacher. I was the person who got executive support for the project. I shared product design responsibilities with our product owner. I would go to bat for the team when hard decisions needed to be advocated to the executive. I did the technical screening of candidates and made my recommendations to the team about hiring. And I was responsible for setting program goals and expectations for technical implementation.

In the end, I parted ways with the team when it became clear to me that the team had become intractably self-determining, which is a potential risk in self-organizing teams when technical competence and directive authority are not invested in the same person. Ultimately, the team failed, a non-trivial percentage of our small company revenue invested in the project was written off, and the entire team was dismissed.

This isn't a typical Agile scenario, but it is very much a possibility faced by teams in situations similar to ours. The failure likely points less to an implicit weakness in Agile and more to an explicit strength in Lean: respect is indeed a cornerstone of success, but if respect isn't holistic, it risks introducing the opportunism that can see to the failure of otherwise meaningful software development efforts.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Lean Reading List

I read Mary and Tom Poppendieck's first book on Lean Software Development, "Lean Software Development: An Agile Toolkit" in 2005. It went in one eye and out the other.

I was the Software Development track chair for Austin's InnoTech conference in November of that year. Mary was gracious enough to accept our invitation to come to Austin to keynote the track and to moderate a panel.

I listened to Mary's talk and got a few more clarifying tid bits from it, but mostly, I dismissed it at the time as some form agile sideshow that didn't quite measure up to the specifics that XP brought to the table. A number of us went out for burgers with Mary after the conference and continued the conversation, but I don't think anyone in the group was fundamentally moved by Lean.

I wasn't equipped with the experience to see Lean for what it was. Later, I realized the deep hubris that encumbered my thinking and the vanity that would lead me to expect that I could intuit a subject as vast as Lean from a single book, the way I could with XP and Scrum.

I suffered a serious set-back in 2007. My career's pinnacle dream project was flirting with disaster.

The company I worked for faced an intractable intellectual property constraint that limited our product's market to a small fraction of the whole. In November of 2006, I sold the company on an ambitious plan to solve the problem by building our own platform that we would have unlimited rights to sell. We started exploratory work and envisioning in December. The undertaking had significant executive sponsorship. When our board of directors voted to not fund the project, the CEO bought out the board and gave us the green light. We started official work in earnest in January.

In November of 2007, I sounded the failure warning alarm to my management, and I kept sounding it. In January of 2008, with the situation becoming increasingly intractable, I parted ways with the project and the company. Three months later, after having given the team a chance to pull itself together, the project was canceled and the team was dismissed from the company - from the most junior technologist, up to the executive overseeing the project.

I was frustrated by having felt handcuffed by Agile development orthodoxies that no longer fit the problems of the team, and yet were followed mechanically, and reinforced by management that was captivated by its first exposure to Agile.

I told some of the details of my experiences to a friend. We talked about Agile, in many variants, including Lean Software Development. My friend recommended that I read about the Theory of Constraints and recommended "The Goal: A Process of Ongoing Improvement" by Eliyahu Goldratt and Jeff Cox. In reading The Goal, I recognized the detailed mechanical process that I faced in the failed project. At that time, I hadn't connected what I had begin to learn in The Goal with what I know about Agile Development and the gaps in Agile that I had begun to see after seven years of immersion.

Following The Goal, I wanted to read more into some of Lean's roots. I had largely avoided the Toyota literature up till that point, and I wasn't convinced that I would get much from The Toyota Production System. I knew a number of people in my community who had read The Machine that Changed the World and The Toyota Production System, but I never really got the sense that the reading had connected them with the transformative learning that I had experienced in Agile development.

Before committing to throwing myself into the Toyota literature, I wanted a sample of what I might be getting into. I started with an article. I read "Decoding the DNA of the Toyota Production System" by Steven J. Spear and H. Kent Bowen published in the Harvard Business Review.

In this article, the authors talk about the things that companies seemed to miss when trying to duplicate Toyota's successes using Toyota's methods. I'm a sucker for stories of unasked questions. The rest of my reading about Toyota and Lean in general would be an exploration of the unasked question. The authors' message: Toyota's fundamental nature as a learning organization is often overlooked, with undue attention paid to Toyota's more obvious practices and mechanics.

And this is what drew me in to the Toyota literature. Here I saw the parallels to my failed organization, my own nature as a learner, a seeker, a pathfinder, and a teacher. I recognized the many of the intractable problems that I had observed in the behavior of my failed organization.

On my team, it had become an unspoken entitlement conferred that people were not required to take direction. This was largely exasperated by a management approach that was fixated on social experimentation and which was not capable of guiding the technical execution or product design imperatives of the project. It was an Agile methods laboratory that produced no real acceptable features for a year.

I learned in Spear's and Bowen's article that the organizational structure, mechanics, and protocols that I felt would benefit the team were the basis of Toyota's organization and culture as I was coming to understand them.

The Toyota Way by Jeffrey Liker was my first read into the Toyota literature. I chose this book to specifically continue learning about the Toyota DNA rather than dig into Toyota's specific process mechanics. Understanding that focusing on the process mechanics led to common problems in learning and adopting Toyota's methods, I wanted to hold off on the obvious aspects longer.

I read the Toyota Way with the observations of the Harvard Business Review article fresh in mind, as well as the mind-opening lessons about work management and problem solving from The Goal providing a backdrop.

Cautious of becoming yet another Toyota disciple, I took a turn away from Toyota-specific literature and back toward Lean in general and read "Lean Thinking: Banish Waste and Create Wealth in Your Corporation" by James P. Womack and Daniel T. Jones. This book tells the story of the companies and people taught by Womack and Jones as they traveled around the world after writing The Machine that Changed the World. It reinforced and what I had learned about learning culture and continuous improvement as well as organizational structure and process in the books that preceded it.

While this reading and studying was happening, I was also tweeting about my experiences and studies, and having numerous long conversations on the phone with some of my peers who had themselves started studying the same material. I also revisited my failed project with the previous product owner, who was still with the company, and still succeeding in his own work.

If I had read this material in a vacuum, without any of the constant interaction with my professional network, and the continual revisitation of previous failure, I believe it would have been a much less informative and transformative experience. My circle of friends and network of colleagues continues to inform my learning, and I expect that it will continue to do so.

I turned the reading back to software, choosing to read Mary and Tom Poppendieck's second book, "Implementing Lean Software Development: From Concept to Cash". Reading a Lean software book from my vantage at that time was a very different experience than when I had read the first Lean software book. The subject had now come to life. It was tangible, and it was deeper than what I had presumed previously. From here, my perspective of Lean Software Development takes meaning beyond my perspective of XP and Agile culture and mechanics.

I had organized the ALT.NET Open Space Conference the previous year, and hadn't want to simply fall into the trap of trying to duplicate an original moment. I wanted another theme for the second annual conference. Over the course of many of those conversations with friends and colleagues, we talked about the essential force of the ALT.NET movement as something akin to Lean's Continuous Improvement. The theme of the second conference became Continuous Improvement, with hopes that we would come to understand it more, and maybe understand better whether ALT.NET is indeed a Continuous Improvement culture.

I subsequently read another Toyota book, "Extreme Toyota: Radical Contradictions That Drive Success at the World's Best Manufacturer" by Emi Osono, Norihiko Shimizu, Hirotaka Takeuchi. This book takes on seeming contradictions in Toyota's culture and organization, such as the simultaneity of both a flat organization and a rigid hierarchical organization and the imperatives of a learning culture and continuous improvement that unify the two. The book also spoke about the climate of contradiction that Toyota uses to stimulate creativity and problem solving.

Tom and Mary accepted our invitation to come to the Continuous Improvement Conference in Austin and share their experience with learning organizations, software development, product development, scientific method, and leadership. I can't imagine that Mary and Tom got as much out of the experience as we did, but they had a lasting impact on our community.

Mary and Tom often say that Lean Software Development is informed more by the Toyota Product Development System than the Toyota Production System. Dave Laribee got to the Toyota Product Development book before I did and warned me that it was quite dry but worth reading, and it was. "The Toyota Product Development System: Integrating People, Process And Technology" by James M. Morgan and Jeffery K. Liker spoke a great deal more about requirements, design, planning for work, and creating workspaces, as well as the risk-mitigation processes that Toyota uses. This book also goes into greater detail about the Toyota's people and roles, and fostering "towering technical competence".

The TPDS book also talks about Lean leadership and the Chief Engineer role. It further reinforces the notion of managers and group leaders as having great technical competence, often knowing the work of their staff better than they do, and being paramountly responsible for teaching their staff though the scientific method and Toyota's A3 report technique.

Dave read David Anderson's book, "Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results" while I was reading the TPDS book, and I followed the TPDS book with David Anderson's book. I wish I would have read this book years ago, but again my biases obstructed my perception of the value I would get from it. In this book, David Anderson ties the Theory of Constraints directly to software development and translates pull systems and throughput accounting to the work.

Dave turned me on to Cory Ladas' writing on the Lean Software Engineering blog. I read a few articles, and then went back to the start of the blog and read forward. Much of that writing has been compiled into his book "Scrumban - Essays on Kanban Systems for Lean Software Development". Cory’s writing goes quite deep into pull systems and Kanban for software development. This material and David Anderson’s book offer a profound exploration of applying the mechanical aspects of Lean.

I put into practice what I learned both from my own experience and from studying and study groups on a project with a distributed team in Austin starting in August 2008. My instinct at first was to overlay their existing organization and process with Scrum. Instead, I looked for signs of what was keeping the organization back, established some basic measurements, and did my best to teach what I knew about how to solve these problems. It wasn't trivial work, but it reaffirmed my experience and study of both the cultural and behavioral aspects of Lean, as well as the mechanics, such as Kanban. The experience also continued to reaffirm XP practices as the tactical foundation of both Lean and Agile strategies.

While in Austin, Mary and Tom mentioned the book "Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead" by John Shook, which tells the story of the techniques and processes used in learning culture and learning organization, and goes much deeper into the the teacher/student relationship and responsibilities that are the foundation of Lean.

The book that really brought Lean Software Development together for me wasn't a book that is necessarily about software development, but about the kind of work that software development is: Product Development.

I'd heard Mary and Tom say for years that Lean Software Development is a closer kin to Lean Product Development than Lean Production, or in reference to Toyota, Lean Software Development is informed more by the Toyota Product Development System (TPDS) than the Toyota Production System (TPS).

In Don Reinertsen's, "The Principles of Product Development Flow: Second Generation Lean Product Development", many of the instincts, intuitions, and insights that had been guiding my own work were given a voice, and augmented with Don's experience and perspective. This is a book that is as much a watershed moment in my career as Kent Beck's "Extreme Programming Explained: Embrace Change". It's a book that stitches together many disconnected pieces of learning through a number of years of experience and observation, and then builds a new level on top of this reinforced foundation.

I think that this book deserves to be read much earlier in the list than in the chronology of my own studies. Even if you don't get past the half-way point - it's a clear elucidation on persistent problem in software development management, and the failure to manage software development from a product development perspective rather than a manufacturing perspective, and the failure of software managers to recognize that they don't know the difference.

I've had Atul Gawande's, "The Checklist Manifesto: How to Get Things Right" on my desk for the the past nine months. I haven't opened it yet. It's been on my mind.

On a recent visit to the Toyota plant in San Antonio, I met Mark Graban, who was also there for the tour. I asked him about the book, whether he had read it, and to try to get a feel for where I might put this book in my reading priority. I suggested to Mark that I've got a feeling that it might give me some insight into mistake-proofing, and for enriching the use and usability of my Lean work management app, Floverse.

Mark highly-recommended this book, and so, to the top of my list it returns. I'll update this article once it's done.

And there are yet more books that I've been meaning to get to, many of them in the nascent Lean Startup field, where Lean is applied as a methodology to entrepreneurship and product and business startup. These books will likely get a mention in this article at some point along the way as well.

Context

I wrote this article because people in my network asked me for some recommendations on books on Lean. The more I thought about it, the more I thought that listing some titles and links might not be entirely responsible.

The books that I read and the order in which I read them is inseparable from the context in which I read them. This isn't a canonical list and it shouldn't be treated that way. There are many, many more resources available.

I've benefited tremendously from the choices I've made for study and for engaging community to enliven that study. I wholeheartedly recommend the books I've talked about here, and I would even recommend going through them in the order that I read them. In retrospect, the reading order worked well as an evolutionary thread and helped me reinforce a deeper understanding of subtler aspects of Lean while continuing to layer on ever broader knowledge.

Nonetheless, your mileage may vary, and if your experiences are quite different from those I've laid out here, then you might disregard my own learning adventure and concoct your own.

Either way, foster a community to learn with, and sit at the feet of as many masters who'll tolerate your presence. Anything you learn from a book is just material until you light it up with experience (or reflection) and turn it into knowledge. Learning in isolation rarely has the yield of learning enlivened by experience and community. That's not always the case, but if you have a tendency to hide in a cave, understand that much of Lean is a social practice.

I still haven't read The Machine that Changed the World or The Toyota production System. I may read them at some point, but my goal isn't to consume every Toyota book that I can find. My goal is to synthesize as much understanding as I can, and the past two years have been very rewarding in this regard.

I recently founded the Lean Software Austin group, and I'm looking forward to continuing the study and the work in Lean principles and software development as this community grows as a learning organization itself. The story doesn't end here, but the narrative reading list does (for now).

For your convenience, here is an actual list of the reading I referenced in this article:

• Lean Software Development, "Lean Software Development: An Agile Toolkit by Mary and Tom Poppendieck
• The Goal: A Process of Ongoing Improvement by Eliyahu Goldratt and Jeff Cox
• Decoding the DNA of the Toyota Production System by Steven J. Spear and H. Kent Bowen (Harvard Business Review)
• The Toyota Way by Jeffrey Liker
• Lean Thinking: Banish Waste and Create Wealth in Your Corporation by James P. Womack and Daniel T. Jones
• Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck
• Extreme Toyota: Radical Contradictions That Drive Success at the World's Best Manufacturer by Emi Osono, Norihiko Shimizu, Hirotaka Takeuchi
• The Toyota Product Development System: Integrating People, Process And Technology by James M. Morgan and Jeffery K. Liker
• Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results by David Anderson
• Lean Software Engineering by Cory Ladas
• Scrumban - Essays on Kanban Systems for Lean Software Development by Cory Ladas
• Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead by John Shook
• The Principles of Product Development Flow: Second Generation Lean Product Development by Don Reinertsen
• The Checklist Manifesto: How to Get Things Right by Atul Gawande

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Myth of Developer Productivity

There are a couple of predictable approaches to an imminent car crash. You could throw your hands in the air, scream, and hope for the best, or you could keep your hands right where they are and try to pilot your way through it to the last responsible moment. If you're the pilot of a software team, offloading the responsibility for productivity is like taking your hands off the wheel before you've left your driveway.

The quickest way to shut the door to productivity is to try to solve it exclusively as a tools and automation problem with tools that promise "Developer Productivity". Tools and automation are essential parts of getting software done, but unless you have a firm grip on why you have productivity problems, taking someone else's word for why a tool or automation will solve your problems amounts to little more than a shot in the dark. And since my word is inevitably "someone else's word", please look more deeply into the issue beyond this article.

Productivity that Matters

There's productivity that matters and productivity that doesn't matter. That probably seems nonsensical - after all, if you could gain productivity, wouldn't it matter? It depends on wether the productivity you gain causes a commensurate obstruction at some other point in your software development bucket brigade.

One heck of a lot of tools that are sold specifically as "Developer Productivity" tools create productivity increases for the developers in your development pipeline, and kill productivity for downstream work centers, like testing, packaging, shipping, installation, configuration, and operations.

Focusing on developer productivity without considering the effects of developer productivity efforts on the rest of the whole pipeline will usually create momentary productivity; productivity that only developers will feel, and often only for a short period of unsustainable time.

Software teams and organizations continue to fail to realize sustainable productivity by continuing to make improvements in one area of the pipeline without realizing the cause-and-effect relationship between the localized improvements and resulting degradations in other parts of the pipeline.

Sustainable productivity is the only productivity that matters, and it's the only productivity that can withstand ever more continuous improvements.

Productivity is about product. It's the activity quality of product, or producing, or production - the "ivity" of product. It's about producing and the production of the only thing that really matters - the final product that can be used to achieve the business's goals.

If the goal of software development is just to create developer artifacts without ever delivering them, then developer productivity itself in isolation would matter a whole lot more. The problem with "developer productivity" is that it is inherently productivity in isolation. Productivity in isolation is often naive local optima.

Local Optima

Local optima problems happen all the time in software organizations. They happen when we try to make improvements in one area of the whole software development workflow without understanding the effects on other areas of the workflow.

Developer productivity as we know it colloquially is inherently a local optima concern. My first concern is the productivity of the entire fire brigade. The moment that there's some obstruction anywhere in the whole workflow, the problem will spread outward and poison adjacent work centers, rippling outward, and sometimes making quantum leaps into parts of the pipeline that are, on the surface, seemingly disconnected.

It's not enough to just understand that a change in one area of team's workflow will have effects on another area, it's also critical to understand which kinds of effects will create obstructions for the whole effort.

Greater productivity often come less from increasing the speed that we can do the work itself, and more from recognizing and decreasing the obstructions to production and producing. One of the most obvious ways to attack the productivity problem is to reduce rework and the things that lead to it.

Reducing rework means undertaking organizational and cultural changes. It means that the making of software and the proving that it's right can't be allowed to work at dramatically different paces, which is usually the case with most software teams (even teams using Agile methods).

Every time that testers get backed up in their work, untested software piles up in front of testing. That pile of untested software is the unproven foundation that developers will continue to build upon wether or not it’s sound. As the pile grows, the likelihood of rework grows geometrically, proportionate to the size of the pile, and the near certainty that structural design is insufficiently precise grows along with it.

When the making of software and the testing are disconnected from each other, and software makers and software testers work at full speed regardless of whether they're building up piles of risky inventory, then productivity is going to degrade.

In this kind of situation, doing things that increase programmers' speed isn't going to help productivity at all. In fact, the right thing to do is to either slow the programmers down until the testing inventory is cleared, or to have the developers change hats and work with testing to clear the obstruction. Increasing a developer's speed will only exasperate the problem.

Optimized developer productivity without simultaneously optimizing the entire pipeline is local optima.

Developer Productivity Myths

There are a number of tools and libraries that are sold under the "Developer Productivity" banner. These tools actually deliver developer efficiency rather than developer productivity.

Unless a tool's productivity proposition takes local optima into account, it flirts with negligence. If it does so knowingly and willfully, then it flirts with corruption; verily stealing value from its users and customers.

Consider Microsoft's re-interpretations of "Rapid Application Development" that drive its developer tools design:

Microsoft's Visual Studio enables developers to create input forms and data processing visually with little expense of time and effort. Developers get this work done very quickly, but the resulting systems features are inordinately difficult to test. Microsoft also packages tools that are specifically geared for "testers" in a separate product package, perpetuating the conclusion that doing work and proving that the work is right is the jobs of different people, creating the handoff boundaries that invite unproven work to collect as piles of costly inventory.

While this allows developers to be very efficient in the work of creating the code and artifacts that go into a feature, that efficiency isn’t realized as productivity. The software designs created by developer efficiency tools are unnecessarily and excessively difficult to test. This discourages developers from preventing the defects that pile up in front of the testing work center which also steals productivity from testers, exacerbating the problem of basing today's work on last week's unproven decisions.

Software organizations who are seeking and realizing higher productivity - that is, they produce end product to business sustainably and timely - come to understand that the inventories built up around handoffs and segregated work centers must be decreased and ultimately eliminated in order to really reclaim lost productivity. This has a profound effect on the shape of software teams and organizations, and in retrospect we come to see that our beliefs about organizational mechanics and culture were what kept us trapped in superstitious dark ages of software development productivity.

Regardless of whether tools like Microsoft Visual Studio (among others) seem impressive on the surface, failure to assess the effects of these tools on an entire pipeline will inevitably lead to mere momentary efficiencies and local optima. Worse yet, they will obstruct the learning that does in fact lead to productivity that matters, and will pull managers away from valuable work like teaching and facilitating and force them to become inventory managers and expediters.

It's not just the younger teams using Microsoft's visual tools who are subject to this problem either. Even advanced "agile" teams indulge in developer efficiency local optima like the hypercoding enabled by a higher order of developers tools. In some cases, hypercoding has even motivated urgent and wide-ranging changes to frameworks to optimize for a tool's use even in the face of compelling evidence that more meaningful sources of productivity are likely elsewhere. The compelling productivity of Ruby on Rails programmers despite the unavailability of extensive hypercoding tooling is a good example.

There's no one true answer to whether a tool is going to contribute to productivity. Sometimes just the creature comforts afforded by a tool are significant sources of productivity. But then, not all indulgences in creature comforts can be considered productivity enhancers either.

Ultimately, any of these tools can be used in a balanced, leveled software development workflow. However, until a meaningful understanding and representation of productivity takes root in software development, any team at any level of maturity is going to trade productivity that matters for mere efficiency.

Fixing the Problem

We can affect the things we control. When software projects go awry, we exert control. Any time we exert control without considering the impact on the entire fire brigade of software development workflow, we're going to create efficiencies at the expensive of our ability to produce.

The further away you are from the whole of the team and the workflow, the less likely you are to exert control constructively. As a senior manager, you might prescribe a suite of "Developer Productivity" tools after seeing a compelling presentation from a vendor that is specifically geared to affect your sensibilities from your perspective. If you're a developer, you might convince your team to adopt a "Developer Productivity" tool that demonstrably makes you a more efficient coding machine.

The decisions made by people who are too far from the whole are often a coin toss. There's little telling whether a team will perform better in getting products into the hands of the people who need them, and it’s often difficult to connect the decisions with the ultimate outcomes.

If you're encouraging or enforcing a team organization that disconnects the work and workers from the validation of their work, whether analysis work, design work, construction work, construction inspection work, packaging work, installation work, or operations work, you will inevitably create the conditions that encourage inventory build up and the subsequent obstructions, rework and general degradation of productivity. The shallow pursuit or mere localized efficiencies are more likely to happen when work centers fail to be shaped to productivity goals.

Making any of a number of mistakes that trade local efficiencies for productivity not only degrades productivity, but creates a cycle where the degradation accumulates, leading to the typical software cost curve that is ultimately a reflection of the degrading productivity curve.

Fixing the problem isn't trivial because no single local optimization will have a predictable effect, and a set of disconnected local optimizations degrade productivity even faster and even more unpredictably.

There's no doubt that we need to act on all levels, and ultimately this means decomposing the problem and working on different levels of an organization an at different work centers. But the local things we do to fix the problem have to extend from a holistic understanding of the software development system.

If you want to fix your software development system, find the problems with the system and then understand how the parts contribute to the problem. Evaluate the success of each effort to fix the parts by the effect that it has on the system.

Developer productivity can just as easily be a reality as a myth. Developers are obviously significant contributors to producing software. But an approach to "developer productivity" that isn't also an approach to organizational productivity is often not likely to do more than transfer value from your organization’s treasury to the coffers of a vendor who is more than happy to assume ownership of your precious resources.

We can’t get software done without software tools – this is true - but choose wisely. Your whole team’s productivity is at stake.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com