The way that Agile works on a team that is new to Agile and the way that it works on a mature team is necessarily different. The sudden breaking with old habits that happens during new adoption is a radical change. The on-going process of improvement on long-lived, mature Agile teams is usually (but not always) less radical.
Mature Agile teams might still undergo radical improvements now and then, but they will have been steeped in continual change and have developed a taste and facility for change, or at least for the methodology of change. New Agile teams are typically not only not steeped in the protocols and social contracts that they are changing to, but they’re also not usually steeped in change itself.
Becoming used to change is a big part of making a deeper connection to Agile. Habituation to change unfolds over time, and over time it cultivates a vision, a pragmatism, and a facility to change in the people on the team. The deepening of the facility with change will change the protocols and social contracts that are necessary to holding things together while change is under way.
Agile usually starts up in the midst of the organizational and cultural chaos that is the typical status quo of software projects. Presently, software teams have a heritage of insufficient technical abilities compounded by ineffective project management and altogether too-weak product definition.
The circumstances of a project beget the particulars of its methodology. The vast majority of Agile teams are in their initial phase, and most of what we hear about Agile methodologies reflects this particular perspective of Agile development. The fact that most Agile teams are just beginning has a tremendous effect on the content of the Agile dialog, and the frequent repetition of this perspective inevitably reinforces itself.
Scrum is a methodology that is really well-suited to getting started with Agile, and consequently a lot of what is believed about Agile at-large is rooted in Scrum. The downside of the self-reinforcement feedback loop surrounding Scrum is that it tends to drown out the rest of Agile - the technical competencies that permit Scrum to be sustainable beyond the initial Agile boot strap phase.
At the outset, Scrum gets away without the necessary technical competencies because the team and surrounding organization are coping with radical changes in the protocols and social contract. Allowances are made for lower productivity in recognition of the team facing radical changes. But once the bloom is off the rose and real expectations come back into effect, colloquial Scrum efforts can fall apart.
Scrum's tighter feedback and greater adaptability demand engineering techniques that permit these things to happen. The engineering practices are what keeps Scrum from shaking itself apart over the long term and from preventing a new Agile team from regressing Scrum into some variant of waterfall.
Scrum's biggest challenge is the calcification of its own orthodoxy. We end up merely with "Scrum teams" rather than high-functioning software product development teams. We end up with teams who continue to apply practices that are appropriate to the early Agile adoption phases long past the adoption phase. We end up with entirely naive and mechanical software development teams flying the Scum banner, and this is exactly the state of affairs that we were trying to move away from.
Filling a cork board with hundreds of index cards that describe features that we're not going to work on for months is really just an updated version of the hundreds of pages of specification documents that come at the outset of traditional software projects. The index card version is a contemporary manifestation of the same presumptions and thinking that drive waterfall.
We end up with the over-specified product backlog by failing to recognize our unrecognized presumptions. There are lots of unasked questions that remain dangerously-silent in mainstream Agile. It's these presumptions that are hurting software product teams' chances of growing beyond the basics and surviving past a guided adoption.
Without growing beyond the basics, Agile efforts end up collapsing under their own weight when the project heat is turned up for real. Many Agile teams have only been taught how to begin and don't really know how to sustain let alone how to improve while simultaneous getting things done. A team that isn't mining for it's invisible, un-asked questions and presumptions is going to have a hard time establishing the improvement practices that will permit it to sustain beyond bootstrapping.
If advanced agile practice is anything, it's a diamond-tipped drill boring into our unconscious presumptions of what makes software product development work. The deeper this drill goes, the longer we can sustain. As soon as the drill stops, the momentum of our projects' challenges catches up and outpaces our ability to deal with them as easily and as effectively as when we were at the very beginning, when things we simpler.
There's no standing still. If you're not moving forward, you're falling behind. I don't typically find admonitions this strong in Scrum orthodoxy, but I do find it readily in Lean. This might suggest that an inevitable result of years of continuous improvement of Agile practices ultimately leads Agile teams to Lean, and to a forming and shaping of Agile fluency to Lean. I've seen this play out a number of times in my own professional network as agilists with many years of day-to-day agile project experience have begun to see answers to their questions in Lean. Maybe this isn't true for everyone, but it's happening with enough regularity that in the very least it deserves consideration.
We're faced with an uncomfortable question: Is Lean an "advanced form" of Agile development.
If Lean is seen as an advanced form, it inevitably confers a beginner stigma onto teams who aren't doing Lean and can be seen to diminish other forms. The problem here isn't that Lean may or not be an advanced form, but that we have a community of software practitioners who see being a beginner as stigmatic.
Nothing but nothing puts software product development efforts in harm's way like a team that is uncomfortable with standing right where it is. If a team is unwilling to be a beginner, then it is incapable of becoming advanced. Wether or not Lean is an advanced form, we seem to have an aversion to accepting that we do indeed have advanced forms, and that they are often incomprehensible and impenetrable from where we stand right now.
Dave West made a great point about the purpose of a well-stocked backlog in his article on InfoQ entitled Lean and Agile: Marriage Made in Heaven or Oxymoron.
Agile works best when there is a huge product backlog and when there is a large amount of churn in the composition of that backlog. Churn results from conversations about story essence; story priority; feedback from developers about stories not understood; stories that turned out to be easier or harder to develop than expected; stories that had to be refactored to make them more understandable or more tractable to development; and feedback from user acceptance of stories completed.
Eventually, churn diminishes and the product backlog becomes stable. The addition of new stories reduces to a trickle and the prioritization of stories changes only nominally. At this point it becomes even more tempting to consider the product backlog as an inventory.
He makes a really astute observation about Agile that I really like and can identify from some of my project experiences. Dave says that agile is a process "for building a consensual theory of the world: with an artifact being a byproduct - an expression - of that theory". And this, I think, is where Agile and Lean can be seen to part ways, and also to not part ways. This is one of those "radical contradictions" that can be seen on the surface to point to a inconsistencies and incompleteness in Lean thinking that is more likely to be one of its enablers.
Here's a bold statement that is likely to raise the hackles of many a latter-day agilst: There is no inherent need for building a consensual theory of the world in product development. There's no doubt that everyone involved in realizing the product vision has to understand what that vision is, and that churn is a way to get there, but there are approaches to managing that process that aren't predicated upon a "huge product backlog".
There are certain circumstances where churning over a huge product backlog is valuable. And there are yet other circumstances where where such churn is unnecessary, and yet others where it can be downright destructive and a significant contributor to failure. The chosen approach can't be cookie-cutter and mechanical. A project’s context has to drive the specifics of its Agile approach. There is no one, single Agile that can work for all levels of maturity.
Some product development efforts start with a strong product vision and some product development efforts start without a strong product vision. And which ever path the project is on dictates the kind of protocols and social contracts that can go into shaping the process, as well as the process of improving that process.
A huge backlog is often one of the necessary trappings of a team that isn't steeped in the practicable know-how of just-in-time development. The huge backlog theory-proving that Dave talks about may be more than just valuable in the absence of just-in-time fluency, it's likely a necessity and working without it might very well be negligent. Working with a large product backlog helps a team to gel and helps a product vision come together. These are truly valuable things to do, but they come with some cost. If it's a necessary cost, then it should be paid.
Filling a product backlog with stories can be a good exercise in socializing the product vision, but there are other ways to socialize the product vision without risking the waste that a pre-mature backlog often generates. The value of just-in-time is great, but the means of doing it isn't typically readily apparent via Scrum.
Lean offers a lot of experience in doing just-in-time development, but the body of knowledge is quite large. It would be a mistake to believe that one or two books by the Poppendiecks will give you the lay of the Lean land the way that one or two books on Scrum can spell out the entire methodology. That's not a sleight against Scrum. One of Scrum's greatest strengths is its relative lightweight. If Scrum were as voluminous as Lean, the Agile movement may have never made it to the mainstream. Scrum’s relative silence on the technical excellence that allows Scrum to succeed is why more organizations begin with Scrum rather than Extreme Programming, regardless of Extreme Programming’s early prevalence over Scrum.
I value my experience in Scrum because it prepared me for eventual undertakings in Lean. Without the well of enthusiasm for software development that I get from Agile development, I might not have undertaken the sheer volume of study and community engagement that helped me to finally see what Lean is about beyond the endeavors of the Poppendiecks to communicate Lean in digest form to the software world. Returning to the Poppendiecks’ books now, I see that I had missed the significance of much of what I was reading.
I've discovered and cultivated a lot of abilities that I didn't have when I started doing agile on a day-to-day basis in 2001. I'm ten times the software product person that I was then, and Agile is responsible for a much-accelerated growth as a developer, a tech lead, and ultimately a product designer and chief.
I can do things now that I couldn't do then. I fit my methodology to where I am in my understanding and and my ability to execute, lead, and socialize product development. Many of the things I choose to do now are downright heresy to mainstream agile, but I'm not exactly where mainstream agile is right now.
Here's are a few examples of things that I'm intimately-comfortable with now that I had only started entertaining just two years ago:
- I don't rely on the embedded (on-site) customer.
- I don't do all-hands team estimation a la XP Planning Game.
- I don't carry a huge product backlog.
I'm simply not abandoning these practices and leaving gapping holes in the methodology. I'm replacing these practices with practices that are commensurate with my current level of understanding and experience, which are radically different than they were even a couple of years ago.
Toyota recognizes a role called the Chief Engineer. I'm not pompous enough to say that I have the depth of experience to confer such a title to myself, but it's the closest approximation that I've found that describes what I do on software projects. I'm conformable playing a relatively equivalent role on software projects, and I've been playing that role in practice in an ever-increasing capacity for a few years.
The distinct organizational qualities and processes that distinguish Lean from mainstream agile permit me to make a departure from the prescriptions of mainstream Agile and to do my Agile work with what I see as greater effectiveness across the board.
Ultimately, I'm taking on more responsibility for definition and specification. I'm doing this because I'm used to working at a pace that requires less of the variation and oscillation that can shake a mainstream Agile project to pieces at high speeds. I'm synthesizing more inputs and factors faster than what can be done by a committee and I'm leaning on considerable technical competence to convert product vision and product design into work items that I can guide a team through. I’m more open to input from people on the team, and I’m willing to see where things go when people feel strongly about outright dissent. And I’m comfortable with putting my foot down when I see that a point I’m making is not understood due to lack of experience and perspective when I can guide the practices that will eventually feed someone’s understanding.
Dave West commented on optimization, "Lean views software development as a process for moving from conception to product. It wants to optimize that process, albeit in a radically different way and with radically different values than traditional (e.g. Taylorism) attempts at optimization."
Lean does seek optimization, but it doesn't view software development as merely a process. An agile perspective of Lean may be biased to see Lean as a process, but Lean's perspective of Lean, internally consistent with itself, doesn't see it this way. Lean is fundamentally a way to cultivate and sustain a learning organization.
Optimization serves a goal that is outside of the typical purview of Scrum. Optimization drives design quality. Design quality drives productivity, sustainability, adaptability, and ultimately customer satisfaction. The reasons for optimization are found in the technical and engineering excellence underpinnings of Agile development. XP has more to say about this than Scrum because delivering technical excellence is not Scrum's purpose. However, XP practices are often how implementation gets done in a Lean software project.
Dave rightly points out that software development is not like production and that we should be cautious about applying a production methodology to software development, "Lean needs to take off the 'production glasses' and look at Agile and elements of the agile development process from a holistic perspective that includes all seven of the Lean principles".
The Poppendiecks often raise this same point and caution that software development is more like product development than production. Specifically, Lean software development is more like Lean product development, and we can look to resources like Toyota's Lean Product Development System rather than the Toyota Production System for insights and a vision of how Lean product development works.
Regardless of whether Lean is applied to product development or production, Lean principles apply equally to both, and these principles are often seen as conflicting with Agile. Indeed, in some circumstances, Lean is in-conflict with Agile and seems to contradict Agile, but which agile, in which circumstances and on which Agile team at which part of the process of maturation and improvement?
I think it's a mistake to look at the practices of Lean software development and conclude that things are being done in a particular way because of a misunderstanding of and a lack of experience with Agile. Many new Lean practitioners have been in deep Agile immersion since the beginning of the 2000’s. I think that it's specifically due to the broad and deep experience with Agile and in constantly pushing the limits that many people are now finding answers to questions in Lean.
That's not to say that Agile itself has any arbitrary limits, but the majority of the Agile population at this point in time is focused on adoption rather than maturation, and insights on problems found at Agile's quantum barrier can be found in Lean. And exposure to the fifty years of methodological and practical experience in applying and improving Lean arse inevitably having an impact on the way that we've been practicing XP and Scrum.
It's not that the recent spate of Lean exploration and adoption is due to inexperience with Agile. It's happening in spite of tremendous experience with Agile. In fact, some rather deep experiences with Agile seems to be leading people to Lean with a rather predictable regularity - even considering the influence of mere hype and relative novelty. But these people don't appear to be abandoning Agile, but rather adapting Agile to fit within Lean as a means to practice Lean as a software development discipline.
I didn't become interested in Lean because I wasn't interested in Agile. I became interested in Lean because I felt that I had started to hit a glass ceiling in Agile. I was trying to get to the next level and the ready part of the community and it's lore, literature, and protocols were just not helping me get there. I'm not saying that the next level for me wasn't already present in pockets in the Agile community, but it was readily-available and present when I went looking in Lean.
When I really started to put my mind and my effort into Lean, I experienced something very similar to what I had first experienced when my mentor from Bell Labs first introduced me to Extreme Programming ideas, and when I read Kent Beck's Extreme Programming Explained. I experienced again what I think of as the "ugly duckling" syndrome, where all of these doubts about the orthodoxy of what I was doing previously were laid out in black and white as well-known anti-patterns, calling me forward into a deeper realization of my own internal notion of common sense.
At the same time, I was irrevocably thrust into an unpopular movement of software development that took several more years before becoming acceptable and loosing it's pariah stigma. At least the ugly duckling went from being an ugly duckling to a beautiful swan. XP practitioners carved out a place for agile in corporate IT shops, product companies, and conferences amidst some non-trivial resistance and even ridicule at times.
Thanks to the brilliant business savvy of the ScrumMaster Certification program, Agile made deep headway into the mainstream, and XP'ers backed into a much-eased Agile climate by the broader path that Scrum had carved. And because it was just common sense, we laid Scrum down on the solid technical foundations of XP and got back to work making software products and improving ourselves.
Here we are ten years later and those same people are doing exactly what they've always done. They've gone where their common sense points them, and for some people, that means adopting Lean. It's rather ironic that this conclusion is as unpopular with the Agile mainstream now that Agile was to the pre-Agile mainstream then.
As not just a Lean student and practitioner but also as an agile veteran, I find the kind of mechanical orthodoxy that has taken root in the Agile mainstream to be categorically wasteful. But I look forward to the churn and contradictions that will inevitably lead to what comes next.
Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience
Learn more about my work and how I can help you at ampgt.com
