Wednesday, February 25, 2009

A Week with a Lean Manufacturer

I recently spent a week working with the team from MTM Recognition, in Oklahoma City. Raymond Lewallen leads the team, and he hired me to work with MTM's developers on context specification, both for .NET code using mSpec and for the web UI using Selenium and RSpec.

MTM is a Lean manufacturer. The company makes awards, trophies, rings, and a host of recognition materials for businesses, schools, and sports teams and events. Most of the work that MTM does is small batches of products that are frequently custom designed and built for a single batch. Its circumstances are almost a cliché of quintessential Lean.

While I was there, Ray and Troy, one of MTM's developers, took me on a on a tour of the plant. Troy had previously worked on the manufacturing side and knows his way around. He transferred to IT when he wouldn't stop building unofficial shadow IT apps to serve his purposes as a manufacturing worker. Talk about domain expertise!

I had a suspicion before we entered the plant that I had sincerely hoped would play out. The plant didn't disappoint!

Walking into MTM's plant, I couldn't find the trappings of Lean. There were some obvious visual indicators in some of the workspaces, but there were no glaring neon signs announcing Lean. What we saw mostly were signs of people making the products that MTM's customers had ordered. We had to ask to have queues and signal cards pointed out, and the folks at MTM pointed out how Lean was manifested and used.

Lean was just how things were done, but it wasn't the whole point. The point was carrying on the company's tradition of building quality products that took MTM from a one-man-in-a-garage operation to a respected and trusted 100 million dollar company.

When software developers latch on to a new thing, we tend to want it's trappings to be so flagrant and obvious that you can't but trip over them in our work areas or in conversation. Considering It's understandable, but it’s also a bit disconcerting.

After living through a catastrophic failure with Agile that ended with the costly write off of a year's worth of time spent and the dismissal of the entire product team, my concerns over Agile's loss of focus on product, producing, and productivity were powerfully punctuated.

Agile teams can become fascinated on the process to the distraction of the purpose that the process serves. In the case of the aforementioned failed project, it had become so bad that at one point the product owner and I mused that it seems like the team believed that it had become charged with producing Agile process rather than software product. When Agile becomes the whole point rather than merely a means, things can go very wrong.

What really struck me during the tour of the MTM plant was the master crafts people on that MTM has on its staff, working in design, production, and quality assurance. MTM has some very competent and talented people plying their trade in service to MTM's customers. Everything from sculptors, to jewelers, painters, metal workers, and typesetters. It's clear that producing at the pace and quality that MTM has built a reputation for requires people of considerable skill.

There's power in the tools that we use, and as crafts people we can even derive pleasure from our appreciation of fine tools themselves, but we are so susceptible to having our attention lured away from the most important aspect of having them and wielding them: productivity, product, and producing.

I appreciate Lean because it is so focused on product and producing, and on shaping an organizing that is hell-bent on the goal. It makes no apologies for remaining focused on the goal and for bending everything to the service of the goal and sustaining its achievement and its betterment.

Agile can be a disruptive technology. It’s vital as software developers that we remain vigilant against our susceptibility to allowing agile to be disruptive to our software development efforts themselves.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Monday, February 23, 2009

Decade of Agile, Dawn of Lean

The way that Agile works on a team that is new to Agile and the way that it works on a mature team is necessarily different. The sudden breaking with old habits that happens during new adoption is a radical change. The on-going process of improvement on long-lived, mature Agile teams is usually (but not always) less radical.

Mature Agile teams might still undergo radical improvements now and then, but they will have been steeped in continual change and have developed a taste and facility for change, or at least for the methodology of change. New Agile teams are typically not only not steeped in the protocols and social contracts that they are changing to, but they’re also not usually steeped in change itself.

Becoming used to change is a big part of making a deeper connection to Agile. Habituation to change unfolds over time, and over time it cultivates a vision, a pragmatism, and a facility to change in the people on the team. The deepening of the facility with change will change the protocols and social contracts that are necessary to holding things together while change is under way.

Agile usually starts up in the midst of the organizational and cultural chaos that is the typical status quo of software projects. Presently, software teams have a heritage of insufficient technical abilities compounded by ineffective project management and altogether too-weak product definition.

The circumstances of a project beget the particulars of its methodology. The vast majority of Agile teams are in their initial phase, and most of what we hear about Agile methodologies reflects this particular perspective of Agile development. The fact that most Agile teams are just beginning has a tremendous effect on the content of the Agile dialog, and the frequent repetition of this perspective inevitably reinforces itself.

Scrum is a methodology that is really well-suited to getting started with Agile, and consequently a lot of what is believed about Agile at-large is rooted in Scrum. The downside of the self-reinforcement feedback loop surrounding Scrum is that it tends to drown out the rest of Agile - the technical competencies that permit Scrum to be sustainable beyond the initial Agile boot strap phase.

At the outset, Scrum gets away without the necessary technical competencies because the team and surrounding organization are coping with radical changes in the protocols and social contract. Allowances are made for lower productivity in recognition of the team facing radical changes. But once the bloom is off the rose and real expectations come back into effect, colloquial Scrum efforts can fall apart.

Scrum's tighter feedback and greater adaptability demand engineering techniques that permit these things to happen. The engineering practices are what keeps Scrum from shaking itself apart over the long term and from preventing a new Agile team from regressing Scrum into some variant of waterfall.

Scrum's biggest challenge is the calcification of its own orthodoxy. We end up merely with "Scrum teams" rather than high-functioning software product development teams. We end up with teams who continue to apply practices that are appropriate to the early Agile adoption phases long past the adoption phase. We end up with entirely naive and mechanical software development teams flying the Scum banner, and this is exactly the state of affairs that we were trying to move away from.

Filling a cork board with hundreds of index cards that describe features that we're not going to work on for months is really just an updated version of the hundreds of pages of specification documents that come at the outset of traditional software projects. The index card version is a contemporary manifestation of the same presumptions and thinking that drive waterfall.

We end up with the over-specified product backlog by failing to recognize our unrecognized presumptions. There are lots of unasked questions that remain dangerously-silent in mainstream Agile. It's these presumptions that are hurting software product teams' chances of growing beyond the basics and surviving past a guided adoption.

Without growing beyond the basics, Agile efforts end up collapsing under their own weight when the project heat is turned up for real. Many Agile teams have only been taught how to begin and don't really know how to sustain let alone how to improve while simultaneous getting things done. A team that isn't mining for it's invisible, un-asked questions and presumptions is going to have a hard time establishing the improvement practices that will permit it to sustain beyond bootstrapping.

If advanced agile practice is anything, it's a diamond-tipped drill boring into our unconscious presumptions of what makes software product development work. The deeper this drill goes, the longer we can sustain. As soon as the drill stops, the momentum of our projects' challenges catches up and outpaces our ability to deal with them as easily and as effectively as when we were at the very beginning, when things we simpler.

There's no standing still. If you're not moving forward, you're falling behind. I don't typically find admonitions this strong in Scrum orthodoxy, but I do find it readily in Lean. This might suggest that an inevitable result of years of continuous improvement of Agile practices ultimately leads Agile teams to Lean, and to a forming and shaping of Agile fluency to Lean. I've seen this play out a number of times in my own professional network as agilists with many years of day-to-day agile project experience have begun to see answers to their questions in Lean. Maybe this isn't true for everyone, but it's happening with enough regularity that in the very least it deserves consideration.

We're faced with an uncomfortable question: Is Lean an "advanced form" of Agile development.

If Lean is seen as an advanced form, it inevitably confers a beginner stigma onto teams who aren't doing Lean and can be seen to diminish other forms. The problem here isn't that Lean may or not be an advanced form, but that we have a community of software practitioners who see being a beginner as stigmatic.

Nothing but nothing puts software product development efforts in harm's way like a team that is uncomfortable with standing right where it is. If a team is unwilling to be a beginner, then it is incapable of becoming advanced. Wether or not Lean is an advanced form, we seem to have an aversion to accepting that we do indeed have advanced forms, and that they are often incomprehensible and impenetrable from where we stand right now.

Dave West made a great point about the purpose of a well-stocked backlog in his article on InfoQ entitled Lean and Agile: Marriage Made in Heaven or Oxymoron.

Agile works best when there is a huge product backlog and when there is a large amount of churn in the composition of that backlog. Churn results from conversations about story essence; story priority; feedback from developers about stories not understood; stories that turned out to be easier or harder to develop than expected; stories that had to be refactored to make them more understandable or more tractable to development; and feedback from user acceptance of stories completed.

Eventually, churn diminishes and the product backlog becomes stable. The addition of new stories reduces to a trickle and the prioritization of stories changes only nominally. At this point it becomes even more tempting to consider the product backlog as an inventory.

He makes a really astute observation about Agile that I really like and can identify from some of my project experiences. Dave says that agile is a process "for building a consensual theory of the world: with an artifact being a byproduct - an expression - of that theory". And this, I think, is where Agile and Lean can be seen to part ways, and also to not part ways. This is one of those "radical contradictions" that can be seen on the surface to point to a inconsistencies and incompleteness in Lean thinking that is more likely to be one of its enablers.

Here's a bold statement that is likely to raise the hackles of many a latter-day agilst: There is no inherent need for building a consensual theory of the world in product development. There's no doubt that everyone involved in realizing the product vision has to understand what that vision is, and that churn is a way to get there, but there are approaches to managing that process that aren't predicated upon a "huge product backlog".

There are certain circumstances where churning over a huge product backlog is valuable. And there are yet other circumstances where where such churn is unnecessary, and yet others where it can be downright destructive and a significant contributor to failure. The chosen approach can't be cookie-cutter and mechanical. A project’s context has to drive the specifics of its Agile approach. There is no one, single Agile that can work for all levels of maturity.

Some product development efforts start with a strong product vision and some product development efforts start without a strong product vision. And which ever path the project is on dictates the kind of protocols and social contracts that can go into shaping the process, as well as the process of improving that process.

A huge backlog is often one of the necessary trappings of a team that isn't steeped in the practicable know-how of just-in-time development. The huge backlog theory-proving that Dave talks about may be more than just valuable in the absence of just-in-time fluency, it's likely a necessity and working without it might very well be negligent. Working with a large product backlog helps a team to gel and helps a product vision come together. These are truly valuable things to do, but they come with some cost. If it's a necessary cost, then it should be paid.

Filling a product backlog with stories can be a good exercise in socializing the product vision, but there are other ways to socialize the product vision without risking the waste that a pre-mature backlog often generates. The value of just-in-time is great, but the means of doing it isn't typically readily apparent via Scrum.

Lean offers a lot of experience in doing just-in-time development, but the body of knowledge is quite large. It would be a mistake to believe that one or two books by the Poppendiecks will give you the lay of the Lean land the way that one or two books on Scrum can spell out the entire methodology. That's not a sleight against Scrum. One of Scrum's greatest strengths is its relative lightweight. If Scrum were as voluminous as Lean, the Agile movement may have never made it to the mainstream. Scrum’s relative silence on the technical excellence that allows Scrum to succeed is why more organizations begin with Scrum rather than Extreme Programming, regardless of Extreme Programming’s early prevalence over Scrum.

I value my experience in Scrum because it prepared me for eventual undertakings in Lean. Without the well of enthusiasm for software development that I get from Agile development, I might not have undertaken the sheer volume of study and community engagement that helped me to finally see what Lean is about beyond the endeavors of the Poppendiecks to communicate Lean in digest form to the software world. Returning to the Poppendiecks’ books now, I see that I had missed the significance of much of what I was reading.

I've discovered and cultivated a lot of abilities that I didn't have when I started doing agile on a day-to-day basis in 2001. I'm ten times the software product person that I was then, and Agile is responsible for a much-accelerated growth as a developer, a tech lead, and ultimately a product designer and chief.

I can do things now that I couldn't do then. I fit my methodology to where I am in my understanding and and my ability to execute, lead, and socialize product development. Many of the things I choose to do now are downright heresy to mainstream agile, but I'm not exactly where mainstream agile is right now.

Here's are a few examples of things that I'm intimately-comfortable with now that I had only started entertaining just two years ago:

  • I don't rely on the embedded (on-site) customer.
  • I don't do all-hands team estimation a la XP Planning Game.
  • I don't carry a huge product backlog.

I'm simply not abandoning these practices and leaving gapping holes in the methodology. I'm replacing these practices with practices that are commensurate with my current level of understanding and experience, which are radically different than they were even a couple of years ago.

Toyota recognizes a role called the Chief Engineer. I'm not pompous enough to say that I have the depth of experience to confer such a title to myself, but it's the closest approximation that I've found that describes what I do on software projects. I'm conformable playing a relatively equivalent role on software projects, and I've been playing that role in practice in an ever-increasing capacity for a few years.

The distinct organizational qualities and processes that distinguish Lean from mainstream agile permit me to make a departure from the prescriptions of mainstream Agile and to do my Agile work with what I see as greater effectiveness across the board.

Ultimately, I'm taking on more responsibility for definition and specification. I'm doing this because I'm used to working at a pace that requires less of the variation and oscillation that can shake a mainstream Agile project to pieces at high speeds. I'm synthesizing more inputs and factors faster than what can be done by a committee and I'm leaning on considerable technical competence to convert product vision and product design into work items that I can guide a team through. I’m more open to input from people on the team, and I’m willing to see where things go when people feel strongly about outright dissent. And I’m comfortable with putting my foot down when I see that a point I’m making is not understood due to lack of experience and perspective when I can guide the practices that will eventually feed someone’s understanding.

Dave West commented on optimization, "Lean views software development as a process for moving from conception to product. It wants to optimize that process, albeit in a radically different way and with radically different values than traditional (e.g. Taylorism) attempts at optimization."

Lean does seek optimization, but it doesn't view software development as merely a process. An agile perspective of Lean may be biased to see Lean as a process, but Lean's perspective of Lean, internally consistent with itself, doesn't see it this way. Lean is fundamentally a way to cultivate and sustain a learning organization.

Optimization serves a goal that is outside of the typical purview of Scrum. Optimization drives design quality. Design quality drives productivity, sustainability, adaptability, and ultimately customer satisfaction. The reasons for optimization are found in the technical and engineering excellence underpinnings of Agile development. XP has more to say about this than Scrum because delivering technical excellence is not Scrum's purpose. However, XP practices are often how implementation gets done in a Lean software project.

Dave rightly points out that software development is not like production and that we should be cautious about applying a production methodology to software development, "Lean needs to take off the 'production glasses' and look at Agile and elements of the agile development process from a holistic perspective that includes all seven of the Lean principles".

The Poppendiecks often raise this same point and caution that software development is more like product development than production. Specifically, Lean software development is more like Lean product development, and we can look to resources like Toyota's Lean Product Development System rather than the Toyota Production System for insights and a vision of how Lean product development works.

Regardless of whether Lean is applied to product development or production, Lean principles apply equally to both, and these principles are often seen as conflicting with Agile. Indeed, in some circumstances, Lean is in-conflict with Agile and seems to contradict Agile, but which agile, in which circumstances and on which Agile team at which part of the process of maturation and improvement?

I think it's a mistake to look at the practices of Lean software development and conclude that things are being done in a particular way because of a misunderstanding of and a lack of experience with Agile. Many new Lean practitioners have been in deep Agile immersion since the beginning of the 2000’s. I think that it's specifically due to the broad and deep experience with Agile and in constantly pushing the limits that many people are now finding answers to questions in Lean.

That's not to say that Agile itself has any arbitrary limits, but the majority of the Agile population at this point in time is focused on adoption rather than maturation, and insights on problems found at Agile's quantum barrier can be found in Lean. And exposure to the fifty years of methodological and practical experience in applying and improving Lean arse inevitably having an impact on the way that we've been practicing XP and Scrum.

It's not that the recent spate of Lean exploration and adoption is due to inexperience with Agile. It's happening in spite of tremendous experience with Agile. In fact, some rather deep experiences with Agile seems to be leading people to Lean with a rather predictable regularity - even considering the influence of mere hype and relative novelty. But these people don't appear to be abandoning Agile, but rather adapting Agile to fit within Lean as a means to practice Lean as a software development discipline.

I didn't become interested in Lean because I wasn't interested in Agile. I became interested in Lean because I felt that I had started to hit a glass ceiling in Agile. I was trying to get to the next level and the ready part of the community and it's lore, literature, and protocols were just not helping me get there. I'm not saying that the next level for me wasn't already present in pockets in the Agile community, but it was readily-available and present when I went looking in Lean.

When I really started to put my mind and my effort into Lean, I experienced something very similar to what I had first experienced when my mentor from Bell Labs first introduced me to Extreme Programming ideas, and when I read Kent Beck's Extreme Programming Explained. I experienced again what I think of as the "ugly duckling" syndrome, where all of these doubts about the orthodoxy of what I was doing previously were laid out in black and white as well-known anti-patterns, calling me forward into a deeper realization of my own internal notion of common sense.

At the same time, I was irrevocably thrust into an unpopular movement of software development that took several more years before becoming acceptable and loosing it's pariah stigma. At least the ugly duckling went from being an ugly duckling to a beautiful swan. XP practitioners carved out a place for agile in corporate IT shops, product companies, and conferences amidst some non-trivial resistance and even ridicule at times.

Thanks to the brilliant business savvy of the ScrumMaster Certification program, Agile made deep headway into the mainstream, and XP'ers backed into a much-eased Agile climate by the broader path that Scrum had carved. And because it was just common sense, we laid Scrum down on the solid technical foundations of XP and got back to work making software products and improving ourselves.

Here we are ten years later and those same people are doing exactly what they've always done. They've gone where their common sense points them, and for some people, that means adopting Lean. It's rather ironic that this conclusion is as unpopular with the Agile mainstream now that Agile was to the pre-Agile mainstream then.

As not just a Lean student and practitioner but also as an agile veteran, I find the kind of mechanical orthodoxy that has taken root in the Agile mainstream to be categorically wasteful. But I look forward to the churn and contradictions that will inevitably lead to what comes next.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Monday, February 16, 2009

Complexity isn't Subjective, Our Reactions to it Are

Software people have significantly-different understandings of what complexity is. Regardless of what software people believe complexity to be, the perception of complexity affects most software developers in the same way: it influences the choices we make about how to build software. What a software developer, development manager, or even a stakeholder believes about complexity might determine whether the project is put in harm’s way.

Complexity isn’t subjective. It’s a hard measure of the number of inputs into a system, the number of moving parts in the system, and the number of decisions that the system makes that lead to different execution paths. But we often use “complexity” to describe what it like getting our heads around something unfamiliar, which can be a purely emotional response.

Two people saying “I don’t want to do it that way, it’s too complex” can have utterly different perspectives of complexity. If so, then they likely have utterly different and likely incompatible approaches to software development. It’s more likely that people who typically use incompatible approaches will not find themselves agreeing on complexity.

Between the start and finish of a project, a lot of things happen. Software is created, and more software is created. Some of that software interacts with other bits of itself. And more software is created and more interconnections are made. The natural increase in code and interconnections during a project while the software gains more features creates real complexity.

With more code, more natural complexity is added to the system. Complexity is going to accrete, the issue is: how fast? And the single most influential factor in how fast complexity increases is the approach we choose based on what we think complexity is. When we don’t make the right decision, we end up with more complexity than we need. Understanding how complexity is created and how it affects software and projects is a essential part of software physics and helps us to understand how to keep projects healthy.

Complexity isn’t an expression of discomfort with approaches that we have no meaningful experience with. Certainly lack of experience can put a project at risk, and some of that risk might manifest from accidental complexity that results from inexperience, but inexperience itself isn’t complexity.

Ultimately, the point of dealing with complexity isn’t the elimination of complexity, it’s the elimination of accidental and unnecessary complexity. Some forms of complexity are desirable and we choose to add some essential complexity in order to improve other desirable design qualities, and even real complexity isn’t the differentiator between software efforts that succeed and software efforts that fail.

When a developer complains of complexity, it's a sign of something that likely needs to be given some attention. It not entirely uncommon, though, to find that what has been called out as complexity isn’t a reflection of the software or the approach itself, but of the programmer’s emotional content. It’s still a matter of concern, but it might not be a technical concern.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Monday, February 09, 2009

Design Flaws, Hernias, and Anemic Quality

Runtime defects are relatively easy to find.  They make undesirable things happen that you can see with your own eyes.  Some defects are certainly harder to observe than others.  They might show up only in certain uncommon circumstances, or you might need to do some search and discovery with some special tools and instrumentation.

Relative to finding design flaws, the effort required to observe runtime defects is often a walk in the park.  And while we put a lot of effort into finding and resolving runtime defects, design flaws can drain more money and value from businesses than the runtime defects that consume the better part of quality assurance efforts and attention.  Not to downplay the risks that runtime defects pose to business, but a business that isn't also mitigating the considerable risks posed by design flaws is ensuring that the erosion of the value of software assets will progress exponentially, signaled by sharp, unexpected drops in productivity.

Design flaws are unseen because they manifest specifically as productivity problems and we tend to not track productivity in software development work in meaningful ways, and we rarely reconcile the suboptimal productivity of businesses in general with their business software.

Runtime defects corrupt data, crash applications, cause outages, enable security breaches, process business transactions incorrectly, and lead business people to the wrong conclusions and decisions.  Defects are “in your face”.  So are design flaws, but if you don't know to be vigilant toward them, you likely won't pay them the decisive attention that they deserve.

Imagine a design flaw as a herniated disk in your spine.  The hernia pushes out on the surrounding muscles, bones, and nerves in a way that is not natural to them.  This kind of misalignment not only causes problems with it's own structure, but also manifests in structural problems in adjacent systems as well. 

For example, a back problem can cause changes in posture and movement to compensate for weakness and pain, changing the way that adjacent and supporting structures and systems align and operate.  A spinal problem might manifest as a knee problem.  The secondary problem might not only be indicative of the original problem, it might also deteriorate to the point of being as bad or worse than the original problem.

The insidious effect of secondary effects is that solving the original problem may not resolve the secondary problem.  The secondary problem might be an indirect result of the original problem, and may continue to deteriorate even once the original problem is resolved.  This happens when new habits are formed when learning to compensate for the original flaw. 

Imagine a graphical representation of the goodness of a good software design as a series of adjacent alignments.  They don't compromise each other's structure or operation.

straightlines

A design flaw in this system would appear as a hernia - a bulging misalignment in the midst of the surrounding order.

hernia

Design flaws often go unnoticed.  As work progresses and a system grows, new features and code that are physically or functionally adjacent to a flaw are inevitably shaped to it.

Often this isn’t done purposefully.  The shape of the code that we build upon determines the shape of the code that we’re building.  Adjacent code will either export order or disorder depending on whether and how much it is flawed.  If it exports disorder, then it spreads productivity problems as well.

adjacent 

The most obvious flaws are known to us by their "workarounds".  But flaws with workarounds are just those few very obvious flaws.  Most flaws don't manifest to our awareness right away, if ever.  We might have a spark of insight one day that illuminates the flaw, or we might have an opportunity to work with someone who wields some non-trivial design flaw mitigation skills who recognizes and points out the disorder.

When we finally become aware of a design flaw and if we have the resources to fix it, we have to be careful of what happens to design when we allow ourselves to become overly-fixated on fixing just that part of the problem that we’re looking at.

The goal is to correct the design and to restore productivity, but often we leave behind the secondary effects, either because we haven't become aware of them yet or because of an excessively-narrow focus on the original problem.

fixed

This is how we end up with those areas of our systems that aren't right and are difficult to work with but when you ask someone why the design is the way it is they respond with something like, "Dunno.  It's always been that way.  We don't touch that code.  There must be a reason it's like that or else it wouldn't be like that.  So we don't change it.  It's just one of those things.  It's the way software is.  You know?"

 Comic

I know that software often ends up like this, and I also know that it doesn't have to end up like this, and I know how to avoid turning software into these messes, and I know that the ways and means can be taught and learned.  Perhaps most importantly, I know why we shouldn't let software degrade into these unknowable messes.

You Can't Afford It

Design flaws are expensive.  They cause the same value loss to ripple through our systems that runtime defects do.  However, with design flaws it can take you a lot longer to realize that the flaws are there and to locate the wound in the system that value and productivity continue to leak out through.

The longer that design flaws remain in your system, the more secondary flaws are introduced.  Those secondary flaws are also the primary causes of their own secondary flaws.  This inability to control the density of flaws in a system only gets worse, accelerating exponentially.  It’s the principle causes of the need to completely re-write systems that may only be two or three years old.

Having tests for our code only makes it possible to continue to make changes to our systems, but it doesn't say that we can make these changes quickly enough or effectively enough to make these necessary changes feasible and practicable.  Code that is awkward to work with has tests that are awkward to work with, doubling the inefficiency of solving the problem.

The simple issue is that you can't afford to have these kinds of problems in your systems.  You can't afford design flaws just as you can't afford runtime defects, and frankly, the more design flaws you have, the more likely that you'll have runtime defects, and the more difficult they will be to find and to fix.

We have a rather naive understanding of quality in the software industry and we’re only now starting to understand the physics and economics of software development enough to understand the importance of quality and how to make it happen.

While we continue to have anemic quality that preoccupied with chasing after only a portion of issues in quality, we’ll continue to suffer.  Our productivity will be far less than it should be and value will erode much faster than it should.  But there are some signs that pockets of the software industry are waking up from the software dark ages and putting quality into play in ways that are much more meaningful than the mere bug-hunting that has dominated software’s concept of “quality” for the longest time.

As we learn to apply the lessons from our own industry as well as from other industries, not only are our abilities to achieve quality improving, but our understanding of the purpose of quality deepens.  We are showing some signs of the industrial maturity that suggests that we may yet reach up and reclaim the productivity that we continue to squander to our primordial struggles with software production.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Tuesday, February 03, 2009

Sunni Brown’s Graphical Recording of My Agile Leadership Talk

Sunni did a graphical recording of my talk at ProductCamp Austin.  The talk was entitled Toyota's Chief Engineer as a Model for Agile Product Development Leadership.

(click the image to view full-size)

Agile Leadership

Thanks, Sunni!  And thanks, ProductCamp Austin!


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Sunday, February 01, 2009

Teaching, Symbology, and Intellectual Materialism – The Chasm is a Vacuum

If you could use a tool and do good works with it, but if you didn't know the name of the tool, would you be less of a craftsperson? How much less? Would your works be less valuable to the people who are served by them?

The world is filled with people who don't understand advanced subjects, but who would like to learn. And the world is blessed with many knowledgeable people who can't understand why the things that they've discovered, learned, and elaborated are not being soaked up by everyone else.

Sadly, the people who are among the first to gain knowledge in a subject often tend to make that knowledge unpalatable to the rest of humanity.

On either side of the issue are people who have fundamentally different reactions to a subject's symbology. Intellectual materialists celebrate and adore a subject's symbology, and the mainstream simply doesn't.

To a mainstream learner, a subject's symbology and pattern language can be a hindrance to learning. This fine point isn't often perceived by those bright and creative people who are quick to recognize, adopt, and deeply-learn an emerging subject, and quick to revel in the accomplishment through the recitation of its symbology within innovator social circles.

When intellectual materialists turn outward from their social circles to readily share their knowledge with the world, the involvement with symbology is often so powerful that they simply can't resist its indulgence. In these moments, the mainstream learner often dismisses many brilliantly-simple and game-changing ideas because they're wrapped up in intimidating symbology, like "Liskov Substitution Principle", "Cyclomatic Complexity", or "Inversion of Control".

These are terms like a lover's whispers to an intellectual materialist. They verily celebrate and decorate the knowledge behind the words, and this celebration of knowledge itself is a deeply-ingrained and vital aspect of intellectual materialism culture. The collection of knowledge and the celebration of knowledge is often as essential as its rightful and meaningful application.

And because the trailblazers aren't often aware of this, they are often only effective as teachers to other early adopters, leaving a chasm between themselves and the mainstream that is a stumble in what should be the continuum of communication along the learning curve.

The Chasm is a Vacuum

Adoption Curve

The chasm in the technology adoption curve is a bit of a stutter-step in what is otherwise a smoothly-flowing transfer of knowledge from people on the left who are earlier adopters than people on the right.

The word "chasm" describes hiccup in the shape of the population distribution curve, but it doesn't describe the social dynamics. Socially, the "chasm" is a "vacuum".

Intellectual materialists from the left-hand side of the adoption curve spend some amount of time amongst themselves incubating emerging ideas before they venture - either accidentally or purposefully - into the mainstream with the ideas. During the incubation, their insular communication patterns are even further reinforced and institutionalized. The celebration of symbology becomes even deeper ingrained through repetition and elaboration.

When intellectual materialists first meet mainstream learners and reach out through these well-ingrained patterns, the ideas and even their representatives are often met with a resistance that doesn't seem to be commensurate with the goodness of the good news of new understanding. If innovators aren't met with resistance, then they've likely just bumped into a new intellectual materialist, pulling them over to the left-hand side of the curve rather than having transformed knowledge - preparing it for, and transitioning it to the right.

When knowledge is first offered across the cultural seam and is met with resistance, the respective cultures recoil into themselves, leaving behind the vacuum. The vacuum shows up on the graph of the technology adoption curve as that blank spot between early adopters and the early majority known as, "the chasm".

Anxiety and Fear

At the point in the continuum where the people who live in the left-hand side of the curve meet the people in the middle of the curve, a lovely handshake should take place. These two neighbors should forge life-long bonds of respect and recognition building on the near-infinite creative potential of diversity, continually transforming and communicating knowledge across the cultural seam.

Far too often an early adopter will bump into someone from the mainstream who will extend his hand across the seam in greeting and say, "Hi, my name is Joe. Lovely day isn't it?" To which the intellectual materialist replies, "Cyclomatic inversion of Liskov test-driven domains of concern segregation substitution interface... in principle". Which in early adopter land roughly-translates to, "Hi-diddly-ho, neighborino!"

The mainstream learner smiles nervously, backs away from the seam, and accelerates away. He returns later with a sign that reads, "Warning: Irrelevant, Academic Clap Trap Beyond this Point", but finds it unnecessary as the intellectual materialist has returned to his capitol city to participate in the ceremonial Reciting of the Terms of the Sacred Symbols at the dedication of an all-new, top-of-the-line, ultra-modern, Class-A echo chamber that his entire society has decided to retreat into for the next five to ten years.

Mainstream learners are often more receptive to new ideas when they're not introduced with the psychic weight of the subject's symbology. It's not because the symbology is just too much to absorb. It's because the symbology and nomenclature, starting with the grandiosity invested in the new subject's name, usually over-states the complexity of the subject.

Innovators and early adopters give names to things that are celebrations of knowledge. They capture and replay the exaltation of the moment of crystallization of the idea by every intellectual materialist that comes to assimilate the new idea.

The name is the idea's material. The name is more than descriptive, it's the psychic texture of the idea. And none of this makes a lick of difference to someone in the mainstream who doesn't also value an idea's texture, but who just wants to learn how do do something so that he can in fact get something done.

Mainstream learners first try to figure out whether some subject is worth learning before they commit their time to learning. There's a lot of anxiety invested in this filtering. The sheer volume of new things bombarding a mainstream learner's priority filter is literally mind-boggling. The subjects that make it through the priority filter are the ones that seem less intimidating, and frankly, the grandiose symbology that is the mark of intellectual materialism is often very intimidating.

As soon as a student's mind touches anxiety about a subject, we've lost a significant portion of their ability to remain present to the subject. The teacher's first duty is to protect the student from anxiety and fear because these things are the anti-matter of learning and knowledge.

Respect the Learner's Tolerance for Intimidating Symbology

The answer isn't to start over with a new symbology (although it would be nice if some thought were given to the effects of symbology on the vacuum when the symbology is established). As teachers, and as the people whose cultural segment on the adoption curve creates most of the intimidating symbology, we have to be conscious of when and where the use of the symbology is appropriate.

The time at which a subject is being introduced to a mainstream learner is often the least appropriate time to introduce a subject's symbology. It might have no effect, or it might have a positive effect, but there's a strong chance that it will have an altogether negative effect - repelling the learner from the subject and the learning experience.

There's no reason to entertain the risk when any subject can be introduced without introducing the symbology. Doing this means that the psychic weight of the intimidation of the symbology isn't brought into play as an obstruction to the learner's capacity to be open and receptive.

Protecting the learner's cognitive space from the anxiety derived through intimidating symbology is a teacher's scared responsibility. It's akin to a physician's oath to do no harm.

There's usually nothing inherent in the symbology that helps a teacher communicate the subject that the symbology represents. It's the subject that it's important, and the subject isn't its name. A subject's symbology can be introduced later, once the subject has been taught past the point of a learner's inherent initial anxiety with new subjects. Until then, there's no reason why a teacher can't introduce a subject, teach it past the intimidation hump, and then and only then introduce the symbology.

Filling in the Vacuum

Sooner or later, enough knowledge is communicated though the vacuum that it eventually dissipates, and the continuum resumes with normal, expected fluidity. This happens when enough early adopters - be they originally from the left of the chasm or the right - work long enough to bridge the gap. If there were more teachers who understood that the role of a teacher bridging the gap is as translator and transformer rather than symbolist, then the vacuum may never even grow from the width of a humble seam.

The strange and predictable phenomenon of the chasm isn't a necessity of the adoption curve, it's the result of a cognitive impedance mismatch between two sub-cultures that fail to recognize the significance of each other's differences.

Intellectual materialists are so over-stimulated by symbology that their behavior borders on the mania of addiction and obsessive compulsion. The solution to the manifestation of the chasm is to respect the human dispositions that turn seams into vacuums.

If we avoid leading with symbology, we avoid inducing the fear response in the mainstream learners, and this goes a long way toward avoiding inducing the chasm. This is well within the reach of the people on the left-hand side of the curve. It simply requires the same discipline as teachers as they require of their students.

When teaching across the void, teach to empower! When empowered, a subject's symbology won't diminish a learner's capacity to be receptive to the subject, and open to learning.

When empowered, a mainstream learner is just as likely to join the celebration of the subject through the subject's symbology just as intellectual materialists do! That doesn't mean that a mainstream learner will have crossed over the seam and become an intellectual materialist, it means that he has been given a chance to see the beauty and power of a subject from his own cultural perspective.

This shared experience from different cultural perspectives is the diversity that will lead to even more creativity and refinement. It's through this diversity that knowledge becomes even more powerful, with new possibilities and applications emerging that lead to the next big idea. And it's through the practice of this diversity that the sheer waste of time and resources that result from the cultural vacuums can be avoided.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com