Testing User Stories - Problem Development Before Solution Development

User stories can be wrong. Business Analysts and Product Owners can be wrong. They can be the best representatives of user interests but they're often still proxy users. The most expensive mistakes to be made in software product development are made when we get the requirements wrong. A quick way to increases the risk of proceeding to solution development on the basis of incorrect requirements is to not prove that they're right. Or, in other words, to not test them.

The only way to definitively validate that the problem statements for a software product are right is to put the working software in users' hands.

User interviews, focus groups, simulations, and mockups are steps along the way, but their purpose is to reduce the time and cost of getting users in front of an experience that is immersive enough to validate assumptions. Allowing users to experience our proposed solutions to their problems clarifies whether the problem is understood.

Generally, we're taking too long to get stories validated - even in colloquial Agile development. Sending a user story through an entire development process in order to validate it is too much. In order to validate user stories, we need to collapse the solution development timeline so that feedback can be had without the expense of the solution development. We commit to the solution development cost once we've validated the problem.

There are two distinct (but inextricably-intertwined) steps that make up a development process that recognizes the value of reducing the time-to-feedback for testing user stories. The later step is the plain-old Solution Development that we use regardless of whether we recognize a preliminary first step. The earlier step is Problem Development.

Problem Development is software development. There's code. There's deployment. There's testing. There's design and analysis. There's process. There's just much more of it in Solution Development (even Agile development) than in Problem Development. And there are different kinds and variations of these activities in each. Problem Development is software development that works hand-in-hand with business development.

Problem Development and Solution Development have different goals. Problem Development is concerned with feeding Solution Development with input that has less risk (at a time when risk is the riskiest).

Assumptions are liabilities. User stories are assumptions until they're validated. To validate user stories, deliver software experiences that propose to solve the problems that user stories describe, and measure the feedback. There's delivery in Problem Development, but not final delivery.

Final delivery is a concern of Solution Development. Final delivery requires long-term sustainability. Solution Development is concerned with long-term sustainability. When we work toward achieving long-term sustainability before we need it, we end waiting until the longer Solution Development process plays out before getting feedback on user story validity. Which is too long.

Problem Development is concerned with short-term sustainability. The reason that it has to be sustainable at all is that the output of Problem Development is the input to Solution Development, and these two steps are ultimately one process that flows together.

It's far too easy to see Problem Development as "throw-away coding", but it's absolutely not. Problem Development is a way to manage some of the biggest risks in software development - risks that often go un-addressed and are frequently realized as wasted effort and resources, and lower productivity. This kind of risk isn't effectively managed with practices that are satisfied by "throw-away" code.

If throw-away code is the output of Problem Development, then Solution Development has to undergo a (mostly) cold start, benefiting from far less of the momentum created by Problem Development. It might be necessary to make the kind of tradeoff that creates these kinds of batch transfer problems, but making a habit of it is a path to a cumulative productivity deficit.

The kinds of practices that characterize Problem Development might look like software development heresy (or at least "Agile" heresy) to many disciplined solution developers. The realities of Problem Development mean that traditional aspects of Solution Development are short-circuited - after all, Problem Development is looking for a shorter circuit.

The Problem Development practices might even seem irresponsible, but recognize whether you're looking through a Solution Development lens. For just as Problem Development practices can seem irresponsible from a Solution Development perspective, applying Solution Development practices to the point in the timeline best suited to Problem Development can be equally as irresponsible.

Problem Development is about testing or validating user stories' problem statements. Solution Development is about building software based on validated problem statements.

The whole of development process is concerned with finding the way with the least wasted effort in getting from concept to cash.

Feeding un-validated user stories into the Solution Development transformer is often the most efficient way to increase wasted effort.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Myth of Iteration 0

"Iteration 0" is the initial phase on an Agile project when collaboration tools such as a source code repository server, a continuous integration server, and distributed build and test agents, as well as document tools like wiki servers, and other tools are set up. It's often also a time to configure individual workstations and team members' tools in preparation for everyone starting work.

"Iteration 0" is a bit of a popular misconception that is a side-effect of software projects initiated by technologists - which sounds a bit ridiculous at first. After all, who else but technologists to do software projects? Of course technologists are the right people to do software projects, but are all technologists the right people to execute project startup? And for that matter, what is "project startup" as differentiated from the rest of the work? Is it something that should be approached differently than the rest of the work?

The technology-focused project initiation isn't necessarily the wrong thing to do, but it's often only the right thing to do from within the technologist's perspective, and that perspective can be limited. A tool-biased perspective is a challenging thing to overcome by someone whose moment-by-moment work is necessarily suffused with a constant focus on tools, or whose initial career path passes through a lengthy time of tool focus.

If we literally codify the "first iteration" as "Iteration 1", we technologists can use a bit of geekery to make allowances for a pre-iteration focused on technology. As programmers, we're accustomed to counting a list of things starting from zero. If the list of iterations starts at the "zeroeth" position, but work is only scheduled to start on the first position, then we get an extra "unclaimed" iteration to work with: "Iteration 0".

Any way you cut it, the first iteration is the first iteration, regardless of the numerical designation assigned to it - and regardless of creative license and word games. If you choose to have a zero-based project schedule, then we should naturally change the schedule to state, "Deliver something user-valued in Iteration 0". Which, consequently, is not an invitation to technologists to insert an "Iteration -1" at the head of the schedule.

Over the past few years the caution to not exit any iteration having only shipped technology has been whittled down to a more technologist-friendly caution against exiting an iteration having only shipped frameworks. It's a narrower interpretation of a deeply-powerful principle. It sets the stage for the institution of a technology-focused "Iteration 0".

Nonetheless, it's sometimes necessary to devote a period of work entirely to technology concerns - especially if technology concerns become obstructions to user-valued concerns. And it's all too easy for technology to become an obstruction to delivery in a technology product delivery effort. But this isn't the problem with "Iteration 0".

"Iteration 0" is a project startup issue that is technology-focused startup rather than a product-focused project startup. There's an implicit assumption in "Iteration 0" that - while perfectly reasonable from a technologist's perspective - isn't necessarily an optimal path for project startup.

If a project startup is executed with "all hands on deck", then coordination of the people involved becomes an issue that must be dealt with right of the bat, including the setup of tools that support the coordination.

So, is it necessary to start a project with an amount of people that necessitates coordination and collaboration tools? If project startup is executed with a small workcell of skilled pathfinders, do they need the tools that necessitate the "Iteration 0" work? Can a project startup be executed without the need for the collaboration tool setup?

And more importantly: Is there a benefit to executing project startup without the full compliment of technologists that we'll need for full-steam-ahead solution development?

The answer to this is usually a resounding "Yes", there is a benefit. A good bit of the reasoning and guidance is found more in the Lean Startup and Lean Product Development bodies of knowledge than the colloquial Agile body of knowledge.

There's a time to activate a full compliment of development staff, but the optimal time is rarely the start of development work.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Least Way: Pivoting Away from Agile Excess

One of the most compelling justifications for Agile development methods and techniques is made through a comparison of the cost curves of both Agile and "traditional", or "non-Agile" methods.

When Agile was new and its case was pleaded more frequently, the following graphic was quite common:

NOTE: The above graphic is intended to represent to relative shape of the cost curves, and not convey exact scales for your organization. Your own organization's cost curve mileage may vary.

The Cost of Traditional Development

The essential proposition is that the cost of a unit of software development effort gets exponentially higher as a project progresses. The same curve represents the cost of change of existing features equally as well as the cost of adding new features.

The underlying principle is pretty straight-forward: the more stuff you have, the more stuff that will go wrong that you can't predict and that you don't detect. The more stuff going wrong, the more rework in-progress. Rework is the crumbly raw material that makes all foundations shaky - be they the foundations of software, or the foundations of the plan.

The number of modules in a software project increases as a project progresses. The number of inter-relationships between modules grows as the number of modules grows. The number of unpredictable repercussions of any change to existing modules or addition of new modules, as well as the extent that repercussions dissipate throughout a system of modules, increases as the number of inter-relationships increases.

When there are fewer modules, it's much more straight-forward to add new modules. However, at a critical turning point, this ease rapidly erodes and it becomes exponentially more difficult to get work done without undoing work that was done previously.

The more modules in a system, the more likely that any new additions or changes will cause adjacent modules to no longer work as they once had, and to no longer work correctly. The longer that they stay in this incorrect state, the more that other adjacent modules will be built incorrectly, or built in a way that reduces development productivity.

The more modules that are in a system, the less likely it is that its programmers can assess any detrimental effects brought about by changes. The system becomes an increasingly unsolvable puzzle that must be solved every time a change is made to add new code or to change existing code. Early on in the life of a system, the puzzle is dead-simple to solve. As the number of modules grow, the possibility of solving that puzzle with every new change closes in on improbable. This is the source of the extra effort that accretes around software development as time goes on.

How Agile Flattens the Traditional Costs

The Agile cost curve is fundamentally different. It was such a radical re-imagining of managing the factors that influence the cost curve that it was frequently dismissed out-of-hand, despite affirmations and reports from people who had given it an honest empirical and experiential trial.

Agile development's power rests in the drive to recognize the conditions that cause the cost of effort to increase over time. It uses techniques that keep effort leveled over the length of a software project, rather than allowing them to spiral toward the inconceivable, unpredictable, and unmanageable.

The cost of change is controllable only when you can predict whether any change will have detrimental consequences. You need to be able to apply countermeasures immediately when any of these detriments are created. The longer the wait to detect a design flaw, the faster the cost grows. This ultimately leads to the vicious circle of rework, where problems are found after the work is delivered to the next phase in the development process, while developers continue to build on that work, believing it to be a sound foundation.

In other words, Agile's radical re-shaping of the traditional cost curve comes from the ability for workers to easily and continuously check that their work isn't a cost magnet while that work is still open on their workbenches. They embrace a principled understanding of the kinds of software designs that are directly and easily tested, and a diligence paid to recognizing, avoiding, and remediating designs that aren't.

And of utmost and paramount concern - concern that can make or break an Agile effort: that organizations realize that its the rigidity of organizations and organizational systems and behaviors that obstruct software development from being done in a way that allows software workers and teams to rehabilitate the traditional cost curve. The impact to organizations - especially entitlements to silos and rigidity - can be profound. Organizations adopting Agile have a responsibility to shift from defensible, garrisoned blame cultures to proactive, optimized learning cultures. If the organizational and management system locks the steering wheel into one, largely immovable position, there should be little surprise when projects end up in the productivity ditch.

The Agile cost curve also suggests that there's a bit of a startup tax to pay up front, but that once the initial bump is navigated, productivity stabilizes and becomes much more predictable.

This is in contrast to traditional methods where the inherent absence of complications early-on lulls developers and managers into a false sense of success. They fail to practice productive countermeasures from the very start by failing to recognize the technical and organizational naivete inherent in the work. The success doesn't last. It's outlived by the realities of the exponentially-increasing number of complications that are natural side-effects of the progress of a software development effort.

The Agile Project Startup Tax

The graphic depicting the Agile cost curve overlaid on the traditional cost curve tells this whole story at a glance. This is what makes it so compelling. And yet, while the Agile cost curve is incredibly compelling, the most interesting point of the graphic isn't so much the benefits that Agile methods avail, but the point on the curve where the lines intersect:


Before that point, Agile development has a higher cost. Before that point, traditional methods should have a lower cost. The time spent using by-the-book, colloquial Agile development techniques before the point where traditional techniques become too costly is a likley volunteer mission for exacerbated costs.


The Best (or Least) of Both

A methodology that uses aspects of traditional development up to the point the costs of traditional methods are on the verge of spiraling out of control, and then makes a methodological "pivot" to techniques that can be considered "Agile" can reap the benefits of each approach's strengths.

Such a "blended" approach takes advantage of traditional methods that, while unsustainable, can deliver more bang for the buck up-front. The resulting path has the least amount of cost if the pivot is well-executed.


The resulting methodology is both traditional and Agile based on where work is in the project timeline and in the smaller development cycles. Neither its traditional aspects nor its Agile aspects are left unchanged by the imperatives of blending what is usually seen as incompatible methods into a continuum. But the most effective and salient principles of development productivity remain in-play.

My own practice of that continuum is informed by Lean Development, and more recently by the Lean Startup movement and its understanding of up-front phases that are often not mentioned by predominating methods like Scrum, and traditional methods that came before.

The Least Way

I was bemoaning failures with Agile and Agile community to Jason Cohen a couple of years ago. He offered the "The Least Way" as a name for the approach that I was embracing to deal with a number of the issues with Agile stasis and orthodoxy that I was trying to work around. I'll ultimately add more meat to those bones on LeastWay.org, and talk about how it underlies some tooling that I'm working on, and how it has affected work on projects. Maybe you'll find it as useful as I have if packaged up in a consumable form.

One of the key differences between an approach that doesn't just cherry pick techniques from different methodologies, but executes a full methodological pivot, is the explicit management of that switch. Planning for it, preparing for it, recognizing the right time to make the pivot, and making a clean entrance and exit from it is a bonafide part of the practice.

Missing the pivot is like missing the last exit before entering the Lake Pontchartrain Causeway. By the time you've realized that you've zoomed past your best chance to take the turn off, you've likely been captured by the gravity well that will start you toward the acceleration of costs. The cost curve begins it's rapid incline immediately after the pivot. The longer it takes to realize that you've missed the turnoff, the faster you'll be traveling, and the more you'll have to spend to put on the breaks and get back on-track toward managing the cost-per-effort.


Shaping the whole methodology around the pivot changes some of the conditions that drive staffing and work management at different phases of a whole software effort. It's not just "changing gears", but more like changing vehicles while they are still moving: you'd want to have vehicles designed for this specific purpose. Modifying the existing vehicles is a good place to start.

With the pivot being a full-fledged part of the methodology, with practices in place to recognize its approach and to execute it cleanly, the cost curve should be smoother, representing the best of what traditional methods have to offer - up to the point that they become unsustainable, and the best of what Agile has to offer - but not before it is sensible.


It's easy to see the methodologies as "one vs. the other" issues - "traditional vs. agile", for example. This perspective can even be helpful when more radical approaches are needed to shake up a rigid, entrenched, silo culture. However, plotting methodologies along a continuum helps us to see those parts of the spectrum where one blends into the other, and to learn to recognize and take advantage of the distinct transitional phases between them that we might otherwise miss or ignore.

Our tendency to look at some colors as "primary" and those in-between as "secondary" is largely an arbitrary side-effect of perceptual bias. The spectrum is one, fluid continuum of material that is neither primary nor secondary. When we choose to see transitional states as secondary, we naturally tend to tune them out. The powerful continuum of abilities that flow smoothly from each other is turned into a checkerboard of disconnected, shallow monotones.

Rapid feedback is the lifeblood of software development that doesn't spiral out of control. Agile protects vital feedback mechanisms from becoming calcified after a projects initial phases. Traditional methods provide rapid - but unsustainable - access to product and customer feedback early-on. Leveraging both and planning for and executing the transition as a fully-fledged part of the methodology rather than a secondary interest is a good part of navigating a "least way" of development obstructions (and costs).

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

User Stories Belong to Everyone

When Agile development is done well, user stories are always visible to everyone involved in turning ideas into working products. And they're visible all the time. When individuals or individual specializations involved presume a sense of "ownership" over user stories, doing Agile development well becomes difficult if not entirely impossible, save for a veneer of "Agile Theater".

As user stories flow through the whole development process from conception to delivery, they pass through the hands of a number of job functions and workers. Any time that people working in the vicinity of a given work step are given to believe that they "own" the user story in their current purview, they are likely to displace the user story from the most public and most visible medium that is common to the whole team. User stories are often then removed to the bowels of tools that are practical and accessible only to workers at the current work step and its immediate vicinity.

User stories can, do, and should change as they march forward through elaboration from concept to working product. Product development is a process of constantly unearthing a clearer understanding of the work we're doing while we're doing it. No amount of up-front analysis and design can stop this from happening. It's the nature of the kind of work that software development is, ie: product development.

If software development were manufacturing work, we'd know what would be happening in each step of the process before it happened. But then, we'd have to be creating exactly the same product again and again, which would be nothing short of an absurdity for software development.

The further we get into the process of turning an idea into concrete, material product, the greater the clarity and amount of thinking that gets invested into what we're doing. The more thinking invested, the more that we come to understand the subject matter we're working with, the circumstances of product's intended audience, and the fit of the decisions we've made earlier in the process to the reality waiting at the end of the process when real workers will try to do real work with the delivered product.

Analysis and design are present in absolutely every kind of work done in software development, and present in every stage and every work step. And while we may indeed have work steps early in software development processes that appear to be characterized by work exclusively in analysis and design, this is ultimately a consequence of not having concrete product yet in-hand at those early stages.

These early stages are less "Analysis and Design" stages as they are "Absence of Material Product" stages. We resort to seeing early stages as "Analysis and Design" due more to a habit of human perception that tends to draw attention to things that are present than to things that are absent. But later stages are no less "Analysis and Design" stages as a "Testing" stage in any less of a "Development" stage.

It's the same cognitive wrinkle that makes negative space visualizations like Rubin's Vase or the FedEx logo so compelling when we finally recognize the secondary images conveyed by the space that our cognitive facilities filter out. It's also what keeps us from being in a constant state of anxiety due to cognitive overload, and what allows us to be mindful of predators (not to mention to be unmindful of camouflage).

The kinds of work we do in software product development are rarely unique to any given stage. The kinds of work we do accumulates continuously so that later stages have responsibilities for nearly all of the kinds of work done in software development throughout the entire process. That's not to say that front-loading software development work with conceptual work is a bad idea. It's obviously an essential part of doing product development work successfully. But it's easy to mistake early-stage work as the domain of "analysis and design" due to the absence of material product this early in the game. The absence of material product at early stages should be a conspicuous absence, but the conspicuousness is typically lost to cognitive filters.

The disastrous side effect of failing to recognize this negative space perspective is the mistaken conclusion that analysis and design is the sole domain of early-stage work. And it starts the people involved in software work on the slippery slope toward believing that work on requirements analysis only happens early in the process, and that requirements - once defined - will not have to change. As if we could really know anything so conclusive so early in the process of uncovering clarity and understanding.

In the end, early stage work is naturally limited in that it can usually only be conceptual work. Conceptual work is realistically the only work that can be done so early. While the accumulation of responsibilities at work steps later in the process continues, analysis and design never wane. New clarity and new understanding never subsides. In fact, clarity and understanding only get sharper. This continues even once a software product has been made operational.

Because understanding continues to get sharper the further along we are, any concrete record of our understanding must be kept up-to-date to make sure that our collective clarity remains collective. Otherwise, workers at different stages of the whole process will have different understandings of what the goal is - as currently defined by all recent learning.

Making sure that everyone is on the same page is of paramount importance in any development effort. Locking down the definition of what's needed isn't the way to do this - although it's often the mistaken path that software organizations take when they fail to recognize the cumulative nature of understanding and clarity as product development unfolds.

To keep everyone on the same page, user stories must be accessible to everyone involved in a development effort, from early-stage conceptual work, all the way through delivery when placeholders like user stories are replaced with concrete, material product. They must be accessible with the least amount of effort by all people involved and at all stages and work steps.

At any point in the whole process, anyone who is empowered to add user stories, change existing ones, remove user stories, or otherwise make use of them should not have to go through a worker with specialized tool skills due to user stories having been removed to a tool that is not generally-accessible with an absolute minimum of effort and a maximum of immediacy.

Stated as a general principle: No single work group at any work step should remove user stories to a medium that, while more expedient to their work, causes user stories to become less accessible by others with different skills and responsibilities.

The inevitable result of workers at a work step feeling that user stories can be removed to a more locally-optimized medium is that separate copies of a user story will be used by workers at different work steps, and these copies will inevitably diverge. When multiple copies of the same user story diverge, workers at different work steps have different understandings of expectations and goals, and work at cross purposes.

In the worst of these situations, workers at different work steps aren't even aware of the others' divergent versions of the truth, and labor under the resulting conflict without any idea as to why expectations are consistently not being met. This drives up rework, reduces the credibility of the team, increases stress and conflict on the team, and generally leads back to the chaos that Agile development was supposed to address.

And yet, there are indeed advantages to specialized tools for specialized work steps. Great care should be taken though when considering specialized tools for artifacts that aren't specialized. It's often a mistake and an oversight when specialized tools are put in place at a work step that makes collective artifacts less accessible for the duration of the work step, or from that work step onward. In the case of user stories, it's hard to find a single artifact that is more collective, public property and thus less amenable to workstep-specialized tools.

At every stage of development, workers can back-slide into misconceptions of authority over artifacts that are responsible for unifying the whole effort and for stitching diverse workflows and workers together. The further we get into a software process, the more likely it is that workers at a later stage will remove user stories from more generally-accessible media and sequester them into the bowels of tools that are accessible only to people with specialized skills.

For example, consider the Cucumber framework and tools developed in the Ruby community, and the family of derivatives and clones that have since been created in a host of programming languages.

This style of tool encourages the displacement of user stories from more generally-accessible media into programmer-specific media. It doesn't do so accidentally. It's done as a recommended practice - reinforced by articles, books, and software conference presentations and workshops.

While Cucumber is built on some fairly compelling technology, and is itself an impressive work, it's not exactly methodologically-sound in that it fails to recognize its own contradiction of Agile development's core value of individuals and interactions over processes and tools.

As an aside, I brought up this issue with Aslak Hellesøy, the Principal of the Cucumber project, during a trip to Sweden last year. At the time, he informed me that work was being done on a tool to extract user stories from the specialized media to make them generally-accessible. But there are three non-trivial oversights inherent with this solution:

Firstly, it disregards the a-priory value of whole-team organizational methodologies like Agile by allowing collective and communal artifacts that are fundamental to higher-order productivity to become appropriated by one specialization.

Secondly, it seeks to address a problem that exists because of overly-elaborate tooling by creating even more elaborate tooling. The path back to simplicity and clarity - i.e.: "elegance" - likely also suggests removing the first wave of tooling rather than adding a second wave of compensating tooling. There are presently solutions that are arguably more holistically-effective as well as arguably more simple and clear. Although, they're based on less elaborate programming technology, and this is likely why the programming specialization remains distracted for alternatives.

Thirdly, it doesn't address the need to allow user stories to be added, removed, combined, divided, or changed by anyone with the authority to do so regardless of whether he or she is comfortable with tools, approaches, and perspectives that are more natural to the programmer specialization.

A Cucumber-style tool can be justified arguably as a general tool because it serves the whole of the process with testing and specification, but merely serving the whole of the process doesn't atone for the reduction in accessibility and immediacy of collective artifacts.

To Aslak's credit, he demonstrated a way of using Cucumber to me where user stories are not removed to its media. Nonetheless, removing user stories to Cucumber media continues to be a practice that is perpetuated in the user community remains a non-trivial methodological problem that is likely rooted in the narrowing of focus of specialized workers due to impressive tooling.

NOTE: There are more methodological oversights inherent in these tools, but that's a subject for a subsequent article.

Here a some principles to keep in the forefront of your Agile practice amidst the distraction of so many Agile-targeted tools:

1. Individuals and interactions over processes and tools has profound meaning. It doesn't suggest that you shouldn't use tools or processes, but that these often become distractions from more powerful means of ensuring success. Consider tooling choices very carefully - especially specialized tooling. Understand how it might inadvertently narrow the perspectives and values of its users.

2. User stories are a whole team artifact. Any tool that moves user stories to a medium that the whole team doesn't have at their finger tips should only be used when there are no viable alternatives. Forcing some team members to go through other team members to work with collective artifacts is categorically not what is intended by individuals and interactions over processes and tools - even if it does cause people to work together to gain access to resources. These resources are supposed to be generally and immediately accessible to begin with.

3. If you choose to remove collective artifacts to workstep-specific tools, put protocols in-place to ensure discrete hand-offs between tools so that multiple versions of the truth are not available to different workers who work in different work steps. When a user story moves from one system to another, destroy the previous version and leave behind a placeholder - a kind of forwarding address for a user story - that tells the interested party how to find the user story's new home.

4. Don't fool yourself into believing that you'll be able to keep multiple versions of user stories in-sync through the life of a project. This kind of work is too costly to keep up and will be quickly abandoned by the project team, leading straight back to the multiple versions of the truth problems. Adding customized electronic synchronization automation to this problem is also not a good answer. This just adds more elaborateness to a situation that is possibly already too inelegant. It's a problem that likely shouldn't exist in the first place.

The inescapable truth of user story management is that the least-elaborate technology is often the most productive solution - even if the technology is no more elaborate than a ten-foot by five-foot cork board with index cards pinned to it. In the very least, this keeps user stories immediately visible and accessible, and makes changing them no more difficult than putting pen to paper.

Of course, the tried and true cork board and index card solution isn't a good foot for all teams. That said, as you seek more elaborate solutions, take tiny steps forward in settling on your solution. A leaping at the most elaborate solution for user story management usually ends up planting a whole team in a tool that serves the specialized perspective of the people doing the tool selection - often to the disservice of everyone else involved in the whole effort.

It's vital that we're not lured into local optima interpretations like, "individuals and interactions over tools and processes except when the tools are intellectually stimulating". All tools are intellectually stimulating. In many cases, tools can be stimulating to distraction. The problem is what we're being distracted from: the essential root cause of success in software product development. Namely, the magnification of understanding and clarity that comes from interactions between individuals over the course of time. The deeper problem is that we believe ourselves to be impervious to this kind of distraction because we have such positive feelings about our tool choices.

The immediacy and accessibility of user stories is a foundational corner stone that your Agile house is built on. Whether it stands strong and continues to be built upon, or whether it crumbles under its own weight can be decided entirely by this salient factor.

Do everything you can to make sure that user stories continue to belong to everyone, in every minute of every day. And be vigilant against any backsliding into specialized expediencies that remove user stories to any medium that is less immediate or less accessible than the simplest and clearest of tools at our disposal. An elegant solution is often not an elaborate one, but greater whole-team productivity is always more stimulating than elaborate, specialized tools.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

User Stories are Temporary

It's obvious, but warrants mention: What we do in the future is likely to be different from what we're doing today. The implications for user stories should be obvious: User stories are temporary. Saving them for posterity doesn't serve the primary purpose of user stories, and doing anything that makes them less temporary can turn user stories from benefit to detriment.

User stories are a bit of semi-refined raw material that systems are created from. They're a way to represent users' needs. Most importantly, they communicate an understanding of users' context to the people who are building the software. User stories help to make it more difficult for software development teams to succumb to the ever-present threat of losing track of the user's bigger picture.

Aside from raw human ideas that user stories represent, they are the most flexible artifact in software development. Then can be created at any time. They can be changed at will. They can be combined. They can be subdivided. They're not intended to become fixed, and anything that serves to make them fixed is a disservice to the value they offer. User stories aren't fixed in a point in time, just as needs aren't fixed. They change because conditions change. They change because our own understandings change and improve as we gain the clarity that comes from working with user stories, and from building the products that satisfy needs.

When we realize new clarity, our responsibility is to reflect that new clarity in the form of new stories, or new subdivisions of existing user stories, or through the combination of existing user stories, and even from the removal of user stories from the product definition entirely.

The user stories that describe our needs tomorrow are going to be different from today's. Knowing this as we do, we have we gone from recognizing the transient, temporary nature of user stories to the point where we now tend to capture them in tools that make them harder and to change, and harder to take in at a glance and to glean new clarity from. When we do this, we're increasingly unlikely to update user stories to reflect the improved understand we cultivate as we work with them, with users, and with the software that we're developing.

The more that we calcify user stories in ever more concrete media, the less we make necessary course corrections along the way. There is little that so obviously contradicts "Agile" more than inviting anything that adds to the calcification of requirements. A principle duty of user story management in Agile development is to protect them from becoming encumbered by the notions of permanence that obstructed software development methodologies that predate the effectiveness of Agile methods.

We pat ourselves on the back for having evolved beyond the quintessentially-primitive "300-page requirements document", but often all we're doing is bringing some of the same old problems forward to our "new" requirements formats. Sure, we've outgrown the massive, big-batch requirements doc, but the genetic material of the disease that infected those documents have evolved into a newer problem for newer artifacts, and ultimately, the problem is still the same essential problem. We're so pleased with ourselves for our new Agile ways that we fail to see that we're repeating the same mistakes, except with new material.

Here are some things to keep in mind that can help with the relapse into the same old mistakes:

1. When you forget why you're doing something the way you do, the answer isn't in a user story archive. The answer is in revisiting what you're doing to make sure that it still holds true. Do this with interactions and with people. Use tools and processes to support the human interactions.

2. User stories are not artifacts for regulatory compliance. User stories are a scratch pad of human understanding at the present moment. They are chuck for the software development grinder. What comes out the other end should be the software that itself explains why things are as they are. If you need a permanent record for the purposes of regulatory compliance, go ahead and create one. Nothing in Agile stops you from doing that. Just don't hang a stasis anchor around the neck of user stories. If you do, you'll strip them of the benefit that they bring to software delivery.

3. Have one master copy of a user story while that user story is in-progress. The master copy should be the one that is most immediately accessible to the most people who are involved in turning ideas into software products. They should be be posted using the most visible medium possible - one that requires the least amount of human interaction to casually-scan a project's gallery of user stories.

4. Throw user stories away once you deliver the working software that addresses the needs expressed by user stories. The software and its supports are enough to describe why it is the way it is. If you no longer remember, and if the software and its supports and the mechanisms and practices of institutional memory aren't sufficiently self-descriptive, revisit all of these issues as an opportunity to improve the production system in a way that doesn't invite more tool-centric bureaucracy.

Here's a bit of a side note to consider: The principle sponsors of the Agile 2010 conference are both "Agile" tool vendors.

If one of Agile development's primary tenants is to value people and interactions over processes and tools, how is it that the only Agile organizations producing enough disposable income to support Agile development itself are tool and process companies specializing in Agile requirements management and requirements compliance? Somehow, we've walked in a circle - yet again. And one of the things we've left on the trail behind us is the understanding of how things went wrong last time, as well as our commitment to avoid expediencies that don't serve sustainability.

The process methodologies that predate Agile are often roundly dismissed by agilists. But there wasn't really inherently wrong with them. Things back then started getting silly when tools started getting silly; when we began to defer to tools as authorities rather than to the people needing and the people solving needs.

One of the first definitive steps into the downward spiral of mindless process bureaucracy hell is the calcification of requirements. And no matter how temping it is to succumb to the materialism of hardened collections of user stories, we don't get to hell without passing through those gates.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Workplace Safety for Software Developers

Software development managers make sure that their workers' physical safety is secured - assuring that office ergonomics are up-to-spec, and in compliance with the prevailing laws and regulations. On the other hand, managers verily require that software development workers undertake some of the riskiest behavior in the contemporary workplace. It's entirely common that software managers require workers to undertake work the crosses the line of being merely questionable, and moves squarely into the territory of negligence. It's not because software development managers are nefarious, but because they are too-often far enough removed from the work of software development to differentiate negligent and hazardous decisions from safe and sound decisions.

Without a tangible understanding of the work undertaken by software workers on a day-to-day basis, and an understanding of that work in great detail, software managers are not equipped to know when they are requiring software workers to supplant safety with expediency.

Workplace safety is an issue of productivity. Lack of safety breeds low productivity. In the case of software development, "safety" is an issue of undertaking work in a fashion that doesn't expose the software worker to unnecessary risk - especially risk that is easily avoided. That is, it's easily avoided for someone who understands the work of software development at a high-enough level of expertise and detail to guide workers away from unsafe practices.


An unsafe practice in software development is any work done in the present moment that makes work in the next moment take longer to complete. A reasonable analogy for this is the notion in physical materials work of a clean workplace.

The purpose of a clean workplace is to remove the clutter that hides unseen, unsafe conditions. It's a lot easier to trip over an errant piece of material in the workplace when its surrounded by a profusion of other errant materials. When hazards are so common that they blend into the background, we simply stop noticing them, and this is how we come to be put at greater risk.

The more unsafe clutter we have in our workplace, the more we have to work around. The more we have to work around, the more time it takes us to do work.

If a manager can't detect the conditions that makes work take longer to accomplish than expected, he doesn't adjust his expectations for cycle time accordingly. This means that he expects more to get done - the original work, plus the workaround work - in an amount of time that would be reasonable for the completion of the original work alone.

This makes workers even more careless, exacerbating the accumulation of obstructions and hazards in the software development workplace as a result. It's a vicious cycle that usually only happens because someone with the authority to avoid problems in the first place doesn't have the experience to recognize the problems and to deal with them.

In the words of the ancient Chinese workflow master, Lao Tzu: "Confront the difficult while it is still easy. Accomplish the great task by a series of small acts."

Although, you can only do those small tasks if you can detect their presence. The finer your ability to perceive counter-productive deviations, the quicker you can respond to them. The longer you wait, the further off-course you'll be when you finally realize that you need to make a course correction. Someone who doesn't do work can't recognize when that work is off-course. Someone who isn't an expert in the work can recognize when the work is off-course, but his will only recognize it after it has become more expensive to deal with than necessary.

And this, as the average software worker would tell you, is the day-to-day conditions in which their work is done. Software work is a constant battle fought against the accumulation of hazardous clutter. Software workers rarely get ahead of the clutter curve, and have to invest significant effort to keep their workplace free of hazards. Paraphrasing Taiichi Ohno, if you're not moving forward, you're falling behind.

Each individual scrap of hazardous material in software development is typically quite small - small enough in fact to be easily deemed negligible - even by people who do the work. The hazard presented by two iota of hazardous waste in software isn't the sum of the hazards - it's the sum of the hazards compounded by some multiplier. Hazards in software don't exist in isolation, they interact with each other creating a higher order of hazards that is greater than the sum of their parts. Two pieces of hazardous software that interact with each other don't create two hazards, they create two hazards compounded by the amount of interactions between these modules. To make matters worse, all modules that interact with hazardous modules in close adjacency also get infected. Lack of day-to-day expertise in software work often leads to negligent underestimation of the risks associated with the "design hernias".

Software hazards compound very quickly. It's very easy to arrive at an explosive accumulation of hazardous software materials. It can happen within the first few days of a software project. If subtle hazards in the tools and frameworks that programmers are slated to use are not recognized immediately, the accumulation begins before the first line of code is even written.


When we ask software workers to continue to work in such conditions, we might as well send them into a mine filled with coal dust and require them to hyperventilate for several hours each day. Sooner or later they are going to have to escape the job to escape the hazards, or they are going to acquiesce to the irreconcilable differences between a manager's expectations and the realities of the working conditions that the manager himself feels entitled to not be exposed to for having already "paid his dues". This creates dispassionate cynicism in software developers. They come to learn that no matter what they do until they escape the work, they are more than likely going to end up on the losing end of the software work proposition. They learn that they'll get hung with the detritus of negligent software development decisions and direction because decision makers are too far removed from the work to see that it is these critical decisions and directions (or the lack thereof) that are the root cause.

In these working conditions, workers constantly face obstructions that are only in-place because they weren't recognized when they were mere seeds of problems rather than towering oaks of looming counter-productivity.

The accretions of hazards into founts of the depressed productivity that is status quo for most software teams is fully and completely avoidable - if only the hazards are dealt with when they are small. To see looming hazards when they're small, you have to have detailed knowledge of the work in the here and now. This is simply not possible when managers have removed themselves from the work.

It's far too common for software development managers to feel entitled to be removed from the work of software development - as if removal from the work is a reward for having done the work for a number of years. The reward for doing software development work for a number of years is not an escape from the work, but an immersion into a far deeper understanding of it so that its expert insight can guide software work away from hazards. Of all the escapes that can be orchestrated by software workers, an escape into management is indeed and in fact an act of pure negligence.

Software work is works in intangibles. Software hazards go unnoticed because they are physically invisible. Taking a bit of liberty with Donald Reinertsen in The Principles of Product Development Flow, we don't see the manifestation of software development safety issues "when we walk through the engineering department." They're "bits on a hard drive, and we have very big hard drives in product development."

Managers often try to compensate for their lack of detailed understanding through the use of summary representations like software diagrams. But software diagrams don't show the looming problems while they are still small; still manageable gathering storms that can be dissipated through judicious application of minimally-invasive countermeasures. Only flagrant hazards can be detected in summary representations. Small, detailed course corrections can't be plotted from coarse, summary information. And yet managers who feel entitled to be removed from the work perpetuate the fantasy that summary representations like diagrams are sufficient to bring their purview into action on the software projects they manage. This is pure folly, and software tool vendors are quite happy to continue to exploit it.

By the time you've detected a software hazard that can be seen in a summary representation like a software diagram, you're looking at a problem that has festered for far too long. You should be able to detect the chemical markers of the disease long before you notice that lump in a vital organ.

Programming work is almost entirely mental. Its effectiveness is influenced by psychology, cognition, awareness, and communication far more than by any material concern like ergonomics, an ultra-fast workstation, or multiple monitors. Software systems are far too large and far too complex to be held in entirety in the conscious focus of any single software worker in any single moment. The devil is always in the details. Summary representations are secondary to the actual raw materials of software: the code. You have to be in the code, and to know enough about code to understand which subtle design differences are looming problems and which aren't.


The average software system is a dark coal mine filled with the poison particulate of tomorrow's case of black lung. We ask developers to work in these conditions every day. We do this because at some point in our careers as managers, we believed that we were entitled to be removed from the details of software work.

Sooner or later, enough hazardous software material accretes in a software system that managers step in, and often the first thing they do is look for a root cause in the workmanship of the software workers. The workmanship is rarely the root cause. The absence of informed, skilled, and insightful software management and guidance is always a preeminent cause to unattended workmanship. The workmanship is, of course, a problem, but it's a side-effect.

This is like blaming miners for an underground explosion due to the accumulation of day-to-day hazards that result from institutionalized negligence of clear and present dangers that should be managed the moment that they show their first signs. We ask workers to undertake work that they know is not in their best interest. We ask them to do this from our organizational perches far about the hazardous conditions of software projects. And then we hang them with the inevitable costs of this kind of mismanagement.

It's the software worker that is required to spend extra time in the mine to balance productivity that is lost to management negligence. It's the software worker who is required to take on the duties on sanitizing data by hand without the safety of the proven automation that should have been built right to begin with. We ask them to validate the decisions that we've made about the tools that they'll use in their work based on little more than a compelling sales pitch from a tool vendor who claims to knows more about the day-to-day work of the software developers in our organizations than we do - vendors who are even further removed from the work than we are.

We ask software developers to work against their own interests and the interest of our organizations because we don't understand the hazards that our decisions create.

Software workers suffer disruptions to their lives and their livelihoods when this kind of institutionalized negligence drives work policy on software projects. They shoulder the accumulated detritus of uninformed and unskilled management decisions. And they suffer the humiliation of blame when lack of management insight can't see it's own reflection in the root cause mirror. And institutions lose the invaluable institutional knowledge when workers escape the organization altogether. It's always a no-win situation. Everyone loses. And it's completely avoidable.

If you're not willing to be in the day-to-day work of software development, you're declaring loud and clear that you're not qualifying yourself for the authority to make decisions that direct software work. Every decision you make risks adding to the accumulation of software hazards. When you do this, you deplete the workplace safety for software workers. You ask them to take risks that the workers themselves know that you can't see, and they know that because you can't see them, that you will likely not understand that the accumulation of avoidable hazards into full-fledged, clear and present dangers is your fault to begin with.

Workplace safety is a serious productivity issue. It's a serious issue for the health of workers. It's a basic expression of human respect for the people who work for software development organizations. Understanding workplace safety for software developers requires a high level of expertise in software development, and it requires day-to-day currency in software development. A manager who is not willing to have insight into the details of the work that he is responsible for is patently disrespectful of his workers. He constantly puts them into harm's way by presuming to express authority without knowing whether his expectations are hazardous to the health, well-being, and viability of software work and software workers. Beyond disrespectful, it's dishonorable.

The software field isn't at risk of programmers forming a united front and leveraging collective bargaining, and frankly such a thing isn't likely what anyone wants. But dealing with the root cause issues that have driven other industries to such actions is a win for everyone when obstructions to productivity are removed in the process. Addressing software development workplace safety is a good place to start - especially in our current economic conditions, where a massive boost of unimaginable untapped productivity would be a welcome change indeed.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Unfortunate Case of Agile Coaching and Servant Leadership

The need for coaches, and especially servant leaders, is a side-effect of an unfortunate organizational pathology.

Consider this quote from James P. Womack: "We've got too little management and therefore need too much leadership".



Someone who has become specialized as an Agile coach necessarily benefits from the perpetuation of Agile coaching. This doesn't mean that Agile coaches are nefarious and interested in suboptimal outcomes, but perpetuated specialization typically creates narrowed perspectives and thus overly-narrow interpretations of conditions and needs.

The result is often the very same counter-productivity that we observe when someone is given the exclusive job of capturing requirements for a software project: they fill all of their time exercising their specialization, creating an inventory of work that is out of sync with the progress of the team.

When we bring Agile coaches into software development efforts, we should recognize the underlying problem that is being signaled: that software development managers are not presently equipped to manage the software development work that they've been tasked to manage.

By failing to recognize the underlying problem, we often go too lightly into engagements with Agile coaches, and we end up failing to use them to solve the root cause problem, and this perpetuates the need for longer engagements with Agile coaches. This, in turn, drives Agile coaches further into the reaches of specialization, and narrower perspectives, accelerating the erosion of their usefulness.

To break this cycle, we need to recognize the most important reason to bring on an Agile coach: The purpose of an Agile coach is to transform managers and the management system so that managers, in turn, can transform their organizations.

Coaches initially take on responsibilities that ultimately need to be the responsibilities of a manager. Even if the outcome is an improvement, when a manager's job is split between an authority position (the Boss), and a guidance position (the Coach), the outcome isn't as good as it can and should be. Nonetheless, if a manager is in a position to need a coach, then the coach is inevitably going to take on some of the manager's responsibilities for teaching and transformation while the manager undergoes rehabilitation of his own.

The coach and the manager must recognize that these conditions are a necessary evil. The goal is a transition away from the split direction of a team between a manager and a coach. The longer that this split is in place, the higher the likelihood that the new behaviors learned by the team will be undone by the manager when the coach moves on.

Agile coaches can find themselves as removed from the work of software development as the managers that they are called to rehabilitate in the work. For an Agile software development coach to remain effective, he has to be an active software developer. Just as managers fall into the trap of losing software development currency, so do Agile coaches. An Agile coach who doesn't also do software product development for a living is a ticking timebomb of counter-productive guidance for a software development team.

Consider this quote from Daniel Markham: "If you're going to train something, you should be able to do it. And I mean do it to a very high level of expertise. An agile coach should be able to code, perform analysis, manage the project, test -- anything that needs doing on a project. If she can't, then how can you talk to her about your particular situation? If your agile trainer was a BA last week, or never slung code in his life, or is a professional trainer, or -- let's be brutally honest -- is making less than the members of the team are, you've got a dud. It seems like common sense but it bears repeating: you can't train something you haven't done. And "done" means a bunch of times, not just on the pilot project."

The qualities and abilities that Daniel outlines are qualities and abilities that we need to expect from software development managers. If they don't have those qualities and abilities, they need to get them, and a coach might be a way to get there. If the coach doesn't have those qualities and abilities, most of what's produced during the engagement doesn't become lasting, transformative value.

Agile coaches who themselves don't know the work of software development can't teach managers to know the work sufficiently to manage it. They can only teach them to become other disconnected Agile coaches. This is how the second half of the first decade of Agile came to be dominated by the doctrine of Servant Leadership. As more senior leaders in the Agile community spend more and more time in roles that are ever distant from the work of software development, Agile itself is becoming more robotic, more mindless, more of an orthodoxy, and even more of a liability than a benefit.

Servant leadership is one approach - one tool - fit for the organizational conditions that it serves best. An Agile coach often embraces the edicts of servant leadership as a universal panacea when his duty to stay sharp in the full curriculum of software development is dulled by the ease of Agile coaching specialization. But any tool used indiscriminately is going to undo any benefits that it provided in the circumstances where it was an appropriate choice.

A servant leader's typical proposition is that an organization's ability to meet expectations will be transformed if only its inherent creativity can be tapped and unleashed. While tapping and unleashing unused creativity is transformative, it's extremely rare that doing so is either sufficient or sustainable. It often creates conditions that not only lead to unmethodical exploration of ideas, but also makes teams feel that they have a right to this kind of behavior. In other words: "self-determination" rather than "self-organization".

Servant Leadership as a panacea is also a real side-effect of past experiences with bully management. It's an over-compensation reaction to some very real and very tangible painful experiences of abusive directed authority. It's an equal and opposite reaction to an extreme. It, as well, is an extreme. It's as much an undesirable specialization of behavior as abusive directed authority. It's often a programmed predisposition, rather than a conscious choice informed by circumstance.

The Servant Leadership doctrine often patently dismisses the greater goal of balance between directed authority and creative enablement, almost entirely eschewing the value of informed directed authority and skilled directed authority in favor of an over-indulged creative enablement.

A job in creative enablement is a lot of fun, and many Agile coaches have come to feel entitled to these kinds of engagements. They're light, they're fluffy, they can come with no measurable expectations, and they're often damned lucrative - especially in software organizations in the thick of the detritus of an unrecognized management crisis.

The increasing prevalence of Servant Leadership also presents a great opportunity for consultants with little real experience in Agile to re-brand themselves as Agile consultants. It allows for anyone with even a modicum of summary knowledge of Agile development to have a role in the Agile bonanza. But we all lose when this happens.

If you don't already know what your problem is, then pre-supposing an Agile(tm) solution further deepens the mindlessness of robotic management, and it diminishes the Agile body of knowledge. And ultimately, it exposes our organizations to the risks of bringing in help that may only have the ability to talk the talk - which is often all that is required of Servant Leadership - and often all that servant leaders assert is required of them.

Again from Daniel Markham: "One guy (famous author again) basically put it like this to me when I told him the team wasn't succeeding: I'm here to demonstrate certain practices and to show that they work, not to just stop everything and attend to what the team is dealing with today."

To move forward through this crisis, we need to set new goals for software development management: To be close enough to the work to be able to manage work, and to use directed authority in service of helping the team meet expectations when creative enablement turns "self-organization" into entitlements for excessive experimentation.

And we need to re-define the role of coaches from servant leaders to temporary co-managers, helping the team to meet expectations while bully management and its detritus is transformed into effective software development management.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Manager or Bully?

Management in the large: managers set expectations for their workers. Of course there's more to it, but "setting expectations" covers a lot of ground.

Management is central to the productivity crisis we face in software development. It's a crisis that pervades software development so profoundly that we usually don't recognize it. It's like water to a fish - not something that our attention is drawn to, not something that we notice. It's the environment that encloses everything we do. It's not that we see it as normal, we just fail to see it at all. Frustration, burnouts, blown budgets, missed expectations, and dissatisfied customers; it's just the way that most software development is done.

We start with the best of intentions, and despite our intentions, we usually end up in the weeds. We end up in the weeds most of the time because that is where software development teams are directed. It's because of one salient, painful truth about software development:

Software development managers don't understand the work of software development sufficiently to be able to direct the work of software development, and the value of hands-on work experience they've had in the past decays so rapidly that it's increasingly unsafe to make new decisions based on it.

Said more plainly: at a time when senior staff should be best informed by their lengthy and varied experience, they remove themselves from the work of software development long enough to lose the understanding of how day-to-day software development work is done. They lose the knowledge needed to direct the work of day-to-day software development. A certain egoic mania takes root that leads senior staff to feel entitled to believe that their insight has moved beyond the necessity of mere practical work experience.

How long does it take to become unqualified to set expectations for software development work? Not long. It happens as quickly as six months and often not longer than two years. This is why the proposition of Self-Directed Work Teams that has been popularized by new, opportunistic interpretation of Agile Methods has become so attractive in software development - even though SDWT is a well-known failure mode in most development industries and is known to be sustainable for no longer than a couple of years in the majority of cases (there are exceptions, but exceptional organizations usually aren't the rule for unexceptional organizations, despite our best hopes and ideals).

Plotting software development management on a continuum between the extremes of Managers and Bullies, we don't find many managers who are only either extreme. A software development authority is rarely either only a manager or only a bully, but software development managers are usually weighted more toward the bully end of the spectrum. It's an innevitable consequence of having lost interest in the details of software development work, and having become quickly dated as a consequence.

Someone who sets expectations but who isn't able to show workers how to fulfill expectations isn't managing, he's using the implicit threat of crossing an authority to make demands without knowing if those demands can be met. And he's hoping for the best. But we need more than hope to manage software development. We need to know that the work being done every day is the work that leads to the desired outcomes.

Setting expectations without knowing how to meet those expectations almost always leads to unqualified or unrealistic expectations. Using authority to require unrealistic expectations to be met always creates a compounding accumulation of obstructions to subsequent work. It's a sure-fire path to lower productivity. Most software development organizations live perpetually under this cloud of depressed productivity.

A bully sets expectations for workers but doesn't have the relevant and necessary capacities to help those workers fulfill the expectations. A bully has been disconnected from work for long enough that he (or she) is no longer able to personally act to ensure that his workers meet his expectations - to show them how the work is done, and to qualify the work that they do.

In other words, the person who needs to be the most experienced and most expert software development worker on the team is often the least capable.

A direct manager knows how the work should be done. He may know this better than his workers. When necessary, a manager can sit down with the worker's current tools, and teach the worker how to meet expectations, and to teach the worker how the specifics and tactics are tied to a greater whole. A bully hides behind assertions that his time and intellect are too valuable to be spent on such trivialities as the "how" of doing the work in his purview.

It's fair for a person with authority to no longer have the capacity to ensure that workers are materially able to meet expectations, but then that authority must defer to someone who does have the necessary and relevant capacity. And this is an entirely undesirable circumstance. It might be the circumstance that you're dealing with, but it's not the one you should settle for.

An authority who isn't capable of knowing whether and how work can be accomplished and whether expectations can be met is qualified to set only summary expectations. An authority who over-steps the material bounds of his capacity will continue to inject productivity obstructions into his organization's workflow by causing workers to make unsafe and counterproductive decisions that innevitably lead to rework, overwork, and relearning.

A bully believes that his workers are soft and lazy, and often puts extra pressure on workers believing that this is what it takes to get merely nominal productivity. A manager recognizes that workers who retreat from work are likely facing some distressing obstructions in the workflow. He uses his skills as a teacher and his insights into the work as an expert worker to help his workers to recognize and clear obstructions.

It's often hard for us to accept the cold reality of the depressed productivity in our organizations. We fail to manage software development as a product development discipline and tend to manage it as a manufacturing discipline (usually without even understanding or recognizing the difference), undermining productivity at every turn by tuning the wrong variables and turning the wrong organizational and procedural knobs.

In an environment where management leans more to the bully side of the continuum, we see work teams stretch the intentions of "self-organization" into the realm of "self-determination" or "self-direction". It's not necessarily an unwarranted reaction, but its a reaction to a root cause that should be the focus of organizational and procedural improvement. It's in these environments that we also see a need for coaches to fill in the management void, but this is also a lesser palliative that doesn't necessarily get to the heart of the matter.

A manager is someone who balances qualified and expert worksmanship with the ability to teach and is comfortable with using directed authority when necessary. He goes with his team into new learning and new understanding, and re-directs the team if the creative drive for experimentation starts to occlude the imperatives of delivering material product.

Doubtlessly, it's sensationalistic to divvy up the software development management landscape into extremes, but doing so is meant to frame the issue so that we can begin to recognize a real problem and begin to do something about it.

The way that software development is practiced at large, it's perfectly understandable for software development workers to become burned out on software development, and to look for a change. Unfortunately, moving upward to software development management isn't where people who are burned out on this work should go. It's little more than an escape, and it needs to be more of a deeper commitment to the work rather than a means to justify avoiding the details.

Ultimately, software development work will remain palatable for much longer when it is managed much better. But we are where we are, and the general population of software development managers will remain skewed heavily toward refugees from the work of software development for some time.

This is the software management crisis: software development managers at large aren't experts in software development work now, and they often weren't expert workers in the days when they were hands-on. They were the people who came to dislike the work enough to want to escape to software development management. They usually don't know enough about the details of the work to understand how subtle decisions can result in tremendous outcomes, and which decisions lead to tremendously good outcomes, or tremendously poor outcomes.

Today's software development managers largely haven't made a study of managing software development and the principles of product development work and work management. Software continues to be managed as one would manage material assembly or manufacturing. Our current generation of managers isn't aware of this problem because software development management hasn't really been a field of interest for most of them - it's just the job that they have, and it's one that keeps them from having to work in the painful niggling details of poorly-managed software.

None of this suggests that we need to wipe the halls of software development management clean and start from scratch with newly-installed personnel lifted from the rank-and-file. One of the worst mistakes that our current management ideology continues to make is the failure to recognize the material cost of losing institutional knowledge when we lose any worker (NOTE: the flip side of this problem is the colloquial horror of not knowing who to hire, and bringing people into our organizations that we dismiss in short order).

The software development mangers that we have today may have become burned out on the details of software development, but those details can be changed to make the work less tiresome, and make the work less like an uphill battle being fought on a cliff face rather than a gentle slope. Their institutional knowledge is far too valuable to lose. Their many years of experience - if paired with a sharper understanding of which decisions make software development good and which make it bad - provides a perspective of the whole of the effort that is invaluable.

It's entirely reasonable to expect that we can transform the software development industry. The root of the problem is in the software development management system. When software development managers are willing to look without flinching into the reality of their situation and the circumstances that have led to their positions, then we can begin to rehabilitate the system. When that happens, many of the software development industry's most futile pathologies will fall by the wayside and we'll begin to tap the potential that is this industry's unclaimed birthright.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Video: Ruby for .NET Developers (From the Norwegian Developers Conference)

My presentation at the Norwegian Developers Conference, "Ruby for .NET Developers" is available for viewing on Vimeo.

There's a bit of off-color humor in the talk for the sake of making a point about orthodoxy, and done specifically-so for the European audience that it was presented to. So, no offense intended! :)

View it here or open it on Vimeo's site for more viewing options.



Here's the description of the presentation:

After having spent many years coding in C#, and after having spent equally as much time in the C# language culture, Ruby seemed like a lot of bad ideas and heresy. In fact, much of Ruby is heretical to a C# or VB.NET mono–culture, but the productivity gains demonstrated by Ruby on Rails teams remains an unavoidable elephant in the room. This presentation looks at C# code examples side by side with some equivalent Ruby code and shines a little light on what it means to have either "ceremony" and "essence". It challenges the claims of static typing's effect on tooling to deliver "developer productivity". And finally, some examples of Ruby meta programming are given to demonstrate direct solutions to programming problems that would require much ado with restrictions in C# that don't end up doing much more than reducing the efficiency of software development efforts.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Praise for the Norwegian Developers Conference - A Conference with Heart

Leaving the Norwegian Developers Conference after the last session of the last day, conference participants were bid a fond farewell at the door by Kjersti Sandberg.

I suppose it's not such a big deal to have people saying goodbye to participants as they leave, but in the case of the NDC, the person doing the goodbyes was the conference organizer herself.

There is a lot to say about the quality and value of a conference like the NDC, but if you trace the outward expressions of quality back to the source of the quality, you inevitably find yourself staring square in the face of leadership. It's Kjersti's leadership that makes the NDC an excellent conference - and in my case, one of the best and most rewarding conference experiences that I've ever had.

When you build something that you truly believe in, and when you have intentions to do something great and hold yourself to extremely high standards, that sense of ownership is palpable in every aspect of the work and of the product. The Norwegian Developers Conference in Oslo last week was an obvious expression of leadership, vision, and commitment.

I get the sense that being at the door, in person, to bid farewell to conference participants wasn't just the execution of good customer experience technique and strategy, but was a moment for Kjersti to enjoy the fulfillment of hard work and excellent execution. I can't remember a conference where the conference director was so deeply invested in her customers' experience that she was there in person to wish them well and to express her appreciation for their participation.

Anyone can fake these kinds of expressions of customer service, but it takes heart to do it and to make it clearly sincere and meaningful.

Kjersti wasn't at the door to wish NDC participants a good journey back home and through the next year because it's just smart, personable technique. She was there because it meant as much to her to enjoy the moment of personal and personable accomplishment as the experience of the NDC meant to the people who came to NDC. And it was definitely a moment of absolute pleasure for me to watch something quite rare: a true appreciation for customer experience reflecting a true investment in product and customer development.

Kjersti wasn't at the door with a big sign over her head letting people know that she is the person behind the NDC. She was there anonymously. She wasn't there for the show and the display, she was there because it means something for her to be there, and to say farewell in person. It wasn't about a demonstration of investment in customer, it was simply investment in truth incarnate. For me, it was probably the most endearing moment of a conference that overflowed with endearment.

There's much more to say about the people involved in making the Norwegian Developers Conference happen. I'd just like to say thanks to everyone at Kjersti's company, Program Utvikling, and to Anders Noras, Rune Grothaug, Henriette Holmen, Jakob Bradford, Thale Sandberg, and a host of people who had a role to play in making the NDC a memorable and rewarding human experience.

The most important thing about creating a good conference is crafting a space for ideas and perspectives to come together - especially conflicting ideas and perspectives - to learn what new ideas can come from the experience, and to form new, higher order ideas that resolve what we had previously assumed were ideas of irreconcilable difference. It takes courage and integrity to create a space to allow this to happen. The space itself is just a blank slate, it becomes a space of courage and integrity when it reflects these qualities that are already qualities of its leaders.

Two salient takeaways for myself from my experience at conferences in Scandinavia:

Firstly, Scandinavian conferences, like NDC, Oredev, and others, don't seem to allow the kind of mindless orthodoxy that is eroding integrity in many conferences in the United States. They encourage encounters of diversity that, while tense, inevitably lead to the greatest ideas of the future, and point the way to the next steps.

And secondly, the United States conference market is in desperate need of a practitioner-level conference like the kind I've experienced in Scandinavia - especially in the Microsoft community, where diversity is increasingly avoided and, in some cases, actively suppressed. My hope beyond hope is that we'll see one of these conferences on our shores soon, and that their influence will help to get the community focused on goals that are more important to society as a whole than they are to the imperatives of a particular player's interests.

---

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com