Friday, July 03, 2009

The Problem with Big Design Up Front is the "Big" not the "Up Front"

The risks of Big Design Up Front isn't the "Up Front" part, it's the "Big" part. Doing too much design without validating it inevitably drives a good bit of the productivity loss that continues to hamper software projects.

It's an issue of large batch sizes - the "Big" in "Big Design Up Front". The larger batch sizes mean that we're always building today's software on yesterday's work and yesterday's decisions before proving that yesterday's work and decisions are sound. The larger the batch size, the larger the risk that the incorrectness of design will lead to ever more expensive countermeasures. And in many cases, design flaws are too subtle to be seen immediately, and collect negative potential energy in the form of on-going degradation in productivity that came to be seen as "normal".

Big Design Up Front has often come to be interpreted as "No Design Up Front", and has led to a lot of mediocrity in design by inappropriately democratizing the responsibility for design quality. This is often done in the name of cross-training, but the best way to teach potential software designers to be good software designers is to constantly expose them to good software design, rather than the mediocre design that can come from a misinterpretation of Big Design Up Front.




Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Designing the Work

More than just designing the software, technical leadership must also design the work of implementing those designs. Taking the shape of the work into consideration along with the shaping of software modules serves the goal of predictability that comes from leveled production, and the awareness of trouble spots. Designing the work serves human and organizational concerns, and predictable manageability.

Decomposing requirements, be they user stories, features, or what ever you use in your process, is inherently a design activity. Division of labor follows the division of software modules, and the division of software modules also follows the division of labor. These design activities have to be done in consideration of each other, likely by the same people and at the same time.

The design should be communicated to the team, and at the very least, to the members of a team who might be tasked with the work, but it isn't necessary to have the whole team involved in creating the design. The best designers should be involved in doing the design work. This should go without saying, but some of the interpretations of "self-organizing team" can contribute to obscuring the obvious: individuals on teams do indeed have strengths, and some are stronger than others.

Communicating design and expectations is a great side effect of all-hands planning and estimation, but it can create poorer designs that are normalized to the average skill on the team while the team learns how to design software. Becoming a competent software designer takes many years of work and the uncanny circumstances that create the precursors of design awareness that, when complimented with experience, creates the talented and mature abstractionists, analysts, implementers, and engineers who lead design efforts. Planning and estimation and teaching and design do share some common ground, but teaching design and growing designers should be an explicit goal with specific work, rather than a magical side effect.

Decomposition, task breakdown, and scheduling are all aspects of design. When we communicate design to the people doing the implementation, we're setting expectations for their work. When we set expectations for people, we (hopefully) activate their critical minds. They inevitably cross-check a work plan and inevitably cross-check the design as they learn about the plan. That's not exactly the same thing as the design-by-committee exercises that are too frequently the result of consensus estimation design, where decomposition is often done as part of whole team sessions that are concerned with dialing in agreements on story points.

Designers front-load the design, bringing their experience and aptitude for design to bear, and provide structure for the ensuing conversations. The tension between the design and the cross-checking can lead to new insights. It can lead to changes in the scheduling of the work and it can lead to changes in the design, but it's not a consensus-based free-for-all that leads to the mediocratization of design. The front-loaded design done by competent designers produces a more balanced equation, especially when they are also designing the work.

This isn't Big Design Up Front or phased-based SDLC though. Designing the work simply means that there's necessary front-loading involved in producing software. Front-loading doesn't preclude inspecting and adapting, responding to change, or emergent design.

The balance between good work that can be continually built upon, work that creates lower standards of productivity, and work that can be managed to expectations hinges on more than good software design. Software development productivity lives at the intersection between well-designed software and well-designed software work. Agile estimation is a good start down the road to manageable software development. Designing the work closes the gap between the beginnings of manageable work, and work that can be managed.





Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Monday, May 25, 2009

Flow, Leveling, and User Stories

Flow and leveling are two sides or the same multi-sided shape. The goal is an understood and controllable cycle time. Without leveling, sustained flow is unlikely. Without flow, improvement is less observable and manageable. Improvement efforts might be rooted more in medieval software superstitions than something closer to a science.

David Anderson talks about the core of Kanban as an agreement that the team has a capacity; that they can only work on so much at a time; and a limit set around that work. Implicit in Lean work management styles is the right-sizing of work items. Reducing the broad variability in work item sizes is a part of the work that goes into flow and into controlling cycle time.

Typically, in a method like Scrum, a team organizes its work efforts around user stories. The user stories that are worked on during an iteration (or sprint) can be any of a number of sizes. Using the Fibonacci estimation technique the variation in story size can be almost infinite.

User stories are a way to communicate expectations, but they're a less effective way to manage work and improvement because of the amount of variation allowed in user story sizes. You could decompose stories into smaller stories so that they're more manageable, but then you'd risk fragmenting the communication value of stories by breaking them into separate chunks that might not carry as much context as the original larger stories.

User stories can be of variable size because they inherently are of variable size. Let user stories be the size that makes sense for the kind of artifact that user stories are: an artifact intended to communicate context and expectations to the development effort. Use right-sized work items to manage work. Decompose user stories into work items and schedule those work items for development in way that takes both customer priority and technical constraints into consideration.

The work items can certainly be linked back the stories that they are decomposed from, but it's the work items rather than stories that go through the development pipeline.

I've sat through enough all-hands Agile planning and estimation sessions to know that while this practice is useful for bring a new agile team together, it's often a terribly ineffective way to establish design. It's a good way to socialize design, but there are alternatives that don't imply the wastefulness of traditional Agile planning and estimation. Using design as a team-building exercise can put the design at risk.

In the worst cases, the all-hands planning and estimation that is common to Agile development methods can hurt the team. It's just as likely that adversarial conditions can break out in the team if an expectation is set that suggests that product design and implementation design is egalitarian and democratic within a team that is necessarily staffed by people of varying experience and ability.

Design should be done by those who are most capable of doing design. Story decomposition should be done by the person or people who are best equipped to do that work. This doesn't preclude the socialization of that design to the team and the proving of that design by the team.

Rather than all-hands estimation and planning, design and planning can be done in short, effective sessions with the team's designers. Since designers are decomposing to work items that are intended to be of a similar size, the team no longer estimates work, but reviews the decomposition and seeks clarification and raises concerns. The collective intelligence of the team is still leveraged, but the potential waste and ineffectiveness of traditional Agile planning is avoided.

The planning and estimation activities change from larger-scale chunks of time, effort, and team participation to smaller units of tiered work and cooperation that can be done just-in-time rather than on those specific days of the week when the whole team's attention can be coordinated and directed to a fixed-schedule meeting. Consequently, any all-hands team meetings can be scheduled on an as-needed basis as well, eliminating any waste that comes from institutionalizing all-hands meetings according to a fixed timebox.

With planning, design, and scheduling being done in smaller units on a just-in-time basis, some of the fixed time boxes in the development process aren't needed. Features can be queued for packaging and deployment on an opportunistic basis as well. There may still be some fixed rhythms at-play in the effort - customer demos and major market events, for example - but these synchronization events don't have to leak their scheduling mechanics into aspects of the work that aren't dependent upon or necessarily coupled to those rhythms.

Work items aren't just same-sized, they're also small-sized. The larger the work item, the more variability there is between the estimate of the work and the actual work. We understand larger work items with less certainty than smaller work items. That's because decomposing work into smaller items means that we have to think about the actual units of design and work that go into delivering those bigger features. Because we're trying to achieve flow in pull-based systems, the variability inherent in larger work items is a likely obstruction.

The specific amount of work that defines what "small" is depends on the kind of work, the team, and a handful of factors. And the actual size of "small" will likely change over time. In some cases, it might just be impossible to get all work items to be of similar size. Consider whether the work can be scheduled so that groups of similarly-sized work items can to be done together.

Without leveling, flow will continue to be difficult, and Kanban a Japanese word that has come to mean "story wall".



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Sunday, May 24, 2009

Pull Harder to Find Problems with Flow, and Improve From There

A focus on flow that leads to continual observation of the health of flow lets us use flow to improve organizational performance. Once flow is happening in a pull system, pull harder. Pulling harder will show you where the next level of obstructions is in your organization that keeps you from the next level of performance.

Lean and Kanban are geared to performance improvement, and their principles are applied regardless of the development process in-play, and in concert with the existing process. As opposed to something like Scrum, Lean doesn't require the wholesale, instantaneous replacement of existing processes with a prescriptive process template like Scrum (or others).

Using Lean principles and Kanban will show you the hotspots in your process where you might need to focus your improvement efforts. The software process mechanics that you end up with could very likely end up looking a whole lot like a prescriptive Agile development process, but the adoption and use of specific practices and mechanics are driven by tangible need, based on the repercussions of several instances of pulling harder and deploying and observing the countermeasures that clear out the chaotic oscillations that result immediately from pulling harder.

Pulling harder means increasing the expectations for the capacity of the system. Lean organizations aren't sitting on their laurels. Successions of pulling harder, solving the resulting organizational, process, and mechanical problems, and understanding and working with resulting performance improvements is Lean's continuous improvement. Continuous improvement means an inculcated value system that doesn't allow its past achievements to distract from the next level of goals, and the inevitable ardors of reaching them.

The organization is there to serve the goal. A Lean organization isn't a goal in and of itself. The organization bends to serve the nexus of value creation at the center of the vaue-add work. An obstruction to the ability to successfully pull harder might be found in a part of the organization outside of the immediate product development team. If that's the case, it's understood that the system as a whole must be optimized in order to continue to improve performance.

Once you've got your bicycle up to speed, pedal harder and see what might get in the way of being able to. And then devise and manage a well-considered plan to adapt the process and the organization to get those obstructions out of the way.

Pulling harder creates process improvement simply by setting expectations that are designed to clearly and brightly illuminate the ugly obstructions the lurk in organizations and processes and even in the tools being used. Simply: improve by improving. Set new expectations for the well-understood process and team and take a principled approach to understanding and working with the team's capacity.

Merely overlaying new process templates on a team isn't really the most reasonable and effective approach to improving performance. A change in process should be goal-driven and executed methodically. You might make some sweeping changes in the process, but the wholesale adoption of something like Scrum's agile project management prescriptions isn't likely necessary, and in some cases such a course might exasperate the effort for performance improvement.



Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Friday, May 22, 2009

Flaccid Kanban

A few shops in my professional community have started flying the Kanban flag. This is potentially a great step forward in opening the agile software development dialog to a more rigorous understanding of value, especially in how value is lost and how to defend development efforts from the pervasive value leaks in end-to-end workflows.

XP and Scrum shops - especially those with people with significant XP and Scrum experience - might trip over their unrecognized biases at first.

Iterations and Rolling Wave

Kanban doesn't require iterations. Scrum and XP work management doesn't magically transist into Kanban when we remove iterations. What we get from iteration-less Scrum and XP is Scrum or XP with rolling wave planning. Regardless of the benefit of rolling wave, it's not really Kanban until you're asking (and answering) some fundamental questions about work management, organizational performance, and value.

What is the purpose of Kanban?

The purpose of Kanban in software isn't to merely ditch the overhead of iterations. That overhead is a form of waste that might be avoided when iterations are taken out of the picture, but depending on the surrounding context, just as much compensatory waste might be created when iterations are removed.

The purpose of Kanban - other than visual control systems - is to limit work-in-progress in relation to a team's capacity in order to become sensitive to opportunities for improvement and to see waste and value loss (attribution: david anderson). But it's not enough to just announce in a team meeting that context-switching is now verboten. There's a point to limiting work in-progress, and there are a good number of inter-related practices and principles that benefit the work and the management of the work that come into play.

Flaccid Kanban

Martin Fowler recently wrote an article entitled Flaccid Scrum. The article calls attention to a Scrum implementations and adoptions that try to achieve agile software development's promise without implementing anything but agile project management, ignoring the engineering practices that allow agile project management to yield the promises of agile software development as a whole.

As Lean gains popularity and adoption, it becomes susceptible to the degradations that come of mainstream opportunism. Kanban adoption into relatively mature XP teams seems to be creating a sort of Flaccid Kanban - implementations of Kanban that more or less disregard the underlying reasons and mechanics of Kanban.

The real loss here is that we tend to stop looking for something once we believe we've found it. What has been frequently adopted lately that has been called Kanban appears to be merely rolling wave planning. Not that there's anything wrong with rolling wave, it's likely a necessary component of Kanban in software development, but there's nothing good about reaching for Kanban and coming up short because of the temptation to use Kanban in-name-only merely for the fashionista value.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

A Metaphor for Flow

Kanban is concerned with flow. Flow brings a group of inter-related concerns into play. Flow isn't as readily-understood in software production as it is in other production industries. Here's a simple metaphor for flow:

Flow is like pedaling a bicycle. As long as you're pedaling, you can continue to pedal, and maybe even pedal faster.

If there's any obstruction to pedaling your bicycle, you certainly won't be able to pedal faster. You'll be pre-occupied with just getting the pedaling happening again. If the chain breaks, you certainly won't be pedaling at anything even approaching your optimum.

So much software development effort is spent trying to get the chain back on the bicycle that we're practically inured to stagnant flow and verily accept it as inevitable.

Waste is only inevitable if you throw your hands up in the air and give up on the field of study and effort that moves software as a production industry into the realm of maturity that other production industries enjoy right now.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Friday, March 20, 2009

Specialization in Recruiting in Hard Times

Recruiting is only one part of the human resources game, but it’s a self-contained sector when businesses are clamoring for talent. It’s a specialization that has a hard time remaining sustainable when the economy isn’t vibrant enough to support specialization.

Recruiters who are having a hard time have an opportunity to un-specialize, and to turn the business into something that is aligned with circumstances in the here and now. That doesn’t mean that they’re guaranteed to survive, but if they do, they’ll be well-placed to emerge quickly from the slump when it ultimately comes to an end, and to do so with more diverse and compelling services. As difficult as things are right now, dealing effectively and creatively with constraints and surviving them is a darned good way to build a foundation that will support a bigger and better business when the economic tide finally turns.

Recruiting is a brokerage business. In good times, recruiters build well-tended relationships with buyers and often terse and purely functional relationships with recruits. The recruiter brokers the relationship between a business and the talent, collecting fees for acting as an effective middleman. At least that’s how it works when there’s flow through the system.

Without demand from buyers, the need for middlemen essentially collapses, but that doesn’t mean that the human resources business simply disappears. It means that the relationship broker’s role has to change to account for the deficient flow. When the market is down, smart recruiters can play the down market by investing!

Sooner of later, businesses will come looking for talent again. A recruiter that emerges from the downturn with a strong portfolio of relationships with good talent to broker with buyers will emerge strong, and can remain strong for the duration.

Here are a handful of things that recruiters can do now to help weather the storm and to prepare to emerge with strength on the other side:

Build Relationships with Employees
Recruiting is a relationship business. If the relationships with the demand side of the business have gotten quieter, then turn your attention to the supply side! Stay in the game: build more relationships with employees. There are more of them to connect with and they don’t know you as well as your buyers do, so trying to pull this off from your desk is likely not going to work. Get out from behind your email app and Facebook and show up in person at community events. Relationships with people should be personal.

Be a Career Counselor
You’re an HR professional with valuable experience. Get an understanding of employees concerns and give them the benefit of your experience as a professional to help them understand the current climate. You have inside knowledge of the kinds of things that employers value and of the things that make employees valuable. Understand their challenges and concerns and help them to stay focused on the personal development that will help to lend more stability to their professional lives. Listen to what they have to say. Giving the benefit of your experience to the employees is not only beneficial to the employees themselves, but it’s also valuable to employers whose employees you serve. This economy can only benefit from well-informed, level-headed employees that have a continuous source of good counsel.

Invest in their Skills
Nothing builds relationships like giving people something of value. Help people to step up to new skills and to improve the skills they have. Host creative learning and education events to help the supply side of your business to go deeper in their work. Connect them with teachers. Partner with teachers to make investments in the market and leverage the access to the supply-side to deepen your understanding of what makes them tick and what they are concerned with. Build them up!

For Goodness Sake, Don’t Recruit Them!
You’re not doing all this to prepare employees to be poached from their employers once the downturn is over. You’re building relationships with employees because relationships with employers are increasingly scarce. Great relationships with employees that have come about because you’ve made an investment in those employees will turn into great relationships with their employers. And if an employee chooses to move on from an employer, you may have an opportunity to help them find their next position, but under no circumstances should you try to actively harvest them.

When the employment market heats up again, the human resources business people with the best reputation building and sustaining a human resource portfolio – especially in a constrained market – will be in demand.

When this economic slump is over, I’d like to work with recruiters who have not only survived the downturn, but who have thrived. When the slump is over, the specialists are going to be crawling out of the woodwork again after having lain low while the going got tough. While the recruiters who have hibernated through the downturn may be just as good as any others, I’m more confident that those who have learned to thrive under constraints are definitely the ones I want to know more about.

In fact, I not only want to connect with them when all this is over, I want to connect with them right now!


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Thursday, March 19, 2009

Productivity, not Bailouts

More than fifty years ago Toyota learned that it was possible to survive tough economic times by facing them head-on; by dealing with drastic reductions in resources through drastic increases in productivity.

If you have fewer resources to do the work at-hand, and the amount of work can't be reduced, then productivity must increase. Reducing waste is a huge part of increasing productivity, but waste reduction on the scale necessary to dramatically increase productivity doesn't happen without organizational and cultural changes.

People must become inter-disciplinary and cross-functional during periods of lower prosperity; times like right now. We can survive and even thrive in tough times by increasing productivity, and we can increase productivity by reducing the amount of time lost to hand offs between individual specialists in a workflow, and by capitalizing on the increased knowledge that comes from people having a greater understanding of the work going on around them.

Greater inter-disciplinary work leads to greater productivity, and greater inter-disciplinary work means challenging our assertions about our entitlements to our particular comforts with remaining generalists or specialists. Rather than continuing to toil in a paradigm that pits generalism and specialism against each other in a constant, irreconcilable tug of war that itself causes more lost time, we need to extend the reaches of individuals into the disciplines that surround their present areas of operation.

We tend to get a bit lazy in times of greater prosperity. We feel entitled to our comfortable, walled gardens. And from this entitlement we generate ever more waste and slack; waste and slack that isn't much of a concern relative to the prosperity that we enjoy. When the prosperity goes away, we're faced with the inefficiencies inherent in the hand offs between walled gardens that accrete along the workflow.

Becoming inter-disciplinary doesn't merely mean trading specialization for generalization, though. It means purging the entire paradigm that pits specialization and generalization against each other as natural opposites.

Specialization and generalization can only exist in relatively pure forms when we can afford the waste that they cause. The waste comes when we have either specialization or generalization. This either/or model is unsustainable. Sustainable productivity is created by a workforce that is both specialized and generalized within the context of someone's area of operation, with the understanding that a person's area of operation will continue to expand over their career.

Failures to reproduce Toyota's successes in the west were originally blamed on the cultural differences between Japanese and western workers. Western companies ultimately learned that the real reason for the failures was that they had missed the essential and immutable organizational imperative of Toyota's methodology: the creation of a learning organization through the fostering of a learning culture. Western organizations consistently failed to replicate Toyota's successes because they often merely adopted Toyota's processes as process improvement efforts.

The world sits on the cusp of the greatest opportunity we've had to rid ourselves of the antiquated mass production methodologies that we continue to wring ever decreasing returns from. At no time in this workforce generation's life have we needed a revolutionary boost in productivity.

Before we had this massive financial crisis, we had a massive productivity crisis. It has been with for years; looming, ever-present, waiting to be triggered. And here we sit in the midst of it. Nothing is so immoral at this time than the continued protectionism of our petty specializations that do nothing but exasperate the productivity crisis that we find ourselves in. And nothing reinforces the entitlement to these specializations like an organization that has little understanding of how profoundly an organization's productivity can be transformed by meaningfully committing to becoming a learning culture, and bending all procedural and organizational imperatives to that goal.

And once we've made the meaningful changes to our organizations and culture to bring about dramatic and sustainable changes in productivity, those new habits can be sustained through the prosperity that follows. We have an opportunity to learn not only how to thrive during crisis, but also how to bolster our society against the cultural rot that regularly brings us to the brink of productivity ruin.

Whether or not the bailouts will serve us in the long run, productivity certainly will. And the productivity that we need is likely not the productivity that comes in a box with a major software company's logo on it. We need a culture of productivity holding together a society of productive people, empowered by vigilant awareness and knowledge.

It's my great great hope that we look to the revolutionaries of human productivity of our time and seize this opportunity as if our society depended upon it.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Thursday, March 12, 2009

Workability

Software development pop vernacular probably doesn't need another "ility" word.  We've got a good collection of the them already: scalability, extensibility, availability, recoverability, among others.  Nonetheless...

Mature production industries already recognize a design quality called "workability".  Workability is at the heart of a number of software design qualities that individually often deliver a fractured message about what makes good software practices good.  Approaching software design and implementation from the perspective of workability helps me stay focused on the higher order objective while simultaneously not loosing track of the principles and practices that permit workability in software development.  That higher order objective is producing.   Or said differently, it’s production.  Or better yet: productivity.

In The Toyota Product Development System, the authors talk about Toyota's concern with workability during digital assembly, a form of virtual reality simulation geared partially to proving that a product's build workflow is practicable:

Use workability DA [Digital Assembly] to study in detail the effects that design changes will have on ergonomic issues involved in assembling a vehicle.  By utilizing DA in conjunction with the assembly plant pilot team (hourly workers assigned for two years to work on preparing a new vehicle launch) during the Kentou [prototyping, modeling] period, they can address both current as well as anticipated human factor issues and identify the ergonomically safest and most efficient way to assemble the vehicle.

Workability is the design quality that simply permits a job to be finished some day.  Workable software allows us to keep working on a software product without being obstructed by the software that we’ve already added to the product.  Without considering workability in design, workers end up painting themselves into corners that they can’t get out of.

We see this all the all the time in software projects whether we recognize it or not.  When productivity degrades sharply after the initial parts of a software product are built and assembled, it’s the absence of workability that is at the root of this all-too-common situation.

Workability is a higher order concept that accounts for  a myriad of principles, properties, tactics, and counter measures that are constantly in-play when high-functioning teams are at work.  In software, workability is the testability, the readability, the maintainability, the SOLID principles, and a host of concerns that are continually being balanced, evaluated, and enacted during design and implementation in the name of defending a software effort from stasis.  In fact, those principles are so essential to productivity that they are far from unique to software.  They permeate all kinds of production across many industries. 

There are a lot of elaborate and expensive tools and ideas in the software industry, but the vast majority of them simply don’t serve workability, and many of them actually obstruct higher orders of productivity.  If productivity can’t be sustained over an entire software effort, then that initial productivity really isn’t much in terms of productivity, is it?  In fact, productivity that can’t be sustained is arguably not productivity at all, and tools that deliver an initial, rapid productivity spike at the beginning of an effort are likely also contributing to it’s decline.

Sustainable productivity isn’t the result of tools, it’s the result of the collection of design qualities and practices that in aggregate can be thought of as workability.  Tools serve workability, but they can’t cause it – people cause it.  As long as people are involved in making things, workability is simply how good things get made.


Ampersand GT

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com