The 7 Software Development Wastes – Part 1

Introduction

The more I learn about Lean, the more I realize how much we can learn from Lean teachings and how they apply to software development practices. Typically, we go about our day-to-day activities without thinking about the bigger picture. Lean specifically addresses the complete end-to-end process with the view of enhancing cycle time and quality.

Lean

Value stream mapping is one of the key areas for helping us learn where we fail, but in particular, what I’d like to address in the next series of posts are the 7 wastes which Lean identifies and which I believe are worth mentioning. Once you hear about these wastes I believe you will be more sensitive to realizing when you see these manifested in your organization and should therefore help you to enhance the overall productivity of your teams.

Mary and Tom Poppendiek are two of the thought leaders in applying Lean to software development and I would highly recommend that you buy their books or attend their courses. Mary in particular is inspirational and their knowledge in this space is profound.

7 Manufacturing Wastes

The 7 manufacturing wastes identified by Shigeo Shingo (from his studies of the Toyota production system) are listed below:

In-process Inventory
Over-Production
Extra Processing
Transportation
Motion
Waiting
Defects

Corresponding 7 Software Development Wastes

The 7 corresponding wastes in software development as defined by Mary and Tom Poppendiek are:

Partially done work
Extra Features
Relearning
Handoffs
Task Switching
Delays
Bugs

Today I’d like to address the 1st waste – Partially done work

Partially done work (In Process Inventory)

Partially done work is probably the biggest killer of all the wastes. Partially done work is essentially work-in-progress. Until this work is done, you don’t know that there are quality issues i.e. you don’t know that the customer will be happy. You don’t know if there are going to be problems one you deploy your software onto your production systems. So the idea should be to complete work-in-progress as soon as possible i.e. minimize work-in-progress as much as possible. Examples of partially done work are:

* code that is completed but not checked in to your version control system – if it’s not checked in, you don’t know if your code changes are going to break the build

* undocumented code – If your code is undocumented, if the developer leaves and someone has to take over, there’s going to take longer for the developer to get up to speed. Additionally, if bugs are found, it will be harder for the original developer to figure out what he has done.

* untested code (both unit tests and functional tests) – if your code is untested, you won’t know till the code is in your customers hands that there is a bug. The further downstream you are in the process the more costly it’s going to be to fix the bugs. So if you build quality in from the start (like writing unit tests) you’ll find out the moment you execute the tests

* code that exists on your staging environment and not your production environment – I hear this all the time – “works on my machine” – enough said. Only once you’re on production can you be sure the software is 100%. Production always surfaces issues, so the sooner you get it on production servers, the better.

* code that is commented out – makes the software less readable and maintainable

My advice to any company doing software development is to figure out how to get things that are in progress fully completed i.e. DONE in accordance with your definition of DONE, as early as possible. Don’t put too many items into the development pipe that you know you can’t complete in a given Sprint. Consider everything in your definition of Done to be completed in one cycle. Set your quality bar really high. This will help you ship faster, make your customers happier sooner.

Next week I will be going over a second type of waste – Extra Features.

Who’s the owner? Shared code vs. code ownership

I think it’s widely recognized that shared code is important in agile teams. It’s an effective way to streamline your development process, making your teams more flexible and productive. Transitioning from code-ownership to shared code can be hard. Lets compare the two, look at the problems in transitioning from one to the other and see how we can ease this transition.

Shared code is only a corollary practice in extreme programming but most agile teams I’ve seen implement it successfully. Many other teams are still in a situation where team members have what I call code ownership, the code base is divided into parts that can only be edited by certain persons or groups within the team. Usually the code is divided along lines of expertise, for example database-professionals develop the database, the back-end and domain logic is developed by one or more experts, and interaction designers and UI developers are responsible for the user interface. The reasons for this division are usually efficiency, people work on the parts of the code that fit their skill set, and responsibility, when something is wrong it’s easy to find the guilty party by looking at what piece of the code failed.

Why Share?
In practice things work a bit different.
Code ownership is actually not very efficient, most non-trivial features will take changes in most of the layers in the application. This will create unnecessary dependencies between team members and unnecessary delays.
Having people take responsibility for pieces of code isn’t very useful either. Eventually implemented features are what count. I have seen many situations where no-one took responsibility for a defect in a feature because of code ownership. The database-guy blamed the back-end guy, the back-end guy blamed the UI team etc.
Code ownership is also risky. Specialization like that will create knowledge silo’s in your team. If the database guy gets hit by a bus who will take his place?

So what’s keeping you?
So if shared code is the way to go, how do we get there? Is just telling everyone to edit any piece of code enough? Of course it’s not that easy. When you remove safeguards you’ll have to replace them with other safeguards, there are a couple of things that need to be in place to enable shared code, responsibility and trust

Responsibilty
We already looked at responsibility, making individuals responsible for parts of the code is not the solution. In teams with shared code the whole team is responsible for the whole code base. But if more than one person is responsible then effectively no-one has responsibility.
But in parts of the code base aren’t that important to our clients. Working features are, user-stories are. Most agile methodologies let people take responsibility for implementing features or user-stories at the start of an iteration. Dividing responsibilities along the same lines as the actual deliverables results in less finger-pointing and more cooperation.

Trust
In many cases developers themselves are opposed to the idea that any other team member can change the code they worked on at any time. Team-members need to trust each other in order for shared code to work, they need to be sure that every time a team member edits some piece of code the code will be better than before the edit. This means the team actually needs to agree on what ‘better’ is. So there need to be coding standards. It’s also good to have a code-review process in place. Of course instant code-review in the form of pair-programming would be even better. A good version control system is a good safety net in case things do go wrong. Practices to get quick feedback on code quality like TDD and continuous integration also help.

Shared code is an important agile practice but getting there takes some effort. Getting rid of code ownership without a support structure can cause problems agile practices in order to guarantee consistent quality and to work in a responsible way. Agile practices like TDD, continuous integration and pair programming help keep quality high and responsibiltiy needs to shift from code to features in order to be successful.

Chopper Shops

Photo (c) Janusz Gorycki

I have been watching an episode of American Chopper the other day. I must admit that I am a bit of a fan of this show – not necessarily because of its main plot premise of a clash of personality between the father and the soon (which is rather cheesy and quite clearly badly acted instead of real), but rather because I like watching “how stuff is built” – in this case, how you can build a gorgeously looking motorcycles from the rusty pieces of metal, using not much more than a set of rather primitive tools.

It was an episode where the guys at OCC bought themselves a super modern, computer-controlled metal-cutting machinery of some sort. At the same time, they had to resort to making order with some misbehaving part of the motorcycle being constructed by whacking it hard with a big hammer.

Watching this show got me thinking: this sort of work is very similar to what I am doing every day creating software. I always cringe when I read somebody likens software development to a factory line. Even if it is an “agile” factory line – like the Toyota Production System, which a lot of people in the agile community hold in such a reverence to that they almost pray to it daily. Because, you see, software development, in teams big or small , organizations huge or microscopic, is always a chopper shop rather than a factory.

The dynamics of what we do is that of a craftsman at OCC or some other chopper shop – we use our skills and creativity to create custom products – each of them is different that the other, no two are alike. We do not create everything from scratch – we do reuse standard components obviously, just like they use standard engines and standard tires. But we put them together so that they become something new and (hopefully) exciting every time.

Now, the factory line is doing something entirely different – its sole purpose is to produce repeatable products, as similar to each other as possible, as fast as possible and as cheaply as possible. A long time ago during my high school days, I had a summer practice in a TV set factory. I watched with admiration how the factory line worked, it was pretty amazing even some twenty five years ago – but I can see no relation in a factory line to what I am doing right now as a software developer.

Now, what are the practical consequences of realizing that your team is a chopper shop? It makes you realize what are the most important thing you should put emphasis on. It becomes obvious that the quality of people is essential for your success. Tools that we use are not all that sophisticated and pretty much every team can afford the best ones available. Same with processes. They have to be simple because they invariably break down when you over complicate them. Nothing to it really – you should not believe when somebody claims that the successful software organizations use some awesome tools and methods – no, they use standard, if not sub-standard ones. It is all in the people, their skills, their talent, their dedication. Often times people with clashing personalities, ego trips, personal problems. Not ideal, not saint, not necessarily disciplined. Still, they are the most important ingredient what makes or breaks the success of your projects.

Additionally, you start to see “waste” in a different light. It is not necessarily a bad idea to spend time and effort on some task, only to decide to throw it away later. In one of the episodes of the American Chopper, the crew decided that the fuel tank they have been working on, does not look “cool”. They “wasted” two days on it. But they decided to redesign it from the scratch anyway, even with a deadline approaching. There was no big fuss over it – it was just the way things are when you try to be creative: there is a danger that your brilliant ideas you may have may prove to not work in practice. But you have to try them anyway, because otherwise you have to resort to safe, but uninspired and uninteresting solutions that nobody cares about. OCC would not sell a single chopper if they all looked like Yamaha or Harley-Davidson ones. The whole point is to produce something that is unique.

Measuring velocity is not enough to determine team productivity

Introduction

Another interesting question was brought up in a LinkedIn discussion thread this week: “Is velocity the right approach to measure productivity of team members working in Scrum. If not, then how can productivity of team members be measured in Scrum?” Here are my thought on this topic.

The purpose of velocity

Like burndown charts, velocity is just another metric which the team can use to reach what I believe is the ultimate goal – sustainable throughput. Velocity in my opinion, is not a metric for determining productivity. Productivity or team efficiency is difficult to measure and probably best left for another blog post.

From my perspective, velocity should be used for planning purposes only i.e. how many user stories I can plan into a Release or Sprint based on my teams previous velocity (using story points of course).

Knowing your teams velocity is an important part of Scrum. Knowing your velocity is also central to lean thinking in that it defines the capacity for how much work your team can manage at any given point in time. It’s important to note that velocity only remains static if the teams conditions remain the same. For example, if the team composition changes, velocity will change. If changes are made to the process (e.g. start pair programming or start writing unit tests), then you can expect velocity to change as well.

I think that Paul Hodgetts, Agile Coach and Certified Scrum Master, said it best when defining the purpose of measuring velocity:

“Velocity is more of a quantity of work completed metric. Useful and important, but not the sole measure of success. I think you would want a set of metrics that together help us understand our current capabilities to deliver as well as if those capabilities are changing from one point in time to another. I don’t believe there is one magic “agility number” to measure [productivity].”

Another important aspect of velocity is that it’s a measure of how much work is fully completed at the end of the Sprint i.e. represented by how many completed stories there are at the end of the iteration.

Relieving pressure

Moreover, why I am such a big fan of velocity is because I believe that once a teams velocity has been established, it takes a lot of unnecessary tension out of the development process. In my 20 year software development career, I have rarely had a situation where the business fully trusted the development team. It always seems to boil down to the development team not working hard enough, leading to many hours of overtime and ultimately burnout.

Velocity on the other hand is the neutralizer. Once this number is established, the business gets used to understanding how much work can be shoved into the development pipeline and have a good chance of being delivered on time Sprint after Sprint.

Why it can’t be used to measure productivity

I think it’s worthwhile explaining why I think velocity is not a great indicator of productivity?

Productivity in my book is how efficient my team is, while velocity can tell you how much you’re getting done; it’s not going to tell you if you should be getting this done faster or not. One would have to look to industry standards to try to measure productivity like function points.

Consider this…

Your team could have reached it’s sustainable throughput, continuously delivering the same amount of software, and still be counter-productive. Why? Because you later find out that what you just delivered didn’t result in any value! At the end of the day, it’s the customers and stakeholders that measure the success of the project. Velocity isn’t the end game, the most important thing is that you’re delivering features that the customers want and with quality that’s expected by them.

Conclusion

In summary, what’s important is that the team is delivering value in a sustainable fashion (value as determined by the customer). So in order to enhance productivity there are many things one can do:

* You have to get the requirements down pat. The best way to do this is to work with the customer to define the user stories and get their intent up-front.
* Ensure you are capturing the acceptance test criteria from the customer up-front. This helps solve the greatest risk of all – getting the wrong product.
* Define your definition of “done” up-front so that there’s no argument at the end.
* Ensure you build quality in right from the start.

If you do this well, your teams will become highly efficient and the need to measure productivity in the first place disappears!

Burndown charts – It’s not about staying below the line

Introduction
I recently participated in a Scrum development discussion thread on Yahoo Groups where one member new to Scrum asked the following: “Our burndown chart’s remaining work line always goes up. As a Scrum Master, what do I have to do to make it go down?” This question, surprisingly, generated a lot of response from the community. I found it puzzling to see how adamant some were to introduce solutions to get the remaining work line to go below the estimated work line.

Goal

I believe that the underlying goal should not necessarily be about staying below the estimated time line in a burndown chart (although this would be nice). Rather, the goal should be about achieving SUSTAINABLE THROUGHPUT (figuring out what your team’s development capacity is and how much you can shove through the pipeline). A burndown chart is just an information radiator that is there to help you to determine whether that goal of reaching optimal capacity is achieved. That’s it. So as a Scrum Master, you should not get too overly hung up on the burndown chart.

Understanding the problem

If you know you are completing tasks but your burndown is still going up, this probably means that you’re adding more tasks during the Sprint than your development team is able to finish. But there’s value in knowing this information. When you see that your remaining work line is above the estimated work line by the end of your Sprint, this is a good indicator of the following:

• Your development team is applying new design choices to the existing code during the sprint.
• The tasks you have scheduled are so large and undefined that the hours keep going up as you figure out the complexity is increasing.
• Or your team is having difficulties estimating – which is common among new Scrum teams (see my previous blog on the significant of story points here).

What you can do about it

These are just a few examples of what a burndown chart can tell you. There could be a whole host of other reasons why you’re above the estimated line. However, there’s a number of things you can do; you can remove tasks off the sprint or you can try and break the tasks down further till you have something more manageable. The challenge is pinpointing the reason and trying in a very practical way, to get the team working at a sustainable capacity in order to manage all tasks for each iteration while minimizing work-in-progress.

Practical use of the tools

Another point that was brought up in the discussion thread was, why have a burndown chart at all if you’re using a task board? Although I do believe the task board provides value, I don’t believe that it should be a substitute for the burndown chart. The Burndown, Kanban boards and burnup charts are all just tools in your project management toolbox to assist you with continuous delivery of value to the customer. And each of those tools serves a different purpose. With a task board, you know which tasks are in queue, you know which are in progress, you may even know how big the tasks are. But without the burndown, you don’t know how far along the complete trajectory you are. Just because half the cards on a Kanban board is in the “done” column midway through a Sprint, doesn’t necessarily mean that you have completed half the story points for that iteration. The reason is that you may have smaller or larger tasks remaining. And the larger the project, the more complicated it is to get a sense of the size of each task just from glancing at a whole bunch of cards on a board. Everyday my teams provide a brand new estimate (in hours) for all tasks in progress (this takes them just a few minutes to update each day). So I take a look at my burndown, and in one second, I know if we are ahead or behind the plan…

Open Space, an agile way of meeting

Last year I went to the European Agile Open conference, two intense days of meeting interesting and interested people and talking about all aspects of agile. I don’t think I ever learned as much in two days. The conference was held entirely in the open space format and one impression I left with was that lots of the ideas we talked about there were also applied in the organization of the conference.

This week I had the pleasure of helping to organize my own open space event. The Open Space Code day in the Netherlands. We copied this idea from Alan Dean who has been organizing events like this for some time in the UK. I got the same impression, the dynamics are different, the goal is different but the event certainly feels like an agile project. I want to have a look at the similarities and see what we can learn from it.

What is Open Space?
Open space is quite popular in the agile world. I think this is partly because it feels familiar to most agilists. But for people who haven’t had the pleasure of participating in an open space event here’s a short introduction. If you want to learn more about this you can visit the open space institute website or read Harrison Owen’s book.
Open Space is a way of making conferences interactive and self-organizing. Speakers and sessions aren’t pre-planned. At the start of the conference attendees take time to propose sessions themselves. Usually you get more session proposals than the available time and space allow. So sessions are voted on and the least popular sessions are cancelled or combined to form new sessions.

Focus on communication
This is what makes an Open Space feel like the coffee break in a normal conference. And I mean this in a good way. I went to the Microsoft dev-days last year and I even skipped some sessions to be able to talk to more people instead of just listening to speakers.

Open space sessions are interactive. There are no prepared sessions but when you have a room full of people interested in the same subject you don’t need preparation. The combined knowledge of people the people in the room is probably greater than that of any one speaker. The trick is sharing that knowledge in an interactive way without anyone dominating. Open Space accomplishes this with the ‘law of two feet’. People are encouraged to leave sessions they’re not participating in or learning from. This makes sure sessions stay interactive or at least interesting.

Fast feedback
Fail Fast is a mantra often used in the agile world. Agile methodologies add many short feedback cycles to projects to make sure that when you’re doing something well you know it and when you’re doing something wrong you spend the least amount of time doing that before things fall over. The ‘law of two feet’ also helps sessions failing fast. Sessions that are not interesting any more bleed-dry to open up space for more interesting sessions.

Prioritized backlogs
Agile teams always try to do the most important work they could be doing at that time. They have a very simple and obvious way of doing that, the prioritized backlog. Just make a list of work-items and make sure that what’s most important to the customer is at the top of the list at any time.
The Open Space planning process is just as simple. Sessions are planned at the start of the event and are prioritized by voting. During the event sessions are merged, re-planned and moved continuously to make sure all sessions are the most interesting sessions possible.

YAGNI
You aint gonna need it! Don’t make assumptions about the future if you don’t need them today. This is how agile teams keep themselves from making large investments of time and resources that will not pay off. Open space also does this. Sessions are not planned until the last moment. People don’t like uncertainty so this might not sound like a good thing to you. But in practice this makes sure you don’t commit to things that later turn out to be less than optimal.

Self organization
Self organization is of course everywhere. Any group of people will self organize into something, groups of football-hooligans are also self organized. The ‘trick’ is to steer this self organization and use it to accomplish your goal. Agile projects are very good at enabling self-organization within teams. Teams pace themselves and divide their own work.
Self organization is even more central in open space than in agile. Given a couple of basic rules all attendees will self-organize into groups for sessions. The

So what can we learn?
The biggest surprise for me was to see self-organization work in the formation of teams. Agile enables teams to self organize to do work. It was fun to see larger groups of people actually self-organize into teams to tackle a set of tasks and then re-organize for the next set of meetings. Many agile organizations are flat at the team-level but the teams themselves are actually organized hierarchically. Look at the scrum of scrums for example. I can’t help but wonder if we can’t be more effective by introducing self-organization on this level too.

Open space can learn from agile too. The only official planning session is at the start of the conference. This makes the session schedule less fluid than it could be. People are allowed to re-schedule sessions but because there is already a plan in people seem uncomfortable doing this. Agile only plans one iteration ahead and this makes the whole process more fluid.

Agile principles work outside of software development. Things may look different because time-scales, tasks or roles are different so things may not be immediately recognizable as agile but it’s very informative to compare these ideas. Right now many agile teams are adopting ideas from lean production for example. But there are more practices that we can learn from. Open space is one of them.

Advertise here

AgileSoftwareDevelopment.com is a popular web site that provides valuable advice on surviving in the world of Agile software development methods. We are focusing most on helping the project managers and team leaders with managing people and requirements and we are ranking exceptionally well on search queries such as “excel product backlog” and “agile software development”. Every month we serve 50K+ pageviews mostly to North American, but also to significant share of European visitors. We are open for custom advertisement campaigns (as long as ad is clearly identified as an ad) and we also have a number of ready-made banner offers below.

 ads-regions

Region #

Region name

Possible ad size, width x height, pixels

Cost per week, eur

1 Top banner 728×90 or smaller 39
    468×60 or smaller 31
    234×60 (will coexist with another ad) 17
2 Wide skyscrapper on the right 160×600 or smaller 16
3 Button above the syndication 160×125 or smaller 5.5
4 Button below the skyscrapper 160×125 or smaller 5.5
5 Button below the skyscrapper 160×125 or smaller 5.5
6 Big rectangle on the right 320×250 or smaller 14
7 Bottom banner 728×90 or smaller 9
    468×60 or smaller 7
    234×60 (will coexist with another ad) 4

Contact Artem (editor-in-chief) by email or via this contact form for ordering these predefined offers or to enquire about the custom advertising possibilities.

Scrum falls short in at least one area in my opinion

Many of us have come to love Scrum, including me. Scrum has been a huge success overall and has turned many a development team around and for the better. Yet in my opinion, I think Scrum falls short in at least one area.

Concept to Cash

Firstly, It’s commonly accepted that Scrum does not consider the whole life-cycle. i.e. it’s a great framework for helping development teams tame the development life cycle but it does not tackle all areas from “concept to cash”. Lean and Lean with Kanban certainly appears to be an alternative to Scrum or at least coupled with Scrum to provide and all round concept to cash highly effective framework for producing software products.

Definition of Done

Secondly and more in particular, where I believe Scrum falls short has to do with it’s definition of Done. Well, truthfully, the definition of Done is not really even a part of Scrum but once again, it’s commonly accepted that defining what Done means on Scrum projects is an important part of it.

Potential Shippable code – what does this really mean?

Most Agile thought leaders define the output of an iteration as an increment of potentially shippable code. And more often than not, it appears to be acceptable (please tell me if I perceive this incorrectly) that this potentially shippable increment of code is not actually live production code. And therein lies my beef with Scrum.

Lean

I think the Lean folks have this right – they are very interested in the value stream and minimizing the cycle time of this value stream from concept to cash. So my feeling is that this definition should be changed from “potentially shippable code” to “shipped code”.

Iteration Output = Live on Production Servers

On out project teams, every iteration represents a new increment of code LIVE ON PRODUCTION servers – no discussion, no arguments.

This takes serious discipline and focus. And most of all it requires very careful planning. If you get too much work-in-progress you will never turn this in to a reality.

Way to many companies fall short

I find way too many companies after each iteration still trying to close things down in order to get this potentially shippable code shipped. More often than not there’s on-going QA etc long after the iteration is over. This is not a healthy state of affairs.

Take the hard line, change the definition, you won’t look back