Weekly Agile Reading. Pack 11

Top writing that caught my attention this week:

» Scrum Tools | Mingle | Review " Scrum 4 You An Agile project management star was expected.
» Scrum Log Jeff Sutherland: Scrum Makes You Smarter Brain research on rats suggests that voluntary pressure in the form of audacious goals taken on by high performance teams makes them produce more neuron stem cells, rewire the brain, and become smarter.
» Lazy Initialization :: Working To End Agile If agile methods are working and the industry is noticing it’s only a matter of time until it stops being “agile” and just becomes “software development.”
» Matt O’ Rama " My First Agile Project: Part 1 This is the first of a couple of posts on what Matt has learned about how they are doing (and not doing, as he’s learned) Scrum on a big project at his work.
» Agile Game Development: Agile values – Responding to change over following a plan A popular misconception about agile is that it doesn’t allow for plans. This isn’t true. Agile focuses on the activity of planning rather than focusing on a fixed plan. A certain amount of knowledge about any project is going to be well known.

You might like having a look at the previous link packs as well.

If you happen to encounter another interesting content published since last Saturday, please, post links in the comments.

Third Annual State of Agile Survey Data Available

This survey was conducted and sponsored by VersionOne in June and July 2008. It received answers from 3061 participants in 80 countries; most of them (70%) were participating to the survey for the first time. The majority of the respondents were agile team leaders, coach or consultants. This could lead to a bias towards a perhaps slightly more optimistic vision of the reality of agile projects. Whether they are agile or not, managers stay managers ;o)

The survey doesn’t try to measure agile adoption, but it gives interesting information on how agile is adopted. A majority (55%) of the participants works in rather small organizations with less than 100 people involved in software development. For most of them agile is also relatively new, as only 34% of the companies have been practicing agile for more than 2 years. The percentage that has adopted agile in the previous year is 36%. We could see that adoption is often partial in software development organizations as 65% of the participants use agile in less than 50% of their projects. Full adoption is realized in 17% of the companies, a similar number that was found by a Methods & Tools survey conducted at the end of 2007.

Scrum is the most followed agile methodology. Amongst the practices, iteration planning, unit testing, daily standup, release planning and continuous integration were adopted by more than 2/3 of respondents. Surprisingly, pair programming, an emblematic practice of the XP approach, is ranked near the bottom of practices adopted with only around 31% of adopters. On-site customer is also poorly adopted. This shows that practices concerning people interactions seem to be the most difficult to implement, as people are the most difficult elements to change in software development processes.

An interesting question deals with the rate of success for agile projects. For 55% of the participants, close to 100% of agile projects are successful. On the other side, 24% estimate than one project out of two fails, mainly due to conflicts between the company culture and agile values or because of the lack of experience with agile approaches. A final part of the report gives interesting information about the tools used in agile projects. We can see that agile and traditional tools are currently used at the same level to manage projects.

Version One Survey Data
http://www.versionone.com/pdf/3rdAnnualStateOfAgile_FullDataReport.pdf

Methods & Tools Survey
http://www.methodsandtools.com/dynpoll/oldpoll.php?Agile2

Secretaries Make the Best ScrumMasters

Okay everyone. First I want to say that I am happy to be here writing for Agile Software Development. I appreciate Artem reaching out and giving me this opportunity. I really hope you guys enjoy what I’ve got to say. I promise, I am not short on opinions, and I am never content with the status quo. So… to get started, let’s kick this thing off with a question… are you ready?

Do you believe the title of this post?

If you do, don’t feel bad, you are not alone. I have been working in project management and around project managers for years. Over that time, I have worked in traditional environments and agile environments. I have worked with PMPs and CSMs. It consistently amazes me the number of people that are converted into project managers for the sole reason they are good at following people around and asking them when they are going to be done.

Before I get into a full blown rant on this, and the potential is there because this issue really hacks me off, let’s be fair… a ScrumMaster is not the same role as that of a project manager. A ScrumMaster is more of an enabler of the team. They are there to remove impediments. They are there to be a facilitator and to make sure the Scrum process is being properly implemented. The team is self-managing, self-organizing, and empowered. The team works with the product owner to get the project done. The ScrumMaster is there to enable the Scrum process.

By design, this is a barely sufficient description of what it means to be a project leader. This perspective breaks down at scale and it breaks down with a team that is not prepared to deal with the full implications of what it means to be empowered and self-organized. On most project teams I work with, the ScrumMaster is a project leader, sometimes even a manager or a director. They have a degree of accountability for the project that is equal to or greater than any other individual team member on the project.

When I’ve got serious money on the line, when I have made hard commitments to the market, I need someone that really understands how all this works. I need someone that can set the context and coordinate between the team and the business. I need a leader that understands methodology, someone that understands what it takes to build software, someone who understands the business domain and how to work with executives. I want someone that can motivate people, stay focused on the vision, and communicate the right information, at the right time, and tailored to the needs of the audience.

A really good project leader, or a really good ScrumMaster, should be able to help the team [not only] remove impediments but anticipate impediments. They need to help the team stay focused and manage risk. They need to be able to broker complex conversations and help the team come to consensus when communication breaks down and egos get in the way. Most importantly, they need the breadth of experience to help the team devise situationally specific strategies for solving complex business problems.

We need to hold our project leaders to a higher standard. We need to nurture and develop leaders that can do more than facilitate a meeting and go get the pizza when the team gets hungry. We need seasoned IT professionals that know how to balance agility and discipline. We need people that have the experience and judgment to take what they know and apply it for the good of the team and the good of the project. We need people that can really lead a self-organized, self-managing, and empowered team. We need people that can hold them accountable.

None of this is easy, and the people that can do this are really hard to find. I am sure there are many great secretaries, that with the right training and experience, could do a really bang up job leading software projects. I have worked with one or two over my career that I would hire in a heartbeat.

As an agile community, I believe that we undervalue the role of project leadership and the breadth of background and experience required to do it well. Much of the problem though is of our own making. Project managers, and even ScrumMasters, have been applying stupid, simple processes for way too long. We have been operating as if we have all the answers and the team is there to do what we tell them. It is our own fault that people don’t generally want to have much to do with us. It is our own fault that we have people defining our role so lightly on agile teams as to make us irrelevant. The problem for us is that the ScrumMaster role is necessary but it is not sufficient.

As project managers, we need to focus more on servant leadership and situationally specific strategies. We need to focus less on checklists, Gantt charts, and documents that no one is going to read. Let’s establish the right balance of agility and discipline and create a context that enables great project teams to deliver great product. When project management begins to make sense again, we’ll start to see project managers recongnized as they professionals they are or should be.

And by the way… I am happy to go get the pizza and occasionally remove an impediment or two. Do you prefer peperoni or sausage?

Welcome Mike

Let me introduce a new permanent contributor to the site. We were lucky to get Mike Cottmeyer on board.

Mike is a Product Consultant and Agile Evangelist for VersionOne. Prior to joining VersionOne, Mike was a Senior Project Manager for CheckFree Corporation where he led a portfolio of projects for their online banking and bill payment business unit. Mike has a traditional project management background and has worked primarily with agile methodologies for the past four years.

Mike is a certified PMP project manager and a certified ScrumMaster. Mike was involved with the creation of the DSDM Agile Project Leader certification, holds this certification at the Foundation, Practitioner, and Examiner levels. Mike was recently named an honorary member of the DSDM consortium and serves on the board of APLN.

Please, join me in welcoming Mike and take a look at his personal blog Leading Agile as well.

Can unit testing be a waste?

Picture courtesy of Kecko@flickr

It is sometimes hard to start implementing new feature (or fix a bug) by writing an unit test even for a very agile and experienced developer, that’s for sure. But it pays off and it usually turns out to be the easiest and the most efficient way. Besides this, developers have to decide what to test. “You should test everything!” some of you may say, but what does it mean?

Should I test all my JavaBeans (as Java is the language I mostly use I will give examples in this technology)? Should I test all my Data Access Objects? Should I test all toString(), equals(), hashCode(), etc. methods? Or maybe I should test only stuff where I “integrate” all those “low-level” components/classes?

I’ll try to answer the question of what level of abstraction should be tested by your unit tests and why some unit tests may be considered a waste in this post.

Testing everything (literally) is a waste
You should start developing new features from unit tests. More so, you should start by writing tests from the highest possible level of abstraction. I mean that if you want to test adding new users to the database feature you should start from the service class that takes input parameters, creates user object and then stores it using some kind of DAO object into the database. You should not start from testing and implementing User class or UserDao class.

Why? Well, if you start thinking of objects you will need for implementing your feature you may invent some overkill architecture, many unnecessary classes that will eventually be unused. With this approach you will start from implementing User class, then UserDao class and eventually you will implement the service class that will use these both classes and expose some public methods to the outer world.
With this approach, if you are still agile and start from writing an unit test, you should end up with at least three unit tests (one for each class). And assuming that User class is stupid JavaBean with only getters and setters you will have to write unit test for those methods, right?

This is a waste (especially unit testing getters and setters) because you could have only one unit test class with less code and have 100% code coverage.
Agile methodologies focus on eliminating waste – that’s why you should not test everything if you want to be really agile.

The only way is to start from the highest possible level of abstraction and then adding necessary classes just-in-time. And the example follows…

What should you test
I will not give you an example of adding users to the database but instead an example of Struts2 action that retrieves announcements from the database and creates RssChannel object (that is processed by the View layer later on and returned as an application/rss+xml content type that is readable by the RSS readers). Take a look at the attached RssAction.java file. This action uses other classes like:

  • AnnouncementDao
  • RssChannel
  • Announcement

Now take a look at the test for this action: RssActionTest.java. At the first sight this unit test tests only com.bielu.annoboard.action.rss.RssAction but in reality I also tests all classes mentioned above.

The effect? Cobertura tells me that:

  • RssAction is covered at 100%
  • RssChannelHelper is covered at 92%
  • RssChannel is covered at 95%
  • AnnouncementDaoIml (which is an implementation of the AnnouncementDao) is covered at 100%
  • Announcement is covered at 100%

Voila! No waste and everything (or almost everything) is tested!

Use your common sense
As always there are some pitfalls lurking – my previous opinions are not always true. For sure you should NOT test getters and setters in your JavaBeans. But what if you implement for example toString(), equals() or compareTo() method? Should you test them or not?

I generally test classes from the highest possible abstraction level but I sometimes also test lower-level classes. Here are some pieces of advice you should consider following:

  • it is a very good idea to test toString() method
  • it is an absolute MUST to test (to death) equals() and hashCode() methods and contract relations between them (refer to “Effective Java” book) – if you use Sets, Maps and collections in general this is a critical part – I even remember the very small bug there in one of my previous projects and two seasoned Java developers looking for the problem for couple of hours – this is my coding horror
  • it is a good idea to also test compareTo() and contract relations to equals() method
  • check with your code coverage tool what is still not tested after the high-level tests
  • consider removing the code that is not tested and see whether it was used at all (see this post)
  • if it is necessary but not tested then test it from the lower level

Use these pieces of advice as well as your commons sense wisely and remember to eliminate waste every time you can (yes! writing unnecessary unit tests is a waste).

I’m really curious about your opinions about writing unnecessary unit tests. Do you think writing unnecessary tests can be even dangerous? Do you consider unnecessary and overlapping unit tests a waste? Share your opinions here.

User Stories: Three Steps to Better Presentations

Slide 'Waterfall' from Scrum Presentation

As a Scrum Coach, I often take on the role of Evangelist. Monday afternoon, I explained Scrum to the Swiss Java User Group[1]. Although not my first such talk, I had to completely rewrite the presentation. When this was 80% done, the realization hit me: How will I know if the “users” are getting what they need? Why am I not writing user stories?

Why didn’t I think of this sooner?

The process is simple: identify the users, figure out what they want to accomplish and why. Give them what they need to accomplish their most important goals. If you can meet their needs even before they are even aware of them, magic happens.

Step 1: Who is coming to the event?

First I brainstormed about who might come. Then I shortened the list based on the probable attendees:

  • Java Developers and Project Managers — The majority.
  • The Speaker
  • Early adopters — the people who will go back to the office and say ‘let’s do Scrum’
  • Skeptics and happy users of other methods — who will say, “what we’re doing now is good.’
  • Human beings with needs beyond software — i.e. everyone!

Who got optimized out? I felt there wouldn’t be enough managers or potential product owners present to justify emphasizing their stories. Various categories of people with potentially adversarial tendencies got consolidated into “Skeptics.”

Step 2: What do these people want, and why?

The next step was to figure out what these people want and why. Their goals are a good place to start. For example, the early adopter: “I am looking for interesting new ideas to try out.”

So I went through the list of people and thought about what each one wanted. I generated a lot of stories. For instance, “As a software developer, I want to know how my job will be impacted, so I can decide whether Scrum is good for me.” That one was not so obvious at the beginning. I added a slide to answer this question.

Step 3: Consolidate and Prioritize

The talk ran from 17:30 to 18:30, plus an hour for discussion. Of course, I had more stories than time, so I had to shorten the list:

  • As a project manager or software developer, I want to

    • understand the Scrum process
    • assess the Scrum’s impact on my job
  • As a project manager, I want to understand Scrum planning and control
  • As an early adopter I want
    • to know what is special about Scrum
    • credible road maps for introducing Scrum
  • As a skeptic, I want to be treated with respect
  • As a Java developer, I want to hear examples based on Java Projects
  • As the Speaker, I want good feedback about the talk.
  • As a human being, I’d like a brownie before we start the talk.

Any surprises here? Actually, most of the stories were unexpected because I had only thought seriously about potential Scrum Masters! Early Adopters are the target audience. Skeptics are potential adversaries, not to be antagonized! Developers might be worried about the impact on their careers. Maybe those brownies will help people maintain their attention to the end.

Although this analysis started out as an example for one slide, it caused many changes to the presentation. I created a new feedback form (based on the Kano model), and ordered brownies for everybody.
What stories didn’t make the cut? There are no slides about scaling Scrum or Java examples. Topics of interest for the Product Owner got held to a minimum, and the story on “assessing the impact of Scrum on my career” got deferred to the discussion round.

The Power of User Stories

User stories aren’t just for software! User stories are about products: something you make or do for other people. The better you can get inside your user’s heads and recognize their true needs, the better the acceptance of your products. Try it the next time you have to give a talk, write some documentation, or even a request for proposals. The results may surprise you!

Notes

[1] The presentation is available online.

Upgr-r-rade

If you are seeing this message we have just upgraded AgileSoftwareDevelopment.com. Not all the functionality has been ported to the new version of the platform yet (some might even no be ported at all) and there might be some elves running here and there, but the basic upgrade is done. You want see any immediate benefits, but it means quite a lot on the administration side – with the new version of the platform it will be way easier to customize the site, make it look nicer and more useful.

If you notice some problems with the site, please, comment about them here, tell about them on the forum or via contact form – whatever way is more convenient for you.

Creativity as the Root of Software Development

Creativity
There is no single, authoritative perspective or definition of creativity. At least 60 different definitions of creativity can be found in psychological literature. However, a widespread conception of creativity is that it manifests itself in the production of things that are both original and useful. (from: Wikipedia)

Original
The basic idea of writing software is to produce code that has not been produced before. Techniques like object-orientation, component-based design, service-oriented architecture, and refactoring are all there to help you in making each line of code unique. Ultimately, software developers think that, in a perfect world, each piece of code exists only once. And in searching for this utopia, trying to prevent any repetition of work, software developers have far more possibilities than, say, writers, painters, architects, and hairdressers. None of these other creative people have a similar array of techniques for abstraction and indirection. (In fact, they probably don’t even know what that is.)

Useful
Likewise, producing useful results is another basic idea of writing software. Quite possibly, no other type of creative activity has increased global productivity levels as much as software has done. The business value of software exceeds that of almost any other creative product, by several orders of magnitude. Software developers can hardly be compared to writers, painters, architects, or even hair dressers for that matter. They often don’t even think of themselves as “creative”, with all the wishy-washy connotations that often come along with that term. Most software developers are not of the poem-writing, ballet-dancing kind. They just want to be practical, and make stuff that is used.

So it appears that the terms original and useful touch the very core of software development. And even though intelligence, discipline and routine all play important parts in many software projects, there is nothing to apply them to if it weren’t for creativity.

Note: You don’t have to agree that creative products are often useful, but that doesn’t change the fact that software development is a creative activity. One that just happens to be useful.

The Creative Process
The best-known model for the creative process was created by Graham Wallas and Richard Smith, in their work Art of Thought. The five-step process, consisting of preparation, incubation, intimation, illumination and verification, is just as applicable to software development as to any other creative activity. For example, let’s suppose you are responsible for improving the performance of a web site. The process might go like this:

  1. preparation: Finding out what the location and dimension of the problem is, like the time it takes for (some) queries to execute on the database server;
  2. incubation: Pondering on the performance problem, both consciously and unconsciously, while taking a shower, playing poker and discussing the latest Batman movie with friends (possibly all at the same time);
  3. intimation: Realizing that the solution might have to be found in a better data model, and not, as you thought earlier, in more efficient queries or better hardware;
  4. illumination: Suddenly having the insight that a solution can be realized by splitting table Foobar in tables Foobear and Fooboar, while flattening the tables Lorem and Ipsum into one;
  5. verification: Trying out the new solution, verifying and improving it, until it has the intended results.

This is creativity. People can be seen using a similar process during requirements gathering, analysis & design, construction, maintenance, management, and all the other disciplines of software engineering.

Our Creative Joy
The creative part of software projects is what motivates most software engineers. Frederick P. Brooks, in The Mythical Man-Month, listed five reasons for it:

  1. making stuff: We like making things that we can call our own, ever since we were able to pick up crayon markers and use them on the wallpaper in our bedrooms.
  2. pleasing others: We like to be complimented by others, either directly or indirectly, for the things that we created for them.
  3. solving problems: We like solving puzzles, as an intellectual endeavor, particularly when our solutions are unlike any other’s.
  4. learning things: We like gaining knowledge of problem domains, as knowledge improves our value to ourselves and to others.
  5. handling tools: We like handling tools. And code (being nothing more than digitally written thoughts) is the most tractable and malleable tool we have ever devised.

All About People
According to scientists, complexity –that interesting state between order and chaos– is the root of all creativity, in physics, biology, psychology and beyond. Only that which is complex can be creative. And with creativity being the root of software development, and people being very effective creative systems, it is clear that software development is all about people.

Note: This post is published automatically while I’m reading a book on a beach in Cuba. I will respond to any replies and comments, but it might take a while.

Productivity

Productivity is often confused with velocity. They’re related, but they’re very different things.

Velocity is the amount of work that you/your team can do in an iteration. In economics terms, productivity is defined as “the ratio of what is produced to what is required to produce” (source: Wikipedia). So, a person who can carve ten statues out of a lump of granite is more productive than a person who can only carve eight statues out of an equal size lump of granite.

At this point it’s understood that we can’t measure productivity when it comes to building software because there’s no good, consistent, accurate way to measure output.

However, while we can’t measure our productivity, we can often improve it. Without measurement we won’t know how much of an improvement we’ve gained (or how much we’ve regressed), but we can measure improvements in velocity, which can indicate corresponding productivity improvements.

Productivity is not improved by adding people. You might increase your velocity by adding people, but productivity often suffers when a person is added to a team (at least until the team adjusts and the new member is running at full speed). At best, productivity will remain the same when people are added.

Productivity is improved by simply being more efficient. Automate tasks that are repeated often. Know your tools and the shortcuts (small automations) that they provide. Reduce waste — don’t solve a problem unless it needs to be solved. Identify blocks and remove them as soon as possible.

Agile and XP practices are, for the most part, about increasing efficiency, focus, and productivity. Automation, including continuous integration and automated unit testing, is important. Pair programming is a good way to learn about the tools your team uses. Test-driven development and pair programming help keep the focus on adding business value. Scrum-style stand-up meetings are a good place to bring up blocking issues and identify the best person to help remove blocks.

Weekly Agile Reading is back. Pack 10

I used to publish weekly links on what was published on the web during a week. It happened to be quite difficult. What was sometimes particularly difficult was to figure out whether the article was fresh or not. I decided to continue publishing interesting links related to Agile. I will still try concentrating on fresh articles and publish link sets weekly, but won’t commit to it. Most of the time the links will be published weekly and will concentrate on fresh content, but will also include links to older content that caught my attention during the week.

Top writing that caught my attention this week:

» thekua.com@work " Defensive programming depends on context How to decide on how much defensive your code needs to be.
» Lasse’s weblog – Best quote from Agile 2008 Possibly the best quote from the Agile 2008 conference.
» NOOP.NL: Managing Software Development: Call for Votes: Top 100 Software Engineering Blogs! Please email me some candidates for the upcoming and all-new Top 100 Most Popular Software Engineering Blogs. (Send me the URL of the blogs.
» AGILE IN ACTION: Be honest about what you are then start climbing Dreyfus Model of Skill Acquisition, which describes how people progress through levels of mastery in a subject

You might like having a look at the previous link packs as well.

If you happen to encounter another interesting content published since last Friday, please, post links in the comments.