Automated testing in agile software development

The biggest difference of agile methods from traditional waterfall is the short feedback loop. The whole concept of agility in essence is no more, than “build the most important piece, evaluate, adjust, and repeat”. Automated tests in the agile methods serve as a very important tool for shortening the feedback loop.

In the most of traditional processes automated tests are mimicking the manual test procedures. The tests are often written not by the code developers. Tests usually involve testing the functionality of the whole system. Test results come to the original developers quite late and tests don’t take into account the latest code changes. Automating functional and system tests is a good thing. It makes testing a little easier, more effective and saves some money for the company. However, it is no more, than automating a small piece of the manual labor.

In agile methods the majority of the automated tests consists of unit tests that verify the smallest possible pieces of software and can be executed very quickly. It makes it possible to execute the test set many times a day or even many times an hour and shortens the feedback loop even more.

Agile Software Development on the Xmas edition of the Carnival of Agilists

The series of Kano-model related posts published on this web-site has been noticed and referenced in the Christmas 2006 edition of the Carnival of Agilists – the blogroll about latest thoughts in the agile community.

What a nice Christmas present!

On a side note:
Even though at the moment I am the only one posting to, everybody can register and post his own thoughts here. Every registered user automatically is granted the own blog space and an opportunity to be published on this site front page.

Importance of following the plan

Any plan is in essence a sequence of actions to be performed in order to achieve a particular goal. Plan acts as some kind of a contract that specifies the agreed way of reaching this goal.

Keeping the plan can be both harmful and useful under different conditions.

In waterfall-based processes people try to specify to death everything before the actual development starts. Plan creation takes huge amounts of time and efforts. Replanning would require similar amount of work. It also might introduce the half-completed features since in waterfall there are no completed features until the very end of the project. No surprise, that waterfall-based processes avoid plan updates as much as possible.

While keeping the plan is very useful in mass manufacturing, in software development too many things can change during the product development. Customer requirements, environment, technologies and even customers themselves do change during the project. Therefore preserving the original plan can very well lead to the useless product that perfectly conforms to the original specification.

Agile software development processes tend to use the easily updatable, short and flexible plans. Ease of the plan updates brings the following benefits:

  • It is easier to respond to the requirements change
  • It is easier to respond to the customer or own company business related changes
  • Early feedback can efficiently influence the plan and lead to the product the customers really need
  • The development team builds the best possible product according to the current understanding. Whenever more information is available, it is easier to adjust the plans accordingly

How important is to keep the plan in your company? How long are your plans and how easy is it to adjust them?

Power of limited forecasting

Most if not all of the agile software development processes are based on the assumption that it is hardly possible or not possible at all to predict the real requirements and real workload size at the beginning on the project. The whole idea of agility is to try implementing the most critical stuff, see how it goes, get customer feedback and adjust the close plans.

Such limitation of the upfront planning minimizes the amount of time wasted on never used overplanning and on developing features, that after later consideration happen to be wrongly understood or unnecessary at all. Also the detailed plan non-existence welcomes changes easier. If there are no planning results to modify, there is no temptation to just follow the obsolete plans.

Unfortunately very limited amount of upfront planning also minimizes the forecasting possibilities. Without the longer time estimations it is not possible to commit to multi-million and multi-year contracts. In most of cases the waterfall’s ability to predict the length and costs of the project is anyway an illusion, but nevertheless long time project estimations are required before committing to any non-in-house project.

There are two major ways to overcome the prediction obstacle:

  • Build trust with your customer so that he trusted in your ability and commitment to deliver results as soon as possible and as well as possible
  • Utilize the agile process’s power not only to adjust to changing requirements, but also to deliver functionality and improved estimations fast. Split the big project into several parts and contract delivery of each of those separately. In fact you don’t even need to tell the customer about the very first phase – behind the scenes you could run the first spikes during the time that waterfall guys would spend on the initial requirements analysis.

Measuring time at work

One of the simplest metrics that some managers are actually paying attention to is time at work, i.e. how much time or overtime each programmer spends in front of his workstation. This simplest metric might work for manufacturing-like tasks: when worker spends 50% more time at the production line, he is likely to produce 50% more goods (those typically he wouldn’t be happy about it). However, the “more time – more goods” approach doesn’t work for complex, intellectually intensive tasks like programming. Everyday every software developer has to write the code that never existed before. It is not just pushing the same buttons as yesterday; a programmer has to literally invent a new thing everyday. And invention needs a fresh mind.

Overtime flaws

Forcing a programmer to work hard when he is tired forces him to:

  • do less thinking. Just do something not really needed to spend time;
  • use simplest possible algorithms instead of the best ones;
  • not to look for reusing the legacy or third party code;
  • perform unnecessary polishing of the already implemented features, just not to start thinking about the new features;
  • spend time on useless web-surfing.

In short, in software industry “tired programmer” is almost equal to “less thinking, more bugs” and missed bugs might cost a fortune to remove at later phases of the project. Especially if they have to be removed under the similar overtime pressure.

Results vs laziness

Often managers watch the time at work only to track laziness and react on it somehow. Unfortunately in software industry hard work and overtimes are closely related to productivity. Even vise versa, a lot of very good programmers can be peak productive only few hrs a day. Those who want to manage the software project have to learn measuring the work results instead of participation rates. Fortunately close customer-developers feedback loop promoted by all the agile methods is exactly aimed at presenting the work results in the customer-understandable form.

Does your manager track your work time? And if he does, does it have positive or negative effect on the team results?

Measures of size

Measurement is fundamental to any engineering discipline and the planning of a software creation work is no different. Whenever you plan to make or renew a piece of software the most important metric as the workload size. Measure of size is important because knowing the amount of work to do and the team skills (i.e. velocity) you can easily derive the work costs and duration. Agile software development methods do strive against the overplanning and overdetalization.

However, being agile doesn’t mean having no clue on when the work is going to be completed. Vise versa most if not all of the agile methods continuously provide team the current “best guess” on the remaining workload and possibly even options to cut some corners.

Traditional measures of size

Lines of code


  • Simple. There can hardly be a simpler measure, than counting lines of code in the program.


  • This kind of measurements prohibits refactoring and optimization. The very nature of refactoring is not changing the code functionality while attempting to make it more manageable and understandable. In many cases it means making the amount of code lines smaller. Not too many programmers will like doing work that makes the total amount of work done look smaller. In fact measuring the workload in lines of code can even encourage the ineffective coding: why use cycles, hashing of calculations if hard coding produces more lines of code and the programmer is paid per line.

Function points


  • Function points are a quite reliable metric based on the number of inputs, outputs and the perceived algorithm complexity.


  • Complexity. There are not too many function point experts out there and it is not always easy to explain how exactly the metric has been derived
  • Function points are easy to use only AFTER the end of the project where there is not that big use of them. Function points computation is based on such a simple artifacts as number of inputs, files, user inquires, etc. Unfortunately we are living in a world of constantly changing requirements and at the beginning of the project it is quite impossible to reliably estimate all the data required for the function points calculation.

Agile measures of size

Ideal days

Ideal days are the imaginable amount of time that would be spent on the project if there were no interruptions (including email checking), everybody was working full time at full speed without any vacations and everything needed was present from the day one.


  • The concept of ideal time is relatively simple to explain both to developers and their bosses. It is easy to provide the real world example of a basketball game that contains 40 minutes of ideal time, while lasting for 3+ hours of elapsed time


  • Since both elapsed and ideal times are measured in the same units, they tend to mix
  • Not everybody is eager to accept that in a real world week there can be only two-three full ideal work days
  • Since team skills and the understanding of the subject area raise over time, ideal day estimations tend to decay and have to be readjusted often
  • Ideal days can be difficult to sum. My ideal days are not equal to yours

User story points

Story points come from extreme programming and propose measuring the workload in imaginable units of complexity. Team assumes one of the relatively simple tasks to be of size 1 or 2 and estimates the size of another tasks by comparing them to the chosen standard.


  • Estimations don’t decay over time like ideal days do. When team skills go up, its velocity changes, not the work size
  • Relative values are what’s important. It is easier to reach an agreement on relative sizes, than on a real world related artifacts like days
  • Pure measurements of work size – easy to sum.


  • More difficult to explain. The benefits of using unitless measurements and the power of relative estimations have to be explained

What kind of work size measurements does your team use? What is the most useful metric to you?

Kano questionnaire survey analysis

Different potential user groups and even users within the same group can have different opinions on the product features. The only way to see the high-level picture is to question many users and to gather some statistics on the user preferences. If the user opinions are gathered with the help of Kano questionnaires, the answers can be analyzed in a simple table. For example, for some web-based word processor features, the user survey results could be as follows:

Feature M L E R Q I Category
Spellchecker 52.7 22.6 8.3 2.5 1.5 12.4 Mandatory
Can post to blogs 38.2 8.2 33.4 1.1 0.8 18.3 Mandatory
Supports many fonts 22.3 44.6 13.6 0.5 1.2 17.8 Linear

From this example it is evident, that:
1. The spellchecker feature is a mandatory one. Users might not applaud too much to it, but they would be really disappointed if it is missing
2. On the other hand support for many fonts appeared to be a linear feature. Users will like if there are many fonts supported, but only few are mandatory
3. The blogposting feature has two clear peaks. It is a sign of the fact that different user groups have different opinions about the feature. In this example, it might happen that one target group wants their current desktop word processor to be replaced with the web-based one, and another group just wants a tool for quick blogging from anywhere

Feature Categorization and Customer Satisfaction series:

Kano questionnaires

One of the simplest ways of dividing the product features into must-haves, linear, exciters and even wrong features is to gather the potential customer’s opinion with the help of the Kano questionnaires ( PDF ). Kano questionnaires include two questions for the every feature or group of features: the functional question “How do you feel if this feature is present?” and dysfunctional question “How do you feel if this feature is NOT present?”

The customer has to choose one of the five possible options for the answers:
1. I expect it to be this way
2. I like it that way
3. I am neutral
4. I can live with it this way
5. I dislike it this way

Asking both functional and dysfunctional questions helps to go beyond the simple “priorities”. For example if the potential user expects some feature to be present, but can live without the feature, it is not really a mandatory feature.

The quickest way to assess the questionnaires is to map them to the following table.

Dysfunctional question
Like Expect Neutral Live with Dislike
Functional question Like Q E E E L
Expect R I I I M
Neutral R I I I M
Live with R I I I M
Dislike R R R R Q

M – Must have
L – Linear
E – Exciter
R – Reverse, i.e. wrong features, that would make the user experience worse
Q – Questionable, i.e. the potential user answers are inconsistent
I – Indifferent, i.e. the potential user doesn’t really care about the feature

For example, if a user tells that he can live with a word processor with a spellchecker, but would dislike if a word processor didn’t have a spellchecker, than a spellchecker is a mandatory feature for the word processor.

Feature Categorization and Customer Satisfaction series:

Kano model of customer satisfaction

According to the Standish group research on average 45% of a software features are never used and only 20% of features are used always or often. It means that on average you could develop two times simpler product and sell it for the same price. Potentially gains can be even bigger, for the enterprise scale systems two times simpler product often means four times shorter schedules and ten times simpler integration and testing.

There is no other way to discover what features would be actually used, than to ask the real or at least potential customers, preferably after letting them to try some features live. One of the tools for aiding the feature categorization is the Kano model of customer satisfaction.

The Kano model separates features into three broad categories:
1. Must-have, mandatory features.
It is necessary to develop these features just to enter the market. However, once the basic requirement is satisfied, customer almost doesn’t care about the further improvement of the feature.
For example, the modern web processor has to contains the spell-checker feature to be sell. However, most customers won’t even notice if the spell-checker contains 50 or 200 thousands words

2. Linear or performance features
The better this kind of features is developed, the more customer is satisfied.
For example, the more storage space a file storage server provides, the more customers would be satisfied

3. Excitement or “wow” features
Customer won’t be disappointed too much if these features are missing. However, the couple of exciter features can provide great customer satisfaction often adding a price premium to a product.
For example, a built-in e-mail client isn’t an indispensable feature for a web browser, but it can really excite some people

Must-have features are required for a product to enter the market and therefore have to be developed as soon as possible. After must-have features are implemented, as many linear features as possible should be implemented, because each of these features increases the customer satisfaction (unless there are already too many of those). Exciter features are to be approached the last, but keep in mind that a couple of exciters can contribute very well to the viral marketing of a product.

What kind of categorization do you use for your products? How often do you re-prioritize your feature list?

Feature Categorization and Customer Satisfaction series: