Pair programming. What researches say on the costs and benefits of the practice.

Photo courtesy of BrianScott@flickr

Last week the entry on risks on solo programming attracted a number of comments debating the value and costs of pair programming (PP). My personal observations and discussions do support the claim of better quality and faster development, when pair programming is done the right way and learned under the coaching of experienced mentor. However, these are just my personal observations and I was rarely focused on measuring and "average" team, most of my observations were based on teams who did want to pursue the path of continuous improvement and were interested in learning how to do the practices the right way.

Gathering data

Anyway, I promised to gather a bit of scientific evidence and here it is. This is not an in-depth scientific review, but rather a result of couple of hours spent on browsing Google Scholar and IEEE Xplore and skimming through the top articles related to "pair programming defect rate" and "pair programming quality". Most of the articles happened to be about the students doing pair programming.  Taking into account the industrial context of original discussion, I decided to ignore most of the articles that involved only students. Note, that for purity I was trying to examine solely the effect of pair programming on quality, defect rate and corresponding costs. A synergy of pair programming, test-driven development and continuous integration might deliver much greater benefits.

The results

Somewhat surprisingly for me there is indeed not too much scientifically valid data about the pair programming. I had to tweak and tune the search query in order to dig out at least some thorough case studies. The resulting message happens to be "it depends". There is definitely no evidence on pair programming being a silver bullet of software development, it seems to be quite clear that during the learning period PP can decrease the team performance and even later it is not going to double it. Pair programming looks to be more predictable and most of the time produce slightly better design and deliver less bugs, however the difference is rarely if ever on the order of magnitude.

For me the results can be translated three more or less clear messages:

  • If your company considers short time to market being one of its major goals, it is better to support Pair Programming -PP is likely to be worth the slightly higher cost per feature.
  • If your company does care much about after-the-release life cycle, it is better to support Pair Programming – lower level of defects will pay off over time.
  • Otherwise the effectiveness of Pair Programming depends much on the interest of people practicing it. If you programmers hate it, it’s unlikely to be effective, if they love it – it is going to make your SW department more effective, while making it happier.

Raw data

If you are interested in making your own conclusions, here are the links to the most representative [from my point of view] articles together with the summary and/or citations.

  • The Costs and Benefits of Pair Programming (pdf)
    They found that for a development-time cost of about 15%, pair programming improves design quality, reduces defects, reduces staffing risk, enhances technical skills, improves team communications and is considered more enjoyable at statistically significant levels.
  • A multiple case study on the impact of pair programming on product quality
    Effort spent on PP is highest in the beginning of a project and in the final iteration. Neither programming style (pair or solo) has consistently higher productivity. PP is most useful for learning, and complex tasks. PP results in lower adherance to coding standards. Code produced by PP has higher comment ratio. Conflicting results on the defect density
  • Analyzing the cost and benefit of pair programming

    The results of the computations provide clear management guidelines when to use Pair Programming or better not. Given that a short time to market is decisive for the success of a project, adding developers to form programmer pairs can speed up the project and increase its business value despite the increased personnel cost.

  • Effects of pair programming at the development team level: an experiment
    The PP teams had 29% lower project productivity than the SP teams. However, the reason was the considerably larger effort they spent for the first three or four use cases. The inefficiency was probably caused by the learning time involved in getting familiar with new people and with the pair programming practice. Later in the projects the PP teams spent 5% less effort than the SP teams for implementing the use cases. If the inefficient learning time is not taken into account, the productivity of the PP teams seems to be equal to that of the SP teams.

    The code written by pair programmers contained 8% less defects per use case when the responsible developers considered the code ready. However, the SP teams were much more successful in finding and fixing the defects, and in the end of the project they delivered systems with a lower number of defects per use case. This indicates that pair programmers write code with fewer defects, but this benefit may be lost unless careful system testing is performed.

    The PP teams had slightly better design quality based on the method size and complexity metric

  • Experimental Evaluation of Pair Programming (pdf)
    Student-based experiment: Pair programming is more predictable than individual one with regard to development time and program size.

  • Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise
    Short experiment, but on many real programmers from the industry: The results of this experiment do not support the hypotheses that pair programming in general reduces the time required to solve the tasks correctly or increases the proportion of correct solutions. On the other hand, there is a significant 84 percent increase in effort to perform the tasks correctly. However, on the more complex system, the pair programmers had a 48 percent increase in the proportion of correct solutions but no significant differences in the time taken to solve the tasks correctly. For the simpler system, there was a 20 percent decrease in time taken but no significant differences in correctness.

Your experience

How does it correlate with your observations? Do you find pair programming more effective for the whole business? Or do you find that the need for pairing irritates most of the teams too much and is not worth the investment?

8 thoughts on “Pair programming. What researches say on the costs and benefits of the practice.”

  1. From my experience I know that PP boosts productivity very much when you do it for solving complex problems where one of the pair is kind of expert in some domain (e.g. SQL) and the second guy has some ideas but is not a domain expert (doesn’t know how to implement those ideas). When you want to spread knowledge about implementation details of your project and be sure everyone can fix every possible bug and implement every possible extension PP works also great. Passing tacit knowledge is very important in complex software projects – I see no other possibility here – even great Wiki docs written by the team members couldn’t help here. PP is the best.

    On the other hand I noticed that PP doesn’t work in every situation. For simple problems and semi-repeatable work like implementing yet another report, or plugin that does not require any kind of rocket-science you can work alone. Everyone else already knows how to write plugins.

    The best thing to do (as always) is to leave decision what is going to be done in pairs to the team. They know better!

  2. I agree that simple, semi-repeatable work *can* be done alone.

    I *have* seen benefits from pair programming for such work, too, though. Mostly because of two reasons: a pair helps spotting oversights that are quite common to me when doing grunt work, and he helps spotting opportunities for automation.

    Having said that, I also agree that the final decision has to be with the team. Still, a team should also be encouraged to experiment, so that they *really* do know better, and not just imagine they do.

  3. Robert Martin’s keynote at Agile2008 and your previous post on pairing brought the topic to the front of my mind recently and it’s nice to see some real data on the subject. It’s nice to see a trend of unreduced productivity with better defect rates across most of the studies. Definitely a lot of food for thought and further reading here.

  4. I have to say I found your posting to be horribly biased and completely non-scientific.

    You obviously think Pair Programming is the way to go and while I agree with you I think they way you tried to influence your results is wrong.

  5. That’s definitely not a full scale good scientific research (that might be a good article BTW if anybody is willing to do a rigorous literature review), however, I was trying to be as objective as I could within a couple of hours of research.

    Indeed, I do think the Pair Programming is more often benefitial, than not, however to me the results don’t look like supporting my point of view much. Quite vice versa, to me they are more like “it really depends, not’ institute PP just because you think it’s good”.

Leave a Reply

Your email address will not be published. Required fields are marked *