Pair programming. What researches say on the costs and benefits of the practice.

August 19, 2008 by Artem Marchenko

Last week the entry on risks on solo programming attracted a number of comments debating the value and costs of pair programming (PP). My personal observations and discussions do support the claim of better quality and faster development, when pair programming is done the right way and learned under the coaching of experienced mentor. However, these are just my personal observations and I was rarely focused on measuring and "average" team, most of my observations were based on teams who did want to pursue the path of continuous improvement and were interested in learning how to do the practices the right way.

Gathering data

Anyway, I promised to gather a bit of scientific evidence and here it is. This is not an in-depth scientific review, but rather a result of couple of hours spent on browsing Google Scholar and IEEE Xplore and skimming through the top articles related to "pair programming defect rate" and "pair programming quality". Most of the articles happened to be about the students doing pair programming.  Taking into account the industrial context of original discussion, I decided to ignore most of the articles that involved only students. Note, that for purity I was trying to examine solely the effect of pair programming on quality, defect rate and corresponding costs. A synergy of pair programming, test-driven development and continuous integration might deliver much greater benefits.

The results

Somewhat surprisingly for me there is indeed not too much scientifically valid data about the pair programming. I had to tweak and tune the search query in order to dig out at least some thorough case studies. The resulting message happens to be "it depends". There is definitely no evidence on pair programming being a silver bullet of software development, it seems to be quite clear that during the learning period PP can decrease the team performance and even later it is not going to double it. Pair programming looks to be more predictable and most of the time produce slightly better design and deliver less bugs, however the difference is rarely if ever on the order of magnitude.

For me the results can be translated three more or less clear messages:

• If your company considers short time to market being one of its major goals, it is better to support Pair Programming -PP is likely to be worth the slightly higher cost per feature.
• If your company does care much about after-the-release life cycle, it is better to support Pair Programming - lower level of defects will pay off over time.
• Otherwise the effectiveness of Pair Programming depends much on the interest of people practicing it. If you programmers hate it, it's unlikely to be effective, if they love it - it is going to make your SW department more effective, while making it happier.

Raw data

If you are interested in making your own conclusions, here are the links to the most representative [from my point of view] articles together with the summary and/or citations.

• The Costs and Benefits of Pair Programming (pdf)
  They found that for a development-time cost of about 15%, pair programming improves design quality, reduces defects, reduces staffing risk, enhances technical skills, improves team communications and is considered more enjoyable at statistically significant levels.
• A multiple case study on the impact of pair programming on product quality
  Effort spent on PP is highest in the beginning of a project and in the final iteration. Neither programming style (pair or solo) has consistently higher productivity. PP is most useful for learning, and complex tasks. PP results in lower adherance to coding standards. Code produced by PP has higher comment ratio. Conflicting results on the defect density
• Analyzing the cost and benefit of pair programming
  

  The results of the computations provide clear management guidelines when to use Pair Programming or better not. Given that a short time to market is decisive for the success of a project, adding developers to form programmer pairs can speed up the project and increase its business value despite the increased personnel cost.

• Effects of pair programming at the development team level: an experiment
  The PP teams had 29% lower project productivity than the SP teams. However, the reason was the considerably larger effort they spent for the first three or four use cases. The inefficiency was probably caused by the learning time involved in getting familiar with new people and with the pair programming practice. Later in the projects the PP teams spent 5% less effort than the SP teams for implementing the use cases. If the inefficient learning time is not taken into account, the productivity of the PP teams seems to be equal to that of the SP teams.
  

  The code written by pair programmers contained 8% less defects per use case when the responsible developers considered the code ready. However, the SP teams were much more successful in finding and fixing the defects, and in the end of the project they delivered systems with a lower number of defects per use case. This indicates that pair programmers write code with fewer defects, but this benefit may be lost unless careful system testing is performed.

  The PP teams had slightly better design quality based on the method size and complexity metric

• Experimental Evaluation of Pair Programming (pdf)
  Student-based experiment: Pair programming is more predictable than individual one with regard to development time and program size.

• Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise
  Short experiment, but on many real programmers from the industry: The results of this experiment do not support the hypotheses that pair programming in general reduces the time required to solve the tasks correctly or increases the proportion of correct solutions. On the other hand, there is a significant 84 percent increase in effort to perform the tasks correctly. However, on the more complex system, the pair programmers had a 48 percent increase in the proportion of correct solutions but no significant differences in the time taken to solve the tasks correctly. For the simpler system, there was a 20 percent decrease in time taken but no significant differences in correctness.

Your experience

How does it correlate with your observations? Do you find pair programming more effective for the whole business? Or do you find that the need for pairing irritates most of the teams too much and is not worth the investment?

Permalink