Peer review is seen as an important quality assurance mechanism in both industrial development and the open source software (OSS) community. The techniques for performing inspections have been well studied in industry;​ in OSS development, peer review practices are less well understood. In contrast to industry, where reviews are typically assigned to specific individuals, in OSS, changes are broadcast to hundreds of potentially interested stakeholders. What is surprising is that this approach works very well, despite concerns that reviews may be ignored, or that discussions will deadlock because too many uninformed stakeholders are involved.

In this work we use a multi-case study methodology to develop a theory of OSS peer review. There are three research stages. In the first stage, we examine the policies of 25 OSS projects to understand the review processes used on successful OSS projects. We also select six projects for further analysis:​ Apache, Subversion, Linux, FreeBSD, KDE, and Gnome. In the second stage, using archival records from the six projects, we construct a series of metrics that produces measures similar to those used in traditional inspection experiments. We measure the frequency of review, the size and complexity of the contribution under review, the level of participation during review, the experience and expertise of the individuals involved in the review, the review interval, and number of issues discussed during review. We create statistical models of the review efficiency, review interval, and effectiveness, the issues discussed during review, to determine which measures have the largest impact on review efficacy. In the third stage, we use grounded theory to analyze 500 instances of peer review and interview ten core developers across the six projects. This approach allows us to understand why developers decide to perform reviews, what happens when reviews are ignored, how developers interact during a review, what happens when too many stakeholders are involved during review, and the effect of project size on the review techniques. Our findings provide insights into the simple, community-wide mechanisms and behaviours that developers use to effectively manage large quantities of reviews and other development discussions.

The primary contribution of this work is a theory of OSS peer review. We find that OSS reviews can be described as (1) early, frequent reviews (2) of small, independent, complete contributions (3) that, despite being asynchronously broadcast to a large group of stakeholders, are reviewed by a small group of self-selected experts (4) resulting in an efficient and effective peer review technique.