Analysis of interrater agreement in ISO/IEC 15504-based software process assessment

The emerging ISO/IEC 15504 standard provides a framework and a model for software process assessment and improvement. There are two requirements for reliable process assessment: internal reliability and external reliability. The objective of the study is to provide an empirical case of external reliability, i.e. the interrater agreement in ISO/IEC 15504-based software process assessment. Interrater agreement implies the extent to which independent assessors agree in their ratings of software process attributes. Our dataset was from two assessments conducted using the ISO/IEC 15504 standard. The results showed "substantial" to "excellent" agreement. This implies that the two assessments acquired external reliability.