Experiences from performing software quality evaluations via combining benchmark-based metrics analysis, software visualization, and expert assessment

Software quality assessments are critical in organizations where the software has been produced by external vendors, or when the development and maintenance of a software product has been outsourced to external parties. These assessments are typically challenging because is not always possible to access the original developers (or sometimes is not even allowed), and in rare cases suppliers keep an account of the costs associated to code changes or defect fixes. In those situations, one is left with the artifacts (e.g., database, source code, and documentation) as the only sources of evidence for performing such evaluations. A major challenge is also to provide fact-based conclusions for supporting decision-making, instead of subjective interpretations based on expert assessments (an approach still very predominant in mainstream industrial practice). This paper describes an instance of a software quality evaluation process performed for an international logistics company, which combined: benchmark-based metrics threshold analysis, software visualization, and expert assessment. An interview was carried out afterwards with a member from the business division of the company, to assess the usefulness of the methodology and corresponding findings, and to explore avenues for future improvement.