Characterizing Architectural Evaluations and Identifying Quality Attributes addressed in Systems-of-Systems: A Systematic Mapping Study

System-of-Systems (SoS) are composed by heterogeneous constituent systems that are autonomous, independently managed and operated, and geographically distributed. Architectural evaluation is a key activity of all systems design, construction and operation; unfortunately, the interrelationships among SoS constituent systems and quality attributes (QA) make hard to evaluate SoS. Several evaluation techniques have been proposed, but this corpus of knowledge has not yet been organized for easy access. This article describes the design, execution and results of a Systematic Literature Mapping (SLM) of architectural evaluation proposals, focusing on the QA's they address and the techniques they use. The SLM yield 1675 articles, of which 22 were selected for detailed analysis; most of them take a quantitative perspective (82%) and the rest are hybrid quantitative/qualitative (18%); most proposals offer no empirical validation at all (64%); and the most addressed QA's are performance and robustness.