Modeling Impact of Human Errors on the Data Unavailability and Data Loss of Storage Systems

Data storage systems (DSSs) and their availability play a crucial role in contemporary datacenters. Despite using mechanisms such as automatic failover in datacenters, the role of human agents and consequently their destructive errors is inevitable. Due to very large number of disk drives used in exascale datacenters and their high failure rates, the disk subsystem in storage systems has become a major source of data unavailability (DU) and data loss (DL) initiated by human errors. In this paper, we investigate the effect of incorrect disk replacement service (IDRS) on the availability and reliability of DSSs. To this end, we analyze the consequences of IDRS in a disk array, and conduct Monte Carlo simulations to evaluate DU and DL during mission time. The proposed modeling framework can cope with different storage array configurations and data object survivability, representing the effect of system-level redundancies such as remote backups and mirrors. In the proposed framework, the model parameters are obtained from industrial and scientific reports alongside field data, which have been extracted from a datacenter operating with 70 storage racks. The results show that ignoring the impact of IDRS leads to unavailability underestimation by up to three orders of magnitude. Moreover, our study suggests that by considering the effect of human errors, the conventional beliefs about the dependability of different redundant array of independent disks (RAID) mechanisms should be revised. The results show that $\text{RAID}1$ can result in lower availability compared to $\text{RAID}5$ in the presence of human errors. The results also show that employing automatic fail-over policy (using hot spare disks) can reduce the drastic impacts of human errors by two orders of magnitude.