Fermilab's multi-petabyte scalable mass storage system

Fermilab provides a multi-petabyte scale mass storage system for high energy physics (HEP) experiments and other scientific endeavors. We describe the scalability aspects of the hardware and software architecture that were designed into the mass storage system to permit us to scale to multiple petabytes of storage capacity, manage tens of terabytes per day in data transfers, support hundreds of users, and maintain data integrity. We discuss in detail how we scale the system over time to meet the ever-increasing needs of the scientific community, and relate our experiences with many of the technical and economic issues related to scaling the system. Since the 2003 MSST conference, the experiments at Fermilab have generated more than 1.9 PB of additional data. We present results on how this system has scaled and performed for the Fermilab CDF and D0 Run II experiments as well as other HEP experiments and scientific endeavors.