For a large scale cloud cluster, providing data access which crosses through high performance computing servers, network and storage servers, and scheduling resources among multi-tenants are challenging jobs, especially with specific level of quality of service (QoS) requirements. Besides the criteria of current availability and reliability in service level agreement (SLA), throughput and latency are also critical QoS performance measurements to the tenants. Any QoS solution, unfortunately, cannot be achieved without introducing additional overhead. A good QoS design should always be cost effective. This paper proposes a cost-effective QoS infrastructure over a distributed storage system, providing work-conserving differentiated QoS for multi-tenants cloud storage. The proposed QoS infrastructure combines global service differentiation with adaptive share allocation to support differentiated QoS and achieves both service isolation among users from the same class and service differentiation among users from different classes.