To provide a more reliable and low power consumption rotor support system for a high-speed compressor, a novel and compact magnetic bearing and damping system is proposed. It is composed of a cup-shaped rotor, an axial passive bearing of reluctance type outside the rotor cup, and a passive damper with Halbach magnet array inside the rotor cup, which not only can provide adequate stiffness, but also good damping ability for the rotor. To enhance the load and stability of the system, an analytical model, which can calculate the system stiffness and damping coefficient with significantly less computing time than the finite element method (FEM), is established. The effects of some important structure parameters on the system performance are discussed. Then, an optimal design method is proposed based on the above analysis. At last, an experimental setup has been developed and both static and dynamic performances have been tested. The experimental results show that the proposed system is capable of producing high axial stiffness and effective in suppressing the rotor vibration. The results also verify the validity of the analysis and design method.