The optical trapping and manipulation of magnetic holes (MHs) dispersed in a magnetic fluid is systematically investigated. It is found that the gradient force, which tends to attract MHs to the beam center, can be completely counteracted by the repulsive force between MHs induced by a magnetic field. As a result, a depletion region is created at the laser beam spot for a sufficiently strong magnetic field. This phenomenon can be easily observed for large MHs with a diameter of 11 μm. However, it does not appear for MHs with a smaller diameter of 4.3 μm. It is revealed that the enhancement in the concentration of magnetic nanoparticles in the laser spot region as well as the clustering of these nanoparticles leads to a much stronger interaction between MHs when a magnetic field is applied. Consequently, the magnetic field strength necessary to create the depletion region is significantly reduced. We also find that the trapping behavior of MHs depends strongly on the thickness of the sample cells. For thin sample cells in which only one layer (or a two-dimensional distribution) of MHs is allowed, we can observe the creation of depletion region. In sharp contrast, MHs can be stably trapped at the center of the laser beam in thick sample cells even if a strong magnetic field is imposed. This phenomenon can be explained by the existence of a gradient in magnetic field strength along the direction perpendicular to the sample cells. Apart from individual MHs, we also investigate the movement of MH chains under the scattering force of the laser beam. It is observed that MH chains always move along the direction parallel to the magnetic field. This behavior can be easily understood when the anisotropy in viscosity caused by the applied magnetic field is considered.