Micropillar cavities are potential candidates for high-efficiency single-photon sources and are testbeds for cavity quantum electrodynamics experiments. In both applications a high quality (Q) factor is desired. It was recently shown that the Q of high-Q semiconductor micropillar cavities exhibit pronounced quasi-periodic variations in the regime from 1 to 4 μm, and a detailed understanding of the variational behavior of the Q is required. Here, we study the origin of these variations using a multi-mode Fabry-Perot model appropriate for this regime. We analyze in detail contributions to the effective reflectivity of the fundamental mode arising from coupling to scattering channels involving higher-order cavity modes and propagating Bloch modes in the distributed Bragg reflectors (DBRs). We show how these weak contributions lead to strong variations of the Q factor, and we relate the average periodicity of these variations to the thickness of the DBRs and the derivative of the effective indices of the guided Bloch modes. We also examine the influence of various geometrical parameters, including the number of DBR layers pairs, the amplitude of the corrugation of the pillar sidewalls and the number of etched layer pairs in the bottom DBR on the Q versus diameter relation. Comparisons are made between extensive numerical simulations and experimental measurements, and a good qualitative agreement is found.