This paper describes how a small helium (4He) peak can be separated from a large adjacent deuterium (D2) peak using a modified quadrupole mass spectrometer (QMS). This QMS utilized a condition of the second stability zone (zone II) in the Mathieu diagram. The minimum detectable peak ratio of 4He to D2 was 10-4 when a mass scan line was set close to the upper tip, and was 10-3 when the scan line was adjusted to the lower tip. The ion transmission rates simulating the peaks were calculated for the upper and lower zone conditions. It was found that a longer peak tail occurs when an unstable orbit has mainly a cosh function. From the experimental and theoretical data, it is concluded that the upper zone condition is the most suitable for realizing a high‐resolution peak.