Since Lim's paper (see IEEE Trans. Circuits Syst., vol.33, p. 357-364, Apr. 1986) on the frequency-response-masking (FRM) technique for the design of finite-impulse response digital filters with very small transition widths, the analysis and design of FRM filters has been a subject of study. In this paper, a new optimization technique for the design of various FRM filters is proposed. Central to the new design method is a sequence of linear updates for the design variables, with each update carried out by semidefinite programming. Algorithmic details for the design of basic and multistage FRM filters are presented to show that the proposed method offers a unified design framework for a variety of FRM filters. Design simulations are included to illustrate the proposed algorithms and to evaluate the design performance in comparison with that of several existing methods.