We present a formalism for analyzing laser resonators which possess nonplanar mirrors and lateral waveguiding [e.g., an unstable resonator semiconductor laser (URSL)]. The electric field is expanded in lateral modes of the complex-index waveguide and is required to reproduce itself after, one roundtrip of the cavity. We show how the waveguide modes, their gain and loss, and hence the criterion for truncation of the infinite set of modes can be derived from the Green's function of the one-dimensional eigenvalue equation for the waveguide. Examples are presented for three cases of interest-a purely gain-guided URSL, an index-guided URSL, and a gain-guided tilted-mirror resonator. We compare theoretical calculations to previous experiments.