A novel hybrid finite-element method for the analysis of leaky-waveguide structures is presented. The possibly radiating structure is enclosed within a fictitious circular contour-C in order to truncate the infinite solution domain. The field in the unbounded domain, outside contour-C, is expressed as a superposition of transverse electric and magnetic modes. Their radial dependence is expressed in terms of Hankel functions, which satisfy the radiation condition at infinity. The bounded area is discretized using hybrid node/edge elements for an accurate and efficient handling of the electric field vector wave equation. The transparency of the fictitious contour is ensured by enforcing the field continuity conditions according to the principles of a vector Dirichlet-to-Neumann mapping. The whole procedure yields a nonlinear eigenvalue problem for the complex axial propagation constant (beta). The nonlinearity is due to the appearance of beta within the argument of the Hankel functions. The final nonlinear problem is solved by employing a matrix Regula-Falsi algorithm. Initial guesses for the Regula-Falsi algorithm and a fast estimation of the eigenvalues spectrum are provided by a linear formulation based on a second-order approximation (beta/ko)2 Lt 1. The proposed method is validated against published numerical and experimental results for both leaky and surface wave modes.