A vectorH-field finite-element method has been used for the solution of optical waveguide problems. The permittivity of the guiding structures can be an arbitrarily tensor, only limited to being lossless. To extend the domain of the field representation, infinite elements have been introduced. To eliminate spurious solutions and to improve eigenvectors, a penalty function method has been introduced. To show the validity and usefulness of this formulation, computed results are illustrated for step channel waveguide, diffused channel waveguide, anisotropic channel waveguide, and channel waveguide directional couplers.