The article describes an optimization algorithm based on coupling SNOPT to finite element analysis for optimization of the insulation layers in a model generator. An electromagnetic and thermal study of the core, winding, and insulation of stator slots is performed with the main idea of finding the optimum shape and characteristics of the insulation layers to minimize electric stress with respect to the thermal limitations of the materials. For this purpose a novel method of coupled SNOPT-FEM analysis is used for the optimization of insulation layers in a generator. The optimization algorithm in this article builds on the wellknown SNOPT packag, which is a general purpose system for large-scale, nonlinearly constrained optimization. For optimization involving electric fields and geometric properties, such as shape optimization, SNOPT can be connected to FEM to perform optimization based on a finite element model. According to this method, the geometry of the insulation layers is optimized, and optimized values for the material properties are obtained with respect to electrothermal limitations to investigate the possibility of improving the HV generator characteristics. Following a short review of the stator winding insulation system components, the developed optimization algorithm for SNOPT and FEM analysis will be given along with the results of a model generator.