A prototype broadband electromagnetic (EM) sensor, GEM-5, has been built and tested as a possible new probe for the future Mars rover to seek an ice-bonded layer at a given depth below the Martian surface. The sensor, with a vertical coaxial coil configuration, will measure the terrain resistivity and susceptibility to determine lateral variations in resistivity and magnetic susceptibility. The lateral variations will indicate regions of resistivity/susceptibility anomalies that may contain ice or water at depth. The forward solution for the sensor geometry over a layered formation and inverse algorithms to convert the EM data into the apparent susceptibility and resistivity are developed to investigate the ability of the sensor in detecting and resolving a buried (wet) ice layer in Mars-like geologic formations. Based on the simulated study, we find that the prototype sensor design should be able to resolve the lateral variations in resistivity/susceptibility under conditions of the Martian subsurface.