Polycrystalline BiMn2O5 and Bi0.9Ce0.1Mn2O5 have been prepared by a solid-state reaction. A crystalline structure study shows that all x-ray diffraction patterns can be indexed successfully in an orthorhombic phase. Substitution of Ce for Bi ions induced in the unit cell a slight distortion and enlargement. Magnetic and dielectric measurements all reveal that Ce substitution induces great effects: BiMn2O5 is in the antiferromagnetic (AFM) phase with a Néel transition temperature at about 42 K, whereas Bi0.9Ce0.1Mn2O5 is in the ferromagnetic (FM) phase with a Curie temperature at about 46 K. The permittivity of Bi0.9Ce0.1Mn2O5 is much higher than that of BiMn2O5, and two dielectric relaxation peaks have been observed in the former, instead of one dielectric loss peak as observed in BiMn2O5. The valent state of Ce ions has been analyzed mainly as tetravalent by the core-level spectrum of Ce 3d from x-ray photoemission spectroscopy. The possible mechanisms of the magnetic transition from AFM to FM and the peculiar dielectric relaxation peaks induced by Ce substitution have been discussed in detail.