With the aid of ab initio calculations, an n-body Fe–Nb embedded-atom potential is first constructed and then applied to study the crystal-to-amorphous phase transition through molecular dynamic simulations. The simulations determine that the glass-forming range of the Fe–Nb system is 18–83 at. % of Nb. In ion beam mixing experiments, five Fe–Nb multilayered films with overall compositions of Fe85Nb15, Fe75Nb25, Fe55Nb45, Fe25Nb75, and Fe15Nb85, respectively, are irradiated by 200 keV xenon ions to doses in the range of (1–7)×1015Xe+/cm2. The result shows that the Fe–Nb metallic glasses can be synthesized within a composition range of 25–75 at. % of Nb, matching reasonably well the theoretical prediction. Moreover, in the Fe55Nb45 sample, a fcc-structured alloy phase with a large lattice constant of a≈0.408 nm was obtained at a dose of 3×1015 Xe+/cm2 and the associated magnetic moment per Fe atom was measured to be 2.41μB. The observed magnetic moment is much greater than the initial value of 1.42μB in the bcc-Fe lattice and can thus serve as evidence confirming the high-spi- n ferromagnetic state of fcc Fe predicted by ab initio calculations. Interestingly, further irradiation induced phase separation in the Fe55Nb45 sample, i.e., irradiation to a dose of 5×1015 Xe+/cm2 results in the growth of a fractal pattern consisting of Fe72Nb28 nanoclusters embedded in Fe35Nb65 matrix. The formation mechanism of the metastable phases as well as that of the fractal pattern observed in the Fe–Nb system was discussed in terms of the atomic collision theory and the well-known cluster-diffusion-limited-aggregation model.