In the equilibrium immiscible Hf–Nb system characterized by a positive heat of formation, five Hf–Nb metallic glasses with overall compositions of Hf84Nb16, Hf65Nb35, Hf45Nb55, Hf38Nb62, and Hf20Nb80 are obtained by ion beam mixing with properly designed Hf–Nb multilayered films, suggesting a glass-forming composition range of 16–80 at. % of Nb. For the special case of Hf45Nb55 located at the ridge point on the convex free energy curve, dual-glass phases are formed at a dose of 2×1015 Xe+/cm2, which results from a spinodal decomposition of the expected Hf45Nb55 amorphous phase. With increasing irradiation dose, fractal growth of nanocrystals (around 20 nm) appears in the major glass phase and the dimension is determined to be from 1.70 to 1.84 within a dose range of (4–7)×1015 Xe+/cm2. In atomistic modeling, a n-body Hf–Nb potential is first constructed with the aid of ab initio calculations. Applying the constructed potential, molecular dynamics simulations using the hcp and bcc solid solution models, reveals an intrinsic glass-forming range to be within 15–83 at. % of Nb, which is compatible with the ion beam mixing - experiments. Moreover, the formation of the metallic glasses and the fractal growth in association with the amorphous spinodal decomposition are also discussed in terms of the atomic collision theory and cluster-diffusion-limited-aggregation model.