Structural and optical properties of single crystal Zn1-xMgxO nanorods (0≤x≤0.17) are studied experimentally and theoretically. Structural analyses indicate that the nanorods grown on Si substrates are oriented in the c-axis direction and the nanorods possess the single-crystalline hexagonal structure with the Mg incorporated within the ZnO nanorods by means of substituting Zn. A blueshift of the near-band edge emission in the photoluminescence spectra by increasing Mg content is observed. Two distinct emission bands are found in the photoluminescence spectra; one is mainly attributed to the delocalized exciton recombination and the other is originating from localized excitons due to the incorporation of foreign impurity of Mg. Enhanced exciton localization with increasing Mg content in Zn1-xMgxO nanorods is mainly due to large ionic characters of Mg-O bonding. Structural stability, band structures, projected density of states, and charge distribution in various Zn1-xMgxO alloy compounds were further investigated by first-principles calculations. A good agreement between experimental and theoretical results is found.