The atomic structure and optical properties of Si-rich silicon nitride thin films have been for decades the subject of intense research, both theoretically and experimentally. It has been established in particular that modifying the chemical composition of this material (e.g., the Si excess concentration) can lead to dramatic differences in its physical, optical, and electrical properties. The present paper reports on how the incorporation of oxygen into silicon nitride networks influences their chemical bonding and photoluminescence properties. Here, by using a combination of analytical scanning transmission electron microscopy and x-ray photoelectron spectroscopy it is demonstrated that the structure of Si-rich silicon nitride with low O content can be described by the co-existence of Si nanocrystals in a Si3N4 matrix, with occasional localized nano-regions of a Si2ON2 phase, depending on the amount of excess Si. Furthermore, it is shown that the structure of silicon nitride with high O content can be adequately described by a so-called random bonding model, according to which the material consists in bonded networks of randomly distributed tetrahedral SiOxN4-x (where x = 0, 1, 2, 3, and 4). Photoluminescence measurements indicate that the effect of O is to introduce a gap state in the band gap of Si3N4 matrix. When a large amount of O is introduced, on the other hand, the photoluminescence measurements are in agreement with a shifted conduction band minimum in the dielectric. For both cases (high and low O content), Si dangling bonds were found to give rise to the deep level in the band gap of the nitride matrix, causing the dominant emission band in the photoluminescence of the films.