The contribution of optical phonons to thermal conductivity has typically been ignored. However, when the system size decreases to the nanoscale regime, optical phonons are no longer negligible. In this study, the contributions of different phonon polarizations to the thermal conductivity of silicon are discussed based on the phonon lifetimes extracted from a first principles approach. The results indicate that around room temperature, optical phonons can contribute over 20% to the thermal conductivity of nanostructures as compared to 5% in bulk materials. In addition, the temperature and size dependence of the contributions from acoustic and optical phonons are fully explored.