Magnetic nanofluids possess some unique properties that can significantly affect their thermal conductivity. We synthesize monodispersed magnetite (Fe3O4) nanofluids in toluene with the particle size from 4 to 12 nm and obtain aqueous nanofluids by a simple “one-step” phase transfer. Even without the effect of external field, the magnetic-interaction-induced self-assembled aggregation can still be significant in magnetite nanofluids. Investigation of the microstructures of self-assembled aggregation is carried out by the dynamic light scattering, which unveils the variation of aggregated configurations with particle concentration and time. Based on the calculation from the existing models, the aggregates decrease the thermal conductivity of both themselves and the entire system, mainly due to the less solid contents and weaker mobility compared with the single particles as well as the increase in interfacial thermal resistance. As the manifestation of the aggregation-structure variation, the measured thermal conductivity is of a wavelike shape as a function of particle concentration. The particle coating layers are also of importance in cluster formation so that nanofluid thermal conductivity can be manipulated for some nanofluids by changing the stabilizer used and thus controlling the particle aggregated structures. Due to the effects of temperature, viscosity and coating layers, the thermal conductivity for aqueous system varies in a different way as that for the toluene system.