Carrier dynamics in silicon photonic crystal (PhC) nanocavities are studied numerically. The results agree well with previous experimental demonstrations. It is shown that the presence of carrier diffusion makes fast switching possible, which is an advantage of nanocavity switches over other types of larger carrier based nonlinear optical switches. In particular, diffusion is effective in PhC nanocavity switches, which makes the switching recovery time even faster than that of silicon waveguide-based optical switches. In addition, calculations suggest that the thermo-optic effect can be reduced if the carriers are extracted within a few 100 ps by introducing a p-i-n structure.