We demonstrate all-optical switching in the telecommunication band, in silicon photonic crystals at high speed (∼50 ps), with extremely low switching energy (a few 100 fJ), and high switching contrast (∼10 dB). The devices consist of ultrasmall high-quality factor nanocavities connected to input and output waveguides. Switching is induced by a nonlinear refractive-index change caused by the plasma effect of carriers generated by two-photon absorption in silicon. The high-quality factor and small mode volume led to an extraordinarily large reduction in switching energy. The estimated internal switching energy in the nanocavity is as small as a few tens of fJ, indicating that further reduction on the operating energy is possible.