In this paper, the optimal probabilistic neural network (PNN) is proposed as the core classifier to discriminate between the magnetizing inrush and the internal fault of a power transformer. The particle swarm optimization is used to obtain an optimal smoothing factor of PNN which is a crucial parameter for PNN. An algorithm has been developed around the theme of the conventional differential protection of the transformer. It makes use of the ratio of voltage-to-frequency and amplitude of differential current for the determination of operating condition of the transformer. The performance of the proposed heteroscedastic-type PNN is investigated with the conventional homoscedastic-type PNN, feedforward back propagation (FFBP) neural network, and the conventional harmonic restraint method. To evaluate the developed algorithm, relaying signals for various operating condition of the transformer, including internal and external faults, are obtained by modeling the transformer in PSCAD/EMTDC. The protection algorithm is implemented by using MATLAB.