Protection of medium- and large-power transformers has always remained an area of interest of relaying engineers. Conventionally, the protection is done making use of magnitude of various frequency components in differential current. A novel technique to distinguish between magnetising inrush and internal fault condition of a power transformer based on the difference in the current wave shape is developed. The proposed differential algorithm makes use of radial basis probabilistic neural network (RBPNN) instead of the conventional harmonic restraint- based differential relaying technique. A comparison of performance between RBPNN and heteroscedastic-type probabilistic neural network (PNN) is made. The optimal smoothing factor of heteroscedastic-type PNN is obtained by particle swarm optimisation technique. The results demonstrate the capability of RBPNN in terms of accuracy with respect to classification of differential current of the power transformer. For the verification of the developed algorithm, relaying signals for various operating conditions of the transformer, including internal faults and external faults, were obtained through PSCAD/EMTDC. The proposed algorithm has been implemented in MATLAB.