A plastic scintillator-based radiation portal monitoring system has played an important role in preventing and detecting illicit trafficking of nuclear and radioactive materials. The limited spectroscopic information of the plastic scintillator material makes it difficult to discriminate radioactive materials of concern from naturally occurring radioactive materials (NORM) or background radiation. This has an impact on operations and surveillance costs. Various studies including energy windowing algorithm have been conducted to deal with this problem. However, few papers have been published on how to determine the optimal boundary of energy windowing algorithm. This paper discusses the algorithmic method for a plastic scintillator-based radiation detection system and how to determine the optimal boundary of the energy windowing. Comparing the calculated and experimental results, it appeared that the algorithmic method using energy window boundary presented in this paper could improve the ability of a plastic scintillator-based radiation detection system to discriminate certain threat materials from NORM or background radiation. Furthermore, nuclear materials (natural and low-enriched uranium) which have the similar spectral distributions with ambient background radiation could also be separated from it.