We propose a systematic approach to identify outlier in transactional data. First, we define a measure to estimate an outlying score for each transaction. Then, based on the estimated scores, we propose a probabilistic method that exploits the beta mixture model to automatically identify outliers. In contrast to existing transactional outlier detection methods, the approach that we propose does not require the target number of outliers in the data. Furthermore, our method is able to automatically discriminate between outliers and inliers without requiring any user-predefined threshold. Experiments on both synthetic and real data demonstrate the superior performance of our approach.