A Distributed Approach to Determine Acronym and Expansion Pairs using Apache Spark

We evaluate the performance of a distributed approach to recognize acronym and expansion pairs using Apache Spark. We train and evaluate the K-NN, Parallel SVM, and Decision Tree algorithms using Apache Spark and compare the performance of the three algorithms with SVM Light model, developed and trained on a single machine without using Spark. The MapReduce paradigm was implemented using Java programming and thus, no wrapping library such as Hadoop-Streaming Library was needed. The results reveal that the accuracy of the newly trained models on Spark is excellent, it gets around 99% for the training phase and around 98% for the testing phase. An additional pruning approach was also introduced and it successfully reduced the number of candidate of acronym and expansion pairs to approximately 13.3%.