Topic modeling of news based on spark Mllib

In recent years, Spark has become one of the hottest data processing frameworks that makes up for Hadoop's flaws in implementing traditional data mining and machine learning algorithms. Based on the Spark platform, this paper deals with the news by using LDA model on MLlib to model the topics of them. The main work of this paper includes using the crawler to collect large-scale news corpus on the Internet, processing the data, and using the Scala language to call the LDA model in MLlib. The optimal number of iterations is determined first, then the optimal number of topics is estimated by the statistical method of V-fold cross validation, and finally the topic modeling of news corpus is realized. The topic probability distribution of each news obtained in the experiment is applied to the construction of the news corpus. In the Spark local mode, the experiment achieves the function of topic modeling by program coding and initial debugging.