Spark-based Network Log Analysis

In order to analyze the rules of the user’s search intent. A statistical method of user search behavior based on Spark is proposed; based on the four indicators of correctness, precision, recall and F1-store evaluated by the classification model, the three classifiers of naive Bayes, logistic regression, and decision tree are compared. , The Naive Bayes classifier was selected as the classification model to realize the classification of the user’s search term intentions. The experimental results show that the number of RDD partitions is closely related to the model training time. In the Sogou Chinese classification data set, the number of RDD partitions is 6 When the time, the model training time is 69.12s, and its training time is the least. Naive Bayes is able to classify 10 topics of user search intent, among which historical and technological topics account for the highest proportions, respectively: 18% and 15%.