Spark-based Network Log Analysis

In order to analyze the rules of the user’s search intent. A statistical method of user search behavior based on Spark is proposed; based on the four indicators of correctness, precision, recall and F1-store evaluated by the classification model, the three classifiers of naive Bayes, logistic regression, and decision tree are compared. , The Naive Bayes classifier was selected as the classification model to realize the classification of the user’s search term intentions. The experimental results show that the number of RDD partitions is closely related to the model training time. In the Sogou Chinese classification data set, the number of RDD partitions is 6 When the time, the model training time is 69.12s, and its training time is the least. Naive Bayes is able to classify 10 topics of user search intent, among which historical and technological topics account for the highest proportions, respectively: 18% and 15%.

I. Introduction

Since 2010, the Internet has developed rapidly and widely used. People can perform various operations through the Internet, such as browsing current affairs, online shopping, e-sports, and playing videos. When people want to get the corresponding information, they can enter the content they want to query in the search box and click the search button to get the desired result, which facilitates people's grasp of the information and saves a lot of time . According to CNNIC (China Internet Network Information Center) released the 48th "Statistical Report on Internet Development in China", as of June 2021, the number of Chinese Internet users has exceeded 1 billion, of which the number of search engine users has reached 795 million, accounting for the total number of Internet users. 78.7%[1]. It can be seen that with the popularity of the Internet, the number of users has reached a very large scale, and a large number of logs have also been generated for users. Wang Yuanzhuo et al. [2] believe that there is valuable information in these network big data, and mining data information is the general trend of future development. From this analysis of the user's search log, the user's behavior information can be obtained from it, and it also provides data support for the company's product optimization.