Online Web Bot Detection Using a Sequential Classification Approach

A significant problem nowadays is detection of Web traffic generated by automatic software agents (Web bots). Some studies have dealt with this task by proposing various approaches to Web traffic classification in order to distinguish the traffic stemming from human users' visits from that generated by bots. Most of previous works addressed the problem of offline bot recognition, based on available information on user sessions completed on a Web server. Very few approaches, however, have been proposed to recognize bots online, before the session completes. This paper proposes a novel approach to binary classification of a multivariate data stream incoming on a Web server, in order to recognize ongoing user sessions as generated by bots or humans. The present approach uses deep neural networks combined with Wald's Sequential Probability Ratio Test to express the relationship between subsequent HTTP requests in an ongoing session and to assess the likelihood of each session being generated by a bot or human before it ends. Experimental results showed the ability of the proposed approach to detect Web bots online with high performance scores and a small number of false negatives, as evidenced by the Recall index, minimizing the impact on human visitors. Another valuable indicator is the speed of decision: the present method allows very quick classification of nearly all sessions, leaving only very few of them undecided.