Clustering Alexa Internet Data using Auto Encoder Network and Affinity Propagation

Non-linear mapping is one of the most popular solutions for complex data structures and distinct patterns to cluster data. Auto encoder Networks (AENs) are widely used in clustering as they improve data representation. In this paper, we collect Alexa.com data by crawling popular websites profiles, where dataset has 84 columns with type number and array of words. Next, an AEN architecture is presented to identify specific websites with exceptional patterns and the encoded data expresses new feature space of our original data. (Our) Encoded data is clustered by Affinity Propagation which is a partitioning algorithm without the need for specifying the number of clusters. There are important results based on 194 clusters and exemplars which are filtered and analyzed. One remarkable fact about results is that the first 11 columns as raw data are not clustered by Affinity Propagation a which re considered all as outlier. The results are summarized by selecting the best option w.r.t. the statistics and charts. Some of the obtained results are useful for website owners and provide some suggestions and solutions for Search Engine Optimization (SEO). Finally, we propose our crawler application which crawls and records data over 4 days. It must be added that the proposed web crawler faces challenges and their solutions can be helpful in most commonly used web crawling algorithms and libraries.