A sample statistical characterization of the World-Wide Web

This paper provides a limited characterization of the World-Wide Web in terms of average document size, average transmission time, character frequencies, Hypertext transport protocol header sizes, document type frequencies, and server types. The data was obtained by recursively collecting statistical information about documents referenced by the www.yahoo.com index. This was done using a tool we call StatFetcher. In the study, 15,000 nodes were visited with an average of six of more documents retrieved per node. The results were used to determine opportunities for improving the Web performance.