Web Crawler of Power Grid Based on Selenium

Power grid companies have a massive amount of structured data that possesses potential appliance value. To protect information safety, power grid companies isolate their intranet from internet and require strong authentication to access databases. We developed a web crawler system to collect data from the intranet web pages, and utilize selenium, an automatic testing tool, to simulate human actions to avoid authentication. We proposed a depth-first search strategy to fulfill the need of topical data acquisition. The system provides a way to collect power data efficiently and it can help researchers and engineers build smart power grid.