Solar Data Tools: Automatic Solar Data Processing Pipeline

The increasing volume of photovoltaic system performance data is creating opportunities for remotely monitoring system health and optimizing operations and maintenance activities. However, this data often arrives in a variety of formats, with varying levels of quality, and missing metadata. The inability to automatically ingest, quality check, filter, and flag data across a large number of unique PV systems is a major roadblock to unlocking the potential of PV system performance data at scale. We have developed an automatic data processing pipeline application, written in open source Python. This software takes as an input generic, tabular PV performance time series data, typically a power signal at some sampling frequency, possibly with some missing or corrupted data. Using advanced data algorithms, large, high-frequency data sets can be standardized, cleaned, filtered, and flagged for basic operational issues in seconds on a standard laptop, with no need for user input or system metadata.