NEPTUNE Canada: Data integrity from the seafloor to your (Virtual) Door

In December 2009, the NEPTUNE Canada ocean observatory successfully launched to the public, allowing anyone with an Internet connection access to observatory data. This event also officially marked a transition from infrastructure development to a fully operational observatory. Given that this occurred only a few months after the instrument platforms were deployed at four nodes, this transition was a considerable achievement. Notably, this quick turnaround presented a significant change for principal investigators who normally have much more time, often years, to conduct their science before releasing data to the public. In this paper, commissioning processes, operational challenges, and future plans are described as they relate to observatory data integrity. The data commissioning phase, beginning even before an instrument enters the water, is a team effort that requires NEPTUNE Canada staff, the principal investigator (PI) and sometimes the manufacturer. For each instrument, there are three qualification stages. First and foremost, the systems team checks that instruments can be powered and perform within acceptable limits (e.g., no ground faults). Most instruments pass these requirements. The next level involves a daily review by NEPTUNE Canada staff of data from each instrument. The main objectives are to ensure raw data meets manufacturer specifications, sensor values are appropriately parsed and calibrated, and basic metadata are appropriately recorded in the database. Most issues at this stage can be relatively easily fixed. The third level focuses on the quality of the data itself. Pis are provided both raw data and derived data products since they are expected to be involved in the verification process. Various routines and visualizations plots are used to ensure values make sense. Anomalies and data gaps are noted and investigated. Instrument configurations may require numerous iterations to conquer challenges like interference between acoustic instruments and to better meet scientific objectives. Online tools (e.g., a screen to view and change configuration parameters) and specialized software (e.g., for interaction between instruments) have been created to support these adjustments. Data products are also being continually developed as a response to PI feedback and requests. As a result of this continual examination of the data, about 30 instruments have successfully passed commissioning phase. These instruments include Acoustic Doppler Current Profilers, Bottom Pressure Recorders, CTDs, fluorometers, a gravimeter, video cameras, hydrophones, methane and oxygen sensors. Post-launch, in addition to ongoing instrument commissioning and system maintenance, there were new operational challenges. The first challenge has been meeting the demands for data access. As of May 28, 2010, there have been over 20,000 data requests (1463 within the first two days) from over 1000 registered users via the Data Search tool in Oceans 2.0 (an online collaborative workspace for interacting with data and metadata and remotely controlling instruments). In response to user demand, more data product options continue to be offered. Web services are being developed as an alternative data access method. The general public has responded with an enormous appetite for video data, which has necessitated instructive tutorials for watching video data and explanations as to why cameras cannot be operated 24/7. A second challenge has been communicating important information to the end-user. Feedback is helping us determine how and what information to convey with regards to data interruptions/delays, instrument malfunctions, metadata and quality. Maintenance cruises present a third significant challenge, as there are significant metadata updates to make and new instruments to commission. Metadata must be quickly obtained, verified and entered, particularly for instruments that are already 'live'. A combination of technologies facilitate necessary communications with the crew at sea: live and archived video from ROPOS (including logs) via our website, Twitter, an installation blog, Skype and email. The first successful maintenance cruise occurred in May, and another deployment at Folger Shallow is scheduled for this summer. Going forward, there are further plans to enhance data integrity on the system, such as: developing automated quality control routines to detect anomalous data, exposing more metadata about instruments (history, documentation, annotations, quality flags, etc.), providing a notification system for events affecting data (e.g., instrument configuration changes and power outages), improving help resources and information within Oceans 2.0, increasing staff (two Scientific Data Specialists start in June). At the time of this conference, NEPTUNE Canada will be actively deploying more instruments including Wally II (an improved version of the crawler Wally), a repaired VPS, and Endeavour Ridge instrument platforms. A new wave of commissioning begins.