National Oceanic and Atmospheric Administration Office of Ocean Exploration end-to-end data management: a success story

The National Oceanic and Atmospheric Administration (NOAA) is the lead federal agency for the ocean exploration initiative, which resulted from the multidisciplinary panel report released in October 2000 "Discovering Earth's Final Frontier: A U.S. Strategy for Ocean Exploration". This report, along with the constituent workshop results, defined a mandate that "Ocean exploration data needs to be readily available to researchers and to the public for education and outreach". Additional data management mandates are found throughout the government, ranging from the President's Management Agenda through the NOAA Strategic Plan. NOAA's Satellite and Information Service is working cooperatively with the Office of Ocean Exploration and other agencies in an integrated product team to meet these data management mandates through the development of a unique, end-to-end data management system. The goals of this system are to collect, manage, archive and disseminate OE data and information in accordance with nationally accepted formats and standards, to produce information that can be used throughout society to inform citizens and empower decision makers, and to make this information available independently and in perpetuity. NOAA's Ocean Exploration program conducts 25-30 expeditions per year, each of which host one or more independent research projects. The multiplicity of data sources, user requirements, and national standards, as well as the diversity of data collected during each expedition, contribute to the complexity of the OE data management challenge. Information about each research project within the expedition must be tracked from the proposal phase through project completion. At the same time, the assimilation, analysis and distribution of the data collected during each research project must be accomplished. A phased approach to meeting the OE data management challenge was implemented by the IPT. Existing systems used by OE and partners were reviewed and leveraged to provide a short-term solution. The longer term, end-to-end process was modeled and requirements were formalized. Metadata and other reports were standardized, and data were made available to the public in two prototypical systems. The Digital Atlas provides access to geospatially enabled data associated with research activities, and the Video Data Management System publishes video, still imagery, annotations and related documents. Still under development, the long term solution will combine research proposal and cruise planning information with specific vessel and instrumentation information, resulting in a customized interface. This interface will support at-sea data management and at the same will provide a metadata record framework. While using the system to document at-sea operations, data managers will also build metadata record components. Post-cruise, the at-sea information will be synchronized with the primary database. The information will be securely accessible to project principles via the Internet for record editing, completion and publishing. The overarching goal is to compile information as it is collected, to build and publish metadata records that support data access. The publication of metadata will support data access in accordance with any proprietary regulations. Once this system is fully operational, it will streamline data collection, reduce the overall data management burden and simplify compliance with national standards through an automated process. The system will also ensure data availability to the public in near real time and in perpetuity, and will be extensible to other applications.