Case: Big geosciences data validation challenges and achievements

Geosciences data vary in a wide spectrum of types/attributes (hundred thousands of data classes) in extra huge volumes (Hundreds of PB's and ZB's) with the complexity of the format/media and versions in addition to the dissemination in different DB's/applications. The current exponential data growth rate “DGR” is >100 PB just in a couple of years. Managing and validating such data is a challenge and complex. The concept of validation through the integration triangle (Technology, People & Best Practices) has been implemented in our project to close the data validation gaps and to eliminate the conventional data issues that represents a hindrance in the interpretation that badly affects the decision time and effectiveness. We used a standard queries and scripts for data mining/analysis to highlight the gaps, duplication/redundancy, inconsistency, incomplete record or attribute, conflict, typing mistakes, naming issues, nonstandard records and any other data issues. The best industry standards and the naming conventions have been applied to the Database in order to have a clean, consistent, complete, standard, unique object and authenticated data sets that complies with the aforementioned standards. During the planning phase, we studied many cases, then started with a pilot and detect the weakness / bottlenecks that may represent any obstacles during the full implementation. Based on that we expected other issues and we put the suggested solution for each. This why it ran seamlessly in the final phase. The implementation of the project took place through the following: 1. Data analysis / mining to quantify the data issues in exact numbers and statistics. 2. Use the best workflows in the industry 3. Standard processes and business rules 4. Sealed data governance strategy. 5. Optimum technology utilization 6. Update and then repeat the cycle again periodically. The E&P Information Management Team is committed to keep the project updated in a continuous improvement phase, DB with the consistent level of accuracy and validation in that dynamic process. The effective cooperation between data managers and senior level of the end users to validate the data in a multi-level of authentication is keeping the new coming data in the best shape. The main benefits of such work contains: 1- Remove the duplicated and redundant data that causes a big miss to the users 2- Complete the missing records and data sets 3- Apply the KOC naming conventions 4- Comply to the best industry standards 5- Continuous phase of improvement 6- Reduction of the required space (storage) 7- Most effective connection by centralizing the validated data in one place 8- Save the end users time (30-40% for data searching/acquisition and 25-40% for check/validation) as per the worldwide benchmark 9- Cost saving eventually 10- Well prepared for the future and the knowledge/information burst 11- Very effective disaster recovery policy 12- Right decision on the right time 13- Data security and governance.