Design and Implementation of Scalable Data Collection Framework for SDMX

With the increasing adoption of the SDMX (Statistical Data and Metadata Exchange) standard by major national statistical institutes and public authorities, researchers and general users now have greater access to high-quality and reliable statistical data. However, collecting and managing data from various SDMX open data sources presents significant challenges. Firstly, despite operating under the same version of the SDMX standard, discrepancies in data exchange formats exist across different SDMX web services. Secondly, SDMX data requires accompanying metadata for accurate interpretation, yet there has been a lack of robust consideration for a data model capable of concurrently storing both statistical value and its associated metadata. To address these challenges, we propose an scalable data collection framework for SDMX. Structurally, the proposed framework is designed as a flexible and scalable architecture that can be seamlessly extended to target various SDMX open data sources. By deploying dedicated response parsers with standardized in/out interfaces, it can dynamically accommodate a wide range of data sources, providing a scalable solution for diverse statistical data collection. It retrieves data from SDMX open data sources and constructs a integrated SDMX data model within local systems. This model facilitates the retrieval, storage, and management of statistical data while preserving the integrity of the Data Structure Definitions (DSD) as specified by data providers. Additionally, our framework offers advanced data management capabilities, enabling users to efficiently request data CRUD (Collect, Read, Update, and Delete). We validated the functionality and efficacy of the framework by applying it to several prominent SDMX web services.