Design Science Research Approach in Developing Information Retrieval and Web Scraping ETL Architecture for Optimizing NSESDN Data in East Java Social Programs

The National Socio-Economic Single Data (NSESDN) presents significant challenges for regional governments due to fragmented and unstructured data, which hampers effective policy and program implementation. This study employs the Design Science Research (DSR) methodology to develop an adaptive architecture that integrates Web Scraping, Information Retrieval (IR), and ETL (Extract–Transform–Load) processes. Unlike traditional Data Warehouses (DWH) or Service-Oriented Architectures (SOA), the proposed system automatically collects, cleans, and indexes heterogeneous legal and socio-economic documents from public sources. The architecture incorporates Python-based scraping, metadata normalization, and IR models such as TF-IDF and BM25 for intelligent search. Evaluation was conducted using both data quality metrics (completeness: 93%, accuracy: 92%, consistency: 95%) and usability testing involving regional stakeholders, demonstrating the system’s effectiveness in improving data access and relevance. The proposed system significantly enhances decision-making capacity in East Java’s social programs by enabling traceable, valid, and integrated public data access.