Optimizing Large Data Processing and Security for Edge Computing Using Embedded Single-File Databases and Blockchain Technology

The practical applications of technology and their advancement are causing a growing need to process traffic for central servers, which has led to the rise of edge computing as a potential solution to alleviate the load by shifting some of the central server’s tasks to the edge of the network. However, there are potential risks and limitations with this approach, such as the possibility of overloading the edge if the work is not optimized properly and security issues that can arise when edge devices become targets for hackers. Improving this issue is crucial for enhancing the effectiveness of edge computing and its practical applications. This article will introduce a novel method to address this challenge by optimizing the processing of large volumes of data at the edge, thereby alleviating the overload caused by excessive data at the edge boundary. This can be accomplished by using DuckDB, an embedded single-file database. In addition, integrating blockchain technology can serve as a solution to improve data security for edge devices. DuckDB, a compact and simple embedded single-file database, offers an optimized solution for speed and performance in processing large data at the edge while maintaining large storage capacity with minimal size. The reduction in data processing time and the improved storage limits will improve performance and prevent edge overload. Combined with the unique security mechanism of blockchain, including immutability and strong encryption through Hash functions, it delivers superior security performance. This combination leads to significant changes in edge processing and edge security that traditional methods cannot provide. The system under review has shown positive results after being evaluated on three main criteria: its speed and performance in handling large datasets, the complexity of its data security measures, and its ability to minimize overload on both edge and central servers. The strict requirements for resources, input data, and evaluation criteria in experiments highlight the strong potential of this method for implementation in the real world. This shows that the approach could meet the demands of practical applications where high performance and security are critical.