A Proposed Secure and Efficient Big Data (Hadoop) Security Mechanism using Encryption Algorithm

Because it can provide consumers with on-demand, dependable, adaptable, and affordable services, cloud computing has become popular for handling large amounts of data. Big data security management has grown in importance for the cloud because of the expansion of cloud applications. In this study, the suggested method was applied to enhance the efficiency of file encryption and decryption by utilising AES and OTP integrated into Hadoop. In the HDFS, data files are indeed encrypted, as well as the Map Task decrypts them. In earlier research, encryption, and decryption were performed using the AES technique, which resulted in a 50% growth in the size of the encoded data file in a big data environment. As the encoded file grew 20% larger than the underlying file, the suggested method boosted this proportion. Additionally, the proposed study deploys this new method to protect HDFS as well as undertook several experimental experiments to confirm its efficacy. The researcher in this study also evaluated it to the pre-existing methodology. This paper's major objective is to investigate the Hadoop data security problem. To protect data on the Hadoop system. It will suggest putting into action a novel method that combines HDFS files using the AES as well as OTP encryption algorithm to speed up upload and download and lower the capacity of the encrypted data file in big data systems.