The Integration of Shared Storages with the CephFS and Rados Gateway for Big Data Accessing

In recent years, high availability shared storage will become a popular information technology industry development orientation. Currently, information technology industries emphasize to reduce high risk data requirements and improve read and write performance of data storage. Therefore, the main purpose of this work is to improve read and write performance with the best way on Ceph Storage Cluster. In this system, the data is stored on Hadoop Distributed File System (HDFS), and the data stored in-memory virtual distributed store system that mentioned as Alluxio automatically. Then, the data would be processed through Hadoop Map Reduce method and the output would be inserted into Hadoop Distributed File System and Alluxio environment. The first experiment is to use S3 as application program interface that will connect to RADOS Gateway stored data into Object Storage Daemon (OSD). The second experiment is based on the first out experiment would be through Ceph File System (CephFS) connected to Object Storage Daemon directly. The data is saved in Ceph environment more secure than in Alluxio as in-memory storage system because OSD can be used for data backup based on object storage level. We can use S3 browser (GUI) to maintain data like grant access, maintain folders maintenance, create user accounts, move data location etc. The last one, we used Inkscope monitors all system. If there is any problem, system will give warning or error responds to users automatically.