Big Data Application in Cancer Classification by Analysis of RNA-seq Gene Expression

Computer Science domain has got massive developments in the past decade, especially in the field of Big Data and Artificial Intelligence. Many industries are adopting these technologies to improve their products and services. Healthcare is one such industry which comes across voluminous data. Big Data platform is used to process the data and subsequently AI models can be used for prediction or classification. This paper focuses on proposing a Spark based pipeline to reduce the time required to process the voluminous data. With the aid of this new pipeline, we classify the type of cancer through analysis of RNA-seq gene expression levels. The Spark-based pipeline is run on default standalone mode and on a cluster. The study recorded that the Spark-based pipeline run on cluster mode improved the performance approximately by 11 times (regarding processing costs) when compared to the performance of Python-based pipeline.