Integrative Day Trading Stock Trend Prediction using Web Scraping and Sentiment Analysis

In the digital era, data is pivotal across disciplines like business, marketing, engineering, and social sciences. This research proposes a new method of acquiring intraday stock data from the National Stock Exchangeand sentiment-laden news articles, via the web-scrapping technology. Spark’s distributed computing framework that provides powerful processing capabilities has been employed to perform sentiment analysis and quantify sentiment scores for the chosen stock over the past month. Concurrently, the XGBoost algorithm processes the historical stock data to predict future values along with the cross-validation process. With the integration between the XGBoostpredictions and the sentiment score, this model offers stock recommendations to the user to either buy or sell a certain stock. The proposed model performance is evaluated for future value predictionsbased on the average RMSE score reported as 1.2479.This extremely systematic take reaches across financial prediction, natural language processing, and machine learning to supply recommendations for investors.