Deep Reinforcement Learning for Enhanced Stock Market Prediction with Fine-tuned Technical Indicators

Since stock data is unpredictable, noisy, and fluctuating, it is critical to analyze and predict it. Traditional statistical models struggle with the high volatility and multifaceted nature of the market, influenced by diverse factors such as economic indicators, investor behavior, company performance, and global events. Machine learning, particularly deep learning techniques like Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory (LSTM), have shown promise in improving predictive performance. Reinforcement learning (RL) methods, including model-based, off-policy, and actorcritic, further enhance predictive capabilities by adapting to changing environments and maximizing rewards. This study aims to build a profit/loss RL model with unique indicator values to improve RL models' efficacy and precision in trend detection and data testing within the stock market. The methodology involves analyzing various stock datasets using five RL algorithms: Advantage Actor-Critic (A2C), Deep Q-Network (DQN), Actor-Critic using Kronecker-Factored Trust Region (ACKTR), Proximal Policy Optimization 1 (PPO1), and Proximal Policy Optimization 2 (PPO2). The study incorporates hyper-tuned technical indicators to optimize RL algorithms' performance. Upon training and evaluation, Deep Q-Network (DQN) outperforms the other algorithms, demonstrating its suitability for the stock market's dynamic and unpredictable nature. The datasets include historical price data and technical indicators.