Optimizing stock market execution costs using reinforcement learning

Stock market trading is a complex process where traders aim to maximize their expected return while minimizing associated risks. With the increasing availability of digital historical records, using automated agents for stock market trading becomes of a significant interest. Reinforcement learning is a machine learning branch which circumvents the problem of defining explicit targets and tackles problems which require sequential decisions. Reinforcement learning has been applied in finance problems, yet execution costs optimization problem among others still gets little attention. The optimization of execution costs in stock markets is a vital problem, where a trader wants to minimize the cost of buying a predefined amount of shares over a fixed time horizon. In this study, we propose a novel reinforcement learning Q-trade model to address the execution costs optimization problem. We tested the Q-trade model using historical data of the Egyptian stock market and Nasdaq stock market, and it showed in both markets a significant improvement (more than 60% for some securities) over the compared strategies.