Research on Portfolio Optimization Models Using Deep Deterministic Policy Gradient

Recently, the research of reinforcement learning and its application in portfolio optimization is gaining popularity. Within the reinforcement learning framework, an agent can learn a satisfying policy while constructing a portfolio, however, this framework does not work so well with complex dynamical environments and the continuous states due to emotions and lots of unexpected factors in financial markets. To address this issue, we propose a reinforcement learning framework for portfolio optimization based on deep deterministic policy gradient algorithm. First, the dynamic portfolio optimization problem is formulated as a Markov Decision Process (MDP) with action space, state space, and reward function. Then, a Deep Deterministic Policy Gradient algorithm is used to learn policies in high dimensional, continuous action space, aiming at controlling the MDP continuously while different optimization policies for portfolios are learned based on different policy networks. Results of the testing on different portfolios consisting of stocks from SP500 demonstrate that learning rate and cash bias are directly related to the activeness of trading strategies while the Average Logarithmic Cumulative Return reward function achieves the best performance among all the suggested reward functions. More importantly, with the help of LSTM layer or full connection layer, the performance of convolution layer is better than other models including all mentioned benchmarks.