Practical numerical methods for stochastic optimal control of biological systems in continuous time and space

Previous studies have suggested that optimal control is one suitable model for biological movement. In some cases, solutions to optimal control problems are known, such as the Linear Quadratic Gaussian setting. However, more general cost functionals and nonlinear stochastic systems lead to optimal control problems to which direct solutions are presently unknown but these solutions would theoretically model behavioral processes. Additionally, in active exploration-based control situations, uncertainty drives control actions and therefore the separation principle does not hold. Thus traditional approaches to control may not be applicable in many instances of biological systems. In low dimensional cases researchers would traditionally turn to discretization methods. However, biological systems tend to be high dimensional, even in simple cases. Function approximation is an approach which can yield globally optimal solutions in continuous time and space. In this paper, we first describe the problem. Then, two examples are explored demonstrating the effectiveness of this method. A higher dimensional case which involves active exploration to learn unobservable parameters and the numerical challenges which arise will be addressed. Throughout this paper, multiple pitfalls and how to avoid then are discussed. This will help researchers to avoid spending large amounts of time merely attempting to solve a problem because a parameter is mistuned.