We study Merton’s expected utility maximization problem in an incomplete market, characterized by a factor process in addition to the stock price process, where all the model primitives are unknown. The agent under consideration is a price taker who has access only to the stock and factor value processes and the instantaneous volatility. We propose an auxiliary problem in which the agent can invoke policy randomization according to a specific class of Gaussian distributions, and prove that the mean of its optimal Gaussian policy solves the original Merton problem. With randomized policies, we are in the realm of continuous-time reinforcement learning (RL) recently developed in Wang et al. (2020) and Jia and Zhou (2022a,b, 2023), enabling us to solve the auxiliary problem in a data-driven way without having to estimate the model primitives. Specifically, we establish a policy improvement theorem based on which we design both online and offline actor–critic RL algorithms for learning Merton’s strategies. A key insight from this study is that RL in general and policy randomization in particular are useful beyond the purpose for exploration – they can be employed as a technical tool to solve a problem that cannot be otherwise solved by mere deterministic policies. At last, we carry out both simulation and empirical studies in a stochastic volatility environment to demonstrate the decisive outperformance of the devised RL algorithms in comparison to the conventional model-based, plug-in method. Joint work with Min Dai, Yuchao Dong and Yanwei Jia.

Bio: Xunyu Zhou is the Liu Family Professor of Financial Engineering and the Director of Nie Center for Intelligent Asset Management at Columbia University in New York.

His current research focuses on developing a foundational theory for continuous-time reinforcement learning and its applications to financial decision making. Previously, he has worked on quantitative behavioral finance, time inconsistency and stochastic control.

Zhou is known for his work in indefinite stochastic LQ control theory and application to dynamic mean-variance portfolio selection, in asset allocation and pricing under cumulative prospect theory, and in general time inconsistent problems. He has addressed the 2010 International Congress of Mathematicians, and has been awarded the Wolfson Research Award from The Royal Society (UK), the Outstanding Paper Prize from the Society for Industrial and Applied Mathematics, the Humboldt Distinguished Lecturer and the Alexander von Humboldt Research Fellowship. He is both an IEEE Fellow and a SIAM Fellow. He was awarded Distinguished Faculty Teaching Award at Columbia University in 2023.

Zhou received his PhD in Operations Research and Control Theory from Fudan University in China in 1989. He was the Nomura Professor of Mathematical Finance and the Director of Nomura Center for Mathematical Finance at University of Oxford during 2007-2016 before joining Columbia.