How Trustworthy Is the Critic? an Empirical Assessment of PPO's Initial Value Estimates

Actor-critic algorithms such as Proximal Policy Optimization (PPO) depend on the critic's value estimate to guide policy updates. Yet, the accuracy of this estimate is seldom quantified. We empirically assess four Atari games-Alien, Breakout, Freeway, and Pong. After training a PPO agent for 10 million frames in each environment, we record 100 pairs of the critic's initial state value and the corresponding realized discounted return. Correlation and ordinary least squares analyses reveal marked disparities: the critic's predictions closely track returns in Alien but diverge sharply in Pong. These findings indicate that value-learning bias is strongly environment-dependent. Future work will examine how value loss and reward curves, reward sparsity, reward normalization, and sharing the feature extractor influence this bias to improve critic reliability and overall agent performance.