Thesis Defence: Critic-Derived Shaping in Proximal Policy Optimization

Alex Deaconu, supervised by Dr. Pourang Irani & Dr. Abbas Milani, will defend their thesis titled “Critic-Derived Shaping in Proximal Policy Optimization” in partial fulfillment of the requirements for the degree of Master of Science in Computer Science.

An abstract for Alex Deaconu’s thesis is included below.

Defences are open to all members of the campus community as well as the general public. This defence will be offered in hybrid format.  Registration is not required to attend in person, but please email pourang.irani@ubc.ca or abbas.milani@ubc.ca to receive the Zoom link for this defence.

Abstract

The simplicity and stability of Proximal Policy Optimization (PPO) in reinforcement learning make it popular, yet its on-policy training loop may be sample-inefficient. This thesis investigates whether PPO can leverage temporal-difference (TD) error, a signal that it already computes, in a regulated manner. We introduce critic-derived advantage shaping (CDAS), a method which employs normalized positive TD-error as an actor-side shaping signal, and critic-derived value shaping (CDVS), which applies the same detached, normalized bonus to the critic target. While CDAS-0 and CDVS-0 employ one-step positive TD-error, we test additional variants called CDAS-λ and CDVS-λ, which propagate future positive critic surprise backward via a GAE-style bonus trace. We find that PPO can benefit from positive critic surprise in advantage-side shaping, and especially critic-side shaping, but only under strict scale control. When the bonus is modest in comparison to the extrinsic advantage, actor-side CDAS improves performance on several MuJoCo tasks and performs particularly well on the Atari game Gravitar. Conservative CDAS-0 and CDAS-λ settings in MuJoCo increase mean PPO-normalized final return by around 4–5% above PPO. However, the improvements are environment-dependent; Venture demonstrates how visually prominent but task-misaligned transitions can draw the TD-error signal and impair learning, while Hopper performance deteriorates when the coefficient is too high. Critic-side shaping is stronger in aggregate—CDVS-0 improves mean PPO-normalized final return by 7.4% across the studied MuJoCo environments, and the strongest critic-only CDVS-λ configuration improves it by 12.5%. In the Atari critic-side sweep, the strongest aggregate CDVS-λ configuration improves mean final return by 14.2% and mean AUC by about 16%, with AUC gains in all five Atari games. Overall, critic-derived shaping is useful until it overpowers PPO’s extrinsic advantage structure: TD-errorbased bonuses must stay secondary to PPO’s extrinsic advantage signal in order to benefit from the guidance inherent in TD-error.