Thesis Defence: Donsker-Varadhan Policy Matching for Reward-Free Offline Reinforcement Learning

Quang Anh Dao, supervised by Dr. Md. Jahangir Hossain, will defend their thesis titled “Donsker-Varadhan Policy Matching for Reward-Free Offline Reinforcement Learning” in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical Engineering.

An abstract for Quang Anh Dao’s thesis is included below.

Thesis are open to all members of the campus community as well as the general public. Please email jahangir.hossain@ubc.ca to receive the Zoom link for this defence.

Abstract

This thesis studies an alternative method for reward-free offline reinforcement learning, investigating how well an agent can learn from a fixed dataset when reward guidance is missing, weak, or unreliable. A common way to handle this problem is behavior regularization, where the learned policy is kept close to the behavior policy that generated the dataset. This thesis studies an actor-to-behavior forward-KL version of this idea using the Donsker–Varadhan representation of KL divergence. In contrast to behavior cloning’s behavior-to-actor KL direction, this actor-to-behavior direction can avoid the action-averaging problem. A reward-augmented alternative is also studied and can be used in settings where reward labeling is meaningful. In controlled settings, our method can recover KL divergence. However, its practical performance depends strongly on witness quality, optimization stability, and dataset structure. Therefore, while the method provides a principled framework for reward-free policy optimization, it does not consistently outperform established benchmark algorithms.