KL divergence has been long used as a policy constrait in the field of reinforcement learning (RL). For example, in online RL, where agents interacts with the environment to update its policy, KL divergence is adopted to limit the search steps. Actually, KL divergence are so widely in the RL that it has become the golden standard. However, it sounds magical to me: why we adopt KL divergence as the constrait of policies?

KL Divergence and Related Meaning in Informative Perspective

KL divergence are first introduced by

Definition of KL divergence

Why KL divergence is useful in policy constrait?

Motivation of policy constrait

Futher thoughts on KL divergence