RLHF讲解

RLHF包含了两个至关重要的步骤：

1. 训练Reward Model
2. 用Reward Model和SFT Model构造Reward Function，基于PPO算法来训练LLM
   1. frozen RM
   2. frozen SFT Model
   3. Actor ${\pi }_{\Phi }^{RL}$ initialized from SFT Model
   4. Critic $V_{\eta }$ initialized from RM

---

参考

RLHF理论篇