Zhuoyun Du 杜卓耘

The objective lives at the end of the trajectory

Single-turn RLHF usually treats a prompt and its completion as one decision. A reward model scores the completed response, and PPO, GRPO, DPO, or another optimizer changes the token policy accordingly. Multi-turn agents add a second temporal scale: tokens form a response, and responses form a trajectory.

The learning problem becomes difficult when useful rewards are delayed. A conversation may receive a single success indicator only after the customer completes a purchase, the student solves a problem, or the environment accepts a final action. An early response has no obvious label of its own, even though it may determine whether the later outcome is reachable.

This is the credit-assignment problem in conversational form. The trainer must propagate evidence from an eventual outcome back to the intermediate decisions that helped or hurt it, without encouraging the model to exploit a brittle simulator or reward proxy.

One agent, two action scales

At the response level, an action is an entire message: ask a clarifying question, offer a plan, invoke a tool, or stop. At the token level, the same action is generated autoregressively. Classical multi-turn RL reasons naturally over response-level actions, while mature LLM post-training software is optimized for token-level updates.

This mismatch explains why simply attaching the final trajectory reward to every generated token is unsatisfying. It gives all decisions the same coarse label and ignores how the value of a response depends on the state in which it was produced. Two recent approaches address the mismatch from different directions: learn a value for each state-response pair, or compare complete trajectories directly.

Route one: use a Q-function as the bridge

Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO reduces multi-turn policy improvement to a sequence of familiar single-turn RLHF problems. The central object is a response-level Q-function, Q(s, a): the expected future return after producing response a in conversational state s and then continuing under the current policy.

If the Q-function is accurate, it compresses the future into a score for the current response. The token-level optimizer no longer needs to unroll every future turn during each update. It can treat Q(s, a) as the reward model for a standard single-turn completion problem.

The Iterative PPO loop

1. Deploy the current policy. Collect a batch of complete multi-turn trajectories and their eventual outcomes.
2. Evaluate the policy. Compute Monte Carlo returns for intermediate state-response pairs and fit a Q-function with supervised regression.
3. Improve the policy. Freeze the learned Q-function and use it as the reward model for token-level PPO.
4. Collect fresh data. Deploy the improved policy, because its new response distribution may move beyond the coverage of the old Q-function.

This is policy iteration expressed through existing LLM tooling: response-level policy evaluation followed by token-level policy improvement. The paper proves that if the new policy improves the expected Q-value at each state, it improves the multi-turn return globally. It implements the update with PPO, while noting that other effective single-turn optimizers, including GRPO, can in principle serve as the improvement step.

The distinction matters. The original method is Iterative PPO, not an experimentally established "Iterative GRPO" algorithm. Replacing PPO with GRPO is a plausible implementation choice supported by the reduction, but it should be evaluated rather than treated as a reported result.

What can go wrong?

The Q-function only learns from actions represented in the logged data. A nearly deterministic policy may produce narrow trajectories, leaving the critic unreliable on alternative replies proposed during policy optimization. Exploration, off-policy evaluation, and periodic data refresh are therefore central parts of the method rather than secondary engineering details.

Route two: optimize trajectory preferences

Building Math Agents with Multi-Turn Iterative Preference Learning begins from a different supervision format. A tool-using math agent alternates between model actions and code-interpreter observations. Instead of fitting a scalar Q-value for each response, the method compares successful and unsuccessful trajectories and extends DPO and KTO to the multi-turn setting.

The key implementation detail is easy to overlook: interpreter outputs are part of the history, but they are not actions generated by the model. Their tokens must be masked out when computing policy log-probabilities. Otherwise the optimizer rewards or penalizes the model for text produced by the environment.

The framework iteratively samples new solutions, separates preferred and dispreferred trajectories using final-answer feedback, and updates the policy with multi-turn DPO or KTO. In the reported Gemma-1.1-it-7B experiment, iterative M-DPO improved accuracy from 77.5% to 83.9% on GSM8K and from 46.1% to 51.2% on MATH over three iterations. The result illustrates the value of online data refresh: each new policy exposes trajectories that were unavailable to its predecessor.

The methods make different assumptions

Q-based policy iteration and trajectory-level preference learning are complementary rather than interchangeable. Their supervision, environmental assumptions, and failure modes differ.

This last distinction is important for general agents. Code execution is deterministic given the submitted program, but a human user or another LLM can respond stochastically. The M-DPO paper shows that its simplified loss is exact under deterministic transitions and points toward value estimation or an adaptive correction when observations are stochastic. Moving from tool-use math to open-ended conversation is therefore not a change of dataset alone; it changes the learning assumptions.

A practical training checklist

Before choosing an optimizer, I would make the trajectory semantics explicit:

1. Define the outcome. Is success a terminal binary event, a graded task score, a human preference, or a vector of safety and quality objectives?
2. Mark policy-owned tokens. Separate model actions from user messages, tool outputs, retrieved documents, and system-generated state.
3. Choose the credit mechanism. Decide whether intermediate actions need an explicit Q-value or whether trajectory comparisons are sufficient.
4. Measure coverage. Track whether training trajectories cover the response styles and states that the optimizer will explore.
5. Refresh on-policy data. A policy that improves changes its own state distribution; stale data eventually describes the wrong agent.
6. Audit the simulator. If another model supplies user reactions or rewards, check whether the policy is learning the intended task or the simulator's quirks.
7. Evaluate whole trajectories. Report task success, cost, length, safety, and recovery behavior, not only average response-level reward.

Takeaways

• Multi-turn post-training changes the unit of optimization. The object of interest is a trajectory, even when the implementation updates token probabilities.
• Value functions connect long-term outcomes to current responses. They make mature single-turn RLHF infrastructure usable for response-level policy iteration.
• Preference objectives must respect the environment boundary. Tool and user observations belong in the state, not in the policy likelihood.
• Online iteration is also distribution management. Fresh rollouts are needed because an improved policy visits new states and produces new actions.
• Algorithm names are less important than assumptions. Before choosing PPO, GRPO, DPO, or KTO, specify where rewards come from, which transitions are stochastic, and how credit reaches earlier turns.

The broader lesson is that training an agent is not just training a better response generator. It is learning a policy whose local decisions remain useful after the environment, the user, and the agent itself have had time to react.