Ujan

Ujan

he/him

NLP Researcher

Changing the action space in LLM RL

less than 1 minute read

Published:

  • Noisy advantage in LLM RL

    Illustrative token-level advantages: each token has a synthetic advantage, with coral indicating negative values and green indicating positive values.

  • Plan

    Prefix activations and five later states are compressed into discrete codes. The first three future tokens are highlighted as an illustrative selection based on high mutual information, estimated across many rollouts.

  • Spec decoding

    A diffusion drafter conditioned on target-model context features proposes five tokens in parallel. Target verification accepts the first three, rejects the fourth, and discards the fifth. The accepted draft length h is an illustrative proxy for plan length.

  • Role of action space in discrete RL settings

You May Also Enjoy

Flow Matching

16 minute read

Published:

Flow matching gives us a way to model complex real world distributions (target) from simpler or known distrubitions (source).