https://saheb.github.io/blog/notes/ Recent content in Notes on Saheb's AI Journey Hugo en-us Mon, 20 Jul 2026 00:00:00 +0000 https://saheb.github.io/blog/notes/rl-generalization-notes/ Mon, 20 Jul 2026 00:00:00 +0000 https://saheb.github.io/blog/notes/rl-generalization-notes/ <p>Source: Kirk et al. 2023 ZSG survey (arXiv 2111.09794) + working sessions, July 2026.</p> <h2 id="1-the-problem-in-one-sentence">1. The problem in one sentence</h2> <p>Train a policy on a finite sample of contexts drawn from p(c); <strong>freeze it</strong>; measure expected return on contexts never trained on. Zero-shot = <strong>no adaptation at test time</strong> (no gradient updates, no practice episodes — nothing to do with &ldquo;one attempt&rdquo;).</p> <ul> <li><strong>Generalization gap</strong> = train return − test return. Always report <strong>two numbers</strong>: gap (overfitting) AND absolute test return (competence). Random agent: gap ≈ 0, useless.</li> <li><strong>Interpolation</strong> = test contexts inside the convex hull of training values. <strong>Extrapolation</strong> = outside. <strong>Combinatorial</strong> = novel combos of seen factor values. Interpolation is a favorable prior, not a proof — the test context was still never seen.</li> <li>Difficulty order (Kirk §4): interpolation &lt; combinatorial &lt; extrapolation. Extrapolation: &ldquo;unlikely to occur at all with standard RL methods.&rdquo;</li> <li>Why zero-shot matters: the only deployment workflow that exists — train in sim/offline, freeze, deploy, must work immediately. Every LLM deployment is zero-shot policy transfer.</li> </ul> <h2 id="2-cmdp-formalism">2. CMDP formalism</h2> <p>Tuple: S&rsquo; (underlying state), A, O, phi (observation fn), T, R, C, p(c), p(s&rsquo;|c). A CMDP = a POMDP with state (s&rsquo;, c): sample c ~ p(c) per episode, then s&rsquo; ~ p(s&rsquo;|c).</p>