Saheb Motiani

RL fine-tuning OLMo to withhold prohibited exam help

Writeup · Code and experiments

• Built a custom PyTorch/Transformers/PEFT GRPO trainer for OLMo-3-7B-Think-SFT with LoRA, vLLM rollouts and binary LLM-judge rewards for policy compliance and answer correctness; compared final-answer and whole-completion loss over 12 updates and 864 rollouts per run.
• Whole-completion loss raised withholding from 68/144 to 81/144 fresh responses to seen prompts. Without policy text, it never refused prohibited help or asked permission (0/24 each on development prompts), so gains remained policy-dependent. One run per loss scope; automated judgments.

TasteBento – graded LLM generation in a live app

Live app · Connects people through shared tastes in films, books, music and food

• Built hard checks for AI-generated facts and quizzes (format, grounding, non-triviality and quiz answers); an LLM judge with capped scoring weight reranked only candidates that passed.
• Shipped parallel best-of-N generation with near-duplicate suppression; logged prompts, candidates and scores to support later post-training.
• Audited 50 outputs from an earlier prompt: 48 passed hard validation checks, but 42 failed a soft specificity check for concrete details (dates, numbers or names). All 10 user-flagged outputs passed the hard checks, showing the hard checks missed what users disliked.

• Built notifications prompting non-admin users at high-value advertisers to appeal mistaken ad-account enforcements, unblocking $6.6M/year in revenue.
• Ran experiments showing broader appeal notifications increased harm, as bad actors exploited gaps in the appeal process. Worked with data scientists and the abuse-prevention team to limit the rollout to high-value advertisers.
• Fixed a multi-recipient concurrency bug, reducing notification failure rates by 70% for Facebook Pages, 30% for ad accounts and 8% for businesses; found the platform logging false-positive delivery successes.

• Designed a Python/Faust pattern for deploying ML risk-prediction models, with outputs consumed by clinical portals and patient-outreach tools.
• Built reusable Flink patterns for ETL, feature engineering and stream processing; the data platform team began rolling out Flink internally.
• Led an 8–10-person cross-functional team delivering a health-scoring product with Applied Scientists; developed Scala services for health-assessment and symptom-checker chatbot workflows.

• Designed distributed services to validate incoming media before transcoding and packaging, connecting on-premises infrastructure and AWS through asynchronous Kafka communication.

Orbit Wars – multi-agent self-play RL system

Kaggle placement: 1,060 / 4,729 · Code and results · Interactive replay

• Built a GPU-vectorized PyTorch simulator, entity-transformer PPO policy, self-play league and standalone export; measured ~4.1k end-to-end environment steps/s on RTX PRO 6000, including policy inference and PPO updates.
• Diagnosed an evaluator that counted opponent crashes as wins; added fail-closed checks, built fixed, seat-balanced evaluation panels and checked train/export parity.
• A post-competition policy won 98.0% against a mid-tier planner and 14.1% against the 13th-ranked agent (256 games each), but 0/32 against a top-10 agent.

Content Moderation OpenEnv – multi-step agent environment

Code and benchmark

• Built a deterministic OpenEnv moderation environment with four task packs and episodes up to 150 steps; implemented thread-context and policy-lookup tools, typed actions and reward shaping.
• Benchmarked four hosted LLMs in one-shot and multi-step tool-using modes; reported task scores separately from accumulated reward because models with equal scores earned different rewards.

Snake – board-size generalization

Study and code · Demo

• An attention-pooling agent with egocentric inputs, trained on 10×10 and 20×20 boards, achieved 97.7% moves toward food in zero-shot 100×100 play. Probes and training-size ablations pointed to training-state coverage as a bottleneck.