[{"content":"The question this post answers How would you actually build a robot foundation model, and what stops you?\nThe answer runs in one chain. Define the object by what it must do and where it must run. Decompose it into a brain and a body. Take the machine that produced large language models, state its three pillars precisely, and test each against robotics — robotics has one of them, plus a constraint text never faced. Five blockers fall out. Each blocker costs real engineering in a specific subsystem. That cost, ordered, is a build order.\nThe conclusion is stated up front so you can argue with it: the compute belongs on the robot, most of the hard parts are measurement and mechanism rather than modelling, and the first thing to build is not a model.\nHow to read this Language-model background — transformers, pretraining, scaling laws, arithmetic intensity — is taught in Part 3 and nowhere else. Skip it if you already know it; nothing before Part 3 depends on it [arXiv:2001.08361].\nThree conventions run throughout, and they are load-bearing.\nEvery claim is typed. Disclosed means a primary source states it. Inferred means we derived it from disclosed facts and say so. Undisclosed means nobody outside the lab knows, and it goes to the gap ledger rather than getting papered over. Marketing means a company blog or press release with no paper and no independent reproduction — labelled inline every time, because in this field the distance between a press release and a result is unusually wide [blog: NVIDIA GEAR GR00T N1.5].\nEvery number carries its reference, and is checked mechanically against a fact ledger. A build script fails if any number in the prose is absent from that ledger, or if any paragraph lacks a citation [arXiv:2604.15483 §V-E].\nWhere the evidence runs out, the post says so rather than reaching for a plausible sentence. Part 6 is explicitly speculative and marked as such [arXiv:2604.15483].\nPart 0 — Two numbers A bimanual UR5e that had never seen a single laundry demonstration folded a shirt at 85.6% task progress and 80% success [arXiv:2604.15483 §IX]. The model driving it had been trained on other robots doing other things.\nThe same task was given to ten expert human teleoperators, each with roughly 375 hours of experience, drawn from the top 2 percentile of operators. On their first attempt with that arm they scored 90.9% progress and 80.6% success [arXiv:2604.15483 §IX]. A generalist model with no task-specific data landed within a fraction of a point of expert humans meeting the hardware for the first time.\nThat is the first number. Here is the second: that model runs on an H100 [arXiv:2604.15483 App. D], and a policy trained to 100% success on one camera pose drops to 0% when the camera is moved 10 cm and rotated 20 degrees [arXiv:2409.03403].\nGenerality is real. Its fragility is mechanical.\nPart 1 — What a robot foundation model is Four axes, not one The shape of the thing, in one line: up to 4 camera images, the current joint configuration, and one sentence of English go in; a block of 50 future joint targets comes out, and your servo loop executes them [arXiv:2604.15483 §IV].\nThe one property to carry into the rest of this part: the sentence is the only thing you change to get a new behaviour [arXiv:2604.15483 §V]. The exact signature, and why the output is a block rather than a step, are in Part 2.\nThe term is used loosely enough to be worth pinning down. A robot foundation model is a system that satisfies four requirements at once, and the interesting thing is how unevenly the field satisfies them [arXiv:2108.07258].\nTask generalization — you prompt it into a new task, through the channel just described. Evaluated across 14 scenarios with 3 to 6 open-ended instructions each, in unseen kitchens and bedrooms [arXiv:2604.15483 §IX-B]. Embodiment generalization — you prompt it onto a new body. The same weights drive robots with different kinematics. Across 22 embodiments and 60 datasets, a cross-embodiment model beat per-dataset specialist methods by 50% on average [arXiv:2310.08864]. Human-grade interaction — it takes open-ended instruction and exposes what it intends to do. The π-series emits a human-readable subtask string refreshed every 4 seconds [arXiv:2604.15483 §VI], and explicit intermediate reasoning lifts task progress from 0.55 to 0.67 [arXiv:2510.03342]. Continual evolution — it improves from its own experience, adapts as the mechanism wears, and absorbs newly assigned work. This is the weakest axis by a wide margin: one published loop demonstrates self-improvement from autonomous rollouts [arXiv:2511.14759], and adaptation to mechanical wear is published by nobody [arXiv:2104.08212]. The fourth axis is the one that turns a model into a product, and it is the one nobody has shipped. Hold onto it; Part 6 comes back to it.\nThe ablation that separates a foundation model from a big policy \u0026ldquo;Trained on a lot of data\u0026rdquo; and \u0026ldquo;foundation model\u0026rdquo; are different claims, and one experiment separates them cleanly. Strip web pretraining out of a cross-embodiment model and it scores 0% on emergent skills and 1% on generalization; keep it, and the same architecture scores 48.7% and 47% [arXiv:2310.08864 Table II].\nThe semantics are inherited, not learned from robot data. Everything downstream in this post follows from that fact, because it is the only reason a robot model gets to start from something the internet already paid for [arXiv:2307.15818].\nWhich axes are actually demonstrated \u0026ldquo;Generalizes\u0026rdquo; does a lot of unearned work in robotics writing. The evidence is uneven across axes, and it is worth being specific.\nNew objects. Demonstrated across labs — roughly double the prior generation on unseen objects and backgrounds, across 280 tasks and 6 thousand evaluations [arXiv:2307.15818]. New scenes. The strongest single result in the field. Training on data from 3, then 12, 22, 53, 82, and finally 104 locations improved performance monotonically, and the 104-location model roughly matched a control model trained directly on the test homes [arXiv:2504.16054]. Cross-embodiment. Demonstrated with a documented ceiling: 75.8% against 27.3% on emergent skills [arXiv:2310.08864]. Long-horizon composition. Demonstrated at roughly 10 to 15 minute horizons, such as multi-stage room cleaning [arXiv:2504.16054]. Novel physical conditions — friction, payload, inertia, compliance, stiffness. This is the weakest axis, and no lab publishes a sweep of it [arXiv:2604.15483]. Lighting and clutter get measured; the mechanical variables do not. Two things should temper all of it, and you should hear them here rather than find them later. Independent work finds that state-of-the-art models exhibit lexical-kinematic shortcuts and semantic feature collapse, with static benchmarks masking the degradation [arXiv:2604.18000], and documents cases where vision simply overrides the language instruction [arXiv:2602.17659]. And the field\u0026rsquo;s leading lab concedes that at its data scale it is \u0026ldquo;practically difficult to definitively determine which tasks are truly seen or unseen,\u0026rdquo; and that the model \u0026ldquo;may well be achieving generalization primarily by remixing\u0026rdquo; skills from other situations [arXiv:2604.15483].\nWhere should the compute live? This is a product decision, and it is usually deferred. It should not be, because it determines what you are allowed to build. Two options: a central service that robots query over a network, or a model that ships on the machine.\nLatency. Off-board inference costs about 13 ms of Wi-Fi on top of 73 ms of model time, for 86 ms total [arXiv:2410.24164]. That is survivable — but only because of an algorithm built to survive it. Real-time chunking tolerates 100 ms and 200 ms of injected delay with no measurable degradation, and fails beyond 300 ms [arXiv:2506.07339]. Meanwhile a current-generation model with four cameras already sits at 300 ms on a single H100 [arXiv:2511.14759]. The network budget is being spent before the network is added.\nPower. An H100 draws 700 W [spec: NVIDIA H100 SXM]. A Jetson Thor draws 40 to 130 W [spec: NVIDIA Jetson AGX Thor]. For scale: a humanoid with an 842 Wh battery and a 4 hour runtime averages about 210 W for the entire machine [spec: 1X NEO], which puts an edge module at 19% to 62% of the whole power budget [computed: 40 to 130 W against 210 W]. An H100 on the robot is not a power question. It is an impossibility.\nCost per unit. An edge module is $3,499 [spec: NVIDIA Jetson AGX Thor] against a datacentre GPU, per robot, forever [spec: NVIDIA H100 SXM].\nFailure mode. A robot whose policy lives on a server stops being a robot when the link drops. This one is reasoning rather than a measured result, and it is stated as such — but it is the argument a buyer makes, not a researcher [arXiv:2506.07339].\nThe verdict is on-device, with an honest qualifier: today\u0026rsquo;s frontier models do not fit. The same model under the same runtime achieves 35.9 Hz on an H100, 10.7 Hz on a Jetson Thor, and 4.6 Hz on an AGX Orin [repo: Isaac-GR00T hardware_recommendation.md]. That gap is exactly why the compression work in Part 5 is load-bearing rather than optional.\nPart 2 — Two subsystems: brain and body The brain, and its contract Strip away the vocabulary and the brain is one function, called twenty to fifty times a second.\nIn: up to 4 camera images at 448×448, up to 6 frames of recent history, the current joint configuration, and one sentence of English [arXiv:2604.15483 §IV].\nOut: an action chunk — a block of H = 50 future joint targets, of which only the first 15 or 25 are executed before the model is called again [arXiv:2604.15483 §VI-B]. The loop runs at 50 Hz on most platforms and 20 Hz on the UR5e [arXiv:2604.15483 §VII].\nThat is the whole interface, and note where its edges fall: this is the brain\u0026rsquo;s boundary, not the machine\u0026rsquo;s. The model is a setpoint generator — it does not replace your servo loop, it feeds it [arXiv:2604.15483 §VII]. It predicts about a second of motion and commits to only a third of it, which is what buys the time to think [arXiv:2506.07339].\nChunking is not an implementation detail, and the ablation behind it is stark. Predicting a single step at a time yields 1% success on a fine bimanual task; predicting 100 steps at a time yields 44% [arXiv:2304.13705]. The mechanism is compounding error: a small deviation moves the robot to a state slightly outside the training distribution, where the next prediction is slightly worse. Committing to a block of motion divides the number of opportunities for that loop to diverge [arXiv:2304.13705].\nRead that input list again, because what it does not contain is the argument of the next two sections. What the brain must carry:\nVision, language, and proprioception. The three channels above, present in every shipped system [arXiv:2604.15483 §IV]. A representative platform exposes a 14-dimensional state [repo: openpi aloha_policy.py]. Reasoning. Not a slogan: an explicit intermediate representation that the rest of the system consumes. Refreshed every 4 seconds in the π-series [arXiv:2604.15483 §VI], worth 0.55 to 0.67 in task progress where it has been ablated [arXiv:2510.03342]. Force and touch. Measured by research systems and absent from every frontier model. Adding a force channel is worth 23.2% on average and takes plug insertion from about 10% to about 80% — from only 244 training trajectories [arXiv:2505.22159]. Tactile sensing takes USB insertion from 5% to 35%, charger insertion from 40% to 90%, and out-of-distribution wiping from 0% to 80% [arXiv:2507.09160]. Audio. No frontier robot foundation model publishes an audio input path. Named here as a gap rather than developed [arXiv:2604.15483]. Action generation. The representation choice, which Part 3 argues is the central axis [arXiv:2501.09747]. The body The requirements read like a hardware specification, because they are one.\nDexterity and fine control. A research-grade arm accepts commands at 1 kHz [spec: Franka Research 3]; the policy above it produces setpoints at 50 Hz [arXiv:2604.15483 §VII]. Contact physics lives in a band the model cannot reach, which is why the harness exists at all. Precision. Repeatability spans under 0.1 mm on a research arm [spec: Franka Research 3] to 1 mm on the low-cost arms much of the field actually trains on [spec: Trossen ViperX 300] — about a 10x spread across the platforms that share a dataset [computed: 1 mm against 0.1 mm]. Robustness and safety. A humanoid designed for human environments reports 95% backdrivability at 30 kg and a head-injury criterion under 250 [spec: 1X NEO]. The standards are explicit: reduced speed is capped at 250 mm/s at PL d and SIL 2 [std: ISO 10218:2025], and contact-force limits run from 65 N at the face to 220 N at the thighs [std: ISO/TS 15066:2016]. Maintenance cost. The field\u0026rsquo;s largest publication hole. Mean time between failures is undisclosed in every fleet-scale paper [arXiv:2104.08212], and cross-unit manufacturing variance is named and never quantified [arXiv:2506.18123]. The harness between them Nobody runs a 5B-parameter transformer inside a servo loop, so every serious system splits into halves that run at different rates: semantics at 1 to 10 Hz, setpoints at 20 to 50 Hz, servo at 200 to 1000 Hz [arXiv:2503.14734]. One published humanoid stack runs its three tiers at 7 to 9 Hz, 200 Hz, and 1 kHz [blog: Figure Helix] — marketing, with no paper behind it. A whole-body control stack runs its policy at 2.5 Hz against a 50 Hz controller [repo: Isaac-GR00T hardware_recommendation.md].\nWhere the labs genuinely disagree is the width of the seam. One system passes a single continuous latent vector between halves [blog: Figure Helix]; another cross-attends to a full sequence of intermediate features [arXiv:2503.14734]; the π-series passes a human-readable subtask string plus a generated image [arXiv:2604.15483 §VI]. That last choice costs bandwidth and buys debuggability: when the robot does the wrong thing, you can read what it thought it was doing.\nAnd the whole path — photon to torque — has never been published end to end, for any platform, by any lab [repo: openpi websocket_policy_server.py].\nPart 3 — The machine that worked, and where it does not transfer To understand why robotics is hard, it helps to understand precisely why language modelling turned out to be easy — not easy in engineering effort, but easy in the specific sense that spending resources on it reliably worked.\nThe philosophy Compress intelligence out of a dataset into weights, and buy progress with compute rather than with hand-designed features. The reason this is a strategy rather than a hope is that the returns are predictable. Loss falls as a power law in data, parameters, and compute, with exponents of 0.095, 0.076, and 0.050, fitted across 7 orders of magnitude [arXiv:2001.08361]. Ten times the data multiplies loss by 0.80; ten times the parameters, by 0.84 [arXiv:2001.08361].\nThat predictability is a budget instrument. It told one lab that a 70B model on 1.4T tokens would beat a 280B model on 300B tokens at equal compute, at roughly 20 tokens per parameter [arXiv:2203.15556]. Scaling laws are how you decide what to buy — which is why a field without one is a field that cannot plan.\nThree pillars made that machine run.\nPillar one — the data organization paradigm Everything is a token. One representation covers translation, summarization, code, and dialogue, which is what makes a single universal law findable rather than a per-task curve [arXiv:2001.08361].\nAnd the corpus already exists. One open pipeline distils 96 Common Crawl snapshots into 15T tokens and 44 TB of text for 1,536 GPU-hours [arXiv:2406.17557] — roughly $10k of compute [computed: 1,536 GPU-hours at commodity rates]. Nobody paid a person to produce that text. It was a by-product of the internet existing.\nEvaluation is nearly free on top of it: held-out perplexity, plus task outcomes that a machine can verify directly. The test cycle is minutes, and it scales with infrastructure rather than with people [arXiv:2001.08361]. And on-policy data is cheap because the sandbox is an API call.\nPillar two — the compute paradigm Here is the part most often stated wrongly. A \u0026ldquo;big model\u0026rdquo; is not one with many parameters. It is one with high arithmetic intensity — floating-point operations performed per byte moved from memory.\nEvery accelerator has a ridge point: peak throughput divided by memory bandwidth. Below it, the chip is waiting on memory and its headline throughput is fiction; above it, the chip is actually computing. The numbers below are derived from vendor datasheets.\ndevice peak bandwidth ridge point H100 SXM 3,958 TFLOPS FP8 3.35 TB/s about 1,181 FLOP/byte Jetson AGX Thor 1,035 TFLOPS dense FP4 273 GB/s about 3,791 FLOP/byte AGX Orin 275 TOPS sparse INT8 204.8 GB/s about 1,343 OP/byte The precisions differ across those rows, so this is not a like-for-like throughput comparison and should not be read as one [spec: NVIDIA H100 SXM]. What is comparable is the ratio each device demands of a workload. Thor\u0026rsquo;s ridge point sits about 3.2x further right than the H100\u0026rsquo;s [computed: 3,791 over 1,181], which means an edge part punishes a memory-bound workload considerably harder than its TOPS number suggests [spec: NVIDIA Jetson AGX Thor].\nThe measured consequence is visible in published timings: the same action expert costs 26.20 ms on Thor against 7.25 ms on a consumer GPU, a 3.6x penalty [arXiv:2602.18397]. Read the GB/s line, not the TOPS line.\nTransformers won because sequence modelling is general and the architecture is compute-dense enough to convert hardware into capability [arXiv:2001.08361]. Both halves of that sentence are necessary.\nPillar three — infrastructure Training at this scale is an engineering artifact in its own right: one 540B-parameter run sustained 46.2% model FLOPs utilisation across its hardware [arXiv:2204.02311], which requires thousands of accelerators and enough fault tolerance to survive weeks of them failing. Inference is the mirror image — low latency at high concurrency, plus test-time scaling and a harness that keeps long multi-turn calls correct [arXiv:2001.08361].\nThe scorecard Robotics satisfies pillar two. It does not satisfy pillars one and three in the forms that made them powerful, and it faces a fourth constraint that text never did: physics does not wait [arXiv:2604.15483 §VII].\nFive blockers This is the hinge of the argument. Each blocker is stated as a principle with the number that settles it, and Part 4 works out what each one costs.\nOne: there is no uniform action vocabulary. Text has one alphabet; robots do not. Action spaces are zero-padded to 18 or 19 dimensions merely to be concatenated into one training mixture [arXiv:2410.24164]. Control rates span 3 Hz to 50 Hz across the standard corpora [arXiv:2310.08864], so an identical 50-step chunk means 1 second of motion on one robot and 2.5 seconds on another [computed: 50 steps at 50 Hz and 20 Hz]. Even the tokenizer\u0026rsquo;s efficiency depends on the body: the same compression scheme yields 1.75x at 5 Hz and 7 degrees of freedom, and 13.2x at 50 Hz and 14 [arXiv:2501.09747]. And the one replicated robotics scaling law is over diversity, not volume — normalized score improves as 0.88 times pairs to the power −0.30, while demonstrations per pair saturate around 800 total [arXiv:2410.18647].\nTwo: there is no unsupervised corpus. The closest analogue to web text is egocentric human video, and the two largest public corpora total 4,956 hours [computed: 3,670 h plus 1,286 h] against web text\u0026rsquo;s 15T tokens [arXiv:2110.07058]. Robot data is manufactured: one pretraining mixture represents about 10,000 hours and 903M timesteps [arXiv:2410.24164], roughly $500k of human labour at a fully-loaded $50 per hour [computed: 10,000 h at $50 per hour]. Another required 100 robots in a 4,000 m² facility to produce 1,001,552 episodes and 2,976.4 hours [arXiv:2503.06669]. A third took 18 rigs, 50 collectors, and 12 months for 76k trajectories [arXiv:2403.12945].\nThree: the compute paradigm transfers, but the extra compute buys nothing on the axis that matters. Loading a vision-language checkpoint and attaching an action head is a genuine transfer, and the web-pretraining ablation in Part 1 proves it. But holding the backbone fixed and varying only the action head, diffusion at 50 denoising steps yields 10.1 Hz at 95.4% success, where continuous regression yields 109.7 Hz at 95.3% [arXiv:2502.19645] — about 11x the throughput for a tenth of a point [computed: 109.7 Hz against 10.1 Hz]. Spending an order of magnitude more compute to gain nothing measurable is the opposite of a scaling law.\nFour: training infrastructure is not the bottleneck; on-device adaptation might be. No robotics result is compute-bound at language-model scale. The largest published compute spend in the field is data generation — 240k samples over 54 hours on 1,500 GPUs [arXiv:2505.12705]. The open question sits at the other end: adaptation on the machine, where an on-device model has been fine-tuned to new tasks from 50 to 100 demonstrations [arXiv:2503.20020].\nFive: inference means a real robot. A language harness that stalls costs a slow answer. A policy that stalls commands a stale trajectory into a world that has moved. That is why real-time chunking exists, why its 100–200 ms tolerance and 300 ms ceiling are load-bearing numbers [arXiv:2506.07339], and why photon-to-torque latency going unpublished by every lab is a measurement failure rather than an oversight [repo: openpi websocket_policy_server.py].\nOne of those five is a modelling problem. The other four are measurement, mechanism, and data-engine problems.\nPart 4 — What that costs you, subsystem by subsystem Six cells. Each one is the local, engineering-level form of a blocker from Part 3.\nBrain / evaluation — from blocker one Build the instrument before the thing it measures, because right now the instrument does not exist.\nStart with the arithmetic nobody runs. Separating a 50% policy from a 60% policy at conventional significance and power requires 387 trials per policy; separating 50% from 55% requires 1,565; even separating 80% from 90% requires 199 [computed: two-proportion test, alpha 0.05, 80% power]. The field runs 10 to 60 [arXiv:2506.18123] — one influential paper reports 10 trials per task per policy [arXiv:2504.16054], another about 125 real rollouts in total [arXiv:2503.14734]. A policy measured at 60% over 20 trials has a 95% interval of 0.39 to 0.78 [computed: Wilson score interval]. That is not a measurement.\nThe consequence is quantified. Across a large audit, only 19.8% of LIBERO claims and 19.7% of SimplerEnv claims were statistically significant [arXiv:2606.04233]. The same work found a 90M-parameter language-free probe reaching 99.0, 100, 98.8 and 92.4% across the four LIBERO suites — a benchmark that a model can pass without reading the instruction is not measuring instruction following [arXiv:2606.04233].\nAnd the benchmarks are sensitive to precisely the variables a hardware team controls. Under a change of robot initial state, one model drops 87.6 points and another 59.9; under a camera viewpoint change, 78.4 and 37.4 [arXiv:2510.13626]. Under position perturbation, models scoring 98% and 92% on standard LIBERO collapse to 0.0% [arXiv:2510.03827]. On a real-robot benchmark, the same task scored anywhere from 0% to 100% depending on the operator [arXiv:2510.17950].\nWhat exists is worth using. Real-to-sim correlation reaches Pearson 0.924 on a curated suite [arXiv:2405.05941]; distributed evaluation across 7 institutions and 4,284 episodes converges after about 100 pairwise comparisons [arXiv:2506.18123]; adaptive trial allocation cuts trials by up to 70% [arXiv:2603.13616]; and a world-model evaluator reports 0.989 rank correlation with real evaluation for 100 H100-hours of replication cost [arXiv:2607.01060].\nBrain / training — from blockers two and three The data does not exist and cannot be scraped, so it has to be engineered. The good news is that the one replicated law tells you what to buy.\nBuy diversity, not volume. The verified recipe is 32 environment-object pairs at 50 demonstrations each, which reached 85% to 92.5% in unseen environments and was collected by 4 people in a single afternoon [arXiv:2410.18647]. The same study, spanning over 40,000 demonstrations and 15,000 real rollouts, found demonstrations per pair saturating around 800 total [arXiv:2410.18647].\nMixture weights matter more than they look: learned mixture weights beat uniform weighting by 38% and human-chosen weights by 32% [arXiv:2408.14037], and a curation method matched full-data performance on under 33% of the data [arXiv:2506.19121]. Heterogeneity is the tax you pay for a shared corpus — 27% wrist-camera and 56% language coverage across a standard open mixture [arXiv:2405.12213], plus the padding and rate spreads from blocker one. One approach compresses every embodiment to 16 fixed tokens across 52 datasets and about 200k trajectories, for a 20% gain [arXiv:2409.20537].\nAgainst a meta-analysis of 327 papers, robotics fits data, model, and compute exponents of 0.276, 0.246, and 0.141 [arXiv:2405.14005] — steeper than the language-model exponents, but fitted over about three orders of magnitude rather than seven.\nBrain / deployment — from blockers three and five Compression has a cliff, and it is not gentle. One method holds 94.8% at 3.0 bits per weight and falls to 48.0% at 2.0 [arXiv:2605.24011]. Robotics-specific quantization is not optional: a W4A8 scheme reaches 97.6% against FP16\u0026rsquo;s 97.1% while cutting the model from 4.27 GB to 1.28 GB [arXiv:2602.20309], where generic language-model post-training quantization applied to the same policy lands at 76.3% — a 21-point drop [arXiv:2602.20309].\nAlgorithm-and-runtime co-design is where the leverage is. Compilation buys 1.5 to 2.9x and an optimised inference pipeline 1.5 to 3.3x [repo: Isaac-GR00T hardware_recommendation.md]. A single-step flow variant takes one policy from 274 ms to 83 ms while improving success from 97.75% to 98.75% [arXiv:2604.05656]. Chunking plus continuous actions took one system from 4.2 Hz to 109.7 Hz and 76.5% to 97.1% — a 26x speedup [arXiv:2502.19645].\nTreat the published record with suspicion. Latency figures for the same model on the same edge part span 46 ms to 246 ms across sources, a 5x spread [arXiv:2602.18397].\nBody / hardware design — from blockers one and five Calibration is a first-order accuracy term, not a maintenance chore. One public dataset shipped improved camera calibrations for 36,000 of its episodes after release [spec: DROID dataset site]. A policy at 100% success collapses to 0% under a 10 cm and 20 degree camera shift [arXiv:2409.03403]. Mounting rigidity and repeatability set a floor the model cannot rise above [spec: Franka Research 3].\nSensing is the other half. Tactile skins that survive replacement matter more than tactile skins that are precise: one design costs 13% in cross-instance performance where a baseline costs 43%, with normalized cross-instance variation of 0.12 against 0.54, and swaps in 12 seconds against 236 ± 64 [arXiv:2409.08276]. A tactile pretraining corpus of 460k images exists [arXiv:2410.24090]. None of it is wired into a frontier model.\nBody / collection pipeline — from blocker two The rig ladder is the clearest cost-benefit table in the field. A handheld station costs $371 — $73 of gripper and $298 of camera — and produces 111 demonstrations per hour against teleoperation\u0026rsquo;s 35, a 3.2x throughput multiple, reaching 71.7% in novel environments and transferring to a research arm at 90% [arXiv:2402.10329]. A low-cost teleoperation interface costs under $300 [arXiv:2309.13037]; an exoskeleton rig costs $0.6k against the roughly $60k platform it replaces [arXiv:2503.03081]; a bimanual station costs under $20k [arXiv:2304.13705] and its mobile version $32k [arXiv:2401.02117].\nOperator economics decide the rest. Disclosed rates run $25.25 to $48.00 per hour at one Western firm [blog: Tesla job posting via Fortune], against about $3 per hour reported at Chinese collection centres [blog: Rest of World] — both marketing-grade sources, labelled as such.\nHuman video is the cheapest source per unit of learning: 1 hour of human video is worth about 1,400 demonstrations where 1 hour of robot time is worth 135 [arXiv:2410.24221]. And it works — laundry success of 88% against 55%, and 85% against 25% on unseen cloth colours [arXiv:2410.24221], with a related method adding 44 points absolute [arXiv:2509.19626]. Simulation and generative augmentation sit above it: real plus sim reached 24.4% and 9.3% where real-only reached 13.6% and 2.6% [arXiv:2406.02523]; one pipeline amplified under 200 demonstrations into over 50,000 [arXiv:2310.17596]; a generative pipeline took novel behaviours in novel environments from 0.0% to 28.5% [arXiv:2505.12705].\nBody / runtime shell — from blocker five The model\u0026rsquo;s own time is the measured third of the problem: 14 ms of image encoders, 32 ms of prefix pass, 27 ms of flow steps [arXiv:2410.24164]. Exposure, transport, preprocessing, transforms, and actuator response are unmeasured by the entire field [repo: openpi websocket_policy_server.py]. A published budget tolerates 0 to 12 timesteps of delay, which at 50 Hz is a 240 ms window [computed: 12 timesteps at 50 Hz].\nSafety is architectural rather than learned. No neural policy will carry a Performance Level rating, so the filter sits outside the policy: a reachability-based safety filter runs at 100 Hz and has been validated over 66 hardware trajectories [arXiv:2410.11157]. The standards moved in 2025 for the first time since 2011 [std: ISO 10218:2025], and a dynamically stable mobile robot standard is still only a Working Draft [std: ISO 25785-1].\nPart 5 — The build order Everything so far has been diagnosis. This part is the answer, and it is the part to argue with, because it is the only one that commits to anything.\nEach earlier section left a hook. Here is where each one gets picked up:\nwhat was left open where it is answered the fourth axis — continual evolution — has one published loop M3 the on-device verdict, with today\u0026rsquo;s models not fitting M4 force and touch measured by the body, consumed by no model M1, recorded from day one blocker 1: no uniform action vocabulary M2, the action-token interface blocker 2: no unsupervised corpus M1, the data engine blocker 3: the extra compute buys nothing M2, the action-head choice blocker 4: on-device adaptation, not training scale M3 and M4 blocker 5: inference means a real robot M0\u0026rsquo;s latency budget, M4\u0026rsquo;s runtime all six cells of the difficulty grid M0 through M5, one cell each Six milestones, each with a gate expressed as a number so it can fail. The ordering principle is that you build the instrument before the thing it measures [arXiv:2506.18123].\nWhere a milestone has been executed publicly at frontier scale, this part describes what was actually built rather than what would be nice — the π-series is the only fully documented instance, and the figures below are carried over from a line-by-line read of it [arXiv:2604.15483].\nBorrowing the recipe, stated once The language-model machine has three stages, and all three transfer in form. What changes is the third, and the reason is blocker two: there is no cheap verifier, so the reward has to come from the robot\u0026rsquo;s own experience [arXiv:2511.14759].\nstage in language in robotics what breaks pretrain next token on a web corpus one generalist across every robot and task, from a VLM checkpoint the corpus does not exist and must be manufactured [arXiv:2410.24164] post-train instruction tuning on curated data condition on subgoals and episode metadata; augment away from the rig supervision is a demonstration, not a preference [arXiv:2604.15483 §V-E] RL preference optimization against a reward model advantage-conditioned policy over the robot\u0026rsquo;s own rollouts no verifier; the reward is sparse and rollouts cost robot-hours [arXiv:2511.14759] M0 — build the instrument You cannot post-train against a measurement you do not trust, and the field\u0026rsquo;s own audit says four comparisons in five are not significant [arXiv:2606.04233].\nWhat to build. A harness that resolves a 10-point difference in fewer than 387 trials [computed: two-proportion test, alpha 0.05, 80% power], combining three things that already exist: sequential testing, a real-to-sim proxy validated at Pearson 0.924 [arXiv:2405.05941], and adaptive trial allocation worth up to 70% of the trials [arXiv:2603.13616]. Distributed pairwise comparison converges after about 100 comparisons if you have partners [arXiv:2506.18123].\nWhat to publish. A photon-to-torque budget for your own platform, because no lab has published one for any platform [repo: openpi websocket_policy_server.py]. You need it in M4 anyway, and measuring it now is the difference between an engineering target and a hope.\nGate. The harness separates two policies you already know to differ, at the trial count above [computed: two-proportion test, alpha 0.05, 80% power]. Closes brain/evaluation.\nM1 — build the data engine It is a flywheel, not a dataset The frontier system draws on 8 sources feeding one annotated mixture [arXiv:2604.15483 §VI-A]. Three of them are worth naming because they are the ones a team building a static dataset never thinks to collect: autonomous evaluation rollouts including failures, human interventions taken inside those rollouts, and data collected during RL training, which the lab describes as a kind of distillation [arXiv:2511.14759]. The deployed policy\u0026rsquo;s rollouts become tomorrow\u0026rsquo;s training data. Build the loop, not the corpus.\nTwo disciplines make the loop safe rather than degenerate.\nKeep the bad data — on purpose. Failures, successes containing mistakes, and older model versions\u0026rsquo; evaluation data are retained deliberately [arXiv:2604.15483 §VI-A]. This inverts the usual instinct to filter, and it only works because of the next section: metadata is what makes keeping it safe.\nExclude evaluation data from training. Autonomous data collected in any generalization-focused evaluation task is excluded from the mixture [arXiv:2604.15483 §VI-A]. Without that exclusion the flywheel quietly trains on the test set, and every number M0 produces becomes fiction.\nWhat one recorded episode has to contain Every episode is stored as a conditioned example rather than a raw trajectory, and the conditioning is plain text in the prompt [arXiv:2604.15483 §V-E]:\n\u0026lt;multi-view observation\u0026gt;\u0026lt;multi-view subgoals\u0026gt; Task: peel vegetables. Subtask: pick up the peeler. Speed: 8000. Quality: 5. Mistake: false. Control Mode: joint. \u0026lt;proprioception\u0026gt; Speed is episode length in timesteps, binned at 500-step intervals [arXiv:2604.15483 §V-C]. Quality is a human integer from 1 to 5 [arXiv:2604.15483 §V-C]. Mistake is a coarse per-segment boolean [arXiv:2604.15483 §V-C]. Control mode distinguishes joint-space from end-effector commands, and is the one field never dropped [arXiv:2604.15483 §V-D].\nThis is the whole idea in one string: condition on how the episode was produced, not only what was done. Because the model trains conditioned on these fields, they become knobs at inference — prompt quality 5 and mistake false and the policy imitates the good half of your data [arXiv:2604.15483 §VII]. Mediocre demonstrations stop being contamination and start being contrast, which is exactly what makes \u0026ldquo;keep the bad data\u0026rdquo; a strategy rather than negligence.\nRecord the wrench channel and the metadata from day one. Retrofitting either means recollecting the corpus, and force is worth 23.2% on average where it has been added [arXiv:2505.22159].\nWhere the episodes come from Start at the recipe with a replicated law behind it: 32 environment-object pairs at 50 demonstrations each, which reached 85% to 92.5% in unseen environments and was collected by 4 people in an afternoon [arXiv:2410.18647]. Four handheld stations at $371 each is $1,484 of capital [computed: 4 stations at $371]. Layer human video in immediately — an hour of it is worth about 1,400 demonstrations against a robot-hour\u0026rsquo;s 135 [arXiv:2410.24221]. Add simulation last, where real plus sim reached 24.4% and 9.3% against real-only\u0026rsquo;s 13.6% and 2.6% [arXiv:2406.02523].\nGate. Diversity, not volume: the count that matters is pairs, and demonstrations per pair saturate around 800 total [arXiv:2410.18647]. Closes body/collection and half of brain/training.\nM2 — the model, parameterized The stack, and where each part sits About 5B parameters on the control path: camera views enter a 400M SigLIP-class vision encoder, pass through a history encoder that applies temporal and spatial compression, and reach a 4B Gemma-3 backbone; an 860M flow-matching action expert reads the backbone\u0026rsquo;s activations and emits the chunk [arXiv:2604.15483 §IV]. Proprioception is a linear projection, one token per history state [arXiv:2604.15483 §VI-B]. A separate 14B world model — 7B understanding plus 7B generation — runs beside the control path rather than inside it, producing up to 3 subgoal images [arXiv:2604.15483 App. C].\nThe lineage matters as much as the snapshot, because it shows which changes actually paid:\nThe firewall, which is the part to copy A FAST-token cross-entropy head trains on the backbone while flow matching trains the expert, and the two are coupled by a stop-gradient: gradients from the action expert do not flow back into the backbone [arXiv:2604.15483 §III]. The discrete head shapes the backbone\u0026rsquo;s representation without the continuous head dragging it around. This is knowledge insulation, and it is the mechanism that lets a model absorb web semantics and motor skill at once without one erasing the other.\nThe attention pattern enforces the same separation. Exactly 50 action tokens, bidirectional among themselves, attending backbone activations; block-causal elsewhere; and FAST tokens and flow actions never attend to each other [arXiv:2604.15483 App. B]. FAST tokens exist only at training time — at inference the discrete branch is simply gone [arXiv:2604.15483 App. B].\nThis is also the answer to blocker three. The action head is where teams reach for diffusion because it is fashionable; the measured trade is 10.1 Hz at 95.4% against 109.7 Hz at 95.3% [arXiv:2502.19645]. Spend the compute on the backbone, not on denoising steps.\nMemory, for free History is six observations at a 1 second stride [arXiv:2603.03596]. The encoder that consumes them adds zero new learnable parameters over a single-image ViT and passes on the same token count as a memoryless VLA — the memory comes from the attention pattern, not from capacity [arXiv:2603.03596]. Post-training it expands to 18 frames and 54 seconds [arXiv:2603.03596].\nGate. Budget the 70 GB VRAM floor for a full fine-tune [repo: openpi README], and beat the incumbent at significance on the M0 harness. Closes brain/training.\nM3 — pretrain, post-train, then RL This is the milestone a hardware team hands to a learning team, so it is written as an algorithm rather than as a plan: what the objects are, what is being maximized, what one training step does line by line, and what happens at inference [arXiv:2604.15483 §III].\nNotation, fixed once symbol what it is instantiated as $o_t$ observation at time $t$ up to 4 images at 448×448 plus the joint configuration [arXiv:2604.15483 §IV] $o_{t-T:t}$ observation history 6 frames at a 1 s stride [arXiv:2603.03596] $C_t$ context subtask string, speed, quality, mistake, control mode, optional subgoal images [arXiv:2604.15483 §V-C] $A_t = a_{t:t+H}$ action chunk $H = 50$ future joint targets [arXiv:2604.15483 §IV] $\\tilde{H}$ executed prefix 15 or 25 of the 50 before re-planning [arXiv:2604.15483 §VII] $\\theta$ backbone parameters 400M vision encoder inside a 4B Gemma-3 [arXiv:2604.15483 §IV] $\\phi$ action-expert parameters 860M flow-matching transformer [arXiv:2604.15483 §IV] $\\omega$ value-model parameters 670M, and only during RL [arXiv:2511.14759] The whole pipeline maximizes one quantity, and every stage below is a different way of getting at it [arXiv:2604.15483 §III]:\n$$\\max_{\\theta,\\,\\phi}\\ \\ \\mathbb{E}_{(o,\\,C,\\,A)\\sim\\mathcal{D}}\\Big[\\log \\pi_{\\theta,\\phi}\\big(A_t \\mid o_{t-T:t},\\ C_t\\big)\\Big]$$One caveat to carry through the rest of this part, because it is what breaks the LLM analogy at stage three: the flow-matching expert optimizes an approximate bound on that log-likelihood rather than the quantity itself, so there is no tractable likelihood to differentiate later [arXiv:2604.15483 §III].\nTwo objectives, one firewall Primer — flow matching. A flow-matching model learns a velocity field that transports a noise sample to a data sample along a path indexed by a flow time. Training regresses that velocity at random points on the path; sampling integrates it. It is the continuous-action analogue of diffusion, with a straighter path and therefore fewer integration steps.\nThe discrete branch trains $\\theta$. FAST compresses the chunk with a discrete cosine transform, quantizes it, and byte-pair-encodes the result into tokens the language model already has, so the action objective speaks the backbone\u0026rsquo;s native language [arXiv:2501.09747]:\n$$y = \\mathrm{BPE}\\big(\\mathrm{quant}(\\mathrm{DCT}(A_t))\\big),\\qquad \\mathcal{L}_{\\mathrm{CE}}(\\theta) \\;=\\; -\\sum_{k}\\log p_{\\theta}\\big(y_k \\mid y_{\\lt k},\\ h_{\\theta}(o_{t-T:t}, C_t)\\big)$$That is ordinary next-token cross-entropy against the loss the backbone was pretrained on, which is the entire reason it does not forget what a colander is [arXiv:2604.15483 §III].\nThe continuous branch trains $\\phi$. Draw noise $\\epsilon \\sim \\mathcal{N}(0, I)$ and a flow time $\\tau$ weighted toward the noisy end of the path, interpolate, and regress the velocity that carries noise to actions [arXiv:2410.24164]:\n$$A^{\\tau} = \\tau A_t + (1-\\tau)\\,\\epsilon, \\qquad \\mathcal{L}_{\\mathrm{FM}}(\\phi) \\;=\\; \\mathbb{E}_{\\tau,\\,\\epsilon}\\Big\\|\\,v_{\\phi}\\big(A^{\\tau},\\ \\tau,\\ \\mathrm{sg}[\\,h_{\\theta}\\,]\\big) \\;-\\; (A_t - \\epsilon)\\,\\Big\\|^{2}$$Signs here follow the convention where $\\tau = 0$ is noise and $\\tau = 1$ is the action chunk; the published statement uses the opposite orientation and is otherwise identical [arXiv:2410.24164].\nThe coupling is one operator. $\\mathrm{sg}[\\cdot]$ is the stop-gradient, and it is the whole firewall: the expert reads every backbone activation and modifies none of them [arXiv:2604.15483 §III]. The total loss is $\\mathcal{L}_{\\mathrm{CE}} + \\lambda\\,\\mathcal{L}_{\\mathrm{FM}}$, and $\\lambda$ is the one number in this section nobody has published — it goes in the gap ledger rather than into a confident sentence [arXiv:2604.15483 §III].\nOne training step, line by line 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 for o, C, A in mixture: # M1\u0026#39;s flywheel, resampled every batch C = drop_fields(C) # rates in stage two; makes every field optional h = backbone(o, C) # theta: 400M SigLIP + MEM history + 4B Gemma-3 # discrete branch -- trains theta, in the language the backbone already speaks y = fast_tokenize(A) # DCT -\u0026gt; quantize -\u0026gt; BPE loss_ce = cross_entropy(fast_head(h), y) # continuous branch -- trains phi, and only phi tau = sample_flow_time() # weighted toward the noisy end eps = randn_like(A) A_tau = tau * A + (1 - tau) * eps v = action_expert(A_tau, tau, stop_gradient(h)) # \u0026lt;-- the firewall, one call loss_fm = mse(v, A - eps) (loss_ce + lam * loss_fm).backward() opt.step() Two lines carry the design. stop_gradient(h) is knowledge insulation, and deleting it is the single most expensive one-character change available to you — the high-variance regression gradient reaches the backbone and erases the web semantics you attached it for [arXiv:2604.15483 §III]. And fast_head(h) without a stop-gradient is what keeps the backbone learning about actions at all; remove that instead and you have a frozen VLM with a policy bolted on [arXiv:2501.09747].\nThe attention mask is the third control, and it is a correctness requirement rather than an optimization: the FAST tokens are the quantized answer, so an expert able to attend to them would learn to copy rather than to look, and would collapse at deployment where they do not exist [arXiv:2604.15483 App. B].\nInference, line by line 1 2 3 4 5 6 7 8 h = backbone(o, C_runtime) # quality=5, mistake=false, speed=15th percentile kv = cache(h) # actions attend the prefix; the prefix never attends back A = randn(50, action_dim) # tau = 0: pure noise for i in range(5): # delta = 1/5, forward Euler A = A + (1 / 5) * action_expert(A, i / 5, kv) execute(A[:25]) # or A[:15]; then re-observe and repeat The prefix is encoded once and reused across all 5 denoising steps, which is a direct dividend of the mask: because the prefix does not attend to the action tokens, nothing in it changes as they are refined [arXiv:2604.15483 §VII]. One masking decision buys both correctness and roughly the cost of the pass.\nThe metadata fields are now knobs rather than annotations, and guidance sharpens them [arXiv:2604.15483 §VII]:\n$$v \\;=\\; v_{\\phi}(\\,\\cdot \\mid \\varnothing\\,) \\;+\\; w\\,\\big(v_{\\phi}(\\,\\cdot \\mid C\\,) - v_{\\phi}(\\,\\cdot \\mid \\varnothing\\,)\\big),\\qquad w \\in \\{1.3,\\ 1.7,\\ 2.2\\}$$Naively that is two forward passes. It is implemented as one, by packing the conditioned and unconditioned branches into the same sequence as an attention tree whose branches do not attend to each other [arXiv:2604.15483 App. B].\nStage one — pretraining One generalist over every robot and every task in the mixture, initialized from the vision-language checkpoint and trained with both objectives above [arXiv:2604.15483 §III]. Nothing here is task-specific and nothing is embodiment-specific: the action space is padded to a common width so that heterogeneous robots share one head, which is blocker one showing up as an implementation detail [arXiv:2410.24164].\nThe point of the stage is the shared representation that makes M5\u0026rsquo;s second embodiment cheap, and the ablation from Part 1 is what justifies paying for it at all — strip web pretraining and generalization goes to 1% [arXiv:2310.08864 Table II].\nStage two — post-training This stage is where the model is taught that its own context is unreliable [arXiv:2604.15483 §V-E]. Formally, the context is corrupted before every forward pass,\n$$\\tilde{C} = D_{\\rho}(C), \\qquad \\Pr[\\,\\text{field } j \\text{ survives}\\,] = 1 - \\rho_j$$and the rates are disclosed in full, which is unusual enough to be worth copying verbatim [arXiv:2604.15483 §V-E, §VI-B]:\nwhat is dropped rate source the entire observation history $p = 0.3$ [arXiv:2604.15483 §VI-B] the rear camera view $p = 0.3$ [arXiv:2604.15483 §VI-B] all episode metadata at once 15% [arXiv:2604.15483 §V-E] each metadata field, additionally 5% [arXiv:2604.15483 §V-E] the subtask string, inside subgoal examples 30% [arXiv:2604.15483 §V-E] subgoal images present at all in 25% of the batch [arXiv:2604.15483 §V-E] Real subgoals are sampled as $p = 0.25$ end-of-segment and $p = 0.75$ uniform up to 4 seconds ahead [arXiv:2604.15483 §VI-C]. Training also simulates inference delay of 0 to 12 timesteps, a 240 ms budget at 50 Hz, which costs nothing at inference because it was paid for in training [computed: 12 timesteps at 50 Hz].\nDropout here is not regularization in the usual sense, and reading it that way is how teams talk themselves out of it. It is what makes every field optional at test time — omit the subtask string, or the metadata, or the history, and you still get a policy [arXiv:2604.15483 §V-E]. It is also what stops the policy binding to a fixed camera rig, the failure Part 4 quantified at 100% to 0% [arXiv:2409.03403].\nTwo rates deserve their own explanation. The subgoal share is capped at a quarter because with a subgoal image present the objective degenerates toward inverse dynamics and trains much faster — left uncapped, the model learns to read the subgoal instead of the scene [arXiv:2604.15483 §V-E]. And control mode is the one field never dropped, because a joint-space command and an end-effector command are different instructions wearing the same numbers [arXiv:2604.15483 §V-D].\nStage three — RL, which is where the recipe stops fitting Two things break at once. There is no cheap verifier — \u0026ldquo;is the shirt folded?\u0026rdquo; has no assertion to call — and the flow-matching policy has no tractable log-likelihood, so a policy gradient has nothing to differentiate [arXiv:2511.14759]. The published comparison is blunt about where that leads: PPO and AWR both underperformed the method below, with PPO struggling in the off-policy regime that real-robot data forces on you [arXiv:2511.14759].\nDefinition — the reward. Deliberately trivial, so that it ports to any task without a per-task verifier [arXiv:2511.14759]:\n$$r_t = \\begin{cases} 0 \u0026 \\text{terminal step, success} \\\\ -c \u0026 \\text{terminal step, failure} \\\\ -1 \u0026 \\text{otherwise} \\end{cases} \\qquad\\quad G_t = \\sum_{k \\ge t} r_k$$Values are then normalized per task by maximum episode length into the interval $(-1, 0)$, which makes the value function readable as \u0026ldquo;how much of this episode is still left, negated\u0026rdquo; [arXiv:2511.14759]. The failure constant $c$ is described only as large, and goes in the gap ledger [arXiv:2511.14759].\nDefinition — the critic. A distributional value model with the same architecture as the policy on a smaller 670M backbone, trained by cross-entropy over 201 bins of discretized Monte-Carlo return [arXiv:2511.14759]:\n$$\\mathcal{L}_{V}(\\omega) \\;=\\; -\\,\\mathbb{E}\\Big[\\log p_{\\omega}\\big(\\mathrm{bin}(G_t)\\ \\big|\\ o_t,\\ \\ell\\big)\\Big]$$Distributional rather than scalar because the return distribution over a real fleet is wide and multi-modal, and a mean would sit in a part of it that never happens [arXiv:2511.14759].\nDefinition — the advantage, and its binarization. The estimator is the one-step difference of the critic, and the threshold is set from the critic\u0026rsquo;s own predictions rather than from zero [arXiv:2511.14759]:\n$$\\hat{A}_t \\;=\\; V_{\\omega}(o_{t+1}) - V_{\\omega}(o_t), \\qquad z_t \\;=\\; \\mathbf{1}\\{\\hat{A}_t \u003e \\varepsilon_{\\ell}\\}, \\qquad \\varepsilon_{\\ell} = \\text{30th percentile for task } \\ell$$Then the move that makes it work. Rather than updating the policy toward high advantage, $z_t$ is rendered as plain text — Advantage: positive — and inserted after the language input and before the actions, so that only action log-likelihoods are affected [arXiv:2511.14759]. Training then proceeds as ordinary supervised learning over the entire dataset, failures included. Reinforcement learning has been converted back into conditional imitation, which is precisely the trick that made \u0026ldquo;keep the bad data\u0026rdquo; safe in M1, applied one level up [arXiv:2604.15483 §V-C].\nThe method has three subroutines — collection with optional corrective interventions, value-function training, and advantage-conditioned policy training — and the loop below is those three in sequence [arXiv:2511.14759 §IV-C, Alg. 1]:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 def recap(task): # ONE call per task; k indexes rounds, not tasks pi = pi_pretrained # round 0 has nothing else to roll out D = seed_demonstrations # scope undisclosed; see below for k in range(K): rollouts = run(pi, task, z=\u0026#34;positive\u0026#34;, human_intervention=True) # ~300 trajectories label_terminal_outcome(rollouts) # success / failure, nothing finer D += rollouts # failures kept, on purpose omega = fit_value(D_every_task) # multi-task: 670M, 201 bins, MC returns eps = percentile([V(o) for o in D if o.task == task], 30) # eps_ell, per task for (o, o_next) in D: z[o] = \u0026#34;positive\u0026#34; if V(o_next) - V(o) \u0026gt; eps else \u0026#34;negative\u0026#34; for seg in human_interventions(D): z[seg] = \u0026#34;positive\u0026#34; # an expert correction is a good action z = randomly_omit(z) # leaves guidance available at inference pi = finetune(pi_pretrained, D, condition=z) # re-init from the BASE, never from pi return pi # a specialist for `task`, and nothing else Read k as a round, not as a task. task is the argument, fixed for the whole call, so a policy only ever rolls out on the task it was just trained on. Improving a second task means a second call, which starts again from pi_pretrained — the two loops never hand a policy to each other. The only thing they share is the critic, which is fitted across everything [arXiv:2511.14759 §V-C]. Specialists therefore cannot contaminate one another, and that falls out of the re-initialization rule rather than being arranged separately [arXiv:2511.14759 §V-D].\nWhere pi comes from, and the one job it keeps. At $k = 0$ there is no previous round, so the collector is the pre-trained checkpoint itself; there is nothing else to roll out [arXiv:2511.14759 §IV-C]. After that, each round\u0026rsquo;s output becomes the next round\u0026rsquo;s collector on that same task, run conditioned on the positive indicator, which is how you ask a conditional policy for its good behavior [arXiv:2511.14759 §V-B].\nNow look at what the last line does not do: pi never appears on the right-hand side of finetune. The policy carries forward as the thing that gathers data, and never as the thing that gets initialized from. That is the entire content of \u0026ldquo;never from the previous iteration\u0026rdquo; — a statement about initialization, not about deployment. Both readings of pi are in the loop simultaneously, and keeping them apart is what stops the rounds from compounding [arXiv:2511.14759 §V-D].\nTwo scopes run at different widths here, and conflating them is the fastest way to misread the algorithm. Because that distinction is also where the source stops being explicit, it is worth separating what is stated from what this post is reading into it.\nStated, and per task. The improvement threshold $\\varepsilon_{\\ell}$ is the 30th percentile of the values the critic predicts for task $\\ell$ [arXiv:2511.14759 §V-D]. Values are normalized per task by maximum episode length [arXiv:2511.14759 §V-C]. Specialists exist as a category and are finetuned from the pre-trained model, while the final generalist is trained from scratch [arXiv:2511.14759 §IV-C]. And the collection budgets are quoted per task: about 300 trajectories per iteration for laundry, 600 autonomous plus 360 intervention trials for box assembly [arXiv:2511.14759].\nStated, and global. The critic is a single multi-task value function over everything collected so far [arXiv:2511.14759 §V-C]. The demonstration corpus behind the pre-trained checkpoint is tens of thousands of hours across numerous tasks and many robots — a corpus, not a per-task seed set [arXiv:2511.14759 §IV-C].\nNot stated. Which slice of that corpus a specialist\u0026rsquo;s finetune actually sees. The paper gives the loop and the per-task threshold; it does not give the dataset scope, which is why the first line above is seed_demonstrations rather than something that looks like an API. This post reads it narrowly — one task\u0026rsquo;s data — because the per-task budgets and the per-task threshold both point that way, and that reading is in the gap ledger rather than presented as the recipe [arXiv:2511.14759].\nSo what makes the output a specialist is not a filtering step inside the algorithm; there is none. It is the scope of D and of the collection, decided before the loop starts. The critic can afford to be global because its job is to score states; the policy update is local because its job is to get this task working [arXiv:2511.14759 §IV-C].\nWhat finetune actually is, and why there is no importance sampling The single most load-bearing fact about that last line: it is not a reinforcement-learning update. It is the same supervised training step from stage two — FAST cross-entropy on the backbone, flow matching on the expert, coupled by the stop-gradient — run on D, with one extra string in the prompt [arXiv:2511.14759 §V-B]. The objective is\n$$\\max_{\\theta,\\,\\phi}\\ \\ \\mathbb{E}_{(o,\\,C,\\,A,\\,z)\\sim\\mathcal{D}}\\Big[\\log \\pi_{\\theta,\\phi}\\big(A \\mid o,\\ C,\\ z\\big)\\Big]$$which is the objective from the top of this part with $z$ appended to the context. Nothing else changes: no ratio, no clipping, no trust region, no KL penalty, no off-policy correction [arXiv:2511.14759 §V-B].\nWhy importance sampling is not needed. Importance weights exist to estimate an expectation under $\\pi_{\\theta}$ from samples drawn by some other policy $\\pi_{\\beta}$ — the mismatch is the problem, and $w = \\pi_{\\theta}/\\pi_{\\beta}$ is the patch. RECAP never forms that expectation. It fits a conditional density on the data distribution as it actually is, and $z$ carries the information that would otherwise have to be corrected for: every sample is on-distribution for the conditional model, because the label says which part of the distribution it came from [arXiv:2511.14759 §V-B]. A failure is not a mislabelled success needing a small weight; it is a correctly labelled sample of Advantage: negative.\nWhy importance sampling is not even available. The ratio requires $\\pi_{\\theta}(a \\mid s)$ in closed form, and a flow-matching policy does not offer one — the same fact that opened this stage [arXiv:2604.15483 §III]. So the reason they do not need the correction and the reason they could not apply it are the same architectural fact. This is also why the published comparison lands where it does: AWR, which is exactly the reweighting cousin of this idea, underperformed conditioning, and PPO struggled in the off-policy regime [arXiv:2511.14759].\nWhat you give up. Conditional density estimation cannot leave the support of its data. The policy will imitate the best behavior in D that carries a positive label, and it will not invent behavior better than that. Improvement has to come from collecting better data — which is what the deployed specialist and the human interventions are for — rather than from squeezing harder on the data you have [arXiv:2511.14759]. The one place the model is pushed past the data is at inference, where the indicator is randomly omitted during training so that guidance can be run above one [arXiv:2511.14759].\nTwo disciplines make it converge rather than drift. The threshold is a percentile of the critic\u0026rsquo;s own predictions for that task, so it tracks a moving policy instead of a fixed bar [arXiv:2511.14759]. And the policy of each iteration is finetuned from the pre-trained checkpoint, never from the previous iteration\u0026rsquo;s policy — the line in the algorithm above that looks like a typo and is not [arXiv:2511.14759]. Round five\u0026rsquo;s weights start where round one\u0026rsquo;s did; what accumulates across rounds is the dataset and its labels, not the parameters. That is the reversible arrangement: a bad round can be dropped from the mixture and retrained, where a bad fine-tune of a fine-tune cannot be undone [arXiv:2511.14759].\nThe critic is the opposite case, and the two lines in the algorithm are easy to read as inconsistent until you see they answer different questions. It is refit every round on everything collected so far, because the MDP itself moves — gripper wear and calibration drift mean a value function fitted on round one\u0026rsquo;s fleet is scoring a robot that no longer exists [arXiv:2511.14759]. Whether that refit warm-starts from the previous critic or from its own initialization is undisclosed [arXiv:2511.14759]. Budget about 300 trajectories per iteration [arXiv:2511.14759].\nWhat the loop produces, and which model you actually ship The question the algorithm above does not answer on its own: the pi that survives the last round is not the thing you deploy to customers. The loop runs per task, and what it produces each round is a specialist — one hard task, finetuned from the pre-trained checkpoint, conditioned on the advantage indicator [arXiv:2511.14759].\nThat specialist has exactly two jobs, which is why the arrow in the figure returns to step one rather than exiting [arXiv:2511.14759]:\nit goes back onto the fleet as the collector for the next round, so round $k+1$\u0026rsquo;s data is better than round $k$\u0026rsquo;s [arXiv:2511.14759]; its rollouts join the dataset permanently, carrying the advantage labels they were scored with [arXiv:2511.14759]. The shipped model is the generalist, and it is trained from scratch. Specialists are finetuned from the pre-trained model, while the final generalist is trained from scratch on the accumulated mixture rather than being finetuned from any specialist [arXiv:2511.14759]. Read the division of labour plainly: the specialists are instruments for manufacturing data better than your teleoperators can produce, and the generalist is the product [arXiv:2604.15483 §VI-A].\nThe evidence that this actually works is the part worth staring at. The generalist exceeds the RL-trained specialists on throughput for diverse laundry folding and box building — beating them on the specialists\u0026rsquo; own axis, having never run a policy gradient itself [arXiv:2604.15483 Fig. 6].\nSo the exit most teams should take. You may not have to run this loop on your generalist at all, and the frontier generation did not: one model runs RL and produces rollouts, and the next model absorbs them as ordinary conditioned training examples, which the lab describes as a kind of distillation [arXiv:2604.15483 §VI-A]. Run RL narrowly where it pays, and let M1\u0026rsquo;s flywheel carry the result into the generalist without a policy gradient ever touching it [arXiv:2511.14759].\nGate. Throughput at least 2x and failure rate at least halved against M2 [arXiv:2511.14759]. Reference platform: two 6-DoF arms, parallel-jaw grippers, 3 cameras, tasks of 5 to 15 minutes [arXiv:2511.14759].\nThis milestone is the fourth capability axis from Part 1, and the one most likely to slip. It is also the only published route to a robot that improves after you ship it [arXiv:2511.14759]. Closes the training half of brain/deployment.\nM4 — move the compute First, stop making the robot wait Before compressing anything, fix the schedule. The serving runtime runs three threads and nothing waits for anything: the high-level policy emits subtasks, the world model generates a subgoal in 1.25 seconds, and the VLA keeps executing against whatever context is latest [arXiv:2604.15483 §VII]. Subgoals refresh every 4 seconds or on a semantic-intent change [arXiv:2604.15483 §VII]. One inference runs per executed prefix, at 5 denoising steps and 38 to 127 ms [arXiv:2604.15483 App. D].\nAsynchrony is the load-bearing idea, and it is free: a 1.25 second world-model call would be fatal in a synchronous loop and is invisible in this one [arXiv:2604.15483 App. D].\nThen compress Distil and quantize to the 130 W ceiling [spec: NVIDIA Jetson AGX Thor] at no more than 1 point of success loss [computed: this plan]. In the order it pays: robotics-specific quantization first, because W4A8 holds 97.6% while cutting 4.27 GB to 1.28 GB where generic language-model quantization lands at 76.3% [arXiv:2602.20309]. Then step reduction — single-step flow takes one policy from 274 ms to 83 ms while improving success [arXiv:2604.05656]. Then compilation, worth 1.5 to 3.3x [repo: Isaac-GR00T hardware_recommendation.md]. Respect the cliff: 94.8% at 3.0 bits per weight becomes 48.0% at 2.0 [arXiv:2605.24011].\nNote what the frontier does with the part that cannot be shrunk: the world model is served on 4 H100s with 8-bit matrix multiplications and a modified attention kernel [arXiv:2604.15483 App. D]. Off-board is a legitimate answer for a component that is not in the control loop.\nGate. Per-robot compute becomes $3,499 against a datacentre GPU, at 40 to 130 W against 700 W [computed: EDGE-19 against EDGE-25]. Closes brain/deployment, and delivers the Part 1 product thesis.\nM5 — run a fleet Publish the mean time between failures and the recalibration cadence — the two numbers no lab discloses [arXiv:2104.08212]. Gate. A second embodiment and a second site with no new pretraining run [arXiv:2408.11812]. Closes body/hardware and body/runtime.\nWhat to expect from scaling Plan against diversity rather than volume, and against a three-order-of-magnitude fit rather than a seven-order one [arXiv:2405.14005]. The diversity law saturates [arXiv:2410.18647]. Assume the published numbers are optimistic until your own instrument reproduces them, because the field\u0026rsquo;s own audit says only about a fifth of its comparisons are significant [arXiv:2606.04233].\nPart 6 — Three open bets This part is speculative and marked as such. Where the fact ledger has nothing, it says so rather than reaching for a plausible sentence.\nA haptic-native model, and first-person data What exists: force channels worth 23.2% on average [arXiv:2505.22159], tactile sensing worth 5% to 35% on USB insertion [arXiv:2507.09160], replaceable skins that hold cross-instance degradation to 13% [arXiv:2409.08276], a 460k-image tactile pretraining corpus [arXiv:2410.24090], and egocentric human data with a demonstrated 44-point transfer [arXiv:2509.19626].\nWhat is missing: no frontier robot foundation model ingests force as a first-class modality, and no tactile representation transfers across sensor instances at foundation-model scale [arXiv:2410.24090]. The bet is that the first team to close the dashed line in the Part 2 diagram gets contact-rich manipulation that nobody else has.\nHardware and policy, optimised together This is the thinnest section in the post, and the reason is worth stating plainly: the fact ledger behind this work contains nothing on joint morphology-and-policy optimisation, so what follows is a question rather than a claim [arXiv:2604.15483].\nThe motivation is grounded, though. Repeatability varies about 10x across the platforms that share a training corpus [computed: 1 mm against 0.1 mm]. No lab publishes a sweep over friction, payload, or stiffness [arXiv:2604.15483]. Cross-unit manufacturing variance is named and never quantified [arXiv:2506.18123]. Every one of those is a case where the body is a free variable that nobody is optimising against the objective that matters, which is the policy\u0026rsquo;s success rate. Putting the morphology inside the optimisation loop — in simulation, jointly with the policy — is the obvious move, and the honest statement is that no published work does it at this scale.\nContinual learning, and adaptation at test time The closest published loop trains a value model on autonomous and intervention data and reports throughput at least 2x with failure rates at least halved, sustaining 13 hours of continuous service [arXiv:2511.14759]. The closest on-device adaptation fine-tunes from 50 to 100 demonstrations [arXiv:2503.20020]. Human-in-the-loop reinforcement learning reaches 100% on 12 of 12 real tasks in 1 to 2.5 hours per task from 20 to 30 seed demonstrations [arXiv:2410.21845].\nWhat nobody publishes: adaptation to mechanical wear, and mean time between failures [arXiv:2104.08212]. This closes the loop back to Part 1\u0026rsquo;s fourth axis. A model that cannot notice its own gripper degrading is not evolving from experience — it is a snapshot with good marketing.\nThe gap ledger The things nobody outside a lab knows, collected in one place. Each is a contribution available to anyone in a position to measure it.\ngap status total pretraining dataset size, per-source mixture weights, optimizer and total compute undisclosed [arXiv:2604.15483] fleet size behind any frontier model undisclosed [arXiv:2604.15483] cost per demonstration, at every lab undisclosed [arXiv:2506.18123] mean time between failures, across all fleet-scale papers undisclosed [arXiv:2104.08212] cross-unit manufacturing variance named, never quantified [arXiv:2506.18123] photon-to-torque latency, any platform undisclosed [repo: openpi websocket_policy_server.py] generalization sweeps over friction, payload and stiffness published by no lab [arXiv:2604.15483] the weight between the cross-entropy and flow-matching losses undisclosed [arXiv:2604.15483 §III] which slice of the corpus an RL specialist\u0026rsquo;s finetune sees undisclosed [arXiv:2511.14759] onboard compute vendor, watts and TOPS, for commercial humanoids undisclosed [blog: Figure Helix] Where this leaves us The generality is real, and Part 0\u0026rsquo;s first number is the reason to take the field seriously. The fragility is mechanical, and Part 0\u0026rsquo;s second number is the reason not to believe a demo. Between them sits a machine that worked once, for language, on three pillars — and robotics has one of them plus a constraint that language never faced.\nThat is not a reason to stop. It is a reason to build in a particular order: the instrument, then the data engine, then the policy, then the loop that improves it, then the compression that puts it on the machine. Four of the five blockers are measurement, mechanism, and data-engine problems, which means most of the work is not modelling work — and that most of the people who can close it do not currently think of themselves as machine-learning researchers.\nBibliography [arXiv:2001.08361] Scaling Laws for Neural Language Models [arXiv:2108.07258] On the Opportunities and Risks of Foundation Models [arXiv:2203.15556] Training Compute-Optimal Large Language Models [arXiv:2204.02311] PaLM: Scaling Language Modeling with Pathways [arXiv:2406.17557] The FineWeb Datasets [arXiv:2604.15483] π0.7: a generalist robot policy [arXiv:2504.16054] π0.5: open-world generalization [arXiv:2410.24164] π0: a vision-language-action flow model [arXiv:2511.14759] RECAP: self-improvement from autonomous experience [arXiv:2506.07339] Real-Time Chunking [arXiv:2310.08864] Open X-Embodiment [arXiv:2307.15818] RT-2: vision-language-action models [arXiv:2212.06817] RT-1: robotics transformer [arXiv:2510.03342] Gemini Robotics 1.5 [arXiv:2503.20020] Gemini Robotics On-Device [arXiv:2503.14734] GR00T N1 [arXiv:2406.09246] OpenVLA [arXiv:2502.19645] OpenVLA-OFT [arXiv:2501.09747] FAST action tokenization [arXiv:2304.13705] ALOHA and ACT [arXiv:2401.02117] Mobile ALOHA [arXiv:2309.13037] GELLO [arXiv:2503.03081] AirExo-2 [arXiv:2402.10329] Universal Manipulation Interface [arXiv:2403.12945] DROID [arXiv:2503.06669] AgiBot World [arXiv:2410.18647] Data scaling laws in imitation learning [arXiv:2405.14005] Scaling laws in robot learning, a meta-analysis [arXiv:2408.14037] Re-Mix: data mixture optimization [arXiv:2506.19121] CUPID: data curation [arXiv:2405.12213] Octo [arXiv:2409.20537] Heterogeneous Pre-trained Transformers [arXiv:2408.11812] CrossFormer [arXiv:2410.24221] EgoMimic [arXiv:2509.19626] EgoBridge [arXiv:2110.07058] Ego4D [arXiv:2406.02523] RoboCasa [arXiv:2310.17596] MimicGen [arXiv:2505.12705] DreamGen [arXiv:2606.04233] Statistical rigour in robot learning benchmarks [arXiv:2510.13626] Perturbation sensitivity of VLA policies [arXiv:2510.03827] LIBERO-PRO [arXiv:2510.17950] RoboChallenge [arXiv:2405.05941] SimplerEnv [arXiv:2506.18123] RoboArena [arXiv:2603.13616] N-SCORE [arXiv:2607.01060] RoboWorld evaluation [arXiv:2602.20309] QuantVLA [arXiv:2605.24011] ActQuant [arXiv:2604.05656] SnapFlow [arXiv:2602.18397] VLA inference on edge accelerators [arXiv:2409.03403] RoVi-Aug [arXiv:2505.22159] ForceVLA [arXiv:2507.09160] Tactile-VLA [arXiv:2409.08276] AnySkin [arXiv:2410.24090] Sparsh [arXiv:2410.21845] HIL-SERL [arXiv:2410.11157] RPCBF safety filters [arXiv:2104.08212] Deep RL at Scale [arXiv:2604.18000] Shortcut learning in VLA models [arXiv:2602.17659] When vision overrides language ","permalink":"https://mzf666.github.io/vla/en/posts/how-to-build-a-robot-foundation-model/","summary":"A first-principles chain: what a robot foundation model is, what it is made of, why the machine that produced large language models does not transfer, what that failure costs in each subsystem, and the order in which to build it anyway.","title":"How to Build a Robot Foundation Model — Routes and Roadblocks"},{"content":" This is the condensed edition of the long-form post — one screen per slide, with a line or two of narration under each. Open the full-screen deck → (arrow keys, or scroll).\nThe chain runs in one direction and every step depends on the one before it. The conclusion is stated on the first screen so the rest can be argued with [arXiv:2604.15483].\nA bimanual UR5e that had never seen a laundry demonstration folded a shirt at 85.6% progress and 80% success; ten expert teleoperators with roughly 375 hours each managed 90.9% and 80.6% on their first attempt [arXiv:2604.15483 §IX]. The same class of policy collapses from 100% to 0% when a camera moves 10 cm and 20 degrees [arXiv:2409.03403].\nPart 1 — What it is Task generalization is demonstrated across 14 scenarios with 3 to 6 open-ended instructions in unseen rooms [arXiv:2604.15483 §IX-B]. Embodiment generalization is demonstrated with a ceiling, across 22 embodiments and 60 datasets [arXiv:2310.08864]. Continual evolution has exactly one published loop [arXiv:2511.14759], and nothing at all on mechanical wear [arXiv:2104.08212].\nStrip web pretraining from the same architecture and emergent skills go to 0% and generalization to 1%; restore it and they are 48.7% and 47% [arXiv:2310.08864 Table II]. The semantics are inherited, which is the only reason any of this starts from something the internet already paid for [arXiv:2307.15818].\nAn H100 draws 700 W [spec: NVIDIA H100 SXM]; an edge module draws 40 to 130 W [spec: NVIDIA Jetson AGX Thor]; a humanoid\u0026rsquo;s whole system averages about 210 W [spec: 1X NEO]. The module is 19% to 62% of the entire power budget [computed: 40 to 130 W against 210 W], and today\u0026rsquo;s models do not fit — 35.9 Hz on an H100 against 10.7 Hz on Thor [repo: Isaac-GR00T hardware_recommendation.md].\nPart 2 — What it is made of Up to 4 images at 448×448, 6 frames of history, a joint configuration and one sentence go in; 50 future joint targets come out, of which 15 or 25 execute before the model is called again [arXiv:2604.15483 §IV]. That is the brain\u0026rsquo;s boundary, not the machine\u0026rsquo;s — the model is a setpoint generator that feeds the servo loop rather than replacing it [arXiv:2604.15483 §VII].\nThe brain writes setpoints at 50 Hz [arXiv:2604.15483 §VII] against an arm that accepts commands at 1 kHz [spec: Franka Research 3]. Repeatability spans 0.1 mm to 1 mm across platforms sharing one corpus [computed: 1 mm against 0.1 mm]. Force and touch are measured by the body and consumed by no frontier model [arXiv:2505.22159].\nNobody runs a 5B transformer inside a servo loop, so every serious system splits into halves at different rates [arXiv:2503.14734]. Where labs disagree is the width of the seam — a latent vector [blog: Figure Helix], a feature sequence [arXiv:2503.14734], or a human-readable subtask string that costs bandwidth and buys debuggability [arXiv:2604.15483 §VI].\nPart 3 — The machine that worked Loss falls as a power law with exponents 0.095, 0.076 and 0.050, fitted across 7 orders of magnitude [arXiv:2001.08361]. That predictability is a budget instrument: 70B on 1.4T beats 280B on 300B at equal compute, at about 20 tokens per parameter [arXiv:2203.15556].\nOne open pipeline distils 96 Common Crawl snapshots into 15T tokens and 44 TB for 1,536 GPU-hours [arXiv:2406.17557], roughly $10k of compute [computed: 1,536 GPU-hours at commodity rates]. The two largest egocentric video corpora total 4,956 hours [computed: 3,670 h plus 1,286 h], and robot data is manufactured — about 10,000 hours of teleoperation is roughly $500k of labour [computed: 10,000 h at $50 per hour].\nRidge points, computed from datasheets: about 1,181 FLOP/byte for an H100, about 3,791 for a Thor, about 1,343 for an Orin [computed: 3,958 TFLOPS over 3.35 TB/s]. Thor\u0026rsquo;s sits about 3.2x further right [computed: 3,791 over 1,181], and the measured consequence is an action expert costing 26.20 ms there against 7.25 ms on a consumer GPU [arXiv:2602.18397].\nSame model, same runtime, three accelerators: throughput tracks the GB/s line rather than the TOPS line [repo: Isaac-GR00T hardware_recommendation.md].\nRobotics satisfies the compute pillar. It does not satisfy the data pillar or the infrastructure pillar in the forms that made them powerful, and it carries a fourth constraint text never faced [arXiv:2604.15483 §VII].\nAction spaces are padded to 18 dimensions and rates span 3 Hz to 50 Hz [arXiv:2410.24164]. Diffusion at 50 steps buys 10.1 Hz at 95.4% where continuous regression buys 109.7 Hz at 95.3% [arXiv:2502.19645]. Real-time chunking survives 100–200 ms and fails past 300 ms [arXiv:2506.07339]. One of the five is a modelling problem.\nPart 4 — What it costs Separating a 50% policy from a 60% policy needs 387 trials [computed: two-proportion test, alpha 0.05, 80% power]; the field runs 10 to 60 [arXiv:2506.18123]. Only 19.8% of LIBERO claims are statistically significant [arXiv:2606.04233], and models scoring 98% and 92% collapse to 0.0% under position perturbation [arXiv:2510.03827].\nThe one replicated law is over diversity: 32 environment-object pairs at 50 demonstrations each reached 85% to 92.5% in unseen environments, collected by 4 people in an afternoon [arXiv:2410.18647].\nA handheld station costs $371 and produces 111 demonstrations an hour against teleoperation\u0026rsquo;s 35 [arXiv:2402.10329]. An hour of human video is worth about 1,400 demonstrations where an hour of robot time is worth 135 [arXiv:2410.24221].\nHolding the backbone fixed, diffusion\u0026rsquo;s extra compute buys about a tenth of a point for roughly 11x the throughput [computed: 109.7 Hz against 10.1 Hz].\nOne method holds 94.8% at 3.0 bits per weight and falls to 48.0% at 2.0 [arXiv:2605.24011]. Robotics-specific W4A8 reaches 97.6% while cutting 4.27 GB to 1.28 GB, where generic quantization lands at 76.3% [arXiv:2602.20309].\nThe model\u0026rsquo;s own time is the measured third — 14 ms of encoders, 32 ms of prefix, 27 ms of flow steps [arXiv:2410.24164]. Everything before and after it is unmeasured by the entire field [repo: openpi websocket_policy_server.py].\nPart 5 — The build order Everything before this was diagnosis. Each earlier hook is picked up by exactly one milestone, and each milestone carries a gate expressed as a number so it can fail [arXiv:2506.18123].\nAll three language-model stages transfer in form. The third one changes shape, because there is no cheap verifier and the reward has to come from the robot\u0026rsquo;s own experience [arXiv:2511.14759].\nRecord every episode as a conditioned example: speed binned at 500 steps, quality as a human 1 to 5, mistake as a per-segment boolean, control mode [arXiv:2604.15483 §V-C]. At runtime those become knobs — prompt quality 5 and mistake false and the policy imitates the good half of your data [arXiv:2604.15483 §VII].\nEight sources feed one annotated mixture, and the deployed policy\u0026rsquo;s rollouts become tomorrow\u0026rsquo;s training data [arXiv:2604.15483 §VI-A]. Failures and mistake-bearing successes are kept on purpose; autonomous data from generalization evaluations is excluded, or the flywheel trains on the test set [arXiv:2604.15483 §VI-A].\nAbout 5B on the control path: a 400M vision encoder and a 4B backbone carrying inherited semantics, an 860M flow expert carrying motor skill, and a 14B world model running beside the loop rather than inside it [arXiv:2604.15483 §IV].\nThe part to copy is the firewall: gradients from the action expert do not flow back into the backbone [arXiv:2604.15483 §III]. FAST tokens exist only at training time and never attend to the flow actions [arXiv:2604.15483 App. B].\nOne step, two losses, two parameter groups. Cross-entropy on FAST tokens trains the backbone; flow matching trains the expert against the velocity that carries noise to actions [arXiv:2410.24164]. stop_gradient is the only coupling, and the relative weight of the two losses is the one number nobody has published [arXiv:2604.15483 §III].\nHistory dropped at p = 0.3, metadata at 15% and 5%, subgoals in 25% of the batch [arXiv:2604.15483 §V-E]. These are not regularization — they are what stops the policy binding to a fixed camera rig [arXiv:2409.03403].\nBefore compressing anything, fix the schedule: three threads and nothing waits, so a 1.25 s world-model call is invisible instead of fatal [arXiv:2604.15483 §VII].\nThere is no cheap verifier and no tractable likelihood, so PPO and AWR both lose to conditioning [arXiv:2511.14759]. Fit a critic, binarize its one-step difference against the 30th percentile for that task, write the result into the prompt as text, and train supervised on everything — failures included [arXiv:2511.14759].\nA 670M distributional value model over 201 bins, a sparse reward, and a binarized advantage indicator inserted as text after the language input [arXiv:2511.14759]. Each iteration\u0026rsquo;s policy finetunes from the pre-trained checkpoint, never from the previous iteration\u0026rsquo;s policy, or it drifts; the critic is the opposite and is refit every round on everything collected so far [arXiv:2511.14759].\nM4 is where the product thesis is paid for: per-robot compute drops to $3,499 at 40 to 130 W, against a datacentre GPU at 700 W [computed: EDGE-19 against EDGE-25].\nPart 6 — Three open bets Speculative, and marked as such. Force is worth 23.2% on average and no frontier model ingests it [arXiv:2505.22159]. Hardware-and-policy co-design has no published work at this scale, so it is posed as a question [arXiv:2604.15483]. The only self-improvement loop reports 2x throughput and halved failures [arXiv:2511.14759].\nFour of the five blockers are measurement, mechanism, and data-engine problems — which means most of the people who can close them do not currently think of themselves as machine-learning researchers [arXiv:2506.18123].\nThe long-form edition carries the full argument, the gap ledger, and the bibliography [arXiv:2604.15483].\n","permalink":"https://mzf666.github.io/vla/en/posts/robot-foundation-model-the-talk/","summary":"The deck edition: the same argument as one slide per screen, readable in fifteen minutes instead of fifty.","title":"How to Build a Robot Foundation Model — Routes and Roadblocks (TLDR)"},{"content":"The claim this post argues Generality in physical AI comes from data diversity — but diverse data is mixed-quality, and naive training averages it into mediocrity. The breakthrough is to condition on how each episode was produced, not only what was done. Rich context turns data quantity from a liability into a monotone benefit, and turns the trained model into something you can steer.\nThe evidence is a scaling result that runs opposite to intuition: with episode metadata, performance keeps improving as the dataset grows even though average data quality falls; without metadata, performance degrades as lower-quality data is added [π0.7 §IX-E, Fig. 18]. The corollary is the hook — π0.7 inherits reinforcement-learning-level capability by distilling π*0.6\u0026rsquo;s RL rollouts through metadata conditioning, without itself running RL [π0.7 §VI-A].\nHow to read this This post assumes you know LLM pretraining and post-training, diffusion and flow matching, and transformers at scale. It assumes you know nothing about robotics. Everything robotics-specific is taught inline, and the prerequisites are quarantined into Primer boxes you can skip if you already have them. The organizing source throughout is the π0.7 paper itself [π0.7 §I].\nThree conventions run throughout, and they are load-bearing rather than decorative [π0.7 §V-E]:\nEvery part opens with an I/O contract. Input → output, with concrete shapes and units wherever the paper discloses them. If you can state what a module consumes and produces, you can reason about it without reading its internals — and you can substitute it [π0.7 §IV].\nEvery claim is typed. Disclosed means a paper states it. Inferred means I reasoned it from disclosed facts and am telling you so. Undisclosed means nobody outside Physical Intelligence knows, and it goes into the gap ledger in Part 10 rather than getting papered over [π0.7 §VI-A].\nEvery number carries its reference. Values in this post are checked mechanically against a fact ledger; a build script fails if any number in the prose is absent from that ledger or any paragraph lacks a citation. That is the only way I know to keep a document this long honest [π0.7 §V-E].\nOne scoping note. \u0026ldquo;Reproduce π0.7\u0026rdquo; here means understand the whole scaling path, the design philosophy, and every disclosed detail of data collection, organization, training and serving — well enough to rebuild the system as an engineering program. It does not mean porting code. π0.7 has no public weights and no public code [π0.7 ref. 42].\nPart 0 — The goal, and why it is hard I/O contract of this part: in — your LLM intuitions; out — a precise definition of \u0026ldquo;generalist physical AI\u0026rdquo; and a working understanding of why the recipe you already know does not transfer.\nWhat \u0026ldquo;generalist\u0026rdquo; actually means here \u0026ldquo;Generalist\u0026rdquo; is the most abused word in this field, so start with the operational definition: a generalist is a model that scores well on the four axes π0.7 actually measures [π0.7 §I].\nOut-of-the-box dexterity means performing hard multi-stage manipulation with no task-specific post-training — the same checkpoint, no fine-tuning per task [π0.7 §I]. Instruction generalization means following open-ended language in environments never seen during training; the protocol is 14 scenarios with sequences of 3–6 open-ended instructions each, run in 4 unseen kitchens and 2 unseen bedrooms [π0.7 §IX-B, Fig. 9]. Cross-embodiment transfer means running a skill on a robot body that never demonstrated that skill — folding shirts on a bimanual UR5e when essentially all folding data was collected on a different, lighter robot [π0.7 §IX-C]. Compositional generalization means recombining known skills into task-and-scene pairings that were never trained together, where performance sits at 60–80% against above 90% in-distribution [π0.7 §X].\nNotice what all four have in common: each is measured by running a physical robot and watching what happens. There is no offline benchmark that substitutes for a trial [π0.7 §IX-E].\nWhy the LLM recipe does not transfer You already know how to build a generalist in text: scrape an enormous corpus, pretrain on next-token prediction, post-train for instruction following, evaluate offline, iterate quickly. Almost every step of that loop breaks when the output is a joint command produced by a physical machine [π0.7 §I].\nThe asymmetries in that figure are my characterization of the gap the paper motivates rather than quoted claims, so treat them as inferred — with one exception. The scale figure is disclosed: the π-series pre-training corpus is described as tens of thousands of hours of demonstrations across numerous tasks and a variety of robots [π*0.6 §IV-C]. That is an enormous robotics dataset and a rounding error against a text corpus.\nThe consequences compound. Because action data must be physically produced, data is the binding constraint, not compute. Because every robot body differs in reach, mass, inertia and gripper geometry, the corpus is not i.i.d. in the way a text corpus is — the UR5e is described as significantly longer, different in morphology, much heavier and higher in joint inertia, requiring a different manipulation strategy and different table positioning [π0.7 §VIII, App. F]. And because evaluation requires physical trials, the iteration loop that makes LLM research fast is simply not available; the paper is explicit that at this data scale it is practically difficult to definitively determine which tasks are truly \u0026ldquo;seen\u0026rdquo; or \u0026ldquo;unseen\u0026rdquo; [π0.7 §IX-E].\nPrimer — teleoperation. A human operator drives the robot through a task while the system records synchronized camera frames and joint positions at the control rate. That recording is one \u0026ldquo;demonstration\u0026rdquo; or \u0026ldquo;episode\u0026rdquo;. This is how nearly all high-quality robot action data comes into existence, and it is why robot data has a wage attached to it. The π0.7 human study gives a sense of the skill involved: the 10 operators recruited were in the top 2 percentile of experience, averaging around 375 hours of teleoperation across all robot platforms [π0.7 App. F, Fig. 21].\nWhat data cannot be collected, and what simulators cannot build The obvious response to expensive data is simulation. It works far less well than outsiders expect, and understanding why explains most of π0.7\u0026rsquo;s design — which is built entirely around real-robot, human-video and web data, with no simulation component [π0.7 §VI-A].\nThe tasks that matter in homes are contact-rich and involve deformable objects: folding laundry, making a sandwich, peeling vegetables, scooping beans, wiping a spill. π0.7\u0026rsquo;s own task list is dominated by exactly these — folding a buttondown shirt from a laundry basket, turning a t-shirt inside out, spreading peanut butter and cutting a sandwich diagonally, slicing a zucchini with a knife on a lanyard [π0.7 App. G]. Cloth, food and liquid have no tractable rigid-body state; a simulator that gets shirt dynamics right at the fidelity needed for grasp selection is a research project in itself, not a tool you reach for.\nThe second barrier is the long tail of appliances and environments. π0.7\u0026rsquo;s tasks include opening an air fryer, toasting a bagel in a toaster oven, operating an espresso machine, and driving through a self-closing closet door [π0.7 App. G]. Each of those is a specific physical object with specific latching behavior. Building faithful models of them is not a scaling problem; it is an unbounded modelling problem, and every home has different ones.\nSo the field is pushed toward real data. That is a inferred reading of why π0.7 invests where it does, but it is strongly implied by where the paper actually spends its effort: on a fleet, on teleoperators, on annotation, and on a data engine [π0.7 §VI-A, §VIII].\nThe central dilemma Now the trap that motivates the entire paper [π0.7 §I].\nGenerality demands diversity. To be a generalist you need many tasks, many environments, many robot bodies, many ways of doing the same thing. But diverse data collected at scale is mixed-quality data. Some episodes are fast, some are slow. Some are clean, some contain a fumble and a recovery. Some come from expert operators, some from a policy evaluating itself and failing [π0.7 §VI-A].\nThe conventional response is to filter: keep the good episodes, discard the rest. This is what a careful practitioner does, and it is what π0.7 argues against. Filtering throws away the majority of the data you paid for, and the discarded episodes contain exactly the information a robot needs about how things go wrong and how to recover. But keeping them naively is worse — train on the union and the model learns the average behavior, which is mediocre, hesitant and error-prone [π0.7 §I, §IX-E].\nπ0.7\u0026rsquo;s answer is the third option: keep everything, and tell the model what it is looking at. Label each episode with how fast it was, how good it was, whether it contained a mistake. Then at test time, ask for the behavior you want. A slow, error-containing episode is no longer a contaminant — it is a labelled example of what \u0026ldquo;slow\u0026rdquo; and \u0026ldquo;mistaken\u0026rdquo; look like, and its presence makes the conditioning sharper [π0.7 §V-C, §VII].\nIf that sounds familiar, it should. It is return-conditioned or advantage-conditioned behavior cloning, generalized from a scalar to a multimodal context [π*0.6 §V-B]. Part 3 develops that connection properly.\nPart 1 — Primer: what a VLA is I/O contract of this part: in — your diffusion and transformer knowledge; out — the vision-language-action model as a black box with a declared contract, plus the four robotics concepts you need for the rest of the post.\nThe contract A vision-language-action model is a policy. It consumes an observation and a context, and produces a chunk of future actions [π0.7 §III].\nFormally, the observation is $o_t = [I_t^1, \\ldots, I_t^n, q_t]$ — $n$ camera images plus the robot\u0026rsquo;s joint configuration — and the model produces an action chunk $a_{t:t+H}$ covering $H$ future timesteps, of which a shorter horizon $\\tilde{H} \u003c H$ is actually executed before the model re-observes and re-plans [π0.7 §III]. The training objective is\n$$\\max_{\\theta}\\; \\mathbb{E}_{\\mathcal{D}}\\left[\\, \\log \\pi_{\\theta}\\!\\left( a_{t:t+H} \\mid o_{t-T:t},\\, C_t \\right) \\right]$$where $C_t$ is the context and $o_{t-T:t}$ is a window of past observations [π0.7 §III, Eq. 1]. One honest caveat for readers who will reach for the likelihood: the flow-matching action expert optimizes an approximate lower bound on this quantity rather than the likelihood in closed form [π0.7 §III].\nFor π0.7 the contract instantiates concretely. Inputs are up to 4 cameras — front, two wrist, and an optional rear view — with up to 6 history frames each, sampled at a stride of 1 second, all resized to 448×448, plus up to 3 subgoal images with the rear view omitted [π0.7 §VI-B]. The output is exactly 50 action tokens, one 50-step chunk [π0.7 §VI-B]. At runtime only 15 or 25 of those 50 steps are executed before re-planning [π0.7 §VII].\nThe most instructive thing about this contract is what is absent from the input. There is no action history, no reward signal, no map, no explicit state estimate beyond proprioception [π0.7 §III]. The policy is close to Markovian in its own outputs: the loop closes through the physical world, not through the context. That single design choice is why compounding error behaves so differently for VLAs than for autoregressive world models, a contrast Part 11 returns to.\nPrimer — control frequency. A robot\u0026rsquo;s low-level controller expects a new target at a fixed rate. π0.7\u0026rsquo;s platforms run at 50 Hz, except the UR5e which runs at 20 Hz [π0.7 §VIII]. At 50 Hz a 50-step chunk is one second of motion. This is why latency is a first-class constraint rather than a nice-to-have: if inference takes longer than the execution window, the robot stalls mid-motion.\nPrimer — action chunking. Rather than predicting one action per forward pass, the model predicts $H$ steps jointly and the robot executes the first $\\tilde{H}$ of them. Predicting jointly avoids the incoherent, jittery motion you get from independently sampled single steps, and it amortizes inference cost across many control cycles. $H$ is a modelling choice; $\\tilde{H}$ is a control choice, and they trade smoothness against reactivity [π0.7 §VII].\nPrimer — embodiment. An \u0026ldquo;embodiment\u0026rdquo; is a specific robot body: its kinematics, its degrees of freedom, its gripper, its camera placement. π0.7 spans bimanual mobile manipulators with two 6-DoF arms plus a lift and a holonomic base, a static bimanual platform, a bimanual UR5e with Robotiq grippers, and single-arm systems [π0.7 §VIII, Fig. 4]. The action space differs across them, which is what makes cross-embodiment transfer a real problem rather than a relabelling exercise.\nHow actions get represented There are two dominant ways to turn continuous robot actions into something a transformer can produce, and π0.7 uses both simultaneously — for different parameters, under different objectives [π0.7 §III].\nFlow matching treats the action chunk as a continuous vector and learns a velocity field that transports noise to data, exactly as you know it from generative modelling [π0.7 ref. 102]. Inference integrates that field for a small number of steps; π0.7 uses 5 [π0.7 §VII]. This representation is precise, which is what actuation demands.\nDiscrete tokenization instead compresses the action chunk with a DCT-based scheme and emits it as tokens from the language model\u0026rsquo;s vocabulary, trained with ordinary next-token cross-entropy [π0-FAST]. This representation is lossy but it lets the action objective speak the backbone\u0026rsquo;s native language.\nThe reason to want both at once is the problem knowledge insulation solves [π0.7 §III].\nKnowledge insulation: the gradient firewall Here is the failure mode, and it is the one knowledge insulation was introduced to fix [KI, π0.7 §III]. You take a pretrained vision-language model that knows a great deal about the world, attach a continuous action head, and train it on robot data. The high-variance regression gradients from that head flow back into the backbone and degrade the very web knowledge you attached it for. You end up with a model that can move but has forgotten what a colander is.\nKnowledge insulation fixes this with two controls that practitioners routinely conflate [π0.7 §III].\nThe first control is the objective: the VLM backbone is supervised with discrete FAST-token cross-entropy — a stable, well-conditioned classification loss of exactly the kind it was pretrained on — while the continuous action expert is trained with flow matching [π0.7 §III].\nThe second control is the gradient: the action expert attends to all backbone activations, but its gradients do not flow into the backbone [π0.7 §III]. The expert can read everything and change nothing. π0.6 used exactly this recipe, training end-to-end on continuous actions and FAST-discretized tokens while a stop-gradient prevented the flow-matching expert from impacting the rest of the model [π*0.6 §V-A].\nThere is a third control, and it is the one most write-ups miss: the attention mask [π0.7 App. B, Fig. 19].\nThe FAST tokens are available only at training time, and the FAST tokens and the flow actions do not attend to each other [π0.7 App. B, Fig. 19]. This is not an optimization detail; it is a correctness requirement. The FAST tokens are the quantized ground-truth answer. An action expert able to attend to them would learn to decode the answer from its context rather than from the observation, and would collapse at deployment where those tokens do not exist.\nSo: the mask governs what each branch may see; the stop-gradient governs what each objective may modify. Both are set deliberately, and knowledge insulation works because of the pair, not either alone [π0.7 §III].\nOne consequence worth internalizing, because it recurs in Part 8: because the action tokens attend to the prefix but the prefix does not attend to them, the prefix can be encoded once and cached across all 5 denoising steps [π0.7 §VII]. One masking decision buys both correctness and inference speed.\nWhy latency is a first-class problem In an LLM, latency is a product concern. In a VLA it is a correctness concern, because the world does not pause while you think — which is why π0.7 trains against a simulated latency budget rather than assuming inference is free [π0.7 §VI-B].\nπ0.7 handles this by training the policy to expect delay rather than by trying to eliminate it. Training uses real-time chunking with simulated delays of 0–12 timesteps, which corresponds to a maximum inference latency of 240 ms on a 50 Hz robot [π0.7 §VI-B]. The policy is thus trained to absorb exactly the latency it will encounter. In deployment the measured numbers sit comfortably inside that budget: the minimal variant runs at 38 ms with 3 camera inputs and 5 denoising steps, rising to 127 ms in the worst case with the MEM vision encoder enabled and subgoal images in context, all on a single NVIDIA H100 [π0.7 App. D].\nKeep that ratio in mind. A 240 ms budget against a 38 ms floor is the headroom that buys everything else in the system — history frames, subgoal images, guidance — and the worst case of 127 ms shows how much of that headroom those features actually consume [π0.7 §VI-B, App. D].\nPart 1 hyperparameter table Parameter Value Source Cameras up to 4 (front, two wrist, optional rear) [π0.7 §VI-B] History frames per camera up to 6, stride 1 second [π0.7 §VI-B] Subgoal images up to 3 (rear omitted) [π0.7 §VI-B] Image resolution 448×448 [π0.7 §VI-B] Action chunk length H 50 steps [π0.7 §VI-B] Executed horizon $\\tilde{H}$ 15 or 25 [π0.7 §VII] Denoising steps 5 [π0.7 §VII] Control rate 50 Hz; UR5e 20 Hz [π0.7 §VIII] RTC simulated delay 0–12 timesteps = 240 ms at 50 Hz [π0.7 §VI-B] Inference latency 38 ms minimal, 127 ms worst case, single H100 [π0.7 App. D] Part 2 — The scaling path: π0 to π0.7 I/O contract of this part: in — the VLA contract from Part 1; out — an understanding of which capability each generation added, what it cost, and why π0.7 needed all of them at once.\nπ0.7 is not a single idea. It is the ninth entry in a line where each generation adds a capability without discarding the previous one, and π0.7\u0026rsquo;s own contribution is the conditioning scheme that lets one model hold them all simultaneously [π0.7 §I, §VI-A].\nThe generations π0 established the template: a flow-matching action expert attached to a pretrained vision-language model, giving continuous dexterous control from web-scale visual knowledge [π0.7 ref. 10]. This is the architecture everything since has refined rather than replaced.\nπ0-FAST introduced DCT-based action tokenization, which made autoregressive training on high-frequency action data practical [π0.7 ref. 104]. Its lasting contribution to π0.7 is not the inference path but the training path: FAST tokens are what the backbone is supervised with under knowledge insulation [π0.7 §III].\nHi Robot added hierarchy — a high-level policy emitting semantic subtasks and a low-level policy executing them — which is what makes open-ended, long-horizon instruction following tractable [π0.7 bibliography]. π0.7 keeps this structure: the high-level policy uses the same architecture as the main model and maps observations, the task specification and past subtasks to the next subtask instruction [π0.7 §V-A, Fig. 2].\nπ0.5 added co-training on heterogeneous data plus explicit semantic subtask prediction, which is what produced open-world generalization to genuinely unseen homes [π0.7 ref. 14].\nKnowledge insulation contributed the stop-gradient firewall analyzed in Part 1 — the mechanism that lets you adapt a VLM to actions without destroying what it knows [π0.7 ref. 103].\nReal-time chunking solved the seam between consecutive action chunks, letting the robot execute continuously instead of pausing while the next chunk is computed [π0.7 ref. 107]. π0.7 uses the training-time variant, which unlike test-time RTC incurs no additional inference-time overhead [π0.7 ref. 108, App. D].\nMEM contributed memory, and π0.7 inherits it almost wholesale — worth its own subsection below [π0.7 ref. 37].\nπ*0.6 contributed reinforcement learning from real deployment experience, and is the most consequential ancestor for the thesis of this post — also its own subsection [π0.7 ref. 50].\nWhat π*0.6 actually did Reduce π*0.6 to \u0026ldquo;they used RL\u0026rdquo; and you miss the mechanism π0.7 depends on [π*0.6 §IV].\nThe method is RECAP — RL with Experience and Corrections via Advantage-conditioned Policies [π*0.6 abstract, §IV]. It has three subroutines: collect data by running the policy and labelling episode outcomes, optionally with human corrective interventions; train a value function on everything collected so far; and train the policy conditioned on an optimality indicator derived from that value function [π*0.6 §IV-C, Alg. 1].\nThe value function is a multi-task distributional critic with the same architecture as the policy but a smaller 670M VLM backbone, also initialized from Gemma 3 [π*0.6 §V-C, Fig. 3]. It discretizes returns into $B = 201$ bins and trains by cross-entropy over Monte Carlo returns [π*0.6 §IV-A]. The reward is deliberately trivial so it applies to any task: 0 at the terminal step on success, a large negative constant on failure, and −1 otherwise, with values normalized per task by maximum episode length into the range $(-1, 0)$ [π*0.6 §V-C, Eq. 5].\nThen the move that matters. Rather than a policy-gradient update — awkward for flow-matching policies, which do not offer a tractable log-likelihood — RECAP conditions the policy on a binarized advantage indicator inserted as plain text: \u0026ldquo;Advantage: positive\u0026rdquo; or \u0026ldquo;Advantage: negative\u0026rdquo; [π*0.6 §V-B]. The indicator sits after the language input but before the actions, so only the action log-likelihoods are affected [π*0.6 §V-B]. The threshold $\\varepsilon_{\\ell}$ is set at the 30% percentile of values the critic predicts for that task, and human corrections are forced to positive on the assumption that expert interventions are good actions [π*0.6 §V-D, §IV-B]. Because the indicator is randomly omitted during training, classifier-free guidance is available at inference [π*0.6 §V-B].\nTwo engineering details generalize beyond this paper. Each iteration finetunes from the pre-trained checkpoint rather than from the previous iteration, which avoids drift across rounds [π*0.6 §V-D]. And specialists are finetuned from the pre-trained model while the final generalist is trained from scratch [π*0.6 §IV-C]. The payoff: RECAP more than doubles task throughput and roughly halves the failure rate on the hardest tasks, enough to make espresso drinks for 13 hours straight and fold novel laundry in a new home for over 2 hours without interruption [π*0.6 abstract, §I].\nLook at what advantage conditioning is: conditioning a behavior-cloning policy on a text label describing how good the demonstrated behavior was. π0.7 takes that exact idea and generalizes the label from one binary scalar to a multimodal bundle [π0.7 §V-C].\nπ*0.6 also incidentally documents its predecessor. π0.6 has no paper — it is a model card [π0.7 ref. 42] — but π*0.6 discloses that it derives from π0.5, uses Gemma 3 4B as its base VLM, increases the action expert to 860M parameters, augments the pre-training set with more robot platforms, and outputs joint angles and gripper commands at 50 Hz [π*0.6 §V-A]. That is most of what an outside team needs about π0.6.\nWhat MEM contributed MEM is Multi-Scale Embodied Memory, and it exists because conditioning a policy on a long raw observation history is computationally hopeless [MEM §I]. Naively passing frames into the backbone drives inference time past the real-time barrier — measured at 300 ms for the π0.6 VLA with 4 camera streams on one H100 [MEM Fig. 3].\nMEM splits memory by timescale [MEM abstract, §III-A]. Long-horizon memory is a language summary that the high-level policy maintains, predicting the updated summary from its own previous summary plus the current observation [MEM §III-B]. Training data for those updates comes from an off-the-shelf LLM that summarizes past subtasks and their success indicators, instructed to remove or compress whenever appropriate [MEM §III-B]. Compression is the point: naively concatenating all past subtask instructions works significantly worse, because at inference a policy repeatedly retrying a failed subtask produces sequences that never occurred in near-optimal training demonstrations [MEM §IV-A, Fig. 6].\nShort-horizon memory is a video encoder built by modifying a ViT\u0026rsquo;s attention pattern: bidirectional spatial attention within each observation, plus causal temporal attention across observations at every 4th layer [MEM §III-C, Fig. 4]. That factorization drops the cost from $O(n^2 K^2)$ to $O(Kn^2 + nK^2)$ [MEM §III-C]. Two properties make it free to adopt: only the current timestep\u0026rsquo;s representation is passed onward, so the encoder emits the same token count as a memoryless single-timestep VLA [MEM §III-C]; and it introduces no new learnable parameters versus a standard single-image ViT, since memory comes from the attention pattern plus a fixed sinusoidal temporal position encoding [MEM §III-C]. At $K = 1$ the initialization exactly matches the source VLM because that embedding is $0$ at $t = 0$ [MEM §III-C].\nMEM pre-trains on 6 observations — 5 past plus the current — at a stride of 1 second, and can be expanded in post-training to 18 frames and 54 seconds [MEM §III-D]. Compare that to π0.7\u0026rsquo;s input contract: up to 6 history frames at a stride of 1 second [π0.7 §VI-B]. π0.7 also takes MEM\u0026rsquo;s state handling, projecting each proprioceptive state into the backbone embedding space with a linear projection instead of π0.6\u0026rsquo;s text tokens [MEM §III-D, π0.7 §VI-B].\nThe philosophy to extract The pattern across nine generations is worth stating explicitly, because it is the actual lesson for anyone building such a program: capability accumulates and is not discarded. FAST tokenization, introduced generations earlier, is still the backbone\u0026rsquo;s training signal [π0.7 §III]. Hi Robot\u0026rsquo;s hierarchy is still the subtask interface [π0.7 §V-A]. MEM\u0026rsquo;s encoder is a component, not a replacement [π0.7 §IV].\nπ0.7\u0026rsquo;s own contribution is the conditioning scheme that lets a single generalist hold all of it at once — explicitly including absorbing the behavior of RL-trained specialists, since data collected by π*0.6 during RL training is used as additional training examples in what the paper calls \u0026ldquo;a kind of \u0026lsquo;distillation\u0026rsquo; process\u0026rdquo; [π0.7 §VI-A]. The generalist inherits specialist capability without running RL itself.\nPart 3 — The π0.7 idea: steerable context conditioning I/O contract of this part: in — the lineage; out — the single reframing that defines π0.7, field by field, with every dropout rate.\nCondition on how, not only what Standard instruction-following conditions the policy on what to do: a task string. π0.7\u0026rsquo;s move is to expand the context $C_t$ from a bare instruction into a multimodal bundle that also describes how the episode was produced [π0.7 §III, §IV].\nThe entire idea is legible in a single string [π0.7 §V-E]:\n\u0026lt;Multi-view observation\u0026gt;\u0026lt;Multi-view subgoals\u0026gt; Task: peel vegetables. Subtask: pick up the peeler. Speed: 8000. Quality: 5. Mistake: false. Control Mode: joint.\u0026lt;Proprioception\u0026gt; The fields Subtask instruction $\\tilde{\\ell}_t$ — an intermediate semantic subtask alongside the overall task $\\ell_t$, following π0.5. At inference it comes from a learned high-level policy, from a human coaching the robot, or it is omitted [π0.7 §V-A].\nSubgoal images $g_t$ — multi-view images of a desired near-future state. The base view carries environment and object outcomes; the wrist views carry arm and gripper outcomes [π0.7 §V-B]. This is a visual instruction, and it turns out to be the channel through which cross-embodiment transfer improves most.\nEpisode metadata $m$ — three human-derived labels. Overall speed is the episode length in timesteps, binned in 500-step intervals, so an episode of 1750–2250 steps is presented as \u0026ldquo;2000 steps\u0026rdquo; [π0.7 §V-C]. Overall quality is an integer from 1 to 5, with 5 best [π0.7 §V-C]. Mistake is a boolean per action segment, coarsely annotated by humans [π0.7 §V-C].\nControl mode $c \\in \\{\\mathrm{joint},\\, \\mathrm{ee}\\}$ — whether the action space is joint positions or end-effector poses [π0.7 §V-D].\nMetadata is the thesis in concrete form. \u0026ldquo;Speed: 8000\u0026rdquo; does not describe the task; it describes the episode the actions were drawn from. A slow, clumsy demonstration is no longer noise to be filtered — it is a correctly-labelled example of slow, clumsy behavior, and its presence sharpens the model\u0026rsquo;s notion of what \u0026ldquo;fast\u0026rdquo; and \u0026ldquo;high-quality\u0026rdquo; mean [π0.7 §IX-E].\nDropout is what makes fields optional A context this rich creates a deployment problem: at test time you may not have subgoal images, or a subtask string, or reliable metadata. π0.7\u0026rsquo;s answer is that every field is dropped during training at a specific rate, so the model must stay competent without it [π0.7 §V-E].\nThe exact schedule, all from the same section [π0.7 §V-E]:\nSubgoal images are added to only 25% of examples per batch [π0.7 §V-E] Within those subgoal examples, the subtask instruction is dropped 30% of the time [π0.7 §V-E] Episode metadata is dropped entirely 15% of the time [π0.7 §V-E] Each metadata component — speed, quality, mistake — is additionally dropped at 5% individually [π0.7 §V-E] Control mode has no dropout at all [π0.7 §V-E] Two of those rates deserve attention, because the paper gives an explicit reason for one and a conspicuous silence for the other [π0.7 §V-E].\nThe 25% subgoal rate is not a regularization heuristic. With subgoal images present, the task degenerates toward inverse dynamics — the model can read off the intended outcome and interpolate toward it — and consequently trains much faster [π0.7 §V-E]. Left unchecked, the model would lean on the easy channel and under-train the hard one. Capping subgoals at a quarter of examples keeps the observation-to-action pathway load-bearing. This is a general principle worth stealing: when one input channel makes the task too easy, ration it.\nControl mode getting no dropout is the mirror-image decision [π0.7 §V-E]. Every other field is advisory — a hint about how to behave. Control mode is a hard fact about the action space itself: joint positions and end-effector poses are different output semantics, and a model uncertain about which one it is emitting is not usefully steerable, it is broken. Optional fields get dropout; semantic contracts do not.\nSteering at runtime Because the model has learned the conditional distribution over behavior modes, deployment becomes a matter of asking for the mode you want [π0.7 §VII].\nThe runtime settings are unapologetic: the speed prompt is set per task to the 15th percentile of that task\u0026rsquo;s episode length — that is, ask for a run faster than the great majority of training demonstrations; quality is always set to 5; and mistake is always set to false [π0.7 §VII]. The system is trained on mediocre data and then asked, at inference, to behave like the best of it.\nAnd because dropout produced a model that works both with and without metadata, classifier-free guidance is directly available. π0.7 applies CFG to the episode metadata with weight $\\beta \\in \\{1.3,\\, 1.7,\\, 2.2\\}$ [π0.7 §VII]. You already know this mechanism from diffusion: extrapolate away from the unconditional prediction to push harder toward the conditioned mode. Here the \u0026ldquo;condition\u0026rdquo; is not a text prompt describing content — it is a description of behavioral quality. Turning up $\\beta$ turns up how emphatically the policy commits to being fast, clean and error-free.\nThat is the whole idea [π0.7 §IV]. Everything in the remaining parts is the machinery required to make it work at scale.\nPart 3 hyperparameter table Parameter Value Source Overall speed episode length in timesteps, 500-step bins (1750–2250 → \u0026ldquo;2000 steps\u0026rdquo;) [π0.7 §V-C] Overall quality integer 1–5, 5 best [π0.7 §V-C] Mistake boolean, per action segment, human-annotated [π0.7 §V-C] Control mode joint or ee [π0.7 §V-D] Subgoal images present 25% of examples per batch [π0.7 §V-E] Subtask dropped (within subgoal examples) 30% [π0.7 §V-E] Metadata dropped entirely 15% [π0.7 §V-E] Each metadata component additionally dropped 5% [π0.7 §V-E] Control-mode dropout none [π0.7 §V-E] Runtime speed prompt 15th percentile of task episode length [π0.7 §VII] Runtime quality prompt always 5 [π0.7 §VII] Runtime mistake prompt always false [π0.7 §VII] CFG weight $\\beta$ on metadata 1.3, 1.7 or 2.2 [π0.7 §VII] Part 4 — Subsystem: fleet and teleoperation I/O contract: in — human intent, expressed by an operator driving a robot; out — recorded trajectories of synchronized images and joint positions on physical hardware.\nThis is where a reproduction actually starts, and where most of the money goes. You cannot train the model before this exists, and the paper devotes a full section to the hardware it runs on [π0.7 §VIII].\nThe platforms π0.7 spans four platform families [π0.7 §VIII, Fig. 4]:\nBimanual mobile manipulators — two 6-DoF arms, a 1–2 DoF lift, and a holonomic base [π0.7 Fig. 4] Static bimanual \u0026ldquo;BiPi\u0026rdquo; — two lightweight 6-DoF arms with a 1-DoF gripper [π0.7 §VIII, Fig. 4] Bimanual UR5e — industrial arms with Robotiq grippers [π0.7 §VIII, Fig. 4] Single-arm 6-DoF systems [π0.7 §VIII, Fig. 4] Grippers are parallel-jaw throughout [π0.7 §VIII]. The uniform 6-DoF arm count across families is not an accident: it is what makes a shared joint-space action representation coherent enough to transfer [π0.7 §VIII, Fig. 4].\nControl and sensing Control rates are 50 Hz for everything except the UR5e, which runs at 20 Hz [π0.7 §VIII]. Sensing is a front-facing camera plus one wrist camera per arm, with mobile robots adding a rear-facing camera [π0.7 §VIII]. That is exactly the \u0026ldquo;up to 4 cameras\u0026rdquo; of the input contract: front, two wrist, optional rear [π0.7 §VI-B].\nActuation is deliberately simple. The model\u0026rsquo;s action output is applied through a PD controller, and in end-effector mode a numerical inverse kinematics solver converts target end-effector poses into target joint positions [π0.7 §VIII]. There is no learned low-level controller, no torque-level policy, no model-predictive layer underneath. The learned policy outputs position targets and classical control does the rest.\nThat is worth pausing on, because it inverts a common expectation. The sophistication is entirely in the high-level policy; the bottom of the stack is textbook [π0.7 §VIII]. Inferred: this keeps the sim-to-real-free pipeline tractable and makes the same policy portable across bodies that share an interface, but the paper does not argue it explicitly.\nWhy the UR5e is the hard target The UR5e is the cross-embodiment stress test, and the paper is specific about why: it is significantly longer, morphologically different, much heavier, and has high joint inertia, so it requires a different manipulation strategy and different table positioning [π0.7 §VIII, App. F]. The team did not collect laundry-folding data with the bimanual UR5e at all, and found it generally harder to teleoperate and less suitable for very precise grasps [π0.7 §IX-C].\nThat last fact reframes the whole cross-embodiment result. Transfer to the UR5e is not merely a benchmark — it is the way to get data from a platform that is painful to teleoperate. A model that transfers to a new embodiment opens the possibility of collecting autonomous data there without human teleoperation at all [π0.7 App. F].\nThe transfer also produced genuinely emergent behavior rather than imitation. On the source robot, operators approach cloth with a tilted end-effector to secure fabric against the table before lifting; on the UR5e, π0.7 instead uses a vertical grasp better suited to that arm\u0026rsquo;s kinematics — a strategy that differs from the training data but fits the target body [π0.7 Fig. 13]. Using world-model-generated subgoal images improves this further, labelled π0.7 (GC) in the results [π0.7 §IX-C, Fig. 12].\nPart 4 hyperparameter table Parameter Value Source Control rate 50 Hz; UR5e 20 Hz [π0.7 §VIII] Arm DoF 6 across platforms [π0.7 §VIII, Fig. 4] Mobile manipulator 2 × 6-DoF arms, 1–2 DoF lift, holonomic base [π0.7 Fig. 4] Static bimanual (BiPi) 2 × 6-DoF arms, 1-DoF gripper [π0.7 §VIII, Fig. 4] Grippers parallel-jaw; Robotiq on UR5e [π0.7 §VIII, Fig. 4] Cameras front + one wrist per arm; rear on mobile [π0.7 §VIII] Low-level controller PD [π0.7 §VIII] End-effector mode numerical inverse kinematics → joint targets [π0.7 §VIII] Fleet size undisclosed [π0.7 §VIII] Teleoperation rig undisclosed [π0.7 §VIII] Part 5 — Subsystem: the data engine I/O contract: in — the fleet, human annotators, and the public web; out — an annotated, mixture-weighted training corpus in which every episode carries a description of how it was produced.\nThe eight sources π0.7 trains on eight categories of data [π0.7 §VI-A]:\nDemonstrations across static and mobile, single-arm and bimanual platforms, in lab-like, home-like, and in-the-wild home environments [π0.7 §VI-A] Autonomous data from a large amount of policy evaluations [π0.7 §VI-A] Human interventions within policy rollouts [π0.7 §VI-A] π*0.6 RL-training rollouts, used as additional examples [π0.7 §VI-A] Open-source robot datasets [π0.7 §VI-A] Egocentric human video [π0.7 §VI-A] Auxiliary non-robot web data — object localization, attribute prediction, VQA, text-only [π0.7 §VI-A] Video-language tasks, including captioning of in-house robot data and web video [π0.7 §VI-A] Note the shape of that list. Only the first four involve the fleet. The rest is leverage: web and video data keeps the backbone\u0026rsquo;s semantic knowledge alive while robot data teaches actuation [π0.7 §VI-A].\nThe deliberate departure: keep the bad data Conventional practice curates. π0.7 does the opposite, making heavy use of suboptimal robot data — failure episodes, successful episodes containing substantial mistakes, and data collected by prior model versions during evaluation [π0.7 §VI-A].\nThis is only safe because of metadata. An episode labelled quality 2 with mistake true is not a corrupted training signal; it is a correctly-labelled sample from a region of behavior space the model should be able to represent and then be steered away from [π0.7 §V-C, §VII]. Filtering and conditioning are alternative treatments of the same problem, and π0.7\u0026rsquo;s scaling result is the argument that conditioning dominates [π0.7 §IX-E, Fig. 18].\nThe distillation move The single most consequential line in the data section: data collected by π*0.6 during RL training is used as additional training examples, which the paper describes as \u0026ldquo;a kind of \u0026lsquo;distillation\u0026rsquo; process\u0026rdquo; [π0.7 §VI-A].\nTrace what that means. π*0.6 ran real RL on real robots, using a value function and advantage conditioning to push specialist policies past the quality of human demonstration [π*0.6 §IV, §I]. Those rollouts — including their improvements over teleoperation — become ordinary conditioned training examples for π0.7. The generalist acquires specialist-level behavior without running RL itself, purely through supervised training on RL-generated trajectories carrying metadata about how good they were [π0.7 §VI-A].\nFor a reader from LLM post-training this should land immediately: it is distillation from an RL-tuned teacher into a supervised student, with the twist that the \u0026ldquo;reward\u0026rdquo; survives into the student as a conditioning variable rather than being baked into the weights [π*0.6 §V-B, π0.7 §V-C].\nAnnotation The context fields must come from somewhere, and mostly they come from people [π0.7 §V-A, §V-C].\nSubtask segmentation gives the semantic subtask strings [π0.7 §V-A]. Quality is a 1–5 score [π0.7 §V-C]. Mistake labels are boolean per action segment, described as \u0026ldquo;provided by humans coarsely annotating our data\u0026rdquo; [π0.7 §V-C]. Speed is the only field derived mechanically, as episode length binned in 500-step intervals [π0.7 §V-C].\nThe word \u0026ldquo;coarsely\u0026rdquo; is doing real work there. This is not a high-precision labelling operation — it is a cheap pass over a very large corpus, and the system is designed to tolerate that. Undisclosed: annotation guidelines, inter-annotator agreement, cost per episode, and how disagreements are resolved [π0.7 §V-C].\nAnnotation quality does have one documented hard dependency. For world-model training, label quality — especially temporal segmentation quality — has a large impact on subgoal quality [π0.7 App. C]. Segment boundaries determine what counts as the achieved subgoal, so sloppy boundaries produce a world model that generates the wrong target.\nEvaluation hygiene One footnote carries enormous weight: autonomous data collected in any generalization-focused evaluation task is excluded from training [π0.7 §VI-A fn. 1].\nWithout that exclusion the flywheel would quietly train on the test set [π0.7 §VI-A fn. 1]. The system deploys a policy, records what it does during evaluation, and feeds that back into training — which is exactly how you contaminate a held-out task while believing you measured generalization. This is the kind of discipline that does not show up in a results table and entirely determines whether the results mean anything.\nHow the data is organized Inferred throughout this subsection. π0.7 discloses nothing about its internal storage format [π0.7 §VI-A]. But the schema is legible from the context fields, and the open-source ecosystem provides a concrete instantiation worth teaching.\nThe open analogue splits by access pattern, mirroring the schema the context fields imply [π0.7 §V-C]. LeRobot stores episodes as Parquet tables plus MP4 video with a meta/tasks index, giving map-style random access — the right choice for fine-tuning, where you shuffle over a dataset that fits your storage. RLDS/TFDS streams shards with a large shuffle buffer and weighted interleaving across many datasets — the right choice for pretraining on a mixture too large to index. The principle generalizing beyond either: format follows access pattern follows scale.\nWhat matters more than the file format is the record schema, and that is dictated by the context bundle: per-episode quality, speed and control mode; per-segment subtask text and mistake flags; per-frame images from up to 4 cameras and proprioception [π0.7 §V-C, §VI-B]. Any reproduction must carry those fields as first-class columns, because they are what training conditions on.\nThe scaling payoff Everything above exists to produce one result [π0.7 §IX-E, Fig. 18].\nWith episode metadata, performance keeps improving as the dataset grows even though average data quality falls; without metadata, performance degrades as lower-quality data is added [π0.7 §IX-E, Fig. 18]. The sign of the scaling curve flips based on whether you told the model what it was looking at.\nA second ablation sharpens what \u0026ldquo;diversity\u0026rdquo; means operationally: removing the most task-diverse 20% of robot data hurts substantially more than removing a random 20% [π0.7 §IX-E, Fig. 18]. Data value is not uniform per hour. The tail of unusual tasks carries disproportionate weight, which is an argument for breadth over depth when allocating a collection budget.\nPart 5 hyperparameter table Parameter Value Source Source categories 8 [π0.7 §VI-A] Quality annotation integer 1–5 [π0.7 §V-C] Mistake annotation boolean per action segment, coarse [π0.7 §V-C] Speed derivation episode length, 500-step bins [π0.7 §V-C] Eval hygiene generalization-eval autonomous data excluded [π0.7 §VI-A fn. 1] Diversity ablation most-diverse 20% removal hurts more than random 20% [π0.7 §IX-E, Fig. 18] Pre-training corpus scale (π-series) tens of thousands of hours [π*0.6 §IV-C] Dataset size in hours or episodes undisclosed [π0.7 §VI-A] Per-source mixture weights undisclosed [π0.7 §VI-A] Annotation guidelines and agreement undisclosed [π0.7 §V-C] Storage format inferred from the open-source analogue [π0.7 §VI-A] Part 6 — Subsystem: model architecture I/O contract: in — up to 4 camera views × up to 6 history frames at 448×448, up to 3 subgoal images, proprioception with its history, and the text context; out — 50 action tokens forming one action chunk [π0.7 §VI-B].\nThe parameter budget π0.7 is ~5B parameters on the control path [π0.7 §IV, Fig. 2]. That decomposes into a 4B VLM backbone initialized from Gemma3 4B, which includes a 400M SigLIP-class vision encoder, plus a MEM-style video history encoder, plus an 860M flow-matching action expert [π0.7 §IV, §VI-B, Fig. 2].\nTwo observations for readers calibrated on LLM scales. First, this is small — 5B total, deployable on a single H100 [π0.7 App. D]. The constraint is latency, not capability: a larger backbone would not fit the control loop. Second, the action expert at 860M is a substantial fraction of the whole, roughly a fifth. Action generation is not a thin head bolted onto a language model; it is a major subnetwork with its own objective and its own gradient isolation [π0.7 §III].\nBeside the control path sits the world model, initialized from BAGEL, a 14B mixture-of-transformers, following SuSIE-style initialization [π0.7 §V-B, App. D]. Internally it routes ViT tokens through a 7B LLM backbone and VAE tokens through a 7B generation backbone [π0.7 App. C]. It is nearly three times the size of the policy it advises, and it does not run in the control loop.\nThe high-level policy uses the same architecture as the main model, also based on Gemma3 4B, mapping observations plus the task specification plus past subtask instructions to the next subtask [π0.7 §V-A, App. D].\nThe history encoder The history encoder is where π0.7\u0026rsquo;s memory comes from, and it is inherited from MEM [π0.7 §IV].\nIt applies temporal and spatial compression, and its critical property is that it outputs a fixed number of tokens for any number of history frames — history is compressed to the same token count as a single frame [π0.7 §IV, §VI-B]. That is what makes history affordable: adding 6 frames of memory costs nothing in sequence length at the backbone.\nConcretely, per MEM, the mechanism is a modified ViT attention pattern — bidirectional spatial attention within each observation, causal temporal attention across observations at every 4th layer — passing only the current timestep\u0026rsquo;s representation onward, and introducing no new learnable parameters beyond a fixed sinusoidal temporal position encoding [MEM §III-C]. π0.7\u0026rsquo;s own settings are up to 6 frames per camera at a stride of 1 second, with the entire history dropped at p = 0.3 and the rear view dropped at p = 0.3 [π0.7 §VI-B].\nState: a small change with a clear rationale π0.6 represented the robot\u0026rsquo;s proprioceptive state as discretized text tokens [π*0.6 §V-A]. π0.7 follows MEM and instead embeds proprioception via a linear projection to the backbone dimension, with each history state as its own token, masked out when its frame is dropped [π0.7 §VI-B].\nThe reason is sequence length. With a history of states, text-token representation quickly produces a very large number of state tokens; a linear projection yields exactly one token per state [MEM §III-D]. This is a clean example of a general rule: a representation that is fine for one timestep can be untenable for a sequence. The moment you add memory, per-timestep encoding costs get multiplied, and lossless-but-verbose encodings stop being affordable.\nThe action expert The action expert is an 860M transformer trained with flow matching, using adaptive RMSNorm to inject the flow timestep [π0.7 §VI-B]. It carries exactly 50 action tokens, which attend bidirectionally among themselves and to the backbone\u0026rsquo;s activations [π0.7 §VI-B].\nBidirectional attention among action tokens is the right choice and worth noting [π0.7 §VI-B]: the chunk is generated jointly, not autoregressively, so there is no reason to impose a causal order on positions within it. Causality is a property of time in the world, not of the sampling procedure.\nAttention, precisely The overall pattern is block-causal [π0.7 §VI-B]. Observation tokens and subgoal-image tokens attend bidirectionally within themselves; goal tokens may additionally attend observations; following text tokens use causal attention [π0.7 §VI-B]. When image goals are absent, the pattern matches π0.5\u0026rsquo;s, with global bidirectional attention between embeddings for all memory-aware image views; when goals are present they form an additional block-causal bidirectional block after the text prompt [π0.7 App. B, Fig. 19].\nAnd the load-bearing constraint from Part 1: FAST tokens are available only at training time, and FAST tokens and flow actions do not attend to each other [π0.7 App. B, Fig. 19].\nOne inference-time detail is genuinely clever. For classifier-free guidance, the positive and negative examples are packed into the same sequence as an \u0026ldquo;attention tree\u0026rdquo; with two branches that do not attend to one another [π0.7 App. B]. CFG normally means two forward passes; here it is one pass with a mask. The world model uses the same trick with a more complex mask, since it has 3 CFG groups rather than 2 [π0.7 App. B].\nThe world model The world model\u0026rsquo;s job is to generate subgoal images: given the current observation and the intended subtask, produce what the near future should look like [π0.7 §V-B].\nIts objective is conditional flow matching against a ground-truth subgoal [π0.7 §V-B]:\n$$\\max_{\\psi}\\; \\mathbb{E}\\left[\\, \\mathcal{L}_{\\mathrm{CFM}}\\!\\left( g^{*}_{t},\\; g_{\\psi}(o_t,\\, \\tilde{\\ell}_t,\\, m) \\right) \\right]$$where the ground-truth subgoal $g^{*}_{t} = o_{t_{\\mathrm{end}}}$ is taken from segment ends with especially high-quality subtask labels [π0.7 §V-B]. Each training example consists of a subtask instruction, 3 camera inputs, and 3 target images, where the target timestep is the last of the segment [π0.7 App. C].\nFollowing BAGEL, camera inputs are processed by both a ViT for semantic understanding and a VAE for fine-grained image detail [π0.7 App. C]. The two paths use different resolutions — ViT inputs at 448×336, VAE inputs including targets at 512×384 — because their patch sizes differ, 14 and 16 respectively [π0.7 App. C]. That is a nice concrete instance of a constraint that only shows up in implementation: resolution is downstream of patch size, and mixing two tokenizers means reconciling their grids.\nPart 6 hyperparameter table Parameter Value Source Total parameters (control path) ~5B [π0.7 §IV, Fig. 2] VLM backbone 4B, from Gemma3 4B [π0.7 §IV, Fig. 2] Vision encoder 400M SigLIP-class, inside the 4B [π0.7 §IV, §VI-B] Action expert 860M, flow matching [π0.7 §IV, §VI-B] Timestep injection adaptive RMSNorm [π0.7 §VI-B] Action tokens 50, bidirectional, attend backbone activations [π0.7 §VI-B] History encoder MEM-style, fixed token count for any frame count [π0.7 §IV, §VI-B] State embedding linear projection, 1 token per history state [π0.7 §VI-B] Attention block-causal; FAST ⊥ flow actions [π0.7 §VI-B, App. B, Fig. 19] CFG implementation attention tree, 2 branches in one sequence [π0.7 App. B] World model BAGEL 14B, SuSIE-style init [π0.7 §V-B, App. D] World-model sub-backbones 7B understanding + 7B generation [π0.7 App. C] World-model resolutions ViT 448×336; VAE 512×384 (patch 14 / 16) [π0.7 App. C] High-level policy same architecture, Gemma3 4B [π0.7 §V-A, App. D] Action dimension per embodiment undisclosed [π0.7 §VI-B] Part 7 — Subsystem: the training pipeline I/O contract: in — the annotated corpus plus initialized weights (Gemma3 4B, SigLIP 400M, BAGEL 14B); out — a trained π0.7, a trained world model, and a trained high-level policy [π0.7 §V, §VI].\nThe coupled objectives Training optimizes two objectives on two parameter groups, coupled by one stop-gradient [π0.7 §III].\nThe backbone is supervised with FAST-token cross-entropy. The action expert is supervised with flow matching. The expert attends to backbone activations but its gradients do not reach the backbone [π0.7 §III]. Everything else in this part is about what goes into the context and how often it is withheld.\nEvery dropout and sampling rate This is the part a reproduction cannot guess, so here it is in full [π0.7 §V-E, §VI-B, §VI-C]:\nContext dropout [π0.7 §V-E]:\nSubgoal images: present in 25% of examples per batch [π0.7 §V-E] Subtask instruction: dropped 30% of the time within subgoal examples [π0.7 §V-E] Episode metadata: dropped entirely 15% of the time [π0.7 §V-E] Each metadata component: additionally dropped at 5% [π0.7 §V-E] Control mode: no dropout [π0.7 §V-E] Observation dropout [π0.7 §VI-B]:\nEntire history: dropped at p = 0.3 [π0.7 §VI-B] Rear camera view: dropped at p = 0.3 [π0.7 §VI-B] Subgoal sampling [π0.7 §VI-C]:\nReal subgoals: p = 0.25 taken from the end of the segment, matching the world model\u0026rsquo;s prediction target; p = 0.75 sampled uniformly 0–4 s ahead [π0.7 §VI-C] Generated subgoals: a large number are sampled from the world model and used to construct additional training examples, which mitigates the train/test mismatch between real and generated images [π0.7 §VI-C] That last item deserves emphasis [π0.7 §VI-C]. At deployment, subgoals come from the world model, not from a real future frame — so training only on real frames would create a distribution shift precisely at the moment the feature matters. Generating training subgoals from the same model that will serve them closes the loop. This is the same discipline as RTC: train under the conditions you will deploy under.\nWhy subgoals appear in only a quarter of examples Repeating it here because it is the most instructive hyperparameter in the paper: with subgoal images present, the task degenerates toward inverse dynamics and trains much faster [π0.7 §V-E].\nIf you supplied subgoals always, the model would learn to be an inverse dynamics model — read the goal image, interpolate toward it — and would never develop the harder capability of inferring intent from observation and language alone. The 25% cap is a deliberate handicap on the easy path [π0.7 §V-E].\nTraining for latency Training-time real-time chunking simulates delays of 0–12 timesteps, corresponding to a maximum inference latency of 240 ms on a 50 Hz robot [π0.7 §VI-B]. Unlike test-time RTC, the training-time variant incurs no additional inference-time overhead [π0.7 App. D, ref. 108].\nWorld-model training The world model is initialized from BAGEL and largely uses the same training recipe [π0.7 App. C].\nIts corpus is a subset of the robot data plus egocentric human video with high-quality segmented language labels, mixed with several open-source image-editing datasets and open-source video datasets in order to preserve the model\u0026rsquo;s semantic knowledge [π0.7 App. C]. That mixing is the same instinct as web co-training for the policy: the generative model must keep general visual competence, not collapse onto the robot distribution.\nThe documented sensitivity: label quality, especially temporal segmentation quality, has a large impact on subgoal quality [π0.7 App. C]. Subgoal targets are defined as segment ends, so segment boundaries are the supervision signal.\nA payoff worth naming: because the world model is trained partly on non-robot data, semantic and physical concepts from the web can transfer into π0.7 indirectly, through generated subgoal images [π0.7 §V-B]. The subgoal channel is a second route for web knowledge to reach the policy, bypassing the backbone entirely.\nWhat training does not tell you Everything above is disclosed. The following is not, and any reproduction must decide it independently [π0.7 §VI]:\nTotal token or step budget, and training compute [π0.7 §VI] Optimizer, learning-rate schedule, warmup, weight decay [π0.7 §VI] Batch size and sequence packing strategy [π0.7 §VI] Hardware and wall-clock time for training [π0.7 §VI] Dataset size in hours or episodes, and per-source mixture weights [π0.7 §VI-A] World-model corpus size and composition beyond its qualitative description [π0.7 App. C] High-level-policy training data and hyperparameters [π0.7 §V-A] That is a substantial list, and it is the honest headline of this part: the context recipe is fully specified; the optimization recipe is not [π0.7 §VI].\nPart 7 hyperparameter table Parameter Value Source Backbone objective FAST-token cross-entropy [π0.7 §III] Action-expert objective flow matching [π0.7 §III] Coupling stop-gradient; expert reads, does not modify [π0.7 §III] Subgoal images present 25% of examples [π0.7 §V-E] Subtask dropped within those 30% [π0.7 §V-E] Metadata dropped entirely 15% [π0.7 §V-E] Each metadata component +5% [π0.7 §V-E] Control-mode dropout none [π0.7 §V-E] History dropped p = 0.3 [π0.7 §VI-B] Rear view dropped p = 0.3 [π0.7 §VI-B] Real subgoal sampling p = 0.25 segment end; p = 0.75 uniform 0–4 s ahead [π0.7 §VI-C] Generated subgoals large number from the world model [π0.7 §VI-C] RTC simulated delay 0–12 timesteps = 240 ms at 50 Hz [π0.7 §VI-B] World-model objective conditional flow matching to segment-end frames [π0.7 §V-B] Optimizer, schedule, batch size undisclosed [π0.7 §VI] Steps, compute, wall-clock undisclosed [π0.7 §VI] Part 8 — Subsystem: the serving runtime I/O contract: in — a live observation stream from up to 4 cameras plus proprioception; out — joint commands at 20 or 50 Hz, continuously, without pauses [π0.7 §VII, §VIII].\nAlgorithm 1, walked The deployment loop has four moving pieces [π0.7 §VII, Alg. 1]:\nSample a subgoal from the world model: $g^{*} \\sim p_{\\psi}(\\cdot \\mid o_t,\\, \\tilde{\\ell},\\, m)$ [π0.7 §VII, Alg. 1] Assemble the context $C = \\{\\ell,\\, \\tilde{\\ell},\\, g^{*},\\, m,\\, c\\}$ — overall task, subtask, subgoal images, metadata, control mode [π0.7 §VII, Alg. 1] Sample an action chunk $a_{t:t+H} \\sim \\pi_{\\theta}(\\cdot \\mid o_{t-T:t},\\, C)$ using 5 denoising steps [π0.7 §VII, Alg. 1] Re-plan: execute $\\tilde{H}$ steps, then re-infer; re-sample subgoals when the subtask $\\tilde{\\ell}$ changes or a $\\Delta$-timer elapses [π0.7 §VII, Alg. 1] The runtime constants: 5 denoising steps; execute 15 or 25 of the 50-step chunk; refresh subgoals on semantic-intent change or every $\\Delta = 4$ s, whichever comes first; CFG weight $\\beta \\in \\{1.3,\\, 1.7,\\, 2.2\\}$ on the episode metadata [π0.7 §VII]. The 4-second interval is chosen to match SuSIE [π0.7 App. C].\nAsynchrony is the load-bearing idea Subgoal generation and subtask generation run in separate threads, and VLA inference always uses the latest available output from each [π0.7 §VII, Alg. 1].\nThe reason is arithmetic. Generating a subgoal image takes 1.25 seconds — 25 denoising steps of a 14B model over a sequence of nearly 10,000 tokens, even with 4-way tensor parallelism across 4×H100 GPUs, 8-bit quantization of all large matrix multiplications, and a modified SageAttention kernel for backbone attention [π0.7 App. D]. Meanwhile the robot needs a new action every 20 ms at 50 Hz. A synchronous design would stall the robot for over a second every time it wanted a new subgoal.\nSo π0.7 executes with what the paper calls a naive asynchronous strategy: it keeps acting on the current subgoal while the world model computes the next one [π0.7 App. D]. Staleness is accepted as the price of continuity — a 4-second-old subgoal is a perfectly serviceable target when subgoals describe intent rather than precise setpoints [π0.7 §V-B].\nPrimer — why async matters. The alternative is a robot that visibly stops between chunks: reach, freeze, reach, freeze. Beyond looking broken, pausing mid-motion changes the physics — a half-completed grasp under gravity is not a stable state you can resume from. Continuous execution is a functional requirement, not polish.\nThe latency budget, end to end The numbers close the loop with Part 7\u0026rsquo;s training assumption [π0.7 App. D]:\n38 ms — the minimal variant, with 3 camera inputs, 5 denoising steps and training-time RTC [π0.7 App. D] 127 ms — the worst case, with the MEM vision encoder enabled and subgoal images in context [π0.7 App. D] 240 ms — the budget the policy was trained to tolerate, from RTC\u0026rsquo;s simulated 0–12 timestep delays at 50 Hz [π0.7 §VI-B] All on a single NVIDIA H100 for both π0.7 and the high-level policy, both being Gemma3 4B based [π0.7 App. D]. The world model gets its own 4 GPUs [π0.7 App. D].\nRead those three numbers together and the engineering discipline is obvious [π0.7 §VI-B, App. D]: the worst realistic case sits at roughly half the trained tolerance. The optional features — memory, subgoals — are affordable precisely because the floor was driven down first.\nPart 8 hyperparameter table Parameter Value Source Denoising steps 5 [π0.7 §VII] Executed horizon $\\tilde{H}$ 15 or 25 of 50 [π0.7 §VII] Subgoal refresh $\\Delta$ 4 s, or on intent change [π0.7 §VII, App. C] CFG weight $\\beta$ 1.3, 1.7 or 2.2 on metadata [π0.7 §VII] Speed prompt 15th percentile of task episode length [π0.7 §VII] Quality prompt always 5 [π0.7 §VII] Mistake prompt always false [π0.7 §VII] VLA inference hardware single NVIDIA H100 [π0.7 App. D] VLA latency 38 ms minimal / 127 ms worst case [π0.7 App. D] World-model hardware 4-way tensor parallel on 4×H100 [π0.7 App. D] World-model quantization 8-bit for all large matmuls [π0.7 App. D] World-model attention modified SageAttention [π0.7 App. D] Subgoal generation cost 25 denoising steps in 1.25 s [π0.7 App. D] World-model sequence length nearly 10,000 tokens [π0.7 App. D] Full serving stack undisclosed [π0.7 App. D] Part 9 — Subsystem: the evaluation apparatus I/O contract: in — a trained policy; out — capability claims you can actually trust.\nFor an LLM researcher this is the most alien part of the system, because there is no held-out split to score — every number below cost robot-hours on physical trials [π0.7 §IX-E].\nThree metrics π0.7 reports success rate, task progress as a percentage, and normalized throughput in successes per hour [π0.7 Fig. 6].\nThroughput is the one that will surprise you, and it is a production metric rather than a research one. A policy that succeeds reliably but slowly is worth less than one that succeeds slightly less often but far faster, because the deployed quantity of interest is tasks completed per hour of robot time. Notably, π0.7 exceeds RL-trained specialists on throughput for diverse laundry folding and box building [π0.7 Fig. 6] — the generalist beating the specialists on the specialists\u0026rsquo; own axis, which is the strongest evidence for the distillation claim in Part 5.\nTask progress requires a rubric, and Appendix G supplies one per task [π0.7 App. G]. These are hand-designed partial-credit scales: Take Out Trash has a maximum of 12 points, Peel Fruits and Vegetables 9, Make Peanut Butter Sandwich 9, Turn a T-Shirt Inside Out 7, Shirt Folding 6 [π0.7 App. G]. Shirt Folding awards a point per completed fold with an alignment tolerance, plus a fold-quality score from 0–3, and a full score of 6 counts as success [π0.7 App. G].\nTwo things follow. First, \u0026ldquo;task progress\u0026rdquo; is not a learned or automatic measure — it is a human scoring a video against a checklist. Second, the rubrics are task-specific, so cross-task progress comparisons are only meaningful after normalization [π0.7 App. G].\nThe protocols Instruction following is tested across 14 scenarios with sequences of 3–6 open-ended instructions each, in 4 unseen kitchens and 2 unseen bedrooms [π0.7 §IX-B, Fig. 9].\nCross-embodiment transfer is tested by training on one robot and evaluating on another, with the bimanual UR5e as the hard target [π0.7 §IX-C]. An appendix comparison found joint-space and end-effector control show no clear advantage for EE across tasks, so the main cross-embodiment experiments use joint space for clarity [π0.7 App. E, Fig. 20].\nAblations carry the thesis. π0.7 without metadata and π0.7 without evaluation data both lose, most prominently in throughput [π0.7 Fig. 7]. And the scaling and diversity results of Figure 18 are what license the central claim [π0.7 §IX-E, Fig. 18].\nThe human baseline The most interesting evaluation in the paper is the one against people [π0.7 App. F].\nTen operators from the top 2 percentile of experience, averaging around 375 hours of teleoperation across all platforms, were asked to fold shirts on the bimanual UR5e [π0.7 App. F, Fig. 21]. Crucially, none had prior experience performing shirt folding on the UR5e — mirroring the zero-shot setting of the learned policy [π0.7 App. F]. Each performed 3 trials for 30 total, with no practice or warm-up before the first attempt, under the same initial configuration, time limit and criteria used for policy evaluation [π0.7 App. F].\nResult: human operators achieved 90.9% task progress and 80.6% success; π0.7 achieved 85.6% task progress and 80% success [π0.7 §IX-C, App. F, Fig. 22].\nRead that carefully before drawing conclusions. It is not \u0026ldquo;robots match humans.\u0026rdquo; It is a specific and narrower claim: on one dexterous task, on a robot body neither the policy nor the operators had used for that task, a generalist policy scores comparably to expert teleoperators attempting it cold [π0.7 App. F]. The baseline is zero-shot human, not practiced human. What makes it compelling is the direction of the comparison — the policy is being measured against the mechanism that produced its own training data.\nWhy physical evaluation is hard Every trial costs wall-clock robot time, so sample sizes are small and variance is large. Scenes cannot be reset exactly, so trials are not independent draws from a fixed distribution. And the honest limitation the authors state themselves: at this data scale it is practically difficult to definitively determine which tasks are truly \u0026ldquo;seen\u0026rdquo; or \u0026ldquo;unseen\u0026rdquo; [π0.7 §IX-E].\nThat last one deserves respect rather than skepticism. With a corpus of tens of thousands of hours spanning many tasks and environments [π*0.6 §IV-C], \u0026ldquo;unseen\u0026rdquo; becomes a claim about the whole training distribution that nobody can fully verify. The eval-exclusion footnote from Part 5 is what makes the claim tractable at all [π0.7 §VI-A fn. 1].\nPart 9 hyperparameter table Item Value Source Metrics success rate; task progress %; normalized throughput [π0.7 Fig. 6] Instruction-following protocol 14 scenarios, 3–6 instructions each [π0.7 §IX-B, Fig. 9] Unseen environments 4 kitchens, 2 bedrooms [π0.7 §IX-B] Human study 10 operators, top 2 percentile, ~375 hours experience [π0.7 App. F, Fig. 21] Human study protocol 3 trials each, 30 total, no warm-up [π0.7 App. F] Human vs π0.7 (UR5e shirt fold) 90.9% / 80.6% vs 85.6% / 80% [π0.7 §IX-C, Fig. 22] Rubric maxima (examples) Take Out Trash 12, Peel 9, Sandwich 9, Shirt Folding 6 [π0.7 App. G] Shirt-fold quality score 0–3, full score of 6 = success [π0.7 App. G] Action-space finding no clear EE advantage; joint space used [π0.7 App. E, Fig. 20] Diversity ablation most-diverse 20% \u0026gt; random 20% in impact [π0.7 §IX-E, Fig. 18] Generalization gap \u0026gt;90% in-distribution; 60–80% unseen [π0.7 §X] Per-trial raw data undisclosed [π0.7 §IX] Part 10 — The reproduction blueprint I/O contract: in — everything above; out — a build order, an honest ledger of what you must decide yourself, and a realistic account of what the whole thing costs.\nBuild sequencing The dependencies are strict, and they run in the opposite direction from where most ML teams want to start [π0.7 §VI, §VIII].\nStage 1, fleet and teleoperation. Robots, cameras, PD control, IK, and the rigs operators use to drive them [π0.7 §VIII]. Nothing downstream exists without this.\nStage 2, data engine and annotation. Collection across environments, plus the annotation pipeline producing subtask segmentation, quality scores, mistake flags and speed bins [π0.7 §V-A, §V-C, §VI-A].\nStage 3, base-model initialization. Gemma3 4B, SigLIP 400M, BAGEL 14B [π0.7 §IV, §V-B]. This stage is a download; it is the cheapest in the list.\nStage 4, training. The KI-coupled objectives, the full context dropout schedule, RTC delay simulation [π0.7 §III, §V-E, §VI-B].\nStage 5, serving. Async threads, denoising, chunked execution, GPU provisioning [π0.7 §VII, App. D].\nStage 6, evaluation. Four axes, three metrics, per-task rubrics, human operators to run trials [π0.7 §IX, App. G].\nThen the flywheel closes: rollouts and interventions from stage 6 re-enter stage 2 as training data — minus anything from generalization-focused evaluation tasks [π0.7 §VI-A, §VI-A fn. 1].\nComponent selection For each module, what π0.7 used and what an outside team would choose. This is component selection for a system build, not a code port [π0.7 §IV, §V-B]:\nModule π0.7\u0026rsquo;s choice Availability VLM backbone Gemma3 4B open weights [π0.7 ref. 106] Vision encoder SigLIP-class 400M, from Gemma3 open weights [π0.7 §IV] History encoder MEM-style video encoder published method, no weights [π0.7 ref. 37] Action expert 860M flow-matching transformer trained from scratch [π0.7 §VI-B] Action tokenizer FAST published, open [π0.7 ref. 104] World model BAGEL 14B open weights [π0.7 ref. 105] Subgoal initialization SuSIE-style published [π0.7 ref. 93] Latency handling training-time RTC published [π0.7 ref. 108] Attention kernel modified SageAttention published [π0.7 ref. 112] The striking thing about that table is how much is available. Every initialization point is a public checkpoint or a published method. What you cannot obtain is the data and the organization [π0.7 §VI-A, App. A].\nThe organizational reality The paper\u0026rsquo;s own contribution list names 7 categories: data collection and operations; annotation and supplemental data; policy training and research; policy infrastructure; robot hardware; robot infrastructure; and writing and illustration [π0.7 App. A]. Counting names, 24 people appear under data collection and operations, 22 under robot hardware, and 10 under robot infrastructure [π0.7 App. A].\nLook at that distribution. The largest single group is data collection and operations, and the second largest is robot hardware. A reproduction that budgets like an ML project — GPUs and researchers — has mis-budgeted by a wide margin. The organization is a component of the system [π0.7 App. A].\nThe gap ledger Everything a reproduction must decide without guidance [π0.7 §VI, §VI-A, §V-A, §V-C, App. C]:\nGap Status Source Dataset size in hours, episodes or trajectories undisclosed [π0.7 §VI-A] Per-source mixture weights undisclosed [π0.7 §VI-A] Optimizer, LR schedule, batch size undisclosed [π0.7 §VI] Total steps, compute, wall-clock undisclosed [π0.7 §VI] Annotation guidelines, inter-annotator agreement undisclosed [π0.7 §V-C] World-model corpus size and composition undisclosed [π0.7 App. C] High-level-policy training data and hyperparameters undisclosed [π0.7 §V-A] Number of robots in the fleet undisclosed [π0.7 §VIII] Teleoperation interface and hardware undisclosed [π0.7 §VIII] Action dimension per embodiment undisclosed [π0.7 §VI-B] Per-trial raw evaluation data undisclosed [π0.7 §IX] Full serving stack undisclosed [π0.7 App. D] Data storage format inferred from LeRobot / RLDS [π0.7 §VI-A] π0.6 architecture details partly recoverable [π*0.6 §V-A] Two entries in that table deserve comment [π0.7 ref. 42].\nπ0.6 is a model card, not a paper [π0.7 ref. 42]. Its immediate predecessor therefore has no citable specification — except that π*0.6 documents it in passing: derived from π0.5, Gemma 3 4B base, 860M action expert, KI training, joint angles and gripper commands at 50 Hz [π*0.6 §V-A]. Reading the successor to recover the predecessor is a habit worth developing.\nπ0.7 has no public weights and no public code. The openpi repository ships π0 and π0.5 only [openpi repository]. Anything you build is a reimplementation from the paper, not a fork.\nWhat this actually costs Stating it plainly, since the parts above imply it without saying it. The published details are densest exactly where reproduction is cheapest — architecture and hyperparameters, which are a download and a config file [π0.7 §IV, §VI-B]. They are sparsest exactly where reproduction is most expensive — the corpus, the annotation operation, and the fleet [π0.7 §VI-A, §VIII].\nThat is not a criticism of the paper, which is unusually generous with its numbers. It is the structural condition of the field: the moat is the data engine, and data engines are not publishable artifacts [π0.7 §VI-A].\nPart 11 — Principles and open problems I/O contract: in — the complete system; out — the transferable ideas, and the honest limits.\nFive principles worth stealing Condition on provenance instead of filtering. The default response to mixed-quality data is to discard the bad half. π0.7 keeps it and labels it, and the scaling curve changes sign as a result [π0.7 §IX-E, Fig. 18]. Generalizes anywhere you have abundant, heterogeneous, unevenly-good data.\nMake quality a variable, not a gate. Once quality is an input field rather than an admission criterion, low-quality data becomes informative rather than harmful, and quality becomes something you request at inference [π0.7 §V-C, §VII].\nLet context select the behavior mode at runtime. Speed, quality and error-freeness are set at deployment — 15th percentile speed, quality 5, mistake false — and can be pushed harder with CFG at β up to 2.2 [π0.7 §VII]. One trained model, many behavior modes, selected without retraining.\nDistill specialists into the generalist. Rather than maintaining a fleet of task-specific policies, run RL where it pays and fold the resulting rollouts back into the generalist\u0026rsquo;s training set as conditioned examples [π0.7 §VI-A, π*0.6 §IV]. The generalist then beats those specialists on throughput for laundry folding and box building [π0.7 Fig. 6].\nAdd capability without discarding it. Nine generations, and FAST tokenization, Hi Robot\u0026rsquo;s hierarchy, KI\u0026rsquo;s firewall, RTC\u0026rsquo;s latency handling and MEM\u0026rsquo;s memory are all still present in the final system [π0.7 §III, §V-A, §VI-B].\nAnd one that is really a corollary of the mask-versus-stop-gradient distinction: separate what a component may see from what it may modify [π0.7 §III, App. B]. Two different mechanisms, two different failure modes, and conflating them is how knowledge insulation gets implemented incorrectly.\nThe open problems The paper is candid about where it stands. In-distribution success exceeds 90%, while unseen tasks or unseen task–robot combinations sit at 60–80% [π0.7 §X]. That gap is the frontier.\nThe proposed direction is to exploit steerability itself for efficient learning on the test task — through detailed language coaching, or through autonomous RL [π0.7 §X]. Note how neatly that closes the circle: the conditioning scheme built to absorb heterogeneous training data becomes the interface through which a deployed robot is taught something new.\nThree harder problems sit underneath. These are mine rather than the paper\u0026rsquo;s — treat them as inferred, though each rests on a disclosed fact [π0.7 §IX-E, §V-C, App. D]:\nEvaluation does not scale with capability. Every claim costs robot-hours, sample sizes stay small, and the authors themselves cannot fully determine what is truly unseen [π0.7 §IX-E]. As models improve, distinguishing 85% from 90% requires more trials than anyone wants to run.\nMetadata quality is a hidden dependency. The entire thesis rests on labels that are, by the paper\u0026rsquo;s own description, coarse [π0.7 §V-C]. Nobody outside the lab knows how much slack the method has before conditioning degrades, because agreement rates are undisclosed [π0.7 §V-C].\nThe world model is expensive and off the critical path. It costs 4 H100s and 1.25 s per subgoal to advise a policy that itself runs in 38 ms on one GPU [π0.7 App. D]. It earns that cost in cross-embodiment transfer [π0.7 §IX-C, Fig. 12], but the ratio invites either a much cheaper subgoal generator or a tighter integration.\nWhere this leaves you If you take one thing from π0.7, take the reframing rather than the architecture. The architecture is a well-executed assembly of published components [π0.7 §IV]. The reframing — that the description of how an episode was produced belongs in the context window, and that doing so converts your worst data from a liability into signal — is the part that changed the scaling curve [π0.7 §IX-E, Fig. 18].\nBibliography The π series and its dependencies, all verified against π0.7\u0026rsquo;s own reference list [π0.7 bibliography]:\nWork Title arXiv π0 π0: A Vision-Language-Action Flow Model for General Robot Control 2410.24164 [π0.7 ref. 10] π0-FAST FAST: Efficient Action Tokenization for Vision-Language-Action Models 2501.09747 [π0.7 ref. 104] Hi Robot Hi Robot: Open-Ended Instruction Following with Hierarchical VLA Models 2502.19417 [π0.7 bibliography] π0.5 π0.5: A VLA with Open-World Generalization 2504.16054 [π0.7 ref. 14] KI Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better 2505.23705 [π0.7 ref. 103] RTC Real-Time Execution of Action Chunking Flow Policies 2506.07339 [π0.7 ref. 107] π*0.6 π*0.6: a VLA That Learns From Experience 2511.14759 [π0.7 ref. 50] MEM MEM: Multi-Scale Embodied Memory for Vision Language Action Models 2603.03596 [π0.7 ref. 37] π0.7 π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities 2604.15483 [π0.7 title page] External components [π0.7 bibliography]:\nWork Role arXiv Gemma 3 VLM backbone initialization technical report [π0.7 ref. 106] BAGEL world-model initialization 2505.14683 [π0.7 ref. 105] SuSIE subgoal-generation approach 2310.10639 [π0.7 ref. 93] Training-time RTC latency handling used by π0.7 2512.05964 [π0.7 ref. 108] SageAttention 8-bit attention for world-model serving 2410.02367 [π0.7 ref. 112] Flow matching action-expert objective 2210.02747 [π0.7 ref. 102] Classifier-free guidance metadata guidance at inference 2207.12598 [π0.7 ref. 109] π0.6 has no arXiv entry — it is a model card [π0.7 ref. 42]. Its architecture is partly recoverable from π*0.6 §V-A.\nAll values in this post are checked mechanically against a fact ledger; every number appearing in the prose is traceable to a cited source, and everything the papers do not disclose is named as such in Part 10 [π0.7 §VI].\n","permalink":"https://mzf666.github.io/vla/en/posts/pi07-generalist-physical-ai/","summary":"A reproduction-grade walkthrough of π0.7 — every disclosed hyperparameter, every subsystem\u0026rsquo;s I/O contract, and an honest ledger of what the paper does not tell you.","title":"How to Build Generalist Physical AI: π0.7 from Beginner to Pro"}]