Introducing PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud

Abstract

Physical AI is advancing rapidly, but the inference infrastructure behind it has yet to keep pace. Real-time deployment at the edge demands ultra low latency; cloud based reinforcement learning requires high throughput and large-scale rollouts; and model architectures such as VLAs and WAMs continue to evolve. As a result, research prototypes often require extensive, repetitive adaptation before they can achieve efficient inference and be deployed on real-world devices.

To address these challenges, we are officially open-sourcing PhyAI, an inference framework purpose built for Physical AI. Designed for both ultra fast edge inference and large scale RL rollouts in the cloud, PhyAI makes performance, ease of use, and flexibility its core priorities. Our goal is to provide unified, extensible infrastructure that connects model training, reinforcement learning rollouts, and real-world deployment, allowing developers to focus on model and algorithm innovation instead of repeatedly wrestling with inference systems.

Compared with implementations from the models’ official repositories, PhyAI delivers approximately 2.85x, 1.82x, and 2.28x speedups for π0, π0.5, and GR00T, respectively, on an NVIDIA GeForce RTX 5090. For GR00T, it also achieves approximately 1.40x speedup on NVIDIA Thor and 4.31x on an NVIDIA A40.

These results are only the beginning. We will continue to advance kernel optimization, execution scheduling, model integration, and numerical accuracy alignment. At the same time, Physical AI operates in a real world where actions cannot always be undone. Inference infrastructure, therefore, must be designed not only for speed, but also for reliability, correctness, interruptibility, and safety guarantees under worst-case conditions.

PhyAI is being built in the open, together with the broader community. We hope it will become shared infrastructure connecting research prototypes, cloud-scale computing, and real-world edge deployment, continually lowering the cost of Physical AI inference and helping bring Physical AI into the world more efficiently, more accessibly, and more responsibly.

Why we need a new infra for Physical AI Models

Physical AI is evolving at a rapid pace, but the infrastructure supporting it hasn’t kept up. Demand for inference is surging across both on-device deployment and cloud-side RL rollout, yet the industry still lacks a truly usable inference framework. A “vLLM/SGLang” for embodied intelligence has yet to emerge.

The difficulty lies in the nature of the field itself. Ideas are abundant and model architectures shift dramatically from VLAs to WAM, the structural gaps between approaches dwarf the variants seen in the LLM world. At the same time, the pipeline from training and RL samplinge.g.: As a backend of RLInf ’s rollout engine. to real-world deployment should ideally be seamless, not a gauntlet of repeated adaptation. This demands that inference frameworks place usability and flexibility at the top of the priority list.

Looking at existing work such as FlashRThttps://github.com/flashrt-project/FlashRT which support Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST, etc. They has fastest model inference on Nvidia Thor for now. and realtime-vlahttps://github.com/Dexmal/realtime-vla Embodied.cpp, Robot.cpp Also did great works on Embodied inference infra, : ) , their explorations into inference speed are admirable, but they are fundamentally purpose-built optimizations tied to specific architectures. Once the structure changes, adaptation costs spike, making it hard to keep pace with the fast-moving research frontier.

So we chose a different approach, and we’re doing it in the open. We are building an inference framework for Physical AI that puts usability first. Designed for the broader community, not for any single organization to narrow the gap between research prototypes and efficient deployment. Developers shouldn’t have to wrestle with inference infrastructure; they should be free to focus on exploring new model architectures. This is not something one person or one team can finish alone, and we hope to build it together with the community.

Inside PhyAI

Architecture

PhyAI separates model-specific execution from a shared latency-first runtime. Each model keeps an execution path that reflects its real structure instead of being forced into an LLM-shaped abstraction. Beneath those paths, the engine provides common payloads, model execution, runtime scheduling, CUDA graph management, parallel dispatch, virtual GPU topology, weight and cache handling, and optimized kernels. The same path can then serve latency-critical robots at the edge or high-throughput reinforcement-learning rollouts in the cloud.

Features

• 🚀 Support for Jetson series edge devices.
• 🚀 Scale to clusters with DP, TP, and CFG parallelism.
• 🚀 High-performance kernel integrations, including FlashInfer and Humming.
• 🤗 Quantization support for W4A8 (NVFP4, MXFP4, INT4), W8A8, and W8A16.

How we optimize PI0.5 and GR00T

PI0.5 and GR00T differ at the model level, but their inference hotspots are similar: both build a large vision-language condition and then repeatedly execute an action or diffusion path. PhyAI applies the same optimization toolkit to both models, focusing on kernel efficiency, memory traffic, and removing repeated runtime overhead.

Optimizations

• Operator fusion. PhyAI provides custom Triton kernels for AdaRMSNorm and fused residual-add plus RMSNorm. Separately, PI0.5 combines Q, K, and V projection into one GEMM, merges the MLP gate and up projections, uses fused GeGLU or SwiGLU activation-and-multiply kernels, and performs gated residual updates with a fused multiply-add. These changes remove launches and intermediate tensor writes from every transformer layer.
• Timestep reuse and one denoising graph. For PI0.5’s standard 10 Euler steps, PhyAI builds the complete sinusoidal timestep-embedding table and every layer’s AdaRMS modulation table once before denoising. The full 10-step loop is then captured in a single CUDA Graph, removing nine additional graph launches together with Python work and device synchronization between steps.
• Prefix and KV reuse. The vision-language prefix is computed once per action chunk, and its KV cache is retained throughout all action or denoising steps. Attention plans, index metadata, and fixed buffers are also reused while shapes remain unchanged, so the repeated path only processes the changing action tokens instead of rerunning the vision tower and language prefix.
• A profiled attention path. PI0.5’s action expert issues a 50-token query against a much longer cached prefix at a head dimension of 256. For this specific workload, PhyAI pins FlashInfer’s FA2 prefill kernel instead of accepting the auto-selected FA3, provisions its workspace during setup, and refreshes the attention plan once per inference before CUDA Graph replay. The resulting plan and static buffers are then shared by all 10 Euler steps.

The benchmark results below use BF16, so the reported BF16 speedups are not attributed to low-precision quantization.

Benchmark results

PI0.5 benefits substantially from batching, but only up to the point where the hardware is saturated. On a Hopper-series GPU, amortized execution time falls from 22.6 to 10.0 ms per sample as batch size grows from 1 to 32, while throughput rises from 44.3 to 100.0 samples per second. This is a throughput metric, not request latency: the corresponding full-batch execution times are 22.6 and 320 ms. Most of the useful throughput gain has already arrived by batch size 16. Thor reaches its practical throughput plateau earlier, around batch sizes 8 to 16.

The stage breakdown explains why batching helps. At batch size 1, the action expert accounts for only 8.8% of estimated FLOPs but 57.2% of latency, indicating that small GEMMs, weight traffic, and launch overhead dominate. As batch size grows, the expert becomes more efficient and its latency share falls to 13.5%; the vision tower and LLM prefix then become the larger targets.

GR00T shows a similar systems pattern across hardware. PhyAI reduces latency relative to the official implementation by 1.40x on Thor, 2.28x on RTX 5090, and 4.31x on A40. It is not the fastest implementation in every configuration: FlashRT leads on Thor and is narrowly ahead on RTX 5090 and A40. The important result is that PhyAI stays close to a specialized runtime while retaining a shared, extensible execution framework.

Latency in real-world control

Lower latency is always useful, but it should not be the only objective of a Physical AI inference system. Under synchronous execution, a robot completes an action chunk, stops, waits for the next inference call, and only then begins moving again. In that setting, every millisecond of model latency appears directly as downtime, so optimizing latency can feel like an endless race.

Real-time chunking (RTC) changes this relationship.Real-Time Action Chunking with Large Models, Physical Intelligence, June 2025. RTC formulates action transitions as an inpainting problem, enabling continuous execution without training-time changes. While the robot executes its current action chunk, the model can already generate the next one. RTC uses the actions shared by the two chunks to produce a consistent handoff, allowing the robot to keep moving instead of waiting for inference to finish. Physical Intelligence showed that RTC remained robust with an additional 100 or 200 milliseconds of injected delay and could still complete precise tasks when total inference delay exceeded 300 milliseconds. Once inference fits comfortably within the remaining execution window, a small reduction in raw model latency may no longer translate into a visible improvement in robot behavior.

This does not make latency irrelevant. If inference takes longer than the remaining action horizon, if tail latency is unpredictable, or if the robot must react immediately to an unexpected event, stale actions can still become unsafe. The target therefore shifts from minimizing average latency at any cost to meeting a real-time latency budget reliably. Tail latency, jitter, observation freshness, action continuity, and interruptibility matter alongside raw speed.

For PhyAI, this means we will continue to optimize PI0.5 and GR00T, but we will not treat a single benchmark number as the finish line. Speed is most valuable when it creates room for larger and more capable models, more frequent replanning, and safer execution. RTC does not eliminate the need for fast inference; it turns latency from an absolute objective into a bounded systems constraint.

How we optimize Cosmos3-Nano-Policy-DROID

Cosmos3-Nano-Policy-DROID is a substantially heavier workload than a conventional VLA action expert. In policy mode, it jointly denoises a rollout-video latent and a sequence of action tokens: the observation frame remains clean, while the future video and the complete action chunk are updated by separate UniPC solvers. With classifier-free guidance (CFG), every denoising step also needs both a conditional and an unconditional transformer evaluation.

Reusing the invariant condition

Cosmos3 separates its transformer into an UND text path and a GEN video-action path. For each CFG branch, PhyAI runs the UND text tower only once, keeps its per-layer K/V tensors, and precomputes the rotary positions for the GEN sequence. The denoising loop then reuses this condition at every timestep and reruns only the changing video-action path. Video and action tokens stay packed into one GEN sequence, so one transformer pass predicts both velocities without launching a separate action model.

Two-dimensional parallelism

The multi-GPU policy runner arranges devices as cfg_size × tp_size. Tensor parallelism operates inside each CFG branch: fused QKV projections and MLP gate/up projections are column-sharded, attention heads are divided across ranks, and the attention-output and MLP-down projections reduce their partial results across the TP group. This splits both model weights and the dominant matrix multiplications.

When cfg_size=2, one complete TP group evaluates the conditional branch while a second TP group evaluates the unconditional branch. After each denoising step, matching TP ranks exchange the video and action velocities with an all-gather along the CFG axis. Every rank then applies the same guidance equation and advances both UniPC solvers, leaving the video and action state identical for the next step. A CFG 2 × TP 4 deployment therefore uses eight GPUs and replaces two sequential branch evaluations with two concurrent four-way tensor-parallel evaluations.

If rollout-video pixels are requested, the replicated WAN VAE uses all cfg_size × tp_size ranks differently: it divides the latent frame into overlapping spatial tiles, decodes one tile per rank, and reconstructs the frame with feathered overlap. This optional decode is outside the action-serving critical path and is not included in the latency numbers below.

Single-GPU benchmark

At batch size 1, PhyAI produces a 32-action chunk in 1.132 seconds on a Hopper-series GPU and 8.121 seconds on NVIDIA Thor. These measurements use the eager single-GPU policy runner; they do not include the multi-GPU strategy illustrated above.

Accelerating VLA Reinforcement Learning with PhyAI

Efficient rollout is critical to reinforcement learning because trajectory generation can consume a large share of total training time. LLM RL systems such as SlimeSlime is an LLM post-training framework that connects distributed training with high-throughput rollout generation. and VeRLVeRL is a distributed reinforcement-learning framework for large language models with pluggable training and rollout backends. pair training with specialized inference engines such as vLLMvLLM is a fast, easy-to-use library for LLM inference and serving, known for PagedAttention, continuous batching, and high-throughput execution. and SGLangSGLang is a high-performance serving framework for large language and multimodal models, with an efficient runtime for fast model execution. . VLA RL still lacks an equally mature high-performance inference backend. RLinfe.g.: As a backend of RLInf ’s rollout engine. supports PPO and GRPO for VLA models, but its rollout path primarily relies on model-native PyTorch execution rather than a dedicated inference runtime.

The opportunity is already visible at scale. In a PI0.5 workload running on 8×A100 GPUs with rollout batch size 40 and 41 Predict calls per training step, the steady-state mean spends 57.966 of 109.1 seconds in Predict: 53.1% of the entire step and 65.6% of rollout time. If PhyAI made only Predict 2× faster while every other phase remained unchanged, the projected step time would fall to about 80.1 seconds, a 26.6% reduction.

We have not yet validated this projection in a representative integrated VLA RL workload because of current hardware and engineering constraints. Connecting PhyAI to large-scale rollout pipelines is therefore an important next step toward filling the cloud-side VLA inference gap.

MaaS For Factory

In an edge-only deployment, every robot carries a full inference accelerator. Factory MaaS instead shares an eight-GPU cluster across the fleet while motor control and safety logic remain on each robot. For a 100-robot line, a practical planning point is to serve up to 32 requests in each inference batch: the robots do not all submit work at the same instant, so the same cluster can process successive groups across the fleet. The real-time constraint is network transit plus full-batch execution time; amortized ms/sample describes throughput, not request latency.

The corresponding hardware-price comparison is:Public US pricing as of July 18, 2026. NVIDIA lists the Jetson AGX Orin Developer Kit at US$3,499, the Jetson Thor Developer Kit at US$5,499, and the NVIDIA RTX PRO 6000 Blackwell Workstation Edition at US$13,250. NVIDIA does not publish an H200 MSRP; the H200 figure uses a reported US$30k–40k per-GPU market range.

At this scale, centralized inference puts the accelerator purchase at US$106k–320k, compared with US$349.9k–549.9k for one edge kit per robot. It also concentrates future upgrades in eight shared GPUs instead of 100 separate devices. Production capacity still depends on the target model, request arrival pattern, network, and action deadline. Server chassis, networking, power, spares, integration, and discounts are excluded.

Security of Physical AI Inference Framework

Unlike LLM-based agents, a physical AI agent operates in a world that offers no undo button. Every action an LLM agent takes, from file reads and writes to code execution and environment configuration, is mediated by layers of operating system constraints and sandboxing. Overlayfs provides copy-on-write isolation. Git offers version rollback. Containers enforce namespace boundaries. Together, these mechanisms form a computing environment that is reversible, auditable, and isolatable. Even if the model makes an egregious error, the worst outcome is a git revert or the destruction of a throwaway container. Unless you’ve confidently wired your LLM into a quantitative trading system or handed it your bank account, it is very difficult for an LLM to cause you serious harm.

The physical world offers none of these safeguards. A robotic arm that applies the wrong torque during a grasp will not un-shatter the glass by rolling back a log file. An autonomous vehicle that makes an erroneous lane-change decision at highway speed cannot undo the collision through a second pass of inference. Physical interaction is, by its nature, irreversible. Once an action is applied to matter, it becomes a lossy inscription on reality.

This is why the reliability of a physical AI reasoning framework matters far more than that of an LLM. When an LLM receives a correction signal mid-generation, it can halt output immediately and re-reason. But a robotic arm mid-swing or a vehicle mid-braking carries physical trajectories with continuity and inertia. Once a decision enters the execution phase, instantaneous rollback is no longer an option. This means interruptibility must be a core design constraint of any physical reasoning framework. A safe physical reasoning framework must be capable of gracefully aborting the current action sequence at any timestep and transitioning the system into a known safe state, regardless of where in the control cycle the interruption occurs.

The irreversibility of physical interaction also shifts the center of gravity for safety evaluation from statistical averages to worst-case guarantees. A control policy that fails in one out of a thousand execution cycles is perfectly tolerable in software; if those failures involve human safety, they are entirely unacceptable. A physical reasoning framework must possess the ability to characterize the boundaries of worst-case behavior, whether through reachability analysis or certified robustness guarantees. This entire class of tooling has no counterpart in the LLM technology stack.

Ultimately, the leap from LLM to physical AI is not merely a shift in modality, from text to sensors and actuators. It is a fundamental change in the safety posture of the system: moving from a world that is observable, isolatable, and tolerant of latency, to one that is irreversible, partially observable, and devoid of isolation guarantees. The reasoning framework must be architected to internalize this transformation, embedding both correctness of inference and interruptibility as first-class capabilities at the system level.

PhyAI’s Responsibility and Our Envision

Human-centricity is at the heart of PhyAI. PhyAI was created to lower the cost of reasoning for physical AI and to expand the imagination of algorithm researchers, enabling them to build larger and more ambitious models. We hope PhyAI can accelerate the democratization of physical AI, advance the progress of physical AI development, and contribute to the broader journey toward AGI. Given the unique nature of physical AI models, safety has always been a core concern for PhyAI. We do not want PhyAI to be used in ways that are harmful to people, and we will continue to strengthen the robustness of the framework to deliver a safer and more reliable inference experience for everyone.

What’s Next?

1. Agent-assisted kernel scaling. We will use an agent kernel-programming framework similar to the one in arXiv:2603.24517v1 presents an agent-based framework for writing and optimizing GPU kernels. to generate, benchmark, and iteratively improve kernels for each hardware and model combination. This lets the optimization loop keep climbing with the target workload instead of relying only on hand-written kernels.
2. Continuous numerical alignment. We will keep aligning supported models with their official implementations through layer-by-layer checks and repeatable validation, reducing numerical drift as new models and kernels are added.
3. Stronger robustness and safety. We will continue improving failure handling, interruption and recovery behavior, validation coverage, and the safeguards needed for reliable Physical AI inference in real environments.

Contributions

1. We thank the students and faculty at Beijing University of Posts and Telecommunications, Peking University, and Tsinghua University.
2. We are grateful to the open-source LLM serving frameworks SGLangSGLang is a high-performance serving framework for large language and multimodal models, with an efficient runtime for fast model execution. and vLLMvLLM is a fast, easy-to-use library for LLM inference and serving, known for PagedAttention, continuous batching, and high-throughput execution. for inspiring our work.
3. We thank outstanding kernel libraries such as FlashInferFlashInfer is a high-performance GPU kernel library and generator that provides optimized attention, GEMM, and MoE operations for inference. , HummingHumming is a lightweight, flexible JIT-compiled GEMM kernel library designed for high-performance quantized inference. , and FLAFlash Linear Attention (FLA) provides hardware-efficient kernels and training-ready layers for linear attention, sparse attention, state-space models, and hybrid LLM architectures. , whose work helps PhyAI run blazingly fast.

Limitations

1. Compared with leading VLA inference frameworks such as FlashRT, PhyAI still trails on devices such as NVIDIA Thor, primarily because of differences in GEMM kernel performance. To close this gap, we are building an AVO-based kernel optimization pipeline and exploring Mirage and TIRx to develop megakernels that reduce the overhead of the DiT portion of the model.
2. For some models, PhyAI has not yet completed layer-by-layer numerical alignment against the official implementations. As a result, these models may exhibit small numerical differences from their reference implementations. We plan to use agents to build an automated numerical validation and alignment pipeline for this labor-intensive and repetitive process.

Sponsors

We are deeply grateful to MingTi and ModelBest for their support. We also warmly welcome more companies to join us by providing a broader range of machines for benchmarking and development, as well as better AI API services for our developers. Together, we hope to build a thriving Physical AI ecosystem.