Robotic imitation learning typically requires policies that capture multimodal action distributions, run at real-time control rates, and accommodate multiple sensing modalities. Recent generative approaches—diffusion, flow matching, IMLE—each satisfy a subset of these requirements but not all three.
We present PRISM, a single-pass multisensory policy that meets all three by operationalizing a batch-global variant of RS–IMLE. A key technical observation: the RS–IMLE rejection mechanism, designed for few-shot image synthesis, silently fails at policy-scale batch sizes. We identify why and provide the calibration that restores it.
On real-world loco-manipulation (Unitree GO2 + 7-DoF arm) and tabletop manipulation (UR5), PRISM outperforms diffusion policies by 10–25 pp in success while sustaining 30–50 Hz closed-loop control. On CALVIN (full ABCD, language-conditioned), it improves success by ~31 pp over diffusion and ~24 pp over flow matching while reducing trajectory jerk by 20×–70×.
Language & audio — vision-conditioned manipulation.
Four tabletop tasks executed by PRISM on a fixed-base manipulator. The policy fuses a frozen CLIP text embedding and contact-microphone audio with the visual context stream, attending to color, shape, and spatial descriptors to select the correct object and grasp.
Vision & tactile contact-rich manipulation on a quadruped.
PRISM running on a Unitree GO2 with a 7-DoF D1 arm. Tactile feedback from the gripper is fused with wrist and shoulder RGB at every timestep, enabling robust contact-rich behaviors that pure vision baselines fail to recover.
Whole-body parking for a 14-DoF locomotion + manipulation policy.
Before contact-rich manipulation can begin, the GO2 base must reach a stable, kinematically reachable pose. PRISM’s 14-D action space (gripper + EE pose + base linear & angular velocity) lets a single policy plan the approach end-to-end from RGB observations.
CALVIN benchmark — full ABCD, language-conditioned.
Trained on the full CALVIN ABCD split with language annotations, PRISM improves success by ~31 pp over Diffusion Policy and ~24 pp over Flow Matching, while reducing trajectory jerk by 20×–70×. The reel below shows representative long-horizon language-conditioned tasks executed by a single trained policy.
If you find PRISM useful, please cite.
@inproceedings{anonymous2026prism,
title = {{PRISM}: Bridging Multimodality and Real-Time Control in Multisensory Policies},
author = {Anonymous},
booktitle = {Conference on Robot Learning (CoRL)},
year = {2026},
}