Sketch2Colab: Sketch-Conditioned Multi-Human Animation via Controllable Flow Distillation

View PDF HTML (experimental)

Abstract:We present Sketch2Colab, which turns storyboard-style 2D sketches into coherent, object-aware 3D multi-human motion with fine-grained control over agents, joints, timing, and contacts. Diffusion-based motion generators offer strong realism but often rely on costly guidance for multi-entity control and degrade under strong conditioning. Sketch2Colab instead learns a sketch-conditioned diffusion prior and distills it into a rectified-flow student in latent space for fast, stable sampling. To make motion follow storyboards closely, we guide the student with differentiable objectives that enforce keyframes, paths, contacts, and physical consistency. Collaborative motion naturally involves discrete changes in interaction, such as converging, forming contact, cooperative transport, or disengaging, and a continuous flow alone struggles to sequence these shifts cleanly. We address this with a lightweight continuous-time Markov chain (CTMC) planner that tracks the active interaction regime and modulates the flow to produce clearer, synchronized coordination in human-object-human motion. Experiments on CORE4D and InterHuman show that Sketch2Colab outperforms baselines in constraint adherence and perceptual quality while sampling substantially faster than diffusion-only alternatives.

Comments: Accepted to CVPR 2026 Main Conference (11 pages, 8 figures)

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Graphics (cs.GR); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)

Cite as: arXiv:2603.02190 [cs.CV]

(or arXiv:2603.02190v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.02190

arXiv-issued DOI via DataCite

Submission history

From: Divyanshu Daiya [view email] [v1] Mon, 2 Mar 2026 18:52:51 UTC (35,854 KB) [v2] Sun, 29 Mar 2026 03:13:10 UTC (13,008 KB)