Robot Learning · Vision-Language-Action

Tingting Du

B.S. student in Computer Science and Mathematics at the University of Wisconsin–Madison. I build robot learning systems that connect language, perception, and physical action — currently on trimanual manipulation in the RT² Lab.

RT² Lab, UW–Madison
Trimanual Manipulation
Madison, WI

Curriculum Vitae Google Scholar GitHub Email

Demos

A look at what the systems I work on actually do.

Trimanual Manipulation — RT² Lab, UW–Madison

Ongoing, with Prof. Mike Hagenow · Robot Teaching & Teaming Lab

Scene Wrist · 1 Wrist · 2 Wrist · 3

Episode 0007

Three robot arms collaborate with a human to assemble a pillow block.

XGenAct — Geometry-Enhanced World Action Models

First author · arXiv 2026 · with the CASE Lab, University of Maryland

XGenAct method: RGB, depth, role segmentation, normals, and actions encoded as RGB videos; a shared DiT trained with one loss; inference that generates future RGB and action images.

World action models predict the next pixels and the next actions, but that is not the same as understanding space. XGenAct turns RGB, metric depth, surface normals, functional role segmentation, and robot actions into RGB videos through fixed codecs, then trains one video diffusion transformer with one objective by sampling those tasks. No modality-specific heads. On held-out RLBench tasks this raises closed-loop success from 26% (strongest evaluated baseline) to 52%.

ROCKET — Spatially-Aware Vision-Language-Action Models

NeurIPS 2026 · with the CASE Lab, University of Maryland

ROCKET architecture: a VLA model's residual stream is aligned layer-by-layer with a frozen 3D foundation model through a shared projector, with Matryoshka-style activation, producing end-effector deltas.

VLA models read pixels well but reason about space poorly. ROCKET aligns the VLA's residual stream layer by layer with a frozen 3D foundation model through a shared projector, using Matryoshka-style activation so depth cues at different ranges land in different parts of the representation. Spatial grounding is distilled into the policy during training, so no depth sensor is needed at inference.

About

I am an undergraduate at the University of Wisconsin–Madison, studying Computer Science and Mathematics. Before Madison I was a visiting student in Computer Science at UC Berkeley, and I began my studies in Linguistics at Ningbo University.

I currently work with Prof. Mike Hagenow in the Robot Teaching and Teaming (RT²) Lab at UW–Madison on trimanual manipulation. Previously I worked with Prof. Ang Li in the CASE Lab at the University of Maryland on world action models and Vision-Language-Action models, with Prof. Meng Jiang at the University of Notre Dame on student modeling and question generation, and with Prof. Alane Suhr at Berkeley AI Research on situated language understanding.

Research Focus

World Action & VLA Models

Training one model to predict RGB, actions, and geometry over time, so policies pick up 3D structure instead of treating pixels as the whole story.

Multi-Arm Manipulation

Coordinating several arms and viewpoints on long-horizon, contact-rich tasks, and learning those behaviours from human demonstration.

Grounded Language & Reasoning

How people and models use language in situated, collaborative settings — and how memory and simulation improve agent reasoning.

Updates

2026.10 XGenAct, my first-author paper on geometry-enhanced world action models, is on arXiv.New
2026.09 ROCKET accepted to NeurIPS 2026.
2026.08 Joined the RT² Lab at UW–Madison with Prof. Mike Hagenow, working on trimanual manipulation.
2026.04 Our Vision-Language-Action survey was accepted to TMLR.
2025.12 Collaborative situated game paper released on arXiv.
2025.01 QG-SMS accepted to ACL 2025.
2024.06 Completed a research internship at Berkeley AI Research.

Publications

arXiv 2026 XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation
T. Du, Z. Wang, G. Sun, A. Li
arXiv:2610.03516
arXiv DOI Details
NeurIPS 2026 ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models
G. Sun, T. Du, K. Feng, C. Luo, X. Ding, Z. Shen, Z. Wang, Y. He, A. Li
Advances in Neural Information Processing Systems
arXiv Code DOI Details
TMLR 2026 Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
Z. Wang, B. Wang, H. Zhang, T. Du, T. Chen, G. Sun, Y. He, Z. Shen, W. Ye, A. Li
Transactions on Machine Learning Research
arXiv DOI Details
ACL 2025 QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulation
B. Nguyen, T. Du, M. Yu, L. Angrave, M. Jiang
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics
Details
arXiv 2025 Characterizing Language Use in a Collaborative Situated Game
N. Tomlin, N. Zhou, E. Fleisig, L. Chen, T. Wright, L. Vinh, L.X. Ma, S. Eisape, T. Du, T. Zhang, A. Koller, A. Suhr
arXiv:2512.03381
arXiv DOI Details
ICML 2025 Workshop Agent KB: A Hierarchical Memory Framework for Cross-Domain Agentic Problem Solving
X. Tang, T. Qin, T. Peng, Z. Zhou, D. Shao, T. Du, X. Wei, H. Zhu, G. Zhang, et al.
Workshop on Collaborative and Federated Agentic Workflows
Details

Education

Jan 2025 – May 2027 University of Wisconsin–Madison B.S. in Computer Science and Mathematics (expected)
2023 – 2024 University of California, Berkeley Visiting Student in Computer Science
2021 – 2023 Ningbo University Undergraduate studies in Linguistics