Research

a workflow of the RAVEN system showing how a BLV user can increase the text size in a 3D scene through natural language command.

RAVEN: Realtime Accessibility in Virtual ENvironments for Blind and Low-Vision People

CHI 26 Paper - ASSETS 25 Demo Paper 🏅 Best Demo Award - CHI 25 Late Breaking Work - Video

Xinyun Cao, Kexin Phyllis Ju, Chenglin Li, Venkatesh Potluri, Dhruv Jain

In this work, we present RAVEN, a system that responds to query or modification prompts from BLV users to improve the accessibility of a 3D virtual scene at runtime. We evaluated the system with eight BLV people, uncovering key insights into the strengths and shortcomings of generative AI-driven accessibility in virtual 3D environments, pointing to promising results as well as challenges related to system reliability and user trust.

Overview of PreView, a VR route rehearsal system for independent navigation with low vision, part one.

Overview of PreView, a VR route rehearsal system for independent navigation with low vision, part two.

PreView: Rehearsing Real Routes in VR to Support Independent Navigation with Low-Vision

Accepted for UIST 2026 Demo track

Yotam Sechayk^∗, Xinyun Cao^∗, Dhruv Jain^†, Takeo Igarashi^† (∗Equal contribution, †Equal advising)

In this work, PreView, we developed a system that turns real 360-degree route imagery into a VR rehearsal of a route that highlights landmarks, layers an imagery-derived soundscape, and answers questions through an AI agent grounded in the user’s current view. Developed in WebXR and powered by Gemini Live API.

System diagram of the AutoTailor framework.

AutoTailor: Automatic, User-Aligned Capability Selection and Adaptation for Web Agents

(💼 internship project at Microsoft Research - Systems and Networking Group.)

ArXiv Preprint

Xinyun Cao, Adriana Szekeres, Fazle Faisal

In this work, we present AutoTailor, a meta-agentic framework for constructing and maintaining a compact set of trajectory-derived Model Context Protocol (MCP) APIs. Our evaluation with WebArena tasks show that our combined workflow of static filtering and dynamic reselection improves accuracy and latency while sharply reducing token usage and end-to-end cost.

A UI to change volume and frequency of individual sounds in a VR environment.

SoundModVR: Sound Modifications in Virtual Reality to Support People who are Deaf and Hard of Hearing

ASSETS 24 Paper + Demo Paper - UIST 24 Demo Paper - Video

Xinyun Cao, Dhruv Jain

In this paper, we explore the possibilities of modifying sounds in VR to support DHH people. We designed and implemented 18 VR sound modification tools spanning four categories, including prioritizing sounds, modifying sound parameters, providing spatial assistance, and adding additional sounds. We evaluated the tools with DHH people and Unity VR Developers.

Three pictures illustrating the three steps in the user study. A red body is picking up a cube, turning, and handing off a cube to a blue body, with words Selection, Manipulation and Coordination.

Dual Body Bimanual Coordination in Immersive Environments

James Smith, Xinyun Cao, Adolfo Ramirez-Aristizabal, Bjoern Hartmann

DIS 23 Paper - Video

Dual Body Bimanual Coordination is an empirical study where users control and interact with the world through two bodies in virtual reality simultaneously. Users select and manipulate objects to perform a coordinated handoff between two bodies under the control of a single user. We investigate people's performance in doing this task, classify strategies for how they chose to coordinate their hands, report on sense of embodiment during the scenario, and share qualitative observations about user body schema.