Part-wise Translation of Upper-body Gestures via Real-time Target Estimation for Virtual Avatars in Dissimilar Environments

Jiho Kang, Jihun Shin, Kwang Bin Lee, Dong Hwan Shin, Yuna Choi, Sung-Hee Lee
KAIST
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2026

Presentation Video

User Evaluation Video

Supplemental Animation Results

Abstract

XR telepresence between two dissimilar spaces A and B

In extended reality telepresence, virtual avatars must accurately convey users' intent across dissimilar environments. We propose a novel neural network framework for real-time, part-wise translation of upper-body gestures to virtual avatars in heterogeneous environments. Our approach advances beyond prior methods in three key aspects: (1) real-time estimation of users' interaction targets; (2) unsupervised learning of target-agnostic motion features that better preserve fine-grained motion details; and (3) part-wise translation in which the head and individual hands can independently reference different targets simultaneously. Together, these advances allow users to select arbitrary interaction targets from multiple candidates and reference different targets concurrently with different body parts, while enabling avatars to better preserve users' motion details. These capabilities are realized through two novel networks: a Target Estimation Network that interprets sparse user tracking signals to infer target positions for the head and hands, and a Body-Part Gesture Translation Network that autoregressively generates avatar poses conditioned on these targets. Despite being trained on data from a single individual, our framework translates natural upper-body gestures across diverse users and avatars. We validate the effectiveness of our approach through comparisons with state-of-the-art methods, qualitative animation results, and a user evaluation in telepresence scenarios.

Method Overview

Overview of the part-wise gesture translation framework
The system consists of two networks operating in sequence. First, the Target Estimation Network (TENet) estimates per-body-part target positions by computing posterior densities over all candidate target combinations from user behavioral features. Next, the Body-Part Gesture Translation Network (BP-GTNet) generates avatar motion through a dual end-effector representation: the encoder extracts target-agnostic motion features from the user's end-effectors, while the decoder transforms them into the avatar's end-effectors conditioned on the estimated targets. The generated end-effectors drive an IK solver to produce the final avatar pose, with autoregressive feedback ensuring temporal coherence.

Real-time Target Estimation and Part-wise Gesture Translation: Diverse Users and Avatars

Test Subject 1 (161 cm, female, right-handed) → Avatar (175 cm)

Test Subject 2 (172 cm, male, right-handed) → Avatar (172 cm)

Test Subject 3 (173 cm, male, left-handed) → Avatar (173 cm)

Test Subject 4 (174 cm, female, left-handed) → Avatar (178 cm)

Test Subject 5 (180 cm, male, right-handed) → Robot Avatar (189 cm)

Test Subject 6 (180 cm, male, right-handed) → Avatar (209 cm)

Test Subject 6 (180 cm, male, right-handed) → Smaller Avatar (148 cm)

Test Subject 9 (187 cm, male, right-handed) → Cartoon Avatar (195 cm)

Test Subject 11 (173 cm, male, right-handed) → Cartoon Avatar (186 cm)

Upper-body gestures of test subjects with diverse heights, arm lengths, and handedness (left) and the avatar motions translated in real time by our framework (right). Target objects are positioned differently for the user and the avatar, and the predicted targets for the head (red), right hand (green), and left hand (blue) are shown; smaller spheres indicate higher estimation consistency, whereas larger ones reflect increased uncertainty during target transitions. Although trained on the motion data of a single 170 cm subject, our framework generalizes to diverse users and avatars.

Comparison of TENet with Alternative Network Architectures

This video contains audio. Unmute the player to listen.

Target estimation results of TENet (ours), Baseline (Real NVP with an MLP embedding network), Diffusion (a residual MLP-based diffusion regression model), and TENet-Indep (separate flow networks for each body part), compared with the ground truth (GT). Joint estimation across body parts lets TENet respond to target transitions most rapidly.

Comparison of BP-GTNet (Ours) with UGNet (Baseline)

Part-wise Translation Audio

This video contains audio. Unmute the player to listen.

UGNet copies the non-dominant hand from the user's head coordinates directly to the avatar's head coordinates, preventing part-wise translation, whereas BP-GTNet translates the head and both hands independently toward their own estimated targets.

Motion Detail and Stability Audio

This video contains audio. Unmute the player to listen.

BP-GTNet generates more stable and detailed motion than UGNet.

Ablation Study of BP-GTNet Architecture Design

Motion Discretization (MD) Audio

This video contains audio. Unmute the player to listen.

Motion discretization helps produce natural and stable motion. Without it, the network tends to generate unstable movements.

Part-wise Auxiliary Decoder (PAD) Audio

This video contains audio. Unmute the player to listen.

Without the part-wise auxiliary decoder, the model fails to learn part-wise control, and the avatar's right hand does not gesture toward its estimated target when it differs from the head's estimated target.

Dual End-effector Representation (EE-Dual) Audio

This video contains audio. Unmute the player to listen.

Without the dual end-effector representation, the model fails to learn target-agnostic motion features, resulting in hands directed toward empty space.

XR Application Demo

#1 Collaboration Audio

This video contains audio. Unmute the player to listen.

Two distant users conduct a chemistry lesson, where three virtual atoms are augmented at different positions and scales in each environment. Users explain molecular structures while gesturing toward different atoms simultaneously.

#2 Teleoperation Audio

This video contains audio. Unmute the player to listen.

The user's left and right hand targets are estimated in real time, and corresponding bimanual pick-and-place motions are generated for a humanoid robot avatar despite substantial differences in body size and object placement.

Limitation: Target Distributions with Larger Dispersion

This video contains audio. Unmute the player to listen.

Our training data covers a 180° frontal range, and performance degrades outside it: target estimation worsens as the user's target is placed rearward, and the translated gestures deteriorate as the avatar's target moves toward its rear, even when the estimated target itself is accurate.