Real-time Retargeting of Deictic Motion to Virtual Avatars for Augmented Reality Telepresence

KAIST
IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2023

Abstract

AR telepresence between two dissimilar spaces A and B

Avatar-mediated augmented reality telepresence aims to enable distant users to collaborate remotely through avatars. When two spaces involved in telepresence are dissimilar, with different object sizes and arrangements, the avatar movement must be adjusted to convey the user's intention rather than directly following their motion, which poses a significant challenge. In this paper, we propose a novel neural network-based framework for real-time retargeting of users' deictic motions (pointing at and touching objects) to virtual avatars in dissimilar environments. Our framework translates the user's deictic motion, acquired from a sparse set of tracking signals, to the virtual avatar's deictic motion for a corresponding remote object in real-time. One of the main features of our framework is that a single trained network can generate natural deictic motions for various sizes of users. To this end, our network includes two sub-networks: AngleNet and MotionNet. AngleNet maps the angular state of the user's motion into a latent representation, which is subsequently converted by MotionNet into the avatar's pose, considering the user's scale. We validate the effectiveness of our method in terms of deictic intention preservation and movement naturalness through quantitative comparison with alternative approaches. Additionally, we demonstrate the utility of our approach through several AR telepresence scenarios.

Method Overview

Overview of the deictic motion retargeting framework
Angle features are computed from the user and the avatar with respect to their own targets, and then transformed into latent vectors by AngleNet. A recurrent module maps the sequence of latent vectors to a control vector. With this vector and the user's scale value, MotionNet regresses the avatar's end-effectors (head, right hand, and right index fingertip), from which an IK solver computes the avatar's upper-body pose. Both AngleNet and MotionNet are mixture-of-experts (MoE) networks whose blending weights are determined by a gating network driven by the target position.

Retargeting Results: Pointing

Test subject: 161 cm

Test subject: 161 cm

Test subject: 161 cm

Test subject: 161 cm

Test subject: 172 cm

Test subject: 172 cm

Test subject: 172 cm

Test subject: 179 cm

Test subject: 179 cm

Test subject: 179 cm

Test subject: 179 cm

Deictic motions of test subjects with different heights (left) and the avatar motions retargeted in real time by our framework (right), shown in top (upper) and front (lower) views. The targets are placed at different locations for the user and the avatar.

Retargeting Results: Touching

Test subject: 161 cm

Test subject: 161 cm

Test subject: 161 cm

Test subject: 172 cm

Test subject: 172 cm

Test subject: 172 cm

Test subject: 172 cm

Test subject: 179 cm

Test subject: 179 cm

Test subject: 179 cm

Although the network is trained on motions of a single 170 cm subject, it retargets the touching motions of users with different heights to their avatars.

Robustness on Varying Target Positions

The avatar stably points at its own target while the target position is randomly adjusted.

Comparisons

Model Structure: MoE vs. MLP

With the same number of parameters, the MLP model tends to mimic the user's motion regardless of the avatar's own target position, whereas our MoE model robustly retargets the deictic motion toward the avatar's target.

Input Representation: Angle vs. Position-Rotation

Our angle-based input representation is robust to variations in deictic motion style and body size, while the position-rotation representation is sensitive to changes in hand orientation.

Ours vs. Inverse Kinematics (IK)

Trained on real human movements, our learning-based method generates more natural head and hand motions than a consumer-grade IK solver.

AR Telepresence Scenarios

Education

Education scenario in AR telepresence
User X teaches user Y about three planets in the solar system: Earth, Venus, and Jupiter. Each user virtually augments the planets at their preferred locations within their own space. As user X sequentially points at the planets in space A (left), avatar X′ points at the corresponding planets in the same order in space B (right). Top: egocentric views. Bottom: room perspective views.

Commerce

Commerce scenario in AR telepresence
User X seeks user Y's opinion on a piece of clothing; user Y has the actual clothing item in space B (right), while user X has its virtual 3D replica in space A (left). When user Y touches a specific part of the cloth (the zipper), avatar Y′ touches the exact same part of the replica, so that the user's focus is accurately relayed. Top: egocentric views. Bottom: room perspective views.

BibTeX

@inproceedings{kang2023real,
  title={Real-time Retargeting of Deictic Motion to Virtual Avatars for Augmented Reality Telepresence},
  author={Kang, Jiho and Yang, Dongseok and Kim, Taehei and Lee, Yewon and Lee, Sung-Hee},
  booktitle={2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
  pages={885--893},
  year={2023},
  organization={IEEE},
  doi={10.1109/ISMAR59233.2023.00104}
}