InteractAvatar: Modeling Hand-Face Interaction in Photorealistic Avatars with Deformable Gaussians
1 Brown University 2 Meta Reality Labs
* Equal contribution
Abstract
With the rising interest from the community in digital avatars coupled with the importance of expressions and gestures in communication, modeling natural avatar behavior remains an important challenge across many industries such as teleconferencing, gaming, and AR/VR. Human hands are the primary tool for interacting with the environment and essential for realistic human behavior modeling, yet existing 3D hand and head avatar models often overlook the crucial aspect of hand-body interactions, such as between hand and face. We present InteracttAvatar, the first model to faithfully capture the photorealistic appearance of dynamic hand and non-rigid hand-face interactions. Our novel Dynamic Gaussian Hand model, combining template model and 3D Gaussian Splatting as well as a dynamic refinement module, captures pose-dependent change, e.g. the fine wrinkles and complex shadows that occur during articulation. Importantly, our hand-face interaction module models the subtle geometry and appearance dynamics that underlie common gestures. Through experiments of novel view synthesis, self reenactment and cross-identity reenactment, we demonstrate that InteracttAvatar can reconstruct hand and hand-face interactions from monocular or multiview videos with high-fidelity details and be animated with novel poses.
Method
Our method combines mesh-based geometry (FLAME, MANO) with 3D Gaussian Splatting for realistic hand-face interactions. The dynamic hand appearance module refines pose-dependent deformations, wrinkles, and shadows, while the Hand-Face Interaction module enhances contact-aware geometry and shading adjustments. This enables high-fidelity animation with lifelike interactions and appearance changes.