V2E Face: Emotionally Disentangled Talking Head Generation with Vector Quantization and Attention Fusion

IEEE MultiMedia (2026) · DOI: 10.1109/MMUL.2026.3724384

Most talking-head generators tangle identity, pose, and emotion together in one latent space, then try to untangle them with extra loss terms. V2E Face takes a different approach: disentanglement is built into the architecture itself.

Core Idea#

1. Multi-Attribute Vector Quantization (MAVQ) — separate, supervised VQ codebooks for yaw, pitch, roll, and facial landmarks, isolating geometry from expression from the very first bottleneck.
2. Multi-head Attention Fusion (MAF) — fuses the three pose codes into one physically coherent pose vector instead of naive concatenation.
3. Hierarchical attention synthesis — builds a stable geometric "canvas" first, then applies emotion as a distinct layer on top.

Everything (pose, landmarks, lip motion) is generated purely from audio + a reference image, no external pose input needed.

Results#

Evaluated on CREMA-D and MEAD against MakeItTalk, EMMN, EAT, and ED-Talk:

• Best PSNR/SSIM/lip-sync on both datasets; emotion accuracy close to the top baseline.
• Human study (17 participants): ranked best among all generative methods on visual quality, motion naturalness, emotion expressiveness, and overall preference (p < 0.01 vs. baselines).
• Ablation: removing hierarchical synthesis causes a 14-point crash in emotion accuracy — confirming structured fusion, not just parameter count, drives the gain.

Ablation study (MEAD):

• Lightweight & fast: 15.6M parameters, 2.4 ms/frame on an RTX 4090, 24 FPS end-to-end.

Human evaluation (blind ranking, 17 participants, 1 = best, 4 = worst):

Known limitation#

Fear and Happy share an open-mouth articulation, so the model sometimes confuses them — a geometry/articulation issue rather than a disentanglement failure.

Ethics note#

As with any photorealistic talking-head generator: risk of non-consensual identity impersonation. The authors recommend pairing this kind of framework with forgery-detection and watermarking.

BibTeX#

@article{vo2026v2eface,
  author  = {Vo, Hoang-Son and Yang, Hyung-Jeong and Kim, Soo-Hyung},
  title   = {Emotionally Disentangled Talking Head Generation with Vector Quantization and Attention Fusion},
  journal = {IEEE MultiMedia},
  year    = {2026},
  doi     = {10.1109/MMUL.2026.3724384}
}