SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting

TL;DR SMG models Gaussians motion as semantic motion, allowing reliable, semantically coherent nodes to guide uncertain motion for robust monocular dynamic Gaussian splatting.

Abstract

We study dynamic Gaussian Splatting from monocular videos. While recent advancements in dynamic Gaussian splatting offer a promising foundation for modeling dynamic scenes, they often overfit to the training views and fail under occlusion or complex scene motion due to the lack of reliable regularization signals in under-constrained regions. We propose Semantic Motion Graph (SMG), a novel approach models the Gaussian motion as the low-rank semantic motion. Our key insight is that the real-world scene motion is often structured by semantic coherence: regions that are spatially close and semantically related tend to exhibit consistent dynamics. To leverage this prior, we construct SMG to model structured motion of the scene. The Gaussian motion is driven by the motion of SMG nodes. We further observe that the uncertainty of Gaussian motion arises from both unreliable off-the-shelf priors and weakly constrained regions during optimization. SMG addresses this by using reliable graph nodes to guide the motion of nearby unreliable nodes. To evaluate dynamic Gaussian splatting under challenging real-world scenarios, we introduce a new multiview dataset collected under an ego-exo setup. Extensive experiments demonstrate that SMG achieves state-of-the-art performance on monocular dynamic Gaussian splatting across challenging real-world benchmarks.

Motivation

Problem: Monocular dynamic Gaussian splatting is challenging. Limited observations and noisy priors leave many regions weakly constrained during optimization. Purely geometric motion propagation causes motion leakage, drifting geometry, and poor novel views.

Solution: Motion coherence follows semantic coherence. We use Semantic Motion Graph (SMG) restricts motion propagation to local semantic groups and lets reliable nodes guide unreliable ones.

Methodology

Given a monocular video with 2D tracks, depth, and dynamic/static masks predicted by off-the-shelf models, our method (1) first initializes the Semantic Motion Graph (SMG) with the 3D lifting field within nearby semantic groups, which ensures that the motion is only propagated locally. (2) Building on the SMG, we propose confidence-aware as-rigid-as-possible (C-ARAP) to guide uncertain nodes with reliable neighbors to preserve the SMG topology. (3) We further design local rigid motion (LRM) to constrain local velocities to rigid motion to prevent structural drift. C-ARAP and LRM are incorporated into the SMG to ensure that the semantic motion is propagated correctly. They handle uncertainty during Gaussian optimization caused by limited observations and unreliable priors. Finally, our proposed SMG improves the robustness of novel-view synthesis and yields dynamic 3D scenes enriched with consistent semantics and 3D tracks.

Qualitative Comparison on iPhone Dataset

SMG Dataset

To evaluate monocular dynamic Gaussian splatting in challenging real-world settings, we introduce the SMG Dataset. It contains 20 casually captured indoor scenes, each recorded with two calibrated cameras in an ego–exo setup. The dataset follows the DyCheck format and includes predicted depth by Depth Anything 3. We show the examples of novel-view synthesis comparisons among MoSca, OriGS, and SMG on the proposed dataset below.

BibTeX

@inproceedings{smg,
  title     = {SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting},
  author    = {Haozheng Yu, Xinyu Yang, Rundong Luo, Jennifer J. Sun, Bharath Hariharan},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}