What is Gaussian splatting

Gaussian splatting is a 3D rendering technique introduced in the 2023 paper “3D Gaussian Splatting for Real-Time Radiance Field Rendering”, this method has quickly gained attention for its ability to deliver great visual representations.

“The main reason why Gaussian Splatting is a game changer is its ease in generating and visualizing these 3D models. A simple camera can film an object with low processing cost, resulting in a video that can generate 3D models that can be viewed in any browser. This ease of use helps with the adoption and spread of the technology.”

— Felipe Miranda Tech Lead Front-End at Vidya Technology

Fundamental concepts

1. Scene representation with 3D Gaussians

At the heart of Gaussian splatting is the idea of modeling a 3D scene using a dense cloud of anisotropic 3D Gaussian primitives, or “splats.” Each Gaussian serves as a small, ellipsoidal component of the scene and carries several attributes:

• Position: Defines its location in 3D space.
• Covariance matrix: Encodes shape, size, and orientation.
• Opacity: Determines its transparency.
• Spherical hrmonics: Encodes color in a view-dependent way for realistic shading.

These Gaussians collectively form an explicit and highly flexible representation of the scene geometry and appearance.

2. Differentiable splatting-based rendering

The rendering process involves projecting these 3D Gaussians onto a 2D image plane, where each splat contributes to the final pixel values. This is done using alpha blending techniques, similar to how particles are rendered in traditional graphics pipelines.

Importantly, the rendering is differentiable, enabling end-to-end optimization. This means the appearance of each Gaussian can be fine-tuned based on discrepancies between rendered and real images—crucial for learning accurate scene representations.

3. Training and optimization

The pipeline begins with input from multiple calibrated images and their associated camera poses (often obtained via structure-from-motion tools like COLMAP). A sparse point cloud is then converted into an initial set of Gaussians.

Through iterative training:

• Gaussian parameters (position, shape, color, opacity) are optimized via gradient descent.
• The system adapts dynamically, adding or removing Gaussians as needed. Large splats may be split to capture fine detail, while nearly transparent or redundant ones are pruned.

Advantages over traditional methods

Real-time performance

Unlike methods like Neural Radiance Fields (NeRF), which rely on computationally expensive ray marching, Gaussian splatting leverages GPU rasterization, achieving 100+ FPS at full HD resolution.

High visual fidelity

Thanks to its explicit representation and adaptive optimization, Gaussian splatting excels at preserving intricate details like hair, leaves, and small structures that often blur in NeRF-based reconstructions.

Efficient and scalable

The use of spatial acceleration structures and GPU-friendly tile-based rendering makes the technique highly efficient for large-scale scenes.

How it compares to NeRFs

Applications

• AR/VR: Enables immersive real-time 3D environments.
• Photogrammetry: Accurately reconstructs scenes from photos or drone imagery.
• Film & Game Development: Ideal for rapid scene previews and prototyping.

Challenges and limitations

Despite its promise, Gaussian splatting has some caveats:

• Storage overhead: Large scenes can consume hundreds of megabytes to several gigabytes.
• Visual artifacts: Can suffer from blurriness under fast camera motion or sparse input views.
• Static scene restriction: Currently not designed for dynamic objects or temporal changes.

Tools and resources

Open-source libraries

• GraphDECO (Inria) – Official implementation.
• SplatStudio – CUDA-accelerated tooling for integration with NeRF pipelines.

Visualization tools

• SIBR Viewer – A versatile, cross-platform renderer for splatted scenes.