Gaussian Splatting Study Notes

Reads

• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
  1. Introduction
     • Plenoxels: Radiance Fields without Neural Networks
     • Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
     • Point-NeRF: Point-based Neural Radiance Fields
     • Datasets
       • Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
       • Deep Blending for Free-Viewpoint Image-Based Rendering
       • Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction
     • Photo tourism: Exploring photo collections in 3D
     • Neural Point-Based Graphics
     • PointBased Neural Rendering with Per-View Optimization
     • ADOP: Approximate Differentiable One-Pixel Point Rendering
     • Pulsar: Efficient Sphere-Based Neural Rendering
  2. Related Work
     • Advances in neural rendering
     • Neural fields in visual computing and beyond
       1. Traditional Scene Reconstruction and Rendering
          • The Lumigraph
          • Light Field Rendering
          • Unstructured Lumigraph Rendering
          • Multi-View Stereo for Community Photo Collections
          • Depth synthesis and local warps for plausible image-based navigation
          • Floating Textures
          • Deep Blending for Free-Viewpoint Image-Based Rendering
          • PointBased Neural Rendering with Per-View Optimization
       2. Neural Rendering and Radiance Fields
          • Deepstereo: Learning to predict new views from the world’s imagery
          • View Synthesis by Appearance Flow
          • Deep Blending for Free-Viewpoint Image-Based Rendering
          • Free View Synthesis
          • Deferred Neural Rendering: Image Synthesis using Neural Textures
          • Soft 3D Reconstruction for View Synthesis
          • Escaping Plato’s Cave: 3D Shape From Adversarial Rendering
          • DeepVoxels: Learning Persistent 3D Feature Embeddings
          • NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
          • TensoRF: Tensorial Radiance Fields
          • FastNeRF: High-Fidelity Neural Rendering at 200FPS
          • Baking Neural Radiance Fields for Real-Time View Synthesis
          • KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
          • Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes
          • Scalable Neural Indoor Scene Rendering
          • PlenOctrees for Real-time Rendering of Neural Radiance Fields
          • Variable Bitrate Neural Fields
          • Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
       3. Point-Based Rendering and Radiance Fields
          • Point-Based Graphics
          • Point Sample Rendering
          • Point-based rendering techniques
          • High-Performance Software Rasterization on GPUs
          • Software Rasterization of 2 Billion Points in Real Time
          • High Quality Surface Splatting on Today’s GPUs
          • Surfels: Surface Elements as Rendering Primitives
          • Object Space EWA Surface Splatting: A Hardware Accelerated Approach to High Quality Point Rendering
          • Surface Splatting
          • SynSin: End-to-end View Synthesis from a Single Image
          • Differentiable Surface Splatting for Point-based Geometry Processing
          • Neural Point-Based Graphics
          • ADOP: Approximate Differentiable One-Pixel Point Rendering
          • Neural Point Catacaustics for Novel-View Synthesis of Reflections
          • PointBased Neural Rendering with Per-View Optimization
          • Pulsar: Efficient Sphere-Based Neural Rendering
          • Differentiable Point-Based Radiance Fields for Efficient View Synthesis
          • Point-NeRF: Point-based Neural Radiance Fields
          • A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation
          • Fast Articulated Motion Tracking using a Sums of Gaussians Body Model
          • VoGE: A Differentiable Volume Renderer using Neural Gaussian Ellipsoids for Analysis-by-Synthesis
          • Mixture of Volumetric Primitives for Efficient Neural Rendering
  3. Overview
     • Input: Images of a static scene (SfM)
       • Structure-from-Motion Revisited
     • Output: Set of 3D Gaussians
       • position (mean)
       • covariance matrix
       • opacity α
       • spherical harmonics (color)
         • Plenoxels: Radiance Fields without Neural Networks
         • Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
     • Algorithm: Sequentially optimize the 3D Gaussian parameters + adaptively control the Gaussian density
     • Rasterization:
       • Tile-based (mobile-style)
       • α-blending
       • sorting
  4. Differentiable 3D Gaussian Splatting
     • Goal: optimize a scene representation from a sparse set of SfMs
     • Solution:
       • 3D Gaussians
         • Differentiable
         • Can be easily projected to 2D splats
           • Allows fast α-blending
         • prev methods assume each point is a small planar circle with a normal
           • normals are noisy (sparsity of SfM points)
           • PointBased Neural Rendering with Per-View Optimization
           • Differentiable Surface Splatting for Point-based Geometry Processing
         • Defined by a full 3D convariance matrix Σ in world space centered at point (mean)
           \(G(x) = e^{-{\frac{1}{2}}\left(x \right)^{T}\sum^{-1}{\left(x\right)}}\)
           • Surface Splatting
         • Project 3D Gaussians to 2D
           \(\Sigma = RSS^{T}R^{T}\)
           • S: scaling matrix on the ellipsoid -> vec3 s
           • R: rotation matrix on the ellipsoid -> quaternion q
           • EWA Volume Splatting
  5. Optimization with Adaptive Density Control of 3D Gaussians
     • p: position
     • α: opacity
     • Σ: covariance
     • c: color (SH coefficient)
       1. Optimization
          • needs to create/destroy/move geometry to better represent the scene’s geometry
          • stochastic gradient descent + custom CUDA kernels
            • Plenoxels: Radiance Fields without Neural Networks
            • Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
          • sigmoid activation function α
            • constrain output into [0, 1) range
            • smooth gradient
          • exponential activation function
            • used for the scale of the covariance
          • initial covariance = isotropic Gaussian with axes equal to the mean of the distance to the closest three points
          • standard exponential decay scheduling technique for positions
            • Plenoxels: Radiance Fields without Neural Networks
          • loss function = L₁ + D-SSIM term
            \(L = \left(1 - \lambda \right)L_{1} + \lambda L_{\textrm{D-SSIM}}\)
            • used λ = 0.2
       2. Adaptive Control of Gaussians
          • densify every 100 iterations
          • remove any Gaussians such that η_&alpha > α
          • areas with over/underfitting Gaussians have large view-space positional gradients
            • densify gaussians with τ_pos > average magnitude of view-space position gradients
              • τ = 0.0002
            • underfitting: clone the gradient and move it in the direction of the positional gradient
            • overfitting: areas with high variance
              • split into two Gaussians
              • divide by the factor of Φ = 1.6 (heuristic value)
              • position is sampled from the original(big) Gaussian’s PDF
            • set α value to zero every N = 3000 iterations

Concepts to Study

• SfM
• Point-based Rendering
• NeRF
• Spherical Harmonics
• Stochastic Gradient Descent (more of a review)
  • Activation functions
  • Loss functions