Real-Time GPU-Accelerated Gaussian Splatting with NVIDIA DesignWorks Sample vk_gaussian_splatting

Gaussian splatting is a novel approach to rendering complex 3D scenes by representing them as a collection of anisotropic Gaussians in 3D space. This technique enables real-time rendering of photorealistic scenes learned from small sets of images, making it ideal for applications in gaming, virtual reality, and real-time professional visualization.

vk_gaussian_splatting is a new Vulkan-based sample that demonstrates real-time Gaussian splatting, a cutting-edge volume rendering technique that enables highly efficient representations of radiance fields. It is the latest addition to the NVIDIA DesignWorks Samples.

The NVIDIA DevTech team envisions this new sample project as a testbed to explore and compare various approaches to real-time visualization of 3D Gaussian splatting (3DGS). By evaluating various techniques and optimizations, the team aims to provide valuable insights into performance, quality, and implementation trade-offs when using the Vulkan API. 

The initial implementation is based on rasterization and demonstrates two approaches for rendering splats, one leveraging mesh shaders and another using vertex shaders. 

Figure 1. Comparison of sorting methods, illustrated for the mesh shader pipeline

Because Gaussian splats require back-to-front sorting for correct alpha compositing, two alternative sorting methods are provided: 

A GPU-based Radix Sort implemented in a compute pipeline A CPU-based asynchronous sorting strategy that uses the multithreaded sort function from the C++ STL

The sample allows you to explore and experiment with multiple aspects of this rendering technique, including:

Several visualization modes to inspect the different aspects of Gaussian splats (spherical harmonics, splats, point density, and more) A complete benchmarking system is available and enables profiling in real time More details about both the RAM and VRAM memory consumption, to understand the stream of data to render GPU timings for each stage of the different techniques investigated, to have an understanding of the workload and potential bottlenecks Graphical reports made generated with all of these numbers Figure 3. Example of report comparing the rendering performance with different data storage formats for a complete dataset

This sample provides a starting point for developers looking to experiment with Gaussian splatting rendering techniques and Vulkan-based optimizations.

To start exploring real-time rendering of neural radiance fields, check out the nvpro-samples/vk_gaussian_splatting GitHub repo.