Introduced at SIGGRAPH 2023 was a paper called 3D Gaussian Splatting for Real-Time Radiance Field Rendering that set the graphics world on fire. A technology for capturing the real world in 3D similar to NeRF (Neural Radiance Fields), Gaussian Splats have one major advantage that is especially important to game developers, they can be rendered in real time!
Aras P, one of the earliest developers at Unity Technologies, has a very easy to understand definition of Gaussian Splatting:
I have seen quite many 3rd party explanations of the concept at this point, and some of them, uhh, get a thing or two wrong about it 🙂
- This is not a NeRF (Neural Radiance Field)! There is absolutely nothing “neural” about it.
- It is not somehow “fast, because it uses GPU rasterization hardware”. The official implementation does not use the rasterization pipeline at all; it is 100% done with CUDA. In fact, it is fast because it does not use the fixed function rasterization, as we’ll see below.
Anyway,
Gaussian Splats are, basically, “a bunch of blobs in space”. Instead of representing a 3D scene as polygonal meshes, or voxels, or distance fields, it represents it as (millions of) particles:
- Each particle (“a 3D Gaussian”) has position, rotation and a non-uniform scale in 3D space.
- Each particle also has an opacity, as well as color (actually, not a single color, but rather 3rd order Spherical Harmonics coefficients – meaning “the color” can change depending on the view direction).
- For rendering, the particles are rendered (“splatted”) as 2D Gaussians in screen space, i.e. they are not rendered as scaled elongated spheres, actually! More on this below.
And that’s it.
I highly recommend checking out his technical blog for even more details (and other great articles). Even better, Aras has actually provided a Gaussian Splatting playground for Unity on GitHub.
Do be aware of the following VERY important caveat:
⚠️ Note: this is all a toy, it is not robust, it does not handle errors gracefully, it does not interact or composite well with the “rest of rendering”, it is not fast, etc. etc. Also, do not file bugs or issues just yet; I will most likely just ignore them and do whatever I please. I told you so! ⚠️
Although in my experiences I had no crashes, encountered no major bugs and found rather impressive performance. If you are an Unreal Engine developer there is also an implementation on the Unreal Marketplace, however has a rather large price tag attached. You can also check out Gaussian Splatting in your browser with this site.
You can learn more about Gaussian Splatting and see the Unity example in action in the video below.