Cube Map Reflections - Part 3

I originally planned on making only a single post on cube map reflections. Then I had so many images that I decided to split it into two posts. Surely, two posts was enough, right? Well, I worked on reflections some more, fixed some problems, added new features, and now I have even more images to show. So here goes Part 3.

For this post, I'm going to list the different problems I had run into and how I solved them. These are mostly the "future work" items from the previous post. I think I have everything more or less worked out now, except for a few minor issues.

Dirty/Imperfect Reflections

This is actually related to planar reflections, not cube map reflections, but the topic is close enough. Someone commented that the reflections in the floors look too clean and perfect. Real floors are scratched, chipped, and dirty. I put a bit of effort into adding support for specular maps into the reflection rendering pipeline so that I could have more realistic surfaces.

My first screenshot shows the museum scene with a scratched floor, using a scratch map found here as a specular map texture. This gives the impression of a floor that has had many large objects pushed across it, damaging the smooth wax coating.

Scratched and scuffed reflective marble floor using a specular map in the museum scene.

Next, I changed the floor of the Crytek Sponza Atrium scene to a dirty and chipped metal material. I tweaked the parameters until I found something I liked.

Dirty, chipped, reflective metal plate floor in the Sponza scene.

Large Numbers of Reflective Objects + Mipmaps

This section involves two different topics. In the previous post, I commented that 3DWorld should scale to over 100 reflective cube map objects, and possibly up to 180. In reality, the max is around 120. I'm not sure why it's lower than I thought. Maybe some memory is needed by MS Windows, or maybe there is allocation overhead or fragmentation. After that (120 spheres) it starts running out of GPU memory and the frame rate drops from ~100 FPS to ~2 FPS. The time to generate reflections increases from a few seconds to a few minutes, and it's not even interactive to view any more.

When there isn't enough GPU memory available, 3DWorld doesn't crash. Instead, the graphics driver starts allocating from system memory (CPU side RAM). I have 16GB of memory on this machine, so it's not going to run out any time soon. Unfortunately, reading from system memory is very slow compared to reading from GPU memory, since the data needs to travel across the motherboard, etc. This really hurts performance! I've seen this effect on both my older machine with an ATI card an on my new machine with an Nvidia card.

So, here is the scene with 100 reflective metal spheres, drawn in realtime after waiting for a few seconds for the inter-reflections to converge.

100 shiny metal spheres reflecting each other and the surrounding environment in realtime.

The reflections look very noisy if you zoom in. There is no filtering or antialiasing. This is related to the lack of mipmaps that I commented on in my previous post. It turns out that the reason mipmaps didn't work was because I wasn't allocating memory for them during texture creation. This was easy to fix. But mipmaps take around 33% additional GPU memory to store, and 100 spheres is slightly too much. The framerate is 2 FPS again! Fortunately, everything works just fine when I reduce the scene to 80 spheres, and the reflections are nice and smooth. Note that I had to enable 4x anisotropic texture filtering to fix the smearing near the cube map borders and the edges of the spheres. I was expecting this to hurt frame rate, but it actually had no measurable effect. Take a look at the screenshot:

80 metal spheres using mipmaps for smooth, antialiased reflection images.

Refraction

I made an attempt at implementing refraction for transparent materials in my cube map reflection shader. It's not much different from reflection, only some changes to the math involved in the cube map direction lookup. Basically, the rays refract instead of reflect, which amounts to calling a different build-in GPU function. In fact, there are both reflections and refractions involved, depending on the view angle, so you can see some reflections near the object edges. Here is a screenshot of a refractive sphere.

Transparent glass sphere with index of refraction of 1.5 showing inverted refracted image.

Yes, the refracted image is upside-down, and yes, this is correct. Take a look at a reference image. However, the refraction isn't entirely correct. There are at least two sources of error:

1. The rays used when creating the reflection image are parallel to the cube map image plane, and the view rays for reflective objects are still mostly uniform and parallel. On the other hand, the refracted rays are no longer oriented in a nice grid facing parallel to the surface. The refraction bends the rays around so that they come out at all sorts of odd angles. Since there is no depth information in the cube map, there is no way to determine which objects the scattered rays hit. The distance of the plane used to clip and evaluate the rays affects the reflected image, which is incorrect. I used the view frustum near clip plane, which works okay for these scenes.
2. There are really two refractions involved, not just one. The first one is viewer->surface, which is an air->glass refraction and is correctly modeled since the shader has the hit point. The second one is from the back side of the glass surface back out to the air. The shader can't evaluate this refraction because it doesn't have the 3D geometry and can't determine the point where the ray exits the glass. Only the first hit point is known.

I'm not sure how much these two sources of error affect the images. They may not be very significant, but I don't have a reference image so I can't tell for sure.

Here is another image showing refraction in a glass dragon statue. Note that the refraction of the blue and brown chair is probably larger than it should be.

Transparent glass dragon with approximate reflection and refraction.

And here are a reflective sphere, reflective cube, and refractive sphere together.

Mirror sphere (left), glass sphere (right), and mirror cube (front).

I experimented with normal mapped reflective objects, and came up with some very strange effects. I think the strength of the normal map may be too strong in some of the examples. Here is a screenshot of a partially reflective, partially transparent/refractive cube with a stucco normal map that looks somewhat like a rough block of ice.

Normal mapped reflective and refractive object that resembles a rough block of ice.

Dynamic Reflective Objects

I worked around the previous problems I was having when I tried to make dynamic objects reflective. Their collision objects are added and removed every frame, but as long as the reflection generation happens at the right point in the control flow it still works. The cube maps are recreated every frame, which is inefficient, but the overhead is small compared to the time taken to actually render the scene from 6 cube faces.

The problem of different shader usage was solved by adding a config file option that switched to drawing the collision model of objects using the respective shader rather than the dynamic physics model + shader. As long as I limited this to simple shapes such as spheres, both object models are close enough that this works.

Here is a short video showing me throwing a (bouncy?) reflective metal ball. The reflection changes dynamically as the ball moves. Sorry the framerate is jittery in this video. I tried to do 1080p video capture at 60 FPS, but the reflections are too slow and the frame rate varied randomly from around 30-60 FPS. Maybe I should have recorded it at a lower resolution, a lower video frame rate, or on a simpler scene.

Player Model

I haven't had a chance to work on improving the player model, and it's not in the scope of this post anyway. So here is a screenshot of the simple smiley face player model from last time reflected in the gold dragon. Note that the tile floor is also reflective. See if you can spot a reflection of the eyes and nose.

Player model reflection (red smiley face) in a gold dragon statue.

That covers most of the future work todo items from the previous post. I still have to make some optimizations to the cube map rendering to allow for more dynamic objects in the scene while still hitting the 60 FPS target. However, I may be out of new and interesting images to show. But who knows, there may be another reflection post in the near future.