Cube Map Reflections - Part 3

I originally planned on making only a single post on cube map reflections. Then I had so many images that I decided to split it into two posts. Surely, two posts was enough, right? Well, I worked on reflections some more, fixed some problems, added new features, and now I have even more images to show. So here goes Part 3.

For this post, I'm going to list the different problems I had run into and how I solved them. These are mostly the "future work" items from the previous post. I think I have everything more or less worked out now, except for a few minor issues.

Dirty/Imperfect Reflections

This is actually related to planar reflections, not cube map reflections, but the topic is close enough. Someone commented that the reflections in the floors look too clean and perfect. Real floors are scratched, chipped, and dirty. I put a bit of effort into adding support for specular maps into the reflection rendering pipeline so that I could have more realistic surfaces.

My first screenshot shows the museum scene with a scratched floor, using a scratch map found here as a specular map texture. This gives the impression of a floor that has had many large objects pushed across it, damaging the smooth wax coating.

Scratched and scuffed reflective marble floor using a specular map in the museum scene.

Next, I changed the floor of the Crytek Sponza Atrium scene to a dirty and chipped metal material. I tweaked the parameters until I found something I liked.

Dirty, chipped, reflective metal plate floor in the Sponza scene.

Large Numbers of Reflective Objects + Mipmaps

This section involves two different topics. In the previous post, I commented that 3DWorld should scale to over 100 reflective cube map objects, and possibly up to 180. In reality, the max is around 120. I'm not sure why it's lower than I thought. Maybe some memory is needed by MS Windows, or maybe there is allocation overhead or fragmentation. After that (120 spheres) it starts running out of GPU memory and the frame rate drops from ~100 FPS to ~2 FPS. The time to generate reflections increases from a few seconds to a few minutes, and it's not even interactive to view any more.

When there isn't enough GPU memory available, 3DWorld doesn't crash. Instead, the graphics driver starts allocating from system memory (CPU side RAM). I have 16GB of memory on this machine, so it's not going to run out any time soon. Unfortunately, reading from system memory is very slow compared to reading from GPU memory, since the data needs to travel across the motherboard, etc. This really hurts performance! I've seen this effect on both my older machine with an ATI card an on my new machine with an Nvidia card.

So, here is the scene with 100 reflective metal spheres, drawn in realtime after waiting for a few seconds for the inter-reflections to converge.

100 shiny metal spheres reflecting each other and the surrounding environment in realtime.

The reflections look very noisy if you zoom in. There is no filtering or antialiasing. This is related to the lack of mipmaps that I commented on in my previous post. It turns out that the reason mipmaps didn't work was because I wasn't allocating memory for them during texture creation. This was easy to fix. But mipmaps take around 33% additional GPU memory to store, and 100 spheres is slightly too much. The framerate is 2 FPS again! Fortunately, everything works just fine when I reduce the scene to 80 spheres, and the reflections are nice and smooth. Note that I had to enable 4x anisotropic texture filtering to fix the smearing near the cube map borders and the edges of the spheres. I was expecting this to hurt frame rate, but it actually had no measurable effect. Take a look at the screenshot:

80 metal spheres using mipmaps for smooth, antialiased reflection images.

Refraction

I made an attempt at implementing refraction for transparent materials in my cube map reflection shader. It's not much different from reflection, only some changes to the math involved in the cube map direction lookup. Basically, the rays refract instead of reflect, which amounts to calling a different build-in GPU function. In fact, there are both reflections and refractions involved, depending on the view angle, so you can see some reflections near the object edges. Here is a screenshot of a refractive sphere.

Transparent glass sphere with index of refraction of 1.5 showing inverted refracted image.

Yes, the refracted image is upside-down, and yes, this is correct. Take a look at a reference image. However, the refraction isn't entirely correct. There are at least two sources of error:

1. The rays used when creating the reflection image are parallel to the cube map image plane, and the view rays for reflective objects are still mostly uniform and parallel. On the other hand, the refracted rays are no longer oriented in a nice grid facing parallel to the surface. The refraction bends the rays around so that they come out at all sorts of odd angles. Since there is no depth information in the cube map, there is no way to determine which objects the scattered rays hit. The distance of the plane used to clip and evaluate the rays affects the reflected image, which is incorrect. I used the view frustum near clip plane, which works okay for these scenes.
2. There are really two refractions involved, not just one. The first one is viewer->surface, which is an air->glass refraction and is correctly modeled since the shader has the hit point. The second one is from the back side of the glass surface back out to the air. The shader can't evaluate this refraction because it doesn't have the 3D geometry and can't determine the point where the ray exits the glass. Only the first hit point is known.

I'm not sure how much these two sources of error affect the images. They may not be very significant, but I don't have a reference image so I can't tell for sure.

Here is another image showing refraction in a glass dragon statue. Note that the refraction of the blue and brown chair is probably larger than it should be.

Transparent glass dragon with approximate reflection and refraction.

And here are a reflective sphere, reflective cube, and refractive sphere together.

Mirror sphere (left), glass sphere (right), and mirror cube (front).

I experimented with normal mapped reflective objects, and came up with some very strange effects. I think the strength of the normal map may be too strong in some of the examples. Here is a screenshot of a partially reflective, partially transparent/refractive cube with a stucco normal map that looks somewhat like a rough block of ice.

Normal mapped reflective and refractive object that resembles a rough block of ice.

Dynamic Reflective Objects

I worked around the previous problems I was having when I tried to make dynamic objects reflective. Their collision objects are added and removed every frame, but as long as the reflection generation happens at the right point in the control flow it still works. The cube maps are recreated every frame, which is inefficient, but the overhead is small compared to the time taken to actually render the scene from 6 cube faces.

The problem of different shader usage was solved by adding a config file option that switched to drawing the collision model of objects using the respective shader rather than the dynamic physics model + shader. As long as I limited this to simple shapes such as spheres, both object models are close enough that this works.

Here is a short video showing me throwing a (bouncy?) reflective metal ball. The reflection changes dynamically as the ball moves. Sorry the framerate is jittery in this video. I tried to do 1080p video capture at 60 FPS, but the reflections are too slow and the frame rate varied randomly from around 30-60 FPS. Maybe I should have recorded it at a lower resolution, a lower video frame rate, or on a simpler scene.

Player Model

I haven't had a chance to work on improving the player model, and it's not in the scope of this post anyway. So here is a screenshot of the simple smiley face player model from last time reflected in the gold dragon. Note that the tile floor is also reflective. See if you can spot a reflection of the eyes and nose.

Player model reflection (red smiley face) in a gold dragon statue.

That covers most of the future work todo items from the previous post. I still have to make some optimizations to the cube map rendering to allow for more dynamic objects in the scene while still hitting the 60 FPS target. However, I may be out of new and interesting images to show. But who knows, there may be another reflection post in the near future.

Cube Map Reflections - Part 2

Here is the second part of my cube map reflections post. This is going to be mostly images, and a few videos. Most of the work I've done since my previous post from a few days ago was just tweaking things and fixing minor bugs. Many of the issues were related to dynamic/moving reflective objects, which is one of the reasons I left the videos for this post. I also made some optimizations to the rendering, which is nice but still doesn't make scenes with multiple dynamic reflective objects render in realtime. On the other hand, reflections on static objects work just fine and cost almost no frame time. I can likely have over a hundred of these objects, assuming there is enough GPU memory, though I haven't tested a scene with more than 20 of them.

[For reference, each of the 6 cube map faces is up to 768x768 pixels of 8-bit RGB image data. That's 6*768*768*3 = 10MB of GPU data. My GPU has 2GB of memory, so I can have up to 200 reflective cube map objects if I use all of it. In reality, the rest of the scene needs around 200MB, so I can only have 180 objects max.]

To start off, here is the gold dragon, but this time in silver, with a few minor improvements.

Reflective silver dragon drawn in realtime.

I created a test scene with two rows of 8 reflective silver spheres. This was mostly for testing object inter-reflections, but also made for a good performance test. Dynamic rendering has to be turned on so that the nested reflections of other spheres are correctly created. It seems to converge after around 1-2 seconds, or 10-20 frames at 10 FPS. Once the images stabilize the rendering mode can be switched back to static to recover the realtime framerate for viewing the scene.

Gold dragon and 16 mirrored spheres reflecting each other in realtime (after several seconds of processing).

The reflections are close to physically accurate, and look like what you would get out of ray tracing. There are many levels of nested reflections of spheres within spheres. This works because each frame, the spheres are iterated over for creation of the reflection images of their 6 faces. In the first frame/pass, none of the spheres start reflective as their images haven't been created yet. They're just white spheres. On the second pass, the reflections are present, but they show the nearby spheres reflected as pure white. On the third pass, the reflection images pick up nearby spheres whose reflections contain other white spheres. This continues until the user stops the simulation, where each new frame increases the level of nested reflections by one. Think of it as simulating once bounce of light per frame. After a short period of time, the nested reflections become so small that the difference is not visible and the images no longer change.

[This reminds me of something. When I was a kid, I had a yellow bucket that I used for playing with sand and water. The bucket had a sheet of paper glued to it that had a picture of another yellow bucket. The bucket in the picture also had a (smaller) picture of a bucket on it. There were 4 or 5 levels of nested pictures of buckets that I could see. I always wondered how they created this (and why). This was back before the time of computer graphics, so it was probably done using cameras. Someone took a picture of a bucket, printed it, put it on another bucket, then took a picture of that ... repeated 4-5 times. Interesting.]

Here is another view showing how the gold dragon also reflects in the 16 spheres.

Closeup of gold dragon reflected in metal spheres.

Next, I changed the material properties of the spheres in the config text file, added a torus and a cube, and reloaded the scene. The new scene contains a combination of shapes, materials, and colors:

Various reflective materials and shapes. They even cast faint shadows on the floor.

The materials shown here are:

• Gold dragon (center)
• Silver torus (above) - not self reflective, sorry
• Silver cube (on the floor in the back)
• Marble spheres (left; front to back: black, blue, red, white)
• Metal spheres (right; front to back: gold, brass, copper, silver)

I used real values for index of refraction and metal specular colors that I found online. Here is the scene from another angle.

Closeup of reflective silver torus and other shapes.

I added a reflective torus and some other objects to the Crytek Sponza atrium scene as well, to convince readers that this works on more than one scene. It should work on just about anything, including outdoor scenes full of grass and trees.

Reflective metal torus and other shapes in the Crytek Sponza atrium scene.

I made one of the metal cubes movable so that the player can push it around the scene. Every time the object moves, the reflection textures are regenerated so that they're always correct. I pushed the cube around the office building and into the basement to make sure the dynamic spotlights and shadows work correctly with it. Everything looks correct in this screenshot.

Movable cube casting a dynamic light shadow and reflecting dynamic lights and shadows

Someone asked me about the lack of player reflection in my previous post. Before I added reflections, there was no need to create a player model because it could never be seen. 3DWorld is a first person game engine (except for universe mode), so the camera is always at the position of the player's eyes and therefore can't see the player's own face and body. I decided that it would be nice to have a player model for reference, even if it was very simple. I used the smiley face enemy AI model because it was already available, but I'll likely replace it with something better in the future. The user can toggle drawing of the player model on and off. This is what it looks like:

Reflections of the player as a simple smiley face 3D model.

If you look closely, you'll see that the smiley is actually floating off the ground. This is because I have the camera height set higher than the AI enemy smiley height in this scene, as it feels more natural when walking around the building. That way, the player's eyes are actually above the desk and tables. The player model is a capsule shape rather than a sphere, but the visual model is spherical, so there is a mismatch.

Videos

I created some short videos showing off reflections with dynamic objects and view directions. The first video is of the camera moving around the reflective objects in realtime after the reflections have been computed.

The second video is of me pushing a reflective cube around the scene to show off the dynamic reflection updates. Note that the sun flare reflection wasn't working at this point, so the sun appears as a small yellow circle in the top of the cube.

Rendering Bugs

Here are some failed attempts at reflective sphere rendering. I'm throwing these in just for fun, because they looked silly and it took awhile for me to figure out what went wrong.

What do you suppose happened to this reflective sphere? Did someone wrap a leaf around it?

Reflection of a ... leaf?

This is what you get when enabling a specular map texture but not binding it to the correct texture unit. The specular map uses a leaf texture, which was the last texture bound to that slot from the previous tree rendering pass. Oops. This object doesn't even use or need a specular map.


What about this one? Surely this is how the reflection is supposed to look in a sphere, right?

Simulation of a micro black hole.

I had the wrong sign/direction for the reflection vector, and light was reflected into the center of the sphere rather than away from it. I encountered several variants of this bug while trying to make the shader code correct. Oops again.


Future Work

I wanted to add a video of me throwing a bouncy reflective ball, but I haven't gotten that working yet. The problem is that the ball uses a different type of physics enabled scene object than the other reflective items I've shown. Each ball is converted to a collision object (the type that can be made reflective) so that it participates in collision detection, but it's added and removed every frame. In fact, its collision model isn't valid at the point where the reflection images are created. A second problem is that physics objects aren't drawn with a shader that supports reflections. I'll have to come up with some solution to these issues later.

All of the images shown here have a high specular exponent and are very glossy. I haven't implemented low-gloss surfaces yet. The traditional way to do this is to filter the reflection texture down to a lower resolution, which blurs the colors together. The simplest way to do this is to use mipmaps on the GPU and select a mipmap level based on a material glossiness parameter. I would like to do this, but I haven't gotten mipmaps to work yet on my cube map render-to-texture flow. Maybe there is an online tutorial that can help me with this problem.

Another interesting effect to simulate would be refraction. Unfortunately, this seems difficult to get right without knowing the geometry of the material for computing the optical thickness effects, etc. I have it working somewhat for a sphere (most people wouldn't know it was wrong), and objects with smooth curved surfaces in general, but it doesn't work at all for cubes. I'll have to think about this some more.

Finally, I need to work on creating a better player model.

Cube Map Reflections

As promised in my previous post, I managed to get cube map environment reflections working in 3DWorld. It was more difficult than I expected, but in the end actually looks better than I expected. Part of the problem was that I kept running into graphics problems that were difficult to debug, such as cryptic OpenGL errors, shaders that didn't work, upside down images, incorrect lighting, half black images, etc. It's very difficult to debug graphics code, especially when you're trying to implement an entirely new system from scratch. There are some helpful tutorials out there such as this, but many of the online info is incorrect, outdated, or doesn't work on my OS/graphics card.

How do cube map reflections work? The scene is rendered for each of the six sides of a cube surrounding the object. Each cube side has a 90 degree field of view looking out from the cube center. This is similar to a 360 degree panoramic photograph created from 4 individual photographs, but in addition has the top and bottom faces that you would get by taking a picture of the ground and the sky. Together, the six faces generate an image that covers all possible directions from the center of the cube. Each image is created by rendering the scene once and caching the resulting image as a texture on the GPU. This is a slow process, but only needs to be done when the reflective object moves or something in the scene changes. The texture images are then used to look up the reflected color for each pixel of the object in the fragment shader. The reflected view vector is used to index into one of the pixels in one of the faces based on the ray direction.

If the background of the scene, the "environment", is sufficiently far away from the reflecting object(s), the same cube map can be used for the entire scene. This is called environment mapping, and the cube map textures form a skybox. It works well for the sky, clouds, and distant mountains, but not as well for building interiors. The problem is that the parallax looks incorrect when the camera moves around the object. Also, the reflections of nearby objects appear more distorted. Therefore, for the scenes I'm using in 3DWorld, I had to create a separate cube map for each reflective object.

I'm attaching a lot of screenshots to this post. In fact, I have so many screenshots that I'll need to split the post into two parts. Keep in mind that everything is rendered in realtime even when the player/camera are moving. The scene and reflective objects can move as well, though the frame rate drops to 40-60 FPS with one reflective object as the scene is rendered up to 8 times (normal scene, 6 cube faces, and reflective floor). Actually, I cheated a bit and skipped drawing of the cube faces that are oriented away from the camera, which isn't always legal (but mostly works for convex reflectors).

I initially put all reflective objects in the same place in the scene so that the screenshots can be compared with different materials. Later screenshots will show more variety. I really just want to get the materials and reflection vectors correct to start with. The office building scene is one I've shown many times before. I put the reflective object above the desk in the lobby. The floor is partially reflective tile, the ceiling is a grid of white and brown, and the walls are white with some tan. The windows and doors are mostly transparent so that the player can see outside.

My first test used a traditional perfectly reflective metal sphere, like a large ball bearing. Here is how it looks when viewed from up close:

Metal sphere with a perfect mirror reflective surface (pure specular reflection).

Later, I realized that the reflection vector was incorrect. I updated the metal sphere image since it was the most incorrect, but didn't update the others because the images changed very little. Here is the new screenshot, with a sphere that extends slightly below the desk so that you can see the reflection of the inside of the desk. I was moving it around and didn't get it back into the original position.

Reflective metal sphere with corrected reflection vector resting on (and partially inside of) a desk.

Note that the tile floor reflections are shown in the sphere reflections, though I don't think it's quite right. It looks like the floor is reflecting the sky instead of the building interior. I'll have to work on that later. [Update: I figured it out, it's a transform matrix problem, but it's not so easy to fix.] Also, the player has no model and therefore creates no reflection.

Next, I changed the metal sphere to a dielectric material. Here is a giant glass marble, using real material parameters (index of refraction, etc.) and a Fresnel reflection equation. Only the edges of the sphere that are viewed at a glancing angle are highly reflective. This is physically correct - try it yourself at home.

White reflective dielectric sphere (smooth marble).

Here is the same sphere, but this time it's black and viewed from the other side. The effect is similar, and there is an additional white specular reflection from the ceiling light near the top of the sphere.

Black reflective dielectric sphere (smooth marble).

I then added some dynamic light sources to test out reflections from dynamic objects. The addition of dynamic objects reduces the frame rate, but it's still realtime.

Black reflective marble sphere with dynamic light sources creating specular highlights.

There are 100 floating, moving, colored lights in this scene. Several of the nearby lights create specular highlights on the sphere, and some of the lights behind the player are visible as reflections. You can also see the reflection of the chair that's directly behind and below the camera, and the table with bottles on the left side of the sphere.

Okay, it's nice to have reflective spheres. But what about real objects? The objects I want to use in 3DWorld are more complex than simple spheres. One likely candidate for a reflective object is the gold Chinese dragon that I placed on the desk in the lobby. I've had that statue sitting there on the desk for months and every time I look at it, I think, "That should be reflective." Well, it finally is!

Reflective gold dragon statue (871K triangles).

The dragon model is 871K triangles, which shows that reflections can be applied to an object of high complexity. I have no idea what the reflections should look like here. Fortunately, I spent all that time making reflections correct for simple spheres and cubes, so I'm pretty confident that the dragon reflections are at least somewhat correct.

Wait, I didn't add any reflective cube images? Well, they're not very interesting, they're basically just mirrors. I did add a reflective cube that the player can push around the scene. I'll add a video of that to a future post.

Note that the reflective object is concave, but is not self-reflective. One part of the object is not reflected in another part of the object. I ran into too many problems with the object intersecting the near clipping plan and had to move the near plane out beyond the furthest vertex of the object. Maybe some sort of back face culling would have helped, if all of my models actually had the correct face orientations. That's too much to expect from free models that I found online.

Here is another view of the gold dragon, during night time with some dynamic floating lights. This is still rendered in realtime.

Reflective gold dragon with dynamic floating lights.

For some reason, only the green lights like to float near the dragon. Here is a closer view with at least one green light reflecting off the gold. Someone must have put a lot of effort into keeping this dragon shiny by cleaning it every day.

Closeup of gold dragon showing specular reflections from a floating green dynamic light.

The dragon statue material can also be changed. Here it is in white marble.

Ceramic dragon with Fresnel surface reflections from the environment.

 And here it is in black marble.

Black glossy ceramic dragon lit only by reflected specular environment lighting.

The white lines near the silhouette edges of the dragon are actually a form of specular aliasing. The view vector is nearly tangent to the surface at those locations, which makes the Fresnel reflectivity approach 1.0 (mirror reflection). In addition, the triangles in those locations come together at sharp edges with high frequency noise, so their normals change very quickly. This makes the reflected rays bounce around randomly. Many of the rays hit the bright white walls and ceiling, creating bright white spots (pixels) in those locations. I'm not sure how to remove those artifacts, or if they're real physical effects. I don't happen to have a shiny black marble dragon around for reference. I could clamp the max specular reflection value, but that would break the properties of my metal mirror surfaces.

I think the small white spots on the dragon are triangles that have incorrect orientation so that their normals have the wrong sign. I've seen those on other images but they really stand out with the shiny black material. I'm not sure how much effort I want to put into fixing this, considering it only occurs in this combination of material and model. I guess I better make the dragon white or gold in the final scene.

I think 10 screenshots is enough for this post. I want to make sure the page doesn't take too long to load for those of you on mobile devices or slow internet connections. I'll make a follow-up post with more screenshots of multiple inter-reflecting objects, reflections in other scenes, and possibly some videos. I'll also try to include a section on "future work".