Cube Map Reflections - Part 3

I originally planned on making only a single post on cube map reflections. Then I had so many images that I decided to split it into two posts. Surely, two posts was enough, right? Well, I worked on reflections some more, fixed some problems, added new features, and now I have even more images to show. So here goes Part 3.

For this post, I'm going to list the different problems I had run into and how I solved them. These are mostly the "future work" items from the previous post. I think I have everything more or less worked out now, except for a few minor issues.

Dirty/Imperfect Reflections

This is actually related to planar reflections, not cube map reflections, but the topic is close enough. Someone commented that the reflections in the floors look too clean and perfect. Real floors are scratched, chipped, and dirty. I put a bit of effort into adding support for specular maps into the reflection rendering pipeline so that I could have more realistic surfaces.

My first screenshot shows the museum scene with a scratched floor, using a scratch map found here as a specular map texture. This gives the impression of a floor that has had many large objects pushed across it, damaging the smooth wax coating.

Scratched and scuffed reflective marble floor using a specular map in the museum scene.

Next, I changed the floor of the Crytek Sponza Atrium scene to a dirty and chipped metal material. I tweaked the parameters until I found something I liked.

Dirty, chipped, reflective metal plate floor in the Sponza scene.

Large Numbers of Reflective Objects + Mipmaps

This section involves two different topics. In the previous post, I commented that 3DWorld should scale to over 100 reflective cube map objects, and possibly up to 180. In reality, the max is around 120. I'm not sure why it's lower than I thought. Maybe some memory is needed by MS Windows, or maybe there is allocation overhead or fragmentation. After that (120 spheres) it starts running out of GPU memory and the frame rate drops from ~100 FPS to ~2 FPS. The time to generate reflections increases from a few seconds to a few minutes, and it's not even interactive to view any more.

When there isn't enough GPU memory available, 3DWorld doesn't crash. Instead, the graphics driver starts allocating from system memory (CPU side RAM). I have 16GB of memory on this machine, so it's not going to run out any time soon. Unfortunately, reading from system memory is very slow compared to reading from GPU memory, since the data needs to travel across the motherboard, etc. This really hurts performance! I've seen this effect on both my older machine with an ATI card an on my new machine with an Nvidia card.

So, here is the scene with 100 reflective metal spheres, drawn in realtime after waiting for a few seconds for the inter-reflections to converge.

100 shiny metal spheres reflecting each other and the surrounding environment in realtime.

The reflections look very noisy if you zoom in. There is no filtering or antialiasing. This is related to the lack of mipmaps that I commented on in my previous post. It turns out that the reason mipmaps didn't work was because I wasn't allocating memory for them during texture creation. This was easy to fix. But mipmaps take around 33% additional GPU memory to store, and 100 spheres is slightly too much. The framerate is 2 FPS again! Fortunately, everything works just fine when I reduce the scene to 80 spheres, and the reflections are nice and smooth. Note that I had to enable 4x anisotropic texture filtering to fix the smearing near the cube map borders and the edges of the spheres. I was expecting this to hurt frame rate, but it actually had no measurable effect. Take a look at the screenshot:

80 metal spheres using mipmaps for smooth, antialiased reflection images.

Refraction

I made an attempt at implementing refraction for transparent materials in my cube map reflection shader. It's not much different from reflection, only some changes to the math involved in the cube map direction lookup. Basically, the rays refract instead of reflect, which amounts to calling a different build-in GPU function. In fact, there are both reflections and refractions involved, depending on the view angle, so you can see some reflections near the object edges. Here is a screenshot of a refractive sphere.

Transparent glass sphere with index of refraction of 1.5 showing inverted refracted image.

Yes, the refracted image is upside-down, and yes, this is correct. Take a look at a reference image. However, the refraction isn't entirely correct. There are at least two sources of error:

1. The rays used when creating the reflection image are parallel to the cube map image plane, and the view rays for reflective objects are still mostly uniform and parallel. On the other hand, the refracted rays are no longer oriented in a nice grid facing parallel to the surface. The refraction bends the rays around so that they come out at all sorts of odd angles. Since there is no depth information in the cube map, there is no way to determine which objects the scattered rays hit. The distance of the plane used to clip and evaluate the rays affects the reflected image, which is incorrect. I used the view frustum near clip plane, which works okay for these scenes.
2. There are really two refractions involved, not just one. The first one is viewer->surface, which is an air->glass refraction and is correctly modeled since the shader has the hit point. The second one is from the back side of the glass surface back out to the air. The shader can't evaluate this refraction because it doesn't have the 3D geometry and can't determine the point where the ray exits the glass. Only the first hit point is known.

I'm not sure how much these two sources of error affect the images. They may not be very significant, but I don't have a reference image so I can't tell for sure.

Here is another image showing refraction in a glass dragon statue. Note that the refraction of the blue and brown chair is probably larger than it should be.

Transparent glass dragon with approximate reflection and refraction.

And here are a reflective sphere, reflective cube, and refractive sphere together.

Mirror sphere (left), glass sphere (right), and mirror cube (front).

I experimented with normal mapped reflective objects, and came up with some very strange effects. I think the strength of the normal map may be too strong in some of the examples. Here is a screenshot of a partially reflective, partially transparent/refractive cube with a stucco normal map that looks somewhat like a rough block of ice.

Normal mapped reflective and refractive object that resembles a rough block of ice.

Dynamic Reflective Objects

I worked around the previous problems I was having when I tried to make dynamic objects reflective. Their collision objects are added and removed every frame, but as long as the reflection generation happens at the right point in the control flow it still works. The cube maps are recreated every frame, which is inefficient, but the overhead is small compared to the time taken to actually render the scene from 6 cube faces.

The problem of different shader usage was solved by adding a config file option that switched to drawing the collision model of objects using the respective shader rather than the dynamic physics model + shader. As long as I limited this to simple shapes such as spheres, both object models are close enough that this works.

Here is a short video showing me throwing a (bouncy?) reflective metal ball. The reflection changes dynamically as the ball moves. Sorry the framerate is jittery in this video. I tried to do 1080p video capture at 60 FPS, but the reflections are too slow and the frame rate varied randomly from around 30-60 FPS. Maybe I should have recorded it at a lower resolution, a lower video frame rate, or on a simpler scene.

Player Model

I haven't had a chance to work on improving the player model, and it's not in the scope of this post anyway. So here is a screenshot of the simple smiley face player model from last time reflected in the gold dragon. Note that the tile floor is also reflective. See if you can spot a reflection of the eyes and nose.

Player model reflection (red smiley face) in a gold dragon statue.

That covers most of the future work todo items from the previous post. I still have to make some optimizations to the cube map rendering to allow for more dynamic objects in the scene while still hitting the 60 FPS target. However, I may be out of new and interesting images to show. But who knows, there may be another reflection post in the near future.

Cube Map Reflections - Part 2

Here is the second part of my cube map reflections post. This is going to be mostly images, and a few videos. Most of the work I've done since my previous post from a few days ago was just tweaking things and fixing minor bugs. Many of the issues were related to dynamic/moving reflective objects, which is one of the reasons I left the videos for this post. I also made some optimizations to the rendering, which is nice but still doesn't make scenes with multiple dynamic reflective objects render in realtime. On the other hand, reflections on static objects work just fine and cost almost no frame time. I can likely have over a hundred of these objects, assuming there is enough GPU memory, though I haven't tested a scene with more than 20 of them.

[For reference, each of the 6 cube map faces is up to 768x768 pixels of 8-bit RGB image data. That's 6*768*768*3 = 10MB of GPU data. My GPU has 2GB of memory, so I can have up to 200 reflective cube map objects if I use all of it. In reality, the rest of the scene needs around 200MB, so I can only have 180 objects max.]

To start off, here is the gold dragon, but this time in silver, with a few minor improvements.

Reflective silver dragon drawn in realtime.

I created a test scene with two rows of 8 reflective silver spheres. This was mostly for testing object inter-reflections, but also made for a good performance test. Dynamic rendering has to be turned on so that the nested reflections of other spheres are correctly created. It seems to converge after around 1-2 seconds, or 10-20 frames at 10 FPS. Once the images stabilize the rendering mode can be switched back to static to recover the realtime framerate for viewing the scene.

Gold dragon and 16 mirrored spheres reflecting each other in realtime (after several seconds of processing).

The reflections are close to physically accurate, and look like what you would get out of ray tracing. There are many levels of nested reflections of spheres within spheres. This works because each frame, the spheres are iterated over for creation of the reflection images of their 6 faces. In the first frame/pass, none of the spheres start reflective as their images haven't been created yet. They're just white spheres. On the second pass, the reflections are present, but they show the nearby spheres reflected as pure white. On the third pass, the reflection images pick up nearby spheres whose reflections contain other white spheres. This continues until the user stops the simulation, where each new frame increases the level of nested reflections by one. Think of it as simulating once bounce of light per frame. After a short period of time, the nested reflections become so small that the difference is not visible and the images no longer change.

[This reminds me of something. When I was a kid, I had a yellow bucket that I used for playing with sand and water. The bucket had a sheet of paper glued to it that had a picture of another yellow bucket. The bucket in the picture also had a (smaller) picture of a bucket on it. There were 4 or 5 levels of nested pictures of buckets that I could see. I always wondered how they created this (and why). This was back before the time of computer graphics, so it was probably done using cameras. Someone took a picture of a bucket, printed it, put it on another bucket, then took a picture of that ... repeated 4-5 times. Interesting.]

Here is another view showing how the gold dragon also reflects in the 16 spheres.

Closeup of gold dragon reflected in metal spheres.

Next, I changed the material properties of the spheres in the config text file, added a torus and a cube, and reloaded the scene. The new scene contains a combination of shapes, materials, and colors:

Various reflective materials and shapes. They even cast faint shadows on the floor.

The materials shown here are:

• Gold dragon (center)
• Silver torus (above) - not self reflective, sorry
• Silver cube (on the floor in the back)
• Marble spheres (left; front to back: black, blue, red, white)
• Metal spheres (right; front to back: gold, brass, copper, silver)

I used real values for index of refraction and metal specular colors that I found online. Here is the scene from another angle.

Closeup of reflective silver torus and other shapes.

I added a reflective torus and some other objects to the Crytek Sponza atrium scene as well, to convince readers that this works on more than one scene. It should work on just about anything, including outdoor scenes full of grass and trees.

Reflective metal torus and other shapes in the Crytek Sponza atrium scene.

I made one of the metal cubes movable so that the player can push it around the scene. Every time the object moves, the reflection textures are regenerated so that they're always correct. I pushed the cube around the office building and into the basement to make sure the dynamic spotlights and shadows work correctly with it. Everything looks correct in this screenshot.

Movable cube casting a dynamic light shadow and reflecting dynamic lights and shadows

Someone asked me about the lack of player reflection in my previous post. Before I added reflections, there was no need to create a player model because it could never be seen. 3DWorld is a first person game engine (except for universe mode), so the camera is always at the position of the player's eyes and therefore can't see the player's own face and body. I decided that it would be nice to have a player model for reference, even if it was very simple. I used the smiley face enemy AI model because it was already available, but I'll likely replace it with something better in the future. The user can toggle drawing of the player model on and off. This is what it looks like:

Reflections of the player as a simple smiley face 3D model.

If you look closely, you'll see that the smiley is actually floating off the ground. This is because I have the camera height set higher than the AI enemy smiley height in this scene, as it feels more natural when walking around the building. That way, the player's eyes are actually above the desk and tables. The player model is a capsule shape rather than a sphere, but the visual model is spherical, so there is a mismatch.

Videos

I created some short videos showing off reflections with dynamic objects and view directions. The first video is of the camera moving around the reflective objects in realtime after the reflections have been computed.

The second video is of me pushing a reflective cube around the scene to show off the dynamic reflection updates. Note that the sun flare reflection wasn't working at this point, so the sun appears as a small yellow circle in the top of the cube.

Rendering Bugs

Here are some failed attempts at reflective sphere rendering. I'm throwing these in just for fun, because they looked silly and it took awhile for me to figure out what went wrong.

What do you suppose happened to this reflective sphere? Did someone wrap a leaf around it?

Reflection of a ... leaf?

This is what you get when enabling a specular map texture but not binding it to the correct texture unit. The specular map uses a leaf texture, which was the last texture bound to that slot from the previous tree rendering pass. Oops. This object doesn't even use or need a specular map.


What about this one? Surely this is how the reflection is supposed to look in a sphere, right?

Simulation of a micro black hole.

I had the wrong sign/direction for the reflection vector, and light was reflected into the center of the sphere rather than away from it. I encountered several variants of this bug while trying to make the shader code correct. Oops again.


Future Work

I wanted to add a video of me throwing a bouncy reflective ball, but I haven't gotten that working yet. The problem is that the ball uses a different type of physics enabled scene object than the other reflective items I've shown. Each ball is converted to a collision object (the type that can be made reflective) so that it participates in collision detection, but it's added and removed every frame. In fact, its collision model isn't valid at the point where the reflection images are created. A second problem is that physics objects aren't drawn with a shader that supports reflections. I'll have to come up with some solution to these issues later.

All of the images shown here have a high specular exponent and are very glossy. I haven't implemented low-gloss surfaces yet. The traditional way to do this is to filter the reflection texture down to a lower resolution, which blurs the colors together. The simplest way to do this is to use mipmaps on the GPU and select a mipmap level based on a material glossiness parameter. I would like to do this, but I haven't gotten mipmaps to work yet on my cube map render-to-texture flow. Maybe there is an online tutorial that can help me with this problem.

Another interesting effect to simulate would be refraction. Unfortunately, this seems difficult to get right without knowing the geometry of the material for computing the optical thickness effects, etc. I have it working somewhat for a sphere (most people wouldn't know it was wrong), and objects with smooth curved surfaces in general, but it doesn't work at all for cubes. I'll have to think about this some more.

Finally, I need to work on creating a better player model.

Cube Map Reflections

As promised in my previous post, I managed to get cube map environment reflections working in 3DWorld. It was more difficult than I expected, but in the end actually looks better than I expected. Part of the problem was that I kept running into graphics problems that were difficult to debug, such as cryptic OpenGL errors, shaders that didn't work, upside down images, incorrect lighting, half black images, etc. It's very difficult to debug graphics code, especially when you're trying to implement an entirely new system from scratch. There are some helpful tutorials out there such as this, but many of the online info is incorrect, outdated, or doesn't work on my OS/graphics card.

How do cube map reflections work? The scene is rendered for each of the six sides of a cube surrounding the object. Each cube side has a 90 degree field of view looking out from the cube center. This is similar to a 360 degree panoramic photograph created from 4 individual photographs, but in addition has the top and bottom faces that you would get by taking a picture of the ground and the sky. Together, the six faces generate an image that covers all possible directions from the center of the cube. Each image is created by rendering the scene once and caching the resulting image as a texture on the GPU. This is a slow process, but only needs to be done when the reflective object moves or something in the scene changes. The texture images are then used to look up the reflected color for each pixel of the object in the fragment shader. The reflected view vector is used to index into one of the pixels in one of the faces based on the ray direction.

If the background of the scene, the "environment", is sufficiently far away from the reflecting object(s), the same cube map can be used for the entire scene. This is called environment mapping, and the cube map textures form a skybox. It works well for the sky, clouds, and distant mountains, but not as well for building interiors. The problem is that the parallax looks incorrect when the camera moves around the object. Also, the reflections of nearby objects appear more distorted. Therefore, for the scenes I'm using in 3DWorld, I had to create a separate cube map for each reflective object.

I'm attaching a lot of screenshots to this post. In fact, I have so many screenshots that I'll need to split the post into two parts. Keep in mind that everything is rendered in realtime even when the player/camera are moving. The scene and reflective objects can move as well, though the frame rate drops to 40-60 FPS with one reflective object as the scene is rendered up to 8 times (normal scene, 6 cube faces, and reflective floor). Actually, I cheated a bit and skipped drawing of the cube faces that are oriented away from the camera, which isn't always legal (but mostly works for convex reflectors).

I initially put all reflective objects in the same place in the scene so that the screenshots can be compared with different materials. Later screenshots will show more variety. I really just want to get the materials and reflection vectors correct to start with. The office building scene is one I've shown many times before. I put the reflective object above the desk in the lobby. The floor is partially reflective tile, the ceiling is a grid of white and brown, and the walls are white with some tan. The windows and doors are mostly transparent so that the player can see outside.

My first test used a traditional perfectly reflective metal sphere, like a large ball bearing. Here is how it looks when viewed from up close:

Metal sphere with a perfect mirror reflective surface (pure specular reflection).

Later, I realized that the reflection vector was incorrect. I updated the metal sphere image since it was the most incorrect, but didn't update the others because the images changed very little. Here is the new screenshot, with a sphere that extends slightly below the desk so that you can see the reflection of the inside of the desk. I was moving it around and didn't get it back into the original position.

Reflective metal sphere with corrected reflection vector resting on (and partially inside of) a desk.

Note that the tile floor reflections are shown in the sphere reflections, though I don't think it's quite right. It looks like the floor is reflecting the sky instead of the building interior. I'll have to work on that later. [Update: I figured it out, it's a transform matrix problem, but it's not so easy to fix.] Also, the player has no model and therefore creates no reflection.

Next, I changed the metal sphere to a dielectric material. Here is a giant glass marble, using real material parameters (index of refraction, etc.) and a Fresnel reflection equation. Only the edges of the sphere that are viewed at a glancing angle are highly reflective. This is physically correct - try it yourself at home.

White reflective dielectric sphere (smooth marble).

Here is the same sphere, but this time it's black and viewed from the other side. The effect is similar, and there is an additional white specular reflection from the ceiling light near the top of the sphere.

Black reflective dielectric sphere (smooth marble).

I then added some dynamic light sources to test out reflections from dynamic objects. The addition of dynamic objects reduces the frame rate, but it's still realtime.

Black reflective marble sphere with dynamic light sources creating specular highlights.

There are 100 floating, moving, colored lights in this scene. Several of the nearby lights create specular highlights on the sphere, and some of the lights behind the player are visible as reflections. You can also see the reflection of the chair that's directly behind and below the camera, and the table with bottles on the left side of the sphere.

Okay, it's nice to have reflective spheres. But what about real objects? The objects I want to use in 3DWorld are more complex than simple spheres. One likely candidate for a reflective object is the gold Chinese dragon that I placed on the desk in the lobby. I've had that statue sitting there on the desk for months and every time I look at it, I think, "That should be reflective." Well, it finally is!

Reflective gold dragon statue (871K triangles).

The dragon model is 871K triangles, which shows that reflections can be applied to an object of high complexity. I have no idea what the reflections should look like here. Fortunately, I spent all that time making reflections correct for simple spheres and cubes, so I'm pretty confident that the dragon reflections are at least somewhat correct.

Wait, I didn't add any reflective cube images? Well, they're not very interesting, they're basically just mirrors. I did add a reflective cube that the player can push around the scene. I'll add a video of that to a future post.

Note that the reflective object is concave, but is not self-reflective. One part of the object is not reflected in another part of the object. I ran into too many problems with the object intersecting the near clipping plan and had to move the near plane out beyond the furthest vertex of the object. Maybe some sort of back face culling would have helped, if all of my models actually had the correct face orientations. That's too much to expect from free models that I found online.

Here is another view of the gold dragon, during night time with some dynamic floating lights. This is still rendered in realtime.

Reflective gold dragon with dynamic floating lights.

For some reason, only the green lights like to float near the dragon. Here is a closer view with at least one green light reflecting off the gold. Someone must have put a lot of effort into keeping this dragon shiny by cleaning it every day.

Closeup of gold dragon showing specular reflections from a floating green dynamic light.

The dragon statue material can also be changed. Here it is in white marble.

Ceramic dragon with Fresnel surface reflections from the environment.

 And here it is in black marble.

Black glossy ceramic dragon lit only by reflected specular environment lighting.

The white lines near the silhouette edges of the dragon are actually a form of specular aliasing. The view vector is nearly tangent to the surface at those locations, which makes the Fresnel reflectivity approach 1.0 (mirror reflection). In addition, the triangles in those locations come together at sharp edges with high frequency noise, so their normals change very quickly. This makes the reflected rays bounce around randomly. Many of the rays hit the bright white walls and ceiling, creating bright white spots (pixels) in those locations. I'm not sure how to remove those artifacts, or if they're real physical effects. I don't happen to have a shiny black marble dragon around for reference. I could clamp the max specular reflection value, but that would break the properties of my metal mirror surfaces.

I think the small white spots on the dragon are triangles that have incorrect orientation so that their normals have the wrong sign. I've seen those on other images but they really stand out with the shiny black material. I'm not sure how much effort I want to put into fixing this, considering it only occurs in this combination of material and model. I guess I better make the dragon white or gold in the final scene.

I think 10 screenshots is enough for this post. I want to make sure the page doesn't take too long to load for those of you on mobile devices or slow internet connections. I'll make a follow-up post with more screenshots of multiple inter-reflecting objects, reflections in other scenes, and possibly some videos. I'll also try to include a section on "future work".

More Reflections

I can't get enough reflections in 3DWorld. They really bring life to the scenes. I recently added reflective surface support for horizontal "floor" polygons of imported models. The fragment shader enables reading from the reflection texture for fragments with z-values within the z-range of the reflection plane. Other than that change, the rest of the reflection texture creation works like I explained in my previous post.


Here is a screenshot showing the reflective marble floor from the Crytek Sponza atrium scene. You can see the reflected colored curtains and fire in the floor.

Realtime reflections on the shiny marble floor of the Crytek Sponza atrium.

I also added support for multiple reflective z-planes so that more than one floor can be reflective. The active reflective surface is selected as the one with z-value closest to the camera position from the visible surfaces below the camera. Surfaces above the camera are ignored because the system assumes only the top of the objects are reflective. Here is a screenshot showing reflections on the upper floor of the Sponza atrium with some moving dynamic light sources.

Reflections of the scene and some colored lights on the upper level of the Sponza scene.

I made the marble floor of the museum scene reflective as well. It makes the scene look much more realistic. Take a look at the way the windows look reflected from the floor.

Reflections on the marble floor of the museum scene.

Here is the museum scene shown from a different viewpoint where the dinosaur skeletons are visible.

Another view of the museum scene showing the reflective floor.

I'm currently working on cube map reflections for small placed reflective objects such as the golden dragon in the office building scene. Cube map reflections are more difficult than planar reflections because they need to be rendered from all six sides of a cube rather than a single reflection plane. It's probably too slow to generate all six cube face images each frame for large scenes, so other tricks need to be played. For example, if the scene is static (no moving objects), the reflection images can be rendered once as preprocessing rather than per-frame. I'll make another post if/when I get this working correctly.

Overhead Map View

This past week I have been making improvements to 3DWorld's overhead map view. This was originally a low resolution, cartoon-ish overhead view of the entire level that showed the level bounds, player position and direction, enemy positions, and item locations. The player can zoom in and pan around in the image interactively using the arrow keys.

I decided to add some improvements to the original system that I wrote many years ago. First, I added mouse wheel zoom and mouse click-and-drag for a "slippy map" interface that works like Google Maps. The update rate of map view is somewhere between 10 and 30 FPS, which seems to be okay for a drag-able map.

Each pixel of the map is computed independently. The colors are determined from a height-based table lookup and match the normal colors you tend to see in maps: blue for water, brown for dirt, green for vegetation, gray for exposed mountain rock, and white for snowy peaks. I added sun lighting by computing the normal at each pixel to make the peaks and valleys stand out more. Here is what this view looks like for a procedurally generated island scene. The player is located at the red dot in the center facing in the direction of the black dot.

Map view of an island with correct lighting applied, which produces mountains that look 3D.

This approach works for heightmap textures as well. The user can interactively pan and zoom within a high resolution heightmap and view a pseudo-colored image of the terrain. It works just like an image viewing tool. In the following screenshot, I used the 16K by 16K pixel Puget Sound heightmap dataset available here.

Overhead map view of Mount Rainier from the 16K x 16K pixel Puget Sound texture with real lighting.

Note that interactive heightmap editing done in 3D mode will automatically update the overhead map view. Some of my custom height editing can be seen in the area around the player on the left. Those small lakes and green peaks aren't in the original heightmap dataset.

Here is a video where I pan around the Puget Sound heightmap and change the sun position/direction to prove that map view really does compute per-pixel lighting. The sun moves slowly because I use a keyboard key to move it a few degrees per key press.

Next, I added ray tracing of each pixel to pick up all of the other scene objects, and another ray toward the light source to generate shadows. I'm getting around 20M primary rays per second throughput on a 4 core CPU using OpenMP, which makes it interactive. I was getting over 30M rays per second during snow coverage map creation from the previous post, so this isn't quite as fast. I suspect the slowdown is due to cache misses looking up the texture data for each hit point on the scene geometry. I didn't need this extra step when creating the snow coverage map. When the view is zoomed out this far, texture samples are spaced pretty far apart, making the texture access pattern pretty random. Texture aliasing artifacts can be seen in the screenshot below. The small red/yellow and blue/yellow circles are red and blue team AI players.

Ray-traced overhead view of building scene with shadows enabled in gameplay mode.

You can see some shadows cast by the buildings and trees here. I'm simply darkening the pixels in the overhead image when the sun is not visible from the primary (vertical) ray hit location. Shadows are enabled by default, but they add additional runtime. Shadow rays are faster to compute than primary rays because the search can terminate as soon as an intersection is found. However, they're less coherent, especially near complex surfaces, and that adds to the query time.

The results remind me of old sprite-based top down PC games and some of the simple phone/tablet games I've played. Here is another screenshot, this time of the house scene. The trees cast shadows on the ground and the rest of the scene. This scene uses a custom heightmap which is not drawn in map mode, so all terrain is green.

Ray-traced overhead view of house scene with shadows enabled in gameplay mode.

Overall, this was a fun side project. It doesn't look as good as what I could have drawn on the GPU, and it's much slower to compute on the CPU, but it does show off 3DWorld's ray-tracing support. It's just a different way to approach the problem. There are other benefits, such as improved ability to debug the collision volumes, ray tracing code, object placement, etc.

Rain and Snow Progress

The past few weeks of 3DWorld development have mostly been spent improving rain and snow effects. I have added several new features, including dynamic wetness/drying effects with puddles, realtime snow accumulation and coverage, ice, and snow footprints. I'll briefly cover each of these below.

Puddles

I continued implementing features from this blog post, and the next item up was puddles. I wanted to have puddles generated procedurally rather than having to place/paint them all by hand. 3DWorld is all about procedural generation and automating as much of the level design as possible. It's too much work to add every puddle by hand to every horizontal surface of the scene, for all of my test scenes. I don't own any 3D modeling tools and I only use free image editing tools such as Gimp.

Normally, puddles are formed in low areas of the ground, but most of the man-made (non-natural) surfaces in 3DWorld such as concrete floors are modeled as flat planes. The normal maps are too regular and finely detailed to use for puddle depth, so some other tiny invisible adjustment to surface height was required. It seemed like this was another place where I could apply Perlin noise, and as expected, it worked pretty well.

The wet/dry/rain cycle works like this:

1. The ground is dry
2. Rain starts, making mostly horizontal surfaces that are open to the sky uniformly wet over time
3. Rain stops, and the ground dries non-uniformly, leaving puddles
4. After some time has passed, the ground becomes dry again (same as 1.)

There are three states of rain surfaces: wet, dry, and full of puddles. These are implemented with shader flags that control how the surface is rendered, affecting diffuse albedo, specular intensity, specular gloss, and reflectivity (Fresnel term). Here is a screenshot of some procedural puddles from state 3 generated on the ground after the rain has stopped.

Procedural puddles on the ground as the concrete is drying out after a heavy rainstorm. Wet surfaces are reflective.

And here is a YouTube video showing the transition from dry ground, to wet reflective ground, to puddles, and back to dry ground. Oh, and yes, I finally added procedurally generated palm trees.

Realtime Snow Accumulation

My previous post was on volumetric/3D snow accumulation. That system produced good results, but required dropping a very large number (1 billion) of snowflakes from the sky and recording where each of them landed. This works well for a static scene where a one-time preprocessing step is acceptable. But I wanted another solution for faster snow accumulation that could be used with 3DWorld's realtime, dynamic weather system.

I decided to add support for a thin dusting of snow that "paints" upward oriented surfaces a white color. This isn't true volumetric snow with depth, but it still produces a nice snowy scene effect. The most difficult problem was determining which parts of each surface receive snow. Outdoor areas only please - we don't want snow accumulating in the lobby, offices, or basement!

My first attempt was to cast rays through scene vertices and tag a surface as "outdoors" if any rays reach the sky, and "indoors" otherwise. This worked somewhat, and was good enough to use for the rain wet mask, but wasn't quite right for snow. It left areas under objects such as benches snowy because there was a single large polygon covering a large surface area. It looked correct to have wet areas under cover during rain because rainwater will flow across flat surfaces over time, but snow doesn't do this.

The next thing I tried was closer to how shadow mapping works. From what I understand, this is how other games mask snowy and wet areas. The scene is drawn in a "shadow" pass where the objects are projected down, or in the direction of the falling snow if there is wind. This is normally done on the GPU, but that would have produced aliasing and other artifacts. Since shadow mapping tends to sample a pixel once, this means that a small/thin object such as a flagpole or lamppost that happened to fall in a bad spot near the pixel center would leave a hole in the snow mask. Have you ever seen a flagpole that blocks the snow from accumulating around it's base? It looks very odd and unnatural.

Instead of rendering the scene to a snow shadow map, I compute the coverage map on the CPU. A group of rays are traced through a pixel in a 3x3 grid pattern, and all ray intersections with the scene are determined. The furthest/deepest distance is used as the snow depth for that grid position (pixel). This way, a thin object such as a flagpole may block some of the rays, but won't block all 9 of them. It took a bit of further experimenting, but in the end I was able to remove all of the incorrect holes in the snow mask. I also tried adding random noise to the samples, which helped remove some of the artifacts from the snow mask, but also added noise to the smooth edges of snow around building walls and overhangs. So in the end I used a simple fixed grid.

The snow coverage mask is sent to the GPU and used in the fragment shader for per-pixel snow masking. The (x,y) position of the surface to be drawn is used to look up the depth (z) value in the snow mask, which tells the GPU how deep the snow penetrates the scene in z at that position. If the surface has a z-value below this point, it's underneath some other surface that blocks the snow. If its z-value is less than or equal to the snow map depth, it's covered with snow. I addition, there is a soft transition from no snow to full snow based on the difference in depth/z between the snow mask and the surface. Finally, I used the slope of the surface (technically the z-component of the normal vector) to blend snow density from 1.0 for horizontal surfaces to 0.0 for nearly vertical surfaces.

The advantage of this approach is that I was able to produce a 256x256 pixel 2D snow coverage mask using only 256*256*3*3 = ~600K samples. Compare this to 1 billion samples required for smooth snow in the 3D volumetric case. Using spatial coherency of the rays to trace the 3x3 rays as a group, the 3DWorld ray tracing system can achieve around 35M rays per second on 4 CPU cores, which translates into a mere 17ms of runtime. That's quite a bit faster than the ~20 min. required for volumetric snow accumulation! This algorithm allows the snow coverage mask to be recomputed in realtime as the scene is modified and the weather changes. As a bonus, no precomputed snow coverage files need to be stored on disk and loaded.

Below is a screenshot showing the results of realtime dynamic show accumulation. Note that only areas of the scene that have an unobstructed view of the sky accumulate snow (except for the grass, which doesn't use the snow mask).

Realtime procedural snow accumulation in the office building scene.

Here is a close-up view of the snow on objects of various shapes. Note that there is no snow under the trees and benches.

Close-up view of snow accumulating on various objects based on a generated sky coverage mask.

I also recorded a video showing how snow accumulates during a heavy snowfall.

Ice

I decided to implement the mixing of rain and snow to produce ice. The ice effect is somewhat different depending on which comes first. Rain on top of snow melts the snow, but snow on top of rain forms a layer of reflective ice on horizontal surfaces. Angled surfaces accumulate less water and eventually turn to pure snow. It looks like this:

Rain mixed with snow produces a layer of reflective ice on horizontal surfaces. Angled surfaces are more snowy.

Note that the water in the fountain has been frozen and covered with a layer of snow.


Snow Footprints

The day after I published the previous blog post, someone asked me if players left footprints in the snow. What a great idea! It wasn't even that hard to add, though I went back and tweaked the parameters several times. I had to define the player stride, foot width, foot spacing, foot height/crush depth, etc. to get realistically shaped and placed footprints. The numbers don't quite work out for my spherical "smiley face" character model, but it's close enough. Both the user/player and AI controlled players leave footprints in deep snow. Here is a screenshot showing footprints in the snow after I walked around for a while. The gaps in the footsteps where the places where I jumped (which doesn't leave footprints). If you look closely, you can see that the footprints are actually changes applied to the snow mesh rather than decals because that footprint on the very edge on the top of the wall leaves a small gap in the shadow on the snow below.

Footprints created using mesh deformation when the player and AI characters walk in the snow.

I made a Fraps video of a walk in the snow, complete with footstep sounds. The footsteps are a bit too fast, so they sound more like I'm running. Note that snow and concrete have different footstep sounds. I'll probably add a larger variety of footsteps and other sounds in the future.

That's all for now. I wonder what other interesting weather effects I can add?

Simulation and Rendering of Snow

As promised, here is my post on snow simulation and rendering in 3DWorld. Last month, before the holidays, I was working on rain. In fact, I'm still working on rain. I traveled back to Pittsburgh, PA to visit my parents for two weeks and expected to see snow, but there was only rain (except for the last day). It's also raining here in California. The weather really isn't motivating me to work on snow. I'll just discuss the original snow system for various modes, since I think snow works pretty well already.

There are several aspects to snow in a 3D game engine. First is drawing all of the snowflakes as they fall through the air. Second, the engine needs to deal with adding snow to the ground/terrain texture so that snow covered peaks are white. Third, snow should be drawn as actual 3D geometry when it's deep and near the player. The challenging part is figuring out where snow will actually accumulate, given the terrain, scene geometry, weather factors (temperature, wind, etc.), and snow depth. This is what I call snow "simulation".


Snow Ground Coverage

First, I will compare 3DWorld's automatic snow map generation based on altitude and terrain slope to a reference photograph. Higher altitudes are cooler and therefore have more precipitation in the form of snow (as opposed to rain). Also, snow does not remain on steep slopes; it falls down the slope, leaving exposed rock behind. Here is a photograph of Mount Rainier covered in a layer of snow.

Photograph of Mount Rainier viewed from the southwest, from Wikipedia.

Here is 3DWorld's realtime rendering of Mount Rainier from a high resolution heightmap, where snow cover has been calculated from the terrain.

Mount Rainier drawn by 3DWorld using only a heightmap. Snow coverage was simulated using terrain slope and altitude.

Note that I did this image comparison after setting the snow parameters in 3DWorld and didn't go back to tweak anything. It happened to turn out very close to the actual snow coverage, validating the height and slope model. I cheated a bit with the clouds though, and set the average cloud height experimentally based on the reference image. The trees aren't really to scale (I wasn't too interested in fixing them here). Also, 3DWorld's mountain appears to be taller because there was no reference to convert the height map values from the image I found online to real height values, so I took a guess at it and overestimated. Oh, and someone pointed out that I'm missing the glacier on the top of Mount Rainier - I haven't gotten to simulating glaciers yet.


Falling Snowflakes

3DWorld has two forms of physical falling snowflakes. The original snowflakes are all dynamic physics objects that use the same game object class as balls, rockets, shell casings, etc. They have very accurate physical models that include collisions with the exact scene meshes, gravity, friction, wind resistance, and thermal effects (melting in high temperature areas). Unfortunately, it takes a large amount of CPU work to simulate tens of thousands of snowflakes this way. I had to find a faster alternative.

The second and newer snowflake model is more GPU based. Each snowflake is a single {x,y,z} point rendered as a point sprite on the GPU. The physics and collision detection is still performed on the CPU, but it's simpler and more efficient. For example, the velocity of each snowflake is not even tracked - all snowflakes have the same velocity based on gravity and wind. Collision detection is performed coarsely against a 3D voxel (volume grid) model of the scene, and the time consuming detailed collision detection is only performed when the voxel test returns a possible collision. Snowflakes are destroyed and respawned high in the sky when they collide, rather than bouncing or sticking as in the slower model. This approach scales up to 100K snowflakes and looks almost indistinguishable from the more accurate model. You can really only tell the difference when running physics in slow motion or freezing the frame. The lighting computation in the shaders is all the same as well.

In the end, I used a mixture of around 1:5 of old model vs. new model snowflakes. That way, most of the snowflakes are fast, but there are also some that actually stick to the scene objects. Here is what a scene looks like with 40K snowflakes. Can you tell which flakes use which model? I think the only way to tell would be to visually track an individual snowflake's path though the scene.

Snow falling in the office building scene and beginning to accumulate in the grass.

The white spots on the grassy ground mesh are the beginning of snow accumulation. When an old model snowflake collides with the terrain, it colors the terrain texture white in that area. After several minutes of snowfall the ground will be mostly white, except for the grass itself, which remains green. The office building surfaces are unaffected. There is no realtime snow accumulation - that is more expensive (in terms of CPU time) and done as a preprocessing step (see below).

One of the more challenging aspects of snow is drawing individual close-up snowflakes. Sure, snow looks fine in the distance when drawn with white points. But when you actually look closely at a snowflake, it's not a white dot. It has real structure. In fact, every snowflake has a unique fractal crystal pattern. I decided to use a single snowflake texture for nearby snowflakes. It looks nice, but it's not very realistic - more of a cartoon effect. Here is a screenshot of a snowflake on the player's eye/"camera lens".

A snowflake falls in front of the camera. There are 40K total snowflakes falling from the sky.

Snow Accumulation

Next, I will explain how snow accumulation works in 3DWorld. To get an immersive snowy scene, snow needs to have real depth, and needs to cover every object. Just painting it onto the terrain mesh texture doesn't work. But adding a constant layer of snow on top of every horizontal surface doesn't look right either - snow depth changes with the slope and geometry of the objects nearby.

Rather than trying to come up with a physical model for snow accumulation, I decide to approach it using a simulation of random falling snowflakes. One billion snowflakes are dropped from the sky with uniform distribution over the scene. The path of each snowflake is traced, taking into account wind and other factors, until a collision with the scene is found. Then, depending on the collision normal/direction, slope of the surface, friction, and wind, the snowflake either sticks or bounces off and continues falling. Snowflakes that stick are added to a sparse 3D accumulation volume implemented with a hash map. For the house scene below, I used a 1000x1000x500 volume. The actual number of nonempty cells (with at least one sticking snowflake) is around 4M, which can easily be stored in memory. The entire process takes around 20 min. on a 4 core CPU.

This first step produces a volume map of snow accumulation. The next step is to convert this volume to an actual surface. First we need to determine how much volume each of the 1B snowflakes contributes. Given a snowflake volume V, and scene size W by H, and voxel size X by Y by Z (=1000x1000x500 in this case):
volume = width*height*depth
V = voxel_width*voxel_height*delta_z
delta_z = V/(voxel_width*voxel_height) = V/[(W/X)*(H/Y)]

As you can see, the final snow depth is a linear function of the volume of a snowflake. Changing the value of V will vary the snow coverage from a light dusting to heavy accumulation to snow that covers the entire scene. The conversion from snow volume to a surface only takes about two seconds. Once the expensive (20 min.) simulation of 1B falling snowflakes is performed we can vary the snow depth interactively, or save the snow map and use it for different scenes with the same geometry but different snow coverage. 3DWorld saves the snow accumulation in a very compact, sparse structure that is only 13MB for the house scene.

The process of converting snow volume to a 3D surface works by extracting contiguous rows of voxels into triangle strips, then collecting adjacent triangle strips into mesh sections. The height (z-value) of each vertex in the triangle strip is determined by the amount of snow accumulated in that voxel in the snow map. If all of the nearby voxels in an area have similar volumes, this produces a flat snow cover (for example roads). If the volumes vary significantly between adjacent voxels, this produces rough snow (for example on trees). Any vertical (Z) stack of nonzero volume voxels will merge into a deep snowdrift, where the Z height of the mesh is taken from the sum of the volumes. This models snow stacked against the side of a house or tree. Areas that have stair-step patterns of filled voxels where adjacent nonempty voxels are one Z column above or below each other will produce steep triangle strips (for example on sloped roofs). This approach is able to extract multiple overlapping surfaces at different heights, such as the snow on top of the fence and the snow below the fence at the same {x,y} position. Here is what this algorithm produces for moderate snow depth on the house scene.

Snowy house scene with falling snow and heavy snow accumulation on the ground and scene collision objects.

This approach works very well for smooth surfaces, which account for most of the scene. The areas that need improvement are near scene objects that are smaller than a few voxels in width or size. In this case, the voxel snow volumes don't have enough resolution to accurately map a triangle mesh surface onto the small objects. This is an example of the Nyquist Samping Theorem. The result is single triangle patches of snow that in some cases are larger than the objects they are resting on. Using a higher resolution volume would help with this issue, though it would consume more runtime, memory, and disk space.


Snow Surface Rendering

Finally, I will discuss how the snow surface is rendered in 3DWorld. It's not all that difficult. The meshes that were formed from triangle strips are grouped together into vertex buffer objects (VBOs) and stored on the GPU for improved performance. They are static for the scene. [Well, there is dynamic snow deformation due to collisions that can be enabled, but that's a topic for another post.] Snow is then drawn using the normal rendering pipeline for 3DWorld with shadow mapping, normal mapping, and sharp specular lighting to get those nice bright highlights. The strength of the normal mapping is reduced for distant snow to avoid specular aliasing and texture tiling artifacts. Here is what the surface of the snow looks like up close, when looking toward the sun.

Snow drawn as a 3D geometry layer on the ground. Snow depth is based on simulating 1 billion falling snowflakes with collisions and tracking volume accumulation.

The snow surface itself casts shadows on the rest of the scene, which requires it to be drawn without texturing or normal mapping a second time to create the shadow map. When there are no dynamic objects, the snow shadows are cached between frames to improve frame rate. Snow rendering is surprisingly fast, considering the snow mesh contains nearly 4M vertices. On my new Nvidia card the addition of snow decreases the framerate from 500FPS to 250FPS. This is less frame time than grass (170FPS), so if the snow covers the grass the framerate actually increases!

Update: Here is a screenshot of player footsteps in the snow, simulated while the player is walking based on stride, foot length, foot spacing, player speed, and player direction.

Player footsteps in the snow.

Update2 (10/16/20): Here is snow accumulation in the house scene with a wind effect added. Snow has been pushed against the near wall of the house, and there's a gap in the snow on the left side of the house.

Snow accumulation with a strong wind coming from the top right direction.