Postprocessing Effects

I've been experimenting with full-screen postprocessing effects in 3DWorld using the fragment shader on the GPU. The shader has access to the color and depth buffers from the current frame, and overwrites the color buffer before displaying it. Some of the effects worked out well, but others did not. Here is a list of what effects have been added (in no particular order). This is a long post, but it's mostly images and videos and not too much text.

Fog
This is a very simple and common shader effect. It's also very cheap, so it doesn't affect the framerate much. However, I'm not really using it as a full-screen postprocessing effect. Instead, it's added as a postprocessing effect at the end of the regular draw shaders for all objects.

I'm doing more than the standard fog here, where the standard fog technique uses the depth (distance from the camera) of each pixel to select the fog color from no fog near the camera to full gray fog in the far distance. The 3DWorld fog is actually computed by ray marching a few steps through the scene and using the precomputed indirect lighting at each step to determine the fog color. Brightly lit areas have light gray fog, and dark shadowed areas have black fog. That way, the basement has black fog rather than the normal white-ish outdoor fog. For performance reasons, I'm doing a few ray steps, and only for some object types. It looks much better (with smoother color transitions) with more steps, but it's slower. Here are two screenshots for comparison.

Without Fog

With fog

God Rays
God rays, or crepuscular rays, are shafts of sunlight that hit small particles in the air, producing a glowing effect. Many games use God rays to increase the realism of bright direct lighting. 3DWorld uses a common approach of sampling the color buffer, ray marching from the sun position outward in screen space, and using the depth buffer to occlude the smear of color. This gives bright lines that radiate outward from the sun between other objects that block it. I have it working for scene objects, trees, and other solid objects, but not clouds (which don't contribute to the depth buffer). Here is another pair of screenshots showing the effect.

Without God rays

With God rays

Here is an image of God rays and fog in tiled terrain mode, looking through some trees. Note how the sunlight shines though the backs of the leaves as well. Real leaves are partially transparent to light.

A foggy morning, with God rays showing through the tree leaves. Also note the sunlight illuminating the backs of the leaves.

Another view. This time I remembered to turn off the onscreen display text.

Bloom
Bloom is a type of HDR (High Dynamic Range) postprocessing effect that makes bright objects appear even brighter than maximum monitor white by bleeding their color into surrounding pixels. If a light source is only a few pixels wide, the addition of a halo around it makes it seem brighter. This works well for small light sources that are only a few pixels wide, where the max white RGB value of {255, 255, 255} is not enough.

I'm using a 9x9 X/Y separable filter kernel (also used by the DOF and underwater blur effects). The color blur is done independently in X and Y using a 9x1 filter and a 1x9 filter. This requires 2*9 = 18 color texture lookups rather than 9x9 = 81. This is much faster and looks almost as good.

I think the bloom effect works well for small local light sources, but I don't like the effect on the sun and clouds. It's not clear how the fragment/pixel shader knows which pixels are sun vs. other light sources. Right now it's only testing for color intensity greater than 75%. Here are two comparison screenshots.

Without bloom

With bloom. The sun and sky are very bright! The courtyard lights are also bright. (but why are they on in the daytime?)

Heat Waves
I added screen space vertical heat waves as a postprocessing effect when the player is near a heat source such as fire. This is just a simple vertical wavy line effect on the entire screen, with decreasing wave amplitude from bottom to top. The shader doesn't actually know where the fire is on the screen, or if it's even visible in the view frustum. All it can do is add the effect or not. Here you can also see the billboard smoke particles effect. Smoke is colored based on lighting and shadows, so the smoke near the fire appears more reddish.

Heat waves rising from the fires. Don't get too close or you'll be burned!

Depth of Field
Depth of Field (DOF) is a common effect used in many games. The general idea is that the player is focusing on some particular object at a particular distance from the eye/camera. Objects both nearer and further than the object are out of focus and appear blurry. This simulates focusing of the human eye and of some types of camera lenses. Personally, I don't like this effect in games. I don't like anything that makes the scene look blurry - better to have sharp, crisp images. I added this effect to 3DWorld because it looked neat, but it's not used in-game. Here is what it looks like - click on the image for a larger view.

Depth of Field test image. The camera is focusing on the lamp in the center of the screen (in the white circle).

Here is a DOF video. I hope the blurriness of the video due to image compression artifacts doesn't mask the effect.

Underwater Blur and Distortion
Underwater blur is used to increase the realism when the player is underwater, and make it more obvious when the water boundary is crossed by the eye/camera. A wavy distortion is also added when the player is out of oxygen and drowning to simulate dizziness. When you see this effect, you know to get out of the water fast! I originally came across this effect through random experimentation, and I wanted to use it for something. It would make a great drunk effect if I ever add alcoholic drinks to 3DWorld. Here is an example of this rather strange effect.

Underwater drowning effect. Or maybe drunken effect? Watching this sure makes me dizzy.

Screen Space Ambient Occlusion
Not all of my postprocessing effects experiments were successful. SSAO was actually my first attempt at a postprocessing shader, but I should have started with something simpler. It works by calculating light occlusion in screen space using the depth and normal information, and then darkening screen pixels based on the amount of occlusion. That way, small folds and crevices appear darker than open areas. It doesn't work perfectly: for example, occluders that are off-screen or blocked by other objects can't be used.

This effect never really worked correctly in 3DWorld, for a number of reasons. First, 3DWorld uses a forward rendering pipeline, not a deferred rendering pipeline, so there is no normal buffer. Without object normals, it's impossible to tell front faces from back faces and which faces point toward each other. This results in incorrect darkening of areas that aren't actually occluded from the light source.

Second, SSAO doesn't really work with transparent objects. If the transparent object such as a glass window writes to the depth buffer, the front of it is darkened as if it was an opaque wall, which looks wrong. If it doesn't write to the depth buffer, the ambient occlusion isn't blended correctly and the dark area appears in front of the window, which also looks wrong. Here is a pair of screenshots with and without SSAO. The effect is subtle, so you probably need to click on the bottom image to get a higher resolution view.

Without SSAO

With SSAO. Not the intended effect. The color banding, black outlines from the transparent windows, false occlusion, etc. ruin the realism of the scene. Oh well, it was worth a try.

I'll see if I can think up any additional full-screen postprocessing effects.

Museum Scene

I found a nice 3D museum model online, so I decided to post some screenshots of how it looks rendered in 3DWorld. The model was taken from Challenge #17 from the challenge page on 3DRender.com. Note that I didn't actually participate in this challenge. I also spent little time preparing. I don't actually have any real image editing or 3D modeling software, and I wanted to render this in realtime, so it's not the highest quality rendering and there are no fancy special effects. However, I still think it looks good.

This museum scene contains around 750K triangles. 3DWorld can handle scenes up to around 12M triangles, so I could have 16 of these museums in my scene and still draw it in realtime. The original file was untextured, with no normals or texture coordinates. I had to use textures that I found online and create the normal maps using a trial version of CrazyBump. The textures are applied using standard {X, Y, Z} triplanar texturing where the texture coordinates are determined by the dominant direction of the normal. It's difficult to generate automatic texture coordinates for scenes with curved geometry without using a 3rd party tool. Vertex normals are generated from the face normals by 3DWorld. Some of the triangle faces in the model have the wrong clockwise vs. CCW winding order, which causes black spots and smeared textures, but it's not too noticeable.

There are several light sources in this scene:

• Sun direct and indirect lighting with shadows
• Sky/cloud indirect lighting
• Local diffuse + indirect indoor light sources (2 yellow lanterns and 4 blue ceiling lights)

There is a one time lighting precomputation of around 10 min., and after that the scene can be drawn at around 100 FPS (frames per second). It takes 10 seconds or so to update the sun indirect lighting component when the sun position is changed. Here are some screenshots of the museum scene.

Museum scene from the front stairs.

Museum scene from the ground floor in the back.

Museum scene from above near the ceiling.

This is a fully collision and physics enabled scene in 3DWorld. The player can walk around and collide with the geometry, shoot things, and play in game mode against the smiley AIs. There are a few holes in the floor that the player can fall though, but I think I mostly fixed those problems.

That's it for now. I'll write something longer next week when I have more time.

Tree, Plant, and Grass Leaf Wind

It's been awhile since my last post, but I haven't really been working on much that I can show in screenshots and videos. Most of the effort has been spent on continued improvements to movable object physics and related tasks. I had it working well enough to show some videos last month, but it took a lot more effort to make the system stable and fix all of the strange bugs, including the crazy large water splash when underwater boxes were destroyed. I'm working on object rotational physics and that sort of thing now, but it's difficult to get right, and I don't have anything to show yet. I may end up using a third party physics library such as Bullet for the more complex effects.

I also spent a few hours improving the 3DWorld leaf wind effect and making it work on more vegetation types, so I'll make this the topic of today's post. This post won't be as long as the previous one, which hopefully means that I'll have time to make another post soon. I already had a leaf wind system for trees that was entirely on the CPU, and it's always bothered me that this method is slow and I find myself disabling it at times to improve the frame rate. In 2015 everyone seems to be doing leaf wind on the GPU rather than the CPU, so I decided to try this approach. I've actually been thinking about and experimenting with GPU wind for years, but I was never able to figure out how to update the leaf normals for lighting as the leaf moves in the wind. The problem is that the vertex shader can normally only see one vertex at a time, and it can't determine the orientation (rotation axis and angle) of the leaf from a single vertex. The leaf needs to bend/rotate at the point where it's connected to the tree branch, a point that the vertex shader doesn't know. Sure, I could pass this extra info as additional vertex data, but that takes more GPU memory and hurts the frame rate. I could use a geometry shader, but that turned out to be very complex and also slow.

A few weeks ago I read an online article on leaf wind. The author ran into the same problem with updating the leaf normals, but he went ahead and implemented it anyway with the normals and lighting remaining constant. He claimed that it looked just fine this way - and I agreed! So all this time I had been worried about what was really a minor issue. The solution: do something simple per-leaf that doesn't need to know about the leaf orientation and just use the original normals. I used a simple time-varying sum of two sine waves and that looked pretty good. Even with fixed normals, the lighting can still be somewhat dynamic. Shadows and indirect lighting are computed per pixel, and don't depend (much) on the normal, so these components of the lighting will still change dynamically as the leaves move in the wind.

I still had to deal with the leaf attachment point to the branch. The leaf tip should move, but not the base part where the stem attaches to the tree branch. How can the shader know which vertex is at the tip vs. the base? It can use the texture coordinate, which is 0.0 and the base and 1.0 at the tip. This data is already available since it's used to texture the leaf. The next problem is how to keep the leaf size constant by moving both vertices of the tip (the two vertices that form the far edge of the leaf quad/rectangle near the tip). I solved this by using the world space coordinates of the leaf to determine the magnitude of the wind displacement. Since leaves are small, the coordinate values of their tip vertices are close together, making their wind displacement values similar, therefore keeping the distance between those points relatively constant.

This system worked so well that I ended up using it for the small branches and leaves/needles of pine trees, which previously had no wind effects. Then I enabled it for plant leaves. I was never quite able to get the wind effect to work for the berries on some plant types, so I simply disabled this effect on those plants. Grass was already using a similar wind system, so I left that code alone. [Grass is easier since each blade is drawn as a triangle, where only the tip vertex moves, and it's a lot simpler to animate a single vertex rather than an edge formed from two vertices for leaves. Also, grass blades tend to have a more uniform length and orientation, so the vertex shader can guess at the positions of the other two base vertices.]

Here is a screenshot of some vegetation moving in the wind. Oh, right, you can't actually see it moving. Well, it's still a pretty screenshot.

Trees, plants, and grass moving in the wind. This is just a high resolution image from the video below.

Here is a video of the same scene. You can see the tree leaves, plant leaves, flowers, and grass blades all moving in the wind. The wind speed and direction can be interactively changed with hotkeys and an onscreen slider UI widget (not shown). The video starts with a small amount of wind and gradually increases wind intensity to an extreme value. I had to choose a strange camera angle in the corner of the scene to get the pine trees, deciduous trees, plants, and grass all in the same view.

Here is another video showing that wind works on all vegetation in infinite tiled terrain mode as well. This scene with it's high detail trees ran at only ~50 FPS (frames per second) with CPU wind, but runs at ~90 FPS with GPU wind. In fact, the wind computation time has almost no effect on frame rate. In addition, wind is applied even to distant trees, rather than being limited to only nearby trees in the old mode. It may be hard to see the wind effect in this video, especially since I couldn't stand still and had to run around at high speed just to show off that it was possible. The max speed in tiled terrain mode is very high and when the player is on the ground everything is a blur. The minor lag near the end when I went in the water is probably due to loading the water sound.

Right, back to the wind. I'll note that the new wind system doesn't completely replace the old mode. I still need CPU mode for player interaction with leaves. For example, player collision with trees moves the leaves, leaves can be shot off using projectile weapons, etc. Oh, all right, I'll add a video of that too. I guess every post must now contain at least one video of me shooting or otherwise destroying something. Keep in mind that the particle effects are only a placeholder for better particle effects that I'll likely show off in some future post.

Did you notice that I added new sounds such as the "click" when switching weapons? I'll try to continue to add more sounds and other effects as time allows.

That's about all I have to say about leaf wind and related topics. Next time I'll probably make a post about full-screen postprocessing effects, unless something else interesting comes up. I was considering making a post about snow accumulation and rendering, but I think I'll save that for winter time - not that we actually get snow here in CA.

Movable Objects Physics

This time I'm not going to talk about graphics, at least not very much. I'm going to talk primarily about physics - in particular, the new movable objects I have been working on over the past few weeks. This post is going to be less technical than usual, since I'm not going to get into the details of how I implemented object physics. I'm just going to explain what the system can do and show lots of short videos. It's just too hard to capture physics in screenshots.

But, for those of you who like to see pretty screenshots, here is the latest version of my trees in tiled terrain mode. They look a bit different from the last tree post a few months back, and in my opinion they look better. I increased the lengths of the smallest level branches so that the leaves are more spread out, rather than being clustered together.

The latest version of tile terrain mode trees, now with more leaf detail.

Okay, back to movable object physics. Movable objects are a new type of physics object in 3DWorld that are somewhere between standard static objects and dynamic objects. They're really pretty close to the way platforms (such as elevators, crushers, and moving doors) are implemented. Here are the basic types of objects supported by 3DWorld:

• Static scene objects. Most objects are of this type, since they are the most efficient to process: No dynamic updates, no physics, can use a static draw buffer, precomputed lighting, etc.
• Destroyable objects: Static objects that don't move but can be destroyed (removed)
• Platform objects: Move along a predetermined path when activated, no physics, some precomputation is still possible.
• Movable objects: Movement controlled by the player, AI, gravity, and other movable and platform objects. Movement is generally sparse/rare so physics is only applied when needed, but can't precompute anything.
• Dynamic objects: Objects that typically have a limited lifetime and are assumed to move each frame (players, weapons, projectiles, pickup items, fragments, particles, etc.) Limited to collision spheres only for simple and efficient collision/physics.

Movable objects are more general than dynamic objects since they can have any standard collision shape, but they are less efficient. 3DWorld can efficiently handle thousands of dynamic objects but probably only a few hundred to a thousand movable objects.

The supported basic primitive shapes for movable objects are {cube, sphere, cylinder, cone, truncated cone, capsule, polygon, ramp, and extruded (cookie cutter) polygon}. Yes, I have implemented exact collision detection between all of these shape classes, some 30 functions in total. It's not entirely complete and stable for all combinations, but it works well enough. This has some advantages over the standard polygon meshes used in most games:

• More efficient (an analytical sphere is much faster than one formed from a thousand triangles)
• Curved surfaces can be rendered at near pixel accuracy using a smooth LOD transition
• Seamless texture coordinates can be auto generated with simple code in most cases
• Collision can be exact, and generally involves solving some equations rather than iteration
• Objects have true volume, which means their mass can be calculated for physics
• Objects always form a closed surface and inside vs. outside is clear (except for polygons)

I have implemented quite a few "effects" where movable objects interact with the rest of the world in interesting ways. Most of these can be classified as physics or collision effects. They have been implemented using object-object collision detection and almost nothing else that must be special cased on the object shape type.

One of the important effects I wanted to have was stacking. Movable objects can be stacked on top of each other and the whole stack moved around. Stacks can be created statically in the scene object file, or dynamically by dropping objects off ledges onto each other. Stacks can also be pushed over by standing on something to get at the top object on the stack. I also added horizontal stacking (like a train of objects) after recording the videos so you won't see them in action yet.

The remainder of this blog post consists of 11 short videos of movable objects in action. I feel that this topic is much easier to demonstrate using video rather than static images and walls of text like most of my previous posts. Since the free version of Fraps limits video recording to 10 seconds, none of them are longer than that, which is why I had to create so many. Keep in mind that there was no special setup for making these videos, I could have done all of these events in one 3DWorld run. Some of the videos have sound, in particular the ones involving weapons. I really need to add more sound effects for pushing, falling, colliding, etc. [Update: I did add more sounds! But I'm not going to re-record all of these videos.]

Ramp 1
This video demonstrates sliding a stack of two wooden crates (cubes) up and down a ramp and over the grass. Note how the crates crush the grass in their path. In addition, it shows how a sphere and a cylinder can be pushed over the edge of the wall. If you look closely, the sphere smoothly rolls over the wall (though the texture doesn't move to show that it's rolling).

Ramp 2
The previous ramp video was truncated at the 10s mark, so I had to split it into two parts. This time I'm sliding a crate up a ramp that's also a movable object. When I push them together, both the crate and ramp slide on the ground, until the ramp hits the wall of the fountain and stops. At this point, I can finish pushing the crate up the ramp and into the water. Then I push a wooden barrel (horizontal cylinder) up the ramp and into the water. The "boing" sound is the placeholder sound for player jumping.

Elevator 1
Here I push a stack of two metal boxes (Borg cubes!) and a brick cube into the elevator, which goes up and then down. On the way down I push the metal cubes off at one floor and then push the bricks off the back of the elevator.

Elevator 2
The first elevator video doesn't use real physics. Can you spot the error? The cubes were directly controlled by the elevator's velocity, which is a hack. In the next video I have fixed the problem so that the cubes are in free fall when the elevator is going down. The video starts with the elevator stopped on the top floor. You'll notice a small jump in each cube when the elevator starts its descent - what is that? The jumps are due to the unrealistic constant velocity of the elevator. When it starts moving at a fixed velocity, it has an infinite downward acceleration. Since the cubes fall due to true gravity, they can't accelerate to the elevator's velocity instantly. Instead, they accelerate due to gravity under free fall from rest and only collide with the elevator when they exceed its velocity and catch up. The bottom cube has to catch up to the elevator, and the top cube has to catch up to the bottom one. After that, the cubes descend with the elevator at a constant velocity. I'm not entirely sure it's correct now, but it looks plausible for an elevator with infinite acceleration.

Falling Bricks
This video demonstrates falling object and stacked object physics. It starts with a brick cube resting on a wooden board. I shoot the board with the M16, shattering it, and the bricks fall to the ground below. Then I accidentally fall off the edge when trying to look at the bricks and land with a bloody splat. Also note the shell casings and the bullet holes that stay on the bricks as they fall. Since bullet hole decals are independent world entities, it took a bit of extra trickery to bind them to a known position on the object so that they track the object's position as it falls. Oh, and yes, I finally gave the M16 some recoil, which is why I keep falling off of things when I use it.

Ball Push
Here I push a rolling ball with a glass cube, then push it off the edge with a movable wooden ramp. The ball is a dynamic object, rather than a movable static object, so the rolling physics/texture rotation works with it. See how the glass cube isn't a uniform transparency? The glass is modeled as a scattering medium with real optical thickness, where each pixel's ray is intersected with the cube and the light attenuation is accumulated along the ray's optical path through the cube. Windows work the same way.

Shadows
In this video you can see that the local light source shadows are updated as the boxes are pushed on the floor. The shadows are only recreated when a dynamic object moves, to save CPU and GPU time. Technically, the movable boxes aren't dynamic objects, so they don't trigger the shadow updates themselves. But, the player is a dynamic object, and moving around near the light will trigger shadow updates. Since only the player can move objects (currently), everything should work well enough.

Indirect Lighting
3DWorld uses precomputed indirect lighting stored in a 3D volume texture. This allows objects that weren't present during lighting computation to still receive light by looking up the lighting texture data in the shader. However, the objects can't actually modify the indirect lighting in an efficient way. This video shows a white cube being pushed around through various lighting conditions, where indirect lighting dominates. The transitions from lit to dark areas are smooth. The hallway in the middle of the video has motion activated blue lights in the ceiling that contribute a direct lighting component as well.

Destroyable Crate
Here is an example of how nearby explosions can destroy moving objects. I push a wooden crate next to an exploding tank, then shoot the tank, causing it to explode and destroy the crate in the process. The shattering glass sound is from one of the lights on the ceiling that is also destroyed.

Floating 1
Physics is fun, especially when it includes water! At least I think so. I have implemented physically-based buoyancy in 3DWorld. Objects with density < 1.0 float, and objects with density > 1.0 sink. Float height and sink rate are functions of density. The video starts with the lake water in the corner of the office building scene frozen as ice, where I (the player) am standing on the ice and all of the objects are resting on the ice. When I increase the temperature using a hotkey, the ice immediately melts and I fall along with the other objects. Some of them float, and some of them sink, depending on their density relative to water (1.0). The floating objects bob up and down with the waves on the surface of the water. The cube object materials are (from left to right in decreasing density):

• Metal, density = 4.0 (typical metal density is ~2-8)
• Marble, density = 2.8
• Glass, density = 2.5
• Brick, density = 2.0
• Wood, density = 0.7 (wood density ranges anywhere from 0.4 to 0.9 so I picked a mid value)
• Styrofoam, density = 0.2 (random guess)

A note on bricks: It seems odd to me that brick has only twice the density of water. I would have thought that bricks are heavier than that, but I guess not. Here is a fun experiment - try picking up a brick that's out of the water, then submerge it in water and try picking it up again. Does it feel like half the weight when it's in water? Maybe the next time you need to move a stack of bricks, you should flood the area first and it will be half the work. Of course then you'll spend extra energy moving yourself through the water, so it might not help that much:)

Floating 2
I forgot to enable the underwater bubbles in the first video, and I didn't want to have to move all of the objects into their correct place and redo the floating/sinking video. So I quickly put them in less orderly positions and recorded a new video that includes bubbles. This time I stacked the Styrofoam onto the bricks so that they separate in the water. If the bricks were on top they would both float, which may not be correct, but the water system treats each object independently and the bricks are technically out of the water. Also note that the underwater effect includes a full screen blur postprocessing pass, which is why the image looks a bit fuzzy.

Teleporter + Fall
This is my favorite video. I push a crate through the lobby teleporter onto the roof, then go through the teleporter myself. I push the crate over a skylight, stand on top, fire the shotgun at the skylight until the glass breaks, then the crate with me on top falls four stories back to the lobby level. The falling damage is nearly enough to kill me, but I survive and get to see the crate with shotgun shell casings sitting on the floor. I think the fall would normally be fatal but for some reason or other the crate reduces the damage. Don't try this at home!

Bonus: Water Fun
Did I say 11 videos? Well, here is #12. It was so much fun creating these videos that I decided to create one that mixed water, destruction, new sounds, and drowning. Here I have a metal box resting on top of a wooden crate on the ice. I push a large wood plank down the stairs and out onto the ice. Then I jump onto it and melt the ice into water. You can see that the metal box floats on top of the crate on the surface of the water. Then I shoot the crate out from under the box with the M16, causing the box to fall into the water. The crate explosion creates a huge wave that's probably a bit excessive, but that's okay. Then I shoot out the plank from under me and fall into the water, taking some damage from the sharp floating wood fragments. Finally, I destroy the metal box, fire some more M16 rounds under the water for good measure (and to show off the bubbles), then drown. That was fun!

Yes, this post is long. But the time taken for you to read it is small compared to the time I took to write it ... which is small compared to the time it took me to implement all of this ... which is small compared to the time it took me to write the thousands of lines of those ~30 primitive object intersection functions. Sure, I could have probably found the code for some of it online - I tried - but it seems like no one can get that sort of code stable and correct in all cases. So many opportunities for divide-by-zeros, so many places where epsilon tolerances are needed. Maybe I should open source that code some day.

Maybe I'll add more videos later. Keep looking for them.

Update: I added water buoyancy physics for two stacked objects that uses their combined mass, the displacement volume of the bottom object, and the water level of the upper object. I'm not sure it's physically correct, but it looks right. A metal box on a styrofoam block now sinks more slowly than the metal by itself. A wooden block on styrofoam floats low in the water. Bricks on styrofoam sink until the bricks go below the water, then sit there with the buoyancy of the styrofoam keeping the bricks from sinking completely. Wood on bricks sinks faster than the bricks alone until the wood goes under the water, at which point they separate and the bricks sink more slowly while the wood floats. I haven't had a chance to make a video of this yet, but I'm sure you can guess what it would look like.

Volumetric Clouds

Awhile back I wrote a system to create and render a volumetric nebula for universe mode. Later I realized that a modified version of that effect also made a good debris/dust cloud for ship explosions. Then I used a more colorful version for teleporter graphics in ground gameplay mode. This week I decided to try this approach for rendering volumetric puffy clouds in infinite tiled terrain mode, and it works! Well, almost, they don't quite look like "normal" clouds from a distance but they look great up close. Plus, the player can fly through them.

Before I show you some pretty screenshots, I need to get some technical details out of the way. I'll try not to make it too bad for the nontechnical readers.

The volume cloud system I'm using isn't actually something I thought up myself. I saw it explained in a video that I found on YouTube through a Google search a few years ago. Unfortunately, I can't seem to find that video again, so I can't link it here. The video described how to create a nebula out of a number of randomly oriented 2D slices containing partially transparent plasma textures created in Photoshop. Since I needed one unique nebula per galaxy, in a game with an infinite number of galaxies, I had to do something a bit different. Instead of manually creating an infinite number of textures in Photoshop (which I don't own), 3DWorld generates the color and alpha (transparency) values directly on the GPU using random 3D noise textures to create ridged Perlin noise. The location of the nebula in space is the seed used to determine the starting and step direction/distance into the noise texture so that each nebula is unique.

Each particle cloud (nebula, etc.) consists of 13 2D quads (squares) that all intersect at their centers. Each quad can be viewed from either direction, so it's actually more like 26 2D billboards. The quad planes are oriented in uniformly distributed directions by taking all {x,y,z} values in {-1,0,1} and folding out the symmetric ones. This gives the following 13 directions before normalization: (1,0,0), (0,1,0), (0,0,1), (1,1,0), (1,0,1), (0,1,1), (-1,1,0), (-1,0,1), (0,-1,1), (1,1,1), (1,-1,-1), (-1,1,-1), (-1,-1,1).

The corners of the quads are attenuated to an alpha of zero (fully transparent) so that they appear as circles with a smooth falloff from opaque to transparent at their outer edges. In addition, quads that are nearly parallel to the view direction are faded to transparent to avoid ugly artifacts from looking at them edge-on. Then the color and alpha channels are modulated by the noise functions to give an interesting, cloud-like effect. The fragment shader evaluates the noise function at a position corresponding to the coordinate of each pixel on the quad in 3D world space. Therefore, the cloud looks correct and has proper parallax when the viewer/camera moves around and through it. That is, as long as the view doesn't pass directly through the center where the quads all intersect, which is a singularity where every quad is viewed edge-on and rendered invisible, causing the cloud to disappear. I guess that's what it's like to be in the eye of the storm.

Here are some of the fun things I can render using this system.

Nebulae
Yes, it can draw a nebula, what a surprise! This system was initially used for drawing nebulae in universe mode. I've shown this image in a previous blog post, and here it is again.

Closeup view of a procedurally generated nebula in universe mode. This was in a previous blog post.

A nebula is drawn using ridged noise (just Perlin noise with some extra math). Okay, if you're curious, here is the math that converts a Perlin noise value 'v' into ridged noise:

v = 2.0*v - 1.0; // map [0,1] range to [-1,1]
v = 1.0 - abs(v); // ridged noise
v = v*v; // square it

Yes, exciting, isn't it? The nebula shader uses two random noise color channels where the colors themselves are also randomly selected (I think it was pink and orange in the screenshot) + one alpha channel. The noise is mostly high frequency and the bounding shape is spherical.

Explosions
Who creates a space game without ship explosions? What is left after a big ship's explosion, anyway? A cloud of glowing dust and particles! This can be drawn with the same technique. Just use lower frequency noise to make it more irregular, make it even more ridged and wispy, use different colors for the interior vs. exterior of the volume, and you get this:

Volumetric debris and dust cloud from blue/white and red/orange ship explosions in universe mode.

Teleporters
That's enough particle clouds in universe mode. It looks good, but I don't want to overuse the effect. Where else can it be used? How about using it for the new teleporters in 3rd person shooter ("ground") mode? Since a teleporter doesn't exist in the real world I can make it look however I want. I decided to make it very bright so that it stands out, adds color to the dull grays of my office building scene, and looks like something that is definitely not natural. The colors and noise are animated as well so that it's clear to the player this thing isn't just a static decoration. It draws attention - this glowing object does something. Teleporters use 4 high frequency ridged Perlin noise channels in the shader: red, blue, orange, and alpha.

Closeup of a teleporter gameplay entity. The dynamic, animated, colored cloud casts light on the ground.

Here is a short video of the teleporter in action, where you can see the animated colored volume and how it actually moves the player to a different location in the scene. Note that teleporters don't just work with the player: they work with enemies, items, projectiles, particles, and any other type of dynamic game object. It's fun to throw grenades into the teleporter trying to hit someone you can't see on the other side. Just make sure you count to 5 before walking into it yourself. Sorry, I didn't record a video of this (yet).

Clouds
The obvious use for a volumetric particle cloud rendering system it for rendering ... clouds. There are a ton of competing ways to draw clouds in games. Sometimes clouds are meant to be viewed from below, for example when the player can walk or drive on the ground. Sometimes clouds are meant to be viewed from above or inside, for example in a flight simulator or space game. In 3DWorld, there's really no constraint on where the player can go. It's a game engine, not a game, so it could be used for a ground-based first person shooter or a flight simulator. These clouds need to look correct, with a good frame rate, when viewed from any location. Well, except from their exact centers (again).

Clouds are drawn in a base color of white but are modulated to match the lighting conditions so that they're red-orange during sunrise/sunset and dark gray at night. Their brightness decreases toward their center to simulate self-shadowing inside the cloud volume. I haven't yet tried to add light scattering to these clouds. In addition, they become opaque near the center where the noise value has less of an effect. They use regular Perlin noise rather than ridged noise for a more puffy and natural appearance, and have a mixture of high and low frequencies. The noise function offset varies slowly so that clouds change shape over time. Here are some clouds viewed from the ground below in infinite tiled terrain mode.

View of volumetric procedural cloud puffs from below. These clouds slowly change over time.

Tiled terrain mode actually draws three cloud layers (listed in the order in which I added them):

• 2D procedural noise cloud plane. The terrain and grass shaders ray cast into this layer for soft cloud shadows. I also have slow God-rays that ray march through the cloud density function. Density depends on weather conditions and atmosphere. Slowly animated/scrolling.
• Static textured upper cloud layer. Provides a more interesting background than pure blue.
• Puffy volumetric clouds. The new mode that's the subject of this post. Doesn't cast shadows yet - unclear how the terrain and grass shader can ray cast into these. Animated, but more slowly.

The sky looks a bit cluttered with all three cloud layers enabled now. Also, it's odd that the first layer casts shadows, but the second layer (which is denser and more apparent) doesn't cast shadows. Maybe I'll remove the first cloud plane layer later. Or maybe the set of enabled layers should be determined by atmosphere and weather conditions. For example, use a large number of puffy gray volume clouds when it's rainy, but only sparse/high 2D cloud cover when it's sunny.

This wireframe view of clouds from above shows how they are composed of multiple 2D quad billboards in various orientations arranged around a common center point, similar to flower petals.

Clouds drawn in wire frame mode. The structure of the 13 intersecting planes can be seen.

 
Here is a short video of the player flying through the cloudy sky over some islands. Since the clouds are rendered in 3D, they appear as real volumes when flown through. Most games seem to use 2D cloud billboards or skybox background images that are only meant to be viewed from a distance.

So far I've come up with four different uses of this particle cloud rendering technology. I wonder what else I can use it for? Rocket explosions? Plasma balls? Smoke effects? Insect swarms? Actually this might work well for smoke, and could be a good replacement for the existing billboard cloud system I use for smoke in 3DWorld. I guess we'll just have to wait and see.

Indirect Lighting for Player Controlled Lights

I'm continuing to work on improving the dynamic lighting in 3DWorld. This post is a short update that builds on the previous two posts (indirect lighting and dynamic lighting + triggers):
Indirect Lighting
Lighting and Triggers

I was reading another blog post from The Witness and it gave me an idea. The contrast is too high between the lit and unlit areas of the basement scene. All of the lighting is coming from the spotlight direct illumination; the indirect reflected light is completely missing. This is why the orange balls look black on the sides facing away from the lights. The scene doesn't look very real.

Let me review how the indirect lighting in 3DWorld works. The scene is divided into a 3D grid of light volumes in {x, y, z} that are uploaded to the GPU and used in the fragment shader to individually light each pixel. During the offline preprocessing phase, each light source emits millions of rays, each of which is traced through the scene using multiple CPU threads. Each ray's weighted RGB (red, green, and blue) light contribution is added to each grid cell that it passes through. This means that the fragment shader can query any point in space within the scene bounds to get the indirect lighting contribution. This uses more memory per volume than lightmaps and therefore the lighting is stored at a coarser granularity. But, it has the advantage that dynamic objects (such as the orange balls) that weren't part of the original static scene can be correctly lit. This approach may also be simpler to implement and more efficient to compute. I haven't implemented lightmaps so I don't know for sure.

Okay, back to the problem. Dynamic lights can be turned on and off when their triggers (light switches) are activated, so the indirect lighting isn't constant. It can't be baked into the (single) global lighting volume of the scene. The indirect lighting can't even be stored per-trigger because it needs to be removed when an individual light is destroyed. What is needed is per-light source volumes that are generated on-the-fly when needed and merged into the final lighting solution when their intensity (or enabled state) changes. Since the light triggering is infrequent, most game frames have the same set of enabled lights. It makes sense to only merge in the new lighting values when they change, and re-upload the merged data to the GPU sparsely. This avoids having to read from multiple 3D lighting textures on the GPU each frame. I haven't actually tried this, but I assume it would have a significant effect on frame rate. The 4-5ms of CPU time updating the lighting every few seconds is negligible.

So what does it look like? Here is a screenshot of the direct + indirect lighting effects on the basement spotlight scene with some orange balls in motion.

Direct + Indirect lighting + Shadows in the basement spotlight scene.

The biggest difference is the reflection of the spotlight hitting the ceiling and floor near the light on the back wall. The sides of the balls facing away from the lights aren't completely black any more. Much better. Unfortunately, now there is a 6 second freeze when the player first turns on the lights as the CPU computes 5 million rays (1M per light source) with 4 bounces each. That really ruins gameplay. Who wants to sit there waiting for the lighting to be computed in the middle of playing the game? It takes longer than loading the scene at the beginning!

One solution is to compute the lighting of each light source once in a preprocessing pass, then write it to disk for later reuse. I modified the scene file reader to accept filenames attached to each light for caching indirect lighting on disk. This works well, reducing lighting computation time from 6s to a few milliseconds.

However, now I'm stuck with multiple 8MB files on disk, one per light source. These files together take up more disk space than the rest of the scene files combined. They need to be compressed. Fortunately, they're easy to compress. The RGB color data is mostly zeros, and 32-bit floating-point numbers have more precision than I need. 8-bit unsigned integers would work just fine - they get converted to 8-bit in the GPU texture later anyway. The first thing I did was to remove most of those zeros. Since these are small local lights, their radius of influence is pretty small. In addition, their light is confined to this one room in the basement. I first filter the lighting values so that any value smaller than 0.1% is clamped to 0. Then I compute the smallest bounding cube that contains all of the nonzero values. This provides a 100-200x reduction in file size and memory usage. The 8MB files are now only 40-80KB. The reduction is enough that it doesn't seem necessary to do the 32-bit => 8-bit data compression.

Here are some screenshots comparing the effect of the different lighting components. In my opinion, the new combined direct + indirect lighting looks much better than direct only. [Ignore the frame rate on the lower left corner - I froze the scene update so the framerate counter wasn't updating. It normally runs at over 200 FPS.]

Uniform lighting shows the base material colors and textures. Some crates were added to provide more interesting shadows.

No lighting. A few emissive objects are visible (light switch and sky visible through the window).

Direct lighting + shadows only. The spotlights themselves are lit by a separate small light. Similar to the previous blog post.

Indirect lighting only. Most of the direct light hits the floor and ceiling near the back wall, reflecting light onto the wall.

Direct and indirect lighting combined form a more realistic global lighting solution for the scene.

Destruction Video

Here is a test video of me shooting out windows, benches, lights, and exploding tanks.