Postprocessing Effects

I've been experimenting with full-screen postprocessing effects in 3DWorld using the fragment shader on the GPU. The shader has access to the color and depth buffers from the current frame, and overwrites the color buffer before displaying it. Some of the effects worked out well, but others did not. Here is a list of what effects have been added (in no particular order). This is a long post, but it's mostly images and videos and not too much text.

Fog
This is a very simple and common shader effect. It's also very cheap, so it doesn't affect the framerate much. However, I'm not really using it as a full-screen postprocessing effect. Instead, it's added as a postprocessing effect at the end of the regular draw shaders for all objects.

I'm doing more than the standard fog here, where the standard fog technique uses the depth (distance from the camera) of each pixel to select the fog color from no fog near the camera to full gray fog in the far distance. The 3DWorld fog is actually computed by ray marching a few steps through the scene and using the precomputed indirect lighting at each step to determine the fog color. Brightly lit areas have light gray fog, and dark shadowed areas have black fog. That way, the basement has black fog rather than the normal white-ish outdoor fog. For performance reasons, I'm doing a few ray steps, and only for some object types. It looks much better (with smoother color transitions) with more steps, but it's slower. Here are two screenshots for comparison.

Without Fog

With fog

God Rays
God rays, or crepuscular rays, are shafts of sunlight that hit small particles in the air, producing a glowing effect. Many games use God rays to increase the realism of bright direct lighting. 3DWorld uses a common approach of sampling the color buffer, ray marching from the sun position outward in screen space, and using the depth buffer to occlude the smear of color. This gives bright lines that radiate outward from the sun between other objects that block it. I have it working for scene objects, trees, and other solid objects, but not clouds (which don't contribute to the depth buffer). Here is another pair of screenshots showing the effect.

Without God rays

With God rays

Here is an image of God rays and fog in tiled terrain mode, looking through some trees. Note how the sunlight shines though the backs of the leaves as well. Real leaves are partially transparent to light.

A foggy morning, with God rays showing through the tree leaves. Also note the sunlight illuminating the backs of the leaves.

Another view. This time I remembered to turn off the onscreen display text.

Bloom
Bloom is a type of HDR (High Dynamic Range) postprocessing effect that makes bright objects appear even brighter than maximum monitor white by bleeding their color into surrounding pixels. If a light source is only a few pixels wide, the addition of a halo around it makes it seem brighter. This works well for small light sources that are only a few pixels wide, where the max white RGB value of {255, 255, 255} is not enough.

I'm using a 9x9 X/Y separable filter kernel (also used by the DOF and underwater blur effects). The color blur is done independently in X and Y using a 9x1 filter and a 1x9 filter. This requires 2*9 = 18 color texture lookups rather than 9x9 = 81. This is much faster and looks almost as good.

I think the bloom effect works well for small local light sources, but I don't like the effect on the sun and clouds. It's not clear how the fragment/pixel shader knows which pixels are sun vs. other light sources. Right now it's only testing for color intensity greater than 75%. Here are two comparison screenshots.

Without bloom

With bloom. The sun and sky are very bright! The courtyard lights are also bright. (but why are they on in the daytime?)

Heat Waves
I added screen space vertical heat waves as a postprocessing effect when the player is near a heat source such as fire. This is just a simple vertical wavy line effect on the entire screen, with decreasing wave amplitude from bottom to top. The shader doesn't actually know where the fire is on the screen, or if it's even visible in the view frustum. All it can do is add the effect or not. Here you can also see the billboard smoke particles effect. Smoke is colored based on lighting and shadows, so the smoke near the fire appears more reddish.

Heat waves rising from the fires. Don't get too close or you'll be burned!

Depth of Field
Depth of Field (DOF) is a common effect used in many games. The general idea is that the player is focusing on some particular object at a particular distance from the eye/camera. Objects both nearer and further than the object are out of focus and appear blurry. This simulates focusing of the human eye and of some types of camera lenses. Personally, I don't like this effect in games. I don't like anything that makes the scene look blurry - better to have sharp, crisp images. I added this effect to 3DWorld because it looked neat, but it's not used in-game. Here is what it looks like - click on the image for a larger view.

Depth of Field test image. The camera is focusing on the lamp in the center of the screen (in the white circle).

Here is a DOF video. I hope the blurriness of the video due to image compression artifacts doesn't mask the effect.

Underwater Blur and Distortion
Underwater blur is used to increase the realism when the player is underwater, and make it more obvious when the water boundary is crossed by the eye/camera. A wavy distortion is also added when the player is out of oxygen and drowning to simulate dizziness. When you see this effect, you know to get out of the water fast! I originally came across this effect through random experimentation, and I wanted to use it for something. It would make a great drunk effect if I ever add alcoholic drinks to 3DWorld. Here is an example of this rather strange effect.

Underwater drowning effect. Or maybe drunken effect? Watching this sure makes me dizzy.

Screen Space Ambient Occlusion
Not all of my postprocessing effects experiments were successful. SSAO was actually my first attempt at a postprocessing shader, but I should have started with something simpler. It works by calculating light occlusion in screen space using the depth and normal information, and then darkening screen pixels based on the amount of occlusion. That way, small folds and crevices appear darker than open areas. It doesn't work perfectly: for example, occluders that are off-screen or blocked by other objects can't be used.

This effect never really worked correctly in 3DWorld, for a number of reasons. First, 3DWorld uses a forward rendering pipeline, not a deferred rendering pipeline, so there is no normal buffer. Without object normals, it's impossible to tell front faces from back faces and which faces point toward each other. This results in incorrect darkening of areas that aren't actually occluded from the light source.

Second, SSAO doesn't really work with transparent objects. If the transparent object such as a glass window writes to the depth buffer, the front of it is darkened as if it was an opaque wall, which looks wrong. If it doesn't write to the depth buffer, the ambient occlusion isn't blended correctly and the dark area appears in front of the window, which also looks wrong. Here is a pair of screenshots with and without SSAO. The effect is subtle, so you probably need to click on the bottom image to get a higher resolution view.

Without SSAO

With SSAO. Not the intended effect. The color banding, black outlines from the transparent windows, false occlusion, etc. ruin the realism of the scene. Oh well, it was worth a try.

I'll see if I can think up any additional full-screen postprocessing effects.

Museum Scene

I found a nice 3D museum model online, so I decided to post some screenshots of how it looks rendered in 3DWorld. The model was taken from Challenge #17 from the challenge page on 3DRender.com. Note that I didn't actually participate in this challenge. I also spent little time preparing. I don't actually have any real image editing or 3D modeling software, and I wanted to render this in realtime, so it's not the highest quality rendering and there are no fancy special effects. However, I still think it looks good.

This museum scene contains around 750K triangles. 3DWorld can handle scenes up to around 12M triangles, so I could have 16 of these museums in my scene and still draw it in realtime. The original file was untextured, with no normals or texture coordinates. I had to use textures that I found online and create the normal maps using a trial version of CrazyBump. The textures are applied using standard {X, Y, Z} triplanar texturing where the texture coordinates are determined by the dominant direction of the normal. It's difficult to generate automatic texture coordinates for scenes with curved geometry without using a 3rd party tool. Vertex normals are generated from the face normals by 3DWorld. Some of the triangle faces in the model have the wrong clockwise vs. CCW winding order, which causes black spots and smeared textures, but it's not too noticeable.

There are several light sources in this scene:

• Sun direct and indirect lighting with shadows
• Sky/cloud indirect lighting
• Local diffuse + indirect indoor light sources (2 yellow lanterns and 4 blue ceiling lights)

There is a one time lighting precomputation of around 10 min., and after that the scene can be drawn at around 100 FPS (frames per second). It takes 10 seconds or so to update the sun indirect lighting component when the sun position is changed. Here are some screenshots of the museum scene.

Museum scene from the front stairs.

Museum scene from the ground floor in the back.

Museum scene from above near the ceiling.

This is a fully collision and physics enabled scene in 3DWorld. The player can walk around and collide with the geometry, shoot things, and play in game mode against the smiley AIs. There are a few holes in the floor that the player can fall though, but I think I mostly fixed those problems.

That's it for now. I'll write something longer next week when I have more time.

Tree, Plant, and Grass Leaf Wind

It's been awhile since my last post, but I haven't really been working on much that I can show in screenshots and videos. Most of the effort has been spent on continued improvements to movable object physics and related tasks. I had it working well enough to show some videos last month, but it took a lot more effort to make the system stable and fix all of the strange bugs, including the crazy large water splash when underwater boxes were destroyed. I'm working on object rotational physics and that sort of thing now, but it's difficult to get right, and I don't have anything to show yet. I may end up using a third party physics library such as Bullet for the more complex effects.

I also spent a few hours improving the 3DWorld leaf wind effect and making it work on more vegetation types, so I'll make this the topic of today's post. This post won't be as long as the previous one, which hopefully means that I'll have time to make another post soon. I already had a leaf wind system for trees that was entirely on the CPU, and it's always bothered me that this method is slow and I find myself disabling it at times to improve the frame rate. In 2015 everyone seems to be doing leaf wind on the GPU rather than the CPU, so I decided to try this approach. I've actually been thinking about and experimenting with GPU wind for years, but I was never able to figure out how to update the leaf normals for lighting as the leaf moves in the wind. The problem is that the vertex shader can normally only see one vertex at a time, and it can't determine the orientation (rotation axis and angle) of the leaf from a single vertex. The leaf needs to bend/rotate at the point where it's connected to the tree branch, a point that the vertex shader doesn't know. Sure, I could pass this extra info as additional vertex data, but that takes more GPU memory and hurts the frame rate. I could use a geometry shader, but that turned out to be very complex and also slow.

A few weeks ago I read an online article on leaf wind. The author ran into the same problem with updating the leaf normals, but he went ahead and implemented it anyway with the normals and lighting remaining constant. He claimed that it looked just fine this way - and I agreed! So all this time I had been worried about what was really a minor issue. The solution: do something simple per-leaf that doesn't need to know about the leaf orientation and just use the original normals. I used a simple time-varying sum of two sine waves and that looked pretty good. Even with fixed normals, the lighting can still be somewhat dynamic. Shadows and indirect lighting are computed per pixel, and don't depend (much) on the normal, so these components of the lighting will still change dynamically as the leaves move in the wind.

I still had to deal with the leaf attachment point to the branch. The leaf tip should move, but not the base part where the stem attaches to the tree branch. How can the shader know which vertex is at the tip vs. the base? It can use the texture coordinate, which is 0.0 and the base and 1.0 at the tip. This data is already available since it's used to texture the leaf. The next problem is how to keep the leaf size constant by moving both vertices of the tip (the two vertices that form the far edge of the leaf quad/rectangle near the tip). I solved this by using the world space coordinates of the leaf to determine the magnitude of the wind displacement. Since leaves are small, the coordinate values of their tip vertices are close together, making their wind displacement values similar, therefore keeping the distance between those points relatively constant.

This system worked so well that I ended up using it for the small branches and leaves/needles of pine trees, which previously had no wind effects. Then I enabled it for plant leaves. I was never quite able to get the wind effect to work for the berries on some plant types, so I simply disabled this effect on those plants. Grass was already using a similar wind system, so I left that code alone. [Grass is easier since each blade is drawn as a triangle, where only the tip vertex moves, and it's a lot simpler to animate a single vertex rather than an edge formed from two vertices for leaves. Also, grass blades tend to have a more uniform length and orientation, so the vertex shader can guess at the positions of the other two base vertices.]

Here is a screenshot of some vegetation moving in the wind. Oh, right, you can't actually see it moving. Well, it's still a pretty screenshot.

Trees, plants, and grass moving in the wind. This is just a high resolution image from the video below.

Here is a video of the same scene. You can see the tree leaves, plant leaves, flowers, and grass blades all moving in the wind. The wind speed and direction can be interactively changed with hotkeys and an onscreen slider UI widget (not shown). The video starts with a small amount of wind and gradually increases wind intensity to an extreme value. I had to choose a strange camera angle in the corner of the scene to get the pine trees, deciduous trees, plants, and grass all in the same view.

Here is another video showing that wind works on all vegetation in infinite tiled terrain mode as well. This scene with it's high detail trees ran at only ~50 FPS (frames per second) with CPU wind, but runs at ~90 FPS with GPU wind. In fact, the wind computation time has almost no effect on frame rate. In addition, wind is applied even to distant trees, rather than being limited to only nearby trees in the old mode. It may be hard to see the wind effect in this video, especially since I couldn't stand still and had to run around at high speed just to show off that it was possible. The max speed in tiled terrain mode is very high and when the player is on the ground everything is a blur. The minor lag near the end when I went in the water is probably due to loading the water sound.

Right, back to the wind. I'll note that the new wind system doesn't completely replace the old mode. I still need CPU mode for player interaction with leaves. For example, player collision with trees moves the leaves, leaves can be shot off using projectile weapons, etc. Oh, all right, I'll add a video of that too. I guess every post must now contain at least one video of me shooting or otherwise destroying something. Keep in mind that the particle effects are only a placeholder for better particle effects that I'll likely show off in some future post.

Did you notice that I added new sounds such as the "click" when switching weapons? I'll try to continue to add more sounds and other effects as time allows.

That's about all I have to say about leaf wind and related topics. Next time I'll probably make a post about full-screen postprocessing effects, unless something else interesting comes up. I was considering making a post about snow accumulation and rendering, but I think I'll save that for winter time - not that we actually get snow here in CA.