Inside Direct3D: Stencil Buffers
http://gamasutra.com/features/20000807/kovach_01.htm
Inside Direct3D: Stencil Buffers
One aspect of advanced rendering we haven't discussed yet is stenciling, a technique that can be useful for developing commercial applications. If you want your 3D applications to stand apart from the crowd, you'd be wise to combine stenciling with the texturing techniques you learned about in earlier chapters. This chapter will detail how to use stenciling and show you the different types of effects you can generate with it.
Many 3D games and simulations on the market use cinema-quality special effects to add to their dramatic impact. You can use stencil buffers to create effects such as composites, decals, dissolves, fades, outlines, silhouettes, swipes, and shadows. Stencil buffers determine whether the pixels in an image are drawn. To perform this function, stencil buffers let you enable or disable drawing to the render-target surface on a pixel-by-pixel basis. This means your software can "mask" portions of the rendered image so that they aren't displayed.
When the stenciling feature is enabled, Microsoft Direct3D performs a stencil test for each pixel that it plans to write to the render-target surface. The stencil test uses a stencil reference value, a stencil mask, a comparison function, and a pixel value from the stencil buffer that corresponds to the current pixel in the target surface. Here are the specific steps used in this test:
- Perform a bitwise AND operation of the stencil reference value with the stencil mask.
- Perform a bitwise AND operation on the stencil-buffer value for the current pixel with the stencil mask.
- Compare the results of Step 1 and Step 2 by using the comparison function.
By controlling the comparison function, the stencil mask, the stencil reference value, and the action taken when the stencil test passes or fails, you can control how the stencil buffer works. As long as the test succeeds, the current pixel will be written to the target. The default comparison behavior (the value that the D3DCMPFUNC enumerated type defines for D3DCMP_ALWAYS) is to write the pixel without considering the contents of the stencil buffer. You can change the comparison function to any function you want by setting the value of the D3DRENDERSTATE_STENCILFUNC render state and passing one of the members of the D3DCMPFUNC enumerated type.
Creating a Stencil Buffer
Before creating a stencil buffer, you need to determine what stenciling capabilities the target system supports. To do this, call the IDirect3DDevice7::GetCaps method. The dwStencilCaps flags specify the stencil-buffer operations that the device supports. The reported flags are valid for all three stencil-buffer operation render states: D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILPASS, and D3DRENDERSTATE_STENCILZFAIL. Direct3D defines the following flags for dwStencilCaps:
- D3DSTENCILCAPS_DECR Indicates that the D3DSTENCILOP_DECR operation is supported
- D3DSTENCILCAPS_DECRSAT Indicates that the D3DSTENCILOP_DECRSAT operation is supported
- D3DSTENCILCAPS_INCR Indicates that the
D3DSTENCILOP_INCR operation is supported
- D3DSTENCILCAPS_INCRSAT Indicates that the D3DSTENCILOP_INCRSAT operation is supported
- D3DSTENCILCAPS_INVERT Indicates that the D3DSTENCILOP_INVERT operation is supported
- D3DSTENCILCAPS_KEEP Indicates that the D3DSTENCILOP_KEEP operation is supported
- D3DSTENCILCAPS_REPLACE Indicates that the D3DSTENCILOP_REPLACE operation is supported
- D3DSTENCILCAPS_ZERO Indicates that the D3DSTENCILOP_ZERO operation is supported
Direct3D embeds the stencil-buffer information with the depth-buffer data. To determine what formats of depth buffers and stencil buffers the target system's hardware supports, call the IDirect3D7::EnumZBufferFormats method, which has the following declaration:
HRESULT IDirect3D7::EnumZBufferFormats (
REFCLSID riidDevice,
LPD3DENUMPIXELFORMATSCALLBACK lpEnumCallback,
LPVOID lpContext
);
|
Parameter
|
Description
|
| riidDevice | A reference to a globally unique identifier (GUID) for the device whose depth-buffer formats you want enumerated |
| IpEnumCallback | The address of a D3DEnumPixelFormatsCallback function you want called for each supported depth-buffer format |
| IpContext | Application-defined data that is passed to the callback function |
If the method succeeds, it returns the value D3D_OK. If it fails, the method returns one of these four values:
- DDERR_INVALIDOBJECT
- DDERR_INVALIDPARAMS
- DDER_NOZBUFFERHW
- DDERR_OUTOFMEMORY
The code in listing 1 determines what stencil buffer formats are available and what operations are supported and then creates a stencil buffer. As you can see, this code notes whether the stencil buffer supports more than 1-bit -- some stenciling techniques must be handled differently if only a 1-bit stencil buffer is available.
Clearing a Stencil Buffer
The IDirect3DDevice7 interface includes the Clear method, which you can use to simultaneously clear the render target's color buffer, depth buffer, and stencil buffer. Here's the declaration for the IDirect3DDevice7::Clear method:
HRESULT IDirect3DDevice7::Clear(
DWORD dwCount,
LPD3DRECT lpRects,
DWORD dwFlags,
D3DVALUE dvZ,
DWORD dwStencil
);
|
Parameter
|
Description
|
| dwCount | The number of rectangles in the array at lpRects. |
|
IpRects
|
An array of D3DRECT structures defining the rectangles to be cleared. You can set a rectangle to the dimensions of the render-target surface to clear the entire surface. Each of these rectangles uses screen coordinates that correspond to points on the render-target surface. The coordinates are clipped to the bounds of the viewport rectangle. |
| dwFlags | Flags indicating which surfaces should be cleared. This parameter can be any combination of the following flags, but at least one flag must be used: |
| D3DCLEAR_TARGET Clear the render-target surface to the color in the dwColor parameter. D3DCLEAR_STENCIL Clear the stencil buffer to the value in the dwStencil parameter. | |
| D3DCLEAR_ZBUFFER Clear the depth buffer to the value in the dvZ parameter. | |
| dwColor | D3DCLEAR_ZBUFFER Clear the depth buffer to the value in the dvZ parameter.. |
| dvZ | A 32-bit RGBA color value to which the render-target surface will be cleared. |
| dwStencil | The new z value that this method stores in the depth buffer. This parameter can range from 0.0 to 1.0, inclusive. The value of 0.0 represents the nearest distance to the viewer, and 1.0 represents the farthest distance. |
| The integer value to store in each stencil-buffer entry. This parameter can range from 0 to 2n-1 inclusive, in which n is the bit depth of the stencil buffer. |
The IDirect3DDevice7::Clear method still accepts the older D3DCLEAR_TARGET flag, which clears the render target using an RGBA color you provide in the dwColor parameter. This method also still accepts the D3DCLEAR_ZBUFFER flag, which clears the depth buffer to a depth you specify in dvZ (in which 0.0 is the closest distance and 1.0 is the farthest). DirectX 6 introduced the D3DCLEAR_STENCIL flag, which you can use to reset the stencil bits to the value you specify in the dwStencil parameter. This value can be an integer ranging from 0 to 2n-1, in which n is the bit depth of the stencil buffer.
________________________________________________________
Configuring the Stenciling State
Configuring the Stenciling State
You control the various settings for the stencil buffer using the IDirect3DDevice7::
SetRenderState method. Listing 2 shows the stencil-related members of the D3DRENDERSTATETYPE enumerated type.
These are the definitions for the stencil-related render states:
- D3DRENDERSTATE_STENCILENABLE Use this member to enable or disable stenciling. To enable stenciling, use this member with TRUE; to disable stenciling, use it with FALSE. The default value is FALSE.
- D3DRENDERSTATE_STENCILFAIL Use this member to indicate the stencil operation to perform if the stencil test fails. The stencil operation can be one of the members of the D3DSTENCILOP enumerated type. The default value is D3DSTENCILOP_KEEP.
- D3DRENDERSTATE_STENCILZFAIL Use this member to indicate the stencil operation to perform if the stencil test passes and the depth test (z-test) fails. The operation can be one of the members of the D3DSTENCILOP enumerated type. The default value is D3DSTENCILOP_KEEP.
- D3DRENDERSTATE_STENCILPASS Use this member to indicate the stencil operation to perform if both the stencil test and the depth test (z-test) pass. The operation can be one of the members of the D3DSTENCILOP enumerated type. The default value is D3DSTENCILOP_KEEP.
- D3DRENDERSTATE_STENCILFUNC Use this member to indicate the comparison function for the stencil test. The comparison function can be one of the members of the D3DCMPFUNC enumerated type. The default value is D3DCMP_ALWAYS. This function compares the reference value to a stencil-buffer entry and applies only to the bits in the reference value and stencil-buffer entry that are set in the stencil mask. (The D3DRENDERSTATE_STENCILMASK render state sets the stencil mask.) If the comparison is true, the stencil test passes.
- D3DRENDERSTATE_STENCILREF Use this member to indicate the integer reference value for the stencil test. The default value is 0.
- D3DRENDERSTATE_STENCILMASK Use this member to specify the mask to apply to the reference value and each stencil-buffer entry to determine the significant bits for the stencil test. The default mask is 0xFFFFFFFF.
- D3DRENDERSTATE_STENCILWRITEMASK Use this member to specify the mask to apply to values written into the stencil buffer. The default mask is 0xFFFFFFFF.
The D3DSTENCILOP enumerated type describes the stencil operations for the D3DRENDERSTATE_STENCILFAIL, D3DRENDERSTATE_STENCILZFAIL, and D3DRENDERSTATE_STENCILPASS render states. Here's the definition of D3DSTENCILOP:
typedef enum _D3DSTENCILOP {
D3DSTENCILOP_KEEP = 1,
D3DSTENCILOP_ZERO = 2,
D3DSTENCILOP_REPLACE = 3,
D3DSTENCILOP_INCRSAT = 4,
D3DSTENCILOP_DECRSAT = 5,
D3DSTENCILOP_INVERT = 6,
D3DSTENCILOP_INCR = 7,
D3DSTENCILOP_DECR = 8,
D3DSTENCILOP_FORCE_DWORD = 0x7fffffff
} D3DSTENCILOP;
These members serve the following purposes:
- D3DSTENCILOP_KEEP Indicates that you don't want the entry in the stencil buffer updated. This is the default operation.
- D3DSTENCILOP_ZERO Sets the stencil-buffer entry to 0.
- D3DSTENCILOP_REPLACE Replaces the stencil-buffer entry with the reference value.
- D3DSTENCILOP_INCRSAT Increments the stencil-buffer entry, clamping to the maximum value.
- D3DSTENCILOP_DECRSAT Decrements the stencil-buffer entry, clamping to 0.
- D3DSTENCILOP_INVERT Inverts the bits in the stencil-buffer entry.
- D3DSTENCILOP_INCR Increments the stencil-buffer entry, wrapping to 0 if the new value exceeds the maximum value.
- D3DSTENCILOP_DECR Decrements the stencil-buffer entry, wrapping to the maximum value if the new value is less than 0.
- D3DSTENCILOP_FORCE_DWORD Forces this enumeration to be compiled to 32 bits; this value isn't used.
Let's walk through some code that uses the stencil buffer while rendering a scene. This code is from a sample that shows how to draw shadows. For now, don't worry about how all this code generates shadows-the algorithm is described later in the chapter.
The shadow-rendering code starts out by disabling the depth buffer and enabling the stencil buffer:
//--------------------------------------------------
// Name: RenderShadow
// Desc:
//------------------------------------------------
HRESULT CMyD3DApplication::RenderShadow()
{
// Turn off depth buffer and turn on
stencil buffer.
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_ZWRITEENABLE,
FALSE );
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILENABLE,
TRUE );
Next the code sets the comparison function that performs the stencil test by calling the IDirect3DDevice7::SetRenderState method and setting the first parameter to D3DRENDERSTATE_STENCILFUNC. The second parameter is set to a member of the D3DCMPFUNC enumerated type. In this code, we want to update the stencil buffer everywhere a primitive is rendered, so we use D3DCMP_ALWAYS:
//
// Set up stencil comparison function,
reference value, and masks.
// Stencil test passes if ((ref & mask)
cmpfn (stencil & mask))
// is true.
//
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILFUNC,
D3DCMP_ALWAYS );
In this sample, we don't want the stencil buffer to change if either the stencil buffer test or the depth buffer test fails, so we set the appropriate states to D3DSTENCILOP_KEEP:
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILZFAIL,
D3DSTENCILOP_KEEP );
m_pd3dDevice->SetRenderState(
D3DRENDERSTATE_STENCILFAIL,
D3DSTENCILOP_KEEP );
The settings in listing 3 are different depending on whether a 1-bit or a multibit stencil buffer is present. If the stencil buffer has only 1 bit, the value 1 is stored in the stencil buffer whenever the stencil test passes. Otherwise, an increment operation (either D3DSTENCILOP_INCR or D3DSTENCILOP_INCRSAT) is applied if the stencil test passes. At this point, the stencil state is configured and the code is ready to render some primitives.
________________________________________________________
Creating Effects
Creating Effects
Now that you've seen how to create stencil buffers and configure how they work, let's look at some of the effects you can render with them. The following sections describe several ways Microsoft recommends using stencil buffers. Each of these approaches produces impressive results, but a few of them have drawbacks.
Composites
You can use stencil buffers for compositing 2D or 3D images onto a 3D scene. By using a mask in the stencil buffer to occlude a portion of the render-target surface, you can write stored 2D information (such as text or bitmaps). You can also render 3D primitives -- or for that matter a complete scene -- to the area of the render-target surface that you specify in a stencil mask.
Developers often use this effect to composite several scenes in simulations and games. Many driving games feature a rear view mirror that displays the scene behind the driver. You can composite this second 3D scene with the driver's view forward by using a stencil to block the portion to which you want the mirror image rendered. You can also use composites to create 2D "cockpits" for vehicle simulations by combining a 2D, bitmapped image of the cockpit with the final, rendered 3D scene.
Decals
You can use decals to control which pixels form a primitive image you draw to a render-target surface. When you apply a texture to an object (for example, applying scratch marks to a floor), you need the texture (the scratch marks) to appear immediately on top of the object (the floor). Because the z values of the scratch marks and the floor are equal, the depth buffer might not yield consistent results, meaning that some pixels in the back primitive might be rendered on top of those in the front primitive. This overlap, which is commonly known as z-fighting or flimmering, can cause the final image to shimmer as you animate from one frame to the next.
You can prevent flimmering by using a stencil to mask the section of the back primitive on which you want the decal to appear. You can then turn off z-buffering and render the image of the front primitive into the masked area of the render-target surface.
Dissolves
You can use dissolves to gradually replace an image by displaying a series of frames that transition from one image to another. In Chapter 8, you saw how to use multiple-texture blending to create this effect by gradually blending two textures together. Stencil buffers allow you to produce similar dissolves, except that a stencil-based dissolve looks more pixelated than a multiple-texture blending one. However, stencil buffers let you use texture-blending capabilities for other effects while performing a dissolve. This capability enables you to efficiently produce more complex effects than you could by using texture blending alone.
A stencil buffer can perform a dissolve by controlling which pixels you draw from two different images to the render-target surface. You can perform a dissolve by defining a base stencil mask for the first frame and altering it incrementally or by defining a series of stencil masks and copying them into the stencil buffer on successive frames.
To start a dissolve, set the stencil function and stencil mask so that most of the pixels from the starting image pass the stencil test and most of the ending image's pixels fail. For each subsequent frame, update the stencil mask to allow fewer pixels in the starting image to pass the test and more pixels in the ending image to pass. By controlling the stencil mask, you can create a variety of dissolve effects.
Although this approach can produce some fantastic effects, it can be a bit slow on some systems. You should test the performance on your target systems to verify that this approach works efficiently for your application.
Fades
You can fade in or out using a form of dissolving. To perform this effect, use any dissolve pattern you want. To fade in, use a stencil buffer to dissolve from a black or white image to a rendered 3D scene. To fade out, start with a rendered 3D scene and dissolve to black or white. As with dissolves, you should check the performance of fades on the target systems to verify that their speed and appearance is acceptable.
Outlines
You can apply a stencil mask to a primitive that's the same shape but slightly smaller than the primitive. The resulting image will contain only the primitive's outline. You can then fill this stencil-masked area of the primitive with a color or set of colors to produce an outline around the image.
Silhouettes
When you set the stencil mask to the same size and shape as the primitive you're rendering, Direct3D produces a final image containing a "black hole" where the primitive should be. By coloring this hole, you can produce a silhouette of the primitive.
Swipes
A swipe makes an image appear as though it's sliding into the scene over another image. You can use stencil masks to disable the writing of pixels from the starting image and enable the writing of pixels from the ending image. To perform a swipe, you can define a series of stencil masks that Direct3D will load into the stencil buffer in a succession of frames, or you can change the starting stencil mask for a series of successive frames. Both methods cause the final image to look as though it's gradually sliding on top of the starting image from right to left, left to right, top to bottom, and so on.
To handle a swipe, remember to read the pixels from the ending image in the reverse order in which you're performing the swipe. For example, if you're performing a swipe from left to right, you need to read pixels from the ending image from right to left. As with dissolves, this effect can render somewhat slowly. Therefore, you should test its performance on your target systems.
Shadows
Shadow volumes, which allow an arbitrarily shaped object to cast a shadow onto another arbitrarily shaped object, can produce some incredibly realistic effects. To create shadows with stencil buffers, take an object you want to cast a shadow. Using this object and the light source, build a set of polygonal faces (a shadow volume) to represent the shadow.
You can compute the shadow volume by projecting the vertices of the shadow-casting object onto a plane that's perpendicular to the direction of light from the light source, finding the 2D convex hull of the projected vertices (that is, a polygon that "wraps around" all the projected vertices), and extruding the 2D convex hull in the light direction to form the 3D shadow volume. The shadow volume must extend far enough so that it covers any objects that will be shadowed. To simplify computation, you might want the shadow caster to be a convex object.
To render a shadow, you must first render the geometry and then render the shadow volume without writing to the depth buffer or the color buffer. Use alpha blending to avoid having to write to the color buffer. Each place that the shadow volume appears will be marked in the stencil buffer. You can then reverse the cull and render the backfaces of the shadow volume, unmarking all the pixels that are covered in the stencil buffer. All these pixels will have passed the z-test, so they'll be visible behind the shadow volume. Therefore, they won't be in shadow. The pixels that are still marked are the ones lying inside the front and back boundaries of the shadow volume-these pixels will be in shadow. You can blend these pixels with a large black rectangle that covers the viewport to generate the shadow.
The ShadowVol and ShadowVol2 Demos
The ShadowVol sample on the companion CD in the \mssdk\Samples\Multimedia\D3dim\Src\ShadowVol directory contains a project that shows how to create and use stencil buffers to implement shadow volumes. The code illustrates how to use shadow volumes to cast the shadow of an arbitrarily shaped object onto another arbitrarily shaped object. The ShadowVol2 sample, which the Microsoft DirectX 7 SDK setup program on the companion CD installs in the \mssdk\Samples\Multimedia\D3dim\Src\ShadowVol2 directory on your hard disk, provides some additional capabilities for producing shadows with stencils.
The sample application provides these features in its Shadow Modes menu:
- Draw Shadows: Allows you to turn on and off shadow rendering.
- Show Shadow Volumes: Draws the shadow volumes used to compute the shadows rather than drawing the shadows themselves.
- Draw Shadow Volume Caps: When you turn this item off, some "extra" shadows might become visible where the far caps of the cylindrical shadow volumes happen to be visible.
- 1-Bit Stencil Buffer Mode: Tells the code to use a different algorithm that uses only 1 bit of stencil buffer, which won't allow overlapping shadows. If the device supports only 1-bit stencils, you'll be forced to use this mode.
- Z-Order Shadow Vols in 1-Bit Stencil Buffer Mode: The shadow volumes must be rendered front to back, which means that if you don't check this option, rendering might be incorrect.
Figure 12-1, Figure 12-2, and Figure 12-3 show three views of the scene generated by the ShadowVol2 sample application. You can see the shadows in Figures 12-1 and 12-3; Figure 12-2 illustrates the shadow volumes.
|
<<"F12xi01.eps">>
|
|
Figure 12-1. Shadow cast
|
| <<"F12xi02.eps">> |
|
Figure 12-2. Shadow volumes
|
|
<<"F12xi03.eps">>
|
|
Figure 12-3. Another view of the rendered shadows
|
The Code So Far
In this chapter, we didn't add any new code to the RoadRage project. To see these effects in action, refer to the ShadowVol and ShadowVol2 demo projects included in the DirectX samples.
Conclusion
In this chapter, you learned about stencil buffers and the exciting effects they can produce. In today's market, making your code stand out is a requisite if you want it to sell your applications and keep your users coming back for more. Incorporating strategic stencil-buffer effects into the introduction and into the body of a 3D real-time game might help you win over even the most discriminating game players.
In Chapter 13, we'll discuss how to load and animate 3D models. Creating animated, lifelike characters that your users can interact with is one of the most powerful capabilities you can add to any game.
Peter Kovach has been involved in computer software and hardware development since the mid-1970s. After 11 years in various levels of development and project management, he was eager to being pushing the envelope in 3D virtual world development. He currently words at Medtronic, where he is the project lead developming programmable, implantable medical devices that use a next-generation graphical user interface.