The only game I know that sets the stencil polygon attribute, is sega rally2. And it's used for the shadow under the car.
However I had long suspected there was something in the culling nodes themselves that set some sort of stencil mask. The API has a clue in this regard
void SetGroupLayerNumber ( long layer_number ) ;
Description
Sets the layer number for group layering. The group layer number sets the drawing priority for all polygons under the culling node.
The important bit
When group layering is used, the geometry being layered must all be coplanar. In addition, all culling nodes whose drawing order is to be controlled by this attribute must be children of the same parent node.
The SDK has a few clues as well
#ifdef NEW_MODEL
#ifdef BIG_ENDIAN
unsigned long tex_lod_tbl_addr : 32 ;
unsigned long spare3 : 21 ;
unsigned long flip_polygons : 1 ;
unsigned long shadow_only : 1 ;
unsigned long valid_shadow_id : 1 ;
unsigned long shadow_object_id : 8 ;
#else
These aren't used on the model3, but it's clear they put some sort of shadow code into the culling nodes on later hardware. But what about the model3 ...
This is what the culling node data looks like (from mame)
x------- -------- -------- -------- Is UF ref
-x------ -------- -------- -------- Is 3D model
--x----- -------- -------- -------- Is point
---x---- -------- -------- -------- Is point ref
----x--- -------- -------- -------- Is animation
-----x-- -------- -------- -------- Is billboard
------x- -------- -------- -------- Child is billboard
-------x -------- -------- -------- Extra child pointer needed
-------- -----xxx xxxxxx-- -------- Node ID
-------- -------- -------- x------- Reset matrix
-------- -------- -------- -x------ Use child pointer
-------- -------- -------- --x----- Use sibling pointer
-------- -------- -------- ---x---- No matrix
-------- -------- -------- ----x--- Indirect child
-------- -------- -------- -----x-- Valid color table
-------- -------- -------- ------xx Node type (0 = viewport, 1 = root node, 2 = culling node)
But you can see there are 5 unused bits in there in the culling node
-------- ?????--- -------- --------
Anyway I did a bit of debugging with virtua fighter and just dumped the raw values for the culling nodes.
I found two that looked like this
node - 8177d412 node id 501
node - 8177d812 node id 502
Those node values set some of those unused bits. Anyway I stepped through the polys and realised they were the shadow polys. That's can't be a coincidence ..
More investigating required
