Metal #13 Tessellation

This note covers tessellation: subdividing a coarse patch into many small triangles on the GPU, for terrain, displacement and adaptive LOD.

Metal's tessellation pipeline is different

In D3D11 and OpenGL 4, tessellation is three fixed stages: hull shader → tessellator (fixed function) → domain shader.

Metal keeps only the last two. Tessellation factors are not computed by a shader stage; they are written into a buffer by an ordinary compute kernel.

Plain Textcompute kernel (yours) → factor buffer → tessellator (fixed function) → post-tessellation vertex function (yours)

The design is confusing at first but it is genuinely more flexible: computing factors becomes a normal compute dispatch, so it can be batched with other GPU work, reuse visibility data computed earlier, or be cached across frames.

Computing the factors

MSLkernel void tessellation_factors(
    device MTLQuadTessellationFactorsHalf *factors [[buffer(0)]],
    constant Patch *patches [[buffer(1)]],
    constant TessParams &params [[buffer(2)]],
    uint pid [[thread_position_in_grid]])
{
    float3 center = patches[pid].center;
    float  dist   = distance(center, params.cameraPosition);

    // closer patches get subdivided more
    float t = saturate(1.0 - (dist - params.nearDist) / (params.farDist - params.nearDist));
    float level = mix(1.0, params.maxTessellation, t);

    // four edges plus two interior directions
    factors[pid].edgeTessellationFactor[0] = half(edgeLevel(patches[pid], 0, params));
    factors[pid].edgeTessellationFactor[1] = half(edgeLevel(patches[pid], 1, params));
    factors[pid].edgeTessellationFactor[2] = half(edgeLevel(patches[pid], 2, params));
    factors[pid].edgeTessellationFactor[3] = half(edgeLevel(patches[pid], 3, params));
    factors[pid].insideTessellationFactor[0] = half(level);
    factors[pid].insideTessellationFactor[1] = half(level);
}

The edge factors are where this goes wrong. Two neighbouring patches share an edge; if they compute different factors for it, the generated vertices do not line up and a crack opens between them.

The fix is to make an edge's factor depend only on properties of that edge, not of the patch: compute it from the distance to the midpoint of the edge's two endpoints, so both patches derive the same number.

MSLfloat edgeLevel(Patch p, uint edge, constant TessParams &params) {
    float3 mid = (p.corner[edge] + p.corner[(edge + 1) % 4]) * 0.5;
    float  d   = distance(mid, params.cameraPosition);
    return clamp(params.maxTessellation / max(d, 1.0), 1.0, params.maxTessellation);
}

A factor of 0 culls the patch, which is a genuinely useful property: the same kernel can do frustum and back-face culling, and culled patches are never tessellated at all.

The post-tessellation vertex function

MSL[[patch(quad, 4)]]
vertex TessOut tessellation_vertex(
    patch_control_point<ControlPoint> control [[stage_in]],
    float2 uv [[position_in_patch]],
    texture2d<float> heightMap [[texture(0)]],
    constant Uniforms &u [[buffer(11)]])
{
    // bilinear interpolation for the position inside the patch
    float3 top    = mix(control[0].position, control[1].position, uv.x);
    float3 bottom = mix(control[3].position, control[2].position, uv.x);
    float3 pos    = mix(top, bottom, uv.y);

    // displacement: raise by the height map
    float2 texUV  = mix(mix(control[0].uv, control[1].uv, uv.x),
                        mix(control[3].uv, control[2].uv, uv.x), uv.y);
    float  height = heightMap.sample(heightSampler, texUV, level(0)).r;
    pos.y += height * u.displacementScale;

    TessOut out;
    out.position = u.viewProjection * float4(pos, 1);
    out.uv       = texUV;
    return out;
}

[[patch(quad, 4)]] declares a quad patch with four control points. [[position_in_patch]] is the parametric coordinate the tessellator produced — uv for quads, barycentric for triangles.

Note the level(0): sampling a texture in the vertex stage requires an explicit LOD, because the vertex stage has no derivatives.

Pipeline setup

SwiftpipelineDescriptor.tessellationFactorStepFunction      = .perPatch
pipelineDescriptor.tessellationPartitionMode           = .fractionalEven
pipelineDescriptor.tessellationOutputWindingOrder      = .clockwise
pipelineDescriptor.maxTessellationFactor               = 16

tessellationPartitionMode decides how non-integer factors are handled:

• .pow2 — powers of two only. Fastest, but LOD changes pop visibly.
• .integer — rounded. Still pops.
• .fractionalEven / .fractionalOdd — fractional factors allowed, so new vertices grow smoothly out of existing edges. This is the only way to avoid LOD popping, and terrain almost always uses a fractional mode.

Drawing uses a dedicated API:

Swiftencoder.setTessellationFactorBuffer(factorBuffer, offset: 0, instanceStride: 0)
encoder.drawPatches(numberOfPatchControlPoints: 4,
                    patchStart: 0, patchCount: patchCount,
                    patchIndexBuffer: nil, patchIndexBufferOffset: 0,
                    instanceCount: 1, baseInstance: 0)

Displacement and normals

Displacement mapping genuinely moves geometry, which is categorically different from normal mapping: it gives you a correct silhouette, correct self-shadowing and correct parallax.

There is a catch: after displacement the original normals are wrong. Three approaches:

1. Derive normals from the height map with finite differences over neighbouring texels. Simple, at the cost of three more samples.
2. Bake a normal map from a high-poly source in the DCC tool and sample it at runtime. Fastest and most common.
3. Use screen-space derivatives in the fragment stage: normalize(cross(dfdx(posWS), dfdy(posWS))). No extra samples, but it produces face normals and looks faceted.

When to use it, and when not

Tessellation is a clear win for terrain: one height map and a coarse grid give you adaptive geometry, dense up close and sparse in the distance, from very little data.

But be clear about the cost. The tessellation stage has limited throughput on many GPUs, especially mobile ones, and an over-generous factor makes it the bottleneck quickly. The rule of thumb is to keep the final triangles above about 8 pixels on screen — smaller than that and quad overhead eats most of the performance, at which point a denser static mesh would have been cheaper.

It is also worth knowing that the industry is moving on. Mesh shaders (object shader + mesh shader in Metal) provide a more general geometry-generation pipeline that can do everything tessellation does, plus things it cannot — meshlet culling, for instance. In a new project targeting hardware that supports them, mesh shaders are usually the better choice. Tessellation remains worth understanding because it is still widespread in existing engines, and because it is conceptually simpler.

The next note covers post-processing.