Normals and Tangents After Mesh Simplification

Simplification changes triangles, and triangles define the surface directions used to build normals and tangents. A candidate can preserve vertex positions closely yet show dents, seams, or flashing highlights because its shading basis no longer matches the source or its normal map.

Treat shading reconstruction as part of simplification, not a cleanup checkbox afterward. Decide which edges are smooth, which are hard, which source normals should transfer, how tangents are generated, and whether the existing bake remains valid.

Geometric face normals#

For triangle positions p0, p1, and p2, an unnormalized face normal is the cross product:

(p1 − p0) × (p2 − p0)

Its direction depends on vertex winding. Its length is twice the triangle area. Normalizing produces a unit direction for lighting or comparison.

Three failure cases need handling:

coincident points produce zero-length edges;
collinear points produce zero area and no valid direction;
reversed winding points the normal the opposite way.

Simplification can create near-degenerate triangles even when exact zero area is filtered. A tiny cross-product magnitude makes the normalized result numerically unstable. Reject or clean triangles below a scale-aware area threshold.

Do not use one global threshold in raw units. An area of 0.000001 square meters may matter on a 2-millimeter component and be noise on a 2-kilometer terrain.

Vertex normals are an interpolation design#

Rasterization interpolates per-vertex attributes across a triangle. A vertex normal is not necessarily the normal of any one face; it defines a smooth shading field.

At a smooth edge, adjacent triangles share a compatible normal at the shared corner. At a hard edge, each side needs its own normal, creating separate rendered vertices even if XYZ is shared.

After a collapse, the incident triangle set changes. Retaining the old endpoint normal without reevaluation can point in a direction unsupported by the new surface. Recalculating by a naive equal average can let many tiny faces dominate one large face.

Common weighting approaches include:

equal face weighting;
face-area weighting;
corner-angle weighting;
area-and-angle combinations;
custom or transferred normals;
weighted-normal modifiers designed for hard-surface shading.

Angle weighting often approximates a stable smooth normal for irregular triangulation because the corner's angular contribution reflects how much of the local fan a face occupies. Area weighting reduces influence from tiny noise faces. Test on the asset rather than assuming one method matches the source tool.

Preserve the smoothing topology#

A simplifier needs to know which normal discontinuities are intentional. Sources express this through sharp-edge flags, smoothing groups, split-normal data, auto-smooth angles, or duplicated vertices.

Build an explicit edge classification before reduction:

smooth geometric edge;
hard shading edge;
UV seam that may also require tangent split;
material boundary;
open boundary;
detected crease without artist intent;
mirrored tangent discontinuity.

Hard edges can be locked, constrained to their feature line, or assigned a high attribute penalty. Allowing a collapse across them can round a corner even if positions remain close.

Conversely, treating every triangulation edge as hard makes a curved surface faceted and multiplies vertices. Inspect actual corner-normal equality rather than inferring hard shading from face boundaries.

Normal deviation as a constraint#

Before applying a collapse, predict normals for affected triangles. Reject it if any triangle flips or if direction change exceeds the policy.

For unit normals n0 and n1, angular deviation is arccos of their clamped dot product. Dot 1 means 0 degrees; dot 0 means 90 degrees; dot −1 means 180 degrees. Avoid calling arccos for every candidate when a cosine threshold is enough. For a 30-degree limit, require dot product at least cos 30°, approximately 0.866.

Separate face-normal change from vertex-normal change. Face normals capture geometry; vertex normals capture the chosen smoothing field. A flat panel can keep its face direction while custom normals shift.

Maximum normal change is conservative and sensitive to tiny faces. Report p95 plus maximum and inspect outliers. A strict 5-degree rule can block simplification of noisy scans unless noise is cleaned first.

Transferring source normals#

Instead of recalculating only from reduced faces, sample normals from the source surface. For each candidate vertex or corner, find a corresponding point and interpolate the source corner normals barycentrically.

Transfer can preserve the appearance of a dense smooth source on a coarse mesh, especially on beveled hard-surface assets. It needs robust correspondence:

restrict to the same semantic component;
prefer cage or normal-direction projection over unconstrained nearest point;
prevent transfer across thin opposing surfaces;
respect source hard-edge regions;
handle boundaries and gaps;
normalize the interpolated result.

Transferred normals can deviate substantially from the coarse geometric surface. Extreme deviations create dark shading, inverted lobes, or problems for effects that assume normals align with geometry. Limit the angle from the reduced face normal and inspect silhouettes.

Custom normals also become attributes that split vertices. Measure exported count.

Tangent space is built from geometry and UVs#

A tangent-space normal map stores directions relative to a basis at the surface: tangent T, bitangent B, and normal N. T generally follows increasing U, while B follows increasing V subject to handedness and convention.

For a triangle, tangent calculation uses position edge vectors and UV edge vectors. If the UV triangle has zero or near-zero area, the system is singular. Overlapping UVs are allowed across different faces, but each individual triangle still needs a usable mapping for tangent generation.

glTF stores TANGENT as four components. XYZ is the unit tangent and W is a sign. The bitangent can be constructed from cross(N, T) multiplied by W. This compact representation handles mirrored UV orientation.

Simplification changes position edges and sometimes UVs. Tangents generated before reduction no longer correspond exactly to the new triangles. Generate them after final topology, triangulation, normals, and UVs.

Why triangulation must be stable#

A quad has two possible diagonals. Normal-map baking and tangent generation operate on triangles. If the baker uses one diagonal and the exporter or runtime uses another, interpolation differs even though the quad outline is unchanged.

Triangulate before baking and do not run a topology-changing modifier afterward. Store the delivery triangles as the basis of approval.

For a non-planar quad, diagonal choice changes geometric surface as well as tangent interpolation. On a twisted panel, one diagonal can show a visible crease. Simplification should output triangles directly or a deterministic triangulation accepted before bake.

Hash the final index buffer in build reports. An unexpected hash change after a tool upgrade flags possible shading changes even when vertex count stays constant.

MikkTSpace and consistency#

MikkTSpace is a widely used tangent-space convention and reference implementation. A normal map baked with a Mikk-compatible basis should render with compatible tangents. Small implementation or input differences can still matter at seams.

Consistency requires:

identical final positions and triangle indices;
identical UV coordinates and set;
compatible vertex normals;
the same tangent-space algorithm and orientation convention;
correct tangent W handedness;
correct normal-map channel convention;
no exporter recomputation that changes inputs.

If the target runtime reliably generates MikkTSpace tangents from the shipped data, omitting tangents can save transfer bytes. It adds runtime work and places correctness on the importer. Shipping tangents avoids regeneration differences but uses 16 float bytes per vertex before quantization. Decide by platform profile.

The glTF specification defines tangent semantics and notes expected tangent calculation behavior when normal maps are used. Test actual loaders.

Mirrored UVs#

Mirroring reverses the UV orientation. The tangent W sign distinguishes the basis. Welding corners with opposite signs breaks normal mapping.

After simplification, report geometric positions that contain multiple tangent signs. They are not necessarily errors; they explain required rendered-vertex duplication.

Use an asymmetric tangent-space test normal rather than a uniform flat map. A directional brushed or colored normal pattern makes a mirrored-basis error obvious. Rotate a light through at least 180 degrees and inspect the seam.

If the texture itself is intended to mirror, tangent space still needs correct handedness. Do not “fix” an inverted lobe by flipping the whole normal map's green channel unless the convention is globally wrong.

Hard-surface weighted normals#

Beveled product models often use a small bevel plus custom normals that make large panels shade flat and let bevels carry the transition. Equal averaging can pull panel normals toward many bevel faces, creating wavy reflections.

Simplification can change bevel segment count and face areas. Reapply the weighted-normal strategy to the reduced topology or transfer approved normals. Protect the outer bevel width and corner geometry; normals cannot repair a silhouette or a bevel that collapsed to zero.

Review with:

a neutral glossy material;
long rectangular area-light reflections;
a moving bright line environment;
flat-shaded geometry view;
custom-normal visualization;
source and candidate reflection overlays.

Diffuse lighting hides panel waviness that reflective product materials reveal.

Normal maps after aggressive reduction#

An existing normal map assumes the low-poly cage used when baking. Moving the reduced surface changes ray correspondence and tangent basis. Mild simplification may retain acceptable appearance; aggressive reduction usually needs rebaking from the high-detail source.

Rebake when:

silhouette or major curvature changes;
UV islands move or repack;
hard-edge topology changes;
triangulation changes materially;
tangent generator changes;
projection errors become visible;
texture resolution or atlas changes.

Use a cage that encloses the high source without capturing unrelated surfaces. Compare tangent-space normal vectors at seams and examine the bake without base color.

Normal-map compression can create different artifacts from geometry. Keep encoding settings constant while evaluating a geometry candidate.

Transforming normals correctly#

Positions transform by the model matrix. Under non-uniform scale, normals need the inverse transpose of the linear transform to remain perpendicular to the surface.

For scale (2, 1, 0.5), multiplying a normal like a position produces the wrong direction. A runtime shader normally handles this with a normal matrix. Baking transforms into geometry before simplification requires recomputing or correctly transforming normals and tangents.

Negative scale reflects orientation and can reverse triangle winding and tangent handedness. Apply or preserve reflection intentionally, then verify culling and normal-map direction in the exported file.

Normalize after transformation. Interpolation and numeric compression can also move normals away from unit length; shaders often normalize before lighting, but stored-data validation should still report poor values.

A float normal has 3 components and 12 bytes; a float tangent has 4 components and 16 bytes. Across 100000 vertices, that explicit basis occupies 2800000 bytes before compression, so correctness and storage should be assessed together.

Quantization and compression#

Normals and tangents can use normalized integer formats or codec quantization rather than 32-bit floats. Reduced precision can create banding in broad specular highlights even when geometry appears unchanged.

Generate candidates at several bit depths. Compare:

maximum and p95 angular error;
moving highlight video;
seam differences;
encoded and decoded bytes;
runtime component type;
shader normalization behavior.

An 8-bit signed normalized component has far fewer possible directions than a 16-bit component, though vector normalization redistributes effective directions. The appropriate choice depends on material roughness and viewing distance.

Do not compare a simplified 8-bit-normal candidate with a source using float normals and attribute all visual change to topology.

Shading validation sequence#

Validate final triangle winding and area.
Render flat face normals with no textures.
Render recalculated smooth normals.
Render intended split or custom normals.
visualize normal vectors and discontinuities.
apply a glossy neutral material under moving light.
generate final tangents and render a flat normal map.
render an asymmetric tangent test map.
apply the production normal map and other materials.
export, reopen, and repeat in the target runtime.

This progression isolates geometry, normal, tangent, texture, and exporter problems. Jumping directly to the finished material makes attribution harder.

Metrics to retain#

For every candidate, store:

degenerate and flipped triangle count;
hard and smooth edge count;
rendered vertices after normal and tangent splits;
maximum and p95 face-normal deviation;
maximum and p95 vertex-normal deviation;
tangent orthogonality error;
count of invalid or zero-length bases;
mirrored-handness seam count;
image differences under fixed glossy lighting;
target-runtime screenshots and tool versions.

Check T dot N near zero, and lengths of T and N near 1 within chosen numeric tolerance. A non-orthogonal basis can be re-orthogonalized, but investigate why it drifted.

Common failures#

Recompute normals with every edge smooth. Designed hard edges become rounded.

Keep old tangents after topology changes. Normal maps use an obsolete basis.

Bake before triangulation. Exporter diagonal choices change interpolation.

Weld mirrored wedges. Tangent handedness is lost.

Transfer from nearest surface without component constraints. Thin opposing sides borrow wrong normals.

Judge under diffuse light. Specular waviness and seam inversions stay hidden.

Apply negative scale late. Winding and basis orientation change after approval.

Shading checklist#

Degenerate triangles are removed with scale-aware tolerances.
Winding remains consistent through coordinate conversion.
Hard and smooth edges follow explicit source intent.
Normal weighting matches the asset's surface style.
Transfer correspondence cannot jump components or thin layers.
Face and vertex normal deviation are measured separately.
Final triangles are frozen before baking.
Tangents use final normals and UVs.
Mirrored islands retain correct handedness.
Normal-map convention matches the target runtime.
Quantization is evaluated independently from topology.
Exported GLB shading passes moving-light review.

Sources and further reading#

Normals and tangents are derived delivery data with visible consequences. Rebuild them from the final topology under a documented convention, compare them with both geometric and image metrics, and approve the exported artifact rather than the pre-simplification viewport.