When Automatic UV Unwrapping Fails: Repack, Transfer, and Prove the Result

Automatic UV unwrapping is often described as a geometry operation: cut a mesh into charts, flatten the charts, pack them into an image, and continue. That description is incomplete for a textured production asset. A useful result must also preserve every rendered triangle, every material input, every sampling transform, and every visible feature after the texture has moved.

The distinction matters because an unwrap can be mathematically plausible and still produce black wedges around an eye, an untextured strip along a collar, or a missing patch on a nose. Those defects are not inevitable consequences of a dense mesh. They are evidence that one stage failed to preserve correspondence, coverage, padding, or material semantics.

The most useful lesson from building and debugging an automatic UV pipeline was this:

When topology is unchanged and the source UVs are valid, generating new geometric charts is not the only safe path. Repacking the authored charts, then transferring texture directly by source-triangle correspondence, can be both more reliable and more faithful. The pipeline should try both strategies under the same strict quality gates.

This article explains why that fallback works, where it can still fail, and what must be measured before an automatic system is allowed to publish the result.

UV rebuilding contains two different problems#

A UV atlas operation has at least two conceptually separate stages:

Chart generation and parameterization: decide where the mesh is cut, then flatten each connected surface region into two dimensions.
Chart packing: scale, rotate, and place the flattened regions into one or more texture pages with enough separation.

The distinction is visible in the xatlas API, which exposes chart computation and chart packing as separate operations. It also accepts an existing UV mesh through AddUvMesh when the caller wants to keep chart shapes and perform packing only. The watlas WebAssembly wrapper exposes the same separation to JavaScript and Node.js.

That API boundary suggests a useful recovery strategy. If new geometric parameterization fails, the system does not have to abandon the asset immediately. It can preserve the source chart topology and ask the packer to place those charts into a new padded atlas.

This is different from keeping the old UV image unchanged. The islands may move to new pages, receive new padding, and be baked into new textures. What remains stable is the source mapping inside each chart and, crucially, the triangle-to-triangle correspondence.

A failure case that changed the design#

Consider a static generated character with:

25,142 nondegenerate triangles;
one opaque base-color texture;
finite UV coordinates inside the 0–1 range;
no source UV triangle with exactly zero area;
2,138 authored UV islands.

A new geometry-driven unwrap was attempted at 1024, 2048, and 4096 pixels. Every size failed. The chart generator assigned only 25,071 of the 25,142 faces, leaving 71 valid 3D triangles without a usable atlas assignment. Small within-chart overlaps also appeared. Increasing resolution did not restore the faces because this was not fundamentally a texture-size problem.

The correct response was not to ignore 71 faces, paint their missing pixels black, or publish because 99.7% sounded close enough. A single omitted face on a prominent feature can occupy many screen pixels. The result had to remain rejected.

The same model already had usable source UVs, so a source-chart repack was tried. It reached every face, but nine very thin UV triangles collapsed to lines when packed. That exposed a second lesson: “valid in floating point” is not the same as “stable after texel-grid quantization.”

The final path isolated under-resolved source triangles for packing, gave only their temporary destination charts a stable shape, retained their original UVs for sampling, and repacked the asset. The accepted result assigned all 25,142 triangles, produced zero degenerate destination triangles, zero out-of-range coordinates, zero measured overlaps, and complete raster coverage.

This is one production case, not a benchmark for all models. Its value is diagnostic: it separates three failure classes that a generic “No safe result” message can otherwise hide—geometric parameterization loss, texel-grid collapse, and visual resampling differences.

Why glTF seam splits complicate geometric unwrapping#

In glTF, UV coordinates are vertex attributes. The core specification defines TEXCOORD_n as a two-component attribute, and all attributes on a primitive must have matching accessor counts. It also defines tangents, normals, vertex colors, joints, and weights in the same per-vertex attribute system; see the glTF mesh attribute specification.

A point on the rendered surface may need several vertex records at the same XYZ location. The records can differ because:

the point lies on a UV seam;
a hard normal separates shading across an edge;
mirrored charts require different tangent handedness;
material or primitive boundaries duplicate an otherwise shared point;
another vertex attribute is discontinuous.

An atlas library sees the submitted index and vertex buffers, not the artist's original topological intention. If seam-split records are treated as unrelated geometry, a connected cheek may look like many disconnected fragments. A chart generator can then create too many islands, fail to infer adjacency, or parameterize tiny pieces poorly.

xatlas explicitly notes that UV seams can add vertices while the output retains the same number of indices. That is expected behavior: one geometric location may need more than one delivered vertex because it has more than one UV value. A topology-preserving UV rebuild should therefore permit seam-related vertex duplication while requiring the rendered triangle surface, winding, positions, normals, and transforms to remain unchanged.

Scale-relative colocal detection can reconnect records that share a position closely enough for chart planning. It should operate on temporary planning data, not weld the delivered mesh. The tolerance must follow model scale: a fixed epsilon meaningful for a one-meter character can be destructive for a one-millimeter part and ineffective for a kilometer-scale scene.

Colocal detection improves the geometry path, but it cannot guarantee that every irregular or generated mesh will parameterize successfully. That is why a second chart strategy remains necessary.

Fresh charts and source-chart repacking solve different needs#

Use fresh geometry-derived charts when:

the source has no UVs;
source UVs are missing on some primitives;
source UV triangles are truly degenerate;
the source intentionally overlaps or tiles and the target requires unique bake space;
the old layout contains pathological fragmentation that must be redesigned;
topology has changed and the old mapping no longer corresponds to the target.

Use source-chart repacking as the preferred or fallback path when:

topology is unchanged;
every nondegenerate source face has finite, nonzero UV area;
the source appearance is already correct;
geometric charting omits faces, overlaps charts, or exceeds the resource budget;
fidelity is more important than reducing island count;
direct triangle correspondence is available.

Repacking is not universally better. A source with thousands of tiny islands can waste large amounts of padding. Repeated, mirrored, or deliberately overlapping source UVs may need charts to be separated before baking. A material using repeat wrap can sample outside the normalized image; the glTF sampler specification defines repeat, mirrored-repeat, and clamp-to-edge behavior, so a transfer tool must reproduce the source sampler before changing the output to a clamp-safe atlas.

Likewise, KHR_texture_transform can apply offset, rotation, scale, or a different texture-coordinate set. Sampling the raw TEXCOORD_0 values while ignoring that transform bakes the wrong image even when the atlas itself is perfect.

The planner should inspect those conditions before expensive work. Unsupported material extensions, procedural graphs, or undecodable images should fail preflight with a specific reason instead of being flattened approximately.

Preserve correspondence with an explicit source reference#

The safest same-topology transfer does not search for the source surface. It already knows it.

For each atlas output vertex, preserve a reference to the source vertex from which it was created. xatlas calls this relationship an xref. For each output triangle, map its referenced corner set back to the corresponding source triangle. Store the original UV coordinates temporarily alongside the new atlas UVs.

Then rasterize each destination triangle:

Find a sample point in the destination atlas triangle.
Compute its barycentric coordinates relative to that destination triangle.
Apply the same barycentric weights to the source triangle's original UV coordinates.
Apply the source texture-coordinate set, KHR_texture_transform, wrapping, and filtering.
Evaluate the material factors and vertex colors.
Write the resulting value into the destination channel.

No ray is needed because no geometric correspondence is being guessed.

This avoids a common failure on thin or layered surfaces. A nearest ray fired from an eyelid may hit an eyeball; a ray from a collar may hit the neck; a cage expanded through close layers may select the wrong shell. Ray projection remains necessary when retopology changes the surface, but same-topology UV rebuilding should not accept that extra ambiguity.

Direct transfer also makes failure accounting exact. Every destination triangle has one known source triangle. If a face receives no destination page, no raster sample, or no source mapping, the job can identify the missing face instead of hiding it inside a mostly complete bake.

The production transfer used 4 samples per destination texel. That was a measured speed-quality compromise rather than a rule for every renderer: sampling cost scales with written texels and active channels, so the chosen pattern should be tested against thin charts, high-frequency color, normal maps, and the job's time budget.

Thin UV triangles need a texel-space test#

A source UV triangle can have positive area and still be unstable at delivery resolution. Imagine a long triangle whose minimum altitude is 0.2 texel. Its floating-point area is nonzero, but integer or texel-aligned packing can place all three destination vertices on the same row.

Test at least two quantities in texel space:

triangle area in texels, not just normalized UV area;
minimum altitude in texels, which catches long, needle-shaped triangles.

If a valid 3D face is under-resolved, isolate it as a temporary packing chart and assign that chart a small stable triangle. Preserve the source xrefs and original UV coordinates. The temporary chart controls only where destination samples are stored; the source image is still sampled through the original triangle.

This technique should be bounded. If thousands of faces require large replacement charts and the atlas no longer fits its page or texel budget, the safe answer may be retopology. The intervention is meant to prevent quantization collapse, not to disguise a fundamentally unusable parameterization.

Do not solve this by lowering the degenerate-triangle gate. A zero-area destination triangle has no interior to rasterize and cannot carry a reliable texture region.

Texture transfer must preserve material semantics#

Base color is only one part of a glTF appearance. Core metallic-roughness materials can use:

base-color RGB and alpha;
metallic-roughness data;
tangent-space normals;
occlusion;
emissive color;
material factors;
vertex color;
alpha mode and cutoff;
double-sided rendering.

The glTF material specification requires base-color and emissive RGB to use the sRGB transfer function, while metallic-roughness and normal data are linear. It also states that correct base-color filtering should decode the transfer function before linear interpolation. A transfer tool that treats every image as ordinary RGBA bytes will change color and scalar values even when UV correspondence is exact.

Material factors and vertex colors are part of the rendered result. glTF defines COLOR_0 as an additional linear multiplier on base color. If page materials are consolidated, differing factors must either remain separate or be baked into the destination textures.

Normal maps need special treatment because their vectors are expressed in the source tangent basis. The glTF specification recommends default MikkTSpace tangent generation when tangents are absent, and defines tangent W as the handedness used to reconstruct the bitangent. The MikkTSpace reference implementation exists specifically to give baking tools a common tangent-space convention.

When UVs change, old tangents are no longer valid. A robust transfer decodes the sampled source normal into the source tangent frame, converts it through object or surface space, expresses it in the new MikkTSpace frame, and writes regenerated destination tangents. If the source had no normal map and geometry did not change, do not invent one.

Padding is an appearance requirement#

Packing charts without overlap is not enough. Bilinear filtering reads neighboring texels. Mipmaps combine progressively larger neighborhoods. Without duplicated border colors, a chart edge can blend with a black, transparent, or unrelated atlas region.

Blender's Pack Islands documentation describes packing existing islands while maintaining a requested margin. The destination workflow must also dilate actual channel values through that allocated margin.

Use channel-safe unused values:

neutral tangent-space normal for normal pages;
nonmetal and a deliberate roughness default for metallic-roughness pages;
fully unoccluded for occlusion;
black for emissive;
alpha appropriate to the material mode for base color.

Then dilate every written channel through the full padding allocation. Export atlas samplers as clamp-to-edge with mip-capable linear filtering when all required repeat behavior has already been baked.

Padding should be specified in pixels at the selected page size. A normalized margin silently changes meaning when the atlas moves from 1024 to 4096. An automatic planner can increase padding with resolution—for example, 4 pixels at 1K, 8 at 2K, and 16 at 4K—while treating any user value as a minimum. Those numbers are a pipeline policy, not a universal law; validate them against the intended mip range and renderer.

Quality gates must describe the whole result#

“The unwrap function returned” is not a quality gate. Before texture transfer, require:

every nondegenerate source face assigned to exactly one destination page;
zero destination UV triangles with zero area;
zero out-of-range destination coordinates;
zero measured interior overlap;
page count and total texels within budget;
adequate triangle-area distribution in texels;
positive padding.

After transfer, rasterize an independent UV mask and compare it with the bake coverage. Require essentially complete coverage and measure the largest connected missing region. A ratio alone is insufficient: four adjacent missing pixels can form a visible wedge while four isolated misses may disappear under filtering.

Keep output structural checks separate. Triangle count, positions, normals, node transforms, and scene structure should match for a same-topology rebuild. More vertex records and more primitives are allowed when UV seams and atlas pages require them.

An accepted production fixture used four 2048 pages with eight pixels of padding. All 25,142 faces were assigned, overlap and destination degeneration were zero, and the coverage mask reported no missing pixels. The output had more vertices and four material primitives because seam duplication and page partitioning are legal parts of the delivery representation.

Render validation needs more than one score#

Structural UV checks cannot prove that the material still looks right. Render source and result from fixed cameras with identical framing, lighting, background, resolution, and material implementation. Include front, back, side, and elevated views so close surface layers are not hidden.

Structural Similarity, or SSIM, compares local image structure rather than relying only on per-pixel error. It is useful, but no single mean score should decide the job. A small black patch can be diluted by a large unchanged background, while harmless texture resampling can lower local SSIM across high-frequency fabric.

Use several complementary measurements:

mean masked SSIM across the foreground;
worst-view masked SSIM;
worst local-window SSIM;
mean absolute pixel difference;
connected clusters of newly dark pixels;
silhouette difference when topology changes.

The case that motivated this article produced a mean masked SSIM of 0.9780, a worst view of 0.9727, and a worst local window of 0.7415. Those SSIM values initially failed. Yet mean absolute pixel difference was 0.000526 on a 0–1 scale, and the largest new-dark cluster was one pixel. Fixed-camera inspection showed minor, widespread resampling differences rather than a missing texture region.

That does not justify deleting the SSIM gates. A bounded equivalence rule can require extremely low absolute difference, a high mean and worst-view floor, a local-window floor, and no meaningful dark cluster. If any one of those safeguards fails, reject the result. The rule distinguishes small resampling noise from the original defect class without allowing actual tears to publish.

Human review remains valuable for important assets, but the automated report should make review efficient: show source and result side by side, include difference images, list every gate, and identify the exact failing view or connected region.

A reliable automatic decision sequence#

A practical same-topology UV rebuild can follow this order:

Preflight appearance. Confirm static supported geometry, decodable images, supported glTF channels and extensions, and valid required UV sets.
Preserve source data. Keep original UVs, material assignment, factors, vertex colors, transforms, and tangent information until transfer finishes.
Try the preferred chart method. Use geometry-derived charts when a cleaner unwrap is valuable; use source-chart repacking first for dense valid generated assets where segmentation cost dominates.
Run bounded candidates. Increase page size and texture cost while preserving padding and a hard total-texel budget.
Validate the atlas. Reject missing faces, overlap, degeneration, out-of-range coordinates, inadequate texel area, or excessive pages.
Fallback by cause. If geometric charts fail and source UVs are valid, repack source charts. Regularize only under-resolved temporary packing faces.
Transfer directly. Rasterize destination triangles and sample corresponding source triangles barycentrically.
Rebuild appearance. Preserve color spaces, factors, transforms, alpha behavior, channel packing, tangents, and samplers.
Measure coverage. Compare an independent UV mask with written pixels and connected missing regions.
Render fixed views. Evaluate global, per-view, local, absolute-difference, and dark-cluster metrics.
Publish or fail closed. Unsupported inputs should end in preflight rejection. Exhausted layout, transfer, or visual retries should end with no safe result, not a partially textured asset.

This sequence is intentionally conservative. Automatic processing creates trust only when failure remains visible.

When retopology is the honest answer#

Source-chart repacking is not a substitute for retopology. Recommend new topology when:

no candidate fits the texture and page budget;
source UV fragmentation spends most texels on padding;
many faces remain subtexel at the maximum useful resolution;
the source has non-manifold or self-intersecting regions that prevent stable parameterization;
geometry must change for animation, deformation, editing, or runtime cost;
ray-based transfer can be constrained and validated more reliably than preserving the old surface.

Retopology changes correspondence, so its transfer path needs measured source-to-target distances, bounded cages or rays, wrong-surface protections, coverage masks, and silhouette checks. Do not reuse same-topology assumptions after the triangles change.

UV rebuild checklist#

Source and destination topology contracts are explicit.
Chart generation and chart packing are treated as separate stages.
Colocal planning reconnects seam-split positions without welding output geometry.
Valid source charts are available as a bounded fallback.
Every output vertex and triangle can be traced to its source.
Thin source triangles are tested in texel space, including minimum altitude.
Same-topology transfer uses barycentric correspondence rather than rays.
Texture transforms, coordinate sets, wrapping, filtering, factors, and vertex colors are evaluated.
Base color and emissive use correct sRGB handling; data maps remain linear.
Normal maps are converted to the new MikkTSpace basis and tangents are regenerated.
Every page has positive pixel padding and full channel-aware dilation.
Every nondegenerate face is assigned; overlap, degeneration, and out-of-range counts are zero.
Coverage and largest missing connected region are measured.
Fixed-camera visual tests include both SSIM and direct defect measurements.
Unsupported or exhausted jobs fail closed instead of publishing as ready.

The central idea is simple: automatic UV work should optimize for a provably complete rendered surface, not for the mere existence of a new UV accessor. Fresh unwrapping and source-chart repacking are complementary tools. Direct correspondence removes unnecessary projection risk. Texel-space and rendered-image gates decide whether the result is safe.