How to Optimize a Photogrammetry Mesh for Real-Time 3D

Photogrammetry reconstructs surface and color from overlapping photographs. Its dense output contains valuable captured evidence, but also noise, holes, floating components, uneven triangle density, baked illumination, and geometry outside the subject. A real-time release must preserve the useful evidence while removing acquisition artifacts and building a coherent delivery surface.

Never optimize the only reconstruction. Keep camera alignment, source images, dense model, coordinate system, and processing settings as immutable evidence.

Define the use and truth standard#

A museum record, ecommerce object, game environment, WebXR prop, and digital-twin measurement require different tolerances. Write which dimensions, wear patterns, colors, inscriptions, edges, and missing areas must remain faithful.

Record:

target runtime and devices;
physical or geospatial coordinate system;
dimensional tolerance;
normal and closest viewing distance;
expected instance count;
movable or selectable parts;
allowable inferred repair;
triangle, texture, material, byte, and memory budgets;
lighting and relighting requirements;
archival and attribution policy.

Distinguish captured data from reconstruction and artist interpretation. Filling the unseen underside of a statue can make a usable asset, but it is not measured evidence.

Preserve the reconstruction project#

Archive original photographs and metadata, masks, control points, camera calibration and poses, alignment components, dense or high-detail mesh, color layers, textures, coordinate reference system, scale constraints, software version, and processing settings.

Hash important inputs and exported source meshes. Keep the source in a lossless or sufficiently precise format before GLB conversion. Record units and axis transforms.

Export fixed source renders from front, rear, sides, top, detail areas, and acquisition boundaries. Include geometry-only, textured, and confidence or quality views where the reconstruction software provides them.

Do not depend on a delivery GLB as the archive. It may quantize positions, compress images, omit cameras, and lose geospatial precision.

Assess alignment before mesh cleanup#

Mesh defects can originate in poor camera alignment. Inspect reprojection error, camera coverage, control-point residuals, and component fragmentation before sculpting the result.

Double surfaces, smeared edges, and repeated features may indicate misaligned images rather than excess geometry. Return to alignment, masks, calibration groups, or input selection when the error is systematic.

Review texture-source images for motion blur, focus, exposure shifts, reflections, moving vegetation, and cast shadows. Disable images that harm texturing when the software permits, while preserving them in the archive.

If a 2-centimeter edge appears as 3 separated ridges, simplification will not know which ridge is true. Correct reconstruction evidence first.

Set scale and coordinates#

Use measured scale bars, surveyed control, known distances, or authoritative product dimensions. Report how scale was established and its residual error. For georeferenced scenes, retain a high-precision master and move the real-time asset into a local origin to avoid floating-point problems.

Choose a delivery origin and orientation for the runtime. Record the matrix from source coordinates to local delivery coordinates so annotations and future rescans can be transformed consistently.

Compare final width, height, and depth against the scaled source. A 0.5% uniform error over a 20-meter structure is 10 centimeters, which may be unacceptable for a measured twin but harmless for a background game prop.

Remove remote reconstruction fragments before calculating bounds. Keep a copy of removed evidence if the workflow requires auditability.

Segment subject from acquisition debris#

Use connected-component size, camera masks, spatial regions, and visual review to isolate the subject. Remove turntable, floor beyond the intended contact, background walls, vegetation motion fragments, photographer remnants, and floating noise.

Do not delete every small component. A buckle, tooth, leaf tip, wire, or broken archaeological fragment may be meaningful. Classify components with reference views.

Clean ragged acquisition borders. Preserve an original border mask and decide whether the release should retain an honest open boundary, close it with an inferred surface, or terminate it on a designed base.

For a terrain or building scan, split into streaming or culling regions before simplification if the runtime needs them. One 5-million-triangle object with a giant bound can defeat visibility even after compression.

Analyze topology without demanding perfection#

Report non-manifold edges, boundary loops, duplicate and degenerate faces, inverted normals, intersections, internal shells, and density variation. These guide operations but do not all require a watertight repair for rendering.

Photogrammetry commonly produces dense small triangles in noisy areas and broad triangles where evidence is weak. Visualize triangle size and reconstruction confidence. Smoothing a low-confidence region may be appropriate; smoothing an inscription can erase evidence.

Repair zero-area faces and broken normals needed for export. Fill only holes supported by the use case. For printing or physics, watertightness may be necessary; for a wall fragment, a fabricated rear cap can be separate and labeled.

Keep a high-detail cleaned source before decimation and retopology.

Choose simplification or retopology#

Use built-in simplification for fast static candidates and when the high mesh already has acceptable surface continuity. RealityScan's documented Simplify tool creates a new reduced model and supports absolute or relative targets, borders, density equalization, part merging, and reprojection options.

Use retopology for deformation, deliberate edge flow, modular assembly, semantic surfaces, severe noisy topology, or a hero asset requiring controlled density. CAD-like manufactured forms may benefit from remodeling primary planes and curves rather than decimating scan noise.

Generate several candidates. A 12-million-triangle object might be tested at 1 million, 250,000, 80,000, and 25,000 triangles. Compare target screen size and use; do not assume the smallest is the goal.

Protect silhouette, holes, chipped edges, inscriptions, relief, and contact geometry. Simplify flat or hidden areas aggressively. Freeze final triangulation before baking tangent-space maps.

Measure geometric loss#

Use point-to-surface distance from the high cleaned source to the candidate and in the reverse direction where useful. Report maximum, mean, percentile, and signed error with units. Visualize a heatmap.

Global numbers can hide important local loss. A 2-millimeter p95 error may be unacceptable on engraved text but irrelevant on a rough 10-meter rock. Define regions of interest and different tolerances.

Render silhouette differences from approved cameras. Inspect grazing highlights and parallax over relief. For a measured object, compare control dimensions after every transform and simplification.

Human review must distinguish removed capture noise from removed real damage or texture-bearing relief.

Build final UVs#

The dense reconstruction may have many texture charts designed for the source mesh. A simplified or retopologized mesh needs a new unwrap sized for the runtime.

Choose seams outside focal regions and reduce excessive chart fragmentation. Allocate texel density by captured detail and camera use. Preserve unique space for inscriptions, faces, damage, and other evidence. Give gutters enough pixels for the final mip chain.

Decide between one atlas and tiled or multiple texture sets based on platform size limits, streaming, and material partitions. A city block may need spatial tiles; a handheld artifact may fit one 2048 or 4096 atlas.

Validate overlap, stretch, orientation, occupancy, and region IDs before reprojection.

Reproject color and surface detail#

Project from the cleaned high-detail textured model to the final unwrapped mesh. RealityScan's Texture Reprojection workflow supports color, normal, and displacement-related output and is specifically intended for simplified or retopologized models.

Use supersampling and an appropriate sampling method. Inspect cage or ray correspondence, seams, occluded regions, and areas where the low surface diverges from ragged source geometry.

Generate a tangent-space normal map to preserve useful high-frequency surface direction, but do not bake gross shape loss into a normal map when it affects silhouette or parallax. Use displacement only when the target runtime deliberately supports and budgets it.

Retexture or reproject after simplification; RealityScan documents that a simplified model loses its original texture and keeps coloring until texture work is performed.

Keep full-quality projected masters before delivery compression.

Address baked illumination#

Photographs record light as well as material color. The texture may contain cast shadows, specular highlights, exposure variations, sky tint, and ambient occlusion. Under new lighting, these can double the shading.

Decide between faithful captured appearance and relightable material. For documentary visualization, keeping capture lighting may be acceptable. For games and product lighting, use color normalization, delighting, manual paint, or material reconstruction.

Do not invent metallic and roughness maps by mechanically converting brightness. Identify actual materials and author their response. A bright pixel can be white paint, a highlight on metal, or overexposure.

Compare neutral unlit color and relit PBR references. Document any color correction that changes evidence.

Optimize textures and materials#

Inventory texture dimensions, encoded bytes, decoded memory, chart occupancy, color space, and use. Remove unused pages and exact duplicates. Resize according to target texel density.

An 8192 × 8192 RGBA image is about 256 MiB decoded at the top level and roughly 341 MiB with mips. Four pages can exceed 1 GiB before compression. Real-time delivery usually needs smaller or GPU-compressed tiers.

Use KTX 2.0 or target-native formats when supported and configured. Compare normal-map and color quality separately. Preserve mips and sufficient gutter.

Combine material regions only when shader and streaming needs align. Keep transparent capture artifacts out of the final material unless transparency is real.

Build LOD, collision, and streaming data#

Create LODs from the approved high or master low source with consistent pivots, materials, and regions. Choose transition by projected error. For a scan environment, hierarchical spatial chunks may matter more than a few global LODs.

Build simple collision from interpreted surfaces. Do not use noisy scan triangles as physics. Separate walkable ground, blockers, and interaction targets.

For large sites, partition geometry and textures along streaming cells with overlap and seam policy. Validate that cell borders do not crack and that texture pages load before visible geometry.

Keep annotations and measurements mapped through stable coordinates, not raw vertex indices that change across LODs.

Validate capture fidelity and runtime behavior#

Review seams and coverage gaps#

Texture seams often indicate camera exposure differences, weak overlap, occlusion, or an unwrap boundary—not merely insufficient resolution. Inspect the contributing photographs and source-camera assignments. Disable a blurred or reflected view, normalize color where appropriate, and reproject before painting over the evidence.

Mark areas with no photographic coverage. Automated fill can produce a visually continuous texture, but it is inferred. For an archival asset, retain a mask or metadata layer describing those pixels. For entertainment use, an artist can paint them deliberately after the distinction is recorded.

At thin edges, verify that front and back projections do not bleed across the bake cage. Separate islands, tighten correspondence, or model actual thickness rather than accepting a muddy border.

Run format validation. Open the final artifact in independent viewers and the target engine. Compare fixed source and output renders, geometry distance heatmaps, and regional detail.

Test:

physical or geospatial dimensions;
source-to-local transform;
silhouette and acquisition boundaries;
protected evidence and inferred repairs;
normals, UV seams, reprojection, and mips;
captured versus relit material behavior;
LOD and streaming transitions;
collision and annotations;
cold load, memory, draws, and frame percentiles;
production population and device tier;
provenance and attribution display.

Archive the final hash and a derivation manifest connecting it to the reconstruction project.

Photogrammetry optimization checklist#

Source images, alignment, dense mesh, scale, settings, and rights are archived.
Truth, tolerance, inference, runtime, camera, and budgets are explicit.
Alignment defects are corrected before mesh editing.
Physical or geospatial coordinates and delivery transform are reproducible.
Acquisition debris is removed without discarding meaningful small features.
Topology analysis guides necessary repair without fabricating needless surfaces.
Simplification or retopology matches deformation, fidelity, and runtime roles.
Geometric and silhouette error are measured globally and by protected region.
Final UVs allocate density and padding to captured evidence.
Color, normal, and other detail are reprojected and reviewed.
Baked illumination, PBR interpretation, textures, LODs, streaming, and collision are deliberate.
Fidelity, performance, provenance, and target delivery all pass.

Sources and further reading#

Optimizing photogrammetry is evidence-preserving reconstruction work. Fix alignment before surface edits, distinguish captured data from inferred repair, build a controlled real-time mesh and unwrap, reproject from the preserved high source, and validate both dimensional fidelity and target performance.