Presenter

Quanyun Wu, MASc candidate in Systems Design Engineering

Abstract

Feed-forward monocular 3D reconstruction networks estimate dense geometry rapidly and without fragile feature matching. Their output, however, has no metric scale and is not separated into individual objects or structural units. Existing approaches either reconstruct such scale-free, fused geometry or generate individual meshes without metric scale or a consistent position in the scene, which limits their use in embodied artificial intelligence and remote sensing.

This thesis presents a framework that converts feed-forward reconstructions into metric, instance-separated 3D representations. A dual-track mechanism recovers metric scale from a reference object whose dimensions are estimated by a vision-language model indoors, and from a reference camera trajectory outdoors. The metric, gravity-aligned geometry then constrains 2D foundation models. In object-centric scenes, tracked instance masks are projected onto the reconstruction and clustered into objects. In repetitive layouts, wall planes extracted from the reconstruction restrict segmentation to one unit at a time, which reduces mask spillover between adjacent instances.

The framework is evaluated in two indoor applications and an exploratory outdoor extension. In a kitchen scene, recovering metric scale and registering generated meshes to the reconstruction raises the mean novel-view semantic IoU from 0.0246 for the SAM3D baseline to 0.5715. In a repetitive office scene, repeated cubicles are registered into a shared canonical frame, so that the observations of one cubicle complete the occluded parts of the others. Compared with TSDF fusion, VGGT, DUSt3R, Hunyuan3D-2, TRELLIS, and SAM3D, only the proposed framework produces geometry that is metric, separated per object, and watertight, recovering all 23 cubicles as individual solids. Outdoors, the trajectory-aligned reconstruction supports island area estimation with a mean absolute error of 1.0\%, compared with 7.1\% when COLMAP replaces the feed-forward reconstruction under the same alignment and 7.8\% when SAM3 replaces the land segmentation. It also supports individual tree segmentation in two forest stands.

These results indicate that metric scale and instance separation can be added as separate steps on top of an unchanged feed-forward reconstruction, which allows metric, per-object 3D assets to be produced from monocular video without ground control points or active depth sensors.

Join us in EC4-2101A!

This seminar counts towards the graduate student seminar attendance milestone!