UC Berkeley's 3D-from-2D trick cuts cost by refining only occupied space

Inferring the 3D shape of an object from a single small 2D image is genuinely hard for computers, and useful: it would help object tracking in AR and VR and creative workflows. The obstacle is arithmetic. A 100-by-100-pixel image maps to a million voxels in a 100-cubed volume, and reaching 128 pixels takes about two million, with every voxel classified as empty or full by analyzing the original image.

The straightforward approach was to compute that whole high-resolution volume, and its cost is what had held back extrapolating 3D forms from 2D images. Christian Häne, a researcher at the Berkeley Artificial Intelligence Research lab, realized the full volume was never actually needed: the goal is only to describe the surface of an object, and the empty space around it and inside it does not matter.

His system starts with a very low-resolution 3D reconstruction, which already shows, for example, that the outer third of the volume is empty, so that space is thrown away. It then renders the region that was kept at higher resolution, finds the next layers of empty space, discards them, and repeats. After a few passes it has a high-spatial-resolution volume produced with comparatively little calculation, because it only ever computes parts it knows contain meaningful information.

Models built this way were compared with ones produced by more traditional means and generally appeared as good or better while requiring far less computation, with more examples in the paper. The TechCrunch coverage was careful about the limits: it is far from a complete solution and humans remain orders of magnitude better, but the workaround mirrors how human vision throws away data it deems superfluous.

Render a coarse volume first, throw away empty space, then refine only what remains

A coarse pass costs almost nothing yet reliably shows where the large empty regions are, so most voxels are eliminated before they are ever computed at high resolution.

Empty space dominates a bounding volume, so concentrating computation on the surviving region focuses the expensive work exactly where structure exists.

Quality does not collapse because each refinement re-renders only what the previous coarser pass showed to be non-empty, keeping error where the algorithm keeps working.

The trick turns one unaffordable full-volume problem into repeated cheap subproblems, so cost scales with the surface being described rather than the whole box.

When most of the solution space is empty, do a cheap coarse pass first, discard what cannot matter, and spend the expensive computation only on what survives.

The paper and a Berkeley AI Research blog post appeared in August 2017, and TechCrunch's write-up, which called the solution computationally clever and forehead-slappingly simple, drew a 161-point Hacker News discussion. Reported comparisons found the hierarchical models as good as or better than traditional full-volume reconstructions with far less computation. The coverage called it a workaround, not a finished answer: humans remain orders of magnitude better at single-image 3D inference, and the method works by borrowing the brain's habit of discarding superfluous data.