TUN3D: 3D scene understanding from video on your phone

▤ Summary

This article introduces TUN3D, a 3D scene-understanding model that works from video captured on a phone. The system targets indoor scene understanding for use cases such as AR/VR and reconstructs layout, detection, and 3D segmentation from sparse video inputs. It combines a 3D sparse convolutional backbone with specialized heads for layout and object detection, and it uses camera extrinsics plus multi-view information to improve results. The model reports strong performance on ScanNet, S3DIS, ARKitScenes, and Structured3D, and it outperforms several prior methods on key metrics.

▦ Linked Companies