LiP-Map: Interacted Planes Reveal 3D Line Mapping

3D line mapping from multi-view RGB images provides a compact and structured visual representation of scenes. We study the problem from a physical and topological perspective: a 3D line most naturally emerges as the edge of a finite 3D planar patch. We present LiP-Map, a line-plane joint optimization framework that explicitly models learnable line and planar primitives. This coupling enables accurate and detailed 3D line mapping while maintaining strong efficiency (typically completing a reconstruction in 3 to 5 minutes per scene). LiP-Map pioneers the integration of planar topology into 3D line mapping, not by imposing pairwise coplanarity constraints but by explicitly constructing interactions between plane and line primitives, thus offering a principled route toward structured reconstruction in man-made environments. On more than 100 scenes from ScanNetV2, ScanNet++, Hypersim, 7Scenes, and Tanks&Temple, LiP-Map improves both accuracy and completeness over state-of-the-art methods. Beyond line mapping quality, LiP-Map significantly advances line-assisted visual localization, establishing strong performance on 7Scenes.

Our approach leverages the geometric synergy between line segments and the edges of 3D planar primitives. By applying any existing 2D line segment detector to multi-view input images, a set of 3D planes is optimized with two key learning objectives:

• The 3D planes should align as closely as possible with the 2.5D depth/normal maps from the input images.
• The edges (or boundaries) of the 3D planes should be consistent with the observed 2D line segments.