Automating Building Feature Recognition in Digital Twin Models Using Computer Vision

Author

• Jaron Harbison, Utah State University

Date of Award

5-2026

Degree Type

Thesis

Degree Name

Departmental Honors

Department

Mechanical and Aerospace Engineering

Abstract

Standard 3D scans treat all surfaces as generic geometry, but Wi-Fi signals don't. A signal drops by a different magnitude through a wooden door than a concrete wall. Without labeling these materials, Internet Service Providers (ISPs) cannot accurately predict "dead zones". This project addresses that gap by creating a machine learning model that identifies structural and architectural features. This includes features such as walls, doors, windows, and columns directly from RGB-D (Red, Green, Blue, Depth) point cloud data. 

I constructed the model on a custom U-Net architecture featuring a symmetric encoder-decoder structure with skip connections, trained on the StructScan3D dataset to identify seven material classes: Background, Wall, Furniture, Door, Window, Structural Beam, and Miscellaneous Obstructions. Input images were resized to 320×240 pixels and normalized. 

A class-weighted Cross-Entropy loss function was employed to counteract severe class imbalance in the training data, where Wall and Background pixels dominate and rare classes such as Structural Beams would otherwise be ignored. The Adam optimizer was used with a learning rate of 1×10^⁻⁴, and the best-performing model checkpoint was saved based on minimized validation loss. At inference time, predictions are rendered as 40% opacity color overlays on raw scan images, enabling direct visual validation of material identification. 

The model achieved consistent convergence and successfully classified all seven target categories. Class-weighted loss proved to be the pivotal design decision, enabling the model to learn discriminative representations for rare but physically significant features. Future work will integrate monocular depth estimation to synthesize depth cues from 2D images, compensating for hardware limitations and enabling full exploitation of the StructScan3D dataset.

Recommended Citation

Harbison, Jaron, "Automating Building Feature Recognition in Digital Twin Models Using Computer Vision" (2026). Undergraduate Honors Capstone Projects. 1111.
https://digitalcommons.usu.edu/honors/1111