Conference Agenda

This invited talk presents Digital Building Analysis (DBA), a unified framework for intelligent, cloud-based building-scale reconstruction and analysis. Building on our prior advances in Gaussian Splatting for photorealistic 3D scene generation and the Gaussian Building Mesh (GBM) framework for accurate mesh extraction, DBA introduces a new layer of integration between cloud mapping and artificial intelligence. The system connects directly to Google Maps Platform APIs to retrieve geospatial data, imagery, and elevation models from a building’s address or coordinates, while employing Gaussian Splatting to reconstruct high-fidelity 3D models from multi-view imagery. This combination enables seamless digital twin creation without ground-based measurements or proprietary datasets and can be operated through natural language queries, allowing users to simply describe a location or request a building analysis conversationally.

The key component of DBA leverages multi-agent large language models (LLMs) for both natural language interfacing and data interpretation. These models autonomously generate Google Maps API calls, interpret retrieved imagery, extract visual features, and compose semantic building descriptions. Working in tandem, the agents merge 3D geometry, visual realism, and semantic understanding into a single automated process.

Together, these innovations mark a major step forward in Canada’s AI-enhanced remote sensing research, enabling interactive, query-driven urban analytics and advancing the next generation of intelligent digital twins for sustainable urban development.

3D building models are powerful visual tools, typically generated with well-established image-matching or LiDAR methods. However, they do not capture the view-dependent colour characteristics possible with Gaussian splatting. Despite the visual potential of Gaussian splatting, there is limited knowledge on its spatial accuracy and influencing factors, particularly for buildings. To address this gap, a two-building dataset was collected with terrestrial laser scans, images, phone LiDAR, and target points, and the visual and spatial effects of numerous factors were analyzed. These factors included the source and quality of the input camera poses and point cloud, the number of images and training iterations, and the Gaussian splat method. Gaussian splats were trained from open source and commercial image-based reconstruction methods, COLMAP and Pix4D, and phone LiDAR reconstructions. Applying Gaussian splatting to these inputs had minimal impact on the target points and the overall structure of the buildings, but the positions of Gaussians deviated from the initial point cloud, particularly before 15,000 iterations, resulting in more floaters and lower spatial accuracy. Image-based reconstruction methods outperformed phone LiDAR methods on visual and spatial metrics. Cleaning COLMAP point clouds considerably decreased Gaussian floaters, while downsampling input point clouds increased the percentage of floaters and yielded similar visual results. 2D Gaussian splatting provided geometric constraints, removing some floaters, but sacrificed visual quality. Increasing the number of images to three loops around the building improved visual and spatial results. Overall, the spatial accuracy of building Gaussian splatting was heavily dependent on the factors studied.

Urban Digital Twins (UDT) are vital tools for smart city development, enabling data-driven management and analysis of urban infrastructure (Sabri and Witte, 2023). A persistent challenge in realizing the potential of UDTs is the interoperability of disparate geospatial datasets, particularly camera imagery and sensor data, requiring precise synchronization, georeferencing, and integration. Existing implementations often

rely on costly, proprietary hardware, limiting scalability and adoption, especially for organizations constrained by limited budgets (Thakkar et al., 2025). This research addresses the need to develop a cost-effective, standardized framework to capture, integrate, and standardize camera imagery and geospatial metadata for Machine Learning (ML)-driven analysis within spatially enabled UDTs.

Roof material identification is a critical component of energy-aware 3D city modeling, supporting applications such as thermal analysis, climate resilience, and digital twins. Traditional approaches relying solely on aerial imagery struggle with shadows, low contrast, and spectrally similar roof materials. This study introduces a dual-branch deep learning framework that combines high-resolution aerial orthoimages with GoPro-based street-view imagery to overcome these limitation and improve roof material classification.

The aerial branch employs a ResNet-18 model fine-tuned on 120 manually labelled roof samples in New Brunswick, Canada, covering four material classes: asphalt, metal, membrane, and gravel. The street-view branch utilizes GoPro field-survey images, where roof regions are extracted using the Segment Anything Model (SAM) before classification with a second ResNet-18. Although street-view imagery captures only materials visible from ground level, it offers rich textural information that complement nadir imagery.

Because the two modalities are unpaired, fusion is performed at the decision level using learnable weights to combine the softmax probabilities of both branches. Experimental results show that street-view imagery achieves 90.9% accuracy, outperforming aerial imagery alone (77.8%). The combined bimodal framework leverages complementary modality strengths, resulting in improved detection performance for all roof material classes.

This study investigates the comparative performance of state-of-the-art 2D and 3D feature-detection models applied to multimodal airborne datasets for digital-twin generation. Using RGB, LiDAR, and nighttime thermal imagery collected over the University of New Brunswick’s Fredericton campus, a fused RGB–LiDAR–thermal point cloud was created to support building-scale analysis of energy-relevant features, specifically windows and doors. Three 2D object-detection models Faster R-CNN, Mask R-CNN, and YOLOv8 were applied to both RGB and thermally registered imagery, incorporating phase-congruency-based alignment to address differences in sensor geometry and spatial resolution.

Complementing the 2D analysis, three 3D semantic-segmentation models KPConv, PointCNN, and RandLA-Net were implemented to evaluate geometry-driven, order-aware, and scalable point-cloud classification strategies using multimodal attributes. The dataset was divided into 70% training and 30% testing, and evaluated using standard metrics such as accuracy, mean Intersection-over-Union, and per-class F1 score. Preliminary results for the 2D methods have been realased in the abstract, with further evaluation of all models currently underway.

The objective of this work is to establish a unified framework for understanding how 2D and 3D feature-detection approaches perform under low-light and thermally dominant conditions, where conventional RGB-based workflows often fail. The outcomes of this study will support improved digital-twin development for building-energy diagnostics and contribute to future thermal-efficiency modeling workflows in partnership with the National Research Council of Canada.