Cgcvit Achieves up to Improvement in Assessing Urban Heat Resilience with UAV and Street-View Imagery

Scientists are tackling the growing threat of urban heat islands, a problem acutely felt in densely populated areas of the Global South, where vulnerable building materials amplify heat exposure! Led by Steffen Knoblauch and Ram Kumar Muthusamy from Heidelberg University, alongside Hao Li of the National University of Singapore, and with contributions from Iddy Chazua, Benedcto Adamu, and Innocent Maholi at OpenMap Development Tanzania, et al., this research presents a novel machine learning framework to assess building heat resilience at scale! By fusing unmanned aerial vehicle (UAV) and street-view imagery with a coupled global context Vision Transformer, the team demonstrates a significant leap in identifying heat-relevant building characteristics , and links these to associated health risks quantified using thermal infrared data! This innovative approach, deployed in Dar es Salaam, Tanzania, not only outperforms existing methods but also reveals crucial insights into how building materials correlate with household-level heat exposure and health outcomes.

UAV, Street View and Thermal Data Fusion

Scientists have developed a novel machine learning framework to assess building heat resilience in urban environments, particularly addressing the intensifying heat exposure faced by communities in the Global South. The research team achieved a breakthrough by fusing openly available unmanned aerial vehicle (UAV) and street-view (SV) imagery using a coupled global context vision transformer (CGCViT), a technique designed to learn heat-relevant representations of urban structures. This innovative approach leverages the complementary strengths of both data sources, providing a more comprehensive understanding of building attributes impacting thermal performance. Thermal infrared (TIR) measurements, obtained from the HotSat-1 satellite, were then employed to quantify the direct relationship between these building characteristics and associated health risks related to heat exposure.

The study reveals that this dual-modality cross-view learning approach significantly outperforms single-modality models, achieving improvements of up to 9.3% in accuracy. This demonstrates the substantial benefit of integrating UAV and SV imagery, as each perspective offers unique insights into urban structures often missed by either source alone. Experiments show that the presence of vegetation surrounding buildings, brighter roofing materials, and roofs constructed from concrete, clay, or wood are all significantly associated with lower TIR values, indicating reduced heat absorption. The research establishes a clear link between building construction and thermal performance, enabling targeted interventions to mitigate heat risks for vulnerable populations. This work opens new avenues for localized, data-driven risk assessment, which is critical for shaping climate adaptation strategies that deliver equitable outcomes and protect public health in rapidly urbanizing regions. Furthermore, the team meticulously constructed a cross-view representation of buildings by integrating SV panoramic imagery, building footprints, and high-resolution UAV imagery.

A CGCViT was trained to classify buildings based on key attributes including structural openness, number of floors, vegetation cover, wall material, and roofing material. These attributes, alongside factors like distance to surrounding buildings and roof/wall brightness, were then statistically correlated with HotSat-1 TIR values, revealing the features most strongly linked to lower thermal exposure and informing building-level heat mitigation strategies. Data acquisition involved collecting SV imagery via the Panoramax API and UAV imagery using a DJI Mavic 2 Pro drone, resulting in a 19.2 km² orthomosaic with 2.4m horizontal and 1.4m vertical accuracy and a 9cm ground sampling distance.

Cross-view learning for building vulnerability assessment models

Scientists developed a novel machine learning framework to assess heat-relevant building attributes using openly available unmanned aerial vehicle (UAV) and street-view (SV) imagery! This work pioneers a dual-modality cross-view learning approach, fusing these data sources via a coupled global context vision transformer (CGCViT) to learn representations of urban structures. Researchers harnessed thermal infrared (TIR) measurements from HotSat-1 to quantify the relationship between building characteristics and associated health risks, enabling precise identification of vulnerable areas. The study employed SV panoramic imagery, building footprints, and high-resolution UAV imagery as inputs to construct a comprehensive cross-view representation of buildings.

A CGCViT was then trained to classify buildings based on structural openness, number of floors, vegetation presence, wall material, and roofing material, key factors influencing thermal performance. Experiments meticulously recorded distance to surrounding buildings, alongside roof and wall brightness, to establish a robust dataset for statistical analysis. These attributes were statistically associated with HotSat-1 TIR values, revealing which features most strongly correlated with lower thermal exposure and informing targeted mitigation strategies. Experiments revealed that their dual-modality cross-view learning approach outperformed single-modality models by up to 10%, demonstrating the complementary value of UAV and SV imagery in understanding urban structures. This breakthrough delivers a scalable method for assessing heat vulnerability in densely populated areas.

The team measured thermal infrared (TIR) values using HotSat-1 data and discovered significant correlations between building attributes and heat retention. Specifically, the presence of vegetation surrounding buildings was associated with lower HotSat-1 TIR values, indicating reduced heat absorption compared to buildings without vegetation. Brighter roofing materials consistently exhibited lower TIR values than darker roofing, suggesting a reflective capacity that mitigates heat gain. Furthermore, roofing constructed from concrete, clay, or wood demonstrated significantly lower TIR values when contrasted with metal or tarpaulin roofing materials. Researchers found that buildings surrounded by vegetation, those with brighter roofing, and those constructed with concrete, clay, or wood consistently exhibited lower TIR values! This suggests that relatively simple, low-cost interventions, such as reflective roof coatings, urban gardening initiatives, or material upgrades, could effectively mitigate heat.