AI Assesses Streetscapes, Linking Visual Data to Resident Perceptions.

A novel framework assesses streetscapes by fusing image analysis with large language models, achieving 84% F1 score for objective features and 89.3% agreement with resident perceptions in Harbin, China. It identifies context-dependent contradictions and nonlinear patterns linking physical attributes to lived experience, supporting sustainable urban development.

Understanding how people experience urban spaces is crucial for effective city planning, yet conventional analytical methods often prioritise quantifiable data over subjective perceptions. Researchers are now developing systems that integrate both, moving beyond simple metrics to capture the nuances of lived experience. Haotian Lan, from Harbin, China, and colleagues detail such a framework in their paper, ‘Interpretable Multimodal Framework for Human-Centered Street Assessment: Integrating Visual-Language Models for Perceptual Urban Diagnostics’. The study presents a novel approach fusing image analysis with natural language processing to assess streetscapes, validated using a dataset of over 15,000 images from Harbin and demonstrating a high degree of correlation with resident perceptions.

Assessing Streetscapes with Integrated Vision and Language Models

Current urban analytics often rely on quantifiable data, neglecting subjective perceptions of the built environment. This research presents the Multimodal Street Evaluation Framework (MSEF), a novel system integrating computer vision and large language models to provide a comprehensive assessment of streetscapes. MSEF evaluates both objective characteristics – such as building height or road surface condition – and subjective qualities, achieving an F1 score of 0.84 for objective feature identification and 89.3 per cent agreement with resident perceptions in a case study of Harbin, China.

MSEF distinguishes itself by identifying contextual contradictions in urban perception. The framework recognises instances where a single feature elicits both positive and negative responses. For example, informal street commerce can simultaneously contribute to a street’s vibrancy while potentially reducing pedestrian comfort. This nuanced understanding moves beyond simple spatial analysis, acknowledging the complex interplay of factors shaping lived experience.

The framework generates natural language rationales to explain its assessments, enhancing transparency and interpretability. MSEF highlights the relevant features and their impact on the overall perception of the environment, allowing users to understand the reasoning behind its conclusions. This transparency builds trust and facilitates informed decision-making in urban planning.

Future work should expand the dataset to encompass diverse geographical locations and cultural contexts, improving the generalisability of the framework. Integrating additional data modalities, such as noise pollution levels or pedestrian flow rates, could further refine the assessment process. Exploring the application of MSEF within real-time urban planning scenarios, potentially through interactive digital twins – virtual representations of physical spaces – offers a promising avenue for practical implementation.

The research makes a methodological contribution through its capacity to identify contingent patterns, challenging the application of universal design principles. MSEF reveals how architectural transparency evokes different reactions depending on the surrounding area, demonstrating the importance of context-specific evaluation.

Attention mechanisms within the model enable it to focus on the most relevant parts of the input data when making a prediction, resulting in a more nuanced and interpretable assessment. These mechanisms effectively prioritise salient features, improving the accuracy and clarity of the evaluation.

This study demonstrates the potential of artificial intelligence to transform urban planning and design, contributing to more livable, sustainable, and equitable cities. MSEF provides a powerful tool for understanding and evaluating urban environments, enabling informed decision-making and proactive planning.

Future research should focus on refining the framework’s ability to capture and incorporate subjective perceptions, exploring new data modalities and machine learning techniques. Investigating the potential of integrating MSEF with other urban data sources, such as social media data and citizen science data, could further enhance its capabilities. Developing user-friendly interfaces and visualisation tools will facilitate the adoption of MSEF by urban planners and designers.