Kaan Ozbay, along with researchers from NYU Tandon School of Engineering’s C2SMART center and the Center for Robotics and Embodied Intelligence, has developed an artificial intelligence system for automated road safety analysis. This system, called SeeUnsafe, combines language reasoning and visual intelligence to identify collisions and near-misses within existing traffic video footage. Published in Accident Analysis and Prevention, the research earned New York City’s Vision Zero Research Award and demonstrates a novel application of multimodal large language models. Tested on the Toyota Woven Traffic Safety dataset, SeeUnsafe correctly classified traffic events with 76.71% accuracy, offering a proactive approach to identifying and mitigating dangerous road conditions.
AI System Identifies Traffic Collisions and Near-Misses
NYU Tandon researchers have developed SeeUnsafe, an AI system that automatically identifies traffic collisions and near-misses from existing traffic video footage. This system combines language reasoning with visual intelligence, offering a way to improve road safety without requiring major new investments from transportation agencies. Winning New York City’s Vision Zero Research Award, SeeUnsafe leverages pre-trained AI models to analyze long-form videos, pinpointing dangerous intersections and conditions proactively.
SeeUnsafe demonstrated strong performance when tested on the Toyota Woven Traffic Safety dataset, correctly classifying videos 76.71% of the time. The system can also identify the road users involved in critical events with up to 87.5% accuracy. Beyond simply detecting incidents, SeeUnsafe generates “road safety reports” explaining its decisions, including factors like weather and traffic volume, enabling agencies to understand why certain events occurred.
This technology allows for proactive intervention, moving beyond reacting to accidents after they happen. By analyzing near-misses—like close calls with pedestrians—agencies can implement preventive measures such as improved signage or signal timing. Researchers suggest the approach could even extend to in-vehicle dash cameras, potentially enabling real-time risk assessment for drivers.
SeeUnsafe System Outperforms Existing Models
NYU Tandon researchers have developed SeeUnsafe, an AI system that automatically identifies collisions and near-misses in traffic video by combining language reasoning and visual intelligence. This system outperforms existing models, correctly classifying videos as collisions, near-misses, or normal traffic 76.71% of the time. It can also identify involved road users with success rates up to 87.5%, offering a way to improve road safety without major new investments in resources or infrastructure.
SeeUnsafe leverages pre-trained AI models to understand both images and text, representing a novel application of multimodal large language models to long-form traffic videos. Crucially, agencies don’t need to be computer vision experts or collect/label their own data—making the technology readily deployable. The system generates “road safety reports” with natural language explanations, describing factors leading to incidents, enabling proactive intervention at dangerous intersections.
By analyzing patterns of near-misses, SeeUnsafe allows for preventive measures like improved signage or optimized signal timing before accidents happen. This contrasts with traditional methods that only address safety issues after incidents occur. Tested on the Toyota Woven Traffic Safety dataset, the system establishes a foundation for using AI to understand road safety context from extensive traffic footage and could extend to in-vehicle dash cameras for real-time risk assessment.
Agencies don’t need to be computer vision experts. They can use this technology without the need to collect and label their own data to train an AI-based video analysis model.
Chen Feng
C2SMART Research Improves NYC Transportation Systems
C2SMART research is improving NYC transportation systems through a new AI system called SeeUnsafe. This system automatically identifies collisions and near-misses in existing traffic video footage, leveraging both language reasoning and visual intelligence. Winning New York City’s Vision Zero Research Award, SeeUnsafe allows agencies to pinpoint dangerous intersections and road conditions before accidents occur, without needing extensive manual review of thousands of hours of footage.
The SeeUnsafe system demonstrated high accuracy in classifying traffic videos, correctly identifying collisions, near-misses, or normal traffic 76.71% of the time. It can also identify involved road users with up to 87.5% success. By analyzing patterns of near-misses—like risky maneuvers—the system enables proactive safety interventions like improved signage or signal timing, shifting from reactive accident response to preventative measures.
This research builds upon a larger body of work from C2SMART, which includes projects studying the impact of electric trucks, analyzing speed camera effectiveness, developing a “digital twin” for faster FDNY response times, and monitoring the Brooklyn-Queens Expressway. The system generates “road safety reports” explaining its decisions, considering factors like weather and traffic volume, adding to its utility for city officials.
