AI-Powered Surgical Video Summarization Using Computer Vision and Large Language Models

On April 28, 2025, researchers Hugo Georgenthum and colleagues published Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI, presenting an innovative method combining visual transformers with large language models to automatically summarize surgical videos. Their approach achieved 96% precision in tool detection and a BERT score of 0.74, showcasing advancements in AI-assisted surgical documentation.

The research presents an AI method for summarizing surgical videos using multi-modal frameworks combining visual transformers and large language models. The approach processes video clips in three stages: feature extraction, caption generation with temporal context, and report aggregation. Tested on the CholecT50 dataset, it achieved 96% tool detection precision and a BERT score of 0.74 for summarization, advancing AI tools for surgical documentation.

In the ever-evolving landscape of healthcare technology, PRIME AI stands out as a pioneering solution designed to revolutionise surgical documentation. Developed by researchers at the University of Toronto, this innovative system leverages advanced computer vision and natural language processing (NLP) techniques to automate the creation of detailed surgical video reports in real time. By addressing the challenges of traditional surgical documentation, which often requires extensive manual effort, PRIME AI offers a more efficient and accurate alternative.

At its core, PRIME AI is built on three key components: object detection, frame captioning, and clip captioning. The system begins by identifying surgical instruments and body parts within the video feed using an object detection module. This data is then processed by a frame captioning component, which employs Vision Transformers (ViT) to analyse individual frames and extract essential details from each moment of the surgery.

To enhance the quality of generated captions, PRIME AI uses a multi-modal fusion approach that combines visual and textual data. This integration allows the system to produce coherent and contextually rich reports, even for complex surgical procedures. The T5 model is utilised for conditional text generation, transforming the fused features into comprehensive narratives that closely resemble human-generated reports.

Initial testing has demonstrated PRIME AI’s impressive accuracy in object detection and its ability to generate coherent reports from lengthy videos. This capability is particularly valuable given the extended duration of many surgical procedures. While laparoscopic cholecystectomy was a key focus during development, PRIME AI shows significant potential for application across various surgical domains and medical imaging tasks.

As with any new technology, several considerations must be addressed to ensure successful implementation. The system’s efficiency in real-time processing suggests it can handle video data without substantial lag, making it suitable for live documentation during surgery. However, the ease of integration into existing hospital systems and hardware requirements remain important factors.

Additionally, the sourcing and diversity of training data are critical considerations, particularly concerning compliance with medical regulations like HIPAA. Understanding the dataset’s breadth regarding procedures and surgeons will be essential for assessing PRIME AI’s adaptability and reliability.

PRIME AI represents a significant advancement in automating surgical documentation, offering potential improvements in efficiency and accuracy. Its ability to generate detailed reports comparable to human efforts could revolutionise medical reporting practices. As research progresses, exploring the system’s performance metrics against human-generated reports will provide further insights into its practical applications and benefits for healthcare settings.

In conclusion, PRIME AI has the potential to transform surgical documentation by streamlining processes and enhancing accuracy. While challenges remain in terms of integration and data compliance, the technology offers a promising glimpse into the future of medical reporting.