Single-View 3D Reconstruction Achieves Efficient Ship Monitoring with Synthetic Data Training

Researchers are tackling the challenge of creating accurate 3D models of ships from single images, a crucial capability for effective maritime monitoring. Borja Carrillo-Perez, Felix Sattler, and Angel Bueno Rodriguez, all from the German Aerospace Center (DLR)’s Institute for the Protection of Maritime Infrastructures, alongside Maurice Stephan and Sarah Barnes, detail a novel pipeline that overcomes limitations of existing methods which often demand multiple images or extensive annotation. Their work is significant because it achieves robust 3D reconstruction using only synthetic training data, effectively bridging the gap to real-world application and offering a scalable solution for real-time visualisation and inspection of vessels without the need for costly or time-consuming real-world 3D data collection.

This research addresses a critical limitation in current state-of-the-art methods, which often require multiple viewpoints, annotated 3D data, or substantial computational resources, hindering real-time deployment in operational maritime environments. The team achieved single-view 3D reconstruction by training entirely on synthetic data, eliminating the need for real-world 3D annotations, a major obstacle in this field. Their approach leverages the Splatter Image network, a technique that represents objects as sparse sets of 3D Gaussians, enabling rapid and accurate reconstruction from individual images.

The study initially fine-tuned the Splatter Image network using synthetic vessel data from ShapeNet, then further refined it with a custom, diverse dataset of 3D ships to bridge the gap between synthetic and real-world imagery. A state-of-the-art segmentation module, based on YOLOv8, was integrated alongside custom preprocessing steps to ensure compatibility with the reconstruction network. This careful integration allows the system to accurately identify and isolate ships within images before initiating the 3D reconstruction process. Quantitative evaluation on synthetic validation data demonstrated strong reconstruction fidelity, confirming the model’s ability to accurately represent ship geometry.
Following reconstruction, the pipeline incorporates postprocessing steps to align the 3D models with real-world scales, centering, and orientation. This is followed by georeferenced placement of the reconstructed ships onto an interactive web map, utilising metadata and homography-based mapping techniques. The use of Automatic Identification System (AIS) data, providing latitude, longitude, and ship length, is crucial for accurate geospatial alignment and scaling. Qualitative results on real maritime images from the ShipSG dataset confirm the potential for transferring this technology to operational maritime settings, offering a practical solution for real-time 3D ship visualization.

This breakthrough reveals a scalable solution for maritime monitoring, providing interactive 3D inspection of real ships without the need for costly and time-consuming real-world 3D annotation. The research establishes a path toward real-time 3D ship visualization, supporting applications such as maritime surveillance, vessel dimension verification, and port logistics planning. By effectively bridging the synthetic-to-real gap, the team has unlocked a new capability for enhancing situational awareness, improving safety, and optimising operations within the maritime domain. An interactive demonstration of the system is available at https://dlr-mi.github.io/ship3d-demo/.

Single-view ship reconstruction using synthetic data

Scientists developed an efficient pipeline for single-view 3D reconstruction of ships, addressing limitations in current maritime monitoring techniques. The study pioneered a method trained entirely on synthetic data, enabling reconstruction from a single image at inference. Researchers harnessed the Splatter Image network, representing objects as sparse 3D Gaussians to achieve rapid and accurate reconstruction. Initially, the model underwent fine-tuning using synthetic ShapeNet vessels, followed by refinement with a custom, diverse dataset of 3D ships designed to bridge the gap between synthetic and real-world imagery.

To ensure compatibility, the team integrated a state-of-the-art segmentation module based on YOLOv8, alongside custom preprocessing steps. Experiments employed this module to accurately identify ship boundaries within images before feeding data into the reconstruction network. Postprocessing involved real-world scaling, centering, and orientation alignment of the reconstructed models, ensuring accurate representation of ship dimensions and pose. Subsequently, scientists georeferenced these models, placing them on an interactive web map utilising Automatic Identification System (AIS) metadata and homography-based mapping techniques.

Quantitative evaluation was performed on synthetic validation data, demonstrating strong reconstruction fidelity and accuracy. The research then applied the trained model to real maritime images from the ShipSG dataset, confirming its potential for transfer to operational maritime settings. This approach delivers interactive 3D inspection of real ships without the need for real-world 3D annotations, a significant advancement for practical applications. The final system provides an efficient and scalable solution for maritime monitoring, paving the way for real-time 3D ship visualisation in challenging environments.

Synthetic data enables single-view ship reconstruction from images

Scientists achieved efficient single-view 3D reconstruction of ships using a pipeline trained entirely on synthetic data, circumventing the need for real-world 3D annotations. The research team leveraged the Splatter Image network, representing objects as sparse 3D Gaussians, to enable rapid and accurate reconstruction from single images. Initial model fine-tuning occurred on synthetic ShapeNet vessels, followed by refinement with a custom dataset of 3D ships, effectively bridging the gap between synthetic and real-world imagery. This approach allows for interactive 3D inspection of real ships without requiring costly and time-consuming real-world 3D annotation.

Experiments revealed strong reconstruction fidelity on synthetic validation data, demonstrating the model’s ability to accurately represent ship geometry. The team integrated a state-of-the-art segmentation module, based on YOLOv8, alongside custom preprocessing to ensure compatibility with the reconstruction network. Postprocessing steps included real-world scaling, centering, and orientation alignment, followed by georeferenced placement on an interactive web map utilising AIS metadata and homography-based mapping. Quantitative evaluation confirmed the model’s performance in controlled synthetic environments, paving the way for real-world application.

Data shows the system successfully processes real maritime images from the ShipSG dataset, confirming the potential for transfer to operational maritime settings. Researchers recorded the use of AIS metadata, including latitude, longitude, and ship length, to accurately position and orient the reconstructed 3D models on the interactive map. The pipeline’s efficiency allows for scalable solutions in maritime monitoring, offering a path toward real-time 3D ship visualisation in practical applications. The final system delivers interactive 3D inspection capabilities without the limitations of traditional methods requiring extensive manual annotation.

Measurements confirm the approach addresses a critical obstacle in maritime 3D reconstruction: the lack of available real-world 3D ground truth data. Tests prove the effectiveness of synthetic-to-real domain adaptation techniques in this context, enabling the deployment of 3D reconstruction in scenarios where real-world data is scarce. The breakthrough delivers an efficient and scalable solution for enhanced maritime situational awareness, supporting applications such as vessel dimension verification and port logistics planning. The work highlights a significant advancement in 3D vision for geospatial visualisation and maritime monitoring.

Single-image ship reconstruction via synthetic data training

Scientists have developed an efficient pipeline for reconstructing three-dimensional models of ships from single images, trained entirely on synthetic data. This approach utilises the Splatter Image network, representing objects as sparse 3D Gaussians, enabling rapid and accurate reconstruction without requiring multiple viewpoints or annotated 3D ground truth. The model undergoes fine-tuning on synthetic datasets before refinement with a custom dataset of 3D ships, effectively bridging the gap between simulated and real-world imagery. The resulting system integrates a segmentation module and post-processing steps to scale, centre, and orient the reconstructed ships, subsequently placing them on an interactive web map using metadata and homography-based mapping.

Quantitative evaluation on synthetic data confirms reconstruction fidelity, while qualitative results from the ShipSG dataset demonstrate potential for operational maritime applications. This work offers a scalable solution for maritime monitoring, enabling interactive 3D inspection of vessels without the need for real-world 3D annotations. The authors acknowledge a limitation in the lack of quantitative validation on real-world data due to the absence of corresponding 3D ground truth. Future research directions include generating real-world 3D ground truth using technologies like LiDAR and multi-camera systems to enable more comprehensive validation and supervised domain adaptation. Further development will explore integrating these reconstructions with higher-fidelity digital twins and experimenting with additional data sources to enhance robustness in diverse conditions. This research establishes a path towards real-time 3D ship visualisation for practical applications, despite challenges with complex or occluded vessels.