15,600 Image Dataset Advances Machine Learning for Fringe Projection Profilometry

Fringe projection profilometry (FPP) increasingly relies on machine learning, yet progress is hampered by limited training data and a lack of standardised evaluation methods. Anush Lakshman S, Adam Haroon, and Beiwen Li, from the Department of Mechanical Engineering at Iowa State University, address this challenge by presenting the first open-source, photorealistic synthetic dataset for FPP. The dataset comprises 15,600 fringe images and 300 corresponding depth reconstructions, generated using Isaac Sim and representing 50 different objects. This research significantly advances the field by benchmarking four distinct neural network architectures , UNet, Hformer, ResUNet, and Pix2Pix , and revealing fundamental limitations in direct fringe-to-depth mapping, with errors consistently representing a substantial proportion of typical object depth. The availability of this resource and standardised evaluation protocols will facilitate systematic comparison and accelerate the development of learning-based FPP techniques.

Synthetic Dataset Accelerates Fringe Projection Profilometry Learning

Scientists demonstrate a significant advancement in machine learning applications for fringe projection profilometry (FPP), a crucial non-destructive technology used in robotic scanning, manufacturing inspection, and 3D printing optimisation. The research addresses a critical limitation hindering progress in the field: the scarcity of large, diverse datasets and standardised evaluation protocols for machine learning models. The team achieved this by constructing the first open-source, photorealistic synthetic dataset for FPP, generated using NVIDIA Isaac Sim, comprising 15,600 fringe images and 300 depth reconstructions derived from 50 diverse objects. This breakthrough leverages the VIRTUS-FPP system, built within NVIDIA Isaac Sim, integrating advanced ray tracing and physics simulation to create a virtual FPP environment with end-to-end camera-projector modelling.

The researchers meticulously calibrated this virtual system using procedurally generated circular boards, achieving sub-pixel accuracy with a stereo reprojection error of 0.055506 pixels and a projector error of 0.048609 pixels. This precise calibration is fundamental to generating accurate ground truth data for training and evaluating machine learning algorithms. The dataset incorporates objects sourced from the YCB datasets and NVIDIA Physical AI Warehouse, ensuring a broad range of shapes and surface characteristics. This suggests that the fundamental limitations lie not within the network architecture itself, but within the approach of directly mapping fringes to depth without explicit phase information. The study reveals that reconstruction errors consistently approach 75-95% of the typical object depth range, indicating that networks primarily learn coarse shape priors rather than precise geometric details. The work establishes a standardised evaluation framework, enabling systematic comparison and development of learning-based FPP approaches.

The researchers employed a GPU-accelerated pipeline capable of capturing patterns at 3 frames per second, exceeding the speed of previous methods. By generating a dataset with perfect ground truth and comprehensive benchmarking protocols, this research unlocks new possibilities for real-time, single-shot 3D reconstruction, paving the way for more efficient and accurate applications in robotics, automation, and advanced manufacturing. This resource will undoubtedly accelerate innovation in the field of structured light metrology and machine learning.

Virtual FPP Dataset and NVIDIA Isaac Sim Framework

Researchers addressed limitations in machine learning for fringe projection profilometry (FPP) by developing a novel, open-source synthetic dataset and benchmarking framework. The work centres around VIRTUS-FPP, a physics-based virtual FPP system constructed within NVIDIA Isaac Sim, integrating OptiX ray tracing, PhysX physics, and Universal Scene Description for 3D composition. This system enabled the generation of 15,600 fringe images and 300 corresponding depth reconstructions, representing 50 diverse objects with known ground truth, a substantial resource for training and evaluating machine learning algorithms. The virtual FPP system employs a calibrated camera-projector pair, with the camera utilising a 960×960 resolution pinhole primitive with a 50cm focal length, and the projector modelled as a 0.625m x 0.5m rectangular light source emitting at 40 nits.

The projector is strategically positioned 0.1m below and 0.125m to the left of the camera to optimise the triangulation geometry essential for accurate depth mapping. A key innovation lies in modelling the projector using an inverse camera model, allowing for accurate dimensional correspondence of projected fringe patterns regardless of distance, circumventing limitations of physical hardware. All objects within the dataset consistently exhibit matte material properties, roughness of 0.95, specular reflectance of 0.15, and an AO-to-diffuse ratio of 0.95, mirroring characteristics of typical structured light scanning scenarios. Virtual calibration was achieved using procedurally generated 5×9 asymmetric circular boards with 10mm diameter circles spaced 20mm apart, capturing 18 poses to yield 936 calibration images within five minutes, achieving a throughput of 10,530 images per hour.

This calibration process resulted in sub-pixel accuracy, demonstrated by a stereo reprojection error of 0.055506 pixels and a projector error of 0.048609 pixels. Data acquisition involved rotating each of the 50 objects, sourced from YCB datasets and the NVIDIA Physical AI Warehouse, around the vertical axis in 60° increments, generating six viewpoints per object with 50% overlap. An 18-step phase-shifting sequence was then captured at each viewpoint, processed using standard N-step phase-shifting, Gray-code temporal unwrapping, and triangulation to generate depth maps with 1.2×10−3mm precision over an 80mm range. The resulting dataset comprises not only the fringe images and depth maps, but also normalization parameters and ground truth mesh geometries, facilitating comprehensive evaluation and development of learning-based FPP techniques.

Photorealistic Dataset Benchmarks Neural Network Performance

Scientists have achieved a breakthrough in fringe projection profilometry (FPP) through the creation of the first open-source, photorealistic synthetic dataset. Generated using NVIDIA Isaac Sim, the dataset comprises 15,600 fringe images and 300 depth reconstructions representing 50 diverse objects. This substantial resource addresses a critical limitation in machine learning for FPP, the scarcity of large, varied datasets and standardized evaluation protocols. The work leverages VIRTUS-FPP, a physics-based virtual FPP system, to deliver accurate ground truth data for benchmarking. Despite significant differences in their architectural designs, the models exhibited comparable accuracy in reconstructing depth from fringe patterns. The virtual system utilizes a calibrated camera-projector pair, with the camera possessing a 960×960 resolution and a 50cm focal length, and the projector emitting light at 40 nits with a pattern resolution of 912×1140 pixels. Measurements confirm that reconstruction errors consistently approach 75-95% of the typical object depth range, approximately 80 millimeters.

This finding reveals a fundamental limitation of directly mapping fringe patterns to depth without incorporating explicit phase information. The team recorded a stereo reprojection error of 0.055506 pixels and a projector error of 0.048609 pixels during the virtual calibration process, ensuring sub-pixel accuracy. The dataset’s composition includes objects sourced from the YCB datasets and the NVIDIA Physical AI Warehouse, encompassing a wide variety of shapes and complexities. The breakthrough delivers a standardized evaluation framework, enabling systematic comparison and development of learning-based FPP approaches.

VIRTUS-FPP’s innovative projector modeling, based on the inverse camera model, allows for accurate dimensional correspondence of projected fringe patterns without hardware constraints. The system captured 936 calibration images in just five minutes, demonstrating a throughput of 10,530 images per hour. This work establishes a powerful tool for advancing research in 3D reconstruction, robotic scanning, and manufacturing inspection.

Fringe Data Limits Semantic Shape Learning

This research presents a novel, large-scale synthetic dataset, comprising 15,600 fringe images and 300 depth reconstructions of 50 diverse objects, designed to facilitate machine learning advancements in fringe projection profilometry (FPP). Through benchmarking four distinct neural network architectures on this dataset, the study demonstrates a surprising consistency in performance, with all models achieving root mean squared errors between 58 and 77 millimetres. The findings reveal fundamental limitations inherent in directly mapping fringe patterns to depth without incorporating explicit phase information, with reconstruction errors frequently reaching 75-95% of the typical object depth. This suggests that current approaches are primarily learning semantic shape priors rather than accurate geometric representations, and that the limiting factor is not model design but rather a lack of sufficient information.

The authors acknowledge that the ResUNet architecture’s comparatively weaker performance may be due to overfitting on a limited training sample. Future research will focus on integrating phase information into learning processes, exploring sim-to-real transfer techniques, multi-view fusion, and uncertainty quantification. By establishing a standardised benchmarking framework and a comprehensive synthetic dataset, this work provides a solid foundation for data-driven development of robust FPP systems applicable to manufacturing, biomedical imaging, and automated inspection.