作者：洛洛 | 来源：3DCV
In the background of the official account "3DCV", reply to the "original paper" to get the pdf and code link of the paper
Add v: DDDSiew, note: structured light, pull you into the group. At the end of the article, industry subdivisions are attached
1. Introduction
Traditional phase measurement deflection measurement techniques have some problems when measuring low-reflectivity surfaces and complex curved surfaces. The captured fringe pattern on a low-reflectivity surface is of poor quality and does not extract the phase correctly. However, complex surfaces produce complex reflective fringe patterns with closed-loop and open-loop features, which makes it difficult to obtain phase by traditional single-shot phase measurement methods. Therefore, this study aims to solve these problems by using deep learning technology, and proposes a deflection measurement method based on deep learning for phase measurement of a single shot to achieve three-dimensional measurement of complex freeform surfaces.
2. Research content
A deep-learning-based single-shot phase measurement deflectometry method for three-dimensional measurement of complex freeform surfaces. This paper introduces two main problems existing in the current single-shot deflectometry method: the measurement of low-reflectivity specular surfaces and the measurement of complex curved surfaces. To address these issues, the research team employed a low-frequency composite grating pattern to improve grating visibility on low-reflectivity surfaces, and a deep learning approach to process phase measurements on complex surfaces. Through the training and optimization of deep learning networks, 3D measurement and defect detection of complex surfaces can be realized in a single shot.
3. Method
The deep learning network model DYnet++ is used to obtain phase information from a single composite pattern. To train the deep learning model, the authors used a deformable mirror with nine actuators to generate large amounts of data on various surface shapes. By comparing the measurement results with the results of the 16-step phase shift method, the feasibility of the proposed single-shot deflection measurement method based on deep learning is verified. The advantage of this method is that it can obtain phase information of complex surfaces in a single shot, avoiding the disadvantage of the traditional phase shift method that requires multiple images. Deep learning network models are able to learn and understand complex pattern features to accurately extract phase information. This method has a wide range of applications in industrial settings and can improve measurement efficiency and accuracy. It should be noted that this method is still in the research stage, and there may still be some limitations and room for improvement. However, the single-shot deflection measurement method based on deep learning has great potential to solve complex surface measurement problems, and may be developed and improved more in practical applications.
3.1. The principle of surface measurement by deflection method
Deflection surface measurement is a slope measurement technique used to measure smooth freeform surfaces. It uses a screen (LCD screen) to display a modulated sinusoidal fringe pattern that is projected onto the measured surface, and then the camera receives the reflected fringe pattern. In this way, we can obtain the slope of the surface for analysis. If we can obtain the slope of two orthogonal surfaces, we can reconstruct the shape and size of the surface by integral. For defect detection, it is not only necessary to detect defects on surfaces, but also to determine the shape and size of the defects. Therefore, we need to project two orthogonal fringe patterns (called sum patterns) onto the surface under test to obtain the slope of the surface for integration. Traditionally, the phase-shift method has been used to obtain surface phase in sum direction because of its pixel-level phase recovery and high resolution. The captured stripe pattern can be represented by the following mathematical expression:
where and is the light intensity of the captured and streaked patterns, is the pixel coordinates of the camera, and is the background, and is the fringe modulation, and is the phase in the and direction, and is the number of phase shifts (), which is the total number of phase shifts. With these captured fringe patterns, the phase in and direction can be recovered by the following formula:
Where, and is the wrapping phase of and. They need to be unwound using a spatial phase unwrapping algorithm. is the phase shift value. In general, if the number of phase steps N is larger, the higher the phase accuracy of the recovery.
3.2、DYnet++
DYnet++ is a deep learning-based single-shot phase measurement deflection measurement method for measuring low-visibility and complex specular surfaces. DYnet++ deep learning network for obtaining phase from a single low-frequency composite fringe pattern for surface measurements. The DYnet++ network accepts one input, the captured low-frequency composite fringe pattern, and outputs two numerators and two denominators in each decoder path. Through the training process, each decoder path will have the best parameters to predict the numerator and denominator. Then, from the arctangent of these quantities, the wrapping phases of x and y are calculated. The structure of the DYnet++ network is inspired by Ynet and Unet++. We try to combine the advantages of these networks for phase detection. The DYnet++ network adds a decoder path to Unet++ to make it symmetrical.
3.3、DM
In this paper, the authors used deformable specular surfaces (DMs) to generate a training dataset. DM is a metal plate that can change its shape by applying force. By changing the shape of the DM, it is possible to change the captured stripe pattern. The advantage of using DM to generate training data is that a large amount of training data can be generated quickly, simply by changing the shape of the mirror. The authors first display the composite pattern on an LCD screen, and then capture the reflected pattern through the camera as input to the training data. The authors then used the phase shift method to display 16-step phase shift images in the x and y directions as the truth of the training data. For each input composite pattern, the corresponding output will be , , , and. By changing the shape, different training data can be generated. The authors randomly changed the shape by changing the position of each actuator 1,000 times, resulting in a total of 4,000 training images and ground truth. Using this method to generate training data, the network can learn how to predict the correct numerator and denominator from a single composite pattern.
4. Experimental results
DYnet++ is implemented using TensorFlow and Keras 2.8.0 deep learning libraries. The training process took place on a desktop computer equipped with an Intel Core i9-12900K CPU, 64 GB RAM, and a Geforce RTX-3090 GPU (NVIDIA). They used different shapes of deformable mirrors (DMs) to generate a training dataset, generating a total of 4,000 training images and corresponding ground truths. The size of the training dataset is 320×240 pixels. They used the mean square error as the loss function and the mean absolute error as the evaluation metric. The training takes about 12 hours in total.
In the test, they used as input a composite pattern generated by a deformation mirror that was different from the training and validation data. They showed some of the test results and compared them to ground truth. The results show that the DYnet++ method has similar performance in shape reconstruction compared to the traditional 16-step phase-shift method, which requires 32 phase-shift images. In addition, they conducted measurements and defect detection experiments on low-visibility surfaces, and the results showed that DYnet++ could detect small defects in a single composite pattern. In general, the DYnet++ method realizes the phase measurement and deflection measurement of a single shot through deep learning, which has good performance and application potential.
5. Summary
Our research proposes a deep learning-based single-deflection measurement method, DYnet++, which is capable of measuring and detecting complex freeform surfaces. This method has a wide range of applicability and efficient performance, and can be used to measure and inspect complex surfaces in real-time or high-speed environments.
3D visual communication group
Hello everyone, the group will release the most cutting-edge paper interpretation and exchange and sharing in the direction of 3D computer vision for the first time, the main directions are:
Vision SLAM, Laser SLAM, ORB-SLAM, Vins-Fusion, LOAM/LeGo-LOAM, Cartographer, VIO, Semantic SLAM, Filtering Algorithms, Multi-Sensor Fusion, Multi-Sensor Calibration, MSCKF, Dynamic SLAM, MOT SLAM, NeRF-SLAM, FAST-LIO, LVI-SAM, LIO-SAM, Event Camera, GPS/RTK/UWB/ IMU/Code Disc/TOF (iToF, dToF), LiDAR, Barometer, Millimeter-Wave Radar, RGB-D Camera, Ultrasonic, etc., Robot Navigation, Camera Calibration, Stereo Matching, 3D Point Cloud, Structured Light (Surface/Line/Speckle), Robotic Arm Grasping (2D/3D), 2D Defect Detection, 3D Defect Detection, 6D Pose Estimation, Phase Deflectometry, Halcon, Light Field Reconstruction, Photogrammetry, Array Camera, Polarization 3D Measurement, Photometric Stereo Vision, LiDAR, NeRF, Multi-View Geometry, OpenMVS, MVSNet, colmap, Texture Mapping, Depth Estimation, Transformer, Millimeter/LiDAR/Vision Camera Sensors, Multi-Sensor Calibration, Multi-Sensor Fusion, Autonomous Driving Complex, etc., Object Detection, 3D Object Detection, Path Planning, Trajectory Prediction, 3D Point Cloud, 3D Point Cloud Segmentation, Model Deployment, Lane Line Detection, Occupancy, Target Tracking, Quadcopter Modeling, Comprehensive communication groups such as drone flight control, job hunting, hardware selection, visual product landing, the latest papers, and the latest 3D vision products.
Add v: dddvision, note: research direction + school/company + nickname (such as 3D point cloud + Tsinghua + Little Strawberry), and the assistant will pull you into the group.