3D measurement using fringe projection profilometry Текст научной статьи по специальности «Физика»

3D measurement using fringe projection profilometry

M.N. AlkhatibY.D. Shmelev 2, O.A. Tyshova 2, I.V. Sinilshchikov 2, A.V. Bobkov1 1 Bauman Moscow State Technical University, 105005, Moscow, Russia, 2nd Baumankaya st. 5;

2 Lomonosov Moscow State University, GSP-1, 119991, Moscow, Russia, Leninskie Gory

Abstract

This work is devoted to measuring the depth of the 3D object using the structured light method, in particular, phase shift profilometry. Theoretical studies on the methods of three-dimensional measurement systems and fringe projection profilometry are presented. The phase shift profilometry method with an improved calculation of the frequency of sinusoidal patterns is applied. In practical implementation in the environment (20 cm x 30 cm), the algorithm is tested on a stepped object consisting of eight steps with a difference of 150 pm between two successive steps. In this case, the achievable error for measuring such an object is 20 pm. Our method has great potential in industrial applications where the measurement of the smoothing of the surface of the object is needed to find the defect in the surface with high accuracy without contacting the object.

Keywords: fringe projection profilometry FPP; structured light.

Citation: Alkhatib MN, Shmelev YD, Tyshova OA, Sinilshchikov IV, Bobkov AV. 3D measurement using fringe projection profilometry. Computer Optics 2023; 47(6): 913-919. DOI: 10.18287/2412-6179-CO-1297.

Introduction

3D reconstruction methods allow scanning an object and get a map of all its spatial coordinates (methods for transferring an object from the real world into digital 3D space. These are used in many applications where it is necessary to accurately measure objects, such as checking the size of an object in industry or analyzing faults or damage in the object, manufacturing inspection, bio-medicine and virtual and augmented reality, 3D measurement also plays a critical role in additive manufacturing, also known as 3D printing [1]. 3D shape measurement techniques can be classified into two different categories, contact and non-contact, or based on the integration of the measurement information from a 3D vision sensor and a coordinate measuring machine (CMM) [2]. Contact methods measure and reconstruct 3D geometry by examining a 3D surface through physical touch. For example, a coordinate measuring machine (CMM) that can measure three-dimensional geometry using a precise carriage system or articulated probe. Disadvantages, the need for physical contact, it is undesirable to measure soft or deformable objects. To solve the problems associated with contact methods, non-contact 3D measurement methods have been developed and also are independent on the skill and the efficiency of the user. 3D optical scanning methods solve all these problems, that can collect data and build 3D object without touching it. There are several different types of 3D optical scanners that can be distinguished based on the technology used to collect data, such as: time of flight [3] (accuracy for the 3D scanner is 4 -10 mm). In a time of flight system, the scanner emits a pulse of laser light and receives the reflected pulse. The time is calculated, then multiplied by the speed of light in air, and divided it by two to get the distance. Structure from

motion SFM [4] or stereo vision (accuracy for the 3D scanner is 5 -1 cm) [5] and structured-light SL (accuracy for the 3D scanner is 100 -10 pm) [6, 7]. SL is a very popular non-contact 3D shape measurement technique with the advantages in terms of high measurement accuracy, high point density, high speed, and low cost. By projecting sinusoidal fringe patterns onto the object and capturing the corresponding deformed patterns modulated by the object surfaces, the depth information is encoded into the phase of the distorted fringe images. These sinusoidal fringe-based SL techniques are often referred to as fringe projection profilometry (FPP) [8]. Over the past twenty years, a huge number of articles and papers have appeared on fringe projection techniques and its application as review [9]. Phase profilometry (PSP) was first introduced in 1984 [10]. PSP multiple-shot methods are more reliable and accurate than Fourier transform profilometry FTP methods [7]. In addition, the PSP measurement is quite robust to ambient light and changing surface reflectance. However, the PSP methods require more time to obtain multiple fringe patterns, and the object should remain motionless during the projection of multiple fringe patterns. Accuracy is more important to us than execution time, so we applied PSP method. The method presented in this article can correct the frequency of the sinusoidal pattern on a reference surface by applying the phase shift profilometry algorithm.

In our case, as shown in the Fig. 1, the optical system consists of one IP camera (Daheng) with high resolution (4096 x 2048) and a projector (XGIMI) with resolution (1920 x 1080); the projector is perpendicular to the reference plane, the angle between the optical axis of the camera and the optical axis of the projector is 20 degrees, the distance between the camera and the projector is d=200 mm, and the distance between the camera and the reference plane, L = 5200 mm, h is the depth of the object

at the point A, the points B,C are located on the reference plane, the coordinate of point C on the X-axis is xc and the coordinate of point B on the X-axis is xb.

d

The rest of the paper is organized as follows. Calibration camera-projector is presented in section 2. In section 3, the algorithm steps are explained. The analysis and reduction of error sources for PSP are discussed in Section 4. The experimental results are displayed in section 5. The last section contains conclusion and future works.

Calibration camera-projector

The projector-camera pair works as a stereo system, with the advantage that a well-chosen projection template simplifies the task of finding point matches. In such systems, the projectors are modeled as inverse cameras, and all the considerations known for passive stereo systems can be applied almost unchanged. However, the calibration procedure must be adapted to the fact that

projectors cannot directly measure the pixel coordinates of 3D points projected onto the projector's image plane, as cameras do. Field of view, zoom, focus, and other parameters must be adjusted in both the projector and the camera to match the size of each target and scan distance. Therefore, structured lighting systems need to be calibrated before each use to ensure the best possible result, making the simplicity of the calibration procedure as valuable as its accuracy. To estimating the coordinates of the calibration points in the projector image plane using local homographs. First, a dense set of correspondences between the pixels of the projector and the camera are found by projecting onto the calibration object an identical sequence of patterns, which is subsequently projected to scan the target. Secondly, the set of correspondences is used to calculate the group of local homographs, which allow one to find the projection of the points of the calibration object onto the image plane of the projector with subpixel accuracy. At the end, the data projector is calibrated like a regular camera. A calibration target (a symmetric circular grid board (rows = 51 x columns = 51)) is selected and the distance between two consecutive centers is equal to 2 mm. A structured light system calibration algorithm is implemented [11]. The software is written in python and Visual Studio using the OpenCV library. The calibration results show that the projection error is 0.19 pixels.

Algorithm steps

The goal of structured light techniques is to provide encoded images (in which the object's height is embedded) obtained by projecting periodic patterns in order to reconstruct the 3D object with triangulation. In this section, phase shift profilometry method is explained. A block diagram of 3D construction is shown in Fig. 2.

Projected fringes

Wrapped phase

Unwrapped phase

Phase to height

3D reconstruction

Fig. 2. Flow chart of fringe projection profilometry

The purpose of the phase shift is to obtain a relative phase map of the scanned object with respect to the plane. The principle of phase shift is as follows: we consider the vertical pattern of sinusoidal stripes, since the intensity is the same for vertical transfers, but changes sinusoidally along the

2k

In(x,y) = A(x,y)+B(x,y)cos(^(x,y)+—n) ne [0,N-1],

N

1 N-1

A(x, y) =—Yjn (x, y),

2 fN- n N- n

B(x,y) = - [Y/n(x,y)sin(2K—)]2 + [j/n(x,y)cos(2K-)]2.

NV n=0 N n=0 N

Where In (x, y) - intensity value in the coordinates (x, y) of frame nth, A (x, y) - constant component (background), B (x, y) - modulation signal amplitude, ^ (x, y) - is the corresponding wrapped phase, which can be extracted using the following Eq. 2 [10]:

horizontal axis. N sinusoidal patterns are generated. Each pattern is shifted from the previous pattern by the angle 2k /N, and projected onto the object. X-axis and Y-axis form the reference plane. The deformed fringe patterns captured by the camera, denoted as In (x, y) defined by Eq. 1:

(1)

Y/n ( x, y)sin(2ïï nn )

0( x, y) = arctan(-N=f-—)•

Yjn ( X, y) cos(2rc — )

n=0 N

(2)

Since the arctangent function contains a 2n modulo discontinuity, $ (x, y) is called a wrapped phase map. So in order to obtain a continuous phase map, unwrapped phase is required to obtain a continuous phase map. The Relationship between the wrapped phase and the unwrapped phase defined by Eq. 3 [12]:

^unwrapped (X, y) = $ wrapped ( X, y) + 2nk (X, y), (3)

where k (x, y) is the integer number to represent fringe orders.

Once a wrapped phase is detected, the value of 2n is either added, or subtracted, to/from this sample and also from all the further samples until the jump is less than n. The difference between the current sample and its directly adjacent left-hand neighbour is calculated.

1. If the difference between the two is larger than + n, then subtract 2n from this sample and also from all the samples until the jump is less than n.

2. If the difference between the two is smaller than - n, then add 2n to this sample and also to all the samples until the jump is less than - n.

Phase to height. The depth value of this reference plane can be determined using simple trigonometry, as shown in the following Fig. 1.

The idea is to capture structured light patterns on a flat surface and view the resulting phase image as a reference. According to Fig. 1, the depth from point A to the base plane can be calculated using the following triangle relation given by Eq. 4:

±.=Lzh «h=M^№ (4)

\BC\ d d + \BC\ d '

Where, | BC | = | xc - xb | equal to the proportional difference of the measured phase with B and C, which is known after the decoding process. Eq. 4 is approximated with a linear relation based on the assumption that L is much larger than h.

In general, $ (x,y) depends on the coordinates x, y in the deformed image. In our case, when the height is constant, $ (x, y) is only depends on x because the intensity of the pattern is the same for vertical transfers but changes sinusoidally along the horizontal axis, So we can write Eq. 5:

$( x, y) = 2nf0 x + const, (5)

where f0 is the fundamental frequency of the sinusoidal pattern.

Relationship between phase and distance | BC |:

When h = 0, from Eq. 5, we can write const = 90, where 90 represents the phase without the object on the reference plane, then the deformed image at points B and C can be expressed by Eq. 6:

O1 (x, y) = A + B cos(2nf0 x + 90),

O1(xB, y) = A + B cos(2nf0xB + 90) = A + B cos(9B), (6)

Oi( xC, y) = A + B cos(2nf0 xC + 90) = A + B cos(9C).

After the object is placed on the reference plane, the point A is projected on the sensor of the camera instead of the point B with different value of the intensity, from Eq. 5, we can write const = 91, where 91 represents the phase when the object on the reference plane. The deformed interference pattern seen through the camera can be written by Eq. 7:

I1(xB,y) = A + Bcos(2nf0xB + 91)= A + Bcos(9A). (7)

The intensity observed at A on the object is the same as that observed at C on the reference plane,then from Eq. 6 and Eq. 7, the following relation can be deduced Eq. 8:

9a = 9c ^2nf) xb + 91 = 2nf> xc + 90,

(8)

A9 = 91 -90 =2nf |BC|.

The Eq. 8 expresses the relationship between the distance |BC| and the phase map. The relation between the phase and the height of the object can be concluded from Eq. 4 and Eq. 8 as given by Eq. 9:

h = A9L (9)

2nf0 d + A9 2nf0 d

Analysis and reduction of error sources for PSP

There are many causes of phase error, the main causes are classified as: intensity noise, non-linearity error, detector saturation.

1. intensity noise. Sources of intense noise include unstable ambient lighting, projector lighting noise, camera/projector flicker, camera noise, and quantization. A captured image with stripes suffers from intensity noise. Assuming that the intensity noise is additive, Gaussian (0, oB). The standard deviation of phase error 0$ is defined by Eq. 10 [13].

G$ = il_°L_. (10)

$ \NB*F

Where N is the number of patterns, B is the intensity modulation, F the number of stripes in the pattern. Increasing the number of patterns can improve accuracy. However, the execution time increases. The number of patterns 3 is chosen from a trade-off be-tween runtime and accuracy then he number of patterns 7 is chosen to enhance the accuracy. Increasing the number of stripes can increase the range of phase change to obtain more detailed information of the measured object and improve the measurement accuracy of the system. However, the higher F of captured fringe patterns is, the fewer pixels in one stripe period (period of sinewave), which causes the lower measurement accuracy of the system and also using Higher F introduces phase ambiguities in the reconstruction process. There is a problem of saturation when sinusoidal wave is projected on the object. To solve this problem, the intensity of the

pattern is scaled in range [20,220] instead of [0,255]. In Our case (N = 7, B = 200, F = 90). 2. Nonlinearity error. The main cause of the nonlinear mapping between projector input and captured image intensity is gamma factor [14], that leads to

distortions in the fringe profiles. In addition, a

non-sinusoidal waveform is approximated by an ideal sinusoidal function if high-order harmonics are not taken into account. Specifically, based on the Fourier series expansion, non-sinusoidal waveform can be written as Eq. 11:

" 2K In = A + Yfik cos[k(2k/0x +—n)] n e [0,N -1]. (11)

k=1

N

An average filter is applied to minimize the influence of higher harmonics, but this improvement results in a decrease in fringe contrast and SNR. In addition to high frequency detail, the object's surface can also be reduced. So, we applied the mean filter with small kernel (7 pixels). Another method is suggested to correct the period of fringe [15]. Where, homographic transform is applied between projection pattern and captured image plane. In fact, standard N-step phase shifting algorithms have certain resistance to the nonlinearity error inherently. In the state-of-the-art research, nonlinearity compensation methods are used, which can be divided into two categories: active methods (correction before image projection) and passive methods (correction after image projection).

3. Intensity saturation. When the exposure time of the camera is too long or the fringe pattern is projected onto a highly reflective object, it causes a truncated intensity of the fringe pattern and thus leads to an error in the phase calculation. Intensity saturation can also be considered as a special kind of non-linearity error, so the large number of patterns can eliminate the saturation error. To compensate for the phase error caused by saturation, many methods have been proposed. According to the reflectivity of the object surface, camera exposure is adjusted, intensity and contrast of the projected fringe pattern is adjusted.

Experiments

The fundamental frequency is constant if the captured image with the stripes on the reference plane is perfect as seen in the Fig. 3, where Fig. 3a perfect pattern with stripes, Fig. 3b one row from the pattern, Fig. 3c fourier transform of one row, Fig. 3d changing fundamental frequency along row. For ideal pattern with stripes, there is one constant fundamental frequency in Fourier transform as illustrated in the Fig. 3c, but in the practical the captured image is not perfect. The fundamental frequency changes with the x-axis. We can consider each period of the sine wave along the axis x in the captured pattern, it has a different fundamental frequency as in Fig. 4d, in order to improve the measurement. In the case of N = 3, the phase of the error is independent on the second harmonic [16]. So, the second frequency in the Fig. 4c can be neglected, as well as to simplify the calculation.

a)

c)

d)

Fig. 3. (a) Perfect pattern with stripes, (b) one row from the pattern, (c) Fourier transform o/ one row, (d) changing fundamental frequency along row

We implemented algorithms using Visual Studio 2022, python3.9, Opencv4.6, and also matlab2021 for drawing 3D on a laptop running Windows 10 with core i7 2.2GHz

processor. The method is tested on many objects with field of view (20 cm, 30 cm). The algorithm was tested on a three-dimensional object with eight steps, each step

000200010201000101010102010102020102020202020002090002

exceeding the previous step by 150 pm. Sinusoidal patterns are projected onto the object by the projector, then the image

a)

is captured by the camera, as shown in Fig. 5a, wrapped phase Fig. 5b, 2D heat map Fig. 5c, 3D map Fig. 5d.

b)

500 1000 1500 2000 2500 3000 3500 4000 4500 x

change fundamental frequency

500 1000 1500 2000 2500 3000 3500 4000 4500

c) f<Hz> d)

Fig. 4. Practical pattern with stripes, (b) one row from the pattern, (c) Fourier transform of one row, (d) changing fundamental

frequency along row

a)

c

d)

Fig. 5. (a) Striped image, (b) wrapped phase, (c) 2D heatmap, (d) 3D map

The profile of the 3D shape is drawn in Fig. 6 in three different cases, with the constant frequency and approximate the relation between phase and height as a propositional (linear) relation, without approximate Eq. 4, without approximate with change frequency.

When the variable fundamental frequency along the x-axis is applied in Eq. 9, the depth will be adjusted at some points up to about 200 pm according to the depth with constant fundamental frequency. Thus, the depth measurement along the X-axis is improved.

The algorithm is carried out with different number of patterns. (N = 3,7,9). The result is shown in the Fig. 7. As noted, the accuracy is increase with increasing the number of patterns. If N is greater than 7 the accuracy is improved slightly. We found that the 3-step phase shift algorithm is sensitive to non-linearity error, with 7-step and above, the algorithm is almost insensitive to non-linearity error. The error in the measurement is less than 20 pm.

According to the experimental results: The amplitude of the noise is measured by applying the algorithm on the

reference plane 10 times, after that the mean value of the amplitudes is calculated. The amplitude of the noise is about 10 (m. Approximation of the relation between the phase and height to the linear relation in Eq. 4 is possible to simplify

and speed up the calculation, with the standard deviation less than 4 (im. Phase shift profilometry algorithm was applied, and good results were obtained with an error of up to 100 (m with N= 3 and 20 |im with N= 1.

0.5

0

2.5 2.0 1.5

E E

]f 1.0 0.5 0.0 -0.5

Fig. 6. Profile (the middle row in the object)

profiles correct tilt (inclination) 3phases and 7phases

- prof3_cor

--prof9_co

Fig. 7. Comparing profiles with different numbers of patterns (3, 7, 9)

Conclusions and future work

In this article, structured light method is presented for 3D measurement. Phase shift profilometry algorithm is reviewed theoretically and applied to measure the depth of 3D object. The fundamental frequency of the projection pattern is considered variable and formulated to improve the measurement. The results are shown, the depth error is less than 20 (im. For better results, we suggest: using lens with long focal length and neutral filter for projector to increase the accuracy and eliminate the spikes in the images. Using telecentric lenses to solve the problem of scaling and perspective projections. Optical scattering behaves differently on the surface of different objects, so optimizing projector brightness and camera exposure is an important factor requiring further research.

Acknowledgments

This work was supported by the Raitec LLC company of Moscow province of Russia in the research and production department.

References

[1] Li Z, Liu X, Wen S, et al. In situ 3D monitoring of geometric signatures in the powder-bed-fusion additive manufacturing process via vision sensing methods. Sensors 2018; 18(4): 1180.

[2] Carbone V, Carocci M, Savio E, Sansoni G, De Chiffre L. Combination of a vision system and a coordinate measuring machine for the reverse engineering of freeform surfaces. Int J Adv Manuf Technol 2001; 17(4): 263-271.

[3] Cui Y, Schuon S, Chan D, Thrun S, Theobalt C. 3D shape scanning with a time-of-flight camera. 2010 IEEE Computer Society Conf on Computer Vision and Pattern Recognition 2010: 1173-1180.

[4] Alkhatib M, Bobkov A, Zadoroznaya N. Camera pose estimation based on structure from motion. Procedia Comput Sci 2021; 186: 146-153.

[5] Hu Y, Chen Q, Feng S, Tao T, Asundi A, Zuo C. A new microscopic telecentric stereo vision system-calibration, rectification, and three-dimensional reconstruction. Opt Lasers Eng 2019; 113: 14-22.

[6] Spagnolo GS, Cozzella L, Leccese F. Fringe projection profilometry for recovering 2.5 D shape of ancient coins. Acta IMEKO 2021; 10(1): 142.

[7] Takeda M, Mutoh K. Fourier transform profilometry for the automatic measurement of 3-D object shapes. Appl Opt 1983; 22(24): 3977-3982.

[8] Zuo C, Feng S, Huang L, Tao T, Yin W, Chen Q. Phase shifting algorithms for fringe projection profilometry: A review. Opt Lasers Eng 2018; 109: 23-59.

[9] Gorthi SS, Rastogi P. Fringe projection techniques: whither we are? Opt Lasers Eng 2010; 48(2): 133-140.

[10] Srinivasan V, Liu HC, Halioua M. Automated phase-measuring profilometry of 3-D diffuse objects. Appl Opt 1984; 23(18): 3105-3108.

[11] Moreno D, Taubin G. Simple, accurate, and robust projector-camera calibration. 2012 Second Int Conf on 3D Imaging, Modeling, Processing, Visualization & Transmission 2012: 464-471.

[12] Saldner HO, Huntley JM. Temporal phase unwrapping: application to surface profiling of discontinuous objects. Appl Opt 1997; 36(13): 2770-2775.

[13] Zuo C, Huang L, Zhang M, Chen Q, Asundi A. Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review. Opt Lasers Eng 2016; 85: 84-103.

[14] Zhang S. Comparative study on passive and active projector nonlinear gamma calibration. Appl Opt 2015; 54(13): 3834-3841.

[15] Yu J, Gao N, Zhang Z, Meng Z. High sensitivity fringe projection profilometry combining optimal fringe frequency and optimal fringe direction. Opt Lasers Eng 2020; 129: 106068.

[16] Pan B, Kemao Q, Huang L, Asundi A. Phase error analysis and compensation for nonsinusoidal waveforms in phase-shifting digital fringe projection profilometry. Opt Lett 2009; 34(4): 416-418.

Authors' information

Majdi Nasib Alkhatib (b. 1983) graduated from the Higher Institute for Applied Science and Technology in Syria with a master's degree in Control and Robotics. Currently, he is a PhD student at Bauman Moscow State Technical University in Russia. Research interests are computer vision and image programming. E-mail: majdi.alkhatib@yahoo.com .

Yury Dmitrievich Shmelev (b. 1997) graduated from Moscow Institute of Physics and Technology in 2021. Currently he is a PhD student at Lomonosov Moscow State University and works as the technical specialist at the Raitec LLC. Research interests are computer vision, additive manufacturing. E-mail: yura0512@gmail.com .

Olga Alexandrovna Tyshova (b. 1997) graduated from Computational Mathematics and Cybernetics faculty of the Lomonosov Moscow State University in 2022, majoring in Machine Learning. Currently she works as a researcher at Raitec LLC. Research interests are computer vision, machine learning and additive manufacturing. E-mail: otyshova@raitec.ru .

Ilya Vasilievich Sinilshchikov (b. 1996) graduated from Physics faculty of Lomonosov Moscow State University in 2019. Currently he is a PhD student at Computational Mathematics and Cybernetic faculty of the Moscow State University and holds the position of Head of R&D at Raitec LLC. Research interests are machine learning, additive manufacturing and telecommunications. E-mail: iv.sinilshchikov@raitec.ru .

Alexander Valentinovich Bobkov (b. 1974), assistant professor of Bauman Moscow State Technical University, candidate of Technical Science, PhD on Computer Science. Research interests: computer vision, automatic control, artificial intelligence. E-mail: ALexander.Bobkov@bmstu.ru .

Received February 27, 2023. The final version - August 8, 2023.