# Automatic 3-D Optical Detection on Orientation of Randomly Oriented Industrial Parts for Rapid Robotic Manipulation

Liang-Chia Chen^{1, 2}, Manh-Trung Le^{2, }*, and Xuan-Loc Nguyen^{2}

^{1}Department of Mechanical Engineering, National Taiwan University, Taiwan^{2}College of Mechanical and Electrical Engineering, National Taipei University of Technology, Taiwan

(Received 7 October 2012; Accepted 22 October 2012; Published on line 1 December 2012)

*Corresponding author: lemanhtrung@gmail.com

DOI: 10.5875/ausmt.v2i4.160

Abstract: This paper proposes a novel method employing a developed 3-D optical imaging and processing algorithm for accurate classification of an object’s surface characteristics in robot pick and place manipulation. In the method, 3-D geometry of industrial parts can be rapidly acquired by the developed one-shot imaging optical probe based on Fourier Transform Profilometry (FTP) by using digital-fringe projection at a camera’s maximum sensing speed. Following this, the acquired range image can be effectively segmented into three surface types by classifying point clouds based on the statistical distribution of the normal surface vector of each detected 3-D point, and then the scene ground is reconstructed by applying least squares fitting and classification algorithms. Also, a recursive search process incorporating the region-growing algorithm for registering homogeneous surface regions has been developed. When the detected parts are randomly overlapped on a workbench, a group of defined 3-D surface features, such as surface areas, statistical values of the surface normal distribution and geometric distances of defined features, can be uniquely recognized for detection of the part’s orientation. Experimental testing was performed to validate the feasibility of the developed method for real robotic manipulation.

Keywords: Automatic robot pick and place; intelligent 3-D imaging; range image; robotic manipulation

## Introduction

Machine vision systems are often employed in industrial robot application for enhancing robot flexibility and intelligence. As such, they have been playing an important role in factory automation. A large number of 2-D (two-dimensional) vision systems have been successfully applied for industrial robot manipulation. To meet for urgent demands in providing accurate 3-D (three-dimensional) information of industrial parts or achieving more intelligent robotic manipulation, smart 3-D imaging has emerged as the most important pursuit, especially when dealing with complicated working geometry or conditions. The existing 2-D imaging technology has limitations in handling complicated part geometry. Thus, this paper proposes a novel method by employing 3-D optical imaging and processing algorithm for accurate classification of an object’s surface characteristics.

In this method, 3-D geometry of industrial parts can be rapidly acquired by the developed one-shot imaging optical probe based on Fourier Transform Profilometry (FTP) by using digital-fringe projection at a speed of 30 frames per second or higher. One of the current significant issues in optical surface profilometry is extracting accurate dimensional and shape information from objects’ underlying detection. Active 3-D vision methods generally project structure light patterns onto the object surface for 3-D imaging. The most common 3-D measurement algorithm includes the Phase Shifting Profilometry (PSP) and Fourier Transform Profilometry. The advantages of FTP over the other methods come from its high detection rate and lower sensitivity to environmental disturbance. Takeda first proposed the ideal use of FTP by employing Fourier frequency analysis on the deformed fringe image for extracting the object phase information [1]. One-shot imaging capable of 3-D surface reconstruction is the main advantage of the method. The key challenge on FTP lies on how to effectively separate the first-order spectral data from the other frequency components, so that 3-D surface reconstruction can be accurately achieved. The possibility of separating the first-order spectral data from background frequency and other spectra has also been thoroughly researched.

One of the most important issues in 3-D imaging is the development of an effective object segmentation algorithm, which needs to happen before any object classification method becomes realistic [2-12]. The main task in object segmentation is to precisely classify scanned point clouds into foreground and background. Through the segmentation process, the points that belong to the same object are grouped and marked, which is helpful for further steps in 3-D feature extraction and object classification. Object segmentation using range images are in general more difficult and complicated than any traditional 2-D image segmentation method due to its data complexity. The foreground is often highly entangled and sometimes mixed with the background. The data acquired from various sensing devices is usually noisy with a low S/N ratio.

Most of the existing object segmentation methods may require intensive computation or lack robustness in dealing with real range data. To account for this, our method acquires a range image in robot pick and place operations that can be effectively segmented into three surface types by classifying point clouds based on the statistical distribution of the normal vector of each detected 3-D point. The scene ground is then reconstructed by applying least squares fitting and classification algorithms. When the detected parts are randomly overlapped on a workbench, a group of defined 3-D surface features, such as surface areas, statistical value of the surface normal distribution and geometric distances of defined features, can be uniquely recognized for detection of the part’s orientation.

## Methodology

Our method, outlined in Figure 1, begins by classifying point clouds in range images to three different surface types, including uniformity, roughness and noise according to the distribution uniformity of normal surface vectors within the neighboring region. The ground is then detected by applying least squares approximation to extract the uniform region as the background for individual object segmentation. The term least square describes a popular technique for solving over determined or inexactly specified systems of equations. As a result, objects in the robot work cell are extracted by using a recursive search algorithm after the scene ground is removed.

Surface classification of point clouds

Surface classification is an important step that initially detects the ground, objects and edges between different regions. The segmentation method is further refined and developed from the original method established in [13]. The method is based on surface normal vector to classify point clouds into differentdetected scene parts. In our method, spherical range images (SRI) are adapted to efficiently compute surface normals. The classification process of point clouds, illustrated in Figure 2 and Figure 3, shows the process of determining the extracted surface normal from noisy sparse data obtained from a general 3-D imager. In general, it is reasonable to state that the distribution of the normal vectors in ground surfaces is more uniform than that in the edge or curve on an object surface. Based on this observation, we can classify three different types of surfaces for further object detection purposes.

A coordinate vector in the spherical coordinate system is defined as $m={{\left( r,\theta ,\phi \right)}^{T}}$, where $r$ is the range, $\theta$ is the azimuth, and $\phi$ is the elevation component. The coordinates are constrained so that \(r\ge 0,-\pi <\theta \le \pi \), and \(-\pi /2<\theta \le \pi /2\). A point in Cartesian coordinates is represented by \(p={{(x, y, z)}^{T}}\). The transformation from the spherical to the Cartesian coordinate system is given by:

where $\upsilon $ is a unit direction vector defined as:

Similarly, the transformation from Cartesian to spherical coordinate system is given by

Given a set of $n$ measurements ${{m}_{i}}$ of a scene observed from a single viewpoint, the unit surface normal ${{n}_{i}}$ at positions ${{p}_{i}}$ can be estimated.

The resulting operator for the SRI is used to obtain the normal vector. The del operator in the Cartesian coordinate system is given by

where \(\hat{x}\), \(\hat{y}\), and \(\hat{z}\) are the unit vectors in the respective coordinate directions. We have:

From Equation (1), (3) and (4), we can obtain

The expressions in parentheses can define a rotation of a vector, so that the previous equation can be expressed as:

where

Equation (5) expresses the Cartesian del operator as a function of variables and derivatives in spherical coordinates.

Now we have a functional dependence of range on the azimuth and elevation components: $r=s\left( \theta ,\varphi \right)$. The normal vectors of a surface can be obtained by computing the derivatives of the function defining the surface. For this, we apply the del operator of Equation (5) to the function $s\left( \theta ,\varphi \right)$ to obtain:

where \({{\hat{R}}_{\theta ,\phi }}=\left[ \begin{matrix} {\hat{z}} & {\hat{y}} & {\hat{x}} \\\end{matrix} \right]{{R}_{\theta ,\phi }}\).

In the proposed method, the average dot product between the normal vector of inspecting point and its neighbors is taken as a characteristic index for classification. The dot product of two vectors can represent the similarity in orientation between them [13]. The average dot product is calculated as follows:

where $m$ is the number of neighbor points and \({{\delta}_{i}}=\frac{\overrightarrow{n}\overrightarrow{{{n}_{i}}}}{\left| \overrightarrow{n} \right|\left| \overrightarrow{{{n}_{i}}} \right|}\) is the dot product between normal vector of the inspecting point \(\overrightarrow{n}\) and the normal vector of its ${{i}^{\text{th}}}$ neighbor \(\overrightarrow{{{n}_{i}}}\).

Scene ground retrieval by least squares plan fitting

After the surface classification process, the ground in the acquired scene can be further reconstructed by least squares plane fitting. It is reasonable to state that the workbench is a relatively uniform surface type with the normal distribution of its vectors. In the proposed method, the largest portion of uniform region is taken for plane fitting since the workbench in most cases occupies the largest area of the scene. The plane fitting is employed to reconstruct a reference plane fitted with the point clouds. The process is performed by using least squares approximations for plane fitting in ${{R}^{\text{3}}}$.

Recursive search process for object marking

After removing the background, a recursive search process is applied to find all individual objects located on the workbench. This process only works on those points belonging to two types of surfaces: the uniform surface and the rough surface. The recursive method is employed to search for each detected pixel until completing the whole object region [13]. A 3-D growing process is necessary to retrieve the complete object boundary in range images. The developed method employs a raster scanning pattern, which starts the searching from the initial seeds identified from the background removal strategy. The recursive algorithm will check all eight neighboring pixels for similarity in property, continuing until all the pixels belonging to the same object are identified. Since the search algorithm does not check the sequence of line or column separately, the problem of misclassifying as stated will not occur in the recursive method.

## Experimental Results and Analyses

The proposed method has been tested using over 60 range images obtained by a high-speed 3-D surface profilometer system; the developed system is shown in Figure 5. It consists of a light fringe generator for producing color encoded fringe patterns, a high resolution CCD with a high speed of up to 60 frames per second, a personal computer with Dual Core Intel Pentium D, 3400Mhz with RAM 2 GB SDRAM for controlling the projector and acquiring the reflected light containing deformed fringe images by using a suitable frame grabber for obtaining the desired characteristic of the color pattern.

Figure 6 illustrates two experimental results obtained by the developed method, in which several electrical components having various surface features were used as the detected targets for object classification. While a more quantitative analysis was performed in another experiment, these examples were conducted for the feasibility test of the developed method. Up to now, the existing object classification methods have mainly focused on some simple cases. The proposed method can effectively resolve the issue by employing the scene ground reconstruction fitting process by extending the largest uniform surface region. As seen in Figure 6 (b), (e) and (f), the backgrounds have been accurately removed from the point clouds. The industrial parts are successfully marked and segmented by grouping points with similar distribution of surface indexes. The detection frame rate can reach up to 15 fps. This would be extremely useful to real-time robot manipulation and pick-and-place automated tasks in various industries.

## Conclusions

In this research, a novel method for accurate classification of object’s surface characteristics through employing the developed 3-D optical imaging and processing algorithm was developed to successfully detect the orientation of randomly oriented industrial parts for rapid robotic manipulation. The proposed method is even suitable for complicated 3-D geometry of industrial parts since its classification process only defines the surface class without need of complicated assumption or algorithm. The work provides a feasible method in 3-D object classification for object recognition in intelligent robot vision and the integration of the developed 3-D vision system with an industrial robot system.

## References

- M. Takeda and K. Mutoh, "Fourier-transform profilometry for the automatic-measurement of 3-D object shapes," Applied Optics, vol. 22, no. 24, pp. 3977-3982, 1983.

doi: 10.1364/AO.22.003977 - H. Gharavi and G. Shaoshuai, "3-D segmentation and motion estimation of range data for crash prevention," in IEEE Intelligent Vehicles Symposium, Istanbul, Turkey, 2007, pp. 386-391.

doi: 10.1109/IVS.2007.4290145 - C. Revol-Muller, F. Peyrin, Y. Carrillon, and C. Odet, "Automated 3D region growing algorithm based on an assessment function," Pattern Recognition Letters, vol. 23, no. 1–3, pp. 137-150, 2002.

doi: 10.1016/S0167-8655(01)00116-7 - J. Neira, J. D. Tardos, J. Horn, and G. Schmidt, "Fusing range and intensity images for mobile robot localization," IEEE Transactions on Robotics and Automation, vol. 15, no. 1, pp. 76-84, 1999.

doi: 10.1109/70.744604 - A. Golovinskiy and T. Funkhouser, "Min-cut based segmentation of point clouds," in IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), 2009, pp. 39-46.

doi: 10.1109/ICCVW.2009.5457721 - L. C. Chen, X. L. Nguyen, and C. W. Liang, "Object segmentation method using depth slicing and region growing algorithms," presented at the International Conference on 3D Systems and Applications, Tokyo, Japan, 2010.

Available: http://www.cc.ntut.edu.tw/~wwwoaa/oaa-nwww/oaa-bt/bt-data/lowan/13.pdf. - M. Roggero, "Object segmentation with region growing and principal component analysis," in Symposium of ISPRS Commission III, Photogrammetric Computer Vision, Graz, Austria, 2002, pp. 289-294.

Available: http://www.isprs.org/proceedings/XXXIV/part3/papers/paper076.pdf - Q. Wang, Q. Li, Z. Chen, J. F. Sun, and R. Yao, "Range image noise suppression in laser imaging system," Optics and Laser Technology, vol. 41, no. 2, pp. 140-147, 2009.

doi: 10.1016/j.optlastec.2008.05.029 - K. Pulli and M. Pietikäinen, "Range image segmentation based on decomposition of surface normals," presented at the Scandinavian Conference on Image Analysis (SCIA), Norway, 1993.

Available: http://www.ee.oulu.fi/~kapu/SCIApap.pdf. - L. Kil-Moo, P. Meer, and P. Rae-Hong, "Robust adaptive segmentation of range images," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 2, pp. 200-205, 1998.

doi: 10.1109/34.659940 - I. S. Chang and R. H. Park, "Segmentation based on fusion of range and intensity images using robust trimmed methods," Pattern Recognition, vol. 34, no. 10, pp. 1951-1962, 2001.

doi: 10.1016/S0031-3203(00)00124-2 - B. Heisele and W. Ritter, "Segmentation of range and intensity image sequences by clustering," in International Conference on Information Intelligence and Systems, Bethesda, MD, 1999, pp. 223-225.

doi: 10.1109/ICIIS.1999.810265 - L. C. Chen, X. L. Nguyen, and S. T. Lin, "Automatic object detection employing viewing angle histogram for range images," in IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2012, pp. 196-201.

doi:10.1109/AIM.2012.6266019

### Refbacks

- There are currently no refbacks.

Copyright © 2011-2017 AUSMT ISSN: 2223-9766

## Comments on this article

by Dalva Morais (2017-10-12)

by nicole braga (2017-10-26)