Simultaneous Hand-Eye-Workspace and Camera Calibration using Laser Beam Projection

Simultaneous Hand-Eye-Workspace and Camera Calibration using Laser Beam Projection

Jwu-Sheng Hu1, 2 and Yung-Jung Chang1, 2, *

1 Mechanical and Systems Research Laboratories, Industrial Technology Research Institute, Taiwan
2 Department of Electrical Engineering, National Chiao-Tung University, Taiwan

(Received 15 April 2013; Accepted 17 May 2013; Published on line 1 March 2014)
*Corresponding author: jshu@cn.nctu.edu.tw
DOI: 10.5875/ausmt.v4i1.205

Abstract: This work presents a novel calibration technique capable of simultaneously calibrating a camera’s intrinsic parameters and hand-eye-workspace relations. In addition to relaxing the requirement of a precise calibration reference to achieve manipulator accuracy, the proposed method functions when the hand is not in the view field of the eye. The calibration method uses a laser pointer mounted on the hand to project laser beams onto a planar object, which serves as the working plane. Collected laser spot images must adhere to certain nonlinear constraints established by each hand pose and the corresponding plane-laser intersection. This work also introduces calibration methods for two cases using single and multiple planes. A multistage closed-form solution is derived and serves as the initial guess to the nonlinear optimization procedure that minimizes errors globally, allowing the proposed calibration method to function without manual intervention. The effectiveness of the proposed method is verified by comparison with existing hand-eye calibration methods via simulation and experiments using an industrial manipulator.

Keywords: Hand/eye calibration; Camera calibration; Laser

1. Introduction

Hand-eye coordination systems can be classified based on two camera configurations: eye-in-hand and eye-to-hand. A camera mounted on the end-effector (eye-in-hand) can be maneuvered to observe a target in detail. However, with a global view, a static camera (eye-to-hand) can detect scene changes more easily. Over the past several decades, many solutions have been proposed to provide accurate parameter estimations for hand-eye calibration of the eye-in-hand configuration. Assuming that the camera is well calibrated, most works focus on calibrating geometrical relationships by using a 3D object, 2D pattern, non-structured points, or just a point. Jordt et al. [1] categorized hand/eye calibration methods depending on the type of calibration reference. Based on their results, the following discussion summarizes the various approaches:

1) Methods using a 2-D pattern or a 3-D object

Most hand/eye calibration methods adopt a reference object with known dimensions or a standard calibration object. Features such as corners or circles on the calibration object are extracted from the images. Each feature is related to a position in the reference coordinate system. The camera pose can then be determined by inversely projecting the features in an image produced in the camera calibration stage. From this viewpoint, 2-D and 3-D patterns are similar. The classical approaches to hand/eye calibration solve the transformation equation in the form, AX=XB, as first introduced by Shiu and Ahmad [2, 3]. Tsai and Lenz [4] developed the closed-form solution by decoupling the problem into two stages: rotation and translation. Quaternion-based approaches, such as those developed by Chou and Kamel [5], Zhuang and Roth [6], and Horaud and Dornaika [7], lead to a linear form to solve rotation relationships. To avoid error propagation from the rotation stage to the translation stage, Zhuang and Shiu [8] developed a nonlinear optimization method with respect to three Euler angles of a rotation and a translation vector. Horaud and Dornaika [9] also applied a nonlinear optimization method to simultaneously solve the rotation quaternion and the translation vector. Albada et al. [7] considered robot calibration based on the camera calibration method using a 2-D pattern. Daniilidis and Bayro-Corrochano [10, 11] developed a method of representing a rigid transformation in a unit dual quaternion to simultaneously solve rotation and translation relationships. While simultaneously considering hand/eye and robot/world problems, Zhuang et al. [12] introduced the transformation chain in the general form, AX=YB, and presented a linear solution based on quaternions and the linear least square method. Similarly, Dornaika and Horaud [13] included a closed-form solution and a nonlinear minimization solution. Hirsh et al. [14] developed an iterative approach to solve this problem. Li et al. [15] used a dual quaternion and Kronecker product to solve it. The method of Strobl and Hirzinger [16] requires only orthogonal conditions in grids, which relaxes the requirement for patterns with precise dimensions.

2) Methods using a single point

Wang [17] pioneered a hand/eye calibration procedure using only a single point at an unknown position. Pure translation without rotating the camera to observe the point yields a closed-form solution for the rotational relationship. Once the rotation is obtained, the translation relationship is solved via least squares fitting. Similar works such as [18] and [1] include finding intrinsic camera parameters. Gatla et al. [19] considered the special case of pan-tilt cameras attached to the hand.

3) Methods using non-structured features

This category can be considered an extension of using a single point. Rather than using a specific reference, the method developed by Ma [20] uses feature points of unknown locations in the working environment and determines the camera orientation by using three pure translations. By moving the camera without rotating it, the focus of expansion (FOE) is determined by the movements of points in the images where the relative translation is parallel to a vector connecting the optical center and the FOE point. Directions of the three translations in the camera frame are related to the direction of hand movement in the robot frame. Based on such relationships, the camera orientation relative to the end-effector can be calculated. After the orientation is obtained, the camera position with respect to the robot can be obtained by analyzing the general motions. Similarly, Wei et al. [21] computed the initial parameters and developed an automatic motion planning procedure for optimal movement to minimize parameter variances. The work of Andreff et al. [22] was based on the structure-from-motion (SfM) method. It provided a linear formulation and several solutions for combining specific end-effector motions.

4) Methods using optical-flow

Optical flow data implicitly contains the information of camera motion. To extract the relative pose from the flows and motions, Malm and Heyden [23] developed a method using pure translation followed by rotation around an axis. This category can be regarded as a unique case of using non-structured features.

Calibration of the eye-to-hand configuration in which the camera is static has seldom been discussed since the transformation can take the same form of eye-in-hand configuration. Most of the methods mentioned above are applicable to either eye-in-hand or eye-to-hand configurations. Dornaika and Horaud [13] presented a formulation to deal with the calibration problems for the both configurations, suggesting that these two problems are identical, but this is only true when the robot hand can be viewed with the eye-to-hand camera, thus limiting installation flexibility and potential applications. An eye occasionally fails to see a hand due to various conditions. For instance, to sort products on a moving conveyor, the camera is often placed at a distance from the arm to compensate for image processing delay and prevent interference when tracking targets on a rapidly moving conveyor belt. In ball catching systems, e.g., [24], cameras focus on the region of the ball’s initial trajectory and may not see the arm. Sun et al. [25] developed a robot-world calibration method using a triple laser device. Despite enabling calibration of the eye-to-hand transformation when the camera cannot see the arm, their method requires a uniquely designed triple-laser device, thus limiting the flexibility of system arrangement. Moreover, the spatial relationship from the working plane to the camera must be known in advance.

Finally, most work to calibrate hand-eye systems is done by separating camera calibration and hand-eye calibration, resulting in the subsequent propagation of errors. In many industrial applications, cameras must be frequently recalibrated. Camera calibration requires reference objects including 3-D objects [26], 2-D patterns [27, 28], or 1-D bars [29]. However, in an environment that humans cannot easily enter to place a reference object, separating the calibration might become inefficient. Hence, Ma [20] developed a method that combines the two calibrations into a single process for an active vision system.

This work presents a novel eye-to-hand system configuration which simultaneously considers camera intrinsic parameters and geometrical relationships of a working plane simultaneously. The proposed approach uses a laser pointer mounted on the hand thus overcoming problems which would otherwise arise when the eye does not see the hand. In other words, this method allows for a longer working range for hand-eye coordination. By manipulating the robot and projecting the laser beam on a plane with unknown orientations, a batch of related image positions of light spots is extracted from camera images. Since the laser is mounted rigidly and the plane is fixed at each orientation, the geometrical parameters and measurement data must comply with certain nonlinear constraints and the parameter solutions can be estimated accordingly. Closed-form solutions are developed by decoupling nonlinear equations into linear forms to estimate all initial values. Consequently, the proposed calibration method does not require manual initial guesses for unknown parameter values. To achieve high accuracy, a nonlinear optimization method is implemented to refine the estimation.

This proposed approach is found to provide a significant improvement over previous results in [30] and [31]. As opposed to [30], the proposed method requires only a single plane to project the laser beam onto multiple planes for a single hand pose. The multi-plane case increases the difficulty of data collection since the laser spot may run outside the camera’s view field when the plane pose is changed. Moreover, whereas the hand-eye-workspace calibration method developed in [31] relies on a pre-calibrated camera, the proposed method can simultaneously calibrate the hand-eye-workspace relationships and the camera’s intrinsic parameters without prior information.

The rest of this paper is organized as follows. Section 2 describes the parameters for calibration and notations. Section 3 introduces a detailed method for solving the calibration problem with a laser beam and presents the closed-form solutions and nonlinear optimization based on the constraint. Section 4 summarizes the experimental results obtained by simulations and using real data. Conclusions are finally drawn in Section 5.

2. Parameters for calibration and notations

This section introduces the parameters of the eye-to-hand system for calibration, including intrinsic and extrinsic camera parameters and the geometrical relationships of the working plane in relation to the robot base coordinate system.

1) Intrinsic camera parameters:

A pin-hole camera with lens distortion is used. A 3D position of the Cartesian coordinate system in the camera frame is denoted by ${}^{c}p={{\left[ {}^{c}x,{}^{c}y,{}^{c}z \right]}^{T}}$. A 2D position in an image is denoted by $m={{\left[ u,v \right]}^{T}}$ and its homogeneous coordinate is defined by $\tilde{m}={{\left[ u,v,1 \right]}^{T}}$.

\[\tilde{m}=K\cdot \left[ \begin{matrix} {{x}_{d}} \\ 1 \\ \end{matrix} \right] \text{with } K=\left[ \begin{matrix} {{f}_{u}} & {{\alpha }_{c}}\cdot {{f}_{u}} & {{u}_{0}} \\ 0 & {{f}_{v}} & {{v}_{0}} \\ 0 & 0 & 1 \\ \end{matrix} \right],\tag{1}\]

where $k$, the intrinsic matrix, has five intrinsic parameters, in which ${{f}_{u}}$ and ${{f}_{v}}$ are focal lengths in pixels; aspect ratio is ${{{f}_{u}}}/{{{f}_{v}}}$ ; ${{\alpha }_{c}}$ is the screw coefficient; and and are principal points.

\[\begin{align} & {{x}_{d}}=\left[ \begin{matrix} {{x}_{d}} \\ {{y}_{d}} \\ \end{matrix} \right] \\ & =\left( 1+{{\kappa }_{1}}{{\left\| {{x}_{r}} \right\|}^{2}}+{{\kappa }_{2}}{{\left\| {{x}_{r}} \right\|}^{4}} \right){{x}_{r}}+\left[ \begin{matrix} 2{{\rho }_{1}}{{x}_{r}}{{y}_{r}}+{{\rho }_{2}}({{\left\| {{x}_{r}} \right\|}^{2}}+2{{x}_{r}}^{2}) \\ {{\rho }_{1}}({{\left\| {{x}_{r}} \right\|}^{2}}+2{{y}_{r}}^{2})+2{{\rho }_{2}}{{x}_{r}}{{y}_{r}} \\ \end{matrix} \right], \\ \end{align}\tag{2}\]

where ${{x}_{r}}={{\left[ \begin{matrix} {{x}_{r}} & {{y}_{r}} \\ \end{matrix} \right]}^{T}}\triangleq {{\left[ \begin{matrix} {{}^{C}x}/{{}^{C}z}\; & {{}^{C}y}/{{}^{C}z}\; \\ \end{matrix} \right]}^{T}}$ denotes the ray direction from camera center to the point; ${{\kappa }_{1}}$ and ${{\kappa }_{2}}$ denote two radial distortion parameters; and ${{\rho }_{1}}$ and ${{\rho }_{2}}$ denote two tangential distortion parameters. For the pinhole model without considering distortion, all distortion parameters are zeros and ${{x}_{d}}={{x}_{r}}$.

2) Eye-to-hand transformation:

Transformation from the camera frame to the robot base frame is determined by a rotation matrix ${}_{C}^{B}R$ and a translation vector ${}_{C}^{B}T$. A point ${}^{C}p$ in the 3D Cartesian coordinate system in the camera frame is transformed to the robot base coordinate system via

\[{}^{B}P={}_{C}^{B}R\cdot {}^{C}P+{}_{C}^{B}T.\tag{3}\]

The rotation matrix is derived by the direction cosine matrix, $R\left( \theta ,\phi ,\varphi \right)$. The rotation sequence is Z-Y-X, where $\theta $ denotes the z-axis rotation, $\phi $ denotes the y-axis rotation, and $\varphi $ denotes the x-axis rotation. In the following context, the robot base coordinate system is identical to the world coordinate of the entire system. Transformation from the end-effector frame to the robot base frame is determined by a rotation matrix ${}_{E}^{B}R$ and a translation vector ${}_{E}^{B}T$.

3) Working plane:

A planar object can be defined by a normal vector ${}_{\Pi }^{B}n$ nd a position ${}_{\Pi }^{B}T$ on the plane in the robot base coordinate system. A plane in space has three degrees of freedom, and, in general, the plane is also defined as the following vector:

\[{}_{\Pi }^{B}a=({}_{\Pi }^{B}{{n}^{T}}\cdot {}_{\Pi }^{B}T){}_{\Pi }^{B}n.\tag{4}\]

3. The proposed method

Calibration is used to reduce systematic errors by correcting the parameters introduced in Section 2. This section provides a detailed description of the calibration methods of an eye-to-hand system. The feasibility of using a plane and a laser pointer to calibrate an eye-to-hand system is discussed first. Two closed-form solutions for multiple plane poses and for a single plane are then proposed. The single plane case calibration assumes that the principal point and the aspect ratio are known in advance. This assumption is reasonable since most cameras are made such that their principal points are located at the center of the image sensor; in addition, two image axes are set perpendicular to each other, i.e. the aspect ratio is approximately zero.

3.1 Calibration with a laser pointer

Figure 1 illustrates the overall configuration of an eye-to-hand system in which a laser pointer is attached to the end-effector and the working plane is in front of the camera. The robot base coordinate system is defined as a world coordinate system as follows.

Figure 2 illustrates the relationships between the laser pointer and the end-effector. In the coordinate system of the end-effector, the direction of the laser beam is described as a unit vector , which has only two degrees of freedom and can also be denoted by Euler angles as

\[{}_{\Lambda }^{E}u={{\left[ \begin{matrix} \cos {}_{\Lambda }^{E}\theta \sin {}_{\Lambda }^{E}\phi & \sin {}_{\Lambda }^{E}\theta \sin {}_{\Lambda }^{E}\phi & \cos {}_{\Lambda }^{E}\phi \\ \end{matrix} \right]}^{T}},\tag{5}\]

where ${}_{\Lambda }^{E}\theta \in [0,2\pi )$ denotes the z-axis rotation angle and ${}_{\Lambda }^{E}\phi \in \left[ 0,\pi \right]$ denotes the angle between the z-axis and ${}_{\Lambda }^{E}u$. At any point on the line, laser position ${}_{\Lambda }^{E}T$ has only two degrees of freedom. Without a loss of generality, let ${}_{\Lambda }^{E}T$ denote the point at which the laser beam intersects the y-z plane of the end-effector coordinate system, i.e.

\[{}_{\Lambda }^{E}T\triangleq {}_{\Lambda }^{E}{{[\begin{matrix} 0 & y & z \\ \end{matrix}]}^{T}}.\tag{6}\]

In the following context, the laser beam is assumed here to not be parallel to the y-z plane of the robot end-effector; otherwise, let ${}_{\Lambda }^{E}T\triangleq {}_{\Lambda }^{E}{{[\begin{matrix} x & 0 & z \\ \end{matrix}]}^{T}}$ or ${}_{\Lambda }^{E}T\triangleq {}_{\Lambda }^{E}{{[\begin{matrix} x & y & 0 \\ \end{matrix}]}^{T}}$. These values are additional parameters that must be calibrated. Since the laser pointer is rigidly installed, the parameters do not change under normal operation.

The 3D position of the laser beam projected onto the plane can be calculated using the following three methods.

1) Intersection of the laser beam and plane:

The 3D position of a spot on the plane is derived via the principle of intersection of a line and a plane in 3D space. The geometrical constraint is

\[{{({{d}_{L}}{}_{\Lambda }^{B}u+{}_{\Lambda }^{B}T-{}_{\Pi }^{B}a)}^{T}}\cdot {}_{\Pi }^{B}a=0,\tag{7}\]

where ${{d}_{L}}$ denotes the distance from the laser origin to the projected point and

\[{{d}_{L}}=\frac{\left\| {}_{\Pi }^{B}a \right\|-{}_{\Lambda }^{B}{{T}^{T}}\cdot {}_{\Pi }^{B}a}{{}_{\Lambda }^{B}{{u}^{T}}\cdot {}_{\Pi }^{B}a}.\tag{8}\]

Hence, the 3D position of the laser spot is

\[{}^{B}{{p}_{L}}={{d}_{L}}{}_{\Lambda }^{B}u+{}_{\Lambda }^{B}T.\tag{9}\]

2) Intersection of the plane and laser beam from the camera:

A laser spot is projected onto an image taken by the camera. The beam direction ${{x}_{r}}$, from the origin of the camera frame to the laser spot, can be obtained using pixel position $m$ by applying the inverse intrinsic matrix and the undistortion method. According to the principle of a line and a plane intersecting in 3D space, the geometrical constraint can be expressed as

\[{{({}_{C}^{B}R\cdot {{d}_{C}}{}^{C}{{\tilde{x}}_{r}}+{}_{C}^{B}T-{}_{\Pi }^{B}a)}^{T}}\cdot {}_{\Pi }^{B}a=0,\tag{10}\]

where ${}^{C}{{\tilde{x}}_{r}}={{\left[ \begin{matrix} {{x}_{r}}^{T} & 1 \\ \end{matrix} \right]}^{T}}$, and then

\[{{d}_{C}}=\frac{\left\| {}_{\Pi }^{B}a \right\|-{}_{C}^{B}{{T}^{T}}\cdot {}_{\Pi }^{B}a}{{}^{B}{{{\tilde{x}}}_{r}}^{T}\cdot {}_{\Pi }^{B}a}.\tag{11}\]

Hence, the 3-D position of the laser spot in the robot frame is

\[{}^{B}{{p}_{L}}={}_{C}^{B}R\cdot {{d}_{C}}{}^{C}{{\tilde{x}}_{r}}+{}_{C}^{B}T.\tag{12}\]

3) Triangulation of a laser beam from the camera:

Position ${}^{B}{{p}_{L}}$ can also be determined by triangulation of a laser beam and the camera. The point, which is the intersection of two lines, has the following position:

\[{}^{B}{{p}_{L}}={{d}_{L}}{}_{\Lambda }^{B}u+{}_{\Lambda }^{B}T={{d}_{C}}{}_{C}^{B}R{}^{C}{{\tilde{x}}_{r}}+{}_{C}^{B}T.\tag{13}\]

This leads to the equation:

\[\left[ \begin{matrix} {}_{\Lambda }^{B}u & -{}_{C}^{B}{{R}^{C}}{{{\tilde{x}}}_{r}} \\ \end{matrix} \right]\left[ \begin{matrix} {{d}_{L}} \\ {{d}_{C}} \\ \end{matrix} \right]={}_{C}^{B}T-{}_{\Lambda }^{B}T.\tag{14}\]

which can be solved by the least squares method.

Analysis results indicate that this system configuration contains redundant information. Positional differences of the same point derived via different methods are caused by systematic errors and/or noise in measurements. Restated, the systematic parameters can be estimated by tuning these parameters to reduce the positional differences of the points computed from two of these three methods. Moreover, creating a batch of projected points allows one to reduce noise effects via various optimization methods.

The optimization problem, which is described later, is nonlinear. A good initial guess is necessary to prevent local minimum and accelerate convergence. Closed-form solutions are derived in the next two subsections by exploring the unique arrangements in the setup. Since no prior guess of parameters is needed, the proposed method can run automatically.

Two cases are proposed, the first with multiple plane poses and the second with a single plane pose. The laser spots that lie on the plane under two or more different poses can provide abundant 3-D information in 3-D space, but only 2-D information under a single plane pose. Hence, it is impossible to explore all intrinsic camera parameters in the single plane case without an assumption. In reality, reasonable prior knowledge is that in which the skew coefficient is extremely small (i.e. the two axes of the image sensor are mutually perpendicular), in which the principle point is at the center of image sensor (i.e., the lens optical axis is through the image center), and/or in which the two focal lengths along two axes are proportional along with the pixel width and length.

Manipulating the end-effector without rotation can generate constraint equations from parallel laser beams. Under the $p$-th plane pose, assume that the end-effector holds at the ($p$,$m$)-th orientation and moves along a direction (without rotation) which is a linear combination of $N$ translating vectors ($j_{p,m,n}^{{}}$ vectors). The laser spot moves along a vector that is the same linear combination of corresponding vectors ($i_{p,m,n}^{{}}$ vectors). In particular, for a fixed hand orientation, the end-effector moves along several directions (Figure 3). The effect of lens distortion is ignored in the closed-form solution. In other words, the pin-hole camera model of (1) without consideration of distortion is adopted. Under the $p$-th plane pose, ($p$,$m$)-th rotation and ($p$,$m$,$n$)-th translation, the ($p$,$m$,$k$)-th laser spot is at position

\[\begin{align} & {{p}_{p,m,k}}={{\left[ \begin{matrix} z\cdot {{x}_{r}} \\ z \\ \end{matrix} \right]}_{p,m,k}} \\ & \text{ }={{H}_{p,m}}\left[ \begin{matrix} {{w}_{p,m,k}} \\ 1 \\ \end{matrix} \right] \\ & \text{ }\triangleq K\left[ \begin{matrix} {}^{C}i_{p,m,1}^{{}} & \cdots & {}^{C}i_{p,m,N}^{{}} & {}^{C}{{T}_{p,m}} \\ \end{matrix} \right]{{\left[ \begin{matrix} w_{{}}^{1} \\ \vdots \\ w_{{}}^{N} \\ 1 \\ \end{matrix} \right]}_{p,m,k}} \\ \end{align},\tag{15}\]

where ${{w}^{n}}$, $n=\text{1}\tilde{\ }N$ denote scales of a combination according to end-effector movement, $z$ is a related scalar, and ${{H}_{p,m}}$ denotes a relative homography matrix.

The closed-form solutions are derived from the aspect of measurements of the camera to the aspect of robot movements. Pure hand translations can eliminate the translation terms of parameters in equations. The relations can then be transformed into several linear forms to find angle related terms of the parameters. After all directions and orientations are obtained, the translation terms and the scaling factors up to real dimensions can be obtained. The step by step calculations of the two closed-form solution are introduced as follows.

3.2 Closed-form solution for a multiple plane case

1) Finding homography matrixes with unknown scales:

The entries of a homography matrix can be obtained by eliminating according to direct linear transformation (DLT) method [1, 20]. Let ${{\bar{h}}^{a}}$ denote the $a$-th row of $H$. Under the $p$-th plane pose and the ($p$,$m$)-th hand orientation, Equation (15) can be rewritten as

\[{{Q}_{H,p,m,k}}\cdot {{x}_{H,p,m}}\triangleq {{\left[ \begin{matrix} \begin{matrix} w_{K}^{T} \\ 0 \\ \end{matrix} & \begin{matrix} 0 \\ w_{K}^{T} \\ \end{matrix} & \begin{matrix} -{{u}_{k}}w_{K}^{T} \\ -{{v}_{k}}w_{k}^{T} \\ \end{matrix} \\ \end{matrix} \right]}_{p,m}}{{\left[ \begin{matrix} \mathop{{{{\bar{h}}}^{1}}}^{T} \\ \mathop{{{{\bar{h}}}^{2}}}^{T} \\ \mathop{{{{\bar{h}}}^{3}}}^{T} \\ \end{matrix} \right]}_{p,m}}=0.\tag{16}\]

By the given K points, the collected ${{Q}_{H,p,m,k}}$, $k=\text{1}\tilde{\ }K$, denoted as ${{Q}_{H,p,m}}$ is a $2K\times 9$ matrix. The solution ${{x}_{H,p,m}}$ lies in the null space of the matrix ${{Q}_{H,p,m}}$ and can be solved by calculating the eigenvector corresponding to the smallest eigenvalue of matrix $Q_{H,p,m}^{T}{{Q}_{H,p,m}}$. This state of the art of eigen decomposition approach is based on singular value decomposition. Then, the estimated matrix ${{\hat{H}}_{p,m}}$, which is equivalent to the homography, is obtained with an unknown scaling factor:

\[{{H}_{p,m}}={{\lambda }_{p,m}}{{\hat{H}}_{p,m}}\triangleq {{\lambda }_{p,m}}\left[ \begin{matrix} \hat{i}_{p,m,1}^{{}} & \cdots & \hat{i}_{p,m,N}^{{}} & {{{\hat{T}}}_{p,m}} \\ \end{matrix} \right].\tag{17}\]

For the same plane pose, the laser points should be all on this plane regardless of orientation. Thus, the constraint exists as

\[{{\hat{T}}_{p,1}}={{a}_{p,m}}\cdot \hat{i}_{p,m,1}^{{}}+{{b}_{p,m}}\cdot \hat{i}_{p,m,2}^{{}}+{{c}_{p,m}}\cdot {{\hat{T}}_{p,m}}.\tag{18}\]

where, ${{\hat{T}}_{p,1}}$ is a linear combination of $\hat{i}_{p,m,1}^{{}}$, $\hat{i}_{p,m,2}^{{}}$, and ${{\hat{T}}_{p,m}}$ with scalars, ${{a}_{p,m}}$, ${{b}_{p,m}}$, and ${{c}_{p,m}}$, and the three scalars can be calculated directly. Consequently, ${{\lambda }_{p,1}}={{c}_{p,m}}{{\lambda }_{p,m}}$; hence only one unknown scale factor is undetermined for each plane pose. Here, ${{\hat{H}}_{p,m}}$ is multiplied by ${{c}_{p,m}}$ and then ${{\hat{H}}_{p,m}}$s’, $m=\text{1}\tilde{\ }M$, are in the same dimension. Let ${{\lambda }_{p}}\triangleq {{\lambda }_{p,1}}$, and these $P$ unknown scale factors can be obtained by the following steps.

2) Finding the laser direction relative to the robot frame and to the end-effector frame:

Since the laser is moved without rotation in the one hand orientation, the laser beams ${{u}_{p,m,k}}$’s($k$=1,…,$K$) are parallel to each other. The translations of the end-effector are related to the laser spot movements on the plane by projecting the translation vectors to the plane along the laser direction (Figure 4). The projecting equation is

\[i_{p,m,n}^{{}}=j_{p,m,n}^{{}}+d_{p,m,n}^{{}}\cdot u_{p,m}^{{}},\tag{19}\]

where $i_{p,m,n}^{{}}$ denotes the laser spot movement; $j_{p,m,n}^{{}}$ denotes hand translation; and $d_{p,m,n}^{{}}$ denotes the distance along the laser beam. This equation has another form,

\[i_{p,m,n}^{{}}=P_{p,m}^{{}}\cdot j_{p,m,n}^{{}}\triangleq \left( I-\frac{u_{p,m}^{{}}\cdot {}_{\Pi }^{{}}n_{p}^{T}}{u_{p,m}^{T}\cdot {}_{\Pi }^{{}}n_{p}^{{}}} \right)\cdot j_{p,m,n}^{{}}.\tag{20}\]

By considering the coordinate systems and the intrinsic matrix, Equation (20) becomes

\[{{\hat{i}}_{p,m,n}}={{\hat{P}}_{p,m}}\cdot {}^{B}{{j}_{p,m,n}}\triangleq \frac{1}{{{\lambda }_{p}}}K\cdot {}_{B}^{C}R\cdot {}_{{}}^{B}{{P}_{p,m}}\cdot {}^{B}{{j}_{p,m,n}}\tag{21}\]

The hand translations ${}^{B}{{j}_{p,m,n}}$s are from robot commands and ${{\hat{i}}_{p,m,n}}$s are from a previous step. Equation (21) forms a set of linear equations,

\[\begin{align} & {{A}_{P,p,m,n}}{{x}_{P,p,m}}\triangleq \left[ \begin{matrix} {}^{B}j_{p,m,n}^{T} & 0 & 0 \\ 0 & {}^{B}j_{p,m,n}^{T} & 0 \\ 0 & 0 & {}^{B}j_{p,m,n}^{T} \\ \end{matrix} \right]{{\left[ \begin{matrix} \mathop{{{{\hat{\bar{p}}}}^{1}}}^{T} \\ \mathop{{{{\hat{\bar{p}}}}^{2}}}^{T} \\ \mathop{{{{\hat{\bar{p}}}}^{3}}}^{T} \\ \end{matrix} \right]}_{p,m}} \\ & ={{b}_{P,p,m,n}}\triangleq {{{\hat{i}}}_{p,m,n}} \\ \end{align},\tag{22}\]

where ${{\hat{\bar{p}}}^{a}}$ denotes the a-th row of matrix ${{\hat{P}}_{p,m}}$. For N hand translations, the equation $A_{P,p,m}^{{}}{{x}_{P,p,m}}=b_{P,p,m}^{{}}$ is formed to obtain a unique solution of ${{\hat{P}}_{p,m}}$. To solve these equations, at least three independent translation vectors are required; otherwise, the matrix $A_{P,p,m}^{{}}$ is singular.

According to equations (20) and (21), the laser beam ${{u}_{p,m}}$ is the kernel of the matrix ${{p}_{p,m}}$ as well as ${{\hat{P}}_{p,m}}$; therefore, ${{\hat{P}}_{p,m}}\cdot {}^{B}{{u}_{p,m}}=0$. Restated, the laser spot does not move if the end-effector translates along the laser direction. Hence, the laser direction with respect to the robot coordinate system can be determined by calculating the eigenvector corresponding to the smallest eigenvalue of ${{\hat{P}}_{p,m}}^{T}\cdot {{\hat{P}}_{p,m}}$.

With all ${}^{B}{{u}_{p,m}}$ and the hand rotations ${}_{E}^{B}{{R}_{p,m}}$, $p=\text{1}\tilde{\ }P$, $m=\text{1}\tilde{\ }M$, the laser direction with respect to the end-effector coordinate system ${}_{\Lambda }^{E}u$ can be obtained from these simultaneous equations

\[{{A}_{u}}x_{u}^{{}}\triangleq \left[ \begin{matrix} {}_{E}^{B}R_{1,1}^{{}} \\ \vdots \\ {}_{E}^{B}R_{P,M}^{{}} \\ \end{matrix} \right]{}_{\Lambda }^{E}u=b_{u}^{{}}\triangleq \left[ \begin{matrix} {}_{{}}^{B}u_{1,1}^{{}} \\ \vdots \\ {}_{{}}^{B}u_{P,M}^{{}} \\ \end{matrix} \right],\tag{23}\]

by applying the least squares method and normalizing the solution to $\left| {}_{\Lambda }^{E}u \right|=1$. According to Equation (23), only one orientation is necessary to calculate a unique ${}_{\Lambda }^{E}u$. But at least two orientations are needed to find an unique laser beam position ${}_{\Lambda }^{E}T$, which is obtained in the following step. It fully reflects the necessary condition stated by Shiu and Ahmad [3] and Tsai and Lenz [4] to solve the translation term of hand-eye relationships.

3) Finding the intrinsic matrix and the hand-to-eye rotation matrix:

According to figure 4, the hand translating vector ${}^{B}j_{p,m,n}^{{}}$, laser beam ${}^{B}{{u}_{p,m}}$ and projected vector ${}^{B}{{i}_{p,m,n}}$ lie on the same plane in space. Thus, we have ${{\left( {}^{B}{{u}_{p,m}}\times {}^{B}j_{p,m,n}^{{}} \right)}^{T}}\cdot {}^{B}{{i}_{p,m,n}}=0$, which is equivalent to,

\[{{\left( {}^{B}{{u}_{p,m}}\times {}^{B}j_{p,m,n}^{{}} \right)}^{T}}\cdot {{G}^{-1}}\cdot {{\hat{i}}_{p,m,n}}=0.\tag{24}\]

Let ${{\hat{n}}_{p,m,n}}\triangleq \left( {}^{B}{{u}_{p,m}}\times {}^{B}j_{p,m,n}^{{}} \right)$ and $\hat{G}={{G}^{-1}}$, where the transformation matrix $G$ is defined as

\[G=\left[ \begin{matrix} {{g}_{11}} & {{g}_{12}} & {{g}_{13}} \\ {{g}_{21}} & {{g}_{22}} & {{g}_{23}} \\ {{g}_{31}} & {{g}_{32}} & {{g}_{33}} \\ \end{matrix} \right]\triangleq K\cdot {}_{B}^{C}R\tag{25}\]

This equation can be placed in another form:

\[\left[ \begin{matrix} \hat{n}_{p,m,n}^{1}\hat{i}_{p,m,n}^{\text{ }T} & \hat{n}_{p,m,n}^{2}\hat{i}_{p,m,n}^{\text{ }T} & \hat{n}_{p,m,n}^{3}\hat{i}_{p,m,n}^{\text{ }T} \\ \end{matrix} \right]\left[ \begin{matrix} \mathop{{{{\hat{\bar{g}}}}^{1}}}^{T} \\ \mathop{{{{\hat{\bar{g}}}}^{2}}}^{T} \\ \mathop{{{{\hat{\bar{g}}}}^{3}}}^{T} \\ \end{matrix} \right]=0,\tag{26}\]

where $\hat{n}_{p,m,n}^{a}$ denotes the $a$-th entry of vector ${{\hat{n}}_{p,m,n}}$ and ${{\hat{\bar{g}}}^{a}}$ denotes the $a$-th row of matrix $\hat{G}$. For all $N$ translations, $M$ orientations, and $P$ plane poses,

\[{{Q}_{G}}{{x}_{G}}\triangleq \left[ \begin{matrix} \hat{n}_{1,11}^{1}\hat{i}_{1,1,1}^{\text{ }T} & \hat{n}_{1,11}^{2}\hat{i}_{1,11}^{\text{ }T} & \hat{n}_{1,11}^{3}\hat{i}_{1,11}^{\text{ }T} \\ \vdots & \vdots & \vdots \\ \hat{n}_{P,M,N}^{1}\hat{i}_{P,M,N}^{\text{ }T} & \hat{n}_{P,M,N}^{2}\hat{i}_{P,M,N}^{\text{ }T} & \hat{n}_{P,M,N}^{3}\hat{i}_{P,M,N}^{\text{ }T} \\ \end{matrix} \right]\left[ \begin{matrix} \mathop{{{{\hat{\bar{g}}}}^{1}}}^{T} \\ \mathop{{{{\hat{\bar{g}}}}^{2}}}^{T} \\ \mathop{{{{\hat{\bar{g}}}}^{3}}}^{T} \\ \end{matrix} \right]=0.\tag{27}\]

Solution ${{x}_{G}}$ lies in the null space of the matrix ${{Q}_{G}}$. The solution can be derived by calculating the eigenvector corresponding to the smallest eigenvalue of matrix ${{Q}_{G}}^{T}{{Q}_{G}}$. Solution ${{x}_{G}}$ has eight degrees of freedom; thus, at least four data sets are necessary to form the eight equations in equation (27). Since each row of the rotation matrix ${}_{B}^{C}R$ is a unit vector, and the last entry of $K$ is 1, the solution is scaled to satisfy ${{g}_{31}}^{2}+{{g}_{32}}^{2}+{{g}_{33}}^{2}=1$ and then the $G$ is obtained. By applying RQ-decomposition, the homogeneous matrix $G$ is separated into the intrinsic matrix $K$ and the rotation matrix ${}_{B}^{C}R$.

4) Finding the scaling factors and the translations of the laser and the camera relative to the robot:

A laser beam is emitted from its origin on the hand to the position on the plane. Under the ($p$,$m$,$k$)-th plane pose and hand pose, the laser is set at position ${}_{\Lambda }^{B}{{p}_{p,m,k}}={}_{E}^{B}{{R}_{p,m}}{}_{\Lambda }^{E}T+{}_{E}^{B}{{T}_{p,m,k}}$ in the robot base coordinate system, and the laser spot is set at

\[{}_{\Pi }^{C}p_{p,m,k}^{{}}=\lambda _{p}^{{}}{{K}^{-1}}(w_{p,m,k}^{1}{}_{{}}^{C}\hat{i}_{p,m,1}^{{}}+\cdots +w_{p,m,k}^{N}{}_{{}}^{C}\hat{i}_{p,m,N}^{{}}+{}_{{}}^{C}{{\hat{T}}_{p,m}}),\tag{28}\]

in the camera frame. After transforming the laser frame to the camera frame, the laser beam from the robot side to the plane is

\[{}_{{}}^{C}{{t}_{p,m,k}}={}_{\Pi }^{C}{{p}_{p,m,k}}-{}_{B}^{C}R{}_{\Lambda }^{B}{{p}_{p,m,k}}-{}_{B}^{C}T,\tag{29}\]

which is parallel to laser beam ${}^{C}{{u}_{p,m}}$, i.e., ${}^{C}{{u}_{p,m}}\times {}_{{}}^{C}{{t}_{p,m,k}}=0$. Consequently,

\[\left[ \begin{matrix} -{{\left\lfloor {}_{{}}^{C}{{u}_{p,m}} \right\rfloor }_{\times }}{}_{B}^{C}R{}_{E}^{B}R_{p,m}^{{}} & -{{\left\lfloor {}_{{}}^{C}{{u}_{p,m}} \right\rfloor }_{\times }} & {{\left\lfloor {}_{{}}^{C}{{u}_{p,m}} \right\rfloor }_{\times }}{{K}^{-1}}{{{\hat{H}}}_{p,m}}\left[ \begin{matrix} w_{p,m,k}^{{}} \\ 1 \\ \end{matrix} \right] \\ \end{matrix} \right]\]
\[\cdot {{\left[ \begin{matrix} {}_{\Lambda }^{E}{{T}^{T}} & {}_{B}^{C}{{T}^{T}} & {{\lambda }_{p}} \\ \end{matrix} \right]}^{T}}={{\left\lfloor {}_{{}}^{C}{{u}_{p,m}} \right\rfloor }_{\times }}{}_{B}^{C}R{}_{E}^{B}T_{p,m,k}^{{}}\tag{30}\]

where

\[{{\left\lfloor {}_{{}}^{C}{{u}_{p,m}} \right\rfloor }_{\times }}=\left[ \begin{matrix} 0 & -{}_{{}}^{C}u_{p,m}^{3} & {}_{{}}^{C}u_{p,m}^{2} \\ {}_{{}}^{C}u_{p,m}^{3} & 0 & -{}_{{}}^{C}u_{p,m}^{1} \\ -{}_{{}}^{C}u_{p,m}^{2} & {}_{{}}^{C}u_{p,m}^{1} & 0 \\ \end{matrix} \right].\tag{31}\]

According to Equation (6), the first portion of Equation (30) should be modified as ${{\left\lfloor {}_{{}}^{C}{{u}_{p,m}} \right\rfloor }_{\times }}{}_{B}^{C}R{}_{E}^{B}\left[ \begin{matrix} {{r}^{2}} & {{r}^{3}} \\ \end{matrix} \right]_{p,m}^{{}}$ , and the first entries of ${}_{\Lambda }^{E}T$ should be eliminated. The $P\times M\times K$ sets of data form the equation

\[{{A}_{T,\lambda }}{{x}_{T,\lambda }}={{b}_{T,\lambda }},\tag{32}\]

where ${{A}_{T,\lambda }}$ denotes a matrix of size $\left( 3PMK \right)\times (5+P)$, ${{b}_{T,\lambda }}$ denotes a vector of size $3PMK$, and ${{x}_{T,\lambda }}=\left[ \begin{matrix} {}_{\Lambda }^{E}y & {}_{\Lambda }^{E}z & {}_{C}^{B}{{T}^{T}} & {{\lambda }_{1}} & \cdots & {{\lambda }_{P}} \\ \end{matrix} \right]{}^{T}$. This equation can be solved using the least squares method. The laser position in the end-effector frame and the camera position in the robot base frame are then obtained. The homography matrices related to robot dimension are scaled as ${{H}_{p,m}}={{\lambda }_{p}}{{\hat{H}}_{p,m}}$.

5) Finding the planes:

Vectors, ${}^{C}i_{p,m,n}^{{}}={{\lambda }_{p}}{{K}^{-1}}\cdot \hat{i}_{p,m,n}^{{}}$, $m$=1,…, $M$ and $n$=1,.., $N$, lie on the $p$-th plane. The plane normal vector is orthogonal to these vectors. Hence, the normal vector of each plane can be obtained from

\[{{A}_{\Pi ,p}}{{x}_{\Pi ,p}}=\left[ \begin{matrix} {}^{C}i_{p,1,1}^{{}} \\ \vdots \\ {}^{C}i_{p,M,N}^{{}} \\ \end{matrix} \right]{}_{\Pi }^{C}{{n}_{p}}=0\tag{33}\]

The laser spot positions, ${}^{C}T_{p,m,n}^{{}}={{\lambda }_{p}}{{K}^{-1}}\cdot \hat{T}_{p,m}^{{}}$, $m$ = 1,…, $M$, are on the $p$-th plane and the plane position can be obtained from these laser spot positions by averaging them as

\[{}_{\Pi }^{C}{{a}_{p}}=\left( \frac{1}{M}\sum\limits_{m=1}^{M}{{}^{C}T{{_{p,m}^{{}}}^{T}}\cdot {}_{\Pi }^{C}{{n}_{p}}} \right){}_{\Pi }^{C}{{n}_{p}}.\tag{34}\]

By transforming into the robot base frame, Equation(34) becomes

\[{}_{\Pi }^{B}{{a}_{p}}=\left( {{\left( {}_{C}^{B}R{}_{\Pi }^{C}{{a}_{p}}+{}_{C}^{B}T \right)}^{T}}\cdot {}_{C}^{B}R{}_{\Pi }^{C}{{n}_{p}} \right)\cdot {}_{C}^{B}R{}_{\Pi }^{C}{{n}_{p}}.\tag{35}\]

3.3 Closed-form solution for a single plane case

As mentioned earlier, some a priori assumptions must be made, namely that ${{\alpha }_{c}}=0$, ${{u}_{0}}$ and ${{v}_{0}}$ are known, and/or $f={{f}_{u}}={{f}_{v}}$.

1) Finding homography matrixes with unknown scales

The step to determine the homography matrix resembles the first step of the multiple plane case, unless the image position is relative to the nominal principal point. Equation (16) is changed into

\[{{Q}_{H,m}}\cdot {{x}_{H,m}}\triangleq {{\left[ \begin{matrix} \begin{matrix} {{w}_{k}}^{T} & 0 & -\left( {{u}_{k}}-{{u}_{0}} \right){{w}_{k}}^{T} \\ \end{matrix} \\ \begin{matrix} 0 & {{w}_{k}}^{T} & -\left( {{v}_{k}}-{{v}_{0}} \right){{w}_{k}}^{T} \\ \end{matrix} \\ \end{matrix} \right]}_{m}}{{\left[ \begin{matrix} {{{\bar{h}}}_{1}}^{T} \\ {{{\bar{h}}}_{2}}^{T} \\ {{{\bar{h}}}_{3}}^{T} \\ \end{matrix} \right]}_{m}}=0.\tag{36}\]

The estimated matrix ${{\hat{H}}_{m}}$ up to an unknown scale is equivalent to the homography ${{H}_{m}}$. Since all laser points are on a plane, only one unknown scale factor $\lambda $, is left according to Equation (18). The homography ${{H}_{m}}$ here is related to focal lengths only, rather than an intrinsic matrix, i.e.

\[{{H}_{m}}=F\left[ \begin{matrix} {}^{C}i_{m,1}^{{}} & \ldots & {}^{C}i_{m,N}^{{}} & {}^{C}{{T}_{m}} \\ \end{matrix} \right],\tag{37}\]

where

\[F\triangleq \left[ \begin{matrix} {{f}_{u}} & 0 & 0 \\ 0 & {{f}_{v}} & 0 \\ 0 & 0 & 1 \\ \end{matrix} \right].\tag{38}\]

2) Finding the laser direction relative to the robot frame and the end-effector frame:

This step is the same as step 2 of a multiple plane case. Notably, the matrix $K$ in Equation (21) should be replaced by $F$; however, this does not alter the process to find the laser direction w.r.t. the robot.

3) Finding the plane normal vectors:

The projection of robot translations ${}^{B}{{j}_{m,n}}$ to laser moving vectors including camera intrinsic parameters is

\[{{\hat{i}}_{m,n}}={{\hat{P}}_{m}}\cdot {}^{B}{{j}_{m,n}}\triangleq \frac{1}{\lambda }F\cdot {}_{B}^{C}R\cdot {}_{{}}^{B}{{P}_{m}}\cdot {}^{B}{{j}_{m,n}}.\tag{39}\]

With the previously obtained ${{\hat{i}}_{m,n}}$s and ${}^{B}{{j}_{m,n}}$s from robot command $n=\text{1}\tilde{\ }N$, ${{\hat{P}}_{m}}$ is calculated by applying the least square method.

According to Equation (20), the projecting matrix can be rewritten as

\[{{\hat{P}}_{m}}=\left[ \begin{matrix} {{{\hat{p}}}_{m,1}} & {{{\hat{p}}}_{m,2}} & {{{\hat{p}}}_{m,3}} \\ \end{matrix} \right]={{s}_{m}}MU_{m}^{\times },\tag{40}\]

where ${{s}_{m}}=\frac{-1}{\lambda \cdot {}_{{}}^{B}u_{m}^{T}{}_{\Pi }^{B}n}$, $M=\left[ \begin{matrix} m_{1}^{{}} & m_{2}^{{}} & m_{3}^{{}} \\ \end{matrix} \right]=F{}_{B}^{C}R{{\left\lfloor {}_{\Pi }^{B}n \right\rfloor }_{\times }}$, and $U_{m}^{\times }=\left[ \begin{matrix} u_{m,1}^{\times } & u_{m,2}^{\times } & u_{m,3}^{\times } \\ \end{matrix} \right]={{\left\lfloor {}_{{}}^{B}u_{m}^{{}} \right\rfloor }_{\times }}$. The values of matrixes ${{\hat{P}}_{m}}$s are obtained from the previous step and ${{\left\lfloor {}_{{}}^{B}u_{m}^{{}} \right\rfloor }_{\times }}$s from step 2. These values are used to calculate a matrix, $\tilde{M}$, which is proportional to $M$. Since ${{s}_{m}}$ is unknown, we can only have a set of similarity relations ${{\hat{P}}_{m}}\propto MU_{m}^{\times }$ for all $m$ where the symbol $\propto $ denotes equality up to an unknown scalar multiplication. Hence, a solution of $\tilde{M}$ can be obtained by the DLT method. Since the plane normal vector ${}_{\Pi }^{B}n$ is a kernel of $M$ as well as $\tilde{M}$, it can be obtained by solving $\tilde{M}{}_{\Pi }^{B}n=0$.

4) Finding the hand-to-eye rotation:

The laser point moving vectors ${}^{B}{{i}_{m,n}}$ in the robot frame now can be calculated by Equation (20) with obtained ${}_{\Pi }^{B}n$ and ${}_{{}}^{B}{{u}_{m}}$. Since all laser points are on the same plane, they can only provide two-dimensional information for camera calibration. According to equations (15) and (17) and the selection of two bases, ${}^{B}{{b}_{1}}$ and ${}^{B}{{b}_{2}}$, orthogonal to ${}_{\Pi }^{B}n$, the homogeneous laser spot moving vector is

\[{}^{B}{{\hat{i}}_{m,n}}\propto F\cdot {}_{B}^{C}R{}^{B}{{i}_{m,n}}=F\cdot {}_{B}^{C}R\left[ \begin{matrix} {}^{B}{{b}_{1}} & {}^{B}{{b}_{2}} \\ \end{matrix} \right]\left[ \begin{matrix} a_{m,n}^{1} \\ a_{m,n}^{2} \\ \end{matrix} \right].\tag{41}\]

where $a_{m,n}^{1}={}^{B}{{b}_{1}}^{T}\cdot {}^{B}{{i}_{m,n}}$ and $a_{m,n}^{2}={}^{B}{{b}_{2}}^{T}\cdot {}^{B}{{i}_{m,n}}$. Denote $F\cdot {}_{B}^{C}R\left[ \begin{matrix} {}^{B}{{b}_{1}} & {}^{B}{{b}_{2}} \\ \end{matrix} \right]$ as a matrix $\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{G}$ with a size 3×2. A solution $\tilde{G}$ of $\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{G}$ up to an unknown scale can be obtained by the DLT method with all $M\times N$ data. The transformation matrix $\tilde{G}$ can be expanded as

\[\tilde{G}=\left[ \begin{matrix} {{{\tilde{g}}}_{11}} & {{{\tilde{g}}}_{12}} \\ {{{\tilde{g}}}_{21}} & {{{\tilde{g}}}_{22}} \\ {{{\tilde{g}}}_{31}} & {{{\tilde{g}}}_{32}} \\ \end{matrix} \right]=F\left[ \begin{matrix} {}^{C}{{b}_{1}} & {}^{C}{{b}_{2}} \\ \end{matrix} \right]=s\left[ \begin{matrix} \begin{matrix} {{f}_{u}}{}^{C}b_{11}^{{}} \\ {{f}_{v}}{}^{C}b_{21}^{{}} \\ {}^{C}b_{31}^{{}} \\ \end{matrix} & \begin{matrix} {{f}_{u}}{}^{C}b_{12}^{{}} \\ {{f}_{v}}{}^{C}b_{22}^{{}} \\ {}^{C}b_{32}^{{}} \\ \end{matrix} \\ \end{matrix} \right].\tag{42}\]

Since ${}^{C}{{b}_{1}}$ and ${}^{C}{{b}_{2}}$ are two orthogonal bases, a set of simultaneous equations is obtained as

\[{{A}_{F}}{{x}_{F}}=\left[ \begin{matrix} {{{\tilde{g}}}_{11}}^{2} & {{{\tilde{g}}}_{21}}^{2} & {{{\tilde{g}}}_{31}}^{2} \\ {{{\tilde{g}}}_{12}}^{2} & {{{\tilde{g}}}_{22}}^{2} & {{{\tilde{g}}}_{32}}^{2} \\ {{{\tilde{g}}}_{11}}{{{\tilde{g}}}_{12}} & {{{\tilde{g}}}_{21}}{{{\tilde{g}}}_{22}} & {{{\tilde{g}}}_{31}}{{{\tilde{g}}}_{32}} \\ \end{matrix} \right]\left[ \begin{matrix} {1}/{{{\left( s{{f}_{u}} \right)}^{2}}}\; \\ {1}/{{{\left( s{{f}_{v}} \right)}^{2}}}\; \\ {1}/{{{\left( s \right)}^{2}}}\; \\ \end{matrix} \right]={{x}_{F}}=\left[ \begin{matrix} 1 \\ 1 \\ 0 \\ \end{matrix} \right],\tag{43}\]

To derive ${{f}_{u}}$ and ${{f}_{v}}$. Consequently, ${}^{C}{{b}_{1}}$ and ${}^{C}{{b}_{2}}$ are computed from Equation (40). The third basis can be found as ${}^{C}{{b}_{3}}={}^{C}{{b}_{1}}\times {}^{C}{{b}_{2}}$, and the rotation matrix from the robot frame to the camera frame is

\[{}_{B}^{C}R=\left[ \begin{matrix} {}^{C}{{b}_{1}} & {}^{C}{{b}_{2}} & {}^{C}{{b}_{3}} \\ \end{matrix} \right]{{\left[ \begin{matrix} {}^{B}{{b}_{1}} & {}^{B}{{b}_{2}} & {}^{B}{{b}_{3}} \\ \end{matrix} \right]}^{-1}}.\tag{44}\]

This solution is useful for a camera with an aspect ratio not equal to 1.

However, the value of the last entry of ${}^{B}{{\hat{i}}_{m,n}}$ is quite small when the plane is nearly parallel to the image plane. This issue can be avoided by selecting the first two entries of ${}^{B}{{\hat{i}}_{m,n}}$ to solve this problem, that is

\[\left[ \begin{matrix} {}^{B}\hat{i}_{m,n}^{1} \\ {}^{B}\hat{i}_{m,n}^{2} \\ \end{matrix} \right]\propto \tilde{G}\cdot \left[ \begin{matrix} a_{m,n}^{1} \\ a_{m,n}^{2} \\ \end{matrix} \right],\tag{45}\]

where

\[\tilde{G}=\left[ \begin{matrix} {{{\tilde{g}}}_{11}} & {{{\tilde{g}}}_{12}} \\ {{{\tilde{g}}}_{21}} & {{{\tilde{g}}}_{22}} \\ \end{matrix} \right]=s\left[ \begin{matrix} {{f}_{u}}{}^{C}b_{11}^{{}} & {{f}_{u}}{}^{C}b_{12}^{{}} \\ {{f}_{v}}{}^{C}b_{21}^{{}} & {{f}_{v}}{}^{C}b_{22}^{{}} \\ \end{matrix} \right].\tag{46}\]

A solution $\tilde{G}$ with an unknown scale can be obtained by the DLT method using all $M \times N$ data. Assume that the aspect ratio is 1 and let $f={{f}_{u}}={{f}_{v}}$,

\[\begin{align} & {{\left( sf \right)}^{2}}=\frac{1}{2}\left( \left( {{{\tilde{g}}}_{11}}^{2}+{{{\tilde{g}}}_{12}}^{2}+{{{\tilde{g}}}_{21}}^{2}+{{{\tilde{g}}}_{22}}^{2} \right) \right. \\ & +\left. \sqrt{\left( {{\left( {{{\tilde{g}}}_{11}}-{{{\tilde{g}}}_{22}} \right)}^{2}}+{{\left( {{{\tilde{g}}}_{12}}+{{{\tilde{g}}}_{21}} \right)}^{2}} \right)\left( {{\left( {{{\tilde{g}}}_{11}}+{{{\tilde{g}}}_{22}} \right)}^{2}}+{{\left( {{{\tilde{g}}}_{12}}-{{{\tilde{g}}}_{21}} \right)}^{2}} \right)} \right) \\ \end{align}.\tag{47}\]

The values of ${}^{C}b_{11}^{{}}$, ${}^{C}b_{21}^{{}}$, ${}^{C}b_{12}^{{}}$, and ${}^{C}b_{22}^{{}}$ are obtained to form ${}_{{}}^{C}B=\left[ \begin{matrix} {}^{C}{{b}_{1}} & {}^{C}{{b}_{2}} & {}^{C}{{b}_{3}} \\ \end{matrix} \right]$. Let ${}_{{}}^{C}B$ denote a rotation matrix, i.e. orthonormal and $\det ({}_{{}}^{C}B)=1$. The rotation matrix ${}_{B}^{C}R$ can be computed from Equation(44). Notably, two solutions are available,

\[{{R}_{1}}={}_{B}^{C}R \text{and} {{R}_{2}}=\left[ \begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -1 \\ \end{matrix} \right]{}_{B}^{C}R\left[ \begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -1 \\ \end{matrix} \right].\tag{48}\]

The ambiguity can be resolved by calculating the image position errors between the observed laser spot position and the projected laser spot position at the end.

5) Finding the focal length(s), scaling factor and translations of the laser and the camera relative to the robot:

Based on the constraints ${}^{C}{{u}_{m}}\times {}_{{}}^{C}{{t}_{m,k}}=0$, as in Equation (29),

\[{{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }}\cdot \left( {}_{\Pi }^{C}{{p}_{m,k}}-\left( {}_{B}^{C}R\left( {}_{E}^{B}{{R}_{m}}{}_{\Lambda }^{B}T+{}_{E}^{B}{{T}_{m,k}} \right)+{}_{B}^{C}T \right) \right),\tag{49}\]

where

\[{}_{\Pi }^{C}{{p}_{m,k}}=\lambda {{F}^{-1}}{{\hat{H}}_{m}}\left[ \begin{matrix} w_{m,k}^{{}} \\ 1 \\ \end{matrix} \right].\tag{50}\]

Let

\[{}_{\Pi }^{C}{{\hat{p}}_{m,k}}=\left[ \begin{matrix} {}_{\Pi }^{C}\hat{p}_{m,k}^{1} \\ {}_{\Pi }^{C}\hat{p}_{m,k}^{2} \\ {}_{\Pi }^{C}\hat{p}_{m,k}^{3} \\ \end{matrix} \right]\triangleq {{\hat{H}}_{m}}\left[ \begin{matrix} w_{m,k}^{{}} \\ 1 \\ \end{matrix} \right],\tag{51}\]

the constraint equations of Equation(49) becomes

\[\begin{align} & \left[ \begin{matrix} -{{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }}{}_{B}^{C}R{}_{E}^{B}R_{m}^{{}} & -{{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }} & {{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }}{{F}^{-1}}{}_{\Pi }^{C}{{{\hat{p}}}_{m,k}} \\ \end{matrix} \right] \\ & \cdot {{\left[ \begin{matrix} {}_{\Lambda }^{E}{{T}^{T}} & {}_{B}^{C}{{T}^{T}} & \lambda \\ \end{matrix} \right]}^{T}}={{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }}{}_{B}^{C}R{}_{E}^{B}T_{m,k}^{{}} \\ \end{align},\tag{52}\]

under the condition that the aspect ratio is not close to 1 and the plane is not nearly parallel to the image plane.

For a situation in which the aspect ratio nears 1, the simultaneous equation set is

\[\left[ \begin{matrix} -{{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }}{}_{E}^{C}R_{m}^{{}} & {{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }} & \left[ \begin{matrix} -{}^{C}u_{m}^{3}{}_{\Pi }^{C}\hat{p}_{m,k}^{2} & {}^{C}u_{m}^{2}{}_{\Pi }^{C}\hat{p}_{m,k}^{3} \\ {}^{C}u_{m}^{3}{}_{\Pi }^{C}\hat{p}_{m,k}^{1} & -{}^{C}u_{m}^{1}{}_{\Pi }^{C}\hat{p}_{m,k}^{3} \\ -{}^{C}u_{m}^{2}{}_{\Pi }^{C}\hat{p}_{m,k}^{1}+{}^{C}u_{m}^{1}{}_{\Pi }^{C}\hat{p}_{m,k}^{2} & 0 \\ \end{matrix} \right] \\ \end{matrix} \right]\]
\[\cdot {{\left[ \begin{matrix} {}_{\Lambda }^{E}{{T}^{T}} & {}_{B}^{C}{{T}^{T}} & \lambda \cdot {{f}^{-1}} & \lambda \\ \end{matrix} \right]}^{T}}={{\left\lfloor {}_{{}}^{C}{{u}_{m}} \right\rfloor }_{\times }}{}_{B}^{C}R{}_{E}^{C}T_{m,k}^{{}}.\tag{53}\]

Since the laser position ${}_{\Lambda }^{E}T$ only has two degrees of freedom, both equations (52) and (53) should be reduced by one dimension. With the $M \times K$ data sets, the solutions of the focal length, scale factor and laser position in the end-effector frame and the camera position in the robot frame are obtained.

6) Finding the plane:

The plane of the normal vector w.r.t. robot base coordinate is already determined in step 3. Averaging the laser spot positions allows us to determine the plane by

\[{}_{\Pi }^{B}a=\left( \frac{1}{M}\sum\limits_{m=1}^{M}{T{{_{m}^{{}}}^{T}}\cdot {}_{\Pi }^{B}n} \right){}_{\Pi }^{B}n.\tag{54}\]

3.4 Nonlinear Optimization

The closed-form solution above provides a good estimate for the system. However, the error could propagate throughout procedural steps. A nonlinear optimization problem is introduced to refine the results. The laser projecting point is constrained on the plane under various arm poses; this yields the following optimization problem:

\[\underset{S}{\mathop{\min }}\,\sum\limits_{p=1}^{P}{\sum\limits_{m=1}^{M}{\sum\limits_{k=1}^{K}{{{\left\| {{m}_{p,m,k}}-\hat{m}(S,{}_{E}^{B}R_{p,m}^{{}},{}_{E}^{B}T_{p,m,k}^{{}}) \right\|}^{2}}}}},\tag{55}\]

where ${}_{E}^{B}R_{p,m}^{{}}$ and ${}_{E}^{B}T_{p,m,k}^{{}}$ are obtained from the ($p$,$m$,$k$)-th robot command when the plane is placed at the $p$-th pose; ${{m}_{p,m,k}}$ is the observed laser spot position from the ($p$,$m$,$k$)-th image; $\hat{m}(\cdot )$ denotes the virtual projected laser spot position to the image according to (1)–(9); and $S$ is the set of parameters which need to be refined, including ${{f}_{u}}$, ${{f}_{v}}$, ${{\alpha }_{c}}$, ${{u}_{0}}$, ${{v}_{0}}$, ${{\kappa }_{1}}$, ${{\kappa }_{2}}$, ${{\rho }_{1}}$, ${{\rho }_{2}}$, ${}_{B}^{C}\theta $, ${}_{B}^{C}\phi $, ${}_{B}^{C}\varphi $, ${}_{B}^{C}T$, ${}_{\Lambda }^{E}\theta $, ${}_{\Lambda }^{E}\phi$, ${}_{\Lambda }^{E}y$, ${}_{\Lambda }^{E}z$, and ${}_{\Pi }^{B}{{a}_{p}},\text{ }p=1,...,P$.

Specifically, Equation (55) can be rewritten as,

\[\underset{{{S}_{1}},{{S}_{2}}\subset S}{\mathop{\min }}\,\sum\limits_{p=1}^{P}{\sum\limits_{m=1}^{M}{\sum\limits_{k=1}^{K}{{{\left\| {{m}_{p,m,k}}-\hat{m}({{S}_{1}},{}^{B}{{{\hat{P}}}_{L}}({{S}_{2}},{}_{\Pi }^{B}{{a}_{p}},{}_{E}^{B}R_{p,m}^{{}},{}_{E}^{B}T_{p,m,k}^{{}})) \right\|}^{2}}}}},\tag{56}\]

where ${{S}_{1}}$ and ${{S}_{2}}$ are the subsets of $S$. Subset ${{S}_{1}}$ comprises the camera’s intrinsic and extrinsic parameters from the robot base to the camera. Subset ${{S}_{2}}$ contains the laser pointer parameters. Notably, ${}^{B}{{\hat{P}}_{L}}(\cdot )$ represents the position in a robot base coordinate system derived by Equation (9). For the single plane case,$p$ = 1, the value of principal point (${{u}_{0}}$, ${{v}_{0}}$) is constant, rather than the parameters to be refined.

As mentioned earlier in subsection 3.1, a 3D position of a laser light reflection point can be derived from the laser-plane intersection, ray-plane intersection, and laser-camera triangulation. An alternative way to form the optimization formula compares the differences in the positions derived by two of these three methods. However, un-distorting the image becomes non-deterministic, since the camera model containing the distortion is nonlinear. Hence, optimization formulas such as equations (55) and (56) derived from Equation (9) are more reliable than those from equations (12) or (14).

Equation (55) can be solved using the Levenberg-Marquardt method. An initial guess of S is required and can be obtained from the closed-form solution described previously. Second, if all system parameters can be acquired from specifications or design settings, these values must be close to the real values and can be used as an initial guess for optimization.

3.5 Summary of calibration procedures

The procedures to calibrate the target parameters are introduced above, and the following is the recommended procedure:

  1. Attach a laser pointer either to the end-effector of the manipulator or to the tool on the end-effector.
  2. If the working plane is assigned, let it be the first plane to project the laser; otherwise, place a plane in front of the camera.
  3. Point the laser towards the plane and then command the robot to translate but not rotate its end-effector along several directions. Record the poses and take an image in each pose.
  4. Command the robot to rotate its end-effector and then repeat Step 3.
  5. Change the plane and repeat Steps 3–5. This step is ignored for the single plane calibration and can be repeated several times for multiple plane calibration.
  6. Extract laser points from the images.
  7. Estimate the camera intrinsic parameters and the eye-to-hand parameters using the closed-form solution respectively described in subsections 3.2 and 3.3.
  8. Optimize the solutions as described in subsection 3.4.

Strum and Maybank [28] addressed the singularity of camera calibration using a plane pattern resulting from plane pose. Using the proposed methods to project the laser spots on the planes, the singularity of the camera calibration is identical to that of pure camera calibration using 2D pattern. In sum, the plane should be selected to avoid parallelizing to the image plane or to the x-axis or y-axis of the camera coordinates.

4. Experimental results

The proposed algorithm is evaluated using simulations and real data.

4.1 Simulations

The proposed method is validated via computer simulations under different noise levels and with different numbers of samples. The parameter values in all cases are as follows: ${{f}_{u}}$=550, ${{f}_{v}}$=550, ${{\alpha }_{c}}$=0, ${{u}_{0}}$=320, ${{v}_{0}}$=240, ${}_{C}^{B}\theta $=-90°, ${}_{C}^{B}\phi $=-10°, ${}_{C}^{B}\varphi $=-170°, ${}_{C}^{B}T$=[510,151,430]T (mm), ${}_{\Lambda }^{E}\theta $=21.80°, ${}_{\Lambda }^{E}\phi $=53.39°, ${}_{\Lambda }^{E}T$=[0,-10,30]T (mm), ${}_{\Pi }^{B}{{a}_{1}}$=[152.41, 138.34, -163.92] T (mm), ${}_{\Pi }^{B}{{a}_{2}}$=[ 48.68, 31.47, -153.81] T (mm), and ${}_{\Pi }^{B}{{a}_{3}}$ is randomly generated.

In the following tests, the relative errors of one focal length are calculated as $\left| {{f}_{u,estimated}}-{{f}_{u,true}} \right|/{{f}_{u,true}}$ (percentage). The hand-eye rotation and translation errors are the Frobenius norms of the difference between the true and estimated ${}_{C}^{B}R$ and ${}_{C}^{B}T$; in addition, the relative errors are calculated as ${{{\left\| {{T}_{estimated}}-{{T}_{true}} \right\|}_{F}}}/{{{\left\| {{T}_{true}} \right\|}_{F}}}\;$ and ${{{\left\| {{R}_{estimated}}-{{R}_{true}} \right\|}_{F}}}/{{{\left\| {{R}_{true}} \right\|}_{F}}}\;$ (percentages). The relative errors of the first plane pose are calculated as ${{\left\| {}_{\Pi }^{B}{{n}_{1,estimated}}-{}_{\Pi }^{B}{{n}_{1,true}} \right\|}_{F}}$. For each test, 100 independent trials are performed.

Methods proposed by Ma [20], Malm and Heyden [32, 33], and Horaud and Dornaika [9] are selected for comparison with the proposed algorithm. The three methods concern intrinsic camera parameters and can represent conventional calibration methods using three reference types, as shown in figure 5. The three methods (originally eye-in-hand) are modified for eye-to-hand configuration. Since Horaud and Dornaika’s method only introduces the calibration of relative translation and rotation between the robot and the reference object, the relative translational and rotational errors are the Frobenius norms of the difference between the true and estimated ${}_{O}^{E}T$ and ${}_{O}^{E}R$ rather than ${}_{C}^{B}T$ and ${}_{C}^{B}R$, where ${}_{O}^{E}T$ and ${}_{O}^{E}R$ respectively denote the reference object position and orientation relative to the end-effector. Three noise types are added together for the following test. The noises are normally distributed with a zero mean and the standard deviation representing the noise level varies according to Table 1. To assess performance under different noise levels, 90 points are independently generated for each method. The number of hand orientations, hand pure translating directions and/or plane poses are listed as follows,

  • M1:
  • Proposed method for multiple plane case: 3 plane poses, 3 hand orientations per plane, 1 starting position and 3 translating directions per orientation, 3 positions along each direction.
  • M2:
  • Ma’s method: 3 non-structure points on the robot hand, 3 hand orientations, 1 starting position and 3 sets of 3 orthogonal pure translations per orientation.
  • M3:
  • Malm and Heyden’s method: 10 points on the plane reference in the robot hand, 3 hand orientations, 1 starting position, and 2 translations per orientation.
  • M4:
  • Horaud and Dornaika’s method: 10 point on the 3-D reference object, and 9 hand poses.

Since the respective performance of these methods is related to poses and movements of the references, i.e. the robot hand motions, some points should be mentioned when generating data. For the proposed method, the end-effector is close to the workspace and thus decreases the influence of hand orientation error. When generating data for Ma’s method, especially when pixel positions are noisy, the pure translations should not be parallel to any camera axis; otherwise, the estimation of the FOE point seriously deviates causing errors in the subsequent camera calibration results. The instructions provided by Malm and Heyden to implement their method place the plane pattern nonparallel to image plane and translate the plane in a direction nonparallel to the plane. Moreover, based on our experience, manipulating the plane pattern closer to the camera produces better results using the Malm and Heyden method and the Horaud and Dornaika method. Figure 6 shows that these methods are comparable to each other in terms of mixing the three kinds of noise. Ma’s method has a satisfactory rotation estimation capability for noise. The proposed method is outperforms the other methods in terms of intrinsic camera parameters and eye-to-hand translation.

The accuracy is also influenced by the number of data points. The proposed method’s performance with a single plane, 2 planes and 3 planes are compared using different numbers of samples. Zero mean Gaussian noise with a 1-pixel standard deviation is added to each point image position. The number of hand orientations is increased to produce different numbers of samples with measurements of camera projections. For each number of samples, 100 independent trials are performed and averages of relative errors are determined. Figure 7 summarizes results of one focal length, fu , to represent the system error. The number of errors decreases as the number of samples increases. Because of limited data information for intrinsic camera calibration, the error resulting from a single plane is larger than that from multiple planes. Notably, for the single plane case, the relative errors fall to reasonable levels, i.e., it could be used to reduce calibration time if the accuracy requirement is met.

4.2 Real data

Figure 8 shows the experimental setup. The manipulator is a TX60 with a CS8 controller manufactured by Stäubli Inc. The camera is a Panasonic WV-CP 480 CCD camera with an adjustable lens. The camera signal in NTSC format is transmitted to the frame grabber (RTV-24) through a coaxial cable. The frame grabber card is embedded in the computer. A Visual C program in the computer can save images, send robot commands to the CS8 controller and receive the robot status from the controller through an Ethernet connection. A 50×50cm planar plate is placed roughly 90cm away from the manipulator. In this configuration, the camera cannot see any part of the manipulator. The laser pointer is a laser diode with an adjustable lens attached on the robot tool (Figure 9).

Following the steps in subsection 3.5, under each hand orientation, this work manipulated the end-effector to a specific position and then moved it in three directions, generating three positions along each direction, resulting in ten hand positions per orientation. Steps 3 and 4 were performed three times to generate three different hand orientations. The plane pose was then changed twice. A total of 3 plane poses and 90 hand poses were generated per plane. All movements were designed to project the laser beam onto the planar object, and corresponding images were saved. Data were applied to the proposed closed-form solutions to obtain initial values. Next, the proposed nonlinear optimization method refined these values. Table 2 summarizes the results. Four tests were performed in this experiment. The first column in the first test, labeled as Initial, is the estimation of the closed-form solution described in subsection 3.3. The second column, labeled Refined, is the estimation of nonlinear optimization. The third column, labeled $\sigma $, is the standard deviation, indicating the uncertainty of the refined result. Test 2 results of nonlinear optimization using more data points are in the right hand column of Table 2. The results of Tests 3 and 4 under 2 plane poses originate from the closed form solution described in subsection 3.2 followed by nonlinear optimization. In each test, the last row, labeled RMS, shows the root mean square of the distance of the image from the estimated points to the predicted points. The results are consistent with simulation results in that the uncertainty of estimations declines as sample size increases.

5. Conclusion

This work presents a novel calibration method to simultaneously and accurately estimate intrinsic camera parameters, the configuration between a camera and a robot, and the pose of the working plane. The proposed method is developed using a laser pointer. Since the laser is rigidly installed and the plane is fixed, camera measurements of light spots projected by the laser beam onto the plane must adhere to the geometrical constraints. Based on these constraints, the proposed closed-form solution obtains initial values, and nonlinear optimization can be applied to refine these values. Simulations are also performed to validate the calibration method and analyze its performance in different conditions. Experimental results demonstrate that the proposed method is effective when a camera cannot see the hand. Experimental results using real data are consistent with the simulation results, thus validating the proposed method.

References

  1. A. Jordt, N. T. Siebel, and G. Sommer, "Automatic high-precision self-calibration of camera-robot systems," in IEEE International Conference on Robotics and Automation (ICRA), Kobe, Japan, 2009, pp. 1244-1249.
    doi: 10.1109/ROBOT.2009.5152570
  2. Y. Shiu and S. Ahmad, "Finding the mounting position of a sensor by solving a homogeneous transform equation of the form AX=XB," in IEEE International Conference on Robotics and Automation (ICRA), Raleigh, NC, USA, 1987, vol. 4, pp. 1666-1671.
    doi: 10.1109/ROBOT.1987.1087758
  3. S. Yiu Cheung and S. Ahmad, "Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB," IEEE Transactions on Robotics and Automation, vol. 5, no. 1, pp. 16-29, 1989.
    doi: 10.1109/70.88014
  4. R. Y. Tsai and R. K. Lenz, "A new technique for fully autonomous and efficient 3D robotics hand/eye calibration," IEEE Transactions on Robotics and Automation, vol. 5, no. 3, pp. 345-358, 1989.
    doi: 10.1109/70.34770
  5. J. C. K. Chou and M. Kamel, "Quaternions approach to solve the kinematic equation of rotation, AaAx=AxAb, of a sensor-mounted robotic manipulator," in IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA, 1988, pp. 656-662.
    doi: 10.1109/ROBOT.1988.12134
  6. H. Zhuang, Z. S. Roth, Y. C. Shiu, and S. Ahmad, "Comments on "Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB"," IEEE Transactions on Robotics and Automation, vol. 7, no. 6, pp. 877-878, 1991.
    doi: 10.1109/70.105398
  7. G. D. van Albada, J. M. Lagerberg, A. Visser, and L. O. Hertzberger, "A low-cost pose-measuring system for robot calibration," Robotics and Autonomous Systems, vol. 15, no. 3, pp. 207-227, 1995.
    doi: 10.1016/0921-8890(95)00038-H
  8. Z. Hanqi and S. Yiu Cheung, "A noise-tolerant algorithm for robotic hand-eye calibration with or without sensor orientation measurement," IEEE Transactions on Systems, Man and Cybernetics, vol. 23, no. 4, pp. 1168-1175, 1993.
    doi: 10.1109/21.247898
  9. R. Horaud and F. Dornaika, "Hand-eye calibration," international journal of robotics research, vol. 14, no. 3, pp. 195-210, 1995.
    doi: 10.1177/027836499501400301
  10. K. Daniilidis and E. Bayro-Corrochano, "The dual quaternion approach to hand-eye calibration," in the 13th International Conference on Pattern Recognition (ICPR), Vienna, 1996, vol. 1, pp. 318-322.
    doi: 10.1109/ICPR.1996.546041
  11. K. Daniilidis, "Hand-eye calibration using dual quaternions," The International Journal of Robotics Research, vol. 18, no. 3, pp. 286-298, 1999.
    doi: 10.1177/02783649922066213
  12. Z. Hanqi, Z. S. Roth, and R. Sudhakar, "Simultaneous robot/world and tool/flange calibration by solving homogeneous transformation equations of the form AX=YB," IEEE Transactions on Robotics and Automation, vol. 10, no. 4, pp. 549-554, 1994.
    doi: 10.1109/70.313105
  13. F. Dornaika and R. Horaud, "Simultaneous robot-world and hand-eye calibration," IEEE Transactions on Robotics and Automation, vol. 14, no. 4, pp. 617-622, 1998.
    doi: 10.1109/70.704233
  14. R. L. Hirsh, G. N. DeSouza, and A. C. Kak, "An iterative approach to the hand-eye and base-world calibration problem," in IEEE International Conference on Robotics and Automation (ICRA), Seoul, Korea, 2001, vol. 3, pp. 2171-2176 vol.2173.
    doi: 10.1109/ROBOT.2001.932945
  15. A. Li, L. Wang, and D. Wu, "Simultaneous robot-world and hand-eye calibration using dual-quaternions and Kronecker product," International Journal of the Physical Sciences, vol. 5, no. 10, pp. 1530-1536, 2010.
  16. K. H. Strobl and G. Hirzinger, "More accurate camera and hand-eye calibrations with unknown grid pattern dimensions," in IEEE International Conference on Robotics and Automation (ICRA), Pasadena, CA, USA, 2008, pp. 1398-1405.
    doi: 10.1109/ROBOT.2008.4543398
  17. C. C. Wang, "Extrinsic calibration of a vision sensor mounted on a robot," IEEE Transactions on Robotics and Automation, vol. 8, no. 2, pp. 161-175, 1992.
    doi: 10.1109/70.134271
  18. X. Haixia, W. Yaonan, C. Wei, and L. Juan, "A self-calibration approach to hand-eye relation using a single point," in International Conference on Information and Automation (ICIA), Changsha, China, 2008, pp. 413-418.
    doi: 10.1109/ICINFA.2008.4608035
  19. C. S. Gatla, R. Lumia, J. Wood, and G. Starr, "Calibrating pan-tilt cameras in robot hand-eye systems using a single point," in IEEE International Conference on Robotics and Automation (ICRA), Roma, Italy, 2007, pp. 3186-3191.
    doi: 10.1109/ROBOT.2007.363964
  20. M. Song De, "A self-calibration technique for active vision systems," IEEE Transactions on Robotics and Automation, vol. 12, no. 1, pp. 114-120, 1996.
    doi: 10.1109/70.481755
  21. W. Guo-Qing, K. Arbter, and G. Hirzinger, "Active self-calibration of robotic eyes and hand-eye relationships with model identification," IEEE Transactions on Robotics and Automation, vol. 14, no. 1, pp. 158-166, 1998.
    doi: 10.1109/70.660864
  22. N. Andreff, R. Horaud, and B. Espiau, "Robot hand-eye calibration using structure-from-motion," The International Journal of Robotics Research, vol. 20, no. 3, pp. 228-248, 2001.
    doi: 10.1177/02783640122067372
  23. H. Malm and A. Heyden, "Self-calibration from image derivatives for active vision systems," in the 7th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore, 2002, vol. 2, pp. 1116-1121.
    doi: 10.1109/ICARCV.2002.1238580
  24. J. S. Hu, M. C. Chien, Y. J. Chang, Y. C. Chang, S. H. Su, J. J. Yang, and C. Y. Kai, "A robotic ball catcher with embedded visual servo processor," in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, 2010, pp. 2513-2514.
    doi: 0.1109/IROS.2010.5648912
  25. L. Sun, J. Liu, W. Sun, S. Wu, and X. Huang, "Geometry-based robot calibration method," in IEEE International Conference on Robotics and Automation (ICRA), New Orleans, LA, USA, 2004, vol. 2, pp. 1907-1912.
    doi: 10.1109/ROBOT.2004.1308102
  26. J. Heikkila and O. Silven, "A four-step camera calibration procedure with implicit image correction," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1997, pp. 1106-1112.
    doi: 10.1109/CVPR.1997.609468
  27. Z. Zhengyou, "A flexible new technique for camera calibration," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, 2000.
    doi: 10.1109/34.888718
  28. P. F. Sturm and S. J. Maybank, "On plane-based camera calibration: A general algorithm, singularities, applications," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Fort Collins, CO , USA, 1999, vol. 1, pp. 432-437.
    doi: 10.1109/CVPR.1999.786974
  29. Z. Zhengyou, "Camera calibration with one-dimensional objects," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 7, pp. 892-899, 2004.
    doi: 10.1109/TPAMI.2004.21
  30. H. Jwu-Sheng and C. Yung-Jung, "Calibration of an eye-to-hand system using a laser pointer on hand and planar constraints," in IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, 2011, pp. 982-987.
    doi: 10.1109/ICRA.2011.5979846
  31. J.-S. Hu and Y.-J. Chang, "Eye-hand-workspace calibration using laser pointer projection on plane surface," Industrial Robot: An International Journal, vol. 39, no. 2, pp. 197-207, 2012.
    doi: 10.1108/01439911211201663
  32. H. Malm and A. Heyden, "Simplified intrinsic camera calibration and hand-eye calibration for robot vision," in IEEE/RSJ International Conference on Robots and Systems (IROS), Las Vegas, Nevada, 2003, vol. 1, pp. 1037-1043.
    doi: 10.1109/IROS.2003.1250764
  33. H. Malm and A. Heyden, "Extensions of plane-based calibration to the case of translational motion in a robot vision setting," IEEE Transactions on Robotics, vol. 22, no. 2, pp. 322-333, 2006.
    doi: 10.1109/TRO.2005.862477

Refbacks

  • There are currently no refbacks.


Copyright © 2011-2017  AUSMT   ISSN: 2223-9766