Automatic and Flexible Robotic Drawing on Complex Surfaces With an Industrial Robot

In industrial applications, planning and executing robot motions are crucial steps for manufacturing processes. Following the trend for customization, more flexible production systems are needed to quickly adapt the planned robot motion to new user inputs. In this work, a user-defined 2-D input pattern has to be drawn by a robot on a given 3-D object in an automated workflow. For this, two projection methods to map the 2-D input pattern to the 3-D object are presented, and robot trajectories are automatically generated based on the result of the projection methods. Furthermore, two control concepts, i.e., a pure motion control and a hybrid force/motion control, are investigated and validated by experimental results. In addition, a precise force estimation is performed to guarantee a constant normal contact force during the drawing process. The proposed automated workflow is applicable to various industrial processes, e.g., spray painting, cutting, and engraving, and provides an easy way to plan and execute robot motions based on user inputs.


Automatic and Flexible Robotic Drawing on Complex Surfaces With an Industrial Robot
Thomas Weingartshofer , Christian Hartl-Nesic , Member, IEEE, and Andreas Kugi , Senior Member, IEEE Abstract-In industrial applications, planning and executing robot motions are crucial steps for manufacturing processes.Following the trend for customization, more flexible production systems are needed to quickly adapt the planned robot motion to new user inputs.In this work, a user-defined 2-D input pattern has to be drawn by a robot on a given 3-D object in an automated workflow.For this, two projection methods to map the 2-D input pattern to the 3-D object are presented, and robot trajectories are automatically generated based on the result of the projection methods.Furthermore, two control concepts, i.e., a pure motion control and a hybrid force/motion control, are investigated and validated by experimental results.In addition, a precise force estimation is performed to guarantee a constant normal contact force during the drawing process.The proposed automated workflow is applicable to various industrial processes, e.g., spray painting, cutting, and engraving, and provides an easy way to plan and execute robot motions based on user inputs.Index Terms-Force control concept, manufacturing process, mapping, path planning, robotic application, user interaction.

I. INTRODUCTION
I N automated production lines, an increasing number of industrial robots are put into operation every year [1].The main driver for this trend is the growing product diversity in the industry, which goes up to full individualization [2].In some production sectors, such as clothing, shoe, and apparel industry, end consumers can customize and personalize products during ordering.Custom labels, logos, and symbols' size, appearance, and location can be specified.For automation, this raises the demand for flexible production systems and workflows.
In order to keep up with these trends, manual online programming of robots becomes infeasible and Computer-Aided Design (CAD)-based offline programming using fully automated Computer-Aided Manufacturing (CAM)-based production workflows is required [3].In most manufacturing processes, the workpiece geometry is already known, e.g., [4], or state-of-the-art 3-D scanners and algorithms can be Thomas Weingartshofer and Christian Hartl-Nesic are with the Automation and Control Institute (ACIN), TU Wien, 1040 Vienna, Austria (e-mail: weingartshofer@acin.tuwien.ac.at; hartl@acin.tuwien.ac.at).
Digital Object Identifier 10.1109/TCST.2023.3345209employed to obtain the shape of the workpiece [5].In this way, a 2-D user-generated pattern could be projected on a 3-D workpiece to automatically generate robot programs without human intervention.
In the literature, automatic generation of robot programs from a user input has been considered in several works, as discussed in the following.In many publications, a drawing process serves as a demonstration example to show the capabilities of the proposed algorithms.Note that in most cases, the considered task can be easily adapted to a different manufacturing process, e.g., engraving, laser cutting, or painting.
In [6] and [7], an edge detection algorithm is used to extract features from portraits of humans, which are drawn on a flat canvas by a humanoid robot.In order to perform this task, a task-space path is generated first, for which a joint-space path is computed using inverse kinematics, and then, it is executed by the robot.Furthermore, Jean-Pierre and Said [8] considered an industrial robot drawing on a flat whiteboard, where robot programs are automatically generated for drawing the edges and important features of ordinary photographs and images.Similarly, in [9], [10], [11], and [12], robotic drawing is presented with an emphasis on artistic and stylistic algorithms for path generation.Drawing on 2.5-D (terrainlike) objects is demonstrated in [13], where a KUKA LBR iiwa 7 R800 is utilized.The drawing is interpolated with Bézier curves and an impedance control is used to draw on unknown nonplanar objects.In [14], a force/torque sensor is mounted on the end-effector of the drawing robot.With this information, the robot is able to draw on objects, whose relative position to the robot is unknown.Even on 3-D surfaces with small curvature, drawing is successful because the pen orientation is controlled to remain normal to the surface.Because the force information is only available during the drawing process and the geometry of the object is unknown, the projection can cause distortions and is not predictable.Most of the works discussed so far focus on the artistic aspects of robotic drawing on planar surfaces, but the manufacturing aspects on 3-D objects are not considered in detail.
Automatic path planning to directly work on 3-D workpieces is examined for processes such as spraying [15], polishing [16], or draping [17].In those works, algorithms generate 3-D paths automatically based on the CAD data and no user input is required.
Another way of planning and executing robot motions from a user input is interactive teach-in methods.A state-of-theart concept based on an instrumented tool is developed by Wandelbots [18].With the so-called TracePen, robot motions are demonstrated by the user and a robot program is generated automatically.In [19], an instrumented tool is used to record the position, orientation, and force of a rope winding task.This demonstrated trajectory is then performed with an industrial robot.Inspired by the gaming industry, a representation of a robotic environment in a virtual reality could also be used to teach-in an industrial task [20].However, those teach-in methods need a trained person to generate the robot program and this is especially cumbersome for frequently changing user inputs or small lot sizes.
A different way to customize the visual design of products is to use inkjet printer heads mounted on the end-effector of industrial robots to directly perform 2-D printing on 3-D surfaces.Thereby, the printer head is used to print custom designs on shoes [21] or cars [22] and provides an easy way for customization.In this process, the robot motion is generated for each product only once because the robot trajectory stays the same if different 2-D inputs have to be printed on the same surface area.The different designs result from different print jobs.Note that the quality of the print on curved surfaces increases for small 2-D inputs because the print head has to be in constant distance from the product.
The goal of this work is to demonstrate the full customization of products in an industrial manufacturing process.To this end, a robotic drawing task for known arbitrary 3-D objects is considered, see Fig. 1, which is the main contribution of this work.This process has high demands on the robotic system to achieve the required flexibility and replication accuracy on the 3-D surface.A user specifies the drawing and its exact location, size, and orientation on the object.Subsequently, the robot trajectory is planned using a mapping and a drawing procedure is executed.To improve the drawing quality, the contact force of the pen is adjusted using a hybrid force/torque controller, which is further extended to control the position and orientation of the pen simultaneously.
This automatic pipeline is applicable to different manufacturing processes.Robot-assisted additive manufacturing processes, such as material extrusion, fused deposition modeling (FDM), material jetting, and directed energy deposition as shown in [23], can also be performed using the proposed automatic trajectory planning and robot execution pipeline.Another example is automated milling or laser engraving, where a user input pattern has to be engraved into large 3-D objects.In subtractive processes such as laser or ultrasonic cutting, this pipeline can also be employed as well as in spray painting processes.
A preliminary version of this work is published in [24].The main contributions beyond [24] are as follows.First, the automatic planning pipeline utilizing a least-squares conformal mapping (LSCM) is evaluated and compared to a standard parallel projection method.Second, both projection methods are investigated in new experiments, where the robot draws on challenging areas with more complex geometry and significantly higher curvature, in contrast to the plane-like area of [24].Therefore, an adapted version of the hybrid force/motion controller, including adaptions to the null-space controller, was derived to be able to execute these challenging experiments.Third, the previously used optical measurement system to calibrate the experimental setup is replaced by an inexpensive passive mechanical method.In order to keep this work self-contained, parts of the preliminary work [24] are summarized and some aspects are elaborated in more detail.
This article is organized as follows.In Section II, two projection methods to transfer a 2-D path onto a 3-D object are introduced.Subsequently, robot trajectories are generated from the projected 3-D patterns.The control concept to control the pose (i.e., the position and orientation) of the pen and the contact force are explained in Section III.The experimental results are presented and described in Section IV.Finally, this work is concluded in Section V.

II. PATH PROJECTION AND TRAJECTORY GENERATION
In this section, two path projection methods to transfer a user-provided 2-D input pattern onto a 3-D object are presented and the robot trajectory generation is explained.First, a LSCM is introduced and explained in detail.Next, a simple parallel projection method is presented.Finally, the robot trajectory is generated based on the result of the two projection methods.
A. Path Projection Using LSCM [25], [26] A flowchart illustrating the individual steps of the path projection with the LSCM is shown in Fig. 2. In order to handle meshes of high complexity, the mesh of the 3-D object is first decomposed into multiple mesh segments, which is explained in Section II-A1.This preparation step limits the local distortions of the transferred 2-D input pattern in the subsequent workflow.Next, the mesh segments are flattened using a LSCM approach, introduced in Section II-A2.The 2-D input pattern is projected on the flattened surface, and then, an inverse mapping transfers the pattern back onto the 3-D object, which is discussed in Section II-A3.The methods in this section are summarized from [25] and [26] and are tightly integrated in the path-planning workflow.Subsequently, these methods are utilized to compare the two projection methods, i.e., the LSCM and the parallel projection method.1) Segmentation: As a first preparation step, the mesh segmentation algorithm presented in [25] is applied to the mesh of the 3-D object before flattening the individual mesh segments.In this way, distortions in the projected 2-D paths are minimized and the replication accuracy on the 3-D object is improved.
The algorithm [25] first finds the boundaries between two segments by computing the so-called sharpness criterion for each edge of the mesh, where n i and n j denote the normal vectors of two adjacent triangles T i and T j , respectively, see [27], and ∥•∥ 2 is the Euclidean 2-norm.Edges for which the angle w i, j in (1) is over a certain threshold value are combined to feature curves.Next, the triangles with the maximum geodesic distance to a feature curve are determined.These triangles are subsequently used as seeds for a region growing algorithm to obtain the individual mesh segments.An example of a segmented 3-D object is shown in Fig. 3.
2) Least-Squares Conformal Mapping: The LSCM approach [25] is used to flatten the mesh segments of the 3-D object into 2-D meshes.In general, a locally isotropic conformal map X : (u, v) → (x, y) preserves the local angles and therefore the shape of small figures, but generally not their size.
In this section, the considered mesh segment T consists of n triangles T i , i = 1, . . ., n, and n vertices.Each triangle T i has a local orthonormal basis and the coordinates of the vertices are given by (x i,1 , y i,1 ), (x i,2 , y i,2 ), and (x i,3 , y i,3 ).Then, the conformal mapping for a single triangle fulfills the condition with the complex number X = x + iy, where i denotes the imaginary unit.The inverse conformal mapping U : (x, y) → (u, v) reads as [28] ∂U ∂ x which is a formulation of the Cauchy-Riemann equations for complex numbers U = u + iv.Since (3) cannot be satisfied for all triangles T i of the mesh segment T simultaneously, the minimization problem min is formulated, where |•| denotes the magnitude of the complex number • and A i is the area of the triangle T i , i = 1, . . ., n.
The gradients in (4b) for a single triangle T i read as with and +(x i,3 y i,1 − y i,3 x i,1 ) (8) while satisfying the Cauchy-Riemann equations see [29].The compact formulation is found with U i, j = u i, j + iv i, j using (5) and Inserting (10) into the minimization problem (4), the optimization problem for the whole mesh segment T with all triangles Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
with the coordinates of the vertices U i,1 -U i,3 of the corresponding triangle T i , i = 1, . . ., n.Finally, to formulate the conformal mapping for the complete mesh segment T , all vertices U T = U 1 • • • U n of this segment have to be considered.Note that a single vertex may be contained in multiple triangles.
In order to solve the optimization problem (12), the cost function C(U) is decomposed in the form with the sparse index matrix N ∈ R n× n and their Hermitian conjugated matrix N * .Thereby, the elements are calculated as with (11).Equation ( 13) is rewritten in the quadratic form The vector of mapped vertex coordinates U is decomposed into where U f denotes the vector of free (unknown) vertex coordinates and U p are the pinned (given) vertex coordinates.As suggested in the original work [25], the two vertices with the maximum distance to each other are pinned for each mesh segment.In this way, ( 12) is reformulated as a least-squares problem in the unknown variables and with where N f and N p denote the sparse index matrices for the free and pinned vertices, respectively, and the superscripts R and I in ( 16) and ( 18) refer to the real and imaginary parts of the complex-valued vectors and matrices, respectively.Finally, the least-squares problem ( 17) is solved in MAT-LAB using a numerical solver in order to find the coordinates of the free vertices U f under the inverse conformal mapping U.For a more detailed explanation of the mapping algorithm, the reader is referred to [25].
3) 2-D Path Projection and Inverse Mapping: In this section, the user-provided 2-D input pattern is transferred to a user-specified location and size onto the 3-D object.To this end, the 2-D input pattern is projected on the corresponding flattened mesh segment and is subsequently mapped back onto the 3-D object by the inverse conformal mapping U, see Section II-A2, using barycentric coordinates [26].
The 2-D input pattern is created by the user with a touchscreen, digitizer, or mouse interface and is given by with the rotation matrix Note that if the 2-D path covers multiple mesh segments, those segments are combined and flattened again.
Next, the correspondence between the flattened mesh segment and the 3-D mesh segment is established via the triangles of both meshes.For each 2-D path point p t,k , k = 1, . . ., K , from the flattened mesh segment, the associated triangle T k is determined.This is performed using the efficient bin-based algorithm published in [26].
Finally, the 2-D path points p t,k are mapped onto the 3-D mesh segment using barycentric coordinates inside the corresponding triangles T k .In general, barycentric coordinates map between two arbitrary triangles based on the corresponding vertices, see, e.g., [30].A given point p t is mapped from one triangle (g 1 , g 2 , g 3 ) to another triangle (w 1 , w 2 , w 3 ) in the form, see Fig. 4 p o = Ã1 w 1 + Ã2 w 2 + Ã3 w 3 (21) with In (22), the function S(•, •, •) calculates the area of the enclosed triangle.In this way, all 3-D path points p o,k , k = 1, . . ., K , are determined.The vertices (w 1 , w 2 , w 3 ) are represented with respect to an object frame O, e.g., located at the origin of the 3-D object.Therefore, the points p o,k are also described with respect to this object frame O.For each 3-D path point, a coordinate frame on the surface with the origin p o,k and the orientation R o,k is constructed such that the z-axis, i.e., the pen, is parallel to the local surface normal vector.The remaining orientation is derived from a suitable reference orientation.

B. Parallel Path Projection
The parallel projection method is explained in this section, which is simple and serves as a comparison approach for the performance evaluation of the experimental results.The parallel projection method is mostly used to project 3-D objects on 2-D planes by projecting the points along parallel projection rays, see Fig. 5.If the 2-D plane is perpendicular to the projection rays, this projection method is called orthographic parallel projection and otherwise oblique, see [31].
In this work, the inverse projection is needed, i.e., the 2-D pattern of the user input has to be mapped to the 3-D object.Therefore, the points of the user input p d,k are given in a 2-D plane.This 2-D plane can be either defined in front of the 3-D object by the user or it can be automatically computed based on the mean value of the normal vectors of the corresponding triangles.Then, parallel projection rays perpendicular to this 2-D plane are generated from each position p d,k .At the intersection points of the parallel rays with the surface of the 3-D object, the 3-D path points p o,k of the 2-D input, described in the object frame O, are found.The intersection points can be computed with, e.g., the ray-triangle intersection algorithm [32].Analogous to Section II-A, local coordinate frames R o,k are generated for each 3-D path point p o,k .The direction of the parallel projection rays is used as the z-axis of this frame.The remaining orientations are again computed based on a reference orientation.Note that the orientations of all path points p o,k are equal since all projection rays are parallel.

C. Cartesian Robot Trajectory
The projection result of Sections II-A and II-B is described by the sequence of 3-D path points p o,k , k = 1, . . ., K , which is located on the surface of the 3-D object.Along this 3-D path, the robot has to draw the user-defined pattern.Furthermore, also the corresponding orientations R o,k are known from the projection.Based on this pose sequence, the executable robot trajectory is generated.A flowchart of the necessary computation steps is shown in Fig. 6.
If a user-provided 2-D input pattern contains multiple disconnected path segments, additional transition points are added to the sequence of 3-D path points.Between two segments, the robot retracts the pen from the surface by performing a linear motion.Subsequently, the robot moves to the starting point of the next path segment and approaches the surface again.The individual sequences of 3-D path points are interpolated according to [33], where a 5th-order polynomial is used for the time parametrization.In this way, the drawing process of the robot starts and ends smoothly.

III. CONTROL CONCEPT
In order to realize the robotic drawing process with a pen on the 3-D object, the generated trajectory is executed by the robot using a suitable control concept, which is detailed in this section.First, a standard task-space controller is described to control the motion of the pen on the surface of the 3-D object.Second, contact force estimation [34] is introduced.Third, a hybrid force/motion controller is adapted from [35] and [36] to control the contact force during the drawing process and improve the robotic drawing task.

A. Motion Control
The dynamic robot model reads as M(q) q + C(q, q) q + g(q) = τ + τ ext (23) with the joint position q, the positive definite mass matrix M(q), the Coriolis matrix C(q, q), and the vector of gravitational forces g(q), see, e.g., [35].The generalized torque τ is considered as control input and the external torques acting on the robot are denoted by τ ext .The forward kinematics of the robot computes the pose of the pen tip, i.e., the contact frame C with respect to the robot base frame B, composed of the position p C B and the unit quaternion o C B .In (24) and the remainder of this work, the notation (•) Y X refers to mathematical objects describing the geometric relation of the frame Y with respect to the frame X , expressed in X .
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Applying inverse dynamics control, see [35] τ = M(q)v m + C(q, q) q + g(q) (25) to (23) and neglecting the external torques τ ext yields the new linear system dynamics in the form with the new input v m .The relation between the task-space velocities of the pen tip, ṗC B and ω C B , and the joint velocities q is given by the geometric Jacobian J C B (q) as ṗC In (27), the geometric Jacobian J C B (q) consists of the Jacobian related to the linear velocity J C B,v (q) and the Jacobian related to the angular velocity J C B,ω (q).A standard task-space controller is implemented by choosing the new input v m as with the control input where A † = A T (AA T ) −1 is the right pseudoinverse of the matrix A. In (29), the desired Cartesian trajectory with the desired frame D, is introduced by transforming the robot trajectory from the object frame O, see Section II-C, to the robot base frame B. The geometric relation of the object frame O with respect to the robot base frame B is determined using a calibration procedure before the experiment, see, e.g., [37].Hence, the position error of the controller (29) is computed as p = p D B − p C B , the angular velocity error is computed as ω = ω D B − ω C B , and the quaternion error e o is the vector part of the quaternion product o D B ⊗ (o C B ) −1 , see [35], [38].The control matrix K ω is diagonal and positive definite, the gain K o > 0, and the choice of the gain matrices K D , K P , and K I and the proof of the asymptotic stability of (29) are given in [39].

B. Contact Force Estimation
In this work, no external force/torque sensor is available to directly measure the interacting forces between the pen and the 3-D object.Instead, the contact force estimation [34] is employed and summarized in this section.
Based on the dynamic robot model ( 23), Magrini et al. [34] showed that using the residual vector r(t) = K c M(q) q − t 0 τ + C T (q, q) q − g(q) + r ds (30) leads to the residual dynamics from which the external torques τ ext ≈ r are estimated with a large positive definite gain matrix K c .
In general, the relation between the contact forces and moments h T = [f T m T ] and the external torques τ ext is established with the geometric Jacobian J C B (q) of the contact frame C as Assuming a point contact of the pen with the object's surface, the moment m in the contact frame C vanishes and the estimated force f results from

C. Hybrid Force/Motion Control
In this section, the hybrid force/motion controller proposed in [35] and [36] is extended for the robotic drawing task.To this end, the lateral position and the orientation of the pen are controlled simultaneously, while the estimated contact force f at the pen tip along the surface normal vector is regulated to the desired value.
Since a point contact is assumed, m = 0 holds for (32), and the inverse dynamics control law for (23) reads as with a new control input a.Note that the external torque τ ext is assumed to be compensated with the last term in (34).The new control input a is chosen as q) q (35) with the motion control input vc and the force control input v f to be defined later.In order to apply the control concept with respect to the contact frame C, the transformation matrix given by is utilized in (35).Moreover, the constant selection matrices 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 are introduced and vc from ( 35) is similar to v c from (29) by replacing p, e o , ṗ, and ω with Furthermore, also the feedforward terms pD B and ωD B have to be transformed into the contact frame C with (36) and ṘC B is neglected in (35) and (38), see [36].The force control input v f is chosen as Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
with the desired force f T d = 0 0 f d,z and suitable positive definite diagonal gain matrices K P f and K I f .The contact force f is assumed to appear only in the z-direction of the contact frame of the pen tip C, i.e., fC,z , cf., (37).Because the selection matrices Y f and Y v are complementary, the force and motion control loops are decoupled, see [35], [40].A proof of the asymptotic stability of ( 34) and ( 35) is given in [35].
Kinematically redundant robots exhibit an additional null space, which is stabilized using the null-space control law τ 2 = M(q)P −b(q) − K dn q − K pn (q − q) (40) with the projection matrix P = I − (J C B (q)) † (J C B (q)), see [41], and the positive definite diagonal gain matrices K dn and K pn .In (40), the barrier function b T is defined by to prevent a robot with m degrees of freedom from reaching joint positions near the mechanical axis limits q T max = [q 1,max q 2,max . . .q m,max ] and q T min = [q 1,min q 2,min . . .q m,min ].The parameters b max and b min are used to tune the barrier functions and q denotes the middle angle between the axis limits q = 1 2 q 1,max + q 1,min q 2,max + q 2,min . . .q m,max + q m,min .
The control torque of the null-space control law τ 2 is added to (34), resulting in the final control input τ = τ 1 + τ 2 .

IV. EXPERIMENTAL RESULTS
In this section, the experimental setup and the results for the robotic drawing process are presented and discussed.First, the parallel path projection and the proposed path projection using a LSCM from Section II are evaluated in simulation with a user-provided 2-D input pattern.Second, the properties of these projection methods are compared.Third, drawing trajectories are generated and executed experimentally on the robot using the motion controller from Section III-A, and fourth, the hybrid force/motion controller from Section III-C is evaluated experimentally.Finally, the drawing results and measurements are compared directly.A video of the experimental drawing process is provided at www.acin.tuwien.ac.at/c1eb.

A. Experimental Setup
The experimental setup for the drawing process is shown in Fig. 1.In this setup, the ceiling-mounted industrial robot KUKA LBR iiwa 14 R820 is equipped with a pen tool attached to the end-effector.The tool comprises a passive compliance mechanism to account for absolute positioning errors and model uncertainties of the robot and its environment.The workpiece in the drawing process is a 3-D-printed rabbit model.The optimal robot base placement relative to the 3-D object is computed offline using the optimization-based  algorithm proposed in [42].The actual robot base placement in the experimental setup is determined using a calibration procedure before the experiment can start, see Fig. 7.During this calibration procedure, the robot is equipped with calibration pins at the end-effector (red rectangles in Fig. 7), which tightly fit into holes at the base of the 3-D-printed rabbit (red circles in Fig. 7).The actual robot base placement is obtained from the measurement of the robot configuration q and the forward kinematics (24), which accurately calibrates the pose of the robot base for subsequent path planning and trajectory execution.
The 2-D input pattern is provided by the user either simply using a computer mouse, a touchscreen, or drawing patterns with a digitizer on a tablet device.Furthermore, a path may also be generated from parametric equations.In the following experimental drawing process, the 2-D input pattern with four disconnected segments T1 ⃝-T4 ⃝ shown in Fig. 8 is used, i.e., a smiley symbol.

B. Path Projection
In the following, the user-provided 2-D input pattern in Fig. 8 is projected at a user-specified location on the 3-D object, which is the face of the 3-D-printed rabbit, see Fig. 9.Both projection methods introduced in Section II, i.e., the parallel projection method and the proposed projection method based on the LSCM, are used.Subsequently, the projection results are compared and robot trajectories for the process execution are generated.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply. 1) Parallel Projection Method: For the simple parallel projection method from Section II-B, parallel green rays are generated originating at each path point p t,k , k = 1, . . ., K , from the blue 2-D input pattern, see Fig. 9(a).Then, the path points p o,k , k = 1, . . ., K , on the 3-D object are found at the intersection points of the green rays with the 3-D object's mesh using the implementation of the ray-triangle intersection from [43].At each path point p o,k , k = 1, . . ., K , a desired end-effector orientation is generated from path planning.Thereby, the z-axis of the end-effector is aligned with the green ray and the xand y-axes are chosen according to a reference orientation.Note that the orientation is equal for every path point.Finally, transition paths are added and a spline interpolation is computed based on the 3-D path points with a suitable time parametrization, see Section II-C.
Examining the parallel projection result in Fig. 9(a), a large distortion can be seen at point P1  ⃝ due to the high curvature of the 3-D object.In comparison, the eyes of the smiley at P3  ⃝ are less distorted because of the smaller curvature.Nevertheless, the eyes are in an elliptic shape and not properly placed.If the parallel projection method is used to project a 2-D input pattern to areas with a small curvature, e.g., P4  ⃝ or P5 ⃝ in Fig. 9(a), the drawing result of the whole 2-D input would be significantly better without notable distortions.
Due to the simplicity of the projection method, the computation is finished after approximately 50 ms on an Intel Core i7-8700K at 3.70 GHz.
2) Path Projection Using LSCM: The path projection on the rabbit according to Section II-A is computed by segmenting the object first, see Fig. 3, and then flattening the individual segments using the conformal mapping.In the next step, the 2-D input pattern is projected on the flattened segments, i.e., the face of the rabbit, see Fig. 10.Note that the two segments located at the face of the rabbit in Fig. 3(b) have to be combined to be able to apply the LSCM for the whole face area.The 2-D input pattern is transferred back to the 3-D object using the barycentric coordinates introduced in Section II-A3.The result of the 2-D input pattern on the 3-D surface of the workpiece is shown in Fig. 9(b).As explained in Section II-C, the robot trajectory is computed by inserting transition points and interpolating the sequence of 3-D path points using a spline with suitable time parametrization.The orientation of the z-direction, i.e., the pen, is chosen based on the normal vector of the surface and the remaining directions are derived from a reference orientation.
Note that the shape of the pattern, in particular the outer circle of the smiley at P1  ⃝ and the circular eyes at P3 ⃝ in Fig. 9(b), is projected on the 3-D object with minimum distortions.
The path projection using conformal mapping is performed on the 3-D-printed rabbit object, which comprises approximately 13 000 faces and 6500 vertices.The computation is executed on an Intel Core i7-8700K with 3.70 GHz base clock frequency.The most time-consuming computation of this offline planning is the segmentation from Section II-A1 taking around 5 s and the LSCM from Section II-A2 can be calculated in 0.3 s.Note that those steps have to be executed only once for each 3-D object, see Fig. 2.

C. Properties of Projection Methods
In this section, further simulations to compare the parallel projection method with the LSCM are conducted, see Fig. 11.The 2-D input pattern from Fig. 8 is projected on the ear of the 3-D-printed rabbit.Due to the high curvature of the area around the ear, projections are challenging and local distortions can occur.
In the first experiment, the parallel projection method is employed.In this experiment, large distortions due to the high curvature occur, see Fig. 11(a).Note that if the 2-D pattern would be larger than the 3-D object, the green rays would not intersect with the 3-D object at all, and therefore, the projection could not be fully computed and the trajectory generation fails.
The second experiment shows the projection of the 2-D input pattern from Fig. 8 on the ear area of the 3-D-printed rabbit with the LSCM, see Fig. 11(b).Using this mapping, the 2-D input pattern is projected visually proper on the 3-D workpiece.Note that due to flattening the segments of the 3-D object into a 2-D form, the 2-D input pattern is wrapped around the ear area at the 3-D object, see also Fig. 10.Using this method, the projection result contains the complete user-provided pattern and the trajectory for the robotic drawing process can be calculated.

D. Drawing Process With Motion Control
In this section, the drawing process using the two presented path projection results from Section IV-B is executed with the pure motion control as introduced in Section III-A.In the following, the planning and measurement results are discussed in terms of the pen motions as 3-D paths, the position control errors, and the joint-space paths.
1) Planned and Executed 3-D Paths: The drawing process is planned with both projection methods and executed on the KUKA LBR iiwa 14 R820 using pure motion control.The resulting 3-D paths are shown in Fig. 12 for the parallel projection and in Fig. 13 for the LSCM projection.Figs. 12 and 13 show the desired trajectory p D B of the contact frame C, i.e., the pen tip, the corresponding pen orientations n D B , and the actual trajectory p C B , computed using the forward kinematics (24), for the respective projection method.The automatically generated transition paths are also visible.For the parallel projection, the pen orientation n D B is constant and aligned with the projection direction, while it remains aligned with the surface normal vector using the LSCM projection.Consequently, for the parallel projection, a high position deviation between the desired and the actual trajectory becomes clearly visible around P2  ⃝.In this area, the local curvature of the 3-D object is very high, see also Fig. 9(a).Due to the constant orientation n D B , the angle between the surface normal vector and the pen at the contact point C becomes large.Due to this large approach angle, the pen slips on the object's surface, which significantly deviates the pen tip from the desired trajectory p D B and results in a large position error.In comparison, the approach angle around the eyes at point P3 ⃝ is much smaller, and therefore, this effect is less pronounced, see Fig. 9.
For the LSCM projection, see Fig. 13, the pure motion control exhibits a good closed-loop control performance when Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
following the desired trajectory in the experimental setup.Even at the points P1  ⃝ and P2 ⃝ with high curvature, compare Fig. 12, the actual trajectory p C B visually does not deviate from the desired trajectory p D B .Because the pen axis n D B is aligned with the surface normal vector, the pen does not slip on the surface due to the perpendicular approach angle.Thus, no position errors appear at areas with high curvature.
2) Position Control Error: In Fig. 14, the position control errors p in the contact frame C for both experiments are presented, see (38).The topmost graph shows the contact state of the pen, where intervals with the value 1 indicate that the pen is in contact with the 3-D object.The variable l denotes the progress of the path from 0 % to 100 %.During the first interval T1  ⃝, where the pen is in contact with the 3-D object, the outer circle of the smiley, at T2  ⃝ and T3 ⃝ the eyes and during T4 ⃝ the mouth is drawn, cf.Fig. 8.In between those intervals, the robot's end-effector moves from the end of the previous pattern to the starting point of the new pattern.
Examining the position control errors p during the intervals T1 ⃝ and T4 ⃝ for the parallel projection experiment, the position deviation due to the large approach angle of the pen on the object can be clearly seen for l = 5 % − 20 % and l = 80 % − 90 % in Fig. 14.Small position errors emerge during the intervals T2  ⃝ and T3 ⃝ due to the small local curvature and the small approach angle.Note that due to the passive compliance of the pen holder in the z-direction, the position error in this direction is small.In contrast, the position control errors p of the LSCM experiment in Fig. 14 (red lines) remain small during the entire execution time.Quantitatively, the position control error p with respect to the contact frame C remains below 1 mm for all Cartesian position coordinates.
Note that the lengths of the intervals T1 ⃝-T4 ⃝ for the two projection methods differ.This is due to the fact that the range of the axis motion during the drawing process depends on the planned trajectory.Using the LSCM, the pen orientation is always normal to the surface; therefore, the robot has to perform wide joint motions and, consequently, the time intervals are longer.Note that the positive definite diagonal gain matrices K ω , K pn , K dn , and K o > 0 are chosen empirically and the gain matrices K D , K P , and K I are found by pole placement.
3) Joint-Space Paths: The joint-space paths qT (l) = q1 q2 • • • q7 for the process execution with both projections are shown in Fig. 15, where the individual joint angles qh are normalized to their axes limits q h,min and q h,max in the form qh = 2q h − (q h,max + q h,min ) q h,max − q h,min , h = 1, . . ., 7.
As the orientation of the pen during the parallel projection experiment remains constant, only small changes in the joint angles are required to follow the corresponding desired path, see the blue lines in Fig. 15.The joint-space path q(l) for the LSCM experiment (red lines in Fig. 15) shows that in this case, significantly larger robot movements are performed.Most of the joints come close to the respective mechanical axis limit at one point during process execution.Therefore, it is necessary to use the advanced null-space control law (40) and ( 41) to be able to execute this process.Consequently, this shows that Fig. 14.Evolution of the pen/surface contact (top) and position control errors p using the parallel projection method and the LSCM using pure motion control.
the execution of the planned trajectory is more challenging for the LSCM, but it yields more accurate results for the drawing process, compare Figs. 12 and 13.

E. Drawing Process With Hybrid Force/Motion Control
In this experiment, the hybrid force/motion controller from Section III-C is employed to perform the robotic drawing process based on the planned trajectory with the LSCM.The results are discussed in terms of the planned and executed 3-D paths, the position control error, and the contact force.39) is applied in this direction.Therefore, the position in the z-direction deviates from the planned trajectory to guarantee a constant contact force.Note that during the transition phase from the end of one pattern to the starting point of the next pattern, the pure motion controller as in Section IV-D is used.To achieve a smooth transition between the hybrid force/motion control and the pure motion control, slightly smaller diagonal entries of the positive definite gain matrix K ω and K o > 0 and smaller diagonal entries of the matrices K D , K P , and K I compared to the experiment with pure motion control in Section IV-D were chosen.In the force control law (39), the first term, i.e., the feedforward term f d , is neglected and the term q) is added with the positive definite diagonal matrix K D f to damp Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.Fig. 15.Joint-space path q(l) using the parallel projection method and the LSCM with pure motion control.The paths of the individual joints are normalized to their respective axes limits q min and q max .the motion along the z-direction of the contact frame C and generate smooth robot motions during the experiment.Note that the direct force feedback from the last term of ( 34) is omitted due to noise in the contact force estimation f and a disturbance observer is added for friction compensation in the robot joints.2) Contact Force: The estimated contact forces f are shown in Fig. 17 for the process execution of the LSCM trajectory with the hybrid force/motion control and the pure motion control of Section IV-D.Clearly, the contact force parallel to the surface normal fC,z is controlled to the desired value of −3 N by the hybrid force/motion controller (red line).At every change of the pen/surface contact state, the controller is switched from the hybrid force/motion controller to the pure motion controller and vice versa.The small contact force peaks in fC,z , see Fig. 17, originate from these controller switching operations.In contrast, the contact force fC,z is not controlled by the pure motion controller, and hence, it varies between −4 N and nearly zero for this experiment (blue line).In addition, a loss of the pen/surface contact can occur due to misalignment of the workpiece and/or inaccuracies of the robot.For both controllers, the estimated forces are approximately zero in phases without contact.The stick-slip effect is observed between the pen tip and the surface if the normal contact force becomes too high.Other friction effects are not perceivable in the estimation because the pen is controlled to be always normal to the surface, and only the normal contact force is estimated.
Remark 1: Note that the contact force estimation [34] requires a precisely calibrated dynamic robot model (23).Otherwise, significant estimation errors may occur, even if no pen/surface contact is present.Alternatively, the contact force estimation can be calibrated for a specific robot trajectory by performing the experiment without any pen/surface contact as a reference motion with f = 0.
3) Position Control Error: In order to evaluate the performance of the hybrid force/motion control, the position control error p of the drawing process is shown in Fig. 18.Overall, small control errors are observed for x and y, whereas the small peaks originate from the controller switching, compare Fig. 17.During the intervals with pen/surface contact, the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.position control error p in z-direction is higher because the force controller adapts the position in the z-direction to control the contact force fC,z .

F. Comparison of the Drawing Results
In this section, the experimental drawing results of both projection methods introduced in Section II and the two control concepts from Section III are shown and compared.The resulting drawing pattern on the 3-D-printed rabbit using the parallel projection method with pure motion control is shown in Fig. 19(a).For comparison, the LSCM pattern is drawn on top of the drawing pattern for the parallel projection.Comparing the two projection methods, distortions from the parallel projection method are seen, especially at areas with high curvature, e.g., at point P1 ⃝.The robot is not able to accurately follow the desired trajectory at areas with a large approach angle using the parallel projection method with pure motion control, e.g., P2  ⃝, compare Figs. 12 and 13.Next, the drawing results with the two presented control concepts are compared for the LSCM trajectory, see Fig. 20.In the drawing result using the pure motion controller in Fig. 20(a), a fluctuating line thickness is observed.In particular, at point P6 ⃝, nearly no pen/surface contact is present, emerging from inaccuracies of the robot kinematics.This thin line results from a low contact force of the pen tip, which can also be seen in the estimated contact force fC,z in Fig. 17

V. CONCLUSION
In this work, an automated workflow for full customization of products using robotic manufacturing is presented on the basis of a drawing process on a complex surface of a 3-D object.In this process, a 2-D input pattern is provided by a user together with the desired size, location, and rotation on the 3-D object and then drawn on the 3-D object using an industrial robot.
The workflow starts with a mapping procedure of the 2-D input pattern to the 3-D object, for which two different mapping methods are presented and explained, i.e., a conformal mapping and a simple parallel projection approach.The conformal mapping procedure comprises a segmentation step, a LSCM to flatten the segments, and an inverse map using barycentric coordinates.In this way, distortions of the 2-D input pattern are minimized and a 3-D path on the 3-D object is obtained.Although the parallel projection method can be used in areas with small curvature and low complexity, only with the more advanced conformal mapping procedure, a visually proper projection is achieved.Based on the result of the two presented projection methods, robot trajectories are planned to be executed in an experimental setup with an industrial robot.For the planned trajectories, two different control concepts, i.e., pure motion control and hybrid force/motion control, are presented and employed in the experiments.In addition, the contact force during the drawing process is estimated.The pure motion controller is able to execute the trajectory with small errors, however, only the hybrid force/motion controller is able to maintain the desired contact force normal to the surface during the whole task execution, which is necessary for achieving a high production quality.This is demonstrated in this work by visually comparing the drawing process results using both mapping procedures and the two control concepts.
In industry, this approach can be used to automatically map 2-D manufacturing paths to different 3-D objects while maintaining the required accuracy in position and contact force during the whole process.This automated workflow is directly applicable to many other manufacturing processes such as automated laser engraving, milling, or ultrasonic cutting.

Manuscript received 30
November 2022; revised 6 July 2023; accepted 13 July 2023.Date of publication 29 December 2023; date of current version 20 August 2024.The authors acknowledge TU Wien Bibliothek for financial support through its Open Access Funding Programme.Recommended by Associate Editor A. Serrani.(Corresponding author: Thomas Weingartshofer.)

Fig. 2 .
Fig. 2. Flowchart of the LSCM to project the 2-D input pattern.

Fig. 4 .
Fig. 4. Mapping from the triangle of the flattened object to the 3-D object, compare [26].(a) Triangle on the flattened object.(b) Triangle on the 3-D object.

Fig. 6 .
Fig. 6.Flowchart of generating a robot trajectory based on the projection result.

Fig. 10 .
Fig. 10.Projection of the 2-D input pattern on the flattened segments of the 3-D object.

Fig. 11 .
Fig. 11.2-D input path projected on the ear of the 3-D-printed rabbit with (a) parallel projection method and (b) LSCM.

Fig. 12 .
Fig. 12. Desired and executed 3-D path p D B and p C B , respectively, and the corresponding surface normal vectors n D B using the parallel projection method with pure motion control.

Fig. 13 .
Fig. 13.Desired and executed 3-D path p D B and p C B , respectively, and the corresponding surface normal vectors n D B using the LSCM method with pure motion control.

1 )
Planned and Executed 3-D Paths: The desired and actual trajectories of the pen tip p D B and p C B , respectively, are shown in Fig. 16.It can be clearly seen that the actual trajectory p C B deviates in the z-direction from the desired trajectory p D B because the force control law (

Fig. 16 .
Fig. 16.Desired and executed 3-D path p D B and p C B , respectively, and the corresponding surface normal vectors n D B using the LSCM method with hybrid force/motion control.

Fig. 17 .
Fig.17.Evolution of the pen/surface contact state (top) and the estimated contact forces fC,z in the z-direction (bottom) during the drawing process with the pure motion control and the hybrid force/motion control.

Fig. 18 .
Fig. 18.Evolution of the pen/surface contact (top) and position control error p using the LSCM with the hybrid force/motion control.
at l ≈ 10%.A uniform line thickness is achieved with the hybrid force/motion control in Fig.20(b).Note that the experiments for this drawing process are executed without an absolute

Fig. 19 .
Fig. 19.Resulting patterns on the 3-D object with pure motion control (a) using the parallel projection and (b) together with the LSCM.

Fig. 20 .
Fig. 20.Resulting patterns on the 3-D object with the LSCM using (a) motion control and (b) hybrid force/motion control.