Thorsten Schüppstuhl Kirsten Tracht Dominik Henrich Editors

Annals of Scientific Society for Assembly, Handling and Industrial Robotics

Annals of Scientifc Society for Assembly, Handling and Industrial Robotics

Thorsten Schüppstuhl · Kirsten Tracht · Dominik Henrich Editors

# Annals of Scientifc Society for Assembly, Handling and Industrial Robotics

*Editors* Thorsten Schüppstuhl Institute of Aircraft Production Technology Hamburg University of Technology Hamburg, Germany

Dominik Henrich Chair for Robotics and Embedded Systems University of Bayreuth Bayreuth, Germany

Kirsten Tracht Bremen Institute for Mechanical Engineering (bime) University of Bremen Bremen, Germany

#### ISBN 978-3-662-61754-0 ISBN 978-3-662-61755-7 (eBook) https://doi.org/10.1007/978-3-662-61755-7

© The Editor(s) (if applicable) and The Author(s) 2020. This book is an open access publication.

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specifc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affliations.

This Springer Vieweg imprint is published by the registered company Springer-Verlag GmbH, DE part of Springer Nature.

The registered company address is: Heidelberger Platz 3, 14197 Berlin, Germany

## **Inhaltsverzeichnis**




## **Virtual Reality based Assembly Process Validation and Rework Assistance with consistent Data Exchange**

Rainer Müller, Leenhard Hörauf, Attique Bashir, Martin Karkowski and Max Eichenwald

Centre for Mechatronics and Automation gGmbH (ZeMA), Group of Assembly Systems and Automation Technology, Gewerbepark Eschbergerweg 46, Geb. 9, 66121 Saarbrücken

m.eichenwald@zema.de

**Abstract.** Due to rising product and process complexity, assistance systems for humans are increasingly used in both the planning and execution of assembly activities or as support systems for reworking. However, developments available so far usually act as isolated solutions. Therefore a continuous dataflow between top and shop floor is not given. This article presents a comprehensive assistance system consisting of a virtual planning environment and various smart devices at an assembly workstation. In the virtual environment, the feasibility of initially created product condition graphs will be validated and a training of the repair scope to be carried out will be done. As a result, a process plan is derived from the virtual environment and transferred to an assisted workstation for reworking. The repair on the real product is carried out using the created process plan so that a continuous data flow is available. Thus the system supports the persons involved in the product analysis and work plan creation up to the execution of repair activities during rework.

**Keywords:** Virtual Assembly, Assembly/Disassembly Planning, Training Platform, Assistance Systems, Consistent Data Exchange

## **1 Introduction**

Planning processes for assembly and disassembly are becoming more and more extensive due to increasing product complexity, variant diversity and customer-specific individualization options [1, 2]. In addition, manufacturing companies are increasingly forced to reduce development costs and time. Since assembly volumes represent an immense cost factor with regard to the product development process [2, 3] productivity, flexibility and efficiency must increase within the scope of the corresponding planning. However, this is becoming more and more difficult to achieve as the feasibility testing of assembly volumes also becomes more extensive. As a result, virtual development methods and tools are becoming increasingly important. The threedimensional representation within Virtual Reality (VR) enables the user to visualize and interact with the digital product in a realistic way. Time-consuming and costintensive planning with the aid of physical prototypes thus becomes superfluous. The possible fields of application of VR technology for manufacturing companies are very diverse. A classification according to task fields can take place according to: design (prototyping), operations management (planning, simulation, training), manufacturing processes (production flows, assembly, inspections) [4] and visualization [5].

This article focuses on the areas of visualization, training and validation of assembly scopes in a virtual environment as well as the consistent data flow to and from the VR-Application. Based on that, the use of the results for other applications can be enabled. Initially created product status graphs, in which information regarding assembly and disassembly scopes are stored, are to be validated in a virtual environment and adapted if necessary. For this purpose, the assembly scopes are created in a standardized platform and transferred to the virtual environment. To achieve a sustainable utilization of validated product condition graphs, information can be transmitted to other applications e.g. to an assisting workstation. After the initial planning phase the virtual environment can be used as a training platform for the further education of employees.

In the following, the current state of research of virtual environments in the field of assembly is described (chapter 2). Since the paper focuses the overall data flow between VR and other applications, a cognitive assistance system is described as an example system which can profit from the validated product and process scopes. An overview of the data transfer and the necessary systems is given in chapter 3. The technological implementation of the developed VR application, functionalities and the realization of the data exchange is described in chapter 4. A summary and outlook is given in chapter 5.

### **2 State of the art and initial situation**

This chapter highlights the current developments and areas of application in the field of virtual environments with regard to assembly planning and production. One focus of the paper is on consistent data exchange. Downstream systems should benefit from the results of the virtual environment. For this paper a cognitive assistance system in the field of rework on the shop floor was used. Therefore the last section of this chapter discusses the current state of the art of this class of assistance systems.

#### **2.1 Virtual Environment and Assembly Planning**

Further developments with regard to the performance of software and hardware, as well as the possibility of realistic depiction of physical products allow Virtual Environments (VE) to become present in the industrial environment. In the course of industry 4.0, VE is a fundamental technology due to the development of new potentials and its broad application spectrum. According to Nikolaos et al. [3], a technological classification of the VE can be divided into


The traditional procedure for planning assembly and disassembly volumes is strongly linked to the expert knowledge of the planner. A fully automatic generation, also with the help of powerful CAD programs, fails due to the mathematical formulation and utilization of this knowledge. Instead of using abstract algorithms for planning the assembly scope, Virtual Assembly (VA) supports the planner within the virtual environment. With regard to the validation of possible assembly and disassembly sequences, various VR applications can be found. One of the first systems developed is the *Virtual Assembly Design Environment* (VADE), in which user interaction, collision analyses and assembly tools are provided. Based on this, further VR environments exist with regard to validation, visualization and simulation of assembly scopes [6–9]. However, virtual environments are often regarded only as isolated solutions. The data transfer usually only takes place one way to the VR application, but not back again. Thus, the feedback and transfer of changes made to the assembly scope to data management systems or further applications is only insufficiently considered.

#### **2.2 Virtual Training System**

Products are becoming increasingly complex and thus more difficult and expensive to manufacture. The training of employees on real objects is therefore only possible to a limited extent. New technological developments for supporting training and further education are therefore becoming more important [2]. In addition to the applications for planning and development, there already exist various VR platforms for mapping training scenarios [10–13]. The digital three-dimensional model, assembly as well as disassembly sequences are simulated realistically. In addition to the purely visual representation within the VR, the increase of the user's performance and learning rate could be shown by implementing auditory and tactile feedback [7]. Assembly work requiring a high degree of psychomotor skills can be trained thoroughly (learning by doing) and cost-effectively within the VR application. The goal of virtual environments as learning platforms is always the transfer and recognition of simulated scenarios from user's in reality [2].

#### **2.3 Cognitive assistance systems**

In the manual assembly of complex products, the use of cognitive assistance systems (also: digital assistance systems [14]) is suitable for supporting employees. In most cases this is done by fading in assembly process information [15, 16]. Extended systems have integrated resources for process testing and documentation, so that processes can be checked for their correct execution and logged for traceability purposes. This is intended to detect and eliminate errors on site and reduce the return rate.

The assistance systems are usually operated from the shop floor. A process list (assembly sequence) is created by the foreman or a comparable employee to enable secure assembly. Using an example of product assembly, all assembly steps are recorded step by step, together with the associated test characteristics. The next step can only be processed if the previous step with the taught-in inspection characteristics has been recognized as correct.

Assistance systems can be used for supporting the reworking of products with many variants [17]. Nevertheless, cognitive assistance systems for the rework are so far the subject of research and hardly available on the market.

## **3 Concept for the creation and validation of assembly sequences**

The main system discussed in this paper is a *VR-Application* for validating assembly scopes. Functionalities to visualize and edit those assembly scopes are implemented in the VR application. By using a standard exchange format, relevant data can be transferred to the VR application and validated. To achieve a flow of information from the VR to subsequent systems, the standardized graph tool (*SGT*) serves as a platform for data exchange. In the SGT the initial lists of the assembly order and processes are created already. The SGT was previously developed at ZeMA [18]. Edited and validated assembly scopes can be reimported in the SGT and later transferred to other systems. In this case a cognitive *assistance system* to support employees in rework where automatically created repair plans provided to the worker allow a faster disassembly and assembly of complex, variant diverse products. A summary of those three systems is given in Fig. 1. The following chapter provides a detailed description of the three systems.


**Fig. 1** Central characteristics of the virtual environment, standardized graph tool (SGT) and assistance system

## **4 Implementation of the environment for the creation and validation of the assembly sequences and transfer of the work plan into reworking**

The following section gives an overview over the previously described systems. Furthermore, the continuous data exchange between the three subsystems is described.

#### **4.1 Scope of functions within the Virtual Environment**

The virtual environment offers the possibility of visualization as well as the validation and editing of the assembly scope. The displayed product can be assembled and disassembled step by step within VR-Application according to the planning data created in the SGT tool. As shown in Fig. 2(a) the main functionalities are "Move all" (scaling and translation of the object in space), "Teleport" (jumping movement of the user, see Fig. 2(b)), as well as "Assembly" (evaluation assembly sequence) and "Reset". Color coding of successive components or assemblies makes visualization easier. Relevant process steps are displayed in a toolbar (see Fig. 2(c)). The correctness of these can thus be continuously validated. If it is necessary to edit them, another process can be selected and saved. Furthermore, there is the option to adjust the given assembly sequence. Based on the current product status, a subsequently planned component or assembly can either be acknowledged or, if desired, replaced by an object determined by the user. For this purpose, all of the following components are displayed transparently, a chosen component for replacement is highlighted (see Fig. 2(d)). Because of this functionality the user is able to step through the parts, so that the current part will be colored and selectable. If the assembly scopes are valid, they can be confirmed and saved by the planner. Verified assembly sequences are saved in tabular form. This makes it possible to identify corresponding product status paths within the JSON file and to mark these with corresponding information regarding there validity. The marking is done by coloring of the respective edges/processes (see Fig. 3(b)).

**Fig. 2** Functionalities of the Virtual Environment - (a) Menu, (b) Teleport, (c) Visualization of assembly scopes, (d) Editing process

Furthermore, validated assembly scopes can be used as training scenarios. Due to higher product complexities and more variants, employees need extensive training. An alternative to training on real prototypes can be the virtual environment.

The setup for the virtual environment includes VR glasses, two sensors to detect the position of the user, as well as a Leap Motion sensor for the detection of hand movements and gestures. On one hand, this enables intuitive handling of the application, and on the other hand, it allows to assign specific functionalities in different gestures. The engine is implemented in the Unity development environment.

#### **4.2 Development and setup of a planning tool for the creation of graph-based assembly sequences**

The standardized graph tool is used to model and depict the relationships between the product, the required processes and the required operating resources (see Fig. 3(a)). To model and edit the assembly sequences, the user can select and insert elements from a configurable selection of product, process and resource templates. In addition to the visual adjustments of the inserted elements, the user can dynamically assign attributes to the elements. Such attributes can contain product-specific information such as product id, included parts of the assembly, manufacturer, etc. Different edges can be used to model the relationships (processes) between the product states. The name of a specific process which is used to get from product state n to product state n+1 can be entered here. To simplify the representation and structuring of elements, it is possible to hide different elements and to nest or cluster them hierarchically via father-child relationships. Created graph structures can be both exported and imported as JSON (*JavaScript Object Notation*) objects. With implementing the planning tool as a Web application, it can be used on a variety of different platforms.

**Fig. 3** Interface of standardized graph tool (SGT), (a) Product states of transmission, (b) valid (green) and non-valid (red) assembly scopes based on VR-Validation

#### **4.3 Data Transfer between Virtual Environment and SGT-Platform**

Using the virtual environment requires two input formats. A three-dimensional model of the product to be viewed (.obj file), as well as the product states defined in the beginning within the SGT platform. The associated assembly volumes are exported from the SGT platform on the basis of the standardized exchange format JSON.

The different possibilities of (dis-)assemble a product result in a branched representation of product states within the SGT platform. In order to visualize all of the reasonable defined scopes in the SGT platform, each path is exported as a linear lists. Therefore the exported JSON file gets parsed and transferred in multiple .csv files, related to the number of possible paths. Those .csv-files are loaded one after another into the virtual environment. Validating of editing the assembly scopes means, that relevant information about assembly order and process will be assigned to the corresponding .csv file. After all of the lists are checked, an algorithm will loop over all of the .csv files an extract the changes. Therefore the changes can also applied to the JSON file. Valid lists are highlighted by a green color, invalid lists are highlighted red. Reimporting the adapted JSON file in the SGT gives the opportunity to quickly understand/visualize which product state branches are valid and which are not (see Fig. 3(b)). An overview of the data exchange between the systems is shown in Fig. 4.

Fig. 4 Information flow between virtual environment, SGT and subsequent systems (e.g. an assistance system)

#### **4.4 Transferring the routing to rework**

The edited JSON-file enables the use of validated assembly scopes for further applications, e.g. the cognitive assistance system for rework. For this assistance system various product states that the employee can reach during the repair of a product must take into account. Therefore, a product structure is used instead of a classical assembly priority graph. The product structure is composed of nodes and edges, with the nodes representing the respective product status and the edges representing the assembly processes leading to a status change. In the first step, the employee enters an error code into the assistance system via the Co-worker guidance system (see Fig. 5) The required repair plan is then output via a database query. This controls the necessary resources in digital form and provides the employee with the necessary process information via the Co-worker guidance system.

In the technical implementation, the graph is stored in a graph and document database. The objects are organized in four categories: inform (informing the employee), parametrize (parameterizing equipment), validate (process testing) and document (documenting processes), since these four areas are covered by the assistance system. Each of these objects has a sub-object "address" and "parameter", which contain the resource to be controlled as well as the necessary parameters. The graph database allows for queries containing graph operations. For example, the shortest path from a product state to the target state can be output with a query.

The validated work plan from the virtual environment is searched by a JSON parser, whereby the nodes are written to the database and the processes are appended to the edges. The graph structure is set up with the available data and made available to the control program for operative operation.

**Fig. 5** Architecture of the cognitive coworker assistance system

### **5 Conclusion and Outlook**

In this article, a VR-supported procedure for the feasibility assurance of reworking processes by consistent data exchange was described. The central element of the planning was a graph tool that allows the representation of product and process relationships and the input of their relevant data. The process relationships are stored as a graph together with the corresponding data in a JSON formatted text file. This serves as an exchange file for other systems. It is loaded into the VR environment, where a planner can start simulating the (dis-)assembly processes. Step by step, individual processes are validated and returned to the graph-based planning system as feasible or non-feasible. In addition, the virtual environment can be used as a training platform. Employees can train on already validated assembly scopes to achieve a faster understanding of the complexity of the product.

An updated graph can be transferred to other systems. This is shown on the example of a rework system, whereby data is transferred for assistance (information and process testing) as well as for process execution and documentation.

In the future, data from the rework area should be able to be fed back. This would allow optimization potentials, uncovered in rework, to be reported in a standardized way and introduced transparently in rework. Furthermore, a process list generation has to be optimized so that the minimum number of process lists is generated, in order to make the validation process effortless and repeatable.

#### **Acknowledgment**

This paper was written in the framework of the research project KomZetSaar (Funding code 01MF117003A), which is funded by the Federal Ministry of Economic Affairs and Energy (BMWi) and supervised by the lead partner German Aerospace Center (DLR).

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Intuitive, VR- and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models**

Maximilian Metzner<sup>1</sup> , Lorenz Krieg<sup>1</sup> , Daniel Krüger<sup>2</sup> , Tobias Ködel<sup>2</sup> and Jörg Franke<sup>1</sup>

<sup>1</sup> FAU Erlangen-Nuremberg, Institute for Factory Automation and Production Systems, Egerlandstraße 7-9, 91058 Erlangen, Germany

<sup>2</sup> SIEMENS AG, Digital Industries, Gleiwitzer Straße 555, 90475 Nürnberg maximilian.metzner@faps.fau.de

**Abstract.** To manage complexity in modern, highly automated production systems, functional simulation for logic validation, known as virtual commissioning, is an important enabler. However, established systems for virtual commissioning are highly specialized software, which are solely used by simulation experts. In this contribution, we present a system for intuitive, user-integrated virtual commissioning. Using virtual reality, we allow for a first person experience of the simulation at runtime. By implementing a stereo-camera-based hand tracking system, we facilitate intuitive manual interaction with the simulation, including physical manipulation. Furthermore, we also integrate common human-machine interfaces, like touch panels, which are directly streamed from the industrial controller, offering the operator comparable interaction experience as with the real system. We conduct a user study quantifying the intuitiveness of interaction for defined tasks of our system compared to a standard simulation setup.

**Keywords:** Virtual Commissioning, Virtual Reality, Motion Capturing.

## **1 Introduction**

Increasing complexity in functional control logic of production system drives the need for a thorough testing of these algorithms. Virtual commissioning (VC), a method for testing programmable logic controller (PLC) functionality by connecting it to a simulation emulating the real system, is an important tool in this context [1, 2]. However, most production systems still require frequent interaction with human operators for value creation, setup or adaption, maintenance or fault remedying. This human factor is thus critical for a comprehensive simulation of system behavior and function. The active integration of humans into VC environments is hence an important research topic. In the next section, we highlight related work for human integration into simulations and VC of production systems. Based on this, we derive the research gap to be addressed in this contribution. An overview of the abstract solution structure including main components and functions is given before a real implementation is described. Based on a robotic handling use-case incorporating many common automation components, we conduct a user study focusing on the interaction capabilities of our system. We describe and discuss the results of this study before summarizing our research and giving an outlook to future research areas in this field.

## **2 State of the Art**

#### **2.1 Related work**

Virtual commissioning is defined as the development and testing of functional logic on a simulation model for later deployment in a real system [1]. For this, a virtual commissioning setup consists of one or more simulation tools and PLCs connected to it. Each PLC can be emulated in separate PLC simulation tool or consist of a hardware PLC connected to the simulation system. For special system components, like industrial robots, the integration of further controllers may be necessary to ensure realistic behavior.[2]

The integration of humans into such simulation systems is still a research focus. This is mainly due to the inherent non-determinism of human behavior, that makes a comprehensive simulation challenging. However, as human interaction with (partially) automated systems is common, it is also a vital part of virtual commissioning.

Especially in the field of human robot collaboration (HRC), the simulation of humans has been extensively researched. Some approaches use ergonomics simulation tools to offline program the behavior of a digital human model (DHM) [3, 4]. While this method can be used for workstation design and ergonomics improvements, real interaction is hardly implementable.

Other approaches focus on a live integration, using different forms of motion capturing systems (MCS) to map a real human's behavior onto a DHM. De Giorgio et al. [5], Kim et al. [6] and Matsas et al. [7] each implement robot models into a Unity® based environment which is used for VR visualization. These environments are designed as standalone applications for each system to be simulated. Also, system and robot behavior are implemented through scripts in the simulation.

Dahl et al. [8] and Metzner et al. [9] implement VR directly into a virtual commissioning tool for robotic applications, Tecnomatix Process Simulate®, to enable live human integration. Motion capturing is implemented using gaming controllers or fullbody tracking. The simulation tool itself does not support physics or general mechatronics simulation.

Metzner et al. furthermore present an approach to directly map the operator's hands, as the most relevant channel of interaction, into a physics-integrated mechatronics simulation tool [10]. In this approach, however, no VR technology is implemented, decreasing intuitiveness of interaction. Allmacher et al. use a similar approach, implementing the user including his hands into a Unity®-based VR simulation of an automated guided vehicle [11].

#### **2.2 Research Gap and Contribution**

In the following section, we derive evaluation criteria and benchmark the described approaches to derive a research gap.

Current research in the field of human integration to VC systems mostly focuses on realistic first person visualization of the simulated system. These systems are however mostly standalone applications developed in gaming engines [5–7, 11]. This not only greatly decreases re-usability of components and functional logic; it also increases implementation efforts and prevents realistic system behavior. As actual logic and motion controllers, like PLCs and robot controllers, are mostly only imitated through customized behavior programming in the gaming application, these systems also do not fulfil the requirements for virtual commissioning.

Furthermore, the sole visualization of the system is not sufficient to enable realistic interaction. The realistic mapping of the operator's actions in the simulation is also vital. Some contributions integrate physical gaming controllers from the VR system to the virtual scene, allowing for an abstract interaction [6, 8]. Other contributions realize the mapping of the operator's hands through motion capturing systems, finding that this increases intuitiveness of interaction [10, 11]. These contributions however either base on gaming setups or lack VR visualization.

The research gap is thus the live integration of a human operator, mainly his hands, into a realistic VR visualization of a simulation controlled by at least one PLC through motion capturing. To allow interaction similar as with the real system, grasping and other physical interaction with parts is necessary. Furthermore, the pressing of buttons and the interaction with HMI panels is also important. As virtual commissioning is used to test real control logic, the integration of PLCs and, optionally, robot controllers and other specialized controllers is necessary. This is complemented by the requirement that device behavior in the simulation is realistic and that the control algorithms can thus be transferred to the real system. To allow scalable use for different systems, a compatibility with standard CAD systems is necessary to avoid redundant modeling efforts. As displayed in Table 1, no current system fulfills these requirements. We thus present a system that aims to combine the approaches presented in [5–7] with interaction capabilities from [10] while maintaining CAD/PLM compatibility.


**Table 1.** Evaluation of human-integrated VC approaches

#### **3 System Concept**

The system is based on the real-time mapping of a simulation environment to a VR scene while also translating inputs and interaction from the user back into the simulation or PLC. Thus, the user is tracked through a motion capturing system during the simulation. We consider the hands as the predominant interaction channel from human to machine. To allow realistic physical interaction, a simulation of physical behavior, especially rigid body physics, such as gravity and collisions is required. As human-machine interaction is often not only done through physical manipulation of parts, buttons and switches but via HMI panels, a corresponding representation in VR is also necessary. To ensure PLM/VC compatibility, the industrial simulation environment is the center component. Physics simulation and interaction capabilities available in the VR engine are not used, as these effects would only affect the visualization rather than the simulation. Hence, the simulation environment is the single source of truth considering system states as well as object and hands placement.

### **4 Implementation and Results**

#### **4.1 System Setup**

The implementation integrates a Leap Motion Controller® (LM) MCS mounted on an HTC Vive® HMD. The LM uses an active infrared stereo camera for hand recognition and tracking. The HMD is tracked via infrared laser-emitting beacons, which trigger sensors on the HMD and allow pose estimation. For fast movements, an inertial measurement unit is also integrated. The simulation is modelled in the Mechatronics Concept Designer® (MCD), a VC tool in the industrial CAD system Siemens NX®, which also features physics simulation capabilities, capable of covering many mechatronics and automation-related simulation tasks [12–14]. The VR rendering of the scene is performed in the Unity® Engine. It is hence open for combination with different simulation tools or frameworks such as ROS. PLCSIM Advanced® is used to emulate a SIMATIC S7® PLC, which is connected to an emulated WinCC HMI in TIA Portal®.

Two computers are used for the evaluation: a standard CAD laptop workstation (Workstation 1) running the mechatronics simulation and PLC emulation, and a VRready workstation (Workstation 2) running the VR environment and processing tracking data (Fig. 1). This setup enables the reusability of the concept for multiple simulation experts without the need of individual VR-ready computers, since any computer capable of running the VC can be combined with a VR-ready workstation.

**Fig. 1.** Hardware experiment setup including the virtual WinCC HMI (1), MCD simulation environment (2), VR visualization (3) and a HTC Vive® HMD with mounted Leap Motion® Controller (4)

The position data of each rigid body in MCD is streamed to the VR scene in Workstation 2 via UDP. The tracking data of the HMD is streamed to Workstation 1 via the same protocol, where a virtual LM sensor is placed accordingly in the simulation. This function is necessary to compensate the kinematic dependency between the HMD and the MCS. The hand tracking data, transferred via a TCP Web Socket connection, is then used to place a virtual hand skeleton model, described in [10], relative to the virtual sensor. The hand links are modelled as collision bodies, allowing for physical interaction with other simulation components. Physical and gesture-based gripping functionalities, described in [10], are implemented. As the skeletal visualization of the hand has proven uncanny to some users in VR, an overlay of a more humanoid hand model in the VR environment is implemented, see Fig. 2.

**Fig. 2.** Simulation (a) and VR environment (b); Virtual hands (1) in skeletal form (left) and with VR overlay (right), gripping object (2) and virtual LM sensor (3)

The HMI is streamed into the VR environment through a virtual network connection (VNC) as an animation when the user turns the left palm towards his face (Fig. 3). The hand tracking data of the right index finger is then used to project a pointer into the WinCC environment, allowing for interaction with the virtual touch display.

**Fig. 3.** Tracking and visualization of the user's hands; a) skeletal representation in MCD and virtual LM sensor, b) combined visualization in VR, c) HMI touch functionality through VNC

#### **4.2 Evaluation Use-Case**

The use-case for the evaluation is a combined assembly and screw-driving cell. Two components are transported via separate conveyors towards an articulated robot arm. Mechanical baffles ensure a correct positioning of the components. The robot arm assembles the components at a processing position. Via conveyor, the assembly is transported to an automated pick-and-place screw driver, mounted to a portal axis, for final processing. The finished assembly is then transported out of the station on the third conveyor, see Fig. 4. The station is controlled through a PLC, monitoring several laser sensors in the station as well as equipment status signals. The entire station, including sensor and actuators, is modelled in MCD. An HMI is used to set the status of the system and the robot separately, as well as monitor signals and performance. The machine setup was not designed for this use case. An already existing simulation model is used for the validation of our system to prove re-usability of existing models. We choose this setup as it unites common automation components, including conveyors, linear axes, automated tools, an articulated robot arm, fixtures and feeders, and thus highlights the universality of the approach.

**Fig. 4.** Evaluation use-case overview: a) WinCC® HMI; b) PLC emulation software PLCSIM Advanced®; c) Simulation environment featuring a lower casing supply (1), an upper casing supply (2), an articulated robot arm (3) for handling and an automatic screw driving unit (4)

#### **4.3 User Study**

The intuitiveness and functionality of the system is tested through a user study. Our system is benchmarked to a mouse-keyboard (KM) setting. The task of the operator is to activate the station and the robot through the HMI panel. Then, new parts to be processed have to be placed on the two inbound conveyors in specific orientation. The parts are located on pillars next to the station. The goal of the simulation is to determine, whether the designed mechanical baffles enable a robust re-orientation of the parts to allow robotic handling. To this end, an automated generation of part instances in the simulation cannot fully satisfy the requirement to map the operators influence on the system. Furthermore, the logic control functionality as well as robot programs are also to be verified through VC. As the station consists of multiple sensors, different actuator types including a robot arm, and mechanical baffles, it represents many of the aspects commonly modelled in VC.

Intuitiveness is rated on an ascending scale from 1 to 5. A total of 30 participants took part in the study. 12 participants had no previous experience with the simulation environment, 27 had no experience with the VR environment and none had used a Leap Motion® controller before. The results of the study are displayed in Fig. 5. The KM setup is rated between 1 and 4 with a mean score of 2, while the proposed VR approach is rated between 3 and 5 with a mean score of 4.5.

It is hence concluded that the proposed concept outperforms the common KM setup in terms of intuitiveness. Furthermore, even though 12 participants had no experience with the simulation tool itself, high ratings are prevalent. This hints at a possible use of the system for training purposes, as it is seemingly well usable also for users with no previous simulation experience. Even for experienced users of the simulation tools with no previous VR experience, the system still achieves better ratings.

**Fig. 5.** Frequency of intuitiveness ratings in the user study comparing the common KM setup with the proposed VR-integrated approach

#### **5 Conclusion and Outlook**

This contribution describes a method to integrate a human operator into a virtual commissioning system at real time. Using VR and motion capturing, an intuitive interaction with the virtual system is enabled. An implementation of the concept for a robot-based production system based on industrial simulation software is shown. The setup is compared to a KM setup in a user study, indicating a significantly higher usability and hinting at new potentials for application in operator training.

In future research, the integration of a full body tracking system will be examined. Furthermore, a field study to assess the potentials of using this system for operator training should be conducted. As the VR engine used in our setup is also compatible with established augmented reality devices, further use-cases basing on this technology, like real-time control of a physical system through manual interaction with a digital representation, can be explored.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Accuracy of Marker Tracking on an Optical See-Through Head Mounted Display**

Michael Brand1,a[0000-0003-3145-1704], Lukas Antonio Wulff1,b[0000-0002-4266-7060], Yogi Hamdani1,c and Thorsten Schüppstuhl1,d[0000-0002-9616-3976]

<sup>1</sup>Hamburg University of Technology, Institute of Aircraft Production Technology Denickestraße 17, 21073 Hamburg, Germany Email: {<sup>a</sup>michael.brand, <sup>b</sup> lukas.wulff, <sup>c</sup>yogi.hamdani, <sup>d</sup> schueppstuhl}@tuhh.de

**Abstract.** This work assesses the accuracy of mono and stereo vision-based marker tracking on the Microsoft HoloLens as a representative of current generation AR devices. This is motivated by the need to employ object tracking in industrial AR applications. We integrate mono and stereo vision-based marker tracking with the HoloLens. A calibration procedure is developed that allows users to assess the accuracy of the calibration alignments by walking around the virtual calibration target. This can be generally applied when calibrating additional tracking systems with ready-made AR systems. Finally, the accuracy that can be achieved with the developed system is evaluated (comparing the influence of different parameters).

**Keywords:** AR, industrial applications, OST-HMD, HoloLens, marker tracking, stereo vision, calibration, accuracy.

## **1 Introduction**

Industrial applications of Augmented Reality (AR) have several domain-specific requirements such as the accuracy of tracking, robustness, safety, ergonomics and ease of use. One especially important requirement is the accurate tracking of real objects in a workplace, for example, to place virtual work instructions on a workpiece. Possible applications include assembly, training, but also more complex applications like ARassisted programming of industrial robots.

The inside-out tracking methods of current generation AR devices such as the Microsoft HoloLens or the Magic Leap One allow to accurately place virtual objects relative to the world. When the user moves around, the objects stay in their assigned position with little jitter and high accuracy [1]. However, the inside-out tracking in and of itself doesn't allow application developers to place virtual objects relative to real objects in the scene. In order to achieve that, dedicated object tracking is required. It can be accomplished by choosing from different AR tracking techniques available: sensorbased (magnetic, acoustic, inertial, optical and mechanical), vision-based (featurebased or model-based) and hybrid techniques based on sensor fusion [2].

All of these techniques have their strengths and weaknesses. Vision-based tracking requires no additional instrumentation, since the aforementioned AR devices already have cameras integrated into them. This allows for very low-cost solutions that can offer high accuracy. Its disadvantages are the limited computing power of the AR device, limited robustness, the reliance on the line of sight and lighting conditions. [2] Fiducial marker tracking is a simple variant of vision-based tracking that offers relatively high robustness at the cost of the effort to place the fiducials accurately. It is readily available through commercial and open-source software. Microsoft endorses the use of Vuforia<sup>1</sup> with the HoloLens. Open-source solutions such as ArUco<sup>2</sup> and AR-Toolkit<sup>3</sup> are alternatives that can be tailored to the application's needs more freely.

The goal of this paper is to assess the accuracy of mono and stereo vision-based marker tracking on the HoloLens as a representative of current generation AR hardware. Our contributions include:


In Chapter 2 the theoretical background and related research is presented. Following, in Chapter 3 the methods that are used to conduct the assessment of the accuracy are described. In Chapter 4 the results are presented and discussed. Finally, in Chapter 5 a conclusion is drawn and an outlook on future work is presented.

#### **2 Related Work**

Many prototypes of applications based on the Microsoft HoloLens employ either a) manual registration by the user [3] or b) marker tracking [4–6]. Option a) offers the advantage of being easy to implement, and can produce accurate registration. Option b) offers the advantage of not requiring user interaction and can dynamically track multiple objects.

Marker tracking uses computer vision algorithms to estimate the pose of a marker based on camera images. It can be abstracted to the Perspective--Point (PP) problem which describes the task of estimating the pose of a set of points (with = 4 for square markers, each corner of the marker representing one point). In a first step, the corner points have to be extracted from the raw camera image. In a second step the pose estimation is calculated. [7] There are different algorithms available that solve the PP problem [8]. ARToolkit 5 uses the Iterative Closest Point algorithm for mono camera marker tracking and Bundle Adjustment for stereo camera marker tracking. ArUco uses

<sup>1</sup> Vuforia: https://developer.vuforia.com/

<sup>2</sup> ArUco: https://sourceforge.net/projects/aruco/files/

<sup>3</sup> ARToolkit 5.4 snapshot, project discontinued: https://github.com/artoolkit/ARToolKit5

homography estimation followed by an iterative Levenberg-Marquardt optimization. There are more advanced algorithms for the special case of 4 coplanar points that can avoid local optima [9]. This is implemented by ARToolkitPlus<sup>4</sup> . There are several studies which characterize the accuracy of different marker tracking solutions [7, 10–12].

AR systems generally need to be calibrated at least once in order to achieve registration of real and virtual content. Calibration is aimed at eliminating systematic errors as well as calculating the correct viewing parameters. There are calibration procedures for the different subsystems, e.g. camera calibration, and the system as a whole needs to be calibrated as well. For Optical See-Through Head Mounted Displays (OST-HMDs) there are manual, semi-automatic, and automatic procedures [13]. Manual procedures such as the classical SPAAM [14], Stereo-SPAAM [15] and SPAAM2 [16] are based on the user creating so called correspondences from manual alignment of real and virtual content. Based on multiple of those correspondences, the viewing parameters are then calculated.

Most of those classical calibration procedures assume that the AR display is uncalibrated, however, with modern devices such as the HoloLens the system as a whole comes pre-calibrated (except for a quick IPD calibration). Qian et al. [17, 18] use this fact as a motivation to present a new manual calibration approach. A linear calibration model that maps from some 3D-tracker space (which can be any tracking system, external or head-mounted) to the 3D-virtual space (the virtual 3D-space created by the display) is proposed. It is calculated from 3D-3D-correspondences created by the user. One of two scenarios being evaluated uses marker tracking based on the RGB camera of the HoloLens. The results indicate a somewhat limited accuracy in the depth axis of the marker tracking camera, when compared to the other two axes. This depth accuracy could be improved by employing stereo vision because it yields a higher information content. Besides, the calibration relies on the depth perception of the display. Our own preliminary experiments have shown that the depth perception of the display can be quite misleading. Furthermore, the working space is confined to arm's length which doesn't fit the requirements for object tracking in most industrial applications. Therefore, we modify the presented calibration procedure, as we believe it is generally well suited for modern AR devices where systems come pre-calibrated.

#### **3 Method**

The notion of accuracy can be divided into two aspects, 1) trueness and 2) precision. In the scope of this work, the focus will lie on the trueness. Striving for high trueness (i.e. small systematic error) is what makes a calibration procedure essential. The precision of a tracking system can be improved by employing filtering techniques such as averaging or Kalman filtering, which isn't considered in this work.

<sup>4</sup> ARToolkitPlus, project discontinued: https://github.com/paroj/artoolkitplus

First, the concepts for a) the integration of the marker tracking into the HoloLens, b) the calibration procedure and c) the evaluation procedure, need to be outlined. The concepts need to take the industrial application's requirements as well as the properties of the used AR system into account.

#### **3.1 Marker Tracking Integration**

The requirements for object tracking in an industrial application are chosen as follows:


We propose a method where, first, the marker tracking determines the pose of an object relative to the camera of the HoloLens. It is then combined with the information from the built-in inside-out tracking, so that the marker pose with respect to the device's world coordinate frame can be computed. A virtual object could then be placed relative to the marker in the world coordinate frame, using the inside-out tracking. This is repeated in small time intervals, so an up-to-date pose of the object is always available. This method offers the advantage that even when losing the line of sight, the procedure can still track the pose of the marker where it was last seen, which addresses one of the core requirements. The disadvantage of this approach is the reliance on the inside-out tracking. Furthermore, only the last known position can be tracked if the line of sight is lost. If the marker moves in the meantime the tracking can be severely inaccurate. We propose to deal with this per use-case, e.g. handling the loss of the line of sight, or employing a completely different tracking method.

The marker tracking is implemented using ARToolkit 5.3 and Unity 2017.1 game engine as a framework. The tracking target is a 3D cube with markers of 10 cm width on each face (see **Fig. 1, right**). For mono vision the RGB camera (resolution of 896x504) in the center of the front of the HoloLens, for stereo vision the central two of the four environment understanding cameras (resolution of 480x640 each, only available in "research mode") of the device are used.

**Fig. 1.** Left: Working space, right: Marker cube

#### **3.2 Calibration procedure**

In the calibration procedure the user needs to create a set of correspondences by aligning a virtual cube with a marker cube manually, following a predefined protocol. It bases on the procedure described by Qian et al. [17]. We give a brief summary of their procedure first, and then explain the modifications we developed.

In Qian's procedure a calibration model that maps from some 3D-tracker space to the 3D-virtual space (the virtual 3D-space relative to the display) is calculated based on 3D-3D-correspondences collected by the user. A linear transformation ∈ ℝ44 is assumed as a calibration model. It maps points <sup>1</sup> , … , ∈ ℝ<sup>3</sup> from the tracker space to points <sup>1</sup> , … , ∈ ℝ<sup>3</sup> from the virtual space:

$$p\_l = T \cdot q\_l \tag{l}$$

Three different models with varying degrees of freedom (dof) are used, namely isometric (6 dof), affine (12 dof), perspective (15 dof).

The correspondences are collected as follows: The points are several points that are fixed with relation to the display space. The points are the corresponding measurements from the tracking system, where the tracking target is a hand-held cube on a stick. To collect correspondences the user is required to coordinate hand and head movements in order to align on the virtual cube with on the marker cube, which is repeated multiple times.

We modify this procedure so that the points are gathered based on a virtual cube that can be positioned freely in the device's world coordinate frame by the user via keyboard input. The marker cube is placed on a table. This makes use of the system's ready-made property and allows users to walk around the virtual calibration target freely while adjusting it to exactly align with the marker cube (see **Fig. 2, left**). Once the two cubes are aligned the user can signal the system to collect a correspondence. The correspondences are collected at locations that are combinations of different angles ( = −45°, 0°, +45°) and distances ( = 60 , 90 , 120 , 150 ) between the HMD and the marker, 12 in total, so as to have a good coverage of the chosen working space (see **Fig. 2, right**). Additionally, for each correspondence 12 individual measurements by the tracking system are carried out and averaged.

This modification has the advantage of allowing for a larger working space beyond arm's length and being able to assess the proper alignment of the cubes more easily, since it doesn't rely on the depth perception of the display which can be very misleading from our experience. Furthermore, this eliminates the need for head-arm coordination which improves ergonomics and accuracy.

In order to calculate the calibration model we minimize the sum of squares of the absolute value of the residual errors over all correspondences ( , ):

$$E = \Sigma\_{l=1}^{n} \|p\_l - T \cdot q\_l\|\_2^2 \tag{2}$$

**Fig. 2.** Left: Aligning <sup>1</sup> with <sup>1</sup> , right: Calibration Setup

#### **3.3 Evaluation Concept**

For the evaluation we adopt Qian's [17] calibrate-and-test procedure. The quality of the computed model will be assessed by collecting a new set of another 12 correspondences and testing how the previously calculated model fits each data point of the test data set. This is done by calculating the residual error :

$$e = p\_l - T \cdot q\_l \tag{3}$$

This residual error is used as a metric for the trueness post-calibration. The new correspondences are collected in locations that are combinations of different angles ( = −45°, 0°, +45°) and distances ( = 75 , 105 , 135 , 165 ). Note that the distances differ from those from the calibration phase, so as to assess if the calibration model works as well in different locations.

It needs to be noted that if the display space possessed large errors when compared to the real world, the calculated residual error would be misleading, since it is expressed in relation to the display space and not the real world. It is assumed that the error of the display space is small, since the HoloLens is a pre-calibrated AR system.

The advantage of the chosen evaluation approach bases on the fact that manual alignments can closely represent an ideal alignment as seen by the user. If the user cannot distinguish the position of a real object and a superimposed virtual object, he has an optimal experience.

Finally, the results obtained with mono vision are compared with those obtained with stereo vision.

### **4 Evaluation**

#### **4.1 Results**

For the evaluation, the calibration procedure is conducted and subsequently evaluated as outlined in the evaluation concept. These two steps are repeated three times. That way it is assured that the calibration procedure yields reproducible results. The parameters that are evaluated are chosen as follows:


For each configuration (e.g. "z, Stereo, Affine") the trueness over all the 12 correspondences (distributed across the working space) and all three runs (i.e. 36 data points per configuration in total) are averaged.

**Fig. 3** shows boxplots for the absolute residual errors of each configuration. With mono vision the absolute residual is (12.0 ± 6.7 mm) with the isometric model, (3.5 ± 2.0 mm) with the affine model and (3.6 ± 1.8 mm) with the perspective model. With stereo vision the absolute residual error that can be achieved is (40.3 ±34.2 mm) with the isometric model, (20.2 ± 17.4 mm) with the affine model and (19.7 ±13.9 mm) with the perspective model.

**Fig. 3.** Boxplots for the *absolute value* of the residual errors

**Table 1** and **Table 2** show a more detailed overview of the trueness that stereo vision and mono vision can achieve with the setup developed in this work.


**Table 1.** Residual error (mean and standard deviation) by model and axis (mono vision)


**Table 2.** Residual error (mean and standard deviation) by model and axis (stereo vision)

#### **4.2 Discussion**

First, it has to be discussed how the values for the mean and standard deviation can be interpreted. As stated above, 36 data points are combined for each configuration. The mean gives a summary of the overall trueness of each configuration across all locations. The standard deviation has to be interpreted with some care. A varying trueness between the different locations would contribute to the standard deviation, as well as a fluctuation in the three runs. However, these two factors can't be differentiated with the data presented.

Bearing that in mind, the results indicate that mono vision yields clearly superior trueness when compared to stereo vision. This is attributed to the fact that the mono camera has got a higher resolution than the two stereo cameras individually, which plays a big role especially at large distances between the HMD and the marker that were included in the experiments. Furthermore, a stereo vision system is geometrically more complex so it apparently can't be calibrated sufficiently with a linear model, let alone an isometric one. This has to be investigated further.

It is concluded that the presented calibration procedure yields good results for simpler systems like the mono vision system. It is notable that the trueness in z-direction is good, which we attribute to the design of the calibration procedure that is not reliant on the depth perception.

It is observed that the affine and perspective models yield better results than the isometric model for both mono and stereo vision. All in all, there is no apparent difference between affine and perspective, which is why we prefer to use the affine model with less degrees of freedom in a future application.

With mono vision the mean trueness of the absolute value is at 3.5 mm with the affine model which satisfies our requirement for a high accuracy. However, we need to take a closer look at the trueness in different locations in the future, since the standard deviation of the trueness indicates fluctuation among the different locations and runs. The residual errors are generally thought to be due to non-linear errors that cannot be compensated by a linear model.

#### **5 Conclusion**

We conclude that the proposed integration concept and calibration procedure are feasible for object tracking in industrial applications. The accuracy that could be achieved by mono vision is sufficient for many potential applications.

However, stereo vision-based marker tracking on the HoloLens yields no improvement over using mono vision in the working space that was examined.

We have achieved 1) the integration of mono and stereo vision-based marker tracking with the HoloLens, 2) the development of a new manual calibration procedure that allows users to assess the accuracy of the calibration alignments by walking around the virtual calibration target which can be applied for ready-made AR systems like the HoloLens and 3) an assessment of the accuracy of both mono and stereo vision-based marker tracking.

In the future we want to further examine and improve the accuracy of mono and stereo-based marker tracking with the HoloLens by fine-tuning different parameters and analyzing the data we collected in more depth. Furthermore, we want to explore how marker tracking can be integrated into prototype applications.

#### **Acknowledgement.**

This work was created as part of the research project "MiReP" and is supported by the Federal Ministry for Economic Affairs and Energy as part of the Federal Aeronautical Research Programme "LuFo V-3".

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## User Guidance and Automatic Completion for Generating Planar B-Rep Models

Dorian Rohner, Maximilian Sand, and Dominik Henrich

Lehrstuhl f¨ur Angewandte Informatik III (Robotik und Eingebettete Systeme), Universit¨at Bayreuth, D-95440 Bayreuth, Germany dorian.rohner@uni-bayreuth.de

Abstract. A representation of the world as a 3D model is a common necessity in robotics and automation. In previous work, we developed a concept to generate boundary representation (B-Rep) models from multiple point clouds using a hand-held depth-camera and to register them without a prior known pose. During the online reconstruction, properties of the sensor and the system (like noise) lead to small holes in the B-Rep. To prevent tedious post-processing, holes should be closed during the reconstruction. Our goal is to automatically close identified holes. However, not every hole can be closed automatically, as it may be unreasonable. For this case we develop a visual indication for the user, so he can close the hole by recording another depth image. In an experimental validation, we conclude the usefulness of the addition to the system.

Keywords: Reconstruction, Volumetric models, Visualization, CAD

## 1 Introduction

To generate a 3D representation of the surrounding world is a necessary skill needed in robotics, automation, and their applications. For example, it is useful for the reconstruction of a factory layout or of an robot cell [12]. This representation ranges from point clouds to more complex models, e.g. boundary representation (B-Rep) models. In our previous work [3], a method to generate planar B-Reps from a single point cloud was presented. Multiple point clouds can be incorporated in real-time, providing a valid B-Rep model in each time-step. By using the known pose of the camera (e.g. in [3] a robotic manipulator was used) two B-Rep models can be merged. The B-Rep is handled by an half-edge data-structure. The requirement of a known pose can be lifted by utilizing the method from our work [4], by using the angles between three linear independent faces to generate a possible rigid body transformation between two B-Reps.

By combining these approaches, it is possible to generate a B-Rep as representation of the world by using a depth sensor, without a known camera pose. This enables the use of hand-held cameras, which leads to applications like object and robot cell reconstruction. The camera is moved by a user through the scene. Every point cloud is reconstructed into a B-Rep and merged into the world representation. However, the resulting model is not necessarily complete. One possible reason may be occlusion, as a result of a complex scene. A similar aspect are missing points due to absorbed or reflected light. Another problem is that a scene can not be reconstructed from too noisy data. A last possible source for errors is the user, e.g. when he misses parts of the scene. There are two methods for dealing with the problem of incomplete models: On the one hand, sufficient small holes inside the representation should be closed automatically to enable a comfortable usage of the system. On the other hand, it is necessary to direct the user during the recording if the holes can not be closed automatically.

This paper consists of the following parts: The next section presents the state of the art how to handle incomplete representations. We present our approach in Section 3. First, we determine attributes to classify holes and afterwards we suggest a method to automatically close them and to generate user guidance for holes, which can not be closed automatically. The approach is validated in Section 4. In the last section, we discuss our contribution and future work.

#### 2 State of the Art

The most simplest way to give user guidance is a visualization of the model [5]. The model is typically rendered from the user's view. This visualization helps the user detect missing parts. A more advanced method is visual feedback to notify the user of missing parts. This can be done by colouring the backsides of the model, as they are only visible, when a hole exists and the user can look inside the model [1]. Finally, user guidance is possible by suggesting a camera pose, from which the hole can be closed. This problem is known as Next-Best-View in the context of robotics [6, 7]. However, a precise pose is unnecessary as it is impossible for a human to take exactly this pose. Additionally, in our domain of live reconstruction there is no set of possible views, and the human operator may have additional knowledge of the scene. Therefore, it is only necessary to identify holes and give a hint from which direction to look. It is not necessary to give a quality measure for each pose or to propose a reasonable one, as all holes should be closed in the end and the selection can be left to the human. Overall it is at least necessary to give a visualization to ensure a simple and intuitive live reconstruction. This is based on the Hand-Eye-Control System presented in [2], which reasons, that a human reacts in such a way, that the difference between the current state and a goal state is reduced [1].

The other problem mentioned are small holes. For this incompleteness user guidance can be generated, but it is not always helpful, especially when the holes exist because of occlusion or properties of the sensor. Therefore it is necessary to automatically close theses holes. This problem is known as Model Repair [8, 9]: In general, these methods can be split into procedures based on point clouds or on meshes. The mentioned methods are typically based on the surface or on volumetric grids. In this case we will focus on the mesh-based methods, due to the B-Rep as the underlying structure. The surface-based methods follow different approaches. One possible way is to split the hole in smaller, easier to repair ones [16]. Another way is based on a triangulation of the hole, with the goal of

Fig. 1: Examples for holes with different orientation and different cardinality k. The blue half-edges are the hole, the red one the transition-half-edges.

Fig. 2: Visualization of the used variables for an outer hole with k = 3. There are two separated transition-half-edges and one twintransition-half-edge, the hole-halfedges (blue), the transition-half-edges (red), transition-vertices (red-dotted), and the intersection-point of the three face's planes (green).

minimizing the amount of necessary triangles [17]. A last group of methods is based on a implicit surface representation and trying to utilize additional knowledge resp. properties of the object [13–15]. The volumetric approach is based on using a volumetric representation, with the idea of expanding the area around the hole, and therefore closing the model. These methods are usually more reasonable for more complex holes. In contrast to these methods, we are using the B-Rep data structure, therefore a direct application of theses methods is not possible. Methods especially for B-Reps use features like edges and vertices, but are also based on a tessellation [10, 11]. Additionally, our goal is to use the hole closing during the live reconstruction, so there may not be enough information to close the hole with the before mentioned methods.

#### 3 Our Approach

#### 3.1 Definition and Handling of holes

Based on B-Rep models, it is easy to determine incomplete parts within a B-Rep due to the half-edge data-structure by testing, whether every half-edge has a twin-edge. Therefore, a hole is a closed loop of half-edges without twin-edges. A hole can be described by its cardinality k, which is the number of adjacent faces of the hole. Using this definition, the external boundary of the B-Rep is also recognized as a hole. This leads to the definition of the orientation: A hole is called inner hole if it can be closed by enlarging every face, which is adjacent to the hole. Otherwise, the hole is called outer. Examples for holes with different attributes are visible in Figure 1. A last definition are the so called transitionhalf-edges T<sup>i</sup> (red edges in Figure 1). These are twin-edges which start (or end) in a vertex on the boundary of a hole, but are not part of the boundary itself. However, they still have to follow the boundary of the corresponding face. Based on this definition, we call the vertices, which are part of the hole and a transitionedge starts/ends in them, transition-vertices (red-dotted circles in Figure 1).

For further formalisms later on, we define the following variables: The involved faces f<sup>i</sup> with i = 1, ..., k of a hole have the normal n<sup>i</sup> ∈ R 3 . It should be mentioned that one physical face may have multiple identifiers following this definition, due to the fact, that the cardinality may count one face multiple times. The transition-vertex, in which the hole-half-edges switch from face f<sup>l</sup> to f<sup>i</sup> is called t<sup>i</sup> with the position T<sup>i</sup> ∈ R 3 . The direction of the transition-half-edge, which ends in t<sup>i</sup> is called h<sup>i</sup> . Based on this definitions, the vector from a transitionvertex t<sup>i</sup> to the next one t<sup>j</sup> is called vij = T<sup>j</sup> − T<sup>i</sup> . For the special case of k = 3, we call the intersection point of the planes defining the three faces as S ∈ R 3 , and the vector from one transition-vertex t<sup>i</sup> to the intersection point S as s<sup>i</sup> = S − T<sup>i</sup> . These identifiers are visualized in Figure 2.

The state of the art indicates two kinds of user support: the automatic closing of holes and the user guidance. However, only because a hole can be closed automatically, it is not always in the user's interest to do so. Therefore, we distinguish between three kinds of holes a) that can be closed automatically and the user agrees, b) that can be closed automatically and the user disagrees, and c) that can not be closed automatically. In our previous work [3] and [4] a userdefined parameter was introduced, the so called structure size δs. The structure size indicates, which geometric features (considering their size) can be ignored. Therefore, all holes smaller than the structure size can be closed automatically, as they are too small to be relevant. Otherwise, user guidance should be provided. The definition of the size of a hole has to take different shapes of holes into account. As the size of a hole the maximum diameter (k = 1), the maximum distance to the missing edge (k = 2) or the distance to the missing corner (k = 3) can be used. For holes with a cardinality k > 3 or the orientation outer, user guidance is the only possibility, as the hole can not be closed automatically.

#### 3.2 Detection of holes

The following algorithm detects all holes of a B-Rep, starting with a half-edge without a twin-edge. This start edge is added to the set hole. Beginning with start, all following half-edges (considering the direction in a B-Rep) are visited until we reach the start again. If we encounter an edge with an opposite, we do not add this to the set (as this half-edge is a transition-half-edge) and return to the hole in the next step. In the end, the set hole is added to the overall set holes. This whole process is repeated until no half-edge without twin-edge exists, which has not been visited yet. The cardinality is determined by counting the participating faces. The procedure is summarized in Algorithm 1. The function twin(c) denotes the half-edge with the twin relationship to half-edge c. Analogously, the function successor(c) denotes the half-edge after c for this hole.

Based on the now known holes and their cardinality, we calculate the orientation of holes, starting with k = 2. The idea is, to check whether the transition-


half-edge points in the same direction as the vector from one transition-vertex to the next one. However, the transition-half-edges are not necessarily twins. Therefore, we take the normals of the corresponding faces into account:

$$\text{outer hole} \iff \begin{cases} (\mathbf{n}\_1 \times \mathbf{n}\_2) \diamond \mathbf{v}\_{12} < 0 & \text{if } f\_1, f\_2 \text{ convex} \\ (\mathbf{n}\_1 \times \mathbf{n}\_2) \diamond \mathbf{v}\_{12} > 0 & \text{if } f\_1, f\_2 \text{ reflexive} \end{cases} \tag{1}$$

The definition can be easily extended for the case of k = 3:

$$\begin{aligned} \text{outer hole} \iff \begin{cases} (\mathbf{n}\_i \times \mathbf{n}\_j) \diamond \mathbf{s}\_i < 0 & \text{if } f\_i, f\_j \text{ convex} \\ (\mathbf{n}\_i \times \mathbf{n}\_j) \diamond \mathbf{s}\_i > 0 & \text{if } f\_i, f\_j \text{ reflexive} \end{cases} \end{aligned} \tag{2}$$

$$\forall i \in \{1, 2, 3\}, \quad j = (i - 1 + k) \text{ mod } k.$$

This calculation fails, if participating faces are linearly dependent, because no intersection point exists. Due to the definition of an inner hole, that faces meet in one point, it is clear that a hole with linear depending faces is an outer hole. To calculate the orientation we use whether two faces are convex or reflexive. This relationship is only defined for edges (as faces can be both on different edges) and is calculated by edge convex ⇐⇒ (vij × ni) ◦ n<sup>j</sup> > 0. In our case, we want to know the convexity of two faces in one transition-vertex. We define a plane π = n<sup>i</sup> × n<sup>j</sup> −(n<sup>i</sup> × n<sup>j</sup> ) ◦ T<sup>i</sup> T which is orthogonal to both faces and contains the transition-vertex t<sup>i</sup> . We project both faces onto this plane and get two lines gi , g<sup>j</sup> . For these lines there are four possible cases, how the transition-half-edges can lie in the plane π. For two of these the decision of the convexity is obvious. The other two are undefined, as both convex or reflexive is possible. However, an estimation is possible by projecting the hole-half-edges onto π and comparing the overlap a<sup>i</sup> resp. a<sup>j</sup> onto the other side (see Figure 3). Depending on which value is greater, the two faces are convex or reflexive in this transition-vertex.

Fig. 3: The four possible cases how transition half-edges h<sup>i</sup> and h<sup>j</sup> can lie in plane π. The convexity in transitionvertex t<sup>i</sup> (red-dotted) is estimated.

Fig. 4: Closing of inner holes (top) by replacing hole-half-edges (middle) and merging of separated edges (bottom).

#### 3.3 Automatic closing of holes and Generation of User Guidance

Based on the attributes, we close holes automatically which are smaller than the structure size δs. For k = 1 the inner boundary is removed, in which all half-edges of the hole lie. For k = 2, all half-edges of the hole are replaced by one twin-half-edge. If there are already half-edges on this physical edge, they are merged afterwards. For k = 3 a new vertex v<sup>s</sup> is generated, which lies in the intersection of all involved faces. All hole-half-edges are deleted and replaced by pairs of half-edges, with v<sup>s</sup> as one vertex, and each transition-vertex as the other vertex. If the edges are separated, they are merged into one (see Figure 4).

Based on the automatic closing, three combinations of cardinality and orientation exist, where an user guidance during the recording is necessary: Inner holes with k ≤ 3, outer holes with k ≤ 3, and holes with k > 3. This separation is based on the idea, that holes with k ≤ 3 can be closed by taking a look from one proper pose. The user guidance aims on the calculation of an arrow consisting of a position P ∈ R <sup>3</sup> and a direction r ∈ R <sup>3</sup> with ||r|| = 1.

The idea for the method for inner holes with k ≤ 3 is based on the functionality of depth sensors. The easiest way to capture planar surfaces occurs when the angle between the face's normal and the sight ray is as small as possible. Therefore, the direction of the arrow can be calculated as the sum of the inverted normals of all involved faces. The arrow directs towards a point which is calculated either as the centroid P<sup>s</sup> of the hole or as the weighted sum of the transition-vertices:

$$\mathbf{r} = \sum\_{i=1..k} -\mathbf{n}\_i / \left| \left| \sum\_{i=1..k} \mathbf{n}\_i \right| \right|\_2, \mathbf{P} = \begin{cases} \mathbf{P}\_s - d \cdot \mathbf{r} & k = 1\\ \frac{1}{2} (\mathbf{T}\_1 + \mathbf{T}\_2) - d \cdot \mathbf{r} & k = 2\\ \frac{1}{3} (\mathbf{T}\_1 + \mathbf{T}\_2 + \mathbf{T}\_3) - d \cdot \mathbf{r} & k = 3 \end{cases} \tag{3}$$

The parameter d ∈ R corresponds to a distance from the hole in the direction of −r which can be adjusted by the scaling of the B-Rep and the user's preferences. For the case of outer holes with k > 3, the missing patches lie on the other side compared to inner holes, therefore it is reasonable to use the inverted direction. However, when a hole is the inner boundary of a surface, the direction may still not fit, because the arrow is on the wrong side of the face. This can be fixed by checking, whether the hole is an inner or an outer boundary of this face:

$$\mathbf{r} = \sum\_{i=1..k} a\_i \cdot \mathbf{n}\_i / \left\| \sum\_{i=1..k} \mathbf{n}\_i \right\|\_2 \quad \text{with} \quad a\_i = \begin{cases} +1 & \text{inner boundary} \\ -1 & \text{outer boundary} \end{cases} \tag{4}$$

The closing of holes with k > 3 is ambiguous. The calculation of the orientation fails, because no intersection point is available. The idea is to determine triangles, which would close the hole and calculate a direction, from which most of them are visible. The first step is to calculate the mean M of the transition-vertices as M = 1 k P i=1..k Ti , which can be used as the position. Together with the transitionvertices T<sup>i</sup> , this point is used to triangulate the hole and calculate the normal ∆<sup>i</sup> = (M − Ti) × vij with j = (i + 1) mod k for each triangle. These normals are not normalized, so they can be weighted by the area of each triangle:

$$\mathbf{r} = -\sum\_{i} \Delta\_{i} / \left\| \sum\_{i=1..k} \Delta\_{i} \right\|\_{2} \tag{5}$$

#### 4 Validation

The generation and merging of B-Reps [3, 4] was implemented and extended by the developed methods. It can be used as an application with common depth sensors (Microsoft Kinect) to reconstruct objects by moving the hand-held sensor through the scene. Also, we implemented an Android application which can be used on devices with a depth sensor (Lenovo Phab 2 Pro) as a proof of concept. One possible validation technique is the complete exploration of the problem space. For this, we limit the cardinality to k ≤ 3 and determine all possible cases considering convexity of faces, orientation, and for outer holes whether the hole is an outer or inner boundary. For each possible case, a synthetic test case is generated and the calculated user guidance evaluated. The decision, whether the user guidance is reasonable, is an individual decision. The cardinality is fixed with k ≤ 3 due to the fact that the number of possible cases increase exponentially in k and no automatic closing is possible with higher values for k. Another method is a validation based on real data. A first step is to evaluate the automatic closing of holes. In this case, the reconstruction is done by hand without any support. This result can be compared to a reconstruction which uses the automatic closing of holes and the same input. A second method is to analyse the amount of user guidance on real world data. This experiment is based on our previous work [12], in which a robot cell is reconstructed in an offline step.

Fig. 5: Inner holes with cardinality k = 3 for all four possible cases considering the convexity of the three edges. Top: User guidance. Bottom: The model after automatic closing. The direction of the face normal is indicated by a black line.

This reconstruction was used for a real-time path planning. The developed user guidance and hole closing should help in generating a complete model.

Based on this setup for the validation we obtain the following results: As a proof of concept we reconstruct complete objects. The user starts in an arbitrary place and surrounds the object. After each merged segment the user guidance is calculated, and the operator may take another image. After one round the model is completed to a high degree, with some indications left (see Figure 6). Considering the complete evaluation we generated synthetic models for every possible case: If the half edges of the hole lie in more than one inner boundary, the resulting user guidance is not very helpful due to a poor direction. The same problem occurs with models in which one face is twisted around another. However, these two cases are not highly relevant as exactly this situation with no other faces changing the cardinality must arise. Additionally, holes which lie in more than one inner boundary are rare in reality. The results for inner holes with k = 3 and different convexity can be seen in Figure 5. The result of the comparison between real world data with and without the automatic closing is presented in Figure 7. The difference between these two reconstructions is small, but several holes are closed in a reasonable way. Due to the fact, that a edge can only be merged when it exists in both input B-Reps (due to the method from [3]), most of the holes exist on edges and corners. If the corresponding faces are well reconstructed, the hole is barely visible. The scene in Figure 6 visualizes the user guidance. The amount of arrows increases in the beginning and reaches the maximum number when the object is circled for the first time. When the user actively closes these holes, the number of arrows decreases until the object is completely reconstructed and no arrows remain.

Fig. 6: Test-scene and the reconstructed model with a Lenovo Phab 2 Pro. During the recording, the user guidance was calculated and displayed.

Fig. 7: A testscene (c) was reconstructed with a Microsoft Kinect for XBOX. Model (a) is the result without automatic closing. In model (b) sufficient small holes were closed automatically. In detail (d+e) several holes were closed.

#### 5 Conclusion

We presented a new approach to close holes automatically and to generate user guidance during recording. We found that different types of holes can exist, which need different methods depending on the attributes of the hole. A method to determine all holes of a B-Rep and their attributes was presented. Based on this, the holes are closed automatically or user guidance was generated. We evaluated our approach with a complete exploration of a problem space and by comparing the results of a real world scene with and without automatic closing of holes. Future work may include a collision test for the arrows to prevent them from being stuck inside the model. Additionally, a user study could verify the usefulness of the user guidance and the decrease in recording time.

Acknowledgements This work has partly been supported by the Deutsche Forschungsgemeinschaft (DFG) under grant agreement He2696/16 HandCAD.

#### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Development of a Shared Environment Model with Dynamic Trajectory Representation for Multiple Mobile Robots

Sebastian Q. Roder<sup>1</sup> , Florian Rothmeyer<sup>1</sup>,<sup>2</sup> , Bernd Spiegelberger<sup>2</sup> , and Gunther Reinhart<sup>1</sup>

> 1 Institute for Machine Tools and Industrial Management, Technical University of Munich, Munich, Germany Sebastian.Roder@iwb.tum.de 2 ITQ GmbH, Parkring 4, DE-85748 Garching, Germany

Abstract. The productivity of groups can be increased by enabling group members to share their perceptions of the environment. We adapt this concept for mobile robots by presenting an object-oriented approach to a shared environmental model. The objects are stored in a graph, which saves memory and computing power and allows the representation of hierarchical and topological relationships. Each object can contain geometric and semantic data as well as information about its current, past, and planned or estimated future movements. An example application shows that modeling future motion can prevent collisions.

Keywords: world models, distributed systems, dynamic objects, collaborative robots

## 1 Motivation

Product customization and shortening of production life cycles increase the demand for flexible manufacturing systems. Mobile robots show great potential due to their high configurability and scalability as well as their freedom of movement. The applications are diverse and range from logistics and cleaning or inspecting to operation at multiple workstations. The large number of mobile resources means that future production facilities will require decentralized intelligence to handle the increasing complexity.

In order to navigate and interact safely and efficiently in such facilities, robots need an understanding of their environment that is technically enabled by environmental models. The combination of the limited perceptions of individual robots into a common knowledge base enables collective intelligence. Those shared models enable improvements firstly in safety, e.g. warning of imminent danger, and secondly in efficiency, e.g. avoiding traffic jams.

Existing models lack the ability to represent object dynamics. At critical points in a production facility, such as intersections and pedestrian crossings, information about the dynamics of surrounding objects can improve overall performance and safety. Dynamic information could either be directly integrated into the map by the performing robot or estimated by observers for agents with no connection to the shared model, e.g. human workers.

This work presents a novel approach for integrating object dynamics into world models. The suggested solution consists of a client-server architecture and a graph-based data model supporting geometry, semantic classification, topology and object dynamics, including past and present as well as predicted and planned future trajectories. First, an overview of environment models with focus on papers related to this work is given in section 2. Then, our approach is presented in section 3. Section 4 describes a proof-of-concept application offering a technical implementation and illustrating the added value of the approach. Finally, the results are discussed in section 5 and perspectives for future research are given.

#### 2 Related Work

Research on shared environmental models focuses on three main areas: methods for obtaining new data and integrating them into an environmental model, solutions for contributing and distributing data in multi-user scenarios and lastly, the actual data modeling, which uses data structures, itself.

Obtaining New Data This area includes the various methods from the fields of perception, data fusion and pattern recognition and is considered only briefly below. Standard approaches to mapping in mobile robotics are "Simultaneous Localization and Mapping" (SLAM) [1, 2] or "HectorSLAM" [3], which generate grid-based maps. Alternatively, the raw sensor data can also be processed by object recognition and integrated into an object-oriented data model [4]. There are several applications in the field of service robotics [5, 6]. Furthermore, data can be processed to form topological graphs, e.g. transforming a 2D grid-based map of a building into a graph with nodes representing rooms [5, 7]. Applications to path planning for multiple mobile robots are given in [8]. Due to inaccuracy in perception, objects need to be registered to their absolute position in the environmental model, referred to as "anchoring" or "data association" [9, 10]. Context-based classification improves anchoring by considering existing knowledge [11].

Data Distribution This area integrates the methods from the fields of network architecture, consistency management and synchronization and, again, is considered only briefly below. In contrast to fully centralized map storage and fully synchronized maps with all participants, intermediate stages are a trade-off between latency and consistency [12]. A client-server structure can be applied to an object-oriented distribution model, preserving consistency by predefined access rights to object changes [13]. Multi-server architectures with environmental subsets enable scalability [14]. Moreover, there are approaches that are tolerant to inconsistency by assuming it to be acceptable at certain points of the data model [15].

47

Data Structures for Environment Mapping Our approach focuses on data structures, therefore the related work is discussed in detail. Common approaches for geometric relationships can be classified as grid-based or object-oriented. In addition to geometric data, environmental models can represent semantics or uncertainty.

Grid-based Structures A fundamental concept of map generation is the "Occupancy Grid Map" (OGM), discretizing 2D or 3D space into independent cells assigned with the estimated occupancy, e.g. "free", "occupied" or "unknown" [1, 3, 5, 16]. The storage effort can be reduced by structuring the grid cells in a hierarchical manner, the so-called "octree" [17]. Combinations of several octrees enable object hierarchies and multiple resolutions [18].

Object-oriented Structures With object trees, not only cubes, but any geometric object can be described. Gregor et al. [19] use homogeneous transformations to indicate the relative spatial arrangement of parent and child objects. Semantic information can be represented by linking the nodes of the object tree to corresponding elements of a "conceptual hierarchy" [5].

An extension to graph structures holding geometric and transformation nodes is presented by Blumenthal et al. [20]. Here, nodes can contain semantic information via key-value pairs and past object movements are traced by transformationtimestamp pairs. The graph structure allows several transformation nodes to point to the same geometry node, which enables the modeling of competing perceptions in multi-robot scenarios. Multi-user operation is considered unproblematic because geometric objects, as well as transformation-timestamp pairs, are invariant. In addition, there is an interface for distributed storage of a graph, but possible consistency problems are not considered. Other approaches extend geometric models by separately stored ontologies [10, 21], which are not in the scope of this work.

Further graph structures, such as in [22], focus on the shared use of the map by several actors. For this purpose, a so-called "generative grammar" is set up, which regulates how parts of the model may change. The emphasis of the research can also be on a systematic class-based representation of geometric information [23].

Modelling of Uncertainty Predictions or measurements result in uncertain data, which must be correctly modeled. By augmenting coordinates with one standard deviation per dimension, locations can be stored as Gaussian distributions [24]. Likewise, uncertain occupancies in octrees can be expressed by probability values [17] and spatial transformations can be extended by additional covariance matrices, expressing 6D spatial deviation [20].

Different approaches introduce confidence values for graph attributes and links [10]. Object classification algorithms provide a classification probability, which is integrated into the class structure by Papp et al. [23]. Additionally, generic uncertainty models can be linked to physical quantities here.

## 3 Scientific Approach

The overview of the scientific approaches to environmental modeling shows that so far there are only solutions that are specialized in one field. If several robots with a shared environmental model are used, their activities and perceptions are very homogeneous. This in turn leads to restrictions in the representation of the perceived world in the model. Comprehensive environmental models, on the other hand, are not prepared for multi-user or distributed systems. It is not possible to enter predicted trajectories of robots or dynamic obstacles into the world model in any of the considered solutions.

### 3.1 Requirements

From the shortcomings of the existing solutions mentioned in the previous section and the generally necessary characteristics of an environmental model, we can derive some requirements for a new, comprehensive system for mapping the environment.

Geometry and Dynamics Real world objects should be depicted in terms of their position and shape. In order to adequately reproduce movements of objects, a distinction must be made between past, present and future movements. The latter must be further subdivided into planned and estimated movements.

Combination of Topological, Geometrical and Hierarchical Relations Three types of relations are defined in the literature:


For compact storage, a single data structure should represent these three types of relations.

#### 3.2 Scientific Concept

Our system is based on a client-server architecture similar to those in [13] and [14]. Each instance contains a copy of the shared environmental data. In the following, we describe how the data is structured and represented, focusing on motion modeling.

Structure of the Data Model Our approach is based on object-oriented representation. Geometric or abstract objects of the real world are mapped to nodes of a data structure whose edges reflect relationships between the objects.

The data structure has two types of object links, vertical and horizontal. The relative position of two objects is stored in vertical links. They form a tree-like structure and therefore also show hierarchies between objects. In contrast to a conventional tree, objects can have more than one parent node in the superordinate level, resulting in a directed acyclic graph (DAG). This means that multiple memberships can be mapped, for example a workpiece that is jointly transported by two robots. If an object has more than one parent node, its absolute position is ambiguous because there are several paths to the root node.

The second type of object connection is the horizontal link. It shows the topological neighborhood of two objects. Horizontal links may interconnect any objects in the DAG of vertical links. The collection of all horizontal links in a data model forms one or more graphs.

Contents of the Data Model Each node of the data model represents a real physical object or a group of several subordinate objects [18, 20]. Thus a node does not necessarily have to contain a geometry. The assignment of a classification and a trajectory is also possible so that all use cases can be covered.

Geometry and Classification In our approach, several types of geometry representation can be stored. Point clouds provide a storage option close to the sensor, while octrees provide a storage-efficient solution. Moreover, triangle meshes can be stored, which are very suitable for the visualization of objects. The technical details are not discussed in detail here; instead, readers are referred to the literature [17, 20].

In addition to geometries, the results of classification algorithms can be stored in the nodes [10, 23].

Fig. 1. (a, b) Dispersing trajectories of velocity and orientation (yaw) angle plotted as mean value and standard deviation (dashed) [25] (c) Discrete trajectory of uncertain positions [25] (d) Test-setup of first robot (top), human worker (middle) and observing robot (right)

Object Dynamics Present and past object dynamics are modeled according to [20]. This is applicable to the linear and angular velocity stored in each object node, as well as to the location of objects stored in the vertical links.

For interfering objects, future motions can be predicted based on present and past motions. Our approach includes an uncertainty model for representing predicted trajectories [25]. Figures 1(a) and 1(b) show dispersing trajectories of velocity and orientation angle, expressed as the mean value and standard deviation. Accordingly, discrete trajectories are formed by time-sampled points where the position is assigned an uncertainty in the form of a covariance matrix, see Figure 1(c). This representation of an uncertain trajectory has the advantage that time and location components are separated. In an environmental model which is based on geometric relationships, the trajectory can therefore be seamlessly inserted.

In case of robots, future motions are predictable. Therefore, our approach includes planned trajectories as continuous paths (NURBS: Non-Uniform Rational B-Spline) mapped to time.

### 4 Proof of Concept

A typical system has been created using open-source software. Due to the early stage of development, an industry-like application has been simulated in an office environment, e.g. an order picking robot in a small parts warehouse.

#### 4.1 Implementation

The described object-oriented data model has been written in C++. Several hierarchical classes represent the different nodes and edges as well as the optional information about classification, geometry and trajectory. They are depicted on the left side of Figure 2. Linux-based client programs were implemented for reading sample geometries from a database, inserting a trajectory and visualizing the model. Another ROS-based client makes the world model accessible to the mobile robot.

The clients communicate via a WLAN with a linux-based server application. If a client makes a change to its instance of the data model, it informs the server. The server forwards the change to the other clients.

#### 4.2 Experiment and Observation

The use example consists of a mobile robot driving along a narrow alley. At a hard-to-observe intersection a human worker crosses the robot's pathway.

The robot, which is only equipped with on-board sensors, would need to reduce its speed at such locations to ensure safety. Therefore the robot's performance decreases even if no interference occurs.

In this scenario it is assumed that a second robot beyond the junction can perceive the person and estimate their future movement. The expected trajectory

Fig. 2. Classes of the implemented data model (left), typical view of the experiment (middle) and matching environment model including predicted trajectory as circles on the floor (right)

is fed into the environmental model available to the first robot. The setup is illustrated in Figure 1(d).

Experiments prove that the first robot can react to the interference by stopping before being able to perceive the human worker based on onboard sensors alone. Figure 2 shows the crossing situation in reality and in the model.

#### 5 Results, Discussion and Conclusion

In this work, a novel shared environmental model for mobile robots was presented. The graph-based data model can represent geometry as point clouds or triangle meshes and semantic information as classification results. The proposed solution is suitable to represent future or past object dynamics and to consider the uncertainty in predicted trajectories, as demonstrated in the section above.

Furthermore, the environmental model offers advantages in the operation of mobile robots. By communicating with other robots, a robot can react to a moving obstacle before it has detected it with it's on-board sensors.

Robots must normally drive at an appropriate speed to be able to brake at dangerous points such as hard-to-observe intersections. It is also conceivable to install stationary sensors at danger points so that potentially interfering objects are always reported to the shared environmental model. Thus overall performance can be increased while maintaining safety.

#### 5.1 Fulfillment of Requirements

Current movements can be stored in the model as single values with a timestamp. When a value with a newer timestamp is added, the previous one moves to the history. Future movements can be stored continuously or discretely with uncertainty. In this way, the degree of detail and the time horizon can be varied arbitrarily. This covers a large number of applications.

Combining hierarchies and location relationships makes the modeling of location-independent hierarchies more difficult. On the other hand, it simplifies changing the location of object groups as only the position of the parent node must be updated.

#### 5.2 Outlook

Further Aspects of Modeling for Multiple Robots In addition to the aspects described above, the presented concept contains mechanisms to ensure consistency and efficient distribution of data. These are of great importance in a shared environmental model. The mechanisms are based on the existence of a central, superordinate server that decides on the approval and distribution of updates. To limit network traffic, it is possible to subdivide the environmental model into several regions. In this way, each robot receives updates matched to its location instead of receiving updates for the whole map. The horizontal links between the model objects that are described above are decisive for the subdivision.

Future Work Several possibilities for geometry storage were mentioned in sec. 3.2. It should be investigated whether further methods need to be added to this selection. Since most types of information occurring in a virtual environment are uncertain, the modeling of uncertainty should also be further investigated. Finally, tests of the environmental model with large amounts of content must be carried out to assess storage efficiency and speed so that the results can be compared with other solutions.

## 6 Acknowledgements

The results presented in this article were developed within the FORobotics (AZ-1225-16) research network. The authors would like to thank the Bavarian Research Foundation and all participating project partners for their funding and support of the project.

## References


Lecture Notes in Computer Science 10505, 249–261 (Springer International Publishing, Cham, 2017). isbn: 978-3-319-67189-5.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Octree-based, multiresolution environment mapping for efcient calculation of collision-free paths for industrial robots**

Benjamin Kaiser, Veronika Breidohr, and Alexandxer Verl

Institute for Control Engineering of Machine Tools and Manufacturing Units (ISW), University of Stuttgart, Seidenstraße 36., 70174 Stuttgart, Germany benjamin.kaiser@isw.uni-stuttgart.de,

**Abstract.** For the automatic calculation of collision-free robot paths, an environment model is required for collision avoidance. The accuracy requirements for the environment model depend very much on the specifc task of the robot. The computation time of the collision calculation increases strongly with higher resolution and dimensions of the environment model. For this reason, the demand for high accuracy and fast calculation of the collision-free path is contradictory. To solve this contradiction, we propose a method based on the multi-resolution property of octree based maps. Using a subdivision of the workspace into diferent subareas and a set of poses to be connected by path planning, a multiresolution map is created. The individual resolution levels depend on the accuracy requirements of the subareas which are defned by the process requirements. The presented method is evaluated using the "KaBa" path planning framework. For the evaluation, point clouds scanned with a line laser are tested in two path planning scenarios.

**Keywords:** path planning, multi-resolution map, octomap, industrial robot, collision avoidance

## **1 Introduction**

In production technology, the trend towards individualization of products is playing an increasingly important role. While the complexity of manufactured products is increasing, product life cycles are becoming shorter and shorter. In the case of highly customizable parts, a variant may only be manufactured once over the entire life cycle of the product. Meeting the requirements down to lot size one poses great challenges for production technology. This applies in particular to the cost-efectiveness of manufacturing. Innovative manufacturing systems with a high degree of automation and fexibility are necessary to enable the manufacturing industry to meet the new challenges of the changing market. Industrial robots ofer this necessary fexibility to produce small series efciently. Constantly decreasing acquisition and operating costs also support their use. Nevertheless, industrial robots have so far been used mainly in the automated production of large series such as in the automotive industry and are

rarely found in small series production. The main reason for this is that the existing difculties in preparing robot tasks prevent their economical use in small series manufacturing. The programming of industrial robots is a difcult task. Despite the use of the CAD/CAM chain, in practice several iterations of robot programming are necessary. This means that the programming time is often longer than the actual production time. This is especially true for small series. In addition, an individual program must be created for each variant. To enable the economic use of industrial robots in small series manufacturing, downtimes resulting from the preparation of tasks such as programming must be reduced or avoided. This can be achieved by automatic path planning for industrial robots. In this case, compliance with the process requirements and avoidance of collisions of the robot with the environment or with itself must be ensured. A complete model of the environment and the robot is required for the planning. Path planning and collision detection should be performed with high accuracy and as fast as possible. However, these two requirements are contradictory. For an exact path calculation, the space must be explored very precisely. At each explored point in space, the inverse kinematics must frst be solved to check whether the point can be reached kinematically. For all reachable points, it has to be checked if there are no collisions of the robot with itself or its environment. The calculation of collisions is computationally intensive and depends mainly on the accuracy of the collision models of the robot and its environment. To resolve the contradiction between fast and accurate path calculation, the use of efcient maps as environment models plays a particularly decisive role. Therefore, this paper presents a novel environment model that allows fast and accurate path planning by using diferent resolutions within the same map.

#### **2 State of the art**

Models for 3D environment mapping represent a key role in the automatic path planning of robots. Especially in the feld of mobile robotics this topic is intensively investigated. With recent advances of automated path planning for industrial applications like welding or assembly, environment modeling gets more important in the feld of industrial robotics as well. Early work mainly focused on the registration of height maps by distance sensors [5]. In this early works, maps were composed of a fxed, two dimensional grid with a height value at each grid point. Multi-height maps followed up on this approach. These can store more than one height value per feld and can therefore also display passages [8]. However, since industrial robots move in three-dimensional space and have 6- DOF for approaching poses, this type of map is not suitable for path planning of industrial robots. First approaches for 3D environmental models were based on a fxed grid whose cells represented voxels [6]. However, due to the fxed grid, each feld must be stored and initialized whether it is occupied or not. This makes the memory requirements for larger maps very high and hence the approach unsuitable for many applications. In addition to grid-based approaches, the use of point data from distance sensors such as LIDAR or depth cameras is widespread. Here

the map information is stored as point cloud data [1]. The work of [2] deals with multi-resolution maps based on point clouds. The disadvantage of using point clouds is that they do not provide a volumetric representation of the environment. This makes collision detection more complicated since collisions between points can remain undetected. Furthermore, these maps are very susceptible to noise from the used sensor [7]. Using octrees as a data structure to store maps solves the main problem of grid-based maps [9]. By using the octree, the entire map does not have to be initialized. Nodes are gradually inserted into the map. The use of octrees is very memory efcient. Initially Octree based maps only modeled the occupancy of space [3] [10]. The work of [13] and [7]presents a novel probabilistic octree based map called Octomap. The Octomap 3D occupancy mapping framework models occupied, free and unknown space. The Octomap is used as an environment map in a framework for camera-based path planning for industrial robots (KaBa) [12]. The path planning framework automatically calculates collision-free paths for industrial robots. The calculated paths can be executed directly on a robot controller. The KaBa framework frst captures a 3D point cloud of the environment and generates an Octomap. The point cloud is used to localize components. This allows the path planning to be adapted to deviations of the component from the target geometry and deviations of the orientation and position. The KaBa framework has the objective of making the use of industrial robots in small series production cost-efcient.

#### **3 Methodology**

#### **3.1 Problem Formulation**

For a given path planning request consisting of a discrete sequence of n poses Pi , i ∈ [0, n − 1], a collision-free path must be determined. The path planning calculates a smooth, collision-free path connecting these poses. For each segment Si−1,i between Pi−<sup>1</sup> and P<sup>i</sup> , a feasible set is defned. The path must be contained within this subset of the space. The requested path and maximum tolerance are process dependent. A welding or assembly process may require very small tolerances. Whereas the tolerance for a handling task, where the only goal may be the collision-free path, can be selected accordingly large. The computation time of automatic path planning depends mainly on two factors. First, the step size of the path planning has a strong impact on the calculation time [4]. For each pose explored in space, the inverse kinematics of the manipulator must be solved. If the pose can be reached kinematically, collisions must be discarded [12]. Second, the resolution of the environmental model. This has a direct infuence on the number of necessary collision checks at each location. Therefore, a reduction of the map resolution directly leads to shorter planning times. However, it also reduces the accuracy of path planning. This may violate the process requirements regarding precision. In order to overcome the contradiction between accuracy and computation efciency, the environmental model has to be reduced without having a negative impact on planning accuracy.

Based on the segments Si−1,i and the corresponding tolerances δ<sup>i</sup> , the area in which the tool center point moves along the planned path is limited. The process takes place in this area. A reduction of the resolution has a direct efect on the process execution. Therefore, the resolution of the environmental model cannot be reduced in this area. Nevertheless, collisions must be checked with the remaining areas of the map. It must be ensured that the robot does not collide with one of its elements during path execution. By splitting the map into several subareas, the overall map size can be reduced without afecting the required accuracy along the path. This reduction reduces the planning time. The environment model, divided into several subareas, is called Multimap. The Multimap is built on octree based maps. There are three reasons for choosing octrees as a basis. First, maps based on an octree as an internal data structure contain several resolutions inherently. This property can be used directly to generate the Multimap. Secondly, an octree represents voxels. A voxel models a volume other than a point cloud. If the resolution of a point cloud is greatly reduced, the distances between the points may become so large that collisions between them cannot be detected. This is not the case with voxels. Third, operations on the octree can be performed very efciently [11].

#### **3.2 Resolution areas**

Subareas in Multimap can be parameterized and are dependent on the desired path and the corresponding process tolerances. Resolution subareas are defned as cylinders with hemispheres at the ends. The subareas are aligned to the path segments Si−1,i. Each subarea has a defned voxel size. The fgure 1 shows the division of the map into subareas. The parts of the map that are not within any subarea are assigned the parameterized minimum resolution.

Fig. 1: Subarea division

The maximum and minimum resolution, as well as the number of resolution subareas, can be confgured. Since the exact voxel sizes are already defned through the underlying octree it is not possible to create the Multimap with arbitrarily sized voxels. Therefore, the desired highest and lowest resolution is mapped to the nearest, higher resolution contained in the octree. The resolutions of the areas between the minimum and maximum level are interpolated and mapped to an octree depth as well.

The parameterization of the multimap is done by the user based on the task and the setup of the robot. The minimum voxel size is determined based on the process requirements and tolerances. Also, the physical dimensions of the robot and especially the end efector must be considered. If the voxels are chosen too large, a path may not be possible due to collisions with the environmental model. The size of the end efector must also be taken into account when defning the size of the areas. For the number and size of the areas as well as the maximum voxel size, the calculation efciency and the resulting movement restriction for the robot must be considered.

#### **3.3 Multimap Generation**

Each path planning request contains an existing octree as an environment model. For each request, a multimap based on this octree is generated in a pre-processing step. This step consists of two parts. In the frst step, the resolution areas around each segment of the requested path are created accordingly to the parametrization. Cylinders are used to specify the diferent areas because whether a point is contained by a cylinder can be checked very efciently. However, any set that provides a method to check if a point is contained or not can be used. The calculated resolution areas specify the maximum depth of the octree for each location. The algorithm 1 shows the procedure for determining which nodes of the octree are included in the multimap.

**Algorithm 1:** Node selection algorithm

First, the algorithm iterates over the complete tree and checks for each node whether it is part of the Multimap or not. All children of a node that is outside the resolution areas are not added to the Multimap. Only the nodes in the depths that correspond to a resolution area have to be checked. The others are set to the minimum parametrized resolution. Since a lot of nodes are ruled out early by not being contained in any resolution area, the determination is fast, even as the number of nodes expands drastically with every depth. The nodes that are part of the multimap are stored in a list.


**Algorithm 2:** Map building algorithm

In the next step, the Multimap is created by building a new octree. The procedure to build the Multimap based on the node list from the frst step is shown in algorithm 2. This algorithm constructs a new octree containing the previously identifed nodes.

#### **4 Experiments**

Since the Multimap is generated for each path planning request based on the already existing octree this step leads to a calculation overhead in comparison to using the existing map. The idea behind the Multimap is to reduce the overall number of voxels per map and hence speed up the path planning due to the reduced amount of collision queries needed.

This section evaluates the efectiveness of the Multimap. For this purpose, two path planning requests are planned and evaluated using the path planning framework "KaBa" from [12]. The framework for camera-based path planning scans the environment and generates an Octomap from the registered point cloud. The framework includes a multi-level collision detection based on bounding volume hierarchy and approximate convex decomposition of the robot geometries for use with the GJK algorithm. Collisions along the requested path are avoided by using algorithms like RRT or A\* without violating the defned path tolerances.

Fig. 2: Pathplanning with octoMap and multiMap

The used Octomap contains a box on a pallet. The map was created from a point cloud scanned with a line laser. The frst test case is a path along one edge of the box. In the second test case, the robot moves the tool into the box. For this experiment, fle-based Octomaps are used. The path planning is done in Matlab. The Octomap import and the generation of the Multimap are done in C++. The goal of the Multimap is to speed up the path planning process. To achieve this, the time-saving in collision detection has to be big enough to compensate for the computation time of the Multimap. If it has almost the amount of nodes that the Octomap has, it cannot speed up the collision detection, and additionally, take a long time to build. In contrast, the Multimap is especially useful if the important areas are small.

Fig. 3: Test case 1

Three diferent parametrizations of Multimap are tested. Multimap 1 has three resolution areas, with the radiuses 70mm,140mm, and 250mm. The minimum requested voxel size is 40 mm. The resolution resulting from that request is 25.6mm. The maximum allowed voxel size is 300mm. This leads to 204.8mm sized voxels. Multimap 2 uses the same minimal and maximal voxel sizes, but only one resolution area with radius 140. Multimap 3 uses two resolution areas with radii 100 and 250, but with the requested maximum voxel size of only 150, resulting in the voxel size 102.4mm. The resulting Multimaps of test case 1 are shown in fgure 3. The Voxels are colored according to their size. Three resolution areas result in four diferent voxel sizes. The three voxel sizes in the corresponding areas, and the voxel size outside of any subarea. The resulting maps for test case 2 are shown in fgure 4.

## Fig. 4: Test case 2

#### **4.1 Reduction of voxels**

The reduction of the voxels by the multimap depends on the parameters. This reduction increases with smaller resolution areas. Table 1 shows the number of voxels in each layer of the Octomap and the generated Multimaps. Which layers are compressed depends on the parameterization of the Multimaps. Multimap reduces the number of voxels in the required resolution (depth 7) between 83% and 97,5% compared to Octomap. In the Octomap all resolutions are included and take up memory space, although they are never used. In the Multimap, the resolutions that are fner than necessary are cut.


Table 1: Octree nodes per layer in Octomap and Multimap

#### **4.2 Planning times**

Table 2 shows the resulting planning times for the two test cases. All times are the averages of six runs. The frst column of 2 shows the time to import the Octomap. This includes reading the binary fle, creating the multimap and converting it into a Matlab-compatible format. The import of the Octomap takes the most time, therefore the time diference between the import of the Octomap and the Multimap is only between 0.05481s and 0.0954s. The times for the path planning are diferent. Multimap times are usually shorter. In test case 1 Octomap is partially faster. The reason for this is that there are no collisions. Altogether the collision calculation takes less time. Since no collisions have to be avoided here, space is only explored along the requested path. Nevertheless, the total planning time, which also includes times for path preprocessing and other additional times, is always shorter with Multimap. Multimap shows an advantage in memory consumption and since its tree is pruned, it can be traversed faster. The use of Multimap accelerates the overall process.


Table 2: pathplanning times of Octomap and Multimap in comparison

#### **5 Conclusion**

This paper presents a method for generating 3D maps for the automatic path planning of industrial robots. The Multimap uses octrees as a basis and allows the direct use of multiple resolution levels of the octree for path planning. The division of the map into subareas of diferent resolution ofers the possibility to solve the contradiction between exact and fast path planning. The frst experiments show that the Multimap leads to overall faster planning times despite the calculation overhead. In further steps, the Multimap needs to be examined further using the example of a more complex path planning scenarios with many collisions and space exploration where the beneft of Multimap can be seen more clearly.

Due to the promising frst results, the multimap should be integrated into the native implementation of the KaBa path planning framework for subsequent work. The change from the Matlab implementation to the native variant of KaBa allows much shorter planning times and allows experimental validation on a real robot cell. Besides, the question of how the parameterization of the multimap can be carried out in practice by the user must be further investigated.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Towards Embedded Force Sensors in Exoskeletons for Evaluating Interaction Forces in Interfaces**

Niclas Hoffmann1, <sup>2</sup> , Samet Ersoysal<sup>1</sup> , and Robert Weidner1, <sup>2</sup>

<sup>1</sup>University of Innsbruck, Institute of Mechatronic, Chair of Production Technology, Austria <sup>2</sup>Helmut-Schmidt-University/University of the Federal Armed Forces Hamburg, Laboratory of Manufacturing Technology, Germany Niclas.Hoffmann@uibk.ac.at/Niclas.Hoffmann@hsu-hh.de; Samet.Ersoysal@uibk.ac.at; Robert.Weidner@uibk.ac.at/Robert.Weidner@hsu-hh.de

**Abstract.** The applicability of exoskeletons in different daily and occupational settings is continually increasing. Due to varying tasks and user groups, the adaptability of exoskeletons is becoming more significant, with increasing demands for smarter devices. The implementation of force sensors in exoskeletal interfaces could be an approach for analyzing the human-machine interaction in terms of, e.g., wearing comfort, support, and motion synchronicity, as well as optimizing this interaction in real time based on the analyzed sensory data. For this, force sensors need to be embedded in interfaces, which implies the consideration of inexpensive sensors to minimize the total purchase price. However, measuring contact forces on the wearer is challenging and inexpensive flexible force sensing resistors have limited accuracy, repeatability, and stability. This paper evaluates the suitability of an interface principle working with two water capsules and two embedded piezo-resistive pressure sensors in different test scenarios derived from real exoskeletal application examples. Finally, a comparison of the capsules' inner pressures reliably detects different load conditions on the interface such as centered, edged, and shear forces. Thus, this principle seems to be suitable for further exoskeletal considerations.

**Keywords:** Force Sensors, Low-Budget, Exoskeletons, Human-Machine Interaction, Evaluation, Adaptability.

## **1 Introduction**

Exoskeletons have become increasingly popular within the industrial sector with various applications [1]–[3] to support the user by decreasing general workloads or work in forced positions, as well as increasing working productivity due to prolonging muscle fatigue [4]. In principle, the successful application of any support system requires a merger of human, machine, and activity [5]. Thus, exoskeletons feature different properties based on their morphological structure such as path of force, structural rigidity, and actuation [3]. To assess the functionality, suitability, and impact of an exoskeleton in a certain context, an evaluation of the system should be conducted first, considering diverse criteria like level of support, kinematics, comfort, and safety [6]. For this, different methods ranging from biomechanical assessments to questionnaires are usually used to get a comprehensive impression of the exoskeleton's performance [7].

The implementation of force sensors in interfaces of exoskeletons may help to conduct a multicriterial evaluation of exoskeletons using only one tool. Statements concerning level of support, control and dynamics, situation and intention detection, path of force as well as the design, fit and comfort of its interfaces are therefore possible.

If such sensors are incorporated in the interface, the sensory data may be analyzed in real-time and initiate corresponding adjustments of the exoskeleton. This may improve human-machine interaction and acceptance of exoskeletons equipped with force sensors. Additionally, this approach includes the use of inexpensive sensors in order to keep total cost of the exoskeleton at a competitive level. Overall, it depicts one way for exoskeletons to become smarter devices by improving their automatic adaptability to different working environments, tasks, and users.

In principle, the accurate measurement of contact forces on human interfaces is challenging [8] and requires diverse sensor characteristics like linearity, low creep, low temperature sensitivity, flexibility, and repeatability [9]. When focusing on inexpensive sensors for this application, only piezo-resistive force sensors seem to be suitable [10]. Diverse approaches have already been made using different Force Sensing Resistors (FSRs) [8], [11], however these sensors usually suffer from low linearity, stability, repeatability, high drifts and hysteresis [10], [12]–[14]. In this context, commercially available pressure sensor pads are also unsuitable due to their high purchase costs and their complexity for further processing sensor data in real-time.

The use of high-resolution pressure sensors instead of FSR seems to be beneficial for measuring surface pressure on interfaces [15], whereby the average pressure on the interface divided by the interface region implies the total support force of the exoskeleton [14]. In this paper, different test scenarios derived from real exoskeleton applications evaluate the general suitability of a developed exoskeleton interface principle with two water capsules and two embedded pressure sensors. These scenarios mentioned in chap. 3.2 may simulate possible load conditions on exoskeleton interfaces to assess the general human-machine-interaction and reveal recommendations for further actions.

#### **2 Interface Prototype with Embedded Pressure Sensors**

The developed interface prototype measures pressure conditions within a flexible capsule with the help of embedded pressure sensors. If a load occurs on the interface, the flexible capsule is compressed and its inner pressure rises, which is detected by the pressure sensor. Overall, the function of the flexible capsule can be broken into twopart: a) pressure detection and b) replacement of foam for bolstering the interface by working as a cushion itself.

In its basic structure, the prototype is inspired by the work of Lee et al. [15], who designed a prototype for a pressure sensor system based on a "micro-sensor inside a flexible capsule filled with a pressure transmissible fluid". Their prototype was built out of a plastisol bladder as a flexible capsule, laminated on an aluminium plate, using hair gel as transmissible fluid, and the pressure sensor MPX2300D.

Here, the interface prototype uses the above basic construction principle, but a cast silicone capsule, water as transmission fluid, the modern high-resolution pressure sensor MS5803-01BA, and a second aluminium plate for detachable clamping of the flexible capsule onto the aluminium baseplate. In addition, the prototype is expandable to additional flexible capsules with multiple pressure sensors. This may be useful, as apart from magnitude and duration, the pressure distribution is of special interest for human interfaces, which is caused by changes in human soft tissue or mismatches of exoskeletal and human axes [11], [14], [16]. In addition, the force magnitude may be solely assessable via power curves of the exoskeleton actuation or simulated human models. In this approach, the prototype does not feature a third chamber in the middle of the interface, due to load usually shifting towardsits edges [11]. Furthermore, this approach is supported by an economic perspective limiting the total amount of sensors.

#### **2.1 Construction**

The construction of the interface prototype is illustrated in the figure below. The left side shows its lateral cross section at sensor level, the right side its total view presented in an exploded view drawing.

Fig. 1. Construction of the interface prototype with embedded pressure sensors.

The constructive base of the interface prototype is a milled rectangular aluminium plate with the dimensions of 100 x 60 x 10 mm, that represent the exoskeletal interface. For later load experiments, holes in the center and the right edge are drilled into the plate, and a hitch on the right lateral face is installed. On the interface, the two pressure sensors are placed in two transfixions on a pressed sleeve and clamped with a crossbar (not illustrated in Fig. 1). This connection is sealed with an O-ring on the outer diameter of the metal cap of each sensor [17].

The used pressure sensor MS5803-01BA, which is based on leading MEMS technology and placed on a breakout board for an easier junction, is a piezo resistive, highresolution, low power, and economical altimeter sensor with adjustable resolution. For the used resolution of 0.012 mbar, the response time is 8.22 milliseconds. The accuracy in the used range is ± 1.5 mbar. Due to a gel protection, the sensor can measure pressures in different mediums like liquids or air. For further technical details, see [17]. In this set-up, the sensor runs on an Arduino UNO via I <sup>2</sup>C communication.

The flexible capsule needs to be shapeable, resilient, sealed, and skin-friendly, which is achieved by the multi-purpose silicone "Dragon Skin 20" [18]. For molding the capsule, this two-component silicone is mixed, stirred, evacuated to release air bubbles, and cast in a milled mold. The benefits of the molded capsule are a return to its original form without load, no recorded irritabilities in case of longer skin contact, and an easy adaptability of its shape to different dimensions of exoskeletal interfaces. The dimensions of each capsule are 60 x 47 x 10 mm and the wall thickness is 1.0 mm on the sides and 1.5 mm on the bottom and the top. After curing, each capsule is positioned under the corresponding transfixion of the interface and clamped by a second milled aluminium plate with four mounting screws, respectively. After that, the capsule is filled with water, vented, and sealed with the pressure sensor by fixing the screws of the crossbar.

#### **2.2 Calibration of the Pressure Sensors**

First, the prototype is placed on a hard surface, e.g., a wooden table. The calibration weights are centered on a 3D printed sheet that is stuck into the center hole on the aluminium plate, in order to consistently distribute the load on each silicon capsule. The following calibration process is derived from [8]. Under barometric pressure and room temperature with relative humidity, the interface is loaded and unloaded in the center with three weights (200 g, 350 g, 500 g) three times with a duration of five seconds each and a sampling rate of 50 milliseconds. Three pressure values after foursecond measurements are taken and averaged. Based on the respective pressure gradient of each load against the pressure without load, a linear three-point calibration is made. The quality of the linear regression using this method has proven very sufficient (R<sup>2</sup> sensor1 = 0,9997 and R<sup>2</sup> sensor2 = 0,9999) and even a little improved compared to [15]. This suggests that the linear performance of each sensor is very good in this measurement range. For later interpretation, the load with weights in gram (g) is translated to an acting vertical force in Newton (N).

#### **3 Experiments**

Since Lee et al. [15] already demonstrated the benefits of this measurement principle against other commercially available, low-cost force sensors in terms of linearity, temperature effect, hysteresis, reliability and creep, the key aims of this work are to place these within the context of exoskeletons:


For this, different experiments are derived from possible application scenarios for an exoskeleton for the upper extremities. Each experimental set-up uses varying weights and weight positions to represent different support forces of an exoskeleton, or load conditions caused by different movements and tasks.

#### **3.1 General Test Set-Up**

In order to focus on the principal applicability, the prototype is tested in a controlled laboratory setting. In reality, the supportive force of an exoskeleton affects the human soft tissue through the interfaces, where the interface distributes the force as surface pressure to the contact area on the body. Various load conditions, distributions, and shear forces may occur (see Fig. 2). In the lab experiments, this is simulated with different weights placed on a 3D printed plate and positioned in the center and on the edge of the interface. A spring scale simulates shear forces. The human soft tissue is modeled with two silicone layers with different shore hardness [19] (upper layer: "Ecoflex 00- 10", lower layer: "Dragon Skin 20").

Fig.2: Derivation of the general test set-up and scenarios from real exoskeletal applications on the example of the upper arm.

#### **3.2 Test Scenarios**

Test scenario A **"Equal Continuous Constant Load"** describes a task that is performed for a prolonged period of time in the same physically demanding posture (i.e. the complex electronic cable insertion in car doors, which is performed overhead/above shoulder level). This is tested with a constant load of 4.9 N placed only in the center of the interface for 120 seconds. Lee et al. [15] tested such a scenario for 144 seconds.

Test scenario B **"Continuous Decreasing Load"** considers a continuous load on the center resp. edge for a prolonged period, which gradually decreases over time. One example may be the emptying of goods from a large box/container and placing them separately onto shelves or into different storage spaces, so that the weight of the load carried decreases. Another example is the installation of pipes on a ceiling with clamps, so that the weight of the pipe decreases with every fixed clamp. In the tests, the interface is loaded with 5.9 N to begin with, and then slowly decreases to 5.8 N, 5.4 N, 4.9 N, and 3.9 N after six seconds each. The respective gradient deliberately varies.

Test scenario C **"Repetitive Constant Load"** represents a repetitive task at head level or above, e.g., driving screws in a ceiling, with the same weights of tools or other components. If users raises their arms, the exoskeleton may support them, hereby applying surface pressure to the human tissue. If users lowers their arms, no pressure is applied due to the lack of load on the upper extremities. In the simulation, the aluminium plate is repetitively loaded five times on the center resp. edge for three seconds with 3.9 N. A pause of three seconds is conducted in between.

In the test scenario D **"Repetitive Varying Load"**, the ability of the prototype of measuring varying repetitive loads is evaluated. In practice, these loading conditions may appear, if the user of an exoskeleton needs to randomly handle light and/or heavy packages (i.e. working in logistics). The support of especially active exoskeletons is usually adaptable, which may lead to varying loads. In the test set-up, the interface is consecutively loaded on the center resp. edge with 4.9 N, 0.2 N, 3.9 N, 1.5 N, and 4.9 N for a duration of three seconds. A pause of three seconds is conducted in between.

Test scenario E **"Repetitive Varying Shear Forces"** evaluates the ability of the prototype to assess shear forces during motion, that may be a cause for friction and may even lead to skin abrasions. Shear forces may be an indicator for a poor fit and consequently lead to rejection of the exoskeleton. In the test, the interface is vertically continuously loaded with 4.9 N and in addition repetitively loaded horizontally in longitudinal resp. lateral direction of the interface, with a varying traction force of 2.0 N, 1.0 N, 2.9 N, 4.9 N, and 2.9 N. A shaped metal crossbar prevents any rolling of the interface to focus only on the influence of horizontal motions of the interface on each capsule pressure.

#### **4 Results**

For every test scenario, the different sensory data are stored and plotted in Matlab R2018b and presented in Fig. 3. Based on the inner pressure and the calibration of each sensor, the respective load conditions may be concluded.

Both sensors accurately measure near the same pressure, if the interface is equally loaded with the same weight for a prolonged period of time (see scenario A). Both sensors also measure similar pressure for equal repetitive loads (see test-retest scenarios C and D). In case of an equal load, both sensory values add up to near the applied load. If an unequal load occurs, the inner pressure of the capsule underneath the load measures the applied load and additionally rises with almost the same gradient like the equivalent of decrease in pressure occurring to the other capsule (see scenario B, C, D). Test scenario B shows that the sensors may even detect marginal changes in load as small as 0.1 N, and further, measure varying consecutive loads with high accuracy (see scenario D).

Fig. 3: Results of the different test scenarios. The abscissa shows the passage of time and the ordinate the inner pressure in the capsules measured by the pressure sensors.

However, small loads may be detected as a rise in the inner pressure but may also be wrongly interpreted as a negative or decreasing force (see scenario D). If horizontal shear forces longitudinally act on the interface, the value of the anterior sensor to the location of applied load rises slightly, while the value of the posterior sensor decreases at a larger rate – this may be true even for small forces. This effect does not appear for laterally acting shear forces (see scenario E) and all vertical loads (see scenario B-D) on the interface.

#### **5 Discussion**

An analysis of the results shows that the interface prototype has a good stability, repeatability, and accuracy in its measurements. Even marginal load changes may be detected, however, their interpretation may be prone to error, due to the constant of the linear regression, on which the calibration of the sensors is based. Thus, although a small load is usually detected, it may be wrongly interpreted as a small decrease in load. However, a comparison of both sensory data may help to identify and objectively interpret the following load conditions on the interface, which may include recommendations for further actions.

For the following ratios, the respective interpreted load identification is presented. The "X" represents a by an unknown factor changing variable, dependent on the corresponding load condition:


With two capsules, the prototype is only capable of detecting load distributions and shear forces on the longitudinal axis of the interface. For statements on the lateral axis, two additional flexible capsules must be implemented on the interface in this direction. This would enable the detection of any spatial load on the interface. At this point, the benefit of an additional centered capsule requires further investigation.

However, the interface prototype depends on the barometric pressure, as the gradient against the barometric pressure is translated into respective load. Thus, the prevailing barometric pressure needs to be detected prior to each measurement to ensure accurate results. In addition, the gravity of water in the capsule has an influence on the interpreted load as well. This may occur, if the interface spatially rotates while performing certain tasks and the sensor does not remain in its vertical orientation, as it was in the calibration. Here, the implementation of an Initial-Measurement-Unit (IMU) on the interface may help calculate an offset. Alternatively, air may be used to replace the water medium, however this could lead to other limitations and challenges, particularly in sealing the entire interface airtight.

Finally, it must be acknowledged that the test set-up itself presents some limitations. First, the model of the human tissue used in the above study is fairly consistent, which is not always the case when working with human subjects due to factors such as varying tissue and bone structures. The loads used may also not be representative for all possible exoskeletal scenarios. However, as long as a mechanical load test of the interface prototype is pending, the loads are limited to 5.9 N in total, for safety reasons. The planar shape of the interface prototype and the tissue model are also abstract, acknowledging curvatures and uneven surface structures in reality for both. Finally, the environmental influences on interfaces vary in terms of temperature and humidity, which are not considered in the controlled laboratory setting.

#### **6 Conclusion**

In this paper, the suitability of an interface prototype with embedded pressure sensors for assessing and optimizing the human machine interaction in the context of exoskeletons is evaluated. In five test scenarios, it has been shown that this prototype is capable of reliably detecting different load conditions, between an interface and a model of human tissue.

In the future, two main areas should be focused on: the construction and the analytical ability of the interface prototype. First, a load and repeatability test of the entire prototype should be conducted, especially of the silicon capsule. Efforts should also be made to install and test this principle on curvatures, as most exoskeletal interfaces are not planar. On the other hand, the prototype should be tested on humans under real conditions and embedded in exoskeletal interfaces, whereby the influence of wearers' body temperature and sweat as well as the ability of replacing foam bolstering with the flexible capsule may be a matter of interest. The favored level of support for diverse tasks or critical pressure levels in terms of subjective perceived comfort may also be considered. At this point, the evaluation may be extended to health issues by looking at blood flow [20] or oxygen supply beneath the pressured surface [21] in association with near infrared spectroscopy (NIRS).

#### **References**


and its Application to Exoskeletons. In: Developing Support Technologies - Integrating Multiple Perspectives to Create Assistance that People Really Want. Springer Verlag.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Numerical simulation and statistical analysis of a cascaded flexure hinge for use in a cryogenic working environment**

Philipp Jahn1 and Annika Raatz1

1Leibniz Universität Hannover, Institute for Assembly Technology, D-30823 Garbsen, Germany jahn@match.uni-hannover.de

**Abstract.** Due to their many advantages, flexible structures are increasingly being used as guide and transmission elements in handling systems. Prismatic solidstate joints with a concentrated cross-sectional reduction are predominantly used as flexure pivots for both microscopic and macroscopic designs. A transfer of these geometries to applications in cryogenic working environments is not easily possible at temperatures below -130 °C due to the changed material properties. In this paper, the further development of swivel joints as cascaded solid state joints for such a cryogenic environment is illustrated by the targeted adaptation of certain joint parameters and dimensions. By means of a comprehensive FEM simulation, it can be shown how the influence of specific parameters affects movement accuracy, process forces and shape stability and to what extent these geometric parameters influence each other in their effect.

**Keywords:** compliant mechanisms, flexure hinges, cryogenic workspace.

## **1 Introduction**

In today's industry, automation is an omnipresent factor or will be, even in niche applications such as cryopreservation. Nevertheless, Manual handling of biological or toxic samples is still the rule in research facilities. Such samples are conserved in so-called biobanks. In their simplest form, these are Dewar vessels are cooled with liquid nitrogen. However, large semi-automated systems are also used, mainly for an indefinite storage time. Even in such systems, the samples are still stored, rearranged or transferred by hand using bulky protective clothing and equipment. Doing so involves a significant risk of injury due to cold burns for the users on the one hand. On the other hand, the sample integrity can be compromised by any kind of heat exposure during the handling process. To overcome the dangers of manually handling test tubes while circumventing the technical challenges a cryogenic work environment presents, a full automation is desirable. The logical consequence is to develop a robot based handling system.

As part of the DFG funded project "Methods for the automation of handling processes under cryogenic environmental conditions", a parallel robot is being developed at the Institute of Assembly Technology at Leibniz Universität Hannover to implement such full automation. Especially the passive joints of the robot have to meet very high demands: The extreme temperatures of below -130 °C do not allow the use of classic rigid body systems such as ball joints due to the freezing of lubricants or the clamping of components due to cold shrinkage. To avoid these disadvantages, solid-state joints in the form of cohesive swivel joints are used. Due to their monolithic construction, clamping is not an issue since no parts moving against each other are involved. In addition, the use of lubricants is also unnecessary.

Due to these and other advantages, joints with concentrated resilience designed as monolithic coupling mechanisms have already become widely used in technical and industrial handling technology. In these one-piece constructions, an inherent compliance is achieved exclusively by reversible deformation of certain areas. Thus, these mechanisms can be used for small angular deflections or displacements instead of conventional form- or force-paired pivot hinges. The most common application to date has been prismatic solid state joints with areas of reduced cross sections as bending points. These mechanisms are mainly found in precision systems in precision mechanics and metrology as well as in applications in precision assembly, i.e. areas in which high demands are placed on positioning accuracy and resolution [1–4]. Applications in macroscopic constructions and for big angular deflections are still the exception.

Solid-state joints are material-coherent mechanisms and their ability to m ove is based on the deformation of the entire joint or on the deformation of areas with a concentrated reduction in cross-section. Therefore, their bending range is severely limited to a few degrees by the maximum allowable stresses in the component. In addition, a rotational movement of a material-locking joint always causes a load-dependent displacement of its axis of rotation, which has a negative effect on the positioning accuracy of the mechanism. It is therefore always desirable to achieve the smallest possible ratio of maximum stress to the rotation achieved, while at the same time minimizing the displacement of the axis of rotation. To achieve this, increasingly complex geometries have been developed in the recent past, or an increasing number of simple solid-state joints have been installed in a common structure.

#### **1.1 Related work**

In contrast to cohesive joints with only one area of concentrated flexibility or with multi-unit flexible coupling mechanisms, the structural and dimensional synthesis of complex flexure pivots has not yet been discussed in detail. The literature mainly describes constructions with concentrated cross-section reductions in rectangular or circular contours, as well as the effect of these contours on the achievable range of motion [5–17]. There is also work on the geometric design of complex swivel joints on the basis of interlinked tape spring joints [18], which allow large angular deflections. The relevant influences on the geometric design of such joints with regard to their ability to move along the main axes and parasitic axes have hardly been investigated. The project "Methods for the automation of handling processes under cryogenic environmental conditions", however, was preceded by the work of Raatz et al. [19–21]., in which the use of cohesive joints in parallel robot structures and the kinetic and kinematic behavior of these joints were examined in detail. The use of solid-state joints in a cryogenic working environment has not yet been investigated. The work on the use of flexible joints for use in aerospace engineering has been described [3, 4, 22], but not for the environmental conditions envisaged in this project.

Following on from the work of Robert Fowler [4] and Simon Heinein [22], in this paper the finite element method (FEM) and the ANSYS program will be used to present a joint geometry that is capable of achieving a required angle of rotation in a cryogenic working environment. The maximum permissible stresses within the components are never allowed to exceed a certain value, so that the deformation of the joint is completely reversible at all times. In addition, by varying certain design parameters, their influence on the necessary process forces, the undesired deformation along certain axes (henceforth called parasitic axes) and the effect of these parameters on each other will be investigated.

#### **1.2 Materials and methods**

The aim of this work is the further development of cascaded solid state joints based on the work of Robert Fowler [4], who in turn bases his work on the Butterfly Pivots by Heinein et al [22, 23]. Especially the Flex-16 large-displacement compliant hinge offers a good starting point for the development and investigation of own joint geometries (Fig.1). Due to the continuum mechanical problem, a sequential approach of concept synthesis and constructive development of the design is not easily transferable to the existing compliant hinges. The displacement of the axis of rotation allows only limited formation of a replacement model based on rigid state mechanisms. In addition to the desired rotational motion, a force-deformation characteristic along the main and parasitic axes as well as the shape stability must also be taken into account in the structural development of compliant mechanisms. The procedure here follows the idea of a topology-optimized structure. The relevant load paths and most stressed points of a prototype are identified. Then the material distribution is varied within the framework of certain constructive and production-related restrictions and the change of the load cases is evaluated afterwards. In particular, it will be investigated whether the motion and deformation behavior can be represented as a function of the geometry parameters and whether there are interdependencies between the geometry parameters with respect to their influence on the motion and deformation behavior. For this purpose, four criteria are to be considered qualitatively and quantitatively as result variables:


**Fig. 1.** Prototype Geometry based on Robert Fowlers Flex-16 large-displacement compliant hinge

The simulation of the joints in ANSYS is to be carried out under the conditions typical for a cryogenic working environment, taking into account the changed material properties of the joint material at temperatures below -130 °C. However, since these temperatures are reached in a biobank by cooling with liquid nitrogen, the components undergo the danger of being exposed to much lower temperatures. For the simulations, the adjusted material parameters Young's Modulus and transverse contraction number are used from tabulated sources [24, 25] for a temperature of -196 °C, which corresponds to the boiling temperature of liquid nitrogen and includes the case that the joints are immersed in the liquid.

### **2 Prototype geometry**

The starting point of the joint construction is the Flex-16 joint, which was presented in multiple variations in [4]. After initial estimations and simulations, the overall dimension was fixed at 50 x 50 x 10 mm³. This limitation was chosen because of the space available in the final construction of the parallel robot, which will be examined in more detail in other works. For a better understanding, a design of this robot and a Cardan joint consisting of six flexure pivots are shown in Fig. 2. In [4] it has already been shown, that this flexible mechanism is able to achieve a bending angle of 90° at room temperature, if titanium was used as the base material.

**Fig. 2.** a) Parallel robot structure and b) Cardan joint

#### **2.1 Simulations of the prototype geometry**

The rotational movement of the joint in the ANSYS simulation is realized by fixing the joint at one end and applying various forces at the other end to investigate all possible load cases. Care has been taken to ensure that the maximum equivalent stresses do not exceed the maximum yield strength that the material can withstand under cryogenic conditions. This is made possible by the fact that the bending angle to be achieved for the entire joint is divided between the 16 serially linked thin sections of the joint. The simulation results of the bending are shown in Fig. 3 a). From this preliminary examination, the most heavily loaded or most heavily deformed sections were determined. As can be seen in Fig. 3a), the sections 3 and 4 are most strongly involved in the rotational movement about the intended bending axis. In Fig. 3 b), the sections 1 and 2 show the strongest local deformation in positional deviation along the parasitic axes under a tensile load.

**Fig. 3.** Flex 16 under a) tensile load and b) bending load

While the bending movement around the axis of rotation should be achieved with as little process force as possible, it is desirable to stiffen the joint in the direction of the parasitic axes as much as possible by adjusting the geometry. The geometry parameters described in the following and their effects on the result variables mentioned in section 1.2 have particularly this stiffening as their goal. After several simulations with different materials, the choice of material falls on titanium. The minimum 20° bending angle of the joints required for robot design could not be achieved with any other material within the limited installation space.

#### **3 Development of the joint geometry**

The flexure joint developed here will later be manufactured from the titanium alloy TiAl6V4 in a laser sintering process. This choice was made, because it is by far the most common titanium alloy available for industrial additive manufacturing processes. The hinge consists of 16 individual thin sections arranged around a central connection point. Four areas with a comparatively large cross-section - so-called shuttles - are arranged around the fulcrum to ensure axial stability. The thin sections form six "wings".

The first geometric parameter to be varied is the thickness of the material. It is expected that an enlargement of the cross-section *h* (see Fig. 4b) will lead to a stiffening of the joint. The extent of this stiffening is to be investigated for a range between *h*=0.8 mm and *h*=1.2 mm in 0.1 mm steps. The second parameter describes the shortening of the thin spots called wings. Here it is assumed that a shortening of the wings will lead to a stiffening of the joint. The wing proportions are shown in here Fig. 4c). Their length *l* is varied between *l*=15 mm and *l*=5 mm in steps of 2 mm. The stiffening by an enlargement of the cross section and the shortening of the thin sections mainly involved in the bending are logical steps. The last geometry parameter to be varied is derived from the results of the prototype geometry from Fig. 3b. As can be seen, the upward bending of the wings seems to be the main reason for the deformation in the y-direction. To minimize this deformation, small corner shuttles are provided in

**Fig. 4.** a) Prototype proportions, b) widened cross-section *h*, c) shortened wing length *l*, d) increased radius for corner shuttles *r*

the wings, whose radius *r* is varied between *r*=0.4 mm and *r*=3.2 mm in steps of 0.2 mm (see Fig. 4d). The minimum variation of the geometries is based on the manufacturing parameters of a laser sintering process, which can generally be guaranteed by the manufacturers [26, 27]. The number of possible parameter combinations resulting from this is two hundred and twenty-two simulations.

Depending on the selected geometric proportions, different bending angles can be achieved before the deformation causes critical stresses in the components. Since a minimum angle of twenty degrees for each solid state joint is sufficient for the parallel robot structure to be realized, it was checked whether all joint variations can represent this angle, and if so, which maximum angle can be achieved by all joints. The preliminary investigations showed that the smallest bending angle that can be achieved by elastic deformation of all joints is 27.75 degrees. This angle was assumed as a fixed value for the simulations. Thus, the necessary applied bending force as well as the maximum von Mises stresses occurring can be used as target values. In addition to the bending, the joints are also loaded with 16 N for tension in the y-direction and with 16 N for shear in the x-direction in order to investigate a deformation along the parasitic axes. The target values for these load cases are the deviations from the initial position along these axes.

#### **3.1 Simulation of the joint geometry**

The basic geometry of the flexure pivot based on the Fowler model [4] is created with solid works. The geometry parameters to be adjusted are then inserted and the model parameterized in ANSYS. The mesh quality of the model significantly influences the calculated stiffness in the simulation [28]. In order to find a compromise between the necessary simulation time and sufficient accuracy, the number of nodes used per model is limited to 8200. This number resulted from the specification that every element has an edge length of no more than two times the material thickness. The aim was a multizone mesh. Areas of the solid state joint that have a simple geometry (mainly cuboids or circular sections) can be swept and thus meshed very evenly. For the areas where these joint sections meet, a finer mesh structure has been strived for. Here a hex-dominant mesh was preferred, if possible. The use of linear tetrahedrals was avoided as this element class usually implies too high a stiffness. The skewness, which should not exceed the value 0.5, was considered as a measure for the quality of the mesh. These restrictions were met for all simulated joints. Due to the quality of the net, it was possible to forgo center nodes, which can also cause excessive stiffening of the simulated component [28]. A Sparse Matrix direct solver was used with 40 substeps and a minimum increment of 0.51×10-4 for the convergence criteria (CC).

**Fig. 5.** Deflection of the flexure pivot und a) bending load, b) shear load, c) tensile load

Two hundred and twenty-two simulations were carried out in this way. Based on this, a comprehensive examination and analysis of the data was carried out. Fig.5 shows an example from the simulation with the corresponding images of the deformed joint under the given load cases. The consideration of the target values (see section 1.2) shows a trend in the effect of the varied geometry parameters. Fig. 6 shows the maximum stresses as an example.

**Fig. 6.** Maximum stresses, sorted by the varied geometric Parameters

The course supports the assumption that the geometric parameters are not independent of each other with respect to the effect they have on the target variables. It can also be seen that the effect of the changed parameters material thickness, wing length and corner shuttle have a different influence on these target values. The widening of the crosssection has by far the strongest influence on our target values, whereby the increase of the necessary bending forces is rated as negative, and the reduction of the deflection of the joints along the parasitic axes is rated as positive. Analogous conclusions result for the wing length and the radius of the corner shuttles in descending significance. Here, too, it can be seen that the effect of the parameter change is not linear, which again supports the assumption that the geometric parameters are not independent of each other. In order to investigate this in more detail, the results of the simulations were divided into different classes. The simulated feature carriers are first divided into six classes sorted according to the cross-section thickness and then the target values are displayed as a function of the wing shortening. The corner shuttles are set to a radius of 0,4 mm. As can be seen in Fig 7, the graphs do not run parallel to each other. Analogous to this procedure, six subclasses are formed for the wing length and then the target values are displayed as a function of the radius of the corner shuttles. This time the material thickness is set to 1 mm. The result forms a similar picture as in the previous consideration. The course of the curves is not constant between the subclasses.

**Fig. 7.** Class comparison: cross-section widening

### **5 Analysis of the results**

The graphical evaluation of the results suggests that there is no linear correlation between the target values and the individual geometry parameters. For a multivariate analysis approach, a regression analysis based on a polynomial is suitable. Based on Fig.7 however, it is assumed that the parameters themselves depend on each other in their effect on the target parameters and therefore no linear regression model can be used as a basis. At this point, however, qualitative statements about the influence of the geometry parameters on the target variables can already be made. These are summarized below. A further procedure on the basis of the obtained data is discussed in the outlook. Based on the results obtained so far, the following qualitative correlations can be concluded:


Because of the data obtained, the following quantitative statements can be made, even if the formula-based relationship is still unknown. Fig.8 sums the results up graphically:


**Fig. 8.** Summary of the results

## **6 Conclusion and outlook**

Due to the assumed non-linearity and lack of independence of the varied geometry parameters from each other, no formula-based procedure for the optimal design of the flexure pivot described here can be proposed at present. However, it could be shown that the geometric adaptation of the joint allows a significant reduction of deviations from the nominal position without limiting the functionality or exceeding the critical comparative stresses. In the following, further analyses of the existing database will be carried out. This includes non-linear regression analyses as well as optimization strategies. In addition, a characteristic value is to be developed which describes the usefulness of the geometric adaptations as a ratio of the positive and negative effects. If this procedure proves to be promising, further joint geometries will be created and evaluated.

## **7 Acknowledgements**

The results presented in this paper were obtained within the project "Methods for the automation of handling processes under cryogenic environmental conditions" (project number: 349906175). The authors would like to thank the German Research Foundation (DFG) for their financial support of this project.

## **References**


based on compliance closed-form equations. Mechanism and Machine Theory 37, Franklin, TN (2002).


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Automated generation of adapter plates for industrial robots**

Lars Niklas Penczek[0000-0002-1704-2843], Matthias Bartelt[0000-0003-4239-3490] and Bernd Kuhlenkötter[0000-0002-5015-7490]

> Lehrstuhl für Produktionssysteme, Ruhr-Universität Bochum, Universitätsstr. 150, 44801 Bochum, Germany penczek@lps.rub.de

**Abstract.** The current market offers a wide range of different robots and end effectors from a variety of manufacturers and in different versions, where the mechanical compatibility between the components has to be ensured. A large part of the flange patterns of the end effectors offered are not designed to conform to standards, which is one of the reasons why adapter plates must be used to ensure compatibility. Particularly in the early phases of engineering, the weight of the adapter plate and the extension of accessibility can be taken into account in the robot configuration when generating an adapter plate. In order to enable the generation of an adapter plate as early as possible, automated generation offers a great advantage and can contribute to the reduction of engineering effort and thus to time savings and cost reduction. This paper describes a general design rule for the generation of adapter plates and, on this basis, implements the generation of CAD geometries of the adapter plates using Constructive Solid Geometry. The results of this paper are the starting point for the implementation of a web service, which will be integrated into the ROBOTOP project.

**Keywords:** Flange patterns, construction instruction, Constructive Solid Geometry.

### **1 Introduction**

Industrial robots and end effectors are offered by a large number of manufacturers and in an abundance of design variants. For example, 422,271 robots were delivered worldwide in 2018, which represents an increase of 15 percent over the previous year. Similar figures are predicted for 2019. [1]. The market situation for end effectors is similar, with the four best-selling end effectors manufacturers alone (Schunk, Schmalz, Destaco and Festo) currently offering over 5,400 different grippers [2-5]. In order to ensure compatibility between the individual components across manufacturers, the interfaces between robot arms and end effectors must be clearly defined. This should make it easier for the customer to configure robot systems and ensure a lower knowledge barrier for the combination of different robot solutions. Small and medium-sized enterprises (SMEs) in particular can be supported by this in production automation. Because of its relatively low degree of automation, the SME market offers a particularly high potential for future automation through robot systems. Decisive factors with the implementation of such robot systems are the high engineering expenditure and the high costs. Thus, many robot solutions cannot be economically represented for system integrators up-to-date or only to a very limited extent.

The ROBOTOP project addresses this problem by developing an online configurator for robot systems. Within the scope of this research project, the mechanical interfaces of the end effectors and robot arms are considered. If the components to be assembled have different flange designs, they have to be brought together with an adapter plate. In addition to bringing the flange patterns together, the adapter plate also offers the possibility of changing the offset position of the end effector on the robot system. Thus, on the one hand, the end effector can be realized in the extension of the axis of the robot, a lateral offset or a rotation of the end effector is also possible. The generation of these adapter plates is currently carried out mainly by the manufacturers when the end effectors are delivered or by the customer himself. Here, the development of special adapter plate shapes is combined with relatively high effort and is not based on any standardized process. Due to the design of the adapter plates from quantity one at the customers themselves, the adapter plates are usually oversized for safety reasons. This leads to an overall higher dead weight of the adapter plate and thus to a reduction of the possible payload of the robot. In addition, the increase in reachability due to the offset of the end effector with the help of the adapter plate is often not used at all or only in the later phases of the planning process. Thus, the generation of an adapter plate at the time of the configuration of the robot system would allow a reduction of the engineering effort and an optimization of the robot system selection during the planning of the robot system.

Within the scope of the ROBOTOP research project, the generation of the adapter plate is to be automated. This process has to be standardized. A generally valid procedure for the generation of an adapter plate is described in this paper and a type code developed for this purpose is listed.

#### **2 Flange patterns**

In order to ensure the compatibility between the flange designs and to guarantee the implementation of the robot systems across manufacturers, the interfaces must be standardised and associated standardisation introduced. If one considers the standardisation situation in the area of robot flanges, ISO ISO 9409 - 1, ISO 9409 - 2 and DIN ISO 29262 should be mentioned here[6-8].

ISO 9409 - 1, published in 2004, forms an important basis for ensuring the interchangeability of manually mounted end effectors in the form of round mechanical interfaces. The application of this standard is not only limited to industrial robots, but can also be applied to hand lifting devices such as insertion devices, manipulators or teleoperators. Here decisive characteristic values (among other things type and number of thread and fixing holes, diameter specifications) are defined, which allow a holistic description of the flange patterns.

The second part of the DIN ISO standard 9409 defines the main dimensions, the designation and the marking of shafts of the mechanical interfaces of industrial robots. In contrast to IS0 9409 - 1, the IS0 9409 - 2 is designed for relatively low load capacities and is suitable for applications in which end effectors move in small gaps. Furthermore, DIN ISO 29262 defines microscopic interfaces for end-effectors and handling equipment in the field of micro and precision engineering. In addition to the mechanical interface, the position and classification of electrical and fluidic coupling elements is also included.

A closer look at the robot systems available on the market reveals that the ISO 9409- 2 and DIN ISO 29262 standards are rarely used. This could primarily be due to the fact that the market for robots with such low load capacities serves a niche in which special designs are used for the respective application, in which the adaptation to the respective robot flange does not represent a large additional expenditure.

On the other hand, the ISO 9409-1 standard is widely used. Studies of LPS carried out that about 81% of all flanges of the seven largest robot manufacturers (ABB, Kuka, YAMAHA, Denso, Kawasaki, Mitsubishi and Yasakawa) are designed according to this standard. Thus, a relatively high degree of standardization can be observed here. The situation is different for the end effectors. A look at the five largest manufacturers (SCHUNK, Schmalz, Destaco, Festo and FeRobotic) shows that only 21 % of the end effectors comply with the standard.

It is noticeable that the Cobot gripper flanges in particular exhibit a very high degree of standardization. Reasons for this could be, among others, the fundamentally higher degree of standardization in the area of human-robot collaboration, the increased safety requirements and a higher requirement for weight and design, which does not permit an additional adapter plate, which would mean additional weight.

In conclusion, it can be said that only ISO 9409-1 is used in industry. This standard is mainly used in the area of robot arm flanges, but only to a small extent for end effectors. Furthermore, this standard only defines round flange patterns, where rectangular flange patterns are also available in significant numbers on the market. This means that there is neither a generally valid description of robot flanges nor does it consistently comply with the existing standards. As a result, there is an inconsistent image on robot flanges, both on the robot side and on the end effector side. Thus, adapter plates are still an important tool in the field of robotics to connect these components with each other. At the same time, it becomes clear that for efficient and automated generation, a clear design rule must be generated.

#### **3 Automated adapter plate generation**

Within the scope of ROBOTOP it is necessary to be able to offer a total solution of a robot system for the customer, whereby primarily industrial robot solutions were concentrated, which cover small to medium sized payloads. The weight of the adapter plate as well as the extension of the accessibility by the adapter plate should already be considered in the design of the robot system. Thus, the feasibility of the robot system can be better validated at a very early stage, which leads to a reduction in the engineering effort and thus to a shortening of the implementation times of the system.

The basic procedure for generating an adapter plate is shown in Fig. 1. Here it is first necessary to select a robot system according to the requirements of the SME. The ROBOTOP configurator can be used for this purpose, for example. Then the flange images have to be defined. Here the contact surfaces of the flange and the boron extinguishers for fixing the adapter plate or the end effector have to be defined. The information can be taken either from construction drawings, CAD models or, in the case of standardized flange pictures, from the product description. This step decides whether or not an adapter plate is required for compatibility of the selected components. In preparation for generating the adapter plate, the requirements for the adapter plate must be defined. Afterwards the automated generation of the adapter plate can be performed. An iteration then searches for a new selection of the robot system and repeats the generation process. Finally, an optimized selection of possible robot system configurations is suggested. In this paper we will concentrate on steps two to four.

**Fig. 1.** Procedure for iterative adapter plate generation.

There is currently no standardization and no tool that would describe or implement the generation of adapter plates. In order to be able to offer an efficient solution for manufacturer-independent and non-standard flange designs, a clear definition of the construction instruction for adapter plates is necessary. Building on this, the corresponding geometry can now be generated using Constructive Solid Geometry.

#### **3.1 Design specification for the generation of adapter plates**

The unique construction instruction forms the basis for the effective and automated generation of an adapter plate. As already mentioned, there is currently no standardization in this area that could be used for this purpose, so it is necessary to define this design rule. A generally valid definition was made in the context of the research work on the LPS, from which the type code shown in Figure 1 results. In this case, adapter plates that connect a robot arm with a gripper were initially used. Furthermore, additional kinematics or sensors in the adapter plate were not considered.

Each adapter plate consists of two or more flanges which have drilled holes for mounting and a structure for connecting these flanges. Optional feedthroughs or brackets for power and information interfaces can be attached to the worktop. All the relevant parameters are briefly listed below.

**Fig. 2.**&RQVWUXFWLRQLQVWUXFWLRQIRUDGDSWHUSODWH

Flange patterns can be categorised as round or rectangular. This is primarily determined by the parameter F. The next step is to define the boreholes, so these are first to be defined in the number (A) and their arrangement on the flange plate (N). Depending on the arrangement, the position and diameter of the holes must then be specified (B, T, A). Thus the flange patterns are defined holistically, whereby now the position and orientation of these to each other must be determined (X, Y, Z, RX, RY, RZ). In order to connect these flange patterns, the characteristics of the structure (S) and the course of the connecting structure in space (X, Y, L) must now be determined. This makes it possible to define the connection structure in such a way that interfering contours which the area of application of the robot entails are taken into account, thus extending the accessibility and versatility of use for parts which are difficult to access. Finally, the type (V) and positioning (P) of the required energy and information interfaces must be defined. A database with conventional power connectors and hydraulic connections is used for this purpose. When positioning, the user only defines the side of the position of the connections. The exact positioning and the required holes or grooves for fastening these holders or bushings are set automatically. Defined space requirements on all sides of the plugs or holders are defined by boarding boxes.

#### **3.2 Generation of geometric flanges using constructive solid geometry**

In order to compute appropriate geometries, we introduce parametric geometries. For this, we define an XML syntax that allows the specification of constructive solid geometries. The syntax specifies three different kinds of objects: solids, mathematical operations, and loops.

Solids are defined by a region within the x/y-plane and an extrude operation. By now, we have two operations implemented. The first one extrudes the specified region linearly in z-direction. The second one revolves the region around the x-axis. In addition, any solid can be moved and rotated arbitrary in space.

Based on the solids, we can create more complex geometries by using mathematical operations. Merge operations are used to combine two or more solids to a single one. Subtract operations will remove one or more solid from another one.

Loops can repeat both solid creation and mathematical operations with a given number of times. An accumulating transformation specifies a relative offset for each interaction step.

```
<?xml version="1.0"?>
<ParametricXML xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
               xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Parameters>
     <Parameter Name="Radius" Symbol="R" Value="100"/>
     <Parameter Name="HolePositionRadius" Symbol="R1" Value="80"/>
     <Parameter Name="NumberOfHoles" Symbol="N" Value="6"/>
  </Parameters>
  <Difference>
  <Color Value="#008800" />
    <Extrude Length="20">
      <Path StartX="-{R}" StartY="0">
        <ArcTo X="{R}" Y="0" CenterX="0" CenterY="0" ClockWise="false" />
        <ArcTo X="-{R}" Y="0" CenterX="0" CenterY="0" ClockWise="false" />
      </Path>
    </Extrude>
  <Loop Count="{N}">
     <Displacement RZ="360/{N}" />
     <Union>
        <Extrude Length="10">
           <Transform X="{R1}" />
           <Path StartX="-5" StartY="0">
             <ArcTo X="5" Y="0" CenterX="0" CenterY="0" ClockWise="false" />
             <ArcTo X="-5" Y="0" CenterX="0" CenterY="0" ClockWise="false" />
           </Path>
        </Extrude>
        <Extrude Length="10">
           <Transform X="{R1}" Z="10" />
           <Path StartX="-10" StartY="0">
             <ArcTo X="10" Y="0" CenterX="0" CenterY="0" ClockWise="false" />
             <ArcTo X="-10" Y="0" CenterX="0" CenterY="0" ClockWise="false" />
           </Path>
        </Extrude>
     </Union>
  </Loop>
 </Difference>
</ParametricXML>
```
**Fig. 3.** XML example for a parametric flange.

**Fig. 4.** Image of the geometry specified by the XML document depicted in **Fig. 3**.

For all three kinds of objects, parameters can be specified. Hence, we can define a parameter-based template for various geometries. By changing the parameters, we can influence the shape in a simple manner. Of course, there are constrains in the construction possibilities. The drawing plane for new regions is always the x/y-plane, for instance. However, the specification is made with respect to simplify the parsing and computation.

Fig. 3 shows an example for a simple flange. A web-service accepts the document and computes the geometry. We have implemented various export options, e.g. STEP, JSON geometry format, and 3D view embedded within an HTML document. Fig. 4 shows the computed geometry.

**Fig. 5.** Image of a complex adapter plate geometry

With the web-service, besides such relatively simple geometries, almost all possible geometric forms are conceivable. The potential of the service is demonstrated by the exemplarily generated geometry in Fig. 5. This adapter plate should connect a gripper with a rectangular flange and a robot with a round flange. This flange, with its extension and angulation, serves to keep the relatively thick flange geometry of the robot further away from the poorly accessible handling position. This means that greater accessibility can be achieved even in confined spaces. The parametrisation happened via the XML schema described above.

In the future, the parameterization will be extended by a user interface, so that the XML schema is generated in the background. The aim is to guide the client through a user centred configuration process that is as simple as possible.

In addition to the generation of adapter plates, the web service is so generically structured that it can also be used for the generation of various geometries in the sense of a micro service architecture. Overall it is currently used to generate simple workpieces on the ROBOTOP platform[9-10].

Furthermore, optimization potential was seen in the geometry of the adapter plates of low stressed areas, which are to be generated in the future on the basis of the finite element structure synthesis.

#### **4 Results**

In conclusion, an early automated generation of adapter plates reduces the engineering effort and enables the consideration of this adapter plate already in the early planning phase of a robot system. In this context, the dead weight and the extension of the accessibility of the robot can play an important role in the design of the system.

An automated generation of such adapter plates was implemented as an example.

In the case of automated generation, a generally valid description of these plates and thus also of the flange images is necessary. At present, however, such standardisation is lacking in the areas of robot flanges, gripper flanges and adapter plates. A generally valid design specification for robot adapter plates was developed in the context of this paper. For the generation of the CAD geometry, an automated generation was realised on the basis of the developed construction regulation, medium Constructive Solid Geometry. Here the developed web-service is fed by means of an XML based syntax and allows an almost unlimited free space in the geometry design.

In this context we were able to prove, that the developed design rule for the definition of an extended adapter plate is working well. The generation via CSG turned out to be practicable.

The parameterisation of the adapter plate is to be extended in the future, by a specially developed user-centred interface and then integrated into the ROBOTOP platform.

Momentarily, the optimisation of the adapter plate geometries still poses a difficulty, here a finite element structure synthesis could be carried out with the aim of reducing the weight of the adapter plate.

#### **Acknowledgements**

The research and development project ROBOTOP is funded by the Federal Ministry of Economic Affairs and Energy of Germany (BMWi). It is part of the technology program "PAICE Digitale Technologien für die Wirtschaft" and is guided by the DLR Project Management Agency "Information Technologies / Electromobility" in Cologne[11].

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Fragmentation in Reconfigured Real-time Production Networks

Christian von Arnim, Armin Lechler, Oliver Riedel, and Alexander Verl

University of Stuttgart, Institute for Control Engineering of Machine Tools and Manufacturing Units, D-70174 Stuttgart, Germany christian.von-arnim@isw.uni-stuttgart.de https://www.isw.uni-stuttgart.de/

Abstract. A reconfigurable production is able to adapt to a new product in a short time. This can be built up from cyber-physical production systems (CPPS) which exchange various data with several devices simultaneously via a convergent network. In these future production systems, convergent networks will connect more participants than real-time production networks do today. In order to enable uninterrupted operation, reconfigurations must not affect existing communication. Due to this limitation, fragmentation of the configuration can occur, restricting future operations. This paper investigates fragmentation caused by reconfigurations using a simulation to determine whether fragmentation can lead to problems in future production networks.

Keywords: real-time production network, fragmentation, reconfiguration

## 1 Introduction

The industry 4.0 production must be able to produce a wide range of variants up to batch size one and at the same time execute the production in the shortest possible time [1]. In order to achieve this, [1] and [15] propose a division into cyber-physical production systems (CPPS) consisting of a set of sensors and actuators integrated into a network. In the context of assembly, this corresponds to a switch from inflexible conveyor belts to automated guided vehicles (AGVs) [2]. CPPSs enable a reconfigurable manufacturing system that produces different products and can be adapted within a shorter period [16] instead of a flexible manufacturing system that can produce different variants of a product.

This concept does no longer fit into the automation pyramid, with clearly divided field and control level for each manufacturing system [4, 11]. Real-time communication occurs only at the lowest levels in locally limited real-time production networks (RTPNs) (e.g. within a machine), whereby sensors and actuators communicate with a single controller. The concept of levels dissolves during the transition to a CPPS-based production environment. The communication across levels increases so that, for example, an AGV independently accepts the next available production order. In parallel, the values of sensors are not only used for control, but are also stored and evaluated by applications, for example in order to improve the early detection of errors. Instead of separation into levels, this results in a network of different applications that communicate with each other. Real-time communication is no longer limited to local networks and spans the entire network from CPPS instead, as shown in Fig. 1 [15].

Fig. 1: The automation pyramid dissolves and separated RTPNs turn into one large RTPN. Adapted from [15].

Two possible network architectures that realize a CPPS-based automation are explained in [7] and [8]. Both topologies integrate all participants into one RTPN, which can be implemented with the vendor-independent time-sensitive networking (TSN) extensions to the IEEE 802.1 standard, for example [10]. The control is done in local cloud platforms, which execute applications for control. [8] also explains that this control can be executed on external cloud platforms. The network architecture is shown in Fig. 2.

Fig. 2: Network architectures in CPPS-based production systems.

The main difference to the previous architecture of different, isolated RTPNs is that all network participants are contained in a shared RTPN. This results in a RTPN with more participants than before, which can span an entire shop floor. Due to the local separation of the sensors/actuators and the controller, the data must be transmitted over several switches, which must be configured in such a way that all real-time critical data reach their destination on time.

Due to its increased number of devices and new flexibility, changes may occur more frequently to this RTPN that require a new configuration of the switches. Examples for changes are the addition or removal of devices from the network. Furthermore, when starting a new application, such as real-time analysis of manufacturing data, additional data may need to be exchanged between the sensors/actuators and the execution platform. This must not affect the current communication. So, the rest of the production is not interrupted by the reconfiguration. In contrast, [9] only takes care during reconfiguration so that as few packets as possible fail and a modification of existing communication is tolerated.

The paper is structured as follows: In Section 2, RTPNs with minimum guaranteed latencies are introduced, followed by a description of the reconfiguration of these networks without interruption of the existing communication and the possibility of fragmentation in Section 3. In Section 4, a simulation of fragmentation is done and evaluated. Afterwards, a summary is given and possible further work is explained in Section 5 and Section 6.

#### 2 Real-time Ethernet

RTPNs require minimal latency and jitter, which can be achieved by transmission at specified time slots. In addition, the CPPS architecture requires support for multiple senders and receivers. This can be accomplished, for example, by using TSN with 802.1Qbv [6]. It is assumed that all participants have a synchronized time, for example through the Precision Time Protocol (PTP) [5]. Within TSN, data are classified into traffic classes. The real-time capability is realized by defining that only data of a certain traffic class are transmitted in a time slot. Data of other traffic classes are not transmitted in this time slot. The communication runs cyclically, whereby the synchronized clocks ensure that all participants use the same synchronized cycle. This cycle is divided into time slots and provides an exclusive transmission path for the data of a traffic class. The division can be defined per port at the switch and does not have to be identical for the entire network.

In Fig. 3 this distribution is visualized in an example. A cycle from t to t + 1 is shown. From t to t+t<sup>1</sup> only data of the traffic class 1 can be sent, from t+t<sup>1</sup> to t + t<sup>2</sup> only traffic class 2 can be sent. The same applies to the third slot. Finally, there is a slot from t + t<sup>3</sup> to t + 1 which allows to send all data. This cycle is repeated continuously.

In the following, a common cycle time is assumed for the entire network. In case of different transmission cycles a hypercycle with cycle time of the least common multiple of all cycle times can be considered equivalent [3]. Also, only the traffic sent in a defined time slot is considered as real-time traffic in this article. Prioritized traffic, which is sent only prioritized to other traffic, is not considered separately and is seen as part of non-real-time traffic.

Fig. 3: A cyclic transmission with reserved time slots (1 to 3) guarantees that only traffic of specific class is transmitted within this period.

## 3 Reconfiguration of real-time production networks

As mentioned in Section 1, due to the larger RTPN and new possibilities to start/stop applications that access real-time data across the network, changes in the configuration of the RTPN are more likely. A reconfiguration adds and/or removes any number of time slots for real-time data. Existing time slots for real-time data cannot be modified (e.g. moved or split), as otherwise guarantees regarding latency and jitter could be violated [12]. This section points out which conditions must be met to reconfigure a network without disturbance to the existing real-time traffic. Afterwards, the problem of fragmentation is addressed as a consequence of the reconfiguration.

#### 3.1 Execution of a reconfiguration in a real-time production network

When executing a reconfiguration that is not intended to restrict the functionality of the entire network, the possibilities of reconfiguration are limited in contrast to a fresh configuration without existing time slots. Time slots without real-time data do not have any requirements. These can be moved at will, shortened by time slots with real-time data (inserting a time slot at the edge) or interrupted (inserting a time slot in the middle).

In general, changing the configuration can be done in two steps. First, all time slots that are no longer required are removed from the current configuration. Afterwards, the newly added time slots are inserted into the available time ranges in the cycle that are not yet occupied with real-time data. Possible incremental algorithms that support the placement of new time slots are, for example, [12] and [13].

#### 3.2 Fragmentation

When removing and inserting time slots of different sizes, gaps may occur between time slots because existing time slots remain in the cycle at their previous time during reconfiguration. Within the context of RTPNs we define fragmentation in a communication cycle when the reserved time slots have gaps between each other so that the remaining free space in the communication cycle is not available as a single block. This can cause problems when adding a larger time slot and there is no sufficiently large gap available even though the total sum of time slots is less than the cycle time. This is similar to external fragmentation of storage systems, when files have been removed and added over time causing separated blocks of free memory [14]. However, the strategies to reduce external fragmentation (e.g. compaction) on storage systems can not be applied to fragmentation in RTPNs, as moving existing time slots within a cycle may violate guaranteed latencies. Fragmentation in the context of RTPNs is not the same as IP fragmentation, when IP packages are splitted during transmission.

An example of fragmentation is shown in Fig. 4, which visualizes a sequence of reconfigurations. At the beginning (step 1), there are five time slots, which occupy the whole cycle. Then, the time slots 1 and 4 are removed and the smaller slots 6 and 7 are inserted at their position (step 3). After removing the time slot 3, there are gaps between all existing time slots in the resulting cycle so that time slot 8 cannot be inserted although there is enough space in total.

Fig. 4: Example for fragmentation during reconfiguration. The time slot 8 can not be scheduled, although the total space is enough.

This demonstrates how fragmentation can cause problems over time, as small gaps between time slots cannot be used for larger time slots. To what extent and after how many reconfigurations this problem occurs is evaluated in the following using a simulation.

#### 4 Simulation

In the simulation, fragmentation is evaluated based on a real-time connection between two network participants by removing and inserting multiple time slots of different sizes. It is not considered that the time slot in a network must be reserved on the complete path from the sender to the receiver. Fig. 5 shows this conflict. A time slot for data from switch A over B to C must be available in both A → B and B → C. On the A → B route and on the B → C route there is enough space. However, on A → B the new time slot can only be inserted in the middle, while on B → C it can only be inserted at the end. Consequently, there is no time for the new slot that is available on both A → B and B → C.

Since a solution for scheduling in networks is NP-hard [12] and different algorithms produce different scheduling results, only a single connection is con-

Fig. 5: A time slot across multiple switches requires free space on each connection at the same time. (Transmission time expected to be zero)

sidered in the simulation (e.g. the A → B connection). With this simplification, the results are expected to be better than in a network with multiple switches.

The simulation considers a cycle that is reconfigured several times. For the simulation a 100 Mbit network equivalent to current field buses is assumed, which exchanges data with a frequency of 1 kHz. Thereby, 12500 Bytes can be transmitted per cycle including the overhead of the protocol (e.g. Ethernet header). Below is a list of all the parameters of the simulation, some of which have been chosen identically to enable a comparison:


The simulated reconfiguration sequence is shown in Algorithm 1. Each time a new time slot is added, its size is randomly determined from a parameterized interval. During initialization, X time slots are inserted into a cycle whose size for this simulation is set to 12500Bytes. The time slots are all inserted into the cycle one after another so that there are no gaps between them (as shown in Fig. 5 in B → C with four time slots). Afterwards, the following steps are repeated: First, Y random time slots are removed. Then, Y random time slots are inserted into the existing gaps again. If several gaps are large enough for a time slot, the smallest gap is selected in order to have larger gaps available for any following, larger time slots. The two steps of removing and adding time slots are performed until either the time slots can no longer be inserted or the maximum number of iterations has been reached. Up to 10000 iterations are executed in this contribution.

Insert X time slots next to each other; for i = 1 to 10000 do Remove Y random time slots; Try to insert Y random time slots with random sizes, each time slot uses the smallest available gap; if Insert failed then break;

For each parameter set (P) listed in Table 1, 1000 runs with different random sized time slots are performed. Each run contains up to 10000 reconfiguration iterations. The first parameter set (P1) of the simulation uses 40 time slots (X) of a size of [249 : 250] and replaces one time slot per round (Y ) so that the interval of 12500 is occupied between 79.68 % and 80.32 % in total. The result is shown as a histogram in Fig. 6. The histogram uses a bucket size of 100 so that the blue bar on the left represents the 133 simulations that failed within the first 100 iterations. The red line represents the total percentage of simulations that failed until an iteration, e.g. at 10000 it has a value of 93.5 %. Thus, 935 from 1000 runs of P<sup>1</sup> failed before iteration 10000.

As a result, only 65 runs out of 1000 were capable to process 10000 iterations, while 25 % of the runs were not able to reach 200 iterations. Transferred to production, this means that in case of this parameterization, in 25 % of the cases no 200 reconfigurations of the network can be performed without the requirement of moving previous scheduled time slots. But depending on the initial configuration and on the required changes, it might also be possible to reach over 10000 reconfigurations.

Fig. 6: Histogram of 1000 runs of parameter set P1.

Table 1 lists all simulations performed with parameters and results. All simulations have in common that they use 80 % of the available cycle (1000 of 12500) in average. The first column is the number of the parameter set. The next three columns contain the parameters used in that simulation, the number of removed and inserted time slots per iteration (Y ), the range of time slot sizes and the total number of time slots (X).

The other columns show evaluated results of the 1000 runs per simulation. "Min./max. iterations" is the minimum and maximum number of iterations. A value of ≥ 10000 means that the simulation has stopped after 10000 iterations. "Failed" contains the count of the 1000 runs in total that do not reach the 10000 iterations, "passed" contains the runs that reach 10000 iterations. The last two columns show the iteration when 25 %/90 % of the runs have failed. When less than 25 %/90 % of the simulation runs failed in total, there is no value ("−"). In Fig. 6 this value is the number of iterations when the red line reaches 25 %/90 %.


Table 1: Parameters and results of the simulation

Regarding the simulations P<sup>1</sup> to P5, the maximum number of iterations decreases from ≥ 10000 to 4059 when the size of time slots varies more ([249, 251] to [245, 255]). So, wider range of time slot sizes can therefore reduce the number of successful reconfigurations. This can be explained by the strategy that the smallest possible gap is used when inserting a time slot. When the largest time slot should be inserted, in P1, only two other time slot sizes ([249, 250]) might occupy a previous removed largest time slot, while in P<sup>5</sup> ten smaller time slots ([245, 254]) can occupy a largest time slot. Thus, the chance of finding a matching gap between two time slots is the smaller the more possible time slot sizes are available.

Using smaller but more time slots in P<sup>6</sup> and P7, compared to P1, fewer runs (11 in P<sup>6</sup> and 1 in P7) failed within 10000 iterations. One explanation for this is that more iterations are needed to fill 20 % of the available space if no smaller gap is found for an inserted time slot, as the time slot is smaller. By doubling the size of the time slots (P<sup>8</sup> compared to P1), the maximum number of iterations decreases from ≥ 10000 to 1465.

Comparing P<sup>1</sup> with P<sup>9</sup> to P13, the amount of passed runs varies within a range of [35, 96]. In P<sup>10</sup> to P<sup>13</sup> the number of iterations till 25 % percent of the runs fail is higher than P1. So, increasing the number of changed time slots per reconfiguration (Y ) from one to two firstly causes more failures. From 3 on (P<sup>10</sup> to P13) more reconfigurations are possible before 25 % of the runs failed. Two effects might cause this behavior. At first, the first iteration may remove only time slots of the minimum size and only inserts slots with a greater size so that all inserted slots must be placed at the end of the previously inserted time slots. When this is done for more slots simultaneously, then more gaps are in the cycle and the buffer at the end becomes smaller more quickly. Otherwise, the more time slots are reconfigured, the more likely it is that the size of some removed time slots will match that of some time slots to be inserted. In case of 40 reconfigured time slots per iteration, all time slots are removed and inserted again, so there will be no fragmentation at all.

In summary, the simulation of a single wire between two switches identifies the following behaviors:


#### 5 Summary

In a reconfigurable production based on CPPS, new network architectures with larger RTPNs are used. The increased number of participants and the common reconfiguration of the production lead to an increased number of reconfigurations of the RTPN in operation, whereby parts of the configuration must be exchanged without affecting the existing communication. If real-time communication is implemented as a cyclic data transport, in which fixed time slots are reserved (e.g. using IEEE802.1Qbv), the reconfiguration cannot move existing time slots. This can lead to fragmentation when time slots are added and deleted during reconfiguration. Within this paper a simulation has been used to check how often a network can be reconfigured for specific sets of parameters. Depending on the parameters, a RTPN can only be reconfigured a few hundred times before gaps between existing time slots prevent a successful reconfiguration.

#### 6 Further work

On hard disks, the problem of fragmentation has been solved by running defragmentation operations that move existing files on the disk. To solve the problem of fragmentation in RTPNs equivalent concepts need to be developed and methods for modifying existing time slots during a reconfiguration are required. One solution is that some applications can be customized to tolerate scheduling the time slot to a new position during runtime within a certain range. In future publications we will try to move existing time slots repetitively only a tiny bit (e.g. only several nanoseconds) over time. This should not interrupt the existing communication and enables drifting of existing time slots to reduce gaps. When this is done during reconfiguration larger gaps for inserted time slots should be achieved in the progress.

### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Towards Spatial Databases using Simulation State Transformations

## Providing the Means for High Performance Simulation of Digital Twins in Three-Dimensional Scenarios

Moritz Alfrink and J¨urgen Roßmann

RWTH Aachen University, Institue for Man-Machine-Interaction, Aachen 52074, Germany, alfrink@mmi.rwth-aachen.de, www.mmi.rwth-aachen.de

Abstract. The growing complexity of industrial plants, enormous advances in autonomous driving and an increasingly digitized world require a quickly rising amount of virtual planning and testing using threedimensional simulations. In addition to the number of technical advances, the complexity of individual objects and scenarios is increasing, pushing the requirements on simulation technology to generate sufficient realistic data within limited time frames.

Simulation frameworks have their means to store the information necessary to represent the properties of all scenario entities – the simulation state - in their database. However, they lack simulation algorithmdependent transformations and augmentations of the simulation state that allow maximizing calculation efficiency. This paper introduces a generalized formalism to describe these transformations and illustrates how they help to bring the simulation framework closer to a specialized simulation database that stores spatial information with a focus on performance - the spatial database.

Further, this paper demonstrates the advantages of the approach with the introduction of two concrete transformations. One targets the efficient representation of three-dimensional spatial relations for industrial robots while the other allows generating different levels of complexity in the definition of materials in the context of autonomous driving.

Keywords: spatial database, simulation state transformation, high performance, digital twins, three-dimensional scenarios

## 1 Introduction

Simulation of large scale, three-dimensional scenarios has become a major demand of Industry 4.0. The introduction of Digital Twins storing complex geometric and dynamic data requires specialized databases, able to store such information efficiently. Furthermore, scenarios as Hardware in the Loop (HIL), including sensors as LIDARs, cameras or other spatial sensors, require sufficient realistic data while maintaining real-time capabilities. To meet both needs, a combination of ray tracing and rasterization algorithms must be used. Another field which demands efficient spatial data structures are dynamic simulations. Since the geometry approximations in the dynamic simulation have significant influence on precision, complex geometry approximations are required for realistic dynamic simulations. To handle collision detection for such complex geometries, efficient collision queries are a precondition to enable interactive modeling. Furthermore, increasingly specialized hardware is developed to accelerate various tasks in the domain of ray tracing, rasterization, general vector operations and machine learning tasks. To exploit these devices, the input data must be transformed into a device-specific form. This paper introduces a generalized approach to satisfy the proposed constraints on real-time performance by exploiting recent hardware acceleration technologies and efficient data structures leading to a spatial database. To reach this aim, a database transformation formalism is introduced and applied to two challenges of spatial simulation.

#### 2 Related Work and Theory

In this section the theoretical bases for this work are explained. Beginning with the idea of spatial database, continuing with a formalism to describe simulation systems and models and finally introducing a formalism to describe simulation state transformations.

The idea of spatial databases is to transform or augment the information in a database - the simulation state - with respect to hardware capabilities, performance constraints and requirements on precision in a way that either threedimensional simulation algorithms can process them more efficiently or to enable them in the first place by providing a specialized representation of the information.

In general, a simulation has the purpose to create an environment, in which a system with all its dynamic processes can be analyzed as an experimentable model in order to reach findings which are transferable to reality [10]. Moreover, since simulations can be arbitrarily complex, it is quite difficult to find a common ground or a superordinate system to describe them. One approach proposed by [11] is the formalism of Discrete Event System Specification (DEVS). DEVS can be applied to implement behaviors defined by basic system formalisms such as discrete time and differential equations. The DEVS structure is defined by

$$M = \left< U, S, Y, \delta\_{\rm int}, \delta\_{\rm ext}, \lambda, ta \right> \tag{1}$$

where


The proposed system M is in a state s ∈ S at any given time. In case no external event occurs, the system remains in its current state for a period ta(s). At the end of the period ta(s), the system output value is evaluated by λ(s) and the state changes to δint(s). For the case that an external event u ∈ U occurs during ta(s), the system changes its state to δext(u, e, s).

In order to introduce a concept of time to the DEVS, [11] defines time as t = (T, <) where T is a set and < is an ordering relation of elements of T with < transitive, irreflexive and antisymmetric [11, p. 128]. Furthermore, the trajectory ω : ht1, t2i → A with A an arbitrary set, describes the motion through the set A beginning at t<sup>1</sup> and ending at t<sup>2</sup> for every t ∈ ht1, t2i. The motion at time t is defined by ω(t). The second formalism in [11] describes an input/output system as

$$S\_{io} = (T, U, \Omega, Y, Q, \Delta, \Lambda) \tag{2}$$

where


The essential functions are ∆(q, ω) and Λ(q, x) with q ∈ Q, u ∈ U and ω ∈ Ω, which describe the state transition function and the output function respectively. In this definition, ω(t) is the input function. Figure 1 visualizes the system Sio.

Fig. 1: Input/output system

The introduced input/output system represents arbitrary simulation functions in

a generalized form. In a simulation framework, both functions map to algorithms implemented in software or specialized hardware, leading to a system or model specific behavior. However, depending on available hardware, requirements on precision or aim for closeness to reality, the specialized forms ∆ˆ of ∆ and Λˆ of Λ should be used to increase performance and to meet the required precision. The application of ∆ˆ and Λˆ also requires the adapted version ˆq of q.

To transform q to ˆq this paper introduces the input transformation δ<sup>∆</sup> of the state transition function ∆ and the input transformation δ<sup>Λ</sup> of the output function Λ as follows

δ<sup>∆</sup> : Q → Q<sup>∆</sup> ⊆ Q ∪ Ienv , the input transformation of the state transition function δ<sup>Λ</sup> : Q → Q<sup>Λ</sup> ⊆ Q ∪ Ienv , the input transformation of the output function

To augment the state q information Ienv about the simulation algorithm or the hardware executing the algorithm can be used to transform the internal representation. Transforming q can also mean to remove information irrelevant for the algorithm, while adding information relevant for the algorithm. Figure 2 visualizes the augmented system.

Fig. 2: Augmented input/output system

The last element to complete the picture is the definition of simulation state q. For a simulation framework [7] proposes the generalized model state to be defined as

$$q = (\underline{\underline{x}}, \underline{\underline{a}}, \underline{\underline{A}}) \tag{3}$$

with

$$\begin{array}{l} \underline{x} \in \mathbb{V}\_A^{n\_A} \text{ model state} \\ \underline{a} \in \mathbb{V}\_a^{n\_a} \text{ model parameters} \\ \underline{A} \in \mathbb{V}\_A^{n\_A} \text{ model algorithms} \end{array}$$

Given the theoretical basis of the DEVS, its specialized form - the input/output system - the transformations δ<sup>∆</sup> and δ<sup>Λ</sup> and the definition of q, the next section illustrates with two examples how these transformations would look like augmenting a simulation state. In both examples the aim is to increase performance and efficiency doing the special tasks of spatial collision queries and rendering materials.

#### 3 State Transformations

This section introduces two state transformations showing the superiority of using transformed states ˆq in the system functions ∆ˆ and Λˆ. The state transformation design is derived from the requirements which three-dimensional simulation scenarios including real-time, sensor simulation, rendering and dynamic simulation impose.

The first simulation state augmentation aims on increasing the efficiency of spatial collision queries, ray tracing and occlusion queries. The common basis of these methods is that they need to determine which objects or geometries are in particular region of the scene. While ray tracing has a lot of application in sensor simulations for LIDARs or ultrasonic sensors, collision queries are a major part of rigid body simulations and occlusion culling in render algorithms. All together they play a primary role in the scenarios mentioned in section 1.

Given the general definition of q from section 3, the model state x contains the information about the object geometries, poses, the physical properties such as mass, material, etc. The model algorithms A are among others: ray tracing algorithms, collision detection algorithms and occlusion culling algorithms.

Since all algorithms require an efficient representation of spatial relations in the scene, the first proposed augmentation of q adds Space Partitioning Structures (SPS) to the model state x, leading to the augmented state ˆqSP S. δ∆,SP S and δΛ,SP S are in this case the same function, containing an algorithm which creates the SPS from the geometry and pose information in x. An SPS divides the three-dimensional space into a hierarchical structure of geometry groups, which are close to each other as shown in figure 3 left. This allows highly efficient broad intersection queries due to the hierarchic structures, similar to a tree as shown on the right of figure 3, which allows skipping a vast amount of geometry intersection tests.

Fig. 3: Two-dimensional Bounding Volume Hierarchy

While there exist many SPS such as Binary Space Partitioning (BSP) [8], Octrees [1] and Bounding Volume Hierarchies (BVH) [3], not all support efficient building algorithms making use of multi-core systems, which is required when dealing with dynamic scenarios. A very efficient BVH construction algorithm was developed by [12]. It allows to build a BVH in linear time, in parallel on CPU or GPU and is therefore be used for further implementation.

BVH recursively enclose larger groups of geometries in axis aligned bounding boxes, beginning with a single geometry and ending with one root box, enclosing the entire scene. Figure 4a and 4b illustrate how the BVH encloses the objects more and more precisely as the tree depth increases. To trace a ray through the BVH, the ray is initially tested against the root box. On intersection, the tree structure is tested recursively, using the depth first approach. The two individual child nodes of each node are traversed in order of the nearest intersection. To find broad intersections between geometries, the geometry to test is enclosed in a box, which is recursively tested against the BVH beginning with the root. A leaf node intersection while testing against the BVH gives a potential collider, which needs to be tested using a narrow collision algorithm. Using an SPS many of broad range problems can be solved efficiently.

(a) rasterized (b) ray-traced

Fig. 4: BVH enclosing scene objects

Figure 5 shows how the BVH can be used to accelerate collision tests in the example of a reconfigurable working cell [6], by successively test for collision against smaller groups of objects. The Digital Twins of the robots execute the task of picking and placing work pieces into specialized machines. Those process the work pieces and the robots move the processed items to the target carriage. To determine whether the robot can pick a work piece an intersection query has to be performed. This test is split into a broad and a narrow phase. Before preforming the narrow collision test, the broad hierarchical test is performed, testing against each level of BVH from top to bottom as explained before.

The next proposed augmentation of q considers A and x, targeting the materials used to describe the properties of the objects in the scenario. The material description has influence on how sensors perceive objects and how objects are visualized. This is especially important for simulated cameras and LIDARs. However, while these sensors require very complex material definitions, others, like contact sensors, only required very basic material information. This requires a general material definition from which different levels of complexity can be derived using the output transformation δΛ,MDL. Since δΛ,MDL depends on the hardware used for rendering, Ienv contains information about its raster and ray tracing capabilities. For each derivation, the rendering functions expressed in shaders and contained in A have to be adapted depending on the complexity of the material in x and Ienv. One material definition, which allows this deriva-

Fig. 5: BVH in reconfigurable working cell

tion, is the Material Definition Language (MDL) by NVIDIA [2]. MDL allows defining arbitrarily complex materials and deriving material models with fixed complexity depending on the users' requirements. Figure 6 shows the usage of different material derivations - generated from a single material definition - in different render techniques.

(a) real-time rastered (b) interactive rastered (c) ray tracing using [9]

Fig. 6: Subset of the distilled static material model

## 4 Implementation

This section describes the implementation of δΛ,MDL using the MDL to transform the simulation state q depending on the available hardware. The transformation is required to provide the necessary information for different rendering techniques such as ray tracing or rasterization, which represent the output functions Λrt and Λra, respectively. The proposed implementation allows deriving the required material representations from a single material definition. In the chosen implementation, the rasterizer uses the OpenGL API [4] and the ray-tracer uses the Optix API [5]. Moreover, the implemented rasterization approach is able to switch between procedural and baked material rendering (as shown at the end of section). The ray tracing is realized with Optix, which a fully customizable ray tracing framework using GPU hardware and software acceleration. Rasterization is implemented in GLSL using OpenGL as graphics API.

The MDL material definition uses recursively-refined expressions to describe the surfaces, beginning with a basic layer consisting of a surface, volume, index of reflection, thin walled flag and a geometry information. Every surface expression in the MDL can be a bidirectional scattering distribution function (BSDF) leading to a structure shown in figure 7 on the left.

Fig. 7: Different MDL complexity representations

Since these BSDF can only be evaluated in a ray-tracer, the rasterizer requires a different representation. For the chosen implementation, the following expressions are evaluated during rasterization: Anisotropy, anisotropy rotation, attenuation color, attenuation distance, base color, clearcoat normal, clearcoat roughness, clearcoat weight, metallic, normal, opacity, roughness, specular, subsurface color, transmission color, transparency, volume index of reflection. To retrieve the expressions, the NVIDIA MDL distilling API is used. The distiller generates the simplified expressions for the required static material model.

The generated static material model can be used in two ways: The first is the runtime evaluation of the expressions as figure 7 middle shows, leading to materials with a resolution up to the precision of the underlying floating representation. Moreover, run-time evaluation enables the usage of procedural material, which use noise functions to generate permutations of different base material layers, resulting in a very realistic and nonrepetitive structures in the material. The second way is to evaluate the expressions within a fixed range and store the result in a texture as figure 7 right shows. This process is called baking and has the advantage that complex run-time calculation (depending on the material) are reduced to a simple texture lookup. The downside is that the baked textures have a limited maximum size, resulting in a fixed sampling frequency and region to be captured. This is especially problematic for procedural textures, since the baking removes their nonrepetitive structure.

In summary, the simulation state transformation is implemented to derive three

material representations from a single definition, leading to a flexible, target device-specific material representation.

#### 5 Evaluation

In this section, the performance of the output functions Λrt Λra introduced in section 4 is evaluated. Λra uses rasterization with either baked or run-time evaluation of the static material model. The measurement consists of two scenarios. The first one is a test scene containing simple geometry with 12016 triangles, which use MDL materials. The second one is the urban test drive scenario shown in figure 6 with complex geometries containing 3085073 triangles. To evaluate the performance the rendering time of one frame ∆t is measured. The measured results in table 1 indicate a big difference in render times between ray tracing and rasterization, which renders ray tracing impracticable on weaker GPU devices. The simulation state transformation is therefore the enabling step for the visualization on weaker devices.


Table 1: Measured frame times in ms using NVIDIA GTX 1080 GPU

#### 6 Summary

This paper introduces the idea of spatial databases as the transformation or augmentation of the information in a database in a way that three-dimensional simulation algorithms can process them more efficiently creating a spatial database. Further the idea is expressed by extending the DEVS formalism by the simulation state transformations δ<sup>∆</sup> and δΛ. Using the formalism two state transformations aiming on increasing computation efficiency and on creating hardware depended representations of q are introduced. For the latter the implementation and evaluation is described. The results gathered from the implementation of the MDL augmentation indicate the need for variable simulation state representations to support multiple levels of hardware compute capabilities. The next steps are to investigate in different levels of geometry representations such as signed distance field or voxelization to provide an even wider range of simulation state representations, enabling more devices to use spatial databases in real-time applications.

### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## A multi-domain networking infrastructure enabling the situation-specific choreography of decentralized things

Marlene Gebhard, Michael Schluse, Martin Hoppen, and J¨urgen Roßmann

Institute for Man-Machine Interaction, RWTH Aachen University, Ahornstraße 55, 52074 Aachen, Germany, gebhard@mmi.rwth-aachen.de, www.mmi.rwth-aachen.de

Abstract. The Internet of Things (IoT) is a network between physical and virtual things, software services and people. The specific services these things offer to set up value-added chains are provided by the computer resources they use – like cloud computing for computationally intensive tasks or edge computing for real-time processing. Concepts such as the Digital Twin enable things to operate autonomously and connect independently. This leads to self-initiated systems which require the development of new infrastructure concepts to combine different computer resources and various Digital Twins. This paper proposes the Smart Systems Service Infrastructure (S 3 I) – a multi-domain networking infrastructure which allows things to choreograph themselves situationspecifically in highly decentralized and dynamic environments.

Keywords: IoT Infrastructure, Digital Twin, Choreography, Value-Added Chains

## 1 Introduction

In the last decade the term Digital Twin emerged and several definitions were proposed. The glossary of platform Industry 4.0 defines Digital Twins as virtual digital representatives of physical assets [6]. More detailed perspectives focus on simulation or the networking of physical assets [2, 5, 14]. In [11] the definition is extended to a representation which includes semantics and is used as a structural element. Depending on the domain-specific development, the scope of operation offered by the Digital Twin varies. In addition to that, the question how Digital Twins can be used to set up decentralized value-added chains, where Digital Twins live, how they interact with software services and humans in a holistic approach is still being researched intensively.

In this work the Digital Twin is considered to represent one asset in the IoT. Figure 1 shows the individual components of the IoT in an hierarchical order. At the lowest level are physical assets that are connected to their Digital Twins via sensor/actuator networks. The corresponding Digital Twins as well as software

Fig. 1. The Digital Twin in the IoT. Figure based on [10].

services and Man-Machine Systems (MMS) are shown in the IoT-levels above. These things – the Digital Twins representing assets, the software services and the MMS – are the nodes of the IoT. MMS provide Human-Machine Interfaces (HMI) which enable the operators to interact with the IoT-nodes in the specific field of application.

For this, Digital Twins, software services and MMS offer various services according to their specific roles as nodes of the IoT. Applications are configured and tasks are carried out because the individual participants of the IoT network and collaborate with each other creating problem-specific configurations of IoTnodes making up the "applications". It is the purpose of all the things – and especially the Digital Twin – to connect with other things i.e. Digital Twins, software services and MMS to create such applications and perform certain tasks. This way, concrete applications are configured by selecting and networking the things provided by the IoT.

All things – Digital Twins, software services and MMS – have different requirements on the computer resources they use. For a machine which has to perceive and react to its environment with the help of a sensor network, a reaction in real-time is only possible if the processing takes place on the machine itself. However, space and energy resources might be limited. Therefore resourceintensive calculations that do not have real-time requirements are outsourced to calculation services which are reached via the IoT. This illustrates that only the combination of different computing resources enables the collective execution of a task.

The sensor-example above describes one thing's interaction with one software service. The example can be expanded to a whole network of things where a multitude of Digital Twins, software services and MMS connect with each other, thus creating applications solving different tasks. Individual things provide their functionality to diverse applications, resulting in a dynamic network which adapts situation-specifically to particular tasks. Orchestration in which the process logic is implemented by a sequence control that calls up the individual processes is no longer possible. Things act autonomously and pursue various goals independently of each other. The orchestration becomes a choreography in which the individual things contain the process logic.

Summing it up, an infrastructure is needed that allows the things to choreograph themselves using the computer resources they need. The work at hand presents such an infrastructure which we will call S 3 I in the following. For that purpose, in the following section, the state of the art is pictured. In Section 3 the requirements for and the modules of the infrastructure are presented. An example application is given in Section 4. We conclude the paper in Section 5.

#### 2 State of the art

There is a wide range of publications on the individual computing concepts (e.g. cloud computing and edge computing) [1, 8, 13, 15]. Approaches that combine the various computer resources, ranging from the cloud to edge devices, in one single infrastructure are rare, though.

Alam et al. [2] propose a combination of cloud and edge computing. They use the cloud for the Digital Twins to ensure the scalability of storage, computation, and cross domain communication capabilities. The physical layer which represents the edge of the network provides real-time sensory fusion. Two things can establish peer-to-peer (P2P) connections either through direct physical communications or through indirect cloud-based Digital Twin connections. The combination of a fuzzy rule base with a Bayesian belief network forms a decision algorithm that selects between the physical-physical interaction, the cyber-cyber interaction and a hybrid way which uses both, the physical and the cloud-based communication.

In addition to scientific approaches there are industrial approaches. The Bosch IoT Things Platform suggests an IoT Gateway Software which provides autonomy and intelligence at the edge [4]. The gateway software supports current connectivity protocols so that the IoT devices can connect to the gateway and benefit from its computing resource. On the other side the gateway offers a local abstraction and allows unified access to the connected devices and their data. Cloud Computing is used as a back end to handle tasks with advanced computing requirements.

A combination of cloud, fog and edge computing is done by Bierzynski et al. [3]. Using improved IoT hardware, edge and fog things can handle not only functional tasks but data processing and analyzing tasks like filtering and determining correlations as well. Therefore, the aim is to integrate the edge and fog things and to distribute the workload optimally on all levels (cloud, fog, edge). For the distribution of tasks machine learning is proposed. As an example the light control of a fair hall is used. Within that example things are orchestrated by a central unit with no need for self-choreography. This approach does not answer the question how the infrastructure looks like supporting heterogeneous things which act autonomously and choreograph themselves.

All in all, from the authors point of view, concepts to combine different computing platforms exit in principle but there are no infrastructural concepts which allow autonomous things distributed over the different areas of the IoT (cloud, edge, etc.) to choreograph themselves.

#### 3 Smart Systems Service Infrastructure (S<sup>3</sup> I)

To allow highly decentralized things which use various computing resources to choreograph themselves there are several requirements for the infrastructure these things are using.


The S 3 I answers these requirements with different modules (Figure 2), which are explained in more detail below.

#### 3.1 Directory

The directory is the infrastructure's core. It provides a central metadata collection for the network's things, which can be searched for, created, modified and deleted according to the client-server principle.

To ensure confidentiality of this data the directory has to provide authentication and authorization of the thing or human-being (called user in the following)

Fig. 2. Structure of the Smart Systems Service Infrastructure. IdP = Identity Provider.

accessing the directory. The authorization part uses password or token authentication to assure that the user is the one he claims to be. Once this is verified, the directory gives access to users on the basis of their respective roles. Access can be restricted or granted down to individual directory entries and is divided into read and write permissions.

In addition to the confidentiality requirement, the directory also enables interaction between the things. Therefore, the directory does not only collect the things data but also provides their communication interfaces via so-called endpoints. The endpoints represent the interfaces of the things through which they make data and services available for other things. To be communication protocol-agnostic, the directory offers all data which is needed to access these interfaces. For example, this additional data might include a topic which is used within the MQTT protocol.

Apart from the protocol-specific information on the endpoints, information on the data behind the endpoint has to be available. This information indicates the structure of the data and the format in which it is presented and using the data model introduced in the following. Additionally the directory enables the endpoints to administrate data structure and data format themselves. That way, dynamically changing data models within the things as well as dynamically changing thing configurations can be accommodated for. This also allows the introduction of undisclosed things.

Data model Figure 3 shows the conceptual data model for the S 3 I directory. The root represents the entry point to the data and comprises the managed entities and things. Entities are users or groups of users that own, manage or otherwise relate to one or more things of the IoT. For the sake of clarity, user and group details are omitted. Things can be associated with a location to allow for spatial queries (e.g. find all nearby assets in a certain radius). Furthermore, each thing can be associated with communication endpoints for access through connectivity protocols like OPC UA, MQTT, HTTP or S 3 I broker. Things might also be hierarchically structured with other (child) things that are separately managed within the S 3 I directory. For example, a machine thing might consist of several machine child things like an engine or a chassis. Finally, the structure of a

Fig. 3. Data model of the S<sup>3</sup> I directory.

thing can be disclosed in more detail within the directory. For that purpose, the utilized data model can be specified. In more detail, (parts of) the thing structure can explicitly be exposed using a generic object model built from objects, values and links between objects. This allows to associate endpoints directly to the object or value they provide access to. This simple object model serves as a meta model on which most data models can be mapped thus supporting arbitrary application domains in the S 3 I directory. Providing only meta data results in low overhead and reduced computational requirements, so that the infrastructure can be set up using low-power devices only.

Implementation From a technical point of view, the directory is a database that provides an API and takes care of authorization. This set-up can be provided by open-source tools like Eclipse Ditto. Eclipse Ditto is an open-source framework for Digital Twins in the IoT. Ditto focuses on solving the responsibilities a typical "back end" has in IoT solutions [7].

To store the data, Eclipse Ditto uses a MongoDB. Besides the data of the directory entries, Ditto also stores user information to authenticate and authorize the user. The modeling of the users as entities is shown in Figure 3 and mentioned above. The authentication of the users is done by user credentials or JSON Web Token (JWT) which correspond to a digital ID card. Authorization takes place via so-called policies, which are assigned to the corresponding directory entries of the things. A policy lists the specific roles and defines the resources the respective roles can access. The resources can be whole entries of the directory or parts within the respective entry. For every resource read or write permission can be granted or revoked. Using a JWT, Ditto authenticates the user and separates its role (e.g. owner or administrator) from the JWT. In combination with the policies, the access rights to a requested entry are described.

Ditto provides a REST API which offers the possibility to configure the policies as well as the specific entries. The computing resources on which the directory is based must therefore have IP-capable interfaces. Besides that, the directory can be set up anywhere, no matter if it is a cloud or an edge device.

#### 3.2 Identity Provider

While authorization in the directory is done using these policies, for the authentication the user (i.e. a thing or an human-being) has to specify his data with user credentials or a token.

Using user credentials, thus password authentication, the directory would have to store the user credentials of each user. On the other side the user has to have the corresponding password for the directory ready each time another thing is invoked or some other call is made. In a network of services and Digital Twins where a lot of access points are used this is not convenient. Also, the user data, as very confidential information, would be sent along every time a request is sent.

For these reasons the S 3 I provides a central Identity Provider (IdP). The IdP manages the user credentials like passwords and certificates. Each user has an identity within the IdP, which is valid for other services at the same time. The identity is carried in form of a digital ID card, which is implemented as a JWT, as explained above. This digital ID card is the access key for the various things within the S<sup>3</sup> I and authenticates the user.

The login of the respective users is done with OpenID connect, which provides an authentication layer on top of the OAuth 2.0 authorization framework. OpenID connect completes the OAuth 2.0 authorization flow with authentication mechanisms [9].

#### 3.3 S<sup>3</sup> I Broker

With the Directory and the Identity Provider participants are given the ability to make their interfaces securely accessible to others. The actual data transmission is handled by the participants themselves. To provide one message-based communication option the S 3 I offers an Advanced Message Queuing Protocol (AMQP) broker.

AMQP supports a request/response and publish/subscribe architecture. In cooperation with the messaging broker the protocol ensures a robust data transfer and allows asynchronous communication by enabling messages to be stored in queues. Within the broker an exchange receives the messages and routes the data into the correct queue. The queue, in which the message belongs, is determined by a binding. The exchange also offers broadcasting of messages and topic-specific forwarding. Although the exchange types and the bindings offer a variety of communication architectures there can be custom exchange types like geodatabase routing.

### 4 Application

One field of application is the forestry and timber industry. It offers a multitude of different stakeholders and a heterogeneous structure [12]. In addition to forest rangers, forest workers and forest owners, facilities such as sawmills and forest machine manufacturers operate in the cluster. Regarding machinery there are forestry machines like harvesters, forwarders and skidders as well as hand-operated machines like motor saws and brushcutters. Furthermore there are services for forest growth simulations and tree species classification.

The machines might have real-time applications running on the machines themselves thus representing edge computing. Services like forest growth simulation might run in the cloud using the computing power provided here.

These things build a highly decentralized network with heterogeneous computing resources. Still they act independently and choreograph themselves to perform solutions in constant exchange.

One example to illustrate the core concepts of the S 3 I is the communication between a sawmill and a harvester. The sawmill determines its need for a specific amount and type of wood. The aim of the sawmill is to place this order with a harvester. Optimally, an order to a harvester which is currently in a suitable forest area and can process the order is created. The sawmill logs into the directory and queries the harvesters within a specific area using a location based search. The directory entry which is shown in listing 1 provides information on the endpoints offering the current work status of the respective harvester. After choosing a suitable harvester the sawmill queries the directory entry of the harvester and uses the denoted endpoint to place its work order.

This example illustrates how the directory is used. Using the REST API, things can query the directory and search explicitly or categorized for things. Each thing can indicate its interfaces declaring endpoints in the directory which is reduced to an AMQP endpoint in the example. Giving information about the exchange and queue which is used in the example, a communication partner is provided with the necessary information about the interface.

With the login at the Identity Provider, the user gets an ID token to authenticate at the Directory, which allows access to the information about the AMQP endpoint of the harvester. This assumes that the harvester has listed the sawmill in the directory as an authorized party before. Since the harvester controls its policies (e.g. using the API of the Directory), it determines the users of its interfaces by itself.

```
1 { " name ": " HarvesterJD_OPS ",
2 " identifier ": " tzccokgc - resc - glpodnchfgtq ",
3 " location ": {
4 " lat ": 5 1.4 5 4 6 0 2,
5 " long ": 8.0 0 6 5 2 6
6 },
7 " objects ": [
8 { " class ": " workOrder ",
9 " identifier ": " tzccokgc - resc - rpkmxnwurzgd ",
10 " endpoints ": [
11 { " type ": " sssiBroker ",
12 " uri ": " amqp :// sssi / forestAuthority ",
13 " queue ": " harvesterJD_OPS ",
14 " exchange ": " sssi "
15 }
16 ]
17 }
18 ]
19 }
```
Listing 1: Directory entry of the harvester.

### 5 Conclusion

This paper presents an infrastructure for highly decentralized things. Since these things require diverse computer resources and connect autonomously and situation-specifically to applications, an infrastructure is needed which supports this choreography and which allows to use the computer resources meeting best the requirements of each individual thing. The requirements are specified and further the modules of such an infrastructure are presented. To allow situation-specifically choreography a directory is proposed, which offers thing- and user-specific access to the information needed for the thing to interact. More precisely, the things describe their interfaces and indicate which other things are allowed to access to the respective interfaces.

Since the directory is reachable via a REST API, the things are offered a convenient way to manage their interfaces and the access to them. This way, the participants are independent of the computing resource they are using. Offering gateway services, things without IP-based communication can be included.

With the Identity Provider a single entity is given which issues identity tokens to provide a comfortable authentication at the services, user interfaces and Digital Twins within the network.

Further work will determine the computing resources that the S 3 I uses. Cloud computing might be convenient for applications where no other computer resource is available that is accessible from all over the respective network. Still, there might be applications that prefer computing resources at the edge level, dealing with privacy issues.

## References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Data Acquisition and Classification of Best Practices for the Configuration of Robot Applications**

Florian Hoffmann1 , André Hengstebeck1 [0000-0002-9142-2254] and Jochen Deuse12 [0000-0003-4066-4357]

<sup>1</sup> Institute of Production Systems, TU Dortmund University, Leonhard-Euler-Straße 5, 44227 Dortmund, Germany <sup>2</sup> Advanced Manufacturing, School of Mechanical and Mechatronic Engineering, University of Technology Sydney, Australia

florian.hoffmann@ips.tu-dortmund.de

**Abstract.** Despite the far-reaching technological developments in the field of robotics (e.g. lightweight robots), the degree of implementation of innovative solutions in industrial practice is still comparatively low. This applies in particular to inexperienced users from small and medium-sized companies (SME). To facilitate the transfer of robot applications into practice, an internetbased configurator has been developed within the ROBOTOP research project. The configurator comprises planning functions along the whole engineering process. With regard to these functions, industrially implemented best practices are a main focus within the concept phase. Furthermore, they form a foundation for all further planning steps. In this context, the term 'best practice' refers to successfully implemented industrial applications. The acquisition, structuring and integration of these best practices into the web platform constitute the core of this paper. In order to highlight the emerging considerable benefits for industrial end-users, the digital implementation of best practices within the framework of the ROBOTOP internet platform is presented in the last step. The continuous use of digital planning platforms can ultimately pave the way for a significant increase in the degree of implementation of innovative robot applications.

**Keywords:** Best practices, data acquisition, classification, robot configuration, Industry 4.0

### **1 Introduction**

In the field of robotics, extensive research results have emerged in recent years. Despite the considerable potential of new technical achievements, e.g. lightweight robots, these systems often have not yet found their way into industrial practice. One reason for this is that the planning and configuration process requires industrial users to have extensive prior knowledge [1].

In order to counteract this problem, a modular and open internet platform with engineering functions along all phases of automation planning is developed in the ROBOTOP research project. The aim is to use industrially implemented best practices as a foundation for a need-based planning process [2, 3]. On the basis of this planning process, inexperienced industrial end-users are introduced to the field of robotics. In this way, the mass market for industrial and lightweight robots will be opened up in the future [4, 5].

#### **2 Planning Approaches**

In order to automate a production process by means of industrial or lightweight robots, various planning steps are necessary. In this context, numerous planning approaches were developed within the last years. These can be divided into application-neutral approaches [6], assembly-related methods [7] and automation-related systems [8].

Application-neutral planning methods are characterized by a more general description of the planning contents which makes them suitable for a wide range of applications. The REFA planning method was developed in the form of a guideline and can be used by industrial users for work design in general [9]. The method is suitable for various application scenarios that can relate to production as well as to the areas of service and administration. However, the main weakness of application-neutral methods lies in the lack of a connection to the field of robotics, e.g. configuration of robot system and safety requirements.

Assembly-related approaches are particularly suitable for detailed planning of work and assembly systems. In this context, Hartel and Lotter [10] developed a method for the planning of production lines from the assembly area. A special feature of this method is its high flexibility, as the individual product and production parameters of the examined application scenario can be varied, e. g. product design, product complexity and sales volume. Another example is the reference planning process [11] which can be used to plan various industrial scenarios, such as production, logistics, assembly or layout planning. Moreover, the Bosch planning guideline is based on lean production and scalable automation concepts and supports industrial users in all phases of production planning [12]. Overall, the assembly-related approaches have many advantages. However, the area of robot-based automation is not sufficiently considered.

Automation-related approaches serve to design work sequences that are carried out by humans or robot systems. A large part of these procedures is based on a skill-oriented allocation of work tasks. Here, the consideration of different levels of automation [13, 14], concept-based assembly planning [15–17] and MABA-MABA diagrams [8, 18] should be mentioned. A central criticism of these approaches is that they only consider a very specific part of the overall planning process and, for example, do not provide approaches for detailed and implementation planning.

In total, there is a large number of different planning approaches available which, however, have specific disadvantages with regard to the underlying planning task. In particular, the usability for industrial users without specific prior knowledge in the field of robotics is very limited. On this basis, the ROBOTOP configuration process for the implementation of robot applications has been developed. Here, inexperienced endusers benefit from digital assistance tools and best practice solutions so that SMEs in particular gain access to robotics.

## **3 Internet-based Configuration Process**

The aim of the internet-based configuration process is to facilitate planning procedures for both robot suppliers and customers and thus link both groups together directly. As indicated in Fig. 1, the ROBOTOP platform offers extensive assistance functions in all phases of automation planning.

**Fig. 1.** ROBOTOP configuration process

In the course of an initial selection of best practices, potential end-users first have the opportunity to describe their individual application scenarios (e. g. manual workstations) using a structured and user-friendly questionnaire. Suitable best practices are derived in an automated selection process.

The best practices serve as an initial concept proposal for the end-user and as a basis for the following individual configuration process. In this course, suitable robot systems, peripheral components and gripping systems can be freely combined with each other, whereby the compatibility of the individual components is guaranteed at all times.

To increase user-friendliness, the assembled systems can be directly transferred to an online 3D simulation. Users can make an initial assessment of the technical feasibility of the assembled solution for their specific application scenarios. Depending on the result, it is then possible to go through upstream process steps again to improve the planning results. Alternatively, potential service providers may be contacted directly to obtain quotations. However, in order to achieve such high quality results in later planning phases (e. g. individual configuration, online 3D simulation), the selection of suitable best practices at the beginning is indispensable.

#### **3.1 Fields of Application for Best Practice Acquisition**

To ensure that best practices can be used for as many planning scenarios as possible, they should represent a b road cross-section of industrial applications. Therefore, frequently occurring industrial tasks were determined and categorized.

Within the framework of a literature research, the results of various studies on the frequency of specific types of robot applications in industry were consolidated [19]. In addition, concepts and guidelines for the description of work tasks for robot systems were examined [20, 21]. The result was a heterogeneous overview of different robot operations.

This overview was revised by experts from industry and research in various surveys. As a result, nine elementary robot applications could be identified. As indicated in Fig. 2, these could be divided into four higher-level categories.

**Fig. 2.** Best practice classification

The common feature of machine loading / unloading, pick and place activities as well as positioning tasks is that individual objects are picked up, moved and placed in a different location. For this reason, the operations are assigned to the "handling" category. Furthermore, composing an object places higher demands on a potential robot system, since one object has to be inserted specifically into another. Similar to the screwing of objects, these activities increasingly occur in industrial assembly. For this reason, assembly and screwing are assigned to this category. Further typical tasks of a robot system are scanning of features and quality inspection. Since optical sensor systems are often used for this purpose, these activities are assigned to the "optical inspection" category. In addition to the handling and inspection of components, robots are increasingly used in industry for coating and machining processes (in particular cutting). These activities have been classified in the "machining" category.

Overall, robot systems are used in industry for a variety of other application scenarios (e. g. welding). However, a central requirement of the web platform is to comprise representative best practices that allow inexperienced users in SME an easy introduction to the field of robotics. This can be well ensured by the described best practices classification.

#### **3.2 Best Practice Database**

Based on the developed structure, industrial application partners within and outside the ROBOTOP research network were approached for the collection of best practices. This includes SME as well as large corporations in the automotive industry, machine and plant construction, packaging industry and air conditioning technology. This way, best practices data was collected for all four categories. However, an aboveaverage number of handling tasks occurred. These are scenarios that can be automated easily and mainly without the aid of complex sensor technology. This underlines the current industrial trend that lightweight robots are used increasingly for simple applications. In addition, there is the significant advantage that these applications offer an ideal planning guideline for inexperienced users from SME.

In order to select suitable best practices in the first step of the configuration process, it is necessary to compare the requirements of the examined user scenario with the properties of the best practices. Therefore, the best practices are described thoroughly on the basis of defined characteristics. This was ensured by the development of a uniform data sheet.

The best practice attributes include metadata of the individual components such as robots, grippers, tools and conveyors, types of material feeding and safety-related equipment. Moreover, the IP protection class, component properties, control systems as well as the upstream and downstream process are taken into account. On the basis of the generated data, ROBOTOP can provide industry-independent best practice solutions for various application scenarios. In this way, the project can significantly promote knowledge transfer to SME.

#### **3.3 Selection of Best Practices**

For the selection of best practices as a planning foundation, some information on the specific investigation scenario, e. g. manual assembly workstation, is required. This data is collected using a structured questionnaire. This includes questions relating to the work task to be examined and questions relating to the handling objects.

The required information on the work task mainly refers to the four categories presented in this paper (i.e. handling, assembly, optical inspection, machining). This allows eliminating unsuitable best practices that do not correspond to the field of investigation.

As a n ext step, information on the handling objects is required since these have significant influence on the automation capability [22]. The queried object data includes the geometric body (ellipsoid, cuboid and cylinder) as well as information on the maximum dimensions. Furthermore, data regarding component stiffness, material properties and component mass is collected. Based on this product data, further improper best practices can be eliminated. For example, the weight of the handling objects has significant influence on the payload and thus on the size and type of the entire robot system. If the weight range of the application scenario does not correspond to the best practice, the best practice is evaluated as unsuitable and excluded from the configuration process. In this way, it is possible to carry out a meaningful pre-selection and to propose suitable best practices to industrial users.

#### **4 Application**

The planning logic as well as the best practice solutions were integrated into the ROBOTOP internet platform as part of a 3D demonstrator. The digital representation contains all CAD models necessary for the visualization of the best practices. This includes material feeding components, the representation of products, load carriers and conveyor elements, the robot system with its gripper or tool components, safety engineering as well as the switch cabinet. By means of this level of detail, the ROBOTOP configurator is provided with sufficient information about the correct functionality and visualization of the production process.

Based on the requirements analysis, suitable best practices are pre-selected for the user in the first step. As it can be seen in Fig. 3, industrial end-users can view suitable best practices in a 3D environment and compare the essential properties with each other. For this reason, the configuration interface displays the corresponding attributes. This includes information on the kinematics used, the end effector and the estimated price. Furthermore, the platform includes a rendering of the scene to get an initial overview of the components used. This allows industrial users to select individually one of these suitable best practices for further configuration steps.

Furthermore, the user has the possibility to customize his robot application in the configuration mode. Various components such as workpieces, feeds or safety features are available for selection. Using the component-positioning menu, suitable elements can be selected and added to the overall system.


**Fig. 3.** Preselection of best practices on the ROBOTOP web platform

This includes working tables, conveyors, robots and end effectors. In this way, the user can select between various manufacturers and choose the right object type. Alignment can take place within all six degrees of spatial freedom. To ensure compatibility between, the configurator displays only suitable solutions (see Fig. 4).

**Fig. 4.** Individual configuration process based on a suitable best practice

As can be seen in Figure 5, it is possible to store realistic data for each added object. In this case, the configurator menu shows the parameters for a new workpiece. At this, information concerning basic geometry, dimensions, weight and material can be edited. Due to the 3D environment and clearly defined input queries, no in-depth knowledge of robotics is required, so that almost every industrial end-user can benefit from the web platform.


**Fig. 5.** Editing of realistic workpiece data

## **5 Conclusion and outlook**

Industrial robot applications must meet significant technical requirements to ensure high availability, quality and safety. In many cases, this results in highly complex applications. The successful planning of these robot systems requires a correspondingly

high level of specialist knowledge, a wealth of experience and extensive knowledge of the standardization situation. Especially for SME, this represents an entry barrier to the use of industrial robot applications.

This problem was a main motivation for the ROBOTOP research project. The web platform enables industrial end-users to go through the entire configuration process from initial concept planning to preparation for implementation. By using industrial best practices as a planning foundation, users gain a practical insight into the field of robotics. In addition, the extensive assistance functionalities of the platform and the detailed 3D representation enable even inexperienced users to run through the entire planning process without any additional help.

Overall, powerful web platforms mapping the complete engineering process are a key technology of the future. These can ultimately pave the way for a s ignificant increase in the degree of implementation of innovative robot applications.

#### **Acknowledgment**

The research and development project ROBOTOP is funded under Grant No. 01MA17009H within the scope of the German "PAiCE" technology program run by the Federal Ministry for Economic Affairs and Energy and is managed by the German Aerospace Center in Cologne. We would like to thank our project partners Infosim, ICARUS Consulting, Robert Bosch, SCHUNK, FAPS, wbk and LPS for their valuable contributions.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Cognitive Effects of Physical Support Systems:** A **Study of Resulting Effects for Tasks at and above Head Level Using Exoskeletons**

Felix Schroeter1 , Svantje T. Kähler1 , Zhejun Yao2 , Thomas Jacobsen1 and Robert Weidner2, 3

<sup>1</sup> Helmut-Schmidt University Hamburg, Experimental Psychology Unit, Holstenhofweg 85, 22043 Hamburg, Germany

<sup>2</sup> Helmut-Schmidt University Hamburg, Laboratory of Manufacturing Technology, Holstenhofweg 85, 22043 Hamburg, Germany

<sup>3</sup> University Innsbruck, Technikerstr. 13, 6020 Innsbruck, Austria

**Abstract.** In the wake of demographic change and increasing knowledge of potentially harmful activities and work areas, technical support systems like exoskeletons are used to minimize the risk of musculoskeletal disorders (MSDs) by supporting employees during tasks with high biomechanical load. In addition, the increasing quality requirement and individualized demand make production activities more complex and changeable. Not least because of the well-trained abilities and skills of humans, employees have a central role in current and future value chains. One possibility for physically supporting manual tasks is the exoskeleton. During the development and implementation of the exoskeletons, many of their effects and performances have been evaluated. Mostly, developers or end-users focus on selected biomechanical effects like reduction of musculoskeletal stress. The effects on cognitive structures as well as the impact on work quality are less known. The current study analyzed the cognitive effects during the use of the active exoskeleton Lucy for tasks performed at or above head level in a real environment (training center for scaffolders). The study was conducted with 18 participants. Our experiment was able to show that participants who were supported by the exoskeleton Lucy made significantly fewer mistakes in a work task (53,8%). In addition, a significantly higher cognitive performance (concentration achievement, processing of target objects and error frequencies) was found in supported participants, compared to the unsupported condition, in a psychometric test after each work task.

**Keywords:** exoskeleton, support system, concentration performance, resulting

effects, industrial application

#### **1 Introduction**

Despite the increasing demand in automation production tasks, humans have a central role in common, future value chains, and organizational structure [1]. The reason can be seen in the unique skills and abilities of the human perception, cognition and motor skills which are still superior to automated solutions in many areas, such as flexibility, sensorimotor skills, cognition and learning abilities, and attentive perception. On the other hand, we still have other areas, where automated solutions are advantageous, e.g. high endurance and accuracy, arithmetic operations or memory capacity.

To support human beings during manual tasks, different approaches are possible, to offer different types of support. They can strengthen the position or substitute the employee [2]. In general, such support technologies can be arranged parallel or serial to the user and support the user in physical and cognitive aspects, e.g., [3-5]. In this paper, we address an exoskeleton, which can be classified as a parallel system. Such systems are characterized by a force path parallel to the human musculoskeletal system. The use of technical support systems in the field of, e.g., production, logistics, and the handcraft may be motivated in different ways. Central drivers are the organizations or the employee. Support systems can be used to increase quality and productivity or decrease musculoskeletal stress and thus the potential for musculoskeletal disorders.

In general, exoskeletons are designed to achieve task-individual properties. In respect to the desired properties, such systems are different in kinematic structure, human-machine-interfaces and actuation, as well as (in the case of active systems) sensors and control units [5]. Depending on the basic concept of each exoskeleton, such systems can support different kinds of tasks. Typical applications of exoskeletons in industrial areas are supporting tasks at head level or above, handling loads, gripping activities or handling tools and components [6-12].

The benefit of a mechanical relief for the workforce seems most obvious. Not least because the mechanical strain caused by the nature of the work task, such as working at head level or above, or handling heavy loads, could lead to fatigue and chronic complaints in the workforce (musculoskeletal diseases) [13]. These negative consequences must be avoided at all costs in order to be able to fulfill the central requirement of the work task: to be able to work without any impairment or damage [13, 14].

At this point, it should be emphasized that working at or above head level is an unfavorable working position for people, but is justified due to the requirements of production and cannot be changed in its actual execution. This may lead to the development of pain, which can also become chronic in the future [15]. A reduction in cognitive performance, which may be reflected, for example, in impaired memory function is directly related to chronic overload as the result of permanent physical strain. In particular, these consequences are the reason for the development of technical support systems for industrial production [16].

The performance of exoskeletons and their ability to reduce physical stress during manual activities has been shown in previous research [7, 9-12]. However, their influence on worker's cognitive performance and working quality are less known. The relationship between the levels of perceived pain and reduced cognitive and physical performance have been discussed extensively in psychological literature [17]. The investigation of the impact of physical support of an exoskeleton on specifically cognitive performance a rather new field in both psychology and production engineering. Psychometric testing is necessary for the collection of objective data measuring cognitive performance. Here, the relationship between the test and the experimental objective was considered to allow a valid quantitative evaluation. Thus, in addition to measuring work task duration and therein occurring errors, the latest version of the d2-R Attention and Concentration Test was chosen as the central psychometric measuring device for data collection, which meets the above requirements. The abbreviation d2-R refers to the target item "d" with two bars that is intermixed with d, p and b with different amount of bars above and/or under the letters ("R" stands for revised). According to the manual of the d2-R Attention and Concentration Test, this psychometric test presents as valid and reliable. According to the TBS TK review (Testbeurteilungssystem des Testkuratoriums der Föderation Deutscher Psychologenvereinigungen), the d2-R Attention and Concentration is suitable for the evaluation of levels of concentration. Thus, all central criteria regarding concentration level following the work task could be recorded [18].

This paper reports a further study, focusing on the investigation of the influence of support systems on human cognitive function in a work place setting. For this purpose, participants performed a task with and without the support system. Afterwards a psychometric test procedure was carried out. A detailed description of the study design, including task setups, observed variables, statistical analysis and results can be found below.

#### **2 Study design**

The aim of this study was to analyze the effects of a physical support system on human cognitive function, especially the concentration levels, the treatment of target objects and corresponding error frequency in a psychometric test, and performance during a working task. In a previous publication it has been shown, that the exoskeleton Lucy successfully reduced physical stress [8] and improved the participants' levels of concentration [19]. To investigate whether the physical relief effect of Lucy may lead to improvements in participants' cognitive abilities within a workplace setting, testing was performed under realistic conditions in a training center for scaffolders.

Key hypotheses were:


#### **2.1 Exoskeleton Lucy**

The exoskeleton Lucy [7] was used, which is illustrated in Figure 2. It is worn like a backpack and supports the user during movements at head level and above. It uses pneumatic actuators to partially absorb and dissipate the additional workload, caused by and dependent on the tasks performed. In order to realize a natural level of support, predefined force curves are implemented in the control system. A detailed technical description can be found in previous work [7]. The user is able to change the level of support manually and individually applicable to each task. For standardization of the tests, all participants used the same level of support parameters. Hereby the level of support was set to approximately 4 kg with a shoulder angle of 90°.

Figure 2. Exoskeleton Lucy for support tasks at head level and/or above.

#### **2.2 Experimental setup**

To investigate the hypotheses, 18 male participants performed a given task in a standardized manner. Participants were all second-year trainees within the scaffolding construction industry. Due to the absence of female trainees, it was not possible to acquire a gender-balanced sample for the survey-period. Mean age was 22.55 years (*SD* = 5.79; range: 17-39 years). Participants' height ranged from 170 to 190 cm.

In order to determine the interventional effects of the exoskeleton, participants were randomly allocated to one of two interventional groups. Participants with odd participant number were assigned in Group 1 and with even participant number in Group 2. Group 1 began the work task with the Lucy system and then performed the same task in an unsupported condition. Participants of Group 2, began in the unsupported and then ended in the supported condition. Participants were instructed to take a break of 10 minutes between the two conditions, respectively. During this time, participants were able to recover from the previous work task as well as from the concentration test. Thereby equal starting points were created for each experimental condition. For the d2- R concentration test a total of 14 consecutive letter rows consisting of "d" and "p" were shown to the participants in a pen and paper test. The task was to find and cross out the letters "d", which were marked with two dashes in the standardized interval of 20 seconds for each line. After the acoustic signal "Stop! Next line!" was given, participants had to move on to the next line following the same procedure as previously. This procedure is in accordance with the manual of d2-R and took overall 280 seconds. To assess the effects of the intervention on concentration, participants performed another d2-R concentration test in both conditions immediately after completing the work task (see Table 1.).


Table 1. Experimental setup.

A standard scaffold was erected, as it is in real life settings (i.e. facades). For the experiment, eight crossbars including type-related fuses and a standard carpenter's hammer, for hammering in the fuses, were used. A complete crossbar weighed 3.5 kilograms and was 1 meter long. All participants were familiar with the materials and methods used in the work task. The work task was designed after an everyday occurrence within the scaffolding industry, consisting of inserting eight crossbars onto the highest level of the scaffold, hammering two fuses into each crossbar. The installation of one crossbar contained three steps (see Figure 1):


Figure 1. Installation steps of a crossbar at the highest installation level which is 2 meters high from the running board.

During each installation, the crossbar was held at knee, hip, chest, and overhead level. The highest installation level was 2 meters from the running board (see Figure 1). This was considered overhead level for all participants. The final installation step was perceived as the most physically demanding step by all participants, since both the crossbar and the hammer required excessive forces above head level. It was therefore chosen as the condition in which the errors during the working task (ErrorWT) were counted. Additionally, and more importantly for this study, Lucy showed to deliver the greatest level of support at overhead level.

#### **2.3 Observed variables**

To assess the impact on cognitive function after each run of the experimental conditions (supported/unsupported), measurements of concentration performance, accuracy in d2-R test processing (error frequency) and speed in test processing (detected targets) were obtained using the d2-R Attention and Concentration Test [20] as quantitative parameter. Additionally, the errors during the working task (ErrorWT) were defined as hits missing the fuse.

Concentration performance (KL) was used as measure of concentration capability. In the test this described the number of crossed-out target objects, minus the number of wrongly marked objects. The processed target objects (BZO) were determined by the working speed. It was measured by the number of processed target objects of the d2R in a given test period. Error frequency (F) was defined as the accuracy during test processing and was indicated by correctly processed items. Therefore, high values in the error frequency meant better results than smaller ones.

#### **2.4 Statistical Analysis**

Four t-tests were conducted for this study. Three were used to analyze baseline data, determining whether there were significant differences between the two groups (separated for Working Order) prior to our experiment in the three variables of the d2-R. Another was used to see if the required working time for the same task under two conditions (supported, unsupported) differed from each other. To assess the effects of the Lucy for the two factors, the method Analysis of Variance (ANOVA) was used. An ANOVA was calculated for the three dependent variables of the d2-R (KL, BZO, F) each, and one for the errors occurring during the work task (ErrorWT). Each ANOVA had the same mixed within-between design with repeated measurement on the withinsubjects factor Support (supported, unsupported) and the between-subjects factor Working Order (Group 1, Group 2). With the within-subjects factor the difference between the two conditions with and without the support of the exoskeleton was estimated. The between-subjects factor indicated whether Working Order of supported and unsupported condition (Group 1, Group 2) had a significant effect. The interaction effect of the two main factors could assess, if the effect of support was modulated by the Working Order.

The experimental data were analyzed and evaluated using the software RStudio (R-3.6.1) (RStudio, 2012). For standardization of the d2-R values, norm tables printed in the manual were used, considering the different age groups. The d2-R Attention and Concentration Test was standardized from the end of 2007 to mid-2008. The manual for the d2-R test did not provide more detailed information on the standardized period. Altogether 4024 persons with an almost balanced gender ratio (50.7% male, 49.4 female) formed the basis for the standardization [20].

#### **3 Results**

The analysis of the t-test regarding the d2-R baseline showed that there were no significant group differences as a function of Working Order for the concentration performance (KL) *t*(16) = 1.81, *p* = .09, the processed target objects (BZO) *t*(16) = 1.88, *p* = .084, as well as for the error frequency (F%) *t*(16) = 1.41, *p* = .18. The degrees of freedom (in brackets) of the t-test and ANOVA are an estimate of the number of independent information that went into calculating the estimated t-value. For the calculation of a between-group design the amount of groups (here 2) are subtracted from the sample size (18). The non-significant results of our analyses (*p* > .05) indicated that the null hypothesis could not be rejected and we could assume our groups had the same level of concentration prior to the experiment. The t-test of the time on task for conditions supported (*mean* = 150.61 s) and unsupported (*mean* = 132.17 s) differed significantly (*t*(16) = 2.13, *p* = .04).

The results of the four within-subjects ANOVA's for the variable Support (supported, unsupported) are summarized in Table 2. An effect size (ηp2) of .01 is considered as small, .06 as medium and .14 large [21]. All four dependent variables (KL, BZO, F and ErrorWT) showed significant main effects for the factor Support. Participants in the supported condition achieved better values in the dependent variables of the d2-R and lower error counts in the work task, than participants in the unsupported condition. This suggests a positive influence of Lucy on the wearers. The main effect for Working Order did not reach statistical significance in any of the ANOVAs.


Table 2. Summary of the main effects Support (supported, unsupported) of the Analysis of Variances (Support x Working Order).

The results of the interaction term of the four ANOVA's between-subject factor Working Order (Group 1, Group 2) as a function of Support (supported, unsupported) are displayed in Table 3. The results show that Working Order had a significant interaction effect on Support for the variables concentration performance and the processed target objects, but neither on the effect of support on the error frequency of the d2-R, nor on the errors in the work task. The significant interaction between Working Order and Support for the variables KL and BZO were due to the better values in the supported condition, when the participants had already executed the task in the unsupported condition (Group 2), compared to when participants started in the supported condition (Group 1, see Figure 3).


Table 3. Summary of the interaction effects (supported, unsupported x Group 1, Group 2) of the Analysis of Variances (Support x Working Order)

Figure 3. Interaction plots of (a) concentration performance (KL), (b) processed target objects (BZO) and (c) error frequency (F) of the d2-R test, as well as (d) errors during work task (ErrorWT). Group 1 had a working order of changing from supported to unsupported, while the Group 2 worked in a reversed order.

#### **4 Discussion**

To evaluate the effects of a physical support system on cognitive function in a field setting (work place of scaffold trainees), the above study assessed participants' performance in a psychometric concentration test (d2-R) and the errors made in the work task. The results of the study showed significantly better concentration performance, processed target objects, and the error frequency in the d2-R test as well as the errors in the work task, when participants were supported by the exoskeleton Lucy, compared to the unsupported condition. This may be an indication, that the decrease in physical strain through Lucy could improve cognitive performance of the user, compared to an unsupported condition. To rule out order-effects, the experiment was run in two methodological conditions (supported first, unsupported first) each with a subset of the sample. The comparison of the standard values in the d2-R baseline measurement between the two groups did not reach statistical significance for each variable of the d2-R. Thus, the effects of the support system on the dependent variables for concentration and order-effects could not be attributed to pre-experimental group differences. For the variable concentration performance and processed target objects, the interaction effect of Support and Working Order reached statistical significance. The group, which first performed the work task without support, followed by the supported condition performed better, compared to the group starting in the supported condition. Based on two of the three dependent measures derived from the d2-R, this may indicate that the beneficial effect of support on concentration was strongest, when participants had already performed the work task once before. This could be due to practice-effects regarding the experimental setting, or differences between the groups, that could not be ruled out with the baseline measurement. As displayed in Figure 3 none of the values of the supported conditions were significantly lower than in the unsupported condition of each group (it must be noted that the interaction effect of error frequency in the d2R did not reach statistical significance). Therefore, comparing the two conditions (supported, unsupported) within each group, the performance was enhanced for every variable in the supported condition. In this context, it must also be discussed whether the above-mentioned results of the experiment can be attributed to the fact that participants took significantly more time to perform the work task in the supported condition than in the unsupported condition. Similar response patterns were commonly found in a variety of tasks, allowing the person to set a criterion for speed at the expanse of accuracy, termed speedaccuracy trade off (for an overview see [22]). To eliminate this possible covariate, the differing time spent on tasks with and without the exoskeleton should be aligned in subsequent studies, for example with a familiarization phase, as the blocks might have been too short to adapt to the unfamiliar exoskeleton. Nevertheless, as the effect sizes for the benefit of support were large and the prior study [19] found the beneficial effect of support on concentration and errors without speed differences between the condi-

tions (supported, unsupported), the speed-accuracy trade-off should not account exclu-

sively for supporting effect of the exoskeleton on concentration enhancement and error decrement.

#### **5 Conclusion**

 The exoskeleton Lucy improved the working results of the participants in the above study. A large number of further studies on this topic is necessary to achieve an optimized fit between user and technical support systems in the future.

The present study showed that a technical support system in this particular setting presents a meaningful addition to the workforce, with regards to a higher concentration capacity and error avoidance in comparison to working without such a system. In addition, mechanical relief may contribute to health-promoting work. The urgency to investigate this topic area further with relevance to social and economic contexts. Psychological research may also play a greater role in the development and evaluation of exoskeletons, emphasizing the importance of interdisciplinary approaches.

However, errors in the work process can never be completely avoided in practice and physically demanding tasks and occupations, such as those in the scaffolding industry, will likely remain unchanged in the future. Nevertheless, the results show that the use of technical support systems may be considered in the industrial sector as a whole, under aspects of concentration levels and error avoidance.

#### **Acknowledgements**

This research was partially conducted in the project "Smart ASSIST Smart, Adjustable, Soft and Intelligent Support Technologie" (16SV7114) and funded by the Federal Ministry of Education and Research (BMBF) program for an interdisciplinary build-up of competence in human machine interaction for demographic changes. Supervision is provided by VDI/VDE INNOVATION GmbH. The sole responsibility for the manuscript contents lies with the authors.

The authors would like to thank Victoria Edwards for proofreading the manuscript.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Industrial Data Science - Interdisciplinary Competence for Machine Learning in Industrial Production**

Lukas Schulte1 [0000-0002-0613-7927], Jacqueline Schmitt1 [0000-0002-5780-702X], Lukas Stankiewicz1 [0000-0003-0155-5932] and Jochen Deuse1,2 [0000-0003-4066-4357]

<sup>1</sup> Institute of Production Systems, TU Dortmund University, Leonhard-Euler-Straße 5, 44227 Dortmund, Germany

<sup>2</sup> Advanced Manufacturing, School of Mechanical and Mechatronic Engineering, University of Technology Sydney, Australia

lukas.schulte@ips.tu-dortmund.de

**Abstract.** Within the increasing digitalization, the widespread application of modern information and communication technologies and the technological ability to systematically and comprehensively capture and store data allow to build data storages of unprecedented size and quality. The evaluation and efficient use of the implicit knowledge in the data to support decision-making processes is becoming increasingly significant in manufacturing companies. Thus, new requirements arise for the qualification and competence development to efficiently solve engineering applications and issues in manufacturing and assembly with advanced data-driven methods. This paper presents the contribution of a qualification concept for Machine Learning in industrial production that has been realised within a recent research project funded by the Federal Ministry of Education and Research. This concept has been designed and validated within the university curriculum for graduate students in mechanical and industrial engineering, computer science and statistics. Taking into account the current challenges in manufacturing and assembly, the contribution of this enhanced interdisciplinary competence development can be considered quite significant. The results, findings, and future enhancements are presented within this paper.

**Keywords:** Competence Development, Industrial Data Science, Industrial Manufacturing, Interdisciplinary Competence, Machine Learning

## **1 Introduction**

The German manufacturing sector employs eight million people, making Germany one of the major business and industrial locations worldwide. However, the production environment is subject to constant change through ever evolving trends and challenges. Alongside high productivity and profitability, a highly diverse and customizable product portfolio has become a central strategic competitive factor for manufacturing companies [1]. This, however, entails more flexible and increasingly complex manufacturing and assembly systems, pushing traditional methods and techniques progressively to their limits.

At the same time, the increasing digitalization and the technological ability to systematically and comprehensively capture and store data allow to build data storages of unprecedented size and quality [2]. This data, however, can be used for optimization and decision-making processes within production. By discovering non-trivial and yet unknown structures and correlations, Machine Learning (ML), as an interdisciplinary subject of computer science and statistics, provides intelligent and automated processing of large amounts of data [3]. While data analysis alone, however, contributes to solving practical problems only to a limited extent, it represents, in particular in connection with the domain-specific expert knowledge, a future success factor of the companies [4]. Therefore, methodological competence in the field of ML as well as a good technical understanding of practical engineering questions are required [5]. This intersection of computer science, statistics and engineering constitutes the field of Industrial Data Science (IDS).

However, companies often lack the resources to adequately assort such interdisciplinary teams for IDS projects. In order to solve this dilemma, the qualification of academic graduates and continuing education of industrial professionals must be modified accordingly. [5] Therefore, the research project "InDaS", funded by the Federal Ministry of Education and Research in the program "Qualification programs and research initiatives in the field of Machine Learning", focuses on the development and validation of a comprehensive and interdisciplinary qualification concept of methodical competences in the field of ML with practical applications in industrial production.

#### **2 State of the Art**

#### **2.1 Machine Learning in Production**

Machine Learning (ML) has proven to be very advantageous in various fields of application. Thereby, the success is encouraged by the invention of increasingly sophisticated ML models [6], the availability of large data sets [7] and the development of software platforms [8] which enable an easy employment of vast computational resources for the training of ML models on large data sets.

In industrial production, ML methods can be used to overcome existing problems and boundaries in a number of application scenarios along the production process chain. Successful applications can be found for various tasks in production planning and optimization [9], quality improvement and prediction [10, 11], predictive maintenance [12] and energy efficiency optimization [13]. The data used ranges from master data, over transaction, log and sensor data to text, voice, video and audio data [14]. Algorithms applied include supervised and unsupervised methods such as tree-based classifiers, Support Vector Machines (SVM), Neural Networks, Generalized Linear Models (GLM), Clustering techniques, etc. [15].

Nevertheless, the existing research gap is a lack of interdisciplinary competence that allows the appropriate formulation of learning tasks and application of algorithms for engineering questions in a situations-oriented manner.

#### **2.2 Educational Concepts**

As mentioned above, an increasing technical and functional competence to handle large amounts of data in production in the era of Industry 4.0 is needed [16]. In this context and in conjunction with the fact that learning is not directly observable, a multitude of divergent theories, models, and approaches emerged to explain human learning [1]. Educational concepts that focus on knowledge generation and transfer for individual target groups are equally diverse.

Practically-oriented approaches such as learning factories have become highly promising for knowledge and innovation transfer in academic as well as professional education [17]. Teaching and learning is based on actual problem situations and issues from research and industry as well as on practice-oriented case studies and defined didactic methods [18]. Thereby, the interconnection of research, industry and education enables the establishment of interdisciplinary, multi-dimensional teaching and learning experiences. As a result, a sustainable transfer of knowledge, the development of competence and the ability of creative and self-organized action can be captured [18, 19].

This kind of approach is also required in order to transfer the essential knowledge and skills of IDS to industrial manufacturing and assembly. Through a strong practical orientation, the ability to apply adequate methods to real questions is trained in addition to the mere imparting of knowledge. There is a comprehensive range of seminars, trainings and certificates for various fields of ML including visual analytics, data preparation and data quality, ML algorithms and methods as well as others [20–25]. Nevertheless, each of these qualification measures is based on a specific methodological topic and does not enable the attendees to experience the extent of actual industrial ML projects. **Table 1** shows a comparison of different lectures for Data Science and their focuses.

**Table 1.** Comparison of Data Science seminars concerning content alignment

**Table 1** shows that, with regard to the seminars offered on the subject of Data Science, there is little or no reference to industrial application and the introduction of engineering domain knowledge. In addition, the consideration of data management and the processing of real use cases is only occasionally part of the seminars.

### **3 Concept and Development**

In order to sustainably strengthen the application and dissemination on ML in industrial production, competence development and enhancement may not be limited to industrial professionals, but address young graduates in university education in particular. Thereby, qualification requirements arise from current trends and developments in industry as well as the state of knowledge and skills of professional experts. To design the education and competence enhancement as efficiently and practice-oriented as possible, a mature concept taking all given requirements fully into account is a necessary precondition.

#### **3.1 Qualification Requirements for Industrial Data Science**

Future concepts for qualification and competence development are exposed to increased requirements and must be adapted and supplemented by additional aspects in order to guarantee sufficient competencies in the age of digitalization. The rising complexity of industrial processes requires an increase in specialist competencies, while the greatest challenge is to transfer theoretical knowledge to practical questions and areas of application [5]. Therefore, new approaches should include practical applications and experimentation so that a comprehensive understanding can be created.

A survey by Bauer et al. in 2018 revealed, that only 5 out of 57 companies (9%) within different industrial sectors such as automotive, electronics and mechanical engineering, already use ML widely [5]. Further 27 companies (47%) stated that they had gained initial experience or used methods on a small scale. The remaining 25 companies (44%) have not yet used ML, but plan to do so in the future.

Thereby, the lack of corresponding methodological competence has been identified as the major obstacle for the use of ML. To overcome this shortcoming, a catalogue of requirements must be fulfilled by adequate qualification and competence development concepts for Industrial Data Science [5]. Based on didactic-methodological knowledge transfer, the subjects of data management, ML and domain knowledge are the threefold foundation of IDS education. These core topics can be illustrated by three overlapping thematic areas (see **Fig. 1**).

**Fig. 1.** Concept for qualification and competence development of Industrial Data Science as part of the research project "InDaS" [5]

#### **3.2 Conception of interdisciplinary competence development**

Within the research project "InDaS", the aforementioned requirements are addressed in a strongly practice-oriented and interdisciplinary qualification concept for academic graduates and industrial professionals. This concept combines engineering knowledge and skills with methodological expertise in statistics and computer science. Through close cooperation with manufacturing companies and an early integration and consideration of practical requirements and applications, a high practical relevance of taught contents is sought. The processing of industrial use cases by students with supervision by industrial professionals and university lecturers represents a new approach to teaching. Students work on problems in a real context and realistic levels of difficulty and make contact with companies during their studies. The educational offer addresses students aiming for a master's degree in mechanical and industrial engineering, computer science or statistics as well as industrial specialists. The course is divided into a theoretical phase to first convey basic contents and a practical phase to apply gained knowledge to practical use cases and questions.

#### **3.3 Development of Methodological Competence**

The basic theoretical contents have to be imparted to enable an independent processing of industrial questions with appropriate methodologies. Hereby, the selected theoretical contents are determined by the specific needs of industry to react to current trends and changes. With regard to the processing of large data sets, many companies criticize the manual acquisition and processing of unstructured data and the data administration [26]. Thereby, especially large data volumes are challenging and a faster data supply is desired. Further identifiable problems include a lack of data cleansing and analysis capabilities, a non-agile business intelligence infrastructure, a lack of access to data sources, a poorly intuitive user interface and errors in data management. [27]

To address the before mentioned issues and requirements, the curriculum comprises the thematic modules of data management, machine learning and engineering domain knowledge, built on the basis of didactic-methodical knowledge transfer (see **Fig. 2**).


**Fig. 2.** Contentual and didactical design of the IDS qualification concept

In the theoretical part, complex contents are reduced to their general idea in order to create a basic understanding of multiple methods and approaches. In addition, potential fields of application, problems and limits of learned methods are discussed to enable the students to take them into account in the later practical implementation. Within the experience-based learning approach, learning takes place in interdisciplinary working groups with the inclusion of industrial application cases as well as a project-oriented development of individual solutions.

The efficient management of data requires knowledge in the areas of database systems, data administration, resource management and cleansing and processing of structured and unstructured data. To enable the students to practically realize these tasks, experiences with cloud solutions, database concepts, and script languages, e. g. Python, R or RapidMiner, for modelling and programming data science solutions are taught.

In addition to mere knowledge of different ML methods and approaches, their demand-oriented application is an essential success factor and is therefore explicitly taken into account in IDS qualification. Contentually, the processing pipeline of data preprocessing, modelling, validation and evaluation is covered through the focal points of data visualization, feature engineering, ML algorithms, validation and evaluation.

From the engineering point of view, it is essential to understand the manufacturing and assembly processes to detect the potential of ML for individual business units. To execute ML projects systematically, respective process models such as CRISP-DM (cross-industry standard process model for data mining) [28] should be known. Additionally, domain-specific structuring models, e. g. the product life cycle, are needed to examine the achieved results for their relevance and cost-effectiveness regarding the underlying business case.

Based on the mentioned modules, the students are capable of solving industrial tasks with data-driven methods and developing innovative solution and improvement approaches in manufacturing and assembly. They are able to lead and support interdisciplinary project teams and to pass on their interdisciplinary competences and skills.

#### **3.4 Transfer to Practical Applications**

It is an essential success factor to ensure that young academic graduates and industrial professionals not only know the theoretical backgrounds of IDS, but also build up a deep understanding and are able to apply methods situatively and correctly. Therefore, the second part of the qualification concept consists of the practical processing of industrial case studies in interdisciplinary working groups. Under the involvement of industrial specialists, different case studies covering ML applications such as pattern recognition and image classification (see **Fig. 3**) enable the enrichment of theoretical knowledge with practical experience through knowledge transfer and experience-based learning.

**Fig. 3.** Case studies of the 2019 practical IDS seminar at TU Dortmund University

After the general introduction of the use cases with support of industrial process experts, the project takes place in form of a three-month self-organized, independent group work with regular scientific supervision by industrial and methodological experts. The groups compose of graduate students from the disciplines of statistics, computer science as well as mechanical and industrial engineering, ensuring interdisciplinary key competencies. The results are presented in a joint seminar, including a discussion of results and outlook to further research activities.

#### **4 Concept Evaluation**

The evaluation of the concept is based on the mandatory course evaluation as well as the results of the learning progress assessment.

A 60-minute written examination at the end of the theoretical phase serves as a learning success control. The imparted topics of data administration, statistical basics of ML, ML methods and engineering domain knowledge are queried in one assignment each, consisting of several subtasks. The degree of complexity ranges from simple inquiry of knowledge, the transfer onto new facts and conditions and the linkage of information to solve more complex problems. The results of the examination and the results of the anonymous evaluation in the form of a written questionnaire point to identical findings.

The results and feedback from students in all three subject areas show that the comprehensibility of the contents from the individual disciplines has been of a similar high standard. As a result, it can be concluded that the preparation of the content can be regarded as quite successful, taking into account the heterogeneous target group [29]. The only criticism received has been directed to the introduction to the programming language R, which has been perceived as too condensed by students without prior knowledge. Great interest in the IDS education has been reflected among the university students as well as on the side of industrial partners. As a key figure, the number of participants which increased from 51 in the winter term 2018/19 to 71 in the winter term 2019/20 (+39%) can be mentioned. A survey among the cooperating industry partners has revealed that all of them rate the achieved results as above their expectations and are willing to cooperate again in the next practical seminar.

## **5 Conclusion and Outlook**

Enabled by the increasing digitalization, the widespread application of modern information and communication technologies and the availability of large amounts of data, ML applications are becoming increasingly significant in industrial manufacturing and assembly. To address the associated qualification requirements, new educational concepts have to be developed. The "Industrial Data Science" qualification concept enables an interdisciplinary competence enhancement on the intersection of computer science, statistics and the engineering domain. Through the combination of teaching of theoretical contents and the transfer to practical applications, a comprehensive and practically-oriented qualification is facilitated. The learning success and feedback from the concept evaluation in university education at TU Dortmund University has shown that the course addresses current industrial questions and requirements and is subject to high interest. With the goal of integrating potential benefits into manufacturing companies sustainably, the qualification concept has to be expanded and continuously improved in order to guarantee ongoing actuality. Therefore, the qualification in IDS must be anchored in the university curriculum for graduate students of relevant disciplines such as statistics, computer science and especially engineering and the target group has to be expanded onto skilled professionals in manufacturing companies.

**Acknowledgment** Part of this work has been funded through the research project "InDaS" (BMBF, funding code 01|S17063A) by the Federal Ministry of Education and Research in the context of "Qualification programs and research initiatives in the field of Machine Learning" program.

## **References**


factories for future oriented research and education in manufacturing" in *CIRP Annals*, vol. 66, no. 2, pp. 803–826, 2017, doi: 10.1016/j.cirp.2017.05.005.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Measurement based robot cutting as preparation for welding tubular connections**

Sascha Lauer<sup>1</sup> , Tina Ambrosat<sup>1</sup> , Martin-Christoph Wanner<sup>1</sup> , Wilko Flügge1,2

1Fraunhofer Research Institution of Large Structures in Production Engineering IGP, Albert-Einstein-Straße 30, 18059 Rostock, Germany

2Chair of Production Technology, University of Rostock, Albert-Einstein-Straße 2, 18059 Rostock, Germany

{sascha.lauer, tina.ambrosat, martin-christoph.wanner, wilko.fluegge}@igp.fraunhofer.de

**Abstract.** In the production of offshore tubular connections, the cutting process is an important factor. Undefined root gaps between the individual pipe connections require time-consuming reworking and increased welding effort, resulting in longer production times and higher costs. The developed procedure is based on the 3D measurement of the pipes in order to consider the real geometries in the planning of the cutting process and to optimize the following joining process by adapting the root gap all over the tubular connection and to get a constant seam cross section. The basic concept is to build the planned connection virtually with the measured and real pipe geometries and to plan the flame cutting process including weld seam preparation on the base of the result. For the measurement in the production environment, a suitable measuring concept was developed to return the results directly into the cutting process with the robot. This is the base for the automation of the tubular connection production.

**Keywords:** offshore structures, sensor-based programming, path planning, robotic cutting

## **1 Introduction**

Jacket structures play an important role in offshore technology. They are used as foundation for converter platforms at sea as well as for wind energy plants. Jackets are complex tubular connections based on pipes (**Fig. 1**). Compared to other offshore foundation structures, the used pipes have smaller diameters and material thicknesses. Accordingly, jackets can be produced with lower material costs than comparable structures with similar functions, such as monopiles. However, the complexity results in higher production costs. An essential cost driver are the deviations in the pipe production, resulting with time-consuming and cost-intensive manual rework in the assembly.

**Fig. 1.** Jacket structure (left), tubular connection (right) [1]

The main reason for the need for reworking was identified as the determination of the cutting contours without taking into account the production deviations of the real pipes. By measuring the real tube geometries and by not using the CAD geometry, the manufacturing process for joining the tubular connections is to be adapted in order to minimize manual reworking and thus reduce manufacturing costs.

**Fig. 2** shows the manufacturing process of a tubular connection as a process chain. The process adapts the cutting process to the tolerances in pipe production and serves as the basis for positioning and joining the pipe.

**Fig. 2.** Process chain for tubular connection production

Tubular connections consist of a main member pipe and one or more branch member pipes (see **Fig. 3**), whereby the pipe diameters used here varied from 500 mm to 3000 mm and the material thicknesses between 20 mm and 120 mm.

By flame cutting, the pipe of the branch member is prepared for the joining process. The contour which is calculated by virtually placing the branch member pipe on the main member pipe is based on the ideal geometries of both pipes. In addition, the weld seam preparation is planned for the branch member pipe by using this contour. For conforming the standards of joining for this tubular connection it requires a defined root gap between 2 and 8 mm [2], which depends on the different zones over the various areas of the weld (see I, II and III in **Fig. 3**).

**Fig. 3.** K-connection with definitions of root gap, dihedral angle (opening angle between surfaces) and the zones (I – heel zone; II – side zone; III – toe zone)

Due to the production deviation of both pipes, the ideal seam course in the pipe joint must be adapted to reality. If this is not the case, the joining technology will not be able to tolerate root gaps, which will lead to serious manual reworking such as grinding of the weld in the cut surface. The conventional manufacturing of tubular connections doesn't consider the real geometries of the main member pipe during weld seam preparation. Only the real pipe geometry of the branch member is used to ensure the quality of the cut.

By planning the contour based on real pipe geometries and by generating constant seam cross sections in the weld seam preparation, automated welding with the specially developed orbital welding system in high quality is possible [3].

### **2 Description of the developed process**

In order to achieve these goals, the manufacturing process has to be adapted and the measurement of the pipes integrated into it. Specifically, the following steps (**Fig. 4**) are necessary:


Compared to the state of the art, the first two steps reduce the expensive and timeconsuming re-working by a slightly increasing production planning.

**Fig. 4.** Process steps for cutting with robots

In the following, the mentioned four steps are examined using an example tubular connection. In the selected configuration, a branch member pipe with a diameter of 508 mm and a material thickness of 20 mm is placed centrally at an angle of 45° on the main member pipe with a diameter of 1016 mm.

#### **2.1 Test setup**

For the execution of tests, a suitable plant design was developed. A welding robot (KUKA kr5 arc) in combination with a turn and tilt positioner (KUKA DKP400) is used to extend the working area of the robot. The positioner can handle pipes up to a mass of 400 kg, which is necessary for the chosen branch member pipe.

**Fig. 5.** Test setup for the robot cutting

In order to work with the highest accuracy during the entire process, the coordinate systems of the tilt and turn positioner and the robot were measured to each other using an external measuring system (laser tracker). After the robot system was measured, it was referenced to a selected global system defined by reference marks. This is used throughout the process.

#### **2.2 Digitization of the pipes**

Both pipes are digitized in the production environment by a laser scanner. The used Z+F IMAGER® 5010 has a linearity error of less than 1 mm and a distance noise of 0.5 mm rms at 10 meter [4]. Distances of more than 10 meter to the measuring object are not provided for in the measuring concept.

The main member pipe is measured horizontally. The coordinate system is defined by the pipe axis, flange plane and a marking of the theoretical intersection point of the pipe axis with the pipe surface.

When measuring the branch member pipe, it is perpendicular to the tilt and turn positioner. The robot has a pose repeatability of ± 0.06 mm [5], which is used throughout the process. Instead of measuring the tube from different angles and positions, only one laser scanner position is required and the pipe is rotated between each scan using the tilt and turn positioner (see **Fig. 6**). The registration of the scans takes place in the coordinate system of the tilt and turn positioner, defined by the rotary axis of the table and a rotary angle. The generated point clouds of the outer surface of the branch member pipe are assembled in the evaluation software by turning them back about the defined rotary angle of the positioner. Due to the overlapping areas of the various scans and the unfavorable impact angle of the laser, data smoothing is necessary.

**Fig. 6.** Measuring setup for the measurement of the branch member pipe

A measurement of the branch member pipe from the inside does not take place, since the effort seems inappropriate due to the inner diameter and difficult referencing. During preparatory measurements of the pipe geometries, it was determined that the material thicknesses of the pipes, with the exception of the weld seam, is approximately constant. Therefore, the data points of the inner pipe surface are simulated on the base of the data points of the outer pipe surface using the pipe axis determined by cylinder segmentation according to [6]. The planned configuration is now virtually simulated with the point clouds of the main member pipe and the branch member. The axis of branch member pipe is transformed to the z-axis and the axis of the main member pipe is aligned at the defined angle (see **Fig. 7** (left)). The breakthrough point of the z-axis on the surface of the main member pipe is known from the measurement concept. By translating the point cloud parallel to the pipe axis, the dimensions of the future socket are virtually taken into account.

**Fig. 7.** Process flow for determining the virtual intersection contour using the example of the outer surface of the extension tube, virtual alignment (left), polygonised intersection areas (center) and intersection contour (right)

Using trigonometric methods, the intersection areas of the point clouds were identified on the base of the configuration and the points lying within them were polygonised (see **Fig. 7** (center)). From the calculation of the section of the polygonised areas, a 3D polyline is obtained, with a C³ continuous, closed BSpline curve approximated by its straight line center points (see **Fig. 7** (right)). The results are the curves (inner and outer contour), in which point cloud of the main member pipe crosses the point cloud of the branch member pipe. The outer curve corresponds to the seam contour (see **Fig. 3**), on which the weld seam preparation takes place.

#### **2.3 Weld seam preparation**

The dihedral angle is used to determine the seam course [2]. **Fig. 8** shows a seam course corresponding to the standard for the production of a pipe joint with the used configuration. If the dihedral angle is smaller than 50°, a fillet weld is prepared (zone I, **Fig. 3**), if it is larger than 90°, a bevel groove is prepared (zone III, **Fig. 3**). The area in between is called the transition area (zone II, **Fig. 3**), because the change between these two seam types takes place. The course of this seam was defined in such a way that it can be flame cut with a single move. The challenge is that the seam opening angles which needs to be cut becomes acute. For later use, this step has to be adapted to a two-part cut.

**Fig. 8.** Seam definition in the selected configuration, depending on the dihedral angle

The shown dihedral angles were determined with the seam contour of an ideal tubular connection with faultlessly pipes. In case of a fillet weld, the main member pipe and the branch member pipe touch each other at the outer contour. In case of a bevel groove, the inner contour of the branch member pipe is in contact.

**Fig. 9.** Target layer structure for tubular connection at different areas (branch member – top, main member – bottom)

The necessity of an accurately planned cut and the contact between both pipes is shown in **Fig. 9**. For a future automated welding process, with the aim of achieving a uniform layer structure, it is essential to achieve a constant root gap and a defined weld cross section. Otherwise, the welding parameter has to be adjusted during the entire joining process.

#### **2.4 Offline programming and Preparation for flame cutting**

Since the flame cut is placed on the outer surface of the branch member pipe, the calculated points for the robot have to be shifted to the outer surface of the branch member pipe along the cutting direction (see **Fig. 10** (left)). For a constant cutting quality, the torch distance to the cutting surface has to be constant. This is achieved by further shifting each point along the corresponding cutting direction by an offset. A retransformation of the data provides the discrete positions and directions of the torch head in the coordinate system of the tilt and turn positioner. The robot is integrated in this coordinate system. Since the tilt-turn positioner has a higher accuracy than the robot, the turntable rotates during the cutting process, while the robot only performs small movements. The robot software interpolates between the position and direction data and calculates the positions of the axis in order to avoid singularity positions. By using synchronized movements from the positioner and the robot, it is possible to cut at constant speed using all eight axes.

**Fig. 10.** Calculated contour (left) and path planning with the robot in SIEMENS RobotExpert (right)

This is the base for the offline-programming of robot motion (see **Fig. 10**, right) and collision control to prevent crashes between the robot/torch and the pipe. In addition, motion commands and switching commands for the cutting process, such as start and stop distances, ignition commands and waiting times, have to be added.

### **3 Verification and Results**

The method is used to manufacture the described tubular connection. The cut branch member pipe, especially the cut surface, is measured with the strip light projection system GOM Atos Triple Scan. A 300 x 300 mm² measuring field is used, with which a measuring accuracy of approx. 0.01 mm can be achieved [7].

First of all the resulting model is compared with an ideal cylinder to determine the production deviations. On the other hand, the model is virtually placed on the digitized main member pipe in the defined configuration and aligned by best-fit method, whereby only rotation around and displacement along the pipe axis are permitted. The root gap can be derived from this. The results are shown in the form of a measurement in **Fig. 11**.

**Fig. 11.** Measurement results to illustrate the production deviations of an extension tube (left) and to illustrate the deviation of the cut surface from the planned one (right).

The production deviations of the attachment pipe are clearly visible in the left measurement. The tolerance of ±5 mm is maintained [8]. The right measurement shows the deviations between the virtually planned and the actual cut surface. The maximum deviation, this means the largest gap between both pipes, is 1.59 mm. The minimum deviation, the largest penetration of the two pipes, is -1.83 mm. They vary by 3.42 mm. Within the bevel groove area (zone III, **Fig. 3**, upper cut area, **Fig. 11**) the deviations differ by less than 1 mm. Within the fillet weld area (zone I, **Fig. 3**, lower cut area, **Fig. 11**), too. The variations in the transition areas are more than 2 mm. The specified fluctuations of the nominal root gap of 6 mm are adhered to. Despite the production deviations, the weld seam can be prepared in accordance with the standards.

#### **4 Conclusion and outlook**

It has been proven that the interaction of measurement technology, cutting contour planning and robot cutting can be successfully applied in practice. Both the tubular connection configuration and the selected seam configuration are to be understood as an example. The process can be used for any tubular connection and seam configuration. Further optimization possibilities are in the next process steps of positioning and joining (see **Fig. 2**). The challenge in positioning is that the approach point planned in the configuration, in which the branch member pipe axis meets the axis of the main member pipe, is calculated and has to be transferred to the real pipe configuration. In order to optimize the joining process, based on a two-part cut, a suitable seam configuration can be developed. Because the seam course is planned and implemented on the base of the actual data, virtual weld layer planning is possible without further measurement, which additionally accelerates the production process. In particular, this enables automation of the welding process.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Information requirements and interfaces for the programming of robots in flexible manufacturing

Arturo Bastidas-Cruz<sup>∗</sup> , Oliver Heimann<sup>∗</sup> , Kevin Haninger, and J¨org Kr¨uger

Fraunhofer Institute for Production Systems and Design Technology (IPK) Division of Automation Technology, Berlin, Germany firstname.lastname@ipk.fraunhofer.de <sup>∗</sup> Equal contribution

Abstract. Flexible automation limits what information is known during robot commissioning, requiring new robot programming methodologies. To adapt robot behavior to product variation, an operator can supply missing information, controlling predefined robot behaviors via a user interface. This operator interface can abstract domain expertise (in robotic programming, assembly planning), allowing the efficient specification of changes to a robot program by a wide range of potential users. Towards designing such interfaces, this paper analyzes the requirements of flexible manufacturing, the required changes to the robot program, and the information needed to make those changes. Two user interfaces are presented, a drag and drop and a gesture interface, implemented on a robotic flexible manufacturing testbed.

Keywords: flexible manufacturing, robot programming, human-robot interaction, usability

## 1 Introduction

In standard industrial robot commissioning, as common in large-scale assembly lines, experts produce a fully-defined sequence of robot commands for a specific task during commissioning. Although providing higher quality and throughput, traditional automation reduces production agility with limited ability to accommodate product variation or new products without recommissioning. For smalland medium-sized companies in saturated markets with a faster product cycle, this is a major barrier. In comparison, on manual assembly lines new products can be quickly deployed (provided no hardware changes are required) and product variation is tolerated with marginal increases in cost or deployment time. Though difficult for standard automation, flexible manufacturing can address many growing business needs: individualized products, smaller lot sizes, shorter innovation cycles, and shorter production times. These capabilities support onshoring; local production at competitive costs. Flexible manufacturing can involve product variation, where the product varies between task iterations, or production changeover, where the assembly line is reconfigured for a new product. The manufacture of multiple products on an automated line can be solved centrally, through methods like a production management system, which indexes all products and adjusts a cell as necessary. Decentralized flexibility is also possible, through the design of adaptive cells which detect and respond to product variation (e.g. with vision systems, or identifying tags). However, these approaches are complex and do not reduce the effort to introduce a new product. A promising approach to flexibility is human robot collaboration (HRC), where human operators can respond to product changes, adapting pre-programmed robot behaviors as appropriate [7]. Designing the robot such that an operator can later complete a task requires consideration of what decisions the human must make, how those decisions are communicated to the robot, and the contextual information required to make those decisions. Programming for flexible automation, moreso than traditional automation, occurs over time, where a variety of users contribute to the final sequence of robot commands. From the user perspective, the interface must include visibility of system status, understandable information presentation, a match between the system and real world, and help users recognize and recover from errors [2], [6]. Several modes of human-robot communication for modifying a robot program have been developed, including speech [9] and gestures [10], [11], [13]. However, more intuitive and compact modalities of communication can have ambiguity in the interpretation of the input. Therefore, the robot and human must have a common understanding of the environment to correctly ground an exchanged symbol (see [14] or [12]). This is a usability requirement for the design of intuitive human machine interfaces.

Human-robot communication can also be achieved through a graphical interface on a touchscreen in the workspace. Such graphical environments can allow for the simultaneous presentation of more complex information, allowing a user to see and adjust the parameters and program flow of a robot. This interaction can be further improved through the use of augmented reality (AR). AR allows GUIs to present information as an overlay to the actual evironment, which enables a strong visual connection between parameters and their real world influence. Lambrecht [10] et al have shown that tablet based augmented reality can significanlty reduce the programming time for free space programs. Guhl [4] et al as well as Ong [5] et al have further improved on the approach through a headworn AR device allowing for comprehensive virtual interaction with the robot program.

This paper explores HRC in flexible manufacturing, examining application requirements and usability perspectives. Programming for flexible manufacturing is presented as the partitioning of expertise and process information within a prior robot program, where the information required is reduced and easily supplied. While other GUI development considers defining a robot task from lower-level commands [3], the GUIs discussed here are targeting flexible manufacturing, where a subset of decisions or changes must be made via the GUI. Two interfaces for the flexible programming of a robot cell are presented, a gesturebased system for workspace communication, and a touchscreen-based graphical interface with predefined actions.

## 2 Flexible Manufacturing

What new design requirements does flexible manufacturing introduce? Consider a tier two supplier for the car manufacturing industry, where different subassemblies must be produced for just-in-time orders of tier one suppliers. The ordered variants will vary between the different customers and on a day-to-day basis, requiring fast change over times. Furthermore, some variants might only ever be produced a hundred times, rendering the programming of a robotic cell for all possible production unviable. Flexible manufacturing can also encompass artisan production, where highly individualized products are manufactured and robot assistants can be directed in response to material variation, artistic expression, or customer individualization. Table 1 presents different aspects of manufacturing which may vary in flexible production with examples.


Table 1: Flexible manufacturing examples, which may require on-/offline changes to the robot program

There are two major types of variation considered here: geometric and plan. In geometric variation, the location(s) which determine a specific action may vary between cell cycles or product change-over. In plan variation, a change in the set or order of robot behaviors must be made. Other changes to the robot program may be necessary (dwell time, contact force, etc), but are typically process parameters which will require more expertise and are outside the scope of this paper.

## 3 Decision making

Product variation, as introduced in the last section, requires corresponding changes to the robot program. This section overviews available and necessary information to make those changes at the various stages of the flexible programming.

## 3.1 Information

Information and expertise is required to translate a product design into an appropriate set of robot instructions. As seen in Table 1 the product model changes least frequently; it is a result of the design process and therefore is considered available as input to the assembly level planning. The assembly planning requires a very high level of expertise to translate a product design into a sequence of manufacturing processes in order to assemble the product. It involves knowledge about machine availability and resource planning. The assembly planning problem is well supported by tools (computer-aided manufacturing), more easily dealing with high-mix, low-volume production compared to lower-level planning layers (e.g. robot programming).

On the basis of a product-specific assembly plan, a corresponding robot task order - the sequence of robot actions - must be produced. Where the assembly plan includes general operation orders, the robot task order must specify the product-specific robot actions which achieve this result. With increasing variants and smaller production lots, this results in an exponential growth in required decisions, and tighter time constraints on every individual decision.

Therefore, a programming system for flexible manufacturing on the task level requires the ability to rearrange the order of processes or even substitute different processes in an efficient manner. An example is the production of gear boxes: while the overall process of interlocking gears, fitting shafts, bearings and seals is comparable between products, materials, sizes or even the number of overall parts might vary. Here, the operator needs to adapt the overall flow of control of a prior program. Furthermore, at the task level, the operator might specify specific process parameters. These include feed velocity, maximum torque for a screw or the slope of a thread to be cut. In order to make these decisions the operator needs expert knowledge regarding the machining processes involved as well as the capability to translate product specifications into the required processing parameters.

Decision speed becomes even more important on the individual action level as again, per task instance, multiple actions must be specified. Here craftmanship is required to determine the correct instantiation of an action on a per product basis. As there may be no purpose made fixtures or computer vision systems for the individual production lot, the human operator needs to supply this information. Where the operation order is programmed in advance, the artisan process requires information in situ. The frequency of online decisions requires efficient in-situ programming to avoid a bottle neck in the manufacturing processes.

Figure 1 gives an overview on the different levels of programming and the corresponding informational requirements. It can be seen that the frequency of a decision increases as it comes closer to program execution. At the same time, the breadth of information required becomes narrower. This increasing specialization of the information provided helps to address the need for fast decisions.

While at the task planning level, an interface needs to provide means to program the entire process. This results in the necessity for the user to navigate a more complex interface. However, on the process level, the interface might just offer a very limited set of options specifically tailored to the individual process, allowing very fast navigation and user interaction.

Fig. 1: The various levels on which a flexible manufacturing task can be programmed, where a prior program can be specialized, with discrete or continuous inputs. The last two stages (task order and action list) are static in standard automation, but here can be adjusted offline or online.

#### 3.2 Interface Usability

The prior section has shown that the frequency of required decision making varies with the level of detail of the process specification. For frequent shopfloor decisions, a user can make adjustments to the robot execution via an HRC interface. The means by which a user provides this information to the robotic system affects the ability to display contextual information, ergonomics, learnability, and flexibility [1]. HRC is very much defined by its context, requiring a common grounding to the symbols exchanged between robot and human.

The interface must provide feasibility, the ability to adjust the robot behavior as necessary for the appropriate task, and safety, such that for any supplied input of the operator the robot maintains safety. On top of the functional requirements, the interface should support usability [2], [6] in terms of:

Visibility: Information required to make a decision is readily available, including relevant states of the robot.

Predictability: The effects of an input are intuitive and repeatable, so that the user can choose the input required to achieve the desired robot behavior.

Efficiency: The necessary action can be quickly and ergonomically given to the robot. For online decisions, this should not significantly disrupt the operator's workflow.

## 4 Implementation

Fig. 2: Integrated flexible manufacturing demonstrator

To investigate feasibility and usability of various types of HRC for flexible manufacturing, the testbed seen in Figure 2 is built. The cobot used is a Universal Robot UR3. Above the table 2 RGB cameras and a projector are mounted. The current use case is the automatic placement of workpieces from a storage position onto the table followed by a simulated welding operation. This testbed can test variation in both geometric and plan information, where decisions must be made by an operator online or offline. Two interfaces are presented, which differ in how they present the available decisions and contextual information to the user, and how the user supplies their decision to the robot.

#### 4.1 Graphical Programming

The graphical task-oriented robot programming system presented in this work was developed by researchers from the Process Automation and Robotics department of the Fraunhofer IPK institute in order to allow visitors without any programming experience to interact with the cobot, so that the user can make modifications to the robot program and execute an assembly task. The system is based on the task-oriented RP-i3 framework developed at Fraunhofer IPK [8] and aims to reach easy accessibility of complex systems by workers in industry. The task-level framework offers a visual structured interface of the assembly process that allows an easy implementation of the two major types of variation discussed in Section 2: an easy rearrangement of the order of processes related to a specific assembly task (plan) as well as the easy specification or modification of specific action parameters (including geometric variations).

The user selects tasks that the robot can perform on the work pieces, where tasks and work pieces are offered as ready-to-use blocks. The blocks are combined to a program by dragging and attaching them to each other. New robot programs and new work pieces can be defined in that way. The graphical programming environment SCRATCH [17] visualizes the assembly environment abstraction of the RP-I3 framework as seen in Figure 3b.

A robot programming language based on C++ implements the RP-I3 abstraction of tasks and workpieces. A server that works as an interface receives the content of the GUI, organize the information in a data base and execute the programs using this information. In order to send the command to the robot, an external interface is used. The setup consist of a Cobot equipped with a two-finger gripper. The RP-i3 framework is located on a Linux-based PC. The visualization of the RP-i3 framework is based on a version of Scratch that was modified in order to allow the implementation of robot-based assembly tasks.

This interface offers a higher information-density, allowing the presentation of a complete robot plan which helps a user predict robot behavior. Although the actions and workpieces are abstracted from the physical workspace, graphics seen in Figure 3b can improve the operator's understanding. However, this interface does not as easily allow online monitoring or decisions, as the screen must be mounted to not obstruct the workspace.

(a) Graphical Programming (b) User Interface

Fig. 3: Programming of an assembly task using the graphical interface RP-i3 and universal robot UR3

#### 4.2 Gesture based Programming using Augmented Reality

The programming interface for the action level aims to provide very fast and precise user input for a very limited breadth of decisions that allows in detail specification of the action at hand. Where the RP-i3 GUI enables the specification of the more abstract task and its parameters (e.g. weld together part A and B with feed velocity of 20mm/s), the gesture based interface allows to specify artisan-level decisions (e.g. exact position and length of a seam). This information is complex (therefore time consuming and expensive) to describe in advance at the task level but very easy to specify once the product is available in the manufacturing cell.

The interface allows machine operators to communicate these decisions in a very fast way through direct interaction with the program on the workpiece using augmented reality [15]. Geometric information (planar locations) can be communicated, as well as discrete decisions via virtual buttons in the workspace, allowing the omission of other inputs device such as a touch panel. Furthermore, the direct projection of paths onto the workpiece provides the user with understanding of the robot state and plan.

Fig. 4: Augmented Reality interface for gesture based programming. The user specifies the details of a given action through pointing gestures on the workpieces (left). The robot program is automatically generated based on the action specification (right).

To achieve the precision required for programming the artisan process, the system relies on 3D computer vision (stereo vision based on 2 RGB cameras with a pixel resolution of 35µm/px). In an intermediate step the system than extracts relevant geometric features such as edges and planes. These are subsequently fed into an inference engine which determines probable process affordances[16] (areas of interest for a given process) based on pre-defined rules for the according process. Using the detected affordances, the user interaction is elevated to a level of mutual understanding. The system presents possible actions to the user who than solves ambiguities and further details the action through pointing.

Figure 4 shows the programming process for a welding process. The two work pieces are placed on the table. The system automatically detects the common edge between the objects (as it presents seam affordance) and highlights it using the projection system. The user than specifies part of the edge for welding using his finger. The system combines the knowledge about the process affordance with the gesture input to compute a precise robot program (limited by the accuracy of the 3D object detection algorithm rather than the gesture tracking).

### 5 Conclusion

In this paper we introduced the application requirements of flexible manufacturing, corresponding programming requirements, and how those requirements can be met with two different user interfaces. Programming for flexible manufacturing introduces challenges as it requires the engagement of a number of stakeholders, with varying expertise and responsibilities. Two user interfaces are presented and analyzed according to their ability to address the various requirements of flexible manufacturing.

In future work, these two interfaces can be combined in new ways to offer new programming modalities - e.g. allowing re-programming of an action through gesture demonstrations. The unified testbed allows for future usability experiments which can quantitatively motivate best-practices for programming in flexible manufacturing.

#### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **User-Centered Design of an Intuitive Robot Playback Programming System**

Christian Colceriu<sup>1</sup>, Michael Riedl<sup>2</sup>, Dominik Henrich<sup>3</sup>, Sigrid Brell-Cokcan<sup>4</sup>, and Verena Nitsch<sup>5</sup>

<sup>1</sup> Krones AG, Corporate Communications, D-93073 Neutraubling, Germany, christian.colceriu@krones.com <sup>2</sup> University of Bayreuth, Chair for Robotics and Embedded Systems, D-95440 Bayreuth, Germany, michael.riedl@uni-bayreuth.de <sup>3</sup> University of Bayreuth, Chair for Robotics and Embedded Systems, 4 D-95440 Bayreuth, Germany, dominik.henrich@uni-bayreuth.de RWTH Aachen University, Chair for Individualized Production in ASchitecture, D-52062 Aachen, Germany, brell-cokcan@ip.rwth-aachen.de <sup>5</sup> RWTH Aachen University, Lehrstuhl und Institut f¨ur Arbeitswissenschaften, D-52062 Aachen, Germany, v.nitsch@iaw.rwth-aachen.de

**Abstract.** A central part of a robot programming system is its user interface. Oftentimes, its design and development are neglected, although the user interface is the connection between the robot and the user. In this work we present the application of a user-centered design process to a robot programming system interface. The iterative process consists of three steps, namely analysis, design, and evaluation that may be repeated until a desired outcome is obtained. The evaluation shows that with the help of the adapted user-centered design process, it is possible to improve existing user interfaces to become more intuitive and easy to use for nonexperts from domains of assembling and construction.

**Keywords:** interface design, graphical user interface, usability engineering, intuitive robot programming

## **1 Introduction**

Usually, programming robots is done by experts by writing source code. This approach is time-consuming and costly and therefore robots do not play a big role in small and medium sized enterprises. To reduce the cost and time needed to program a robot system, a fast and intuitive programming system that can be used by non-experts with little to no programming experience is needed. In [1] we described our fast playback-programming system that is developed for non-experts. Programming the robots with this system is done by manually guiding them along the desired path. Afterwards, the recorded robot program is displayed along a timeline in the user interface. The target group for this programming system consists of non-experts that usually work in domains of assembling and construction. However, the playback-programming system still needs to be improved in terms of usability and intuitiveness. Therefore this work shows an approach towards how to design an intuitive user interface for touch screens and how to evaluate it. Touch screens tend to allow more intuitive interactions than keyboard-mouse combos [2]. Our contribution is the process of how an interface for an intuitive robot programming system that is controlled with a touch screen may be designed. At the beginning we describe our iterative three-stage design process consisting of analysis, design, and evaluation, where the changes to the existing robot programming system are shown. After that we show the application of the design process using the example of playback robot programming. Finally we show the outcomes of the evaluation steps and how the user interface improved compared to the previous version.

### **2 Related Work**

To get an overview on the field of designing intuitive robot programming systems, we have to take a look at two fields of study. The first part deals with designing interfaces in general, and the second part shows current research in intuitive robot programming systems.

Design is usually an iterative process. This approach can be found in many process models. Four approaches to the design process were selected as examples. All of these approaches relate to a user-centered design process. The individual phases differ greatly in terms of formulations and details. A uniform picture is not recognizable at first glance. The design process as it appears in the German DIN EN ISO 9241-210 standard contains the phases *plan the user-centered design process*, *understand and specify context of use*, *specify user requirements*, *produce design solutions*, *evaluate designs against requirements*, and *design solution meets user requirements*. The design process according to [3] consists of the phases *analysis*, *design*, *evaluation*, and *revision*. The design process according to [4] contains the phases *analyze*, *design*, *develop*, *evaluate*, and *implement*. The Design Thinking process according to [5] contains the phases *understand*, *observe*, *point of view*, *ideate*, *prototype*, and *test*.

**Fig. 1.** Approach to a design process for robot programming systems.

All these design processes can be generalized to an iterative, three-step design approach that can be used to improve the interface design of a robot programming system. These steps are shown in Figure 1 and are called *analysis*, *design*, *evaluation*. This work does not deal with the *implementation* step.

After taking a look at existing design processes, we have to look at existing intuitive robot programming systems to specialize our design process on robot programming systems. Recent research is trying to make programming robots as easy as possible by utilizing methods that do not require the users to write source code. There are multiple approaches that try to achieve this. One approach is to use the *Programming by Demonstration* approach that is either guiding the robot manually through its task [6] or that is learning the robot motion by observing the user with sensors [7]. Other intuitive approaches are to utilize hand held devices [8], smartphones [9], touch screen [10], visual programming in CAD [11], or virtual reality [12] to program robots.

All these approaches have in common that they need some kind of interface to exchange information with the user. Most of the times, a screen is used to interact with the robot, but the design process itself for the development of an intuitive user interface is not shown. Therefore, a need for a design process for robot programming user interfaces can be identified. With the next sections, we want to fill this gap by introducing our design process for robot programming user interfaces.

#### **3 Design Process for Robot Programming Systems**

The research aspect of this paper is to take the proven contents of a user-centered design process and apply them exploratively to the design of an interface for a robot programming system. The goal is to design an intuitively operable interface. For this purpose, the state of the art will be examined in relation to the design process for interfaces. The three phases analysis, design, and evaluation are studied and selected methods are transferred to the robot programming system. The choice of methods is based on the approach of using intuition and personal experience to obtain an optimal result[13].

#### **3.1 Analysis**

The analysis phase is the first step in the user-centered development of interfaces. In the analysis, the interaction partners, man and machine, are examined. The investigation of the user behavior gives a complete picture of the work tasks, user groups, and working environment [14, p. 39]. One approach of the analysis is the division into goal definition, basic idea, and context of use [15, pp. 177 ff.]. Methods of analysis include field observation, identification of personas, structured, and free interviews [14, pp. 47 ff.]. In the intuitive robot programming system, the context of use consists of the operator, the robot, and the object for handling. In addition, the basics of the theory of intuitiveness are researched in order to work out valid methods within the process.

#### **3.2 Design**

The design phase deals with the transfer of requirements from the analysis phase to an interaction platform [14, p. 65]. One approach is to divide it into different aspects: screen design, information design, interface design, interaction design, animation design, and sound design [16, pp. 12 ff.]. Methods for designing interfaces can be divided into sketching and prototyping. Methods of sketching are manual sketches, computer-generated layouts, storyboards, and wireframes. Prototypes can be paper prototypes, video prototypes, Wizard-of-Oz prototypes and physical prototypes [17, pp. 115 ff.].

Here, the implementation of the design has the goal to develop an intuitively operable interface. For this purpose, a paper prototype and later an interactive virtual mockup is generated.

### **3.3 Evaluation**

The evaluation phase defines test criteria with verifiable parameters for specific tasks. The interface can only be assessed in predefined contexts of use and depending on specific user types [18]. Methods of evaluation are divided into quantitative and qualitative methods, heuristic evaluation, and general usability investigation methods [16, pp. 315 ff]. The prototypes here will be evaluated with methods that measure the intuitiveness of the interface. Since the researched evaluation methods measure the usability of the interface, an adaptation of the evaluation method is necessary here. The MINERIC Toolkit is designed to evaluate robotic systems regarding intuitiveness [19].

#### **3.4 Summary**

Existing design processes aim to develop an interface that improves its usability. The research object of the robot programming interface intends to develop an interface that improves its intuitiveness. The approach is to use an already developed theory of intuitiveness in the context of human-technology interaction and to explore methods that design an intuitive interface within the adapted design process.

## **4 Example: A Playback Programming System**

After describing the design process in detail, we verify it by showing its execution on an example. In this example, we want to improve the already existing user interface for playback programming [1] in two iterations.

#### **4.1 First Iteration**

The first iteration contains the phases analysis, design, and evaluation. In the analysis phase, a research on the state of the art is carried out. This includes the research on programming systems, standards, and on scientific literature to analyze the theory of intuitiveness. In the design phase, insights are incorporated into the interface design, which results in a paper prototype. The paper prototype is evaluated with three usability experts in an expert review.

**Analysis** In the analysis phase, various systems from the field of video and audio editing are tested. Systems with and without touch screen functions are used to analyze how elements of a recording timeline can be edited in the systems. The topic of *intuitiveness* in the context of human-technology interaction was examined by a working group in 2006 [20]. 24 experts for man-machine systems were asked how intuitive work is related to the seven principles of dialogue design, as described in the standard DIN EN ISO 9241-110. The similarity of the principles in connection with intuitiveness was inquired [21].

**Design** The interface is oriented towards word processing software (e.g., Microsoft Word, Open Office Writer). It is designed in accordance with the standard DIN EN ISO 9241-110. Several dialog principles have been incorporated into the design. *Conformity with user expectations* was adopted by implementing well-known conventions in the toolbar and icons that correspond to the functions in word processing software. *Self-descriptiveness* has been implemented by functions that are communicated nonverbally, only icons are shown on the buttons of the toolbar. *Suitability for the task* was adopted by active buttons that are clearly visible and buttons, like Paste, that appear only when a section of a recorded

**Fig. 2.** Design of the paper prototype.

robot program along a timeline has been copied to the clipboard.

A paper prototype is created for evaluation, which is shown in Figure 2. First modifications are made to the original system for keyboard-mouse combo. The arrangement of the buttons on the toolbar is included. For loops and branch functions, the additional dialog window is replaced by buttons.

The interaction is based on the operation of word processing software with keyboard-mouse combo. In word processing software, a word is marked by clicking on it and dragging the mouse along the word. In the paper prototype, a part of the robot program along the timeline can be marked by touching it with the finger and moving the finger along the recorded timeline. When a section of a robot program is highlighted, the edit functions become active. The interface consists of a toolbar with functions. The functions are divided into file management functions (*new file*, *open file*, *save file*), undo/redo functions, edit functions (*copy*, *paste*, *cut*, *delete*), and special functions (*loops*, *branches*)[1].

When recording a new robot program for a timeline, the interface changes to the recording mode and the robot has to be programmed by manual guiding. The *open gripper* button is only active when the gripper of the robot is closed and the *close gripper* button only is active when the gripper of the robot is open. The resulting paper prototype is shown in Figure 2.

**Evaluation** The evaluation is conducted as an expert review. The two methods *Cognitive Walkthrough* [22] and *Heuristic Evaluation* [23] were used for this. It was done with three usability experts who also do this job in their profession. The paper prototype from the design phase was evaluated in aspects of intuitiveness. It was done individually with each expert. First, in a Cognitive Walkthrough, the expert was guided through the individual functions of the system. Three tasks to be solved were described. At the end, every expert had to complete the questionnaire for the heuristic evaluation.

#### **4.2 Second Iteration**

The second iteration again contains of the phases analysis, design, and evaluation. In the analysis phase, further research on standards and design laws is done. In the design phase, insights are incorporated into the interface design, which result in an interactive virtual mockup. The MINERIC toolkit is used to evaluate the virtual mockup with 20 workers from the assembling domain[19].

**Analysis** In the analysis phase, more guidelines from standards and design laws are used. The design of a virtual mockup requires a closer examination of the coloring, shaping, and composition of individual interface elements. Here, the standard DIN EN ISO 9241-110 serves again as a support to meet important criteria of usability.

Design laws play a role in the design of intuitive user interfaces outside the ISO standards [21]. Several design laws can be found in the literature [15]. The law of similarity states that related elements are recognized by similarities. The law of proximity states that spatially close elements are recognized as belonging together.

**Design** The suggestions for improvement, which were written down in the expert review, have been incorporated into the design. The interface has also been designed in the second iteration according to DIN EN ISO 9241-110 and design laws of the analysis step.

Conformity with user expectations has been further developed by changing the vocabulary, so that terms from information technology (e.g. *capture stimulus*) have been changed to everyday terms (*capture image*). Self-descriptiveness is adopted by designing the record button especially present when a recording is expected. Suitability for the task is implemented by designing all buttons consistently inactive, if the function is not available at the time. In the record

**Fig. 3.** Design of the interactive virtual mockup.

mode, the standard interface is heavily dimmed. Law of proximity is taken up by clustering buttons in the toolbar to show that they belong to different topics. Law of similarity is taken up by icons for *open gripper* and *close gripper* that look similar in record mode, so they can be recognized as belonging together in the topic, but they differ in colors so they can be distinguished. The interactive virtual mockup is shown in Figure 3.

**Evaluation** During the second iteration of the design process, we conducted a user study according to [19]. The study took place on the 23rd and 24th September 2019 with 20 participants. All participants were workers from the assembling domain aged 21 to 57 with a mean age of M = 38.7 years and a standard deviation SD = 9.7 years. Ahead of the study, each participant saw a short video of the programming system and got an introduction into the interface. Then, each of the three different tasks were explained to the participants. During the study, the participants had to demonstrate the tasks to a scaled, non-automated robot model with the help of the designed interactive virtual mockup. The first task was a pickand-place task with post editing where two red cylinders had to be picked up from a conveyor belt and deposited in the corresponding box, so that both, the recording and the editing of robot programs can be evaluated. The second task was a sensor-based repetition, where objects had to be picked up from the conveyor belt and deposited in the corresponding box, as long as the objects

**Fig. 4.** Schematic representation of the three tasks the participants of the user study had to complete.

were red cylinders. The third task was a sorting task where objects had to be picked up from a conveyor belt and had to be sorted according to their color. A schematic representation of the tasks is shown in Figure 4.

### **5 Evaluation Outcomes**

For each of the two iterations of the design process, an evaluation phase has been performed. The first time, an evaluation with usability experts was conducted, and the second time an evaluation with non-experts from the domains of of assembling and construction was conducted.

#### **5.1 Evaluation with Usability Experts**

In a first evaluation, we used a Cognitive Walkthrough in combination with a Heuristic Evaluation to identify possible improvement points with the help of usability experts. The main comment was, that the usage of the system needs to be consistent. For some functions, sections had to be marked first and then the function was triggered, for other functions it was vice-versa. Another comment was, that the used vocabulary should be adjusted to the target group, so it is understandable for them. The last major comment from the experts was, that functions that are expected to be used next should be placed more prominently within the graphical user interface. With the help of these major comments and some minor notes, the interactive virtual mockup was designed.

#### **5.2 Evaluation with Domain Experts**

The purpose of this evaluation was to get an intuitiveness rating of the improved interface from non-experts, namely workers from the assembling domain. Since intuitiveness is defined as a combination of effectivity, efficiency, and satisfaction according to the MINERIC toolkit, we used the presented approach to evaluate the prototypes. The detailed evaluation process is described in [19].

The *effectivity* score is shown in the left chart of Figure 5. It is measured on a scale from 0 and 100. The rating for the designed mockup is M = 69. This indicates a rather effective to use system. Furthermore, the score for each task shows that the effectivity is continously increasing, which indicates that there is a learning effect and it is easier to use the mockup with each task.

The *efficiency* score is shown in the middle chart of Figure 5. Its value ranges from 0 to 220, where lower scores indicate a better efficiency. The overall rating of M = 39.2 is directly translatable to *a little effort* to use the interface. This means that it is quite easy to use the designed mockup. A slight increase of the efficiency over the three tasks may be explained with the fact that the last two tasks contained sensor-based loops and branches that are harder to understand.

The *satisfaction* score is shown in form of the QUESI score and its subscores in the right chart of Figure 5. It is measured with five subscores on a scale from 1 to 5, where 5 means that the participants are fully satisfied with the system and 1 means that they are not satisfied. An overall QUESI score of 3.7 shows that the participants were rather satisfied. The subscores of the QUESI score are *subjective mental workload (W)*, *perceived achievement of goals (G)*, *perceived effort of learning (L)*, *familiarity (F)*, and *perceived error rate (E)*. The score of 4.2 for the perceived achievement of goals shows that the participants were able

**Fig. 5.** Outcomes of the user study. The left chart shows the results of the effectivity. Higher values are better. The middle chart shows the results of the efficiency. Lower values are better. The right chart shows the QUESI scores and subscores representing the satisfaction. Higher values are better.

to achieve their goals, whereas a score of 3.4 for the familiarity shows that the participants were familiar with the interface, but there is room for improvements.

Overall, the three evaluated scores, namely the effectivity, the efficiency and the satisfaction show that with the help of the design process, the interface for the playback programming system was improved and is usable by non-experts working in assembling and construction domains.

#### **6 Conclusion**

To sum up, we applied a three-step user-centered design process to an interface for a robot programming system. With the example application of the design process to improve an existing user interface in the field of playback programming, we have shown that the user-centered design process results in an intuitive user interface that is usable by non-experts from domains of assembling and construction. Limitations are the early stage of research of the theory of intuitiveness for human-robot interaction and the use of a scaled, non-automated robot model for the evaluation. Future work may be done by also taking the implementation phase into account when designing the user interface.

**Acknowledgment** This work has partly been supported by the Bayerische Forschungsstiftung (BFS) in the project FORobotics.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Circular Path Generation for Toroidal Cavity Inspection

Lukas Bath[0000-0002-4715-1661] , Maik Dammann[0000-0003-3245-814X] and Thorsten Schuppstuhl ¨ [0000-0002-9616-3976]

Institute of Aircraft Production Technology, Hamburg University of Technology, Denickestraße 17, Hamburg 21073, Germany, lukas.bath@tuhh.de, https://tuhh.de/ifpt

Abstract. Borescopes are a widely used technology for optically inspecting machinery from the inside. Recent solutions include enhanced metrology techniques for acquiring image data in high resolution as well as 3D surface information. These already foster more detailed health and damage inspections. However, both inspection and diagnosis strongly rely on the expertise of professionals. Despite their strong coupling, an increasing separation between these tasks is trending. Furthermore, many regions often cannot be inspected properly due to limitations in accessibility and hardware, which is particularly true within toroidal cavities. As a consequence, significant demands regarding emerging digitalization and automatization such as reproducibility, completeness or the steady and high quality of results cannot be fully satisfied. These drawbacks can be faced with a defined probe guidance while providing adjustability towards sensor characteristics. In this publication a novel approach with great automation capability is presented, facing the contactless and complete inspection of toroidal cavities, requiring no disassembly. The proposed solution is related to continuum robotics while based on tube forming technology coupled with a path planning strategy.

Keywords: borescope inspection, endoscopy, continuum robotics, path generation, tube forming, cavity, probe guidance, material compound

## 1 Introduction

Increasing complexity of machinery being used worldwide, demands for nominal downtimes and the availability of sophisticated, non-destructive measurement equipment have led to more frequent routine inspections using borescopes. Recent developments have pushed these devices to smaller scales while making them more powerful by adding new and more accurate measurement principles. The areas of interest inside of machines are usually those where significant wear is expected due to moving parts, flowing liquids or chemical reactions. By inspecting these cavities regularly, damages that are invisible from the outside can be identified and their growth being monitored. Possible damages can include cracks, deformations, missing material or parts, burn throughs, abrasion, corrosion or changes in color. Typically, flexible or rigid borescopes are inserted manually through available gaps, openings or bores in order to investigate the inner health condition of the machine. Hereby, inspection intervals can be further extended and a possible disassembly may be reasonably postponed.

Dedicated professionals need to be available at the site to carry out diagnoses and inspections together, since each diagnosis highly relies on the subjective and variant way of conducting the corresponding inspection. Therefore, special expertise is mandatory for both aspects. For the sake of more valuable results and an improved overall efficiency a separation of these two tasks promises potential. Like that, inspections and diagnoses can be fulfilled independently from each other and thus, experts are only required for the latter. This demands for an inspection approach being invariant of the employee, promising a higher capacity of inspections and the conduction of remote diagnoses.

In some cases, especially found unfavorable when inspecting toroidal cavities, the borescope may be subject to gravity. This, in combination with limited steering and controlling abilities through the inner spaces, can make it challenging satisfying the high standards or inspecting these parts at all. Hence, a generic and more robust approach is needed to help overcome these limitations.

The novel solution proposed in this work aims for a defined and yet contactless probe guidance on a circular path, being adjustable towards certain sensor characteristics. The path is described by the distal end of a material compound tube, which is formed in a certain manner. The overall system is related to continuum robotics and is particularly suitable for inspecting toroidal cavities with hard to reach areas. In contrast to known high-precision solutions [1] this highly automatable and minimally invasive system requires no disassembly of parts.

Next, related work is reviewed followed by a description of the proposed solution and an experimental investigation of its capabilities. Certain characteristics of the system are then identified and possible limitations discussed. This paper is concluded with required future work.

#### 2 Related Work

In this section an overview of recent developments in the field of endoscopy is given followed by selected continuum robots that can be used for inspection purposes and tube forming as the means on which this work is based on.

#### 2.1 Endoscopy

An endoscope, as shown in Figure 1a, mainly consists of a flexible hose or a rigid tube. Devices being used in the technical sector are typically referred to as borescopes. The hose can hold different kinds of conductors, optical fibers or tethers and is very resistive against environmental impacts as well as mechanical stresses. Endoscopes are available in various diameters and lengths. Special optics for imaging as well as for lighting are located at the distal end of the hose. Conventionally, optical fibers are used for lighting and in the form of bundles for imaging also. A respective light source is then located inside of a control device which also offers a representation of the image on a screen. Typically, the probe directly contains multiple sensor types for imaging and other purposes. Hence, no optical fibers are needed anymore and several electrical conductors can be used. The distal end can mostly be actuated in order to manipulate the direction the head is facing. This can be controlled using tethers via the control device.

Recent devices offer intuitive user interfaces simplifying the navigation through the part of interest as well as offering numerous aids for measuring and documenting features of interest. High resolution imaging sensors, which can also be arranged in pairs to allow for stereo vision, provide the detail of information needed. Recent publications describe advanced localization and real time tracking techniques based on these. Examples include the use of optical models for improvements in accuracy of endoscopic measurements [8], photogrammetry for obtaining 3D maps on a larger scale [3], [4] or approaches for localizing the probe relatively against the inner spaces using triangulation and other optical solutions [11] or even using shape sensing of the hose itself [14]. Additionally, some authors have claimed applications in microscopy using endoscopes [9].

#### 2.2 Continuum Robotics

In contrast to conventional industrial robots respective systems in continuum robotics usually consist of joints in high quantities that are chained together (Figure 1b). Beyond the capability of industrial robots of moving the flange within any quantity of poses these systems additionally allow for keeping the robots structure itself close to a defined trajectory in space. This enables for reaching out to inaccessible areas while still being able of avoiding obstacles. Such robots exist in various magnitudes of scale while especially those from the fields of endoscopic robotics as well as surgical applications are of greatest relevance to the objective addressed in this work.

Endoscopic robots are designed to fit through small hollow spaces while being required to describe winding curves. This is typically achieved by involving numerous individual joints. These may be actuated using conventional tethers, compressed or decompressed fluids, electromagnetically by coils or local motors in combination with gear boxes in the joints or even using piezoelectric elements for instance [12]. Possible configurations may be divided into those whose shape evolves with them being fed into the inner spaces of interest and after that cannot be changed anymore and more complex systems that are capable of freely transforming their shape at any time. The latter are known as snake robots [5].

The field of soft robotics comprises mechanisms that don't rely on rigid joints but on flexible ones rather. These often attempt to imitate natural organisms and take advantage of various physical phenomena. These may be pressure, thermoelastic behavior, friction, electromagnetism or the use of tethers for instance. Advantages are adaptability to and conservation of their respective environment, enhanced safety of operation in the presence of humans or smaller magnitudes of scale of the mechanisms being possible.

Robots being combinations of both fields exists such as concentric tube robots

for instance [6]. These actually do not consist of distinct joints but allow for following certain curves due to pre-bent concentric tube segments. Their constitution is not rigid but flexible. These robots do not suffice the requirements within this application, therefore tube forming is introduced in the next subsection as a means for achieving the task.

#### 2.3 Tube Forming

Various tube bending forming techniques have been developed along with the arising respective demands for strong and lightweight bended tubes in different industries such as aviation, aerospace, automotive and more. Hence, certain manufacturing parameters have improved and further developed comprising possible bending radii, bending angles and shapes, material characteristics and manufacturing tolerances. Tube forming can be conducted either cold or warm and under different loading conditions like multipoint bending, compression bending, stretch bending, roll bending, rotary draw bending or laser bending. Different tube materials can be processed and different basic cross-sectional shapes of tubes be used such as circular, rectangular or even arbitrary profiles. Fabrication can occur seamlessly or welded for instance. Depending on the chosen process parameters and the wrought material highly diverse magnitudes of forces can be involved in the fabrication process. Tube forming introduces some challenges though like possible defects in form of wrinkling, over thinning, cross section distortion or spring back due to complex and uneven tension and stress distributions as well as special requirements to the tools. A robust prediction and control of the phenomena is a common topic of research by using experimental, physical, analytical or numerical approaches [13].

Most commonly used for obtaining complex multi dimensional geometries is roll push bending (Figure 1c), where continuously applied mechanisms like feed and torsion are used in combination with certain displacements of the rolls and furthermore investigations on process models exist [2], [7], [10]. This technology is also available fully computer controlled.

Each forming process constitutes of plastic and elastic deformation behavior of the material used. The latter occurs in the form of spring back of the material.

(a) Schematic endoscope (b) Continuum mechanism (c) Roll push bending

Fig. 1. Related work

#### 3 System Description

In the following section the novel inspection system as well as the path planning strategy are described and their potential investigated using an experimental setup. The system's characteristics are evaluated in more detail eventually.

#### 3.1 System Definition

This work aims at the generation of circular paths for the purpose of probe guidance. The underlying core mechanism is an adapted and extended kind of tube forming technology. By forming a special material compound tube the evolving trajectory described by its distal end is used to approximate the desired path and thus, to carry a probe. Further key components of the system include a guide tube (Figure 2a) and a possible actuation unit.

The forming process may be conducted manually under certain limitations regarding the achievable accuracy and repeatability of the path. Incorporating an optional actuation unit helps enhancing the results. But still, generating the desired path is not trivial, unwanted deviations can occur easily and in addition multiple curvatures are required to enter the cavity of interest successfully. Similar to conventional forming strategies, where both plastic and elastic deformational behavior occurs, this approach makes use of both of these effects. Here, especially the elastic component is evoked intentionally. Along with basic material feed an additional and distinct portion of torsion may be used to influence the evolving circular path's direction and tilt. Furthermore, distinct elastic pre-deformations of different magnitudes and orientations can be induced using forming mechanisms like bending the material forth and back.

(a) System schematic (b) Roll arrangement (c) Material compound

Fig. 2. Proposed solution

The entire forming strategy is organized into two stages. Abovementioned complex forming mechanisms are conducted outside of the toroidal cavity to be inspected. An additional, subsequent shaping of the material takes place using a guide tube where also the majority of the circularity is induced. Prior to the path generation, the guide is inserted and positioned through one of the possible inspection openings. The guide tube's shape follows certain geometric rules. Clearances between the material compound tube and the guide tube's inner surfaces allow the previously applied forming steps to influence the shape, unfolding at the guide tube's outlet inside of the cavity. The guide tube furthermore allows for gaining access to hollow spaces that can only hardly be reached, through small openings or bores, for instance.

The material compound tube used here (Figure 2c) consists of a thin aluminum strip wrapped longitudinally, thus forming the shape of a tube. The aluminum strip is laminated with a protective foil on either side and is adhesively encapsulated by a jacket of polyethylene. Even though the material was originally developed for shielding telephone cables and preventing those from corrosion due to moisture <sup>1</sup> , its layer structure leads to certain material characteristics that are beneficial to this work. These do not only include its capability of carrying cables or tethers inside but also its low specific weight per length, its high robustness, its highly repeatable plastic deformability and its high dimensional stability when bending curves. These especially allow for the contactless and self-supporting realization of the generated path while carrying a probe at its distal end.

By adjusting the forming parameters, the guide tube's geometry and the material's dimensions paths of different magnitudes of scale may be realized and thus the method be used for inspecting a variety of parts. Beyond the use of a basic material and process model, additional localization techniques may be used to further improve accuracies and compensate deviations like sag, for instance.

#### 3.2 Path Planning

The path planning strategy feeds the two-stage forming process while taking different boundary conditions into account regarding the part, the sensor, the tube material and the forming capabilities. Figure 3 illustrates the path planning strategy. As an initial step, a geometric representation of the desired circular

Fig. 3. Path Planning Process

path is generated with respect to the toroidal cavity's 3D model facilitating an optimization and equalization of the distances between the sensor and the part's inner surfaces. Subsequently, the geometric representation of the circular path is partitioned into segments. The size of these equally sized segments is calculated based on the sensor's characteristics such as its aperture angle or the desired measurement density, for instance. The forming will therefore be conducted intermittently. Next, the different forming mechanisms such as feed, torsion and

<sup>1</sup> Jachimowicz, L.: Corrosion proof shielding tape for shielding telephone cables. US Patent 3,233,036 (1963).

bending needed for the first segment are quantified in order to achieve a certain shape unfolding at the guide tube's outlet with the distal end approximating the desired path as close as possible. The forming can immediately be carried out either manually or using a programmable actuation unit. The forming may be followed by several measurements using the probe at the approached location. After the completion of the measurements the path planning process returns to the computation of the next individual segment's forming parameters. By incorporating an optional localization technique as an additional part of the measurement sequence at each individual location a pose estimation may be implemented allowing for the closed loop control of deviation residues that may not be compensated using the open loop path generation. During the entire process the distal end's trajectory is crucial indeed, but not the evolving shape itself.

#### 3.3 Experimental Setup

An experimental setup is used to evaluate the proposed system. At first, the material compound tube is fed through an adjustable roll arrangement (Figure 2b) which is mounted on a rigid frame (Figure 4a). Like this, parameters can be tuned in order to reach the targeted shape and thus distal end trajectory. The roll arrangement is further used to derive and validate a suitable geometry of the guide tube and therefore miniaturization of the system. Possible future guide tubes are also tested in form of additively manufactured specimen of plastic or metal. An industrial robot serves as a programmable actuation unit and is able to handle the tube material using a gripper and allows to continuously monitor process forces using a load cell. The robot conducts the intermittent forming using the mechanisms described above.

#### (a) Setup with rolls and industrial robot (b) Lasertracker plot

Fig. 4. Experimental setup and circular trajectory plot

During the experiments it has been found that several influencing factors for the resulting curvatures exist. These do not only incorporate the plastic deformation due to the forming process but also effects in form of elastic spring back after exiting the guide tube, gravitation having an effect on the tube's own mass, conductors running inside or a connected probe, intended pre-deformations or inhomogeneity of the tube's layer structure.

Experiments have been conducted for gaining knowledge about realizable diameters of the circular path, control possibilities regarding diameter and trajectory deviations, load capabilities regarding probes or conductors, repeatability and reversibility of deformations, forces required for the forming work and the system's oscillation behavior.

#### 3.4 Results and System Characteristics

Using the tested material compound tube's diameter of 8 mm circular paths of up to 1 m in diameter could be formed, while an idealized circular shape could still be approximated. Accuracies can be reached down to a few centimeters. This was evaluated in detail using a laser tracker device. The plot (Figure 4b) shows the obtained path, which differs from the idealized circle with proceeding length, due to bending, caused by the tube's mass. In addition to its own weight the system is able to carry small loads of up to 50 g at its distal end as well as conductors inside of the tube. The forming mechanisms mentioned in the previous subsections can be used to control the circular path within some certain boundaries and to compensate for deviations, if identified. The tests revealed that the chosen material compound tube can be formed several dozens of times after which it tends to break. Even in case of fatigue no residues of the tube will remain since the polyethylene jacket still holds the compound together well. The system is prone to oscillations easily which originate from the forming process itself but also from external vibrations and from loads applied to the tube. Forming process loads were up to 80 N when pushing the material, but strongly depend on the roll arrangement, also. As it can be seen in Figure 4b, fast load application leads to significant oscillations (from 180 ° on), while smooth load application allows to avoid these. The optimized guide tubes derived from the roll arrangements allow for entering parts through small openings or bores far smaller than 20 mm.

#### 4 Discussion

Since the material compound only features a limited lifespan it needs to be a disposable component of the overall inspection system. The circular diameters that can be obtained from the forming process using the guide tube are only representative for the tested material diameter. Other magnitudes of scale are possible using different material sizes. Since the overall concept is self-supporting and thus aims at contactless path generation with respect to the part and furthermore no residues will remain inside in case of fatigue the system may also be used for inspecting sensitive areas. Making use of the forming mechanisms mentioned above with the intention of controlling the path would imply having information about the current pose available. While stationary operation is already proven, mobile operation at different sites may still be developed. Hereto, reducing the process forces would be crucial in order to be able to miniaturize possible mobile forming units. Using drive units the forming can be conducted synchronously with possible ongoing measurements, also. Loads like probes may be carried by the system but care must be taken of the oscillations the system is prone to. This may also be done with help of the forming strategy. On the other hand side this fact formulates certain requirements with respect to the sensors as well. The chances to miniaturize the roll setup in form of the guide tube enables for insertion of the system through tiny openings or bores. Special about the suggested solution is the fact that the forming is conducted bipartite. Like this, hard to reach inner spaces can be accessed and a circular path be unfolded.

## 5 Conclusion

#### 5.1 Summary

In this work a performant new approach for the contactless inspection of toroidal cavities using circular path generation is presented, especially characterized by its high automation capability. It fosters the separation of inspection and diagnosis tasks while eliminating several drawbacks of conventional inspections. The approach enables simplified access to inner spaces, that are hard to inspect. Intermittently generating distinct circular paths allows for systematic inspection while meeting the boundary conditions of possible probes as well as parts of interest, leading to more valuable and reliable results. The resulting probe guidance system is self-supporting and therefore clearances between sensors and surfaces to be inspected can be well adjusted for. The solution promises to be configurable for various parts of interest. A first evaluation of the novel solution has been conducted experimentally while its capabilities and limitations were pointed out. Using this systematic approach completeness and reproducibility of measurements can be significantly improved.

#### 5.2 Future Work

Certain actions need to be taken in order to further develop the proposed solution towards an applicable condition. These include:


Furthermore, an optimization of the forming process parameters needs to be conducted in order to reduce oscillations for instance. Using the advanced knowledge obtained by respective models of the system a design strategy for adapting the system for various inspection subjects is to be derived.

#### Acknowledgements

The work was performed in collaboration with Lufthansa Technik AG and funded by German Federal Ministry of Economic Affairs and Energy under grant number 20X1709.

#### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Multi-agent path planning: comparison of different behaviors in the case of collisions**

Michaela Krä<sup>1</sup> , Ludwig Vogt<sup>1</sup> , Valentin Spannagl<sup>1</sup> and Johannes Schilp1, 2

<sup>1</sup> Chair of Digital Manufacturing, Faculty of Applied Computer Science, Augsburg University, Eichleitnerstr. 30, 86159 Augsburg, Germany

<sup>2</sup> Fraunhofer Research Institution for Casting, Composite and Processing Technology – IGCV, Provinostr. 52, 86153 Augsburg, Germany

#### **Abstract.**

In the context of Industry 4.0 and cyber-physical production systems, the role of production logistics is perceived as more and more important in order to reach the overall manufacturing targets. One central aspect in organizing the flow of material consists in task allocation and path planning for transport resources disposing of growing autonomy. There are various approaches for multi-agent path planning as well as the way of dealing with collisions. Collisions are possible due to traffic volume and can either be treated on planning level or in a short-term way on control level.

The paper presents existing strategies for path finding before giving an overview of methods to deal with autonomous transport resources that meet in a manufacturing environment. Then, different existing behaviors and reactions in the case of collision detection based on several criteria are compared. This step allows classifying the strategies depending on the manufacturing environment and its organization.

**Keywords:** production logistics, autonomous transport resources, collision avoidance.

## **1 Introduction**

With the introduction of the concepts of Industry 4.0 like digitization and decentralization, various challenges arise in production environments. In addition, customer demand regarding increasing number of product variants as well as shorter product life cycles result in smaller lot sizes. [1, 2] Production logistics play an important role in the manufacturing process and companies understand more and more its influence on the throughput time [3].

The utilization of autonomous resources is supposed to turn logistics more efficient due to their possibility to organize themselves on a decentral level. This means that the operational level obtains decision-making autonomy within defined constraints. A challenge in this context is the balance between taking advantage of the autonomy and reaching central planning targets. One central aspect is the interaction of various resources with possibly different skill sets. [4–6]

Mobile autonomous resources, i.e. self-directed vehicles also called autonomous guided vehicles (AGV), take center stage in current research. There exist different concepts for multi-agent path finding (MAPF) presenting each advantages and disadvantages. [7, 8] Considering the range of concepts for MAPF and especially collision avoidance an evaluation of these approaches comparing their applicability in a specific manufacturing environment is missing. An approach is needed in order to identify a suitable strategy or set of strategies for avoiding collisions in production environments.

The scope of this paper is to give an overview of existing approaches for dealing with and avoiding collisions of autonomous transport resources. In addition, these ways are compared based on a choice of valuable criteria.

#### **2 State of the art**

Relevant literature concerning production logistics, autonomous resources and MAPF is summed up in this section to present applicable concepts. Before taking a deeper look on path planning and collision avoidance, a short introduction to logistics and mobile resources in general is given.

#### **2.1 Logistics and mobile resources**

As described in the introduction, AGVs are used for a variety of single and multi-agent applications but the main sector is logistic systems. The most common logistic scenarios are the transportation of dock containers in a terminal [9–11] and transportation services in warehouses as well as production lines [12]. While in the past, AGVs were only used for transportation services recent projects have also focused on expanding their skill set. Hoff and Sarker [13] have distinguished between six different major types of AGVs that include unit load, towing, parallel truck, forklift, light load and assembly line vehicles. The newest addition to these classic AGV systems have been drones which are tested in warehouses [14]. The vehicle design and fleet combination is just one of many central research topics in AGV Systems. Others involve the sensors, navigation, localization, vehicle control and the communication between the agents [15].

#### **2.2 Path planning for multiple resources**

The classical MAPF problem for a set of *n* agents receives an input tuple *(G,s,t)* where *G = (V,E)* is an undirected graph with *V* vertices denoting locations and *E* edges. *s : [1,…,n] - V* maps an agent to a start vertex , and *t : [1,…,k] - V* maps an agent to a target vertex. Often the graph is given with its corresponding adjacency matrix that displays the connected vertices and the weight of the edges (c.f. figure 1). For the solution of this problem, one can find methods, which take different constraints into consideration.

The Dijkstra´s algorithm is one of the most commonly known algorithms for the shortest path search [16]. It calculates the shortest path between two vertices in an undirected grid.

**Fig. 1.** Representation of an undirected graph with its adjacency matrix.

An evolution of Dijkstra´s algorithm is the A\*Algorithm, which estimates the distance, i.e. costs, from the start vertex to the target vertex to preselect appropriate vertices with a heuristic [17]. Warren [18] proposes a modified A\*Algorithm which is faster than the normal A\*Algorithm due to a further reduced solution quantity.. Both the Dijkstra and A\*Algorithm are not flexible and cannot react to a changing environment [19]. Therefor the D\*Algorithm (Dynamic A\*) was developed by Stentz [19] as an extension of the A\*Algorithm. Koenig and Likhachev [20] propose the D\*Lite Algorithm which combines the heuristic approach of the A\*Algorithm with the incremental one of the D\*Algorithm. Other extensions of the D\*Algorithm are the Focussed D\* and Field D\*Algorithm. The Focussed D\*Algorithm focusses on the repair of the cost map, thus reducing the computational costs [21]. While all of these approaches are limited by a discrete set of possible transitions in the grid the Field D\*Algorithm is interpolation based and calculates smooth paths for the agents [22].

These algorithms were developed for single agent path planning but can also be used for MAPF problems. The M\*Algorithm is specified for MAPF problems and minimizes the global cumulative cost function of all agents [23]. An extension of the M\*Algorithm is the M\*UM\*Algorithm which considers uncertainty in the position and pose of the agent [24, 25]. Nebel et al. [26] propose an M\*Algorithm for uncertainty in the destination of the agents path while Ma et al. [27] developed a MAPF algorithm with delay probabilities.

#### **2.3 Collision detection and avoidance**

One of the main topics within the MAPF problem is the collision detection and avoidance between the agents. There are two main approaches for the collision detection: one is a centralized and the other one a decentralized. For the centralized approach the information about the actual position, path and speed must be collected in a central location [28]. Felner et al. [29] are adding a heuristic to the Conflict-Based Search (CBS) by aggregating collisions among the agents. While this approach uses discretized time steps and assumes that all actions have the same time duration Andreychuk et al. [30] propose a Continuous Time Conflict-Based Search (CCBS).

The decentralized approach enables the agents to detect collisions by themselves and react accordingly. Wei et al. [31] present an approach where only the affected agents need to communicate with each other in case of a collision. Trodden and Richards [32] use a Model Predictive Control (MPC) where only one agent replans, while all others continue with their path. The agent that replans its path is identified via a bidding system, where the highest bidder receives the replanning token in each iteration. Desaraju and How [33] use a two-step approach where the first step is the individual component and the second step is the interaction component. While in the first step each agent minimizes the cost of its own paths, the interaction component minimizes the total cost across all agents.

If a collision is detected, researchers propose a variety of strategies for the behavior of the involved resources. Liu et al. [34] plan paths for a multi-agent pickup and delivery problem where no paths cross each other during the duration of the exercised task. A similar approach is used by Kiarostami et al. [35] where a Monte-Carlo Tree Search is used to find non-overlapping paths.

Sun et al. [36] developed a decoupled approach where every agent's path is planned with an A\*Algorithm. Every agent continuously checks its front area and if another agent is detected in a critical region, a behavior-based collision avoidance strategy is executed. The behaviors are specialized for an intralogistic problem description and selected by specified traffic rules. Another decentralized approach where possible collisions are detected by the agent itself is proposed by Gochev et al [37]. Instead of a rule based collision avoidance the agents switch in a collision avoidance mode. In this mode the agents create a roundabout along the original path which is faster than stepping back and changing directions. Chang et al. [38] are using a detection shell around each agent and according to the nearest object inside this detection shell breaking and gyroscopic forces are calculated.

A five layer neural network with three hidden layers which is able to generate crossing paths and the associated velocity vectors of the agents so that no collision occurs is presented by Chen et al [39]. This approach is based on the optimal reciprocal collision avoidance (ORCA) which is often used for the collision avoidance in free navigating agents [40].

#### **3 Concept**

Based on the approaches presented in chapter 2, criteria to be considered in manufacturing systems are identified before giving a comparison of the concepts for collision avoidance. For the use of autonomous transport resources in real manufacturing environments, this is a necessary step in order to choose appropriate strategies depending on individual production constraints.

#### **3.1 Criteria to compare collision avoidance strategies**

To evaluate the different collision avoidance strategies it is necessary to define the environmental conditions. The criteria to compare the different collision avoidance strategies are not to be confused with the environmental settings. Since there exists no past work on the ranking or comparison of these influences for the different algorithms we have identified the following environmental influences which were the most common research topics in chapter 2.2:


All of these aspects have to be considered not in an isolated test area, but in real manufacturing conditions, the previously mentioned environmental settings. These comprise the next parameters based on Adam [41]:


#### **3.2 Comparison of different collision avoidance strategies**

There exist many different approaches for AGVs to behave in the case a collision is detected. We have identified five basic strategies from the research done in chapter 2.3 and the collision avoidance behavior presented by Sun et al. [36]. Though they explain eight different behaviors, all of them can be reduced to two basic strategies. That is why we adapted the avoiding and waiting strategies of Sun et al. plus the variation of the start time, no crossing paths and a change of velocities for our evaluation. All strategies are further described in the following segment. To avoid repetitions it is assumed that the paths of the agents cross at vertex *n*, the velocities of the agents are *v* and the distance from the start vertex to vertex *n* is *d*.


**Fig. 2.** Two crossing paths where the agents have different priorities. agent 1 passes vertex *n* before agent 2 because of a higher priority.


### **4 Application and evaluation settings**

The environment for the test cases is a ROS based path planner and its simulation environment. A test scenario is set up to evaluate different criteria for the collision avoidance strategies presented in chapter 3.2. The parameters mentioned in chapter 3.1 are meant to be analyzed and structured regarding their influence on the choice of suitable collision avoidance strategies. They make the simulation better comparable to real life applications but also computational expensive.

Workers are seen as obstacles and these will cross and obstruct the planned paths at a certain percentage. The duration of the obstruction is determined with a Gaussian normal distribution. In addition, we assume that the workstations have different states,

so the vertices of the grid are changing. Vertices will have the capacity one, so only one agent can occupy each vertex at a specific time step. The edges also have a capacity of one. Despite this, it is possible that multiple agents are in the same cross or pass each other in a lane because the ROS path planner builds its own grid on the prescribed map.

To evaluate and compare the different strategies we analyze a set of key performance indicators in the simulation. Since most researchers try to minimize the sum of costs of all agents this is one of the identified parameters. By analyzing additional parameters, the influences of the different collision avoidance strategies can be shown. Furthermore, we analyze the traffic which is generated from the different strategies and also the grid capacity utilization. With these parameters, we can make a first assessment of how the applied strategies influence workers or how easily additional orders can be implemented in the plan:


The first tests in a simulation environment compare the average throughput time of four AGVs with each executing orders of two or three jobs for the strategies priority and path crossing (cf. chapter 3.2). The results for the mean value of the order duration show that avoiding collisions by circumnavigating already planned paths leads, as expected, to 10 up to 30 % longer throughput times. It is evident that these strategies depend highly on the considered manufacturing layout and the existing traffic. This needs to be regarded in a more detailed way.

### **5 Conclusion**

Current challenges linked to the concepts of Industry 4.0 affect in particular production logistics. This comprises among others the need of more flexible solutions for material transport in manufacturing environments. Thus, the use of autonomous resources increases as these means of transport present advantages compared to standard logistic solutions and as they allow relocating decisions on resource level.

This contribution presents an approach to compare different ways of dealing with collisions between AGVs based on valuable criteria. The treated types of collision avoidance were path crossing, priorities, start time, velocities and internal collision avoidance mode. Relevant criteria to evaluate these strategies are order duration, time constraints and prioritization.

The next step will be a more extensive evaluation, validation and verification of the concept presented in chapter 3 within a simulation scenario. These analyses will aim at verifying the applicability of the stated criteria and complete them if necessary. Additional varying speeds of the vehicles on their way through the production environment have to be implemented to make the application more realistic.

#### **Acknowledgements.**

The presented results have been developed within the research project FORobotics (www.forobotics.de). The Bavarian Research Foundation funds this project.

## **References**


on Automation Science and Engineering 13(4): 1433–1447. doi: 10.1109/TASE.2016.2603781


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Trajectory Optimization Methods for Robotic Cells Considering Energy Efficiency and Collisions

Elias Knochelmann ¨ 1 , Dwayne Steinke<sup>1</sup> , Joel Greenyer<sup>2</sup> , Svenja Spindeldreier<sup>1</sup> , and Tobias Ortmaier<sup>1</sup>

1 Institute of Mechatronic Systems, Leibniz Universitat Hannover, Hannover, Germany ¨ elias.knoechelmann@imes.uni-hannover.de,

2 Software Engineering Group, Leibniz Universitat Hannover, Hannover, Germany ¨

Abstract. In production robots are moved at maximum speed whenever possible in order to achieve the shortest overall cycle time. This can lead to individual waiting times, especially in interlinked production processes. These waiting times offer opportunities for optimization. Due to high energy prices and political efforts, energy efficiency has become the focus of trajectory optimization in recent years. Robot cells with a common intermediate circuit offer the possibility of energy exchange across individual axes or robots. By adapting the robot trajectories, the total power consumption of a robotic cell on the grid side can be significantly reduced. This paper focuses on trajectory optimization, whereby a detailed collision detection of individual robots is included within the analysis. It is shown that with collision detection energy optimization for cramped robot cells becomes possible and the losses in efficiency compared to the optimization without it are minute.

### 1 Introduction

Reducing CO<sup>2</sup> emissions through higher energy efficiency is one of the main goals of our time. Operating costs of factories can also be reduced by lowering the energy consumption of its components. As automation advances, the number of robots in production is constantly increasing [22]. This has also been recognised by the EU. An energy roadmap [7] has been issued in 2014 with the goal of increasing energy efficiency by 27 % until 2030. The aim of this work is to reduce the energy consumption of industrial robot cells by adapting robot movements. The method is evaluated using the robot cell shown in Fig. 1.

The reduction of the energy consumption of robots can either be hardware-

Fig. 1: Investigated robot cell packing a pallet, with several potential collisions.

based or software-based [5]. Examples for hardware changes are e.g. extension of the

*Handling and Industrial Robotics*, https://doi.org/10.1007/978-3-662-61755-7\_21

T. Schüppstuhl et al. (Hrsg.), *Annals of Scientific Society for Assembly,*

DC link to several robots [27] or the installation of energy storage devices [13]. With these two methods recuperated braking energy from the motors is stored or used for other robots. This means that recuperated energy for overvoltage protection does not have to be converted into heat [27]. However, these methods have the disadvantage that additional adaptation costs are incurred per robot and that systems for installation have longer downtimes.

In the case of software solutions, costs are only incurred during development, and the transfer to several systems is negligible. Software solutions are a geometrically or time sequence change to a robot trajectory [5]. These solutions are a part of trajectory optimizations and usually refers to one robot. In [30] a robot trajectory was decomposed into viapoints and the position of these were optimized. In [15] the energy absorption of a trajectory was optimized based on maximizing the energy exchange in the DC link. Another possibility is a different time sequence of robot trajectories for a given task. In most cases several robots are involved. One method in this field is, instead of waiting after fast movements, the robots movement may be slowed down. In [9] different robot trajectories were investigated and it was found that slowing down trajectories almost always saves energy. However, movements can also be shifted in time so that braking movements and acceleration, from different robots/axis, directly overlap over each other. Of course this only has advantages if robots can exchange energy.

In industry this is not possible at the moment, but by implementing a DC factory, an energy exchange at factory level becomes possible [27]. In this paper it is assumed that the robots have such a connection. The method of sliding over each other can also be combined with the slowing down of fast movements. In [14] such an optimization for a multi-axis system was presented. Losses from overvoltage protection are reduced for one cyclic trajectory per axis. A method for scheduled energy optimal trajectories was presented in [29] and great energy savings were shown for a single robot. When considering geometric path changes or when changing the time sequence of several robot movements, it is important that no collisions occur. [28] took collisions into account, but only for a restricted area and [24] used simple bounding boxes. In [2] an iterative collision detection for a robot was presented, which guarantees collision freedom for the next pose. However, to the best knowledge of the authors, for larger linked movements of several adjacent robots with a detailed collisions detection, as shown in Fig. 1, no uniform method for trajectory optimization was published.

An automated methodology to exploit braking energy recuperation though scheduling and slowing down trajectories was proposed in [12]. The proposed methodology accomplishes this in two levels. In the first level, the sequential control optimization (SCO), the overall discrete control of the system is optimized to maximize the times in which the acceleration and deceleration phases of individual robots beneficially overlap. Usually, the order of robot movements can be arranged in multiple ways, and some control strategies offer more potential for braking energy recuperation than others. Then, in the second level, the detailed trajectory optimization (DTO), for each set of braking and acceleration phases that an optimized control strategy brought to overlap, the trajectories of the involved axes must be aligned in detail. For example, the braking phase of one axis can be shifted, stretched, or compressed to match the energy demand for accelerating another axis. In this paper, we focus on the DTO, a method for the overlying SCO was already published in [12].

The SCO calculates the acceleration and deceleration phase of robots that should overlap to maximize the energy efficiency gain. These time points and related robot paths are given to the DTO. Here the robot trajectories are improved in an model-based offline optimization. The two internal robot control variables (*start delay* and *compression factor*) are used as optimization parameters. This guarantees an intuitive transferability to industrial plants. The entire process is model-based and offline, therefore only the newly calculated parameters must be changed in the controller. In addition, a collision detection is presented, which can be used as a nonlinear constraint within the optimization. The collision detection method was only applied to DTO, because collision in robot cells, as the one sketched in Fig. 1, are a matter of trajectory timing. This collision detection routine is a two-layered process [4] to guarantee a fast and accurate detection. The method will be tested in the simulated robot cell shown in Fig. 1.

In Sect. 2 the energy model used to simulate the overall energy consumption of a production process is described. Sect. 3 builds on the model of Sect. 2 to optimize the overall energy consumption. The optimization approach and the applied constrains are outlined in Sect. 3. To prevent collisions, which can occur during optimization from Sect. 3, an collision detection is described in Sect. 4. Results are evaluated in Sect. 5 and the paper concludes in Sect. 6.

#### 2 Modeling of Industrial Robots

In this section the robot model used in optimization will be presented. Since the model is often invoked within the optimization, one of the main requirements, besides the accuracy, is a short computing time. The complete model was already presented in [12] but it will be described roughly for the sake of completeness. The general structure of the energy flow is shown in Fig. 2.

The power flows from the main grid Pgrid to the constant consumers Pl,grid. These are normal losses caused by fans and controllers. Besides that, the main part of Pgrid flows thought the rectifier TDC,<sup>1</sup> into the DC intermediate circuit. The DC intermediate circuit mainly supplies the inverters TAC1−<sup>n</sup> , which in turn supply the motors M1−n, whereby n represents the numbers of inverters/motors of each robot. Considering the gear, the motors generate a joint torque τ and a joint speed ϕ˙. This drives the segments of the robot to follow a given path

Fig. 2: Detailed energy flow in the investigated robots.

q(t), q˙(t), q¨(t) with an additional

weight fext. When the robot slows

down, the motors turn into generators and recuperate energy. This energy is converted to DC and can supply other robots via the DC link. If too much power flows into the DC bus, the DC bus voltage increases. As overvoltage protection, a braking resistor Rchp converts the excess energy into heat. The complete energy model of the robot can be written as:

$$E = \int\_0^{t\_{\rm end}} P\_{\rm grid}(t) \mathrm{d}t = \mathbf{f}\_{\rm m}(\mathbf{q}(t), \dot{\mathbf{q}}(t), \ddot{\mathbf{q}}(t), \mathbf{f}\_{\rm ext}). \tag{1}$$

For a known robot, a robot path q,q˙ ,q¨ and an extra weight fext can be used to calculate the required energy. The model is nonlinear and therefore no gradient based methods can be used for optimization.

The energy consumption model is validated for multiple different KUKA industrial robots [17] and has a remaining model deviation of approx. 5 % over the whole spectrum of operating temperatures [8].

### 3 Optimization Strategy

Common industry robots have six joints, but the axes movements are all synchronized to the slowest joint. The output of a production, therefore, depends on the slowest axis movement. An unnecessarily strong acceleration of the other axes has no advantage with regard to the quantity produced. The synchronization can, therefore, lead to energy savings, but there is no recuperation from decelerating, because all the joints are slowing down at the same time. The main goal of the optimization is to make all robot movements equally fast/slow and prevent unnecessarily strong acceleration. The optimization problem thus consists of optimization variables for compression of the trajectory δov,j and a start delay τs,j for a robot j to synchronize accelerating and decelerating from different robots. The effect of these variables on a single joint robot axis can be seen in Fig. 3.

Fig. 3: Visualisation of the used optimization variables.

The start delay τs,j for a robot j shifts the trajectory n - time steps into the future. The lower limit is zero and the upper limit is the travel time for that trajectory of the slowest robot τtr,krit subtracted by the travel time τtr,j of the robot j. By optimizing multiple robots with only one trajectory each, the travel time of the slowest robot τtr,krit equals the total travel time τtot, otherwise it is the sum of the the slowest trajectory τtrl,krit of the respective l multi-robot movement. The compression variable δov,j reduces the maximal velocity q˙<sup>j</sup> and acceleration q¨<sup>j</sup> of the trajectory:

$$
\dot{q}\_{\text{new},j} = \delta\_{\text{ov},j} \cdot \dot{q}\_{\text{org},j} \quad \wedge \quad \ddot{q}\_{\text{new},j} = \delta\_{\text{ov},j} \cdot \ddot{q}\_{\text{org},j}.\tag{2}
$$

The travel time of the robot j changes to <sup>τ</sup>tr,j δov,j . The lower limit is 0 and the upper limit is 1. The optimization problem finally can be described by:

$$\min\_{\boldsymbol{\pi}\_{\text{v}}, \boldsymbol{\delta}\_{\text{ov}} \in \mathbb{R}} E\_{\text{grid}}(\boldsymbol{\pi}\_{\text{s}}, \boldsymbol{\delta}\_{\text{ov}}) \text{ s.t. } 0 \le \boldsymbol{\delta}\_{\text{ov}, j} \le 1 \tag{3}$$

$$0 \le \tau\_{\mathbf{s},j} \le \tau\_{\mathbf{tr},\mathbf{k}\text{rit}} - \tau\_{\mathbf{tr},j} \tag{4}$$

$$t(\tau\_{\sf s}, \delta\_{\rm ov}) \le \tau\_{\rm tot} \tag{5}$$

$$c(\tau\_\ast, \delta\_\text{ov}) < 1,\tag{6}$$

where two new nonlinear constraints t(·) and c(·) are introduced. The first constraint makes sure that the total travel is at least not slower than before the optimization. c(·) is a nonlinear function that returns one if a collision is detected. The collision detection will be described in the following section and is skipped here. With this the optimization problem for multiple robots and multiple trajectories is fully defined.

The non-linear optimization problem and especially the nonlinear constraints result in gaps in the solution area. For that reason a particle swarm optimizer (PSO [26]) was chosen. The PSO does not guarantee that the global minimum will be found, but returns an acceptable solution in reasonable time. A more detailed view on constraints handling and the gaps in the solution area will follow in the next section.

For runtime reasons it makes sense to split the optimization into several sub-problems and solve them individually. Sub-problems are single trajectories of each of the participating robots, so τtr,krit equals τtot. The described optimization is only the secondary level, so the SCO matches the trajectories with the most energy saving potential. Since it is already known from the SCO which trajectories of the different robots have to be superimposed, there is no need to optimize ntr trajectories of m robots. It is sufficient if only one trajectory per robot, selected by SCO, is optimized in one sub problem.

The drawback of this method is that the success of the underlying optimization depends on the SCO. By dividing the optimization problem, a satisfactory solution in a satisfactory time is obtained, but if the SCO does not find the best matches, the solution is not the best possible. However, without dividing an exemplary test case with 3 robots and 10 trajectories each can no longer be solved on a PC in a satisfactory time.

The fact that only one trajectory per robot is considered simplifies the constraints of optimization. The lower limit for the compression factor δov,j of robot j is now limited by the travel time of the slowest robot in the considered subproblem. Before there were multiple trajectories of the same robot in a row, so there was no defined end of a single trajectory, because trajectories could overlap. The start and end time of the critical robots are known by the SCO and therefore the trajectories can be separated. The upper limit for the start delay now also depends on the compression factor. In contrast to the standard problem, again without overlap, the formulation of the constraints is easier. This simplifies the constraints to:

$$\frac{\tau\_{\text{tr},j}}{\tau\_{\text{tr},\text{krit}}} \le \delta\_{\text{ov},j} \le 1,\tag{7}$$

$$0 \le \tau\_{\rm s,j} \le \tau\_{\rm tr,krit} - (\tau\_{\rm tr,i} \cdot \delta\_{\rm ov,j}),\tag{8}$$

$$c(\tau\_s, \delta\_{\rm av}) < 1. \tag{9}$$

So there is no need for a nonlinear time constraint, because only through linear constraints it can be ensured that the total travel time τtot remains the same. The collision detection c(·) stays the same and will be described in the next section.

#### 4 Collision Detection

Using the model from Sec. 2 the optimization strategy from Sec. 3 works well if the robots do not share a common workspace. However, if two or more robots share a workspace there is the possibility that optimized trajectories may lead to collisions even if non-optimized trajectories do not collide. For example, there are two robots (A and B) standing across each other. Robot A is moving vertically and robot B is moving horizontally, so there is a possibility that these two are colliding in the middle. Assuming that robot B is the critical/ slowest robot, then there are two optimization parameters: The compression factor (override) and trajectory start delay of robot A. The collision detection leads now to a divided solution space as shown in Fig. 4. This (red) barrier depends on the compression factor and the start delay and marks a collision. Because of this every generated trajectory of the optimization needs to be checked for collision to only consider valid solutions.

Fig. 4: Solution space divided by collisions of a two robot cell optimization problem

The collision model uses CAD data of the robots as well as other objects in the cell and imports it as a polygon mesh model. For every sample time step the positions of the objects are recalculated from the robot joint angles of the current trajectory. If no collision is found for all sample time steps, the current parameter vector is valid and can be used in optimization.

The repeating collision detection routine is, as shown in [4, 18, 6], divided into a broad phase that approximates the objects with bounding boxes and a narrow phase which determines exactly if two objects collide. The broad phase calculates candidates that may collide and need further investigation in the narrow phase. In the past lots of algorithms and solutions for use in broad phase [20, 6, 11, 19, 16] and narrow phase [21, 23, 10, 3, 1, 25] with advantages and disadvantages depending on the number of objects, complexity of the meshes or compactness of the objects have been developed.

Therefore for the collision detection routine within the trajectory optimization three different methods for the broad phase (axis-aligned bounding box pair test (AABB) [6], the Sweep and Prune algorithm (S&P) [6] and an oriented bounding box pair test (OBB)[11]) and two for the narrow phase (V-Clip [23] and the Gilbert–Johnson–Keerthi separating-axis algorithm (GJK) [1]) were implemented and evaluated. The selection was made on the basis of speed, implementation effort and availability of literature.

To compare the selected algorithms three runtime tests were performed. This and all other test were conducted in MATLAB on an i5-6400 desktop computer. In order to guarantee comparability with the robot cell application, three tests with one trajectory each time were performed for the set-up of the test cell (Fig. 1) . Table 1 shows the average calculation times in ms for the collision check for each case. Each test was repeated 200 times and the average value was obtained. Due to the small number of tests, the deviation is very high (up to 10%), but a generalized statements can still be derived: Sweep-and-Prune shows its advantages in this application-related test, because the system is made up of many objects that move only partially, which is the strong point of this algorithm. Further detailed tests have shown that GJK is faster than V-Clip for the narrow phase, therefore GJK and S&P were chosen as the best solutions.

To speed up the optimization it is also possible to only examine those objects that can collide. With the proposed optimization strategy the robots change their trajectory but not their path. Two objects can only collide if the spatial paths, for a given optimization parameter set, cross. The collision detection approach is therefore expanded by an extra precomputation phase. There, in advance the objects that have the potential to collide will be determined. The precomputation phase is not exact but an approximation, because the reconstruction of a polygonal mesh from multiple meshes is too complex for a preexamination. This generates from any parameter set a point cloud for each object consisting of all vertices from all collision sample time steps. These point clouds represent the paths with all time dependencies eliminated. If two points from different



Fig. 5: Power demand and relative robot speeds for the same trajectory

point clouds are within a detection radius r they are marked as possibly colliding and relevant for the collision detection routine. All other objects in this robot cell e. g. robot bases, fences, conveyor or similar are irrelevant, as they don't change their position and can never collide with the robots.

To integrate the collision detection into the PSO three types for restraining the nonlinear constraints are available: absorbing walls, reflecting walls and invisible walls [26]. Fig. 4 shows that a divided solution space can result from the layout of the cell. Since the particles of the chosen PSO algorithm must be able to diverge through these boundaries, only the invisible walls [26] are suitable for constraints handling.

#### 5 Results

Fig. 5 shows the simulation results of an example system consisting of three robots as displayed in Fig. 1. In Fig 5 the overall grid power demand and the DC bus power flow between each individual robot and the DC bus in comparison to the relative robot speed Vref is shown. The relative robot speed is defined as the absolute sum of all robot joint velocities standardised to the maximum value of the slowest axis of the slowest robot. The travel time of robot 2 is the longest. Therefore, robot 2 is the critical robot of this example system. Its trajectory parameters are not optimized and the movement does not change between the portrayed optimization types. Fig. 5(a) shows the non-optimized

Fig. 6: Comparison of the energy demand with different optimization strategies

case. Each robot moves from its start point to its endpoint as fast as possible. This results in wait times for robots 1 and 3 and high grid power demands while acceleration.

Fig. 5(b) shows the case of optimization without considering collisions. The optimization leads to slower movements of robots 1 and 3 in such a way that all robot trajectories are synchronized. However these trajectories lead between 1.68 s and 1.78 s to a collision between robot 1 and 2 and are therefore not feasible. The time of the collision is marked in red. Using the nonlinear collision detection constraint as shown in Fig. 5(c) this problem is solved because robot 1 moves faster and avoids the collision.

Fig. 6(a) shows a comparison between the energy demands of the different optimization strategies. Without any kind of optimization the example system consumes 4563 J. An optimization without collision detection leads to an energy demand of 3983 J which is a saving of approximate 12.7 %. Changes to the energy demand of the example system through collision detection are minimal and lead to 3984 J. The optimisation results in Fig. 5(b) and (c) indicate that it is not as efficient to match acceleration and deceleration as it is to just compress the trajectories, because there the trajectories were simply stretched. This has two possible causes: First, the DC link capacity consists of three times the amount of a single robot. This means that more energy can be stored for a short time. Since this capacity is not used as an energy storage but mainly as a smoothing capacity, it will probably be smaller for industrial applications, because smaller means less investment costs. The second reason is, that by slowing down the trajectory the acceleration and deceleration was also reduced and thus also less braking energy is recuperated. These observations can be confirmed for a larger system consisting of 3 robots with 39 trajectories each, as shown in Fig. 6(b). By the sub problem optimization the overall energy saving is approximately 9 % compared to trajectories in which each robots moves as fast as possible and needs to wait for the critical robot. The overall results can be seen in Table 2.


Table 2: Simulation results for different types of trajectory optimisation for the test cell in Fig. 1 consisting of 3 robots with 39 trajectories each.

## 6 Conclusion

In this paper the second part of a novel engineering methodology for the development of energy-efficient production system controllers is presented. The functional principle and results of the first optimization level (SCO) have already been published in [12]. This publication presents the underlying optimization strategy (DTO) together with a collisions detection. The complete optimization problem was set up, with two optimization parameters i. e. start delay and compression factor. Based on the already presented SCO the optimization problem could be divided into multiple independent sub problems. Simplification for the optimization constraints were also be derived from this division, so that no non-linear constraints are required except for collision detection. Only by these simplifications it is possible to solve systems with more than 10 consecutive trajectories.

The disadvantage of this division is, however, that the success of this procedure depends heavily on the SCO. If the SCO does not find a good solution, the result of the DTO is also not optimal. For small cells with few trajectories (< 10) it may make sense to solve the general optimization problem.

The implemented two-step collision detection was presented, as well as a short runtime comparison of the different methods. In addition, a method was presented which will remove robots from the collision detection if they can never collide with other robots. All methods presented were tested on a virtual test cell ( Fig. 1). The DTO reduced the energy consumption when loading the pallet by 9 %. Especially due to the large DC-link capacity, the optimization did not match acceleration and deceleration phases, but slowed down the trajectories as much as possible. The collision detection has only marginally reduced the energy saving. With this method it is possible to optimize DC-powered robots even though the space in the cell is limited.

## Acknowledgment

The presented research is funded by German Research Foundation (Deutsche Forschungsgemeinschaft/DFG).

### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Localization and grasping of small load carriers with autonomous unmanned aerial vehicles

Markus Lieret<sup>1</sup> , Michael Lallinger<sup>1</sup> , Markus Tauscher<sup>1</sup> , and J¨org Franke<sup>1</sup>

Institute for Factory Automation and Production Systems (FAPS), Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg (FAU), Germany {markus.lieret,joerg.franke}@faps.fau.de, {michael.lallinger, markus.tauscher}@fau.de

Abstract. The application of unmanned aerial vehicles (UAV) in the area of inspection, survey or urban logistics has become a rapidly developing research domain. While the feasibility of material transports with UAVs has already been shown in the scope of different projects, the payload is thereby usually transferred manually into the UAV's load handling device. A decisive factor for the economic usability of UAVs for aerial transportation, however, is a fully automated system including the autonomous recognition and pick-up of the cargo. We therefore present a solution for the automated detection, localization and grasping of small load carriers with UAVs. The system includes a specialized load handing device, a camera-based real-time tracking solution for small load carriers and a fusion of the global and relative position measurements to achieve the in-flight positioning accuracy required for the autonomous cargo pick-up.

Keywords: autonomous aerial vehicles, aerial grasping, object localization

## 1 Introduction and related work

In recent years, UAVs have found their way into various industrial applications. In addition to tasks in the area of inspection or surveying, there are also various use cases in the field of material transport, as UAVs enable fast, highly flexible and direct material transport in the air. In addition to the transport of parcels in retail or the delivery of urgent medical products or devices, UAVs are also suitable for the transport of spare parts or urgently needed parts on the company premises or within a production hall. Due to their three-dimensional scope of movement UAVs are able to operate in the unused space above the production systems, extend the flow of materials by the third dimension and offer a highly flexible and fast transportation. [1]

While different publications have already shown that autonomous UAVs can provide an efficient addition to existing logistic systems there are still only few solutions or publications describing fully functional systems for the automated aerial in-house transportation [2]. Existing research and pilot projects on the aerial transportation with autonomous UAVs commonly do not focus on the autonomous cargo pick-up. [3] [4]

To enable autonomous aerial gripping of objects, different operating principles including suction-gripping, interlocking gripping or magnetic gripping have been developed during the recent years. Among the most common design are mechanic finger or jaw grippers as they are easy to produce, robust and can handle objects independently of their material and shape. However, they require a precise positioning of the UAV or an underactuated and flexible design to be able to grasp objects properly.

To compensate the displacement between the UAV and the target object caused by inaccuracies during manual flight Pounds et al. utilize an underactuated four finger gripper for the reliable in-flight grasping of different objects. Each finger features two elastic joints to compensate the contact forces between gripper and object, resulting from the positional fluctuation of the UAV. [5] [6]

Optical tracking systems such as motion capturing systems allow a precise tracking of UAVs with an accuracy below 1 mm. The usage of such systems allows the application of fully actuated grippers in combination with autonomous drones. Qi et. al. utilize a basic single-DOF finger gripper for the successful aerial gripping of a cuboid [7].

The usage of interlocking connections for aerial grasping has been shown in [8] and [9]. Thereby pins or clamps are actuated by servo motors to penetrate the cargo objects and to attain a secure fastening of the object.

While the automated cargo pick-up is considered a key requirement for the cost-efficient in-house aerial transportation, the state of the art lacks of working solutions in this area. We thus present a system architecture for the autonomous detection and grasping of small load carriers with autonomous UAVs. The system includes an autonomous hexarotor system equipped with a camera system to detect and locate the cargo objects and an appropriate load handling device to realize an autonomous grasping.

#### 2 System architecture

To ensure an economic usage of the transportation of small load carriers with autonomous UAVs both the cargo pick-up and the drop-off need to be fully automated. Therefore, we want to achieve the following transportation scenario: The UAV receives a transportation task including an approximate position of the load carrier (e.g. the position of the stockyard) and the drop point. After the approach to the transmitted pick-up position, the UAV detects and locates the load carrier and positions itself precisely above the object to allow a reliable pick-up. After the pick-up using an automated load-handling device the cargo is transported to the transmitted destination. Subsequent to the aerial transport the load carrier is either precisely set down on a landing area defined by optical markers or simply at the transmitted drop-off coordinates.

As we focus on the indoor transportation we use an ultra-wide band (UWB) tracking system for global positioning. The system allows a tracking with 10 cm accuracy and can cover areas of several ten-thousand square meters. When flying in outdoor areas this localization system can be easily replaced with tracking data provided by satellite-based radio-navigation systems without modifying the remaining system components.

#### 2.1 Mechatronic cargo handling device

To ensure an automated pick-up and drop-off a mechatronic load handling device is developed and attached to a modified version of a commercially available hexarotor system. The load handling device is designed to be set down vertically on the load carrier. To compensate inaccuracies and fluctuations in the positioning of the UAV and to center the load carrier during the landing phase the system provides four lead-in chamfers (cf. Figure 4).

To fasten the load carrier after the landing, retaining bolts are used to create an interlocking connection with the indentations on the outer contour (1). As depicted in Figure 1, the individual bolts (2) are guided by linear bearings (3) and actuated by servo motors (4). A servo lever with a slotted hole (5) is used to convert the rotary motion into a linear one. As the servos are only actuated while the UAV is resting on the load carrier, no additional force or momentum other than the friction force needs to be overcome to move the bolts. During the flight phases, the weight force of the load carrier presses the bolts onto the bearing thus creating an efficient load restraint additional to the counter-momentum of the motor.

Fig. 1: Fixation of the load carrier by an interlocking connection between the undercut of the load carrier (1) and the retaining bolts (2). The bolts are guided by linear bearings (3) and actuated by a servo motor (4) and a servo lever (5)

#### 2.2 Object localization and tracking

Machine learning approaches are often considered as state of the art for object segmentation tasks. There are various neural networks available for object segmentation such as Mask R-CNN, DeepLab or FuseNet that allow an instance segmentation based on colour or combined colour and depth information [10– 12] as well as advanced pose estimation pipelines for segmenting and grasping various objects [13]. Light-weight models and neural network architectures also allow a near real-time image analysis low-cost hardware such as the NVidia Jetson Nano Board or different single board computers equipped with the Intel Neural Compute Stick.

However, as machine learning approaches still require large amounts of suitable and annotated training data especially when the load carriers is filled with different objects, we favour a solution based on color and shape segmentation.

Our approach for the detection, segmentation and localization of the individual load carriers is based on their known colour (blue) and shape (rectangular with known size). The detailed algorithm to segment and locate the objects is presented in detail below.

To extract the load carrier from the image, in a first step a basic colour segmentation algorithm is used. Based on a tolerance range centred on the actual colour value of the load carrier all blue pixels are segmented and converted into a binary mask. As depicted in the left image in Figure 2 the resulting mask contains most of the load carrier's pixels but is still affected by noise and falsely identified areas.

Afterwards Gaussian smoothing is applied to the binary image to reduce the effect of the remaining noise that can especially occur at the edges of the segmented objects. The Gaussian filtering prevents the incorrect detection of the noise as the edge of an object during the subsequent edge detection using the Canny-algorithm (cf. Figure 2, centre).

By applying topological structural analysis the contours of the remaining objects can be extracted. As only rectangular shapes are required, we apply the Douglas-Peucker-algorithm to approximate the remaining contours with the minimal required number of points. As an rectangle is fully described by its four corner points all contours with more or less points are discarded. To extract the rectangles actually representing a load carrier their enclosed area is analysed. As the intrinsic camera parameters, the size of the load carrier and the current flight altitude is known, the expected size of the load carrier in the camera image can be estimated and all rectangles that do not fulfill this criterion can be discarded.

Next, the scalar product of the vectors defining the rectangle's borders is calculated to get an estimate of the angle between them. This is used to only store rectangular shaped contours with corner angles of approximately 90 degrees. Based on this information, it can be ensured that e.g. for trapezoidal shapes, which also consist of only four corner points, the related contours are no longer stored for further processing.

Finally, the contours are sorted by their enclosed area and only the contour with the biggest area is kept. This is to avoid calculating the position of smaller rectangles within the box resulting from the possible influence of shading or overexposure (cf. Figure 2, right). After the segmentation the transform between the camera and the computed centroid of the load carrier is determined using the camera's inverted intrinsic matrix.

Fig. 2: Binary mask resulting from the color segmentation process (left), output image of the contour detection (center), detected load carrier with bounding box and centroid (right)

#### 2.3 Pose estimation and software architecture

As the optical tracking system features a significantly lower measurement noise than the UWB tracking system, the provided transform is not only used as setpoint but also used to increase the in-flight positioning accuracy of the UAV. This is necessary as the positioning accuracy that can be achieved by the data of the UWB tracking system and the UAV's intertial measurement unit is oftentimes not sufficient to pick up the load carriers. To improve the estimation of the current flight position we apply the covariance intersection algorithm presented by Niehsen [14].

Thereby the relative pose between the UAV and the load carrier is transformed into global coordinates and fused with the pose provided by the UWB tracking system. The inverse transformation is based on the UAVs global position at the time of the initial localization of the load carrier.

Using the covariance matrix Kxx of the global UWB tracking data and the covariance matrix Kyy of the pose provided by the optical tracking a combined covariance matrix Kzz can be calculated as

$$\mathbf{K\_{xz}} = \left(w \ast \mathbf{K\_{xx}}^{-1} + (1 - w) \ast \mathbf{K\_{yy}}^{-1}\right)^{-1}.\tag{1}$$

To find the optimal solution for Kzz the value of w is computed in such a way that the determinant of Kzz is minimized. Using the available closed-loop formulations a fused position estimation

$$P\_F = \mathbf{K\_{xz}} \ast (w \ast \mathbf{K\_{xx}}^{-1} \ast P\_C + (1 - w) \ast \mathbf{K\_{yy}}^{-1} \ast P\_G) \tag{2}$$

is calculated and forwarded to the internal kalman filter of the flight control unit (FCU). Thereby the position values provided by the UWB tracking system and the box detection are denoted as P<sup>G</sup> and P<sup>C</sup> .

The implemented process for cargo pick-up, transportation and drop-off is depicted in Figure 3. Initially the overlying task management sends an approximate position setpoint to the FCU, indicating where the cargo object is located

Fig. 3: Flowchart of the implemented software system used for object segmentation, pose estimation and controlling of the load-handling device

(not shown in the chart). As soon as the UAV reaches the provided position the process for locating and grasping the load carrier becomes active.

Based on the provided RGB-data the load carrier is segmented and localized using the algorithm presented in section 2.2. The resulting transform and the known transforms between the camera and the FCU and the load handling device and the FCU are then used to calculate an updated position setpoint that is forwarded to the FCU.

Using the covariance intersection algorithm an improved position estimation is calculated whenever valid relative tracking data is available and forwarded to the internal kalman filter of the FCU to improve the in-flight positioning accuracy.

The internal land detector of the FCU provides information whether the UAV is already landed or still in-flight. Based on this feedback, the load handling device controller drives the individual servo motors to fasten or release the cargo.

After a successful pick-up (which is determined by the increment in the required take-off thrust) the UAV heads for the provided drop-off position. When the drop-off position is referenced by additional optical markers, they are tracked using the algorithms presented in [15] and the resulting relative position is used as input to the covariance intersection algorithm, otherwise the positioning during the landing phase only relies on the UWB tracking data.

#### 3 Evaluation

The resulting system is depicted in Figure 4. The developed load-handling device is attached to a DJI F550 airframe with the corresponding electronic speed controllers and motors. As FCU we use the open-hardware system Pixracer running the 1.9.2 stable release of the PX4 autopilot [16] and a NVidia Jetson Nano on-board companion computer (OBC) for object detection and data fusion.

The implemented software utilizes the Robot Operating System (ROS) and the provided ROS communication layer. To allow a precise tracking of both

Fig. 4: Hexarotor system equipped with the developed load handling device

the load carrier and the landing area two down-facing cameras are attached to the system. The first is located in the center of the mounting plate of the load handling device and the second one is mounted to the outer frame of the UAV. The individual servo motors are connected to a suitable driver and controlled by the OBC.

While the FCU runs the real-time modules such as attitude and position controller the OBC runs additional software modules required for general flight processes and navigation. Core element of our previously developed software framework is the mission controller, a software module that is responsible for basic navigation tasks, path-following, system diagnosis and failure detection.

Figure 5 compares the achieved in-flight accuracy based on the pose estimation of the internal kalman filter when hovering around 1 m above the load carrier. To simplify the evaluation and depiction the corresponding static setpoint is subtracted from the position values. All flights are conducted with the same closed-loop controller but it's effect on the positioning accuracy and potential improvements will not be discussed in detail. When only the UWB tracking information is used, the resulting in-flight position deviates around ±5 cm from the setpoint in x- and y-direction. However, the deviations in z-direction reach values of up to 20 cm. This can be traced back to the fact that the noise in the measurement values of the UWB tracking system in z-direction is significantly higher then the noise in x- and y-direction.

When fusing additional optical tracking information, the deviations decrease to maximal values of ±4 cm in every direction. Compared to the UWB-only

Fig. 5: Comparison of the achievable positioning accuracy when only relying on UWB tracking information and when fusing UWB and optical tracking data using the covariance intersection algorithm

dataset, especially the accuracy in z-direction improves significantly enabling a consistent and precise descending. Compared to the z-direction, the positioning accuracy in x- and y-direction increases less which can be traced back to inferior measurement inaccuracies of the UWB tracking system.

While the optical tracking of the load carrier improves the flight accuracy and improves the success rate of the cargo pick-up further improvements are required. The lead-in chamfers of the current load handling device can compensate movement of up to ±5 cm, however, the automated pick-up currently only succeeds approximately every second time (evaluated on the basis of 20 test flights). Besides the in-flight position oscillations also faulty tracking information, friction on the lead-in chamfers and oscillations caused by the ground effect are accountable for the poor success rate.

#### 4 Conclusion and Outlook

Within this paper we have presented an approach to the autonomous aerial grasping of small load carriers with UAVs. Using a camera-based solution for segmenting and locating the cargo objects, a newly developed load-handling device and the covariance intersection algorithm for optimized pose estimation, the automated pick-up of small load carriers was realized. While it was shown that the autonomous pick-up of small load carriers is possible with the present system, the reliability and rate of successful pick-ups still needs to be improved.

To optimize the flight and grasping performance within future research, the influence of optimized and model-based controllers will be examined. Also improved cameras with optimized field of views and additional sensors to measure the current movement speed will be integrated and the influence of vibrations caused by the spinning rotors will be examined.

### Acknowledgment

The project receives funding from the EU ECSEL Joint Undertaking under grant agreement n737459 (project Productive4.0) and the German federal ministry of education and research under grant agreement 16SE0198.

#### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Determination of Gripping Forces for Handling Surfacesensitive Objects**

Michael Krampe[0000-0003-3996-0255] , Michael Miro[0000-0003-2347-7496] , Miguel Angel Villanueva Portela[0000-0002-3769-9068] and Bernd Kuhlenkötter[0000-0002-5015-7490]

> Lehrstuhl für Produktionssysteme, Ruhr-Universität Bochum, Universitätsstr. 150, 44801 Bochum, Germany krampe@lps.rub.de

**Abstract.** Even for modern robot attached gripping solutions, surface-sensitive object handling remains a challenge. A minimal gripping force is required to grip and handle a surface-sensitive object safely. At the same time, this gripping force should not exceed a certain level to avoid damaging the sensitive surface of the object. Because of this, it is necessary to determine the mostly small range of optimal gripping forces to ensure a safe handling. Therefore, this paper describes an approach to evaluate this range of gripping forces for surface-sensitive objects using by means of chocolate covered marshmallows. For this purpose, a test bench with the same characteristics as a planned gripper is developed. Throughout the project, this planned gripper will be controlled with the help of the detected range of gripping forces. In addition to these static measurements, it is planned, that the gripper will calculate the handling speeds, accomplished by a six-axis industrial robot, with its acceleration forces. The achieved results of this paper will be the starting point for the upcoming measurements with the robot. Furthermore, the influence of the temperature on the range of gripping forces was evaluated, to show the relevance of consistent environmental conditions for the handling of surface-sensitive objects.

**Keywords:** Gripper, Handling, Force control.

#### **1 Introduction**

In areas like the food industry, robots are often used for heavy load handling, e.g. packaging and palletising. In contrast, the handling of sensitive objects is still a major challenge for industrial robots. Recent articles [1, 2] illustrate the need of a sophisticated control strategy for sensitive object handling in various industries. Furthermore, the acquired sensor data is often used for the handling process alone and not broadcasted towards a MES or CPPS for further process analysis and decision making [3]. Available gripping solutions usually focus on the process of gripping and grasping without taking translational movements during transit into account. Especially sensitive objects are prone to be damaged by dynamic and abrupt changes of acceleration. This can be explained by the corresponding inertia that adds up to the applied static gripping force. As soon as the sum of dynamic and static forces exceeds the object's limits, it is damaged and may no longer be suitable for its designed purpose. In order to provide a suitable solution, the project *Sensitive Robot Based Gripping* (SenRobGrip) aims to develop a cyberphysical gripping system (CPS), which is capable of handling sensitive objects whilst improving the cycle time. To handle surface-sensitive objects, it is required to know the range of optimal gripping forces to handle it safely without damaging it. In some cases, this range can be so small, that even little changes of the environment can affect and lead to a shift of this range. The gripper should include this range of gripping force and the acceleration of the moving robot to optimise the pick-andplace task while transit. The base design of the gripper and the motion planning approach for the robot movements is described by the authors in former publications within the SenRobGrip project [4, 5]. Subsequently, this paper determinates the range of optimal gripping forces using the example of chocolate covered marshmallows and the influence of temperature. These forces are necessary to adjust a later process control with the real gripper. For that, a test bench is developed which covers the same gripping characteristics as the planned cyberphysical gripping system [6].

#### **2 Use Case**

The food industry provides many different practical examples for the demand of sensitive object handling. The range of products is varying in consistency, geometry and is often covered by pressure-sensitive surfaces, which break or show signs of maltreatment if not handled correctly. This maltreatment is directly linked to customer satisfaction and may lead to decreasing sales. In order to automate pick-and-place processes for products like these, it is necessary to consider static and dynamic forces within the handling processes. To provide a realistic use case, the developed gripper is designed to be able to handle chocolate covered marshmallows at high velocities and accelerations without damaging the surface. The proposed scenario makes it necessary to develop a handling device for these size variable and extremely fragile products in order to provide a solution for a broader product spectrum. Furthermore, environmental circumstances like temperature and humidity have a distinct influence on the product and therefore have to be monitored throughout the production.

## **3 Gripper Hardware**

The first step is the development of a new and versatile gripper as shown in Figure 1. The design aims to provide high stiffness for an accurate gripping force measurement as well as for being flexible enough to handle various object geometries. The gripper consists of a hollow cylinder to which matching flange adapters for different robot models can be attached. For communication and control purposes, a Raspberry and Arduino are placed inside the cylinder and mounted to a 3D-printed inlay. The finger drivetrains connect to the bottom of the cylinder. To improve stiffness and reduce friction, the fingers are attached to linear motion guides and are propelled by a belt drive.

**Fig. 1.** Gripper hardware design.

Four 24 V DC motors actuate the motion and generate a finger movement speed of up to 0.4 m/s which allows the gripper to close in less than 0.15 seconds.<sup>1</sup> The device is equipped with four exchangeable piezo-resistive force sensors – one in each finger – which provide precise measurements on a scale of 0 to 10 N and an accuracy of up to 0.01 N at 860 samples per second.<sup>2</sup> In addition to this, a 6 degree of freedom (DoF) inertial measurement unit (IMU) MPU-6050 provides the needed information about the robot's movements. Thus, the accuracy of up to 0.598 mm/s<sup>2</sup> for translational motions and 0.0153°/s for rotational movements at 1 kHz allows robot-autarkic control. This enables the manufacturer-independent usage, as no proprietary interface is needed. To monitor environmental influences, a dedicated sensor gathers data on environmental influences like temperature and humidity in order to provide a feedback to the production management system to maintain stable process parameters.

1 Faulhaber 2250S024BX4 CSD with 1:14 227 gearhead

2 Honeywell FSG010WNPB with HX711 analog-digital-converters

## **4 Gripping Force Test Bench**

The aim of the gripping force test bench is to simplify the testing using chocolate covered marshmallows and to define the object specific range of gripping force in this example. The general construction is shown in Figure 2. The chocolate covered marshmallow lies on a small podium in the centre of the device and is surrounded by four single grippers. Each of these grippers have its own carriage combined with a rail for the linear movement powered by a servo drive with the help of a belt drive. The gripper jaw is a 30° segment of a circle with a radius of 30 mm and a height of 10 mm made of Polylactide. The podium in the center can be lifted up and down by a seesaw to see if a chocolate covered marshmallow is held safely by the grippers immediately.

**Fig. 2:** CAD model of the gripping force test bench

The sensors for force measurement are built into these gripper jaws. To read out the values of the sensors and implement a controller, an analog digital converter is used and connected to an Arduino Due microcontroller by an I2C Bus System.

## **5 Experimental Design**

The main objective of this experiment is to evaluate the range of the gripping force to handle chocolate covered marshmallows without damaging them. This range is limited by the minimal force which is needed to hold the object and the maximum force to hold the object without damaging it. First, the minimal force can be evaluated with the podium and the seesaw in the centre of the test bench. After the gripper is closed around the object, the podium can be lowered to see if the chocolate covered marshmallow stays in position. In a second step, the maximum forces can be evaluated by searching for damages of the sensible chocolate surface. Within the following experiments, the surface damages are divided into five stages. Stage zero represents a not damaged surface and the other four stages describe the level of damaging from a visible light up pressure spot (stage 1), over one crack (stage 2) and many cracks (stage 3) to a complete break of the chocolate (stage 4). These four stages of damage are shown in Figure 3.

**Fig. 3:** The four stages of damage

Before explaining the evaluation and showing the first test rounds a technical problem needs to be explained. We identified a random effect of shaking between two opposite grippers. After some adjustments of the controlling and the Arduino software it was possible to nearly eliminate the shaking after some test runs. Afterwards it occured for a few cases only, but it mostly effects the minimum gripper force which is needed to handle the chocolate covered marshmallow. These cases will be marked in the upcoming results. The shaking results from the combination of fast high-end sensors with very high resolution for the measurements and simple motors providing limited step sizes to change the gripper jaws positions. The final gripper with the more expensive motors was already in production so we were not able to use these motors for the gripping force test bench. This leads to problems in the force controlling which is shown by the shaking sometimes. For the final gripper we will not expect these problems thanks to the high-quality motors that will be implemented.

To evaluate the minimum gripping force the gripper is adjusted to test in specific ranges.

The forces are measured when the gripping process is finished, which means that the motors stopped moving and the object is gripped without the podium underneath. After some iterations the range for the minimum gripping force at room temperature (RT) is detected between 0.15 – 0.35 N based on each single gripper. In order to later include the movements of the robot into the calculation, each gripper finger is individually controlled. For the following 30 tests, a three-point controller with the determined range was tested without losing grip at any test. The results are shown in Figure 4. The four values of the individual grippers are combined in the graph to form an overall gripping force for simplification. The developed lower limit for the range of gripping force lies at around 0.88 N.

**Fig. 4:** Measurements for the minimal gripping force at room temperature

To define the upper limit for the range of gripping force every tested chocolate covered marshmallow is gripped twice to ensure that no damage is missed out in the first attempt. After some iterations the range for the maximum gripping force at room temperature is detected to range between 0.8 – 1.2 N based on each single gripper. Again, random force values within this previously determined range were tested. 26 chocolate covered marshmallows are tested twice, which results in 52 measurements. The results are shown in Figure 5.

Two objects are damaged after the first grasp and four objects are damaged after the second grasp. This leads to eight negative measurements compared to 44 positive ones. The measurements marked with "\* Stage 2" are from a marshmallow with a very rough and bumpy chocolate cover, which results in higher pressure peaks. This shows the importance of a good production quality for future handling tasks.

The objects 20 and 25 were damaged at the second grasp. The single gripper forces from the four measurements are shown in Table 1. For object 20 the crack is between gripper 2 and 3. The cause of this crack could be the rise of the gripping force from 1.03 to 1.08 at gripper 2 or 0.88 N to 0.97 N at gripper 3. For object 25 the crack is between gripper 1 and 4. In this case the rise from 1.11 N to 1.15 N at gripper 1 or 0.99 to 1.09 at gripper 4 could be the reason for the crack. This shows the small range of tenths and hundredth of Newton gripping force that can lead to a crack in the chocolate. Because of these small fluctuations the upper limit for the overall range of gripping force is set to 3.5 N as shown in Figure 5.


**Tab. 1:** Detailed look on two tested objects

Furthermore, the effect of temperature on the range of gripping forces is evaluated. The chocolate covered marshmallows are cooled down to 8°C and the procedure for finding the range of gripping forces is the same. As shown in Figure 6, the lower limit for the range of gripping forces at 8°C is higher compared to the equivalent limit at room temperature. While testing the maximum gripping forces for the upper limit of the range, the previously mentioned shaking of the test bench is a problem again. This shaking results in some gripping force peaks in Figure 7. Furthermore, some chocolate covered marshmallows got a crack from the temperature differences. Overall, the results shown in Figure 7 lead to an upper limit at around 5 N at 8°C. The different results between the measurements at 8°C and room temperature were expected due to the adhesive and melting characteristic of chocolate.

**Fig. 6:** Measurements for the minimal gripping force at 8°C

Linear (upper limit)

**Fig. 7:** Measurements for the maximum gripping force at 8°C

## **6 Results**

First of all, the results show that it is possible to determinate the range of gripping forces for surface-sensitive objects like chocolate covered marshmallows. Also, the problems with the shaking grippers will not be expected when using the higher quality motors in the final gripper. Even with the applied cheaper motors for these experiments have shown that the required sensitivity is possible. The evaluated range of gripping force for the example of chocolate covered marshmallows is found between the limits of 0,88 N to 3,5 N at room temperature and 1,2 N to 5 N at 8°C. These results show the limits for the controlling of the surface-sensible gripper developed in this project. The force sensitive object handling needs to determine and actuate the controlling in these defined limits to allow a safe handling of surface-sensitive objects. It should be emphasised that these tests are made on a specific test bench without the acceleration forces of the robot. This will reduce the range of motion and emphasize the importance of controlling the individual fingers. We are currently trying to transfer these results to the real gripper (shown in Figure 8).

**Fig. 8:** The real gripper

Furthermore, the results show the importance of the temperature controlling of the product environment. The handling of surface-sensitive objects in winter or summer could make a difference in product quality if the season affects the ambient temperature, yet this paper shows that with some information about the effects of the temperature – and by integrating the measurement into the cyberphysical gripper control - it is even possible to enable a safe handling.

#### **Acknowledgements**

The research project *Sensitive Robot based Gripping* (SenRobGrip) is funded with kind support of the European Union's EFRE program *Investments in growth and employment*. Within the time span of March 2017 to February 2020, IBG Automation GmbH, IBG Robotronic GmbH, and the Chair of Production Systems of Ruhr-Universität Bochum are participating in the SenRobGrip project.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Disturbance Force Estimation for a Low Pressure Suction Gripper Based on Differential Pressure Analysis.**

Benjamin Wirth1 , Sven Coutandin1 and Jürgen Fleischer1

1 Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany benjamin.wirth@kit.edu sven.coutandin@kit.edu juergen.fleischer@kit.edu

**Abstract.** In times of ever increasing performance requirements imposed on newly developed products, fiber reinforced plastics have become essential for lightweight applications. While mass related stiffness and strength are huge advantages, automatic handling of the raw materials, such as fabrics, can be very challenging. Low-pressure suction grippers have been proven to be capable of handling fragile air-permeable materials in a non-destructive manner, giving them a clear advantage over potentially damaging competing gripping technologies such as needle grippers. However, they also consume large amounts of energy to ensure successful handling operations. This research aims to estimate disturbance forces applied to the manipulated workpiece by monitoring its internal suction pressure, which will in turn allow for running the grippers at a bare minimum power while still being able to react to disturbances enabling safe handling operations.

**Keywords:** Process Monitoring, Gripping Technology, Textile Handling.

## **1 Introduction**

While weight related stiffness and strength of continuous-fiber reinforced plastics (FRP) are superior to most classic metallic materials [1]. There is a lack of automation solutions in the handling of "*limp and air-permeable semi-finished textile products*" [2]. Low pressure suction grippers have been identified as a technology with great potential for automated handling of semi-finished textile materials [3, 4].

They can maintain a moderate suction pressure at high volumetric flow rates [5], which allows for sufficient holding forces even when air is able to pass into the suction cavity of the gripper either by leaking airstreams passing the material or by penetrating the porous material directly.

For automated handling of air-permeable sheet materials, low pressure suction grippers have been used in a variety of research applications [6–9].

However, due to a lack of compensation mechanisms for '*disturbances during the handling process'* they are often run at maximum suction power to ensure reliable material transportation even when exterior forces may interact with the handled textiles. This mode of operation is very energy-inefficient and thus not desirable. [10]

This work focuses on estimation of disturbing forces applied during handling of airpermeable blanks by monitoring the interior pressure inside the grippers' central cavity.

## **2 Low Pressure Suction Grippers**

Grippers using partial vacuums at moderate volumetric flow rates have been identified as one of the most promising technologies for automated textile handling [11].

Fig. 1 shows a sketch for a simple suction gripper.

**Fig. 1.** Sketch of a simple suction gripper and Forces acting on a gripped Fabric. Based on [10]

These systems usually feature central cavities, which are partially evacuated to a low pressure throughout this paper the differential pressure between suction cavity and surrounding atmosphere will be depicted by.

A suction plate, which may contain small holes preventing major material deformations and restricting airflow in non-covered areas, is usually attached to the gripper. This avoids "short circuits", that would lead to a rapid decline in suction pressure] 12[.

In a one dimensional lifting operation the handled materials are generally subjected to four different forces [10]:


$$F\_H = A\_{eff} \cdot \Delta p \tag{l}$$


The contact force with which any gripped material is pressed against the suction plate can be described by [10]:

$$F\_{CP} = F\_H + F\_G + F\_N \tag{2}$$

### **3 Evaluation of the State of the Art**

#### **3.1 Pressure Monitoring for Suction Grippers**

Götz measured differential pressures inside a low pressure handling system for leather blanks during operation for validation purposes, however, he did not relate them to significant process properties. [9]

The gauge pressure inside a suction Gripper has been used for gripper control by Ehinger. The interior pressure was controlled by a PI-controller to match predefined values found by experimental analysis. Allowing for secure draping of a textile material even with some non-covered holes on the suction plate. [6] While controlling the interior pressure will lead to a response on disturbing forces acting on the handled material, these forces are not quantified. For that an additional observation of the supplied suction energy would be necessary.

Fleischer et al. have shown a dependence of the internal negative pressure inside a coanda-effect based grippers' suction cavity on the number of manipulated layers for air-permeable materials. In experiments placing up to 2 layers of textile materials beneath a suction plate and gradually decreasing the input power-stream to the gripper it was found, that for any given input power the interior pressure approaches a higher vacuum with additional layers adhering to the manipulator. This allows the system to directly estimate the number of sheets separated from a stack-magazine. [8]

#### **3.2 Gripping Force Estimation for Low Pressure Suction Grippers**

Fleischer et al were able to measure holding forces between a suction gripper and carbon fiber materials by attaching brass electrodes to the suction plate. The measured electrical resistance between the electrodes has been shown to directly depend on brasscarbon contact resistances, which in turn decrease with higher holding forces applied to the handled textile. By designing control loops for energy efficient operation, they were able to create a dynamic system capable of quickly reacting to disturbances, while also reducing energy consumption by nearly 90% compared to a conventional static energy supply. Furthermore, they were able to control the holding force, such that a reliable separation of singular sheets off of a stack-magazine was possible. [8]

This approach, however, relies on the electrical conductivity of the handled textile and is therefore not applicable for a wide variety of common materials.

Brink et al. developed a "*Force Controlled Variable Fan Gripping Module*" incorporating an electrically powered fan as a vacuum generator. Strain gauges were integrated into the fan grill, which serves as a suction plate in this system. With increasing handling forces, increasing deformations in the grill can be observed, which were in turn registered as a change in resistance for the strain gauges. Using this principle, Brink et al. were able to monitor process forces occurring during the handling of non-crimp fabrics. They furthermore noted, that a dependence of the gripping force on the number of attached layers is observable. [7]

Retrofitting sensor integrated suction plates introduces undesired complex alterations to commercial gripping-systems. A less invasive approach for measuring disturbances acting on the gripper is required. Monitoring the pressure inside the suction cavity promises to be an efficient solution to this problem. This research shows an approach for generation of a process model relating negative pressure to disturbance forces based on an offline calibration. This system can be used for online monitoring of handling operations. It may also be used as part of a state estimation algorithm allowing the gripper system to control contact forces at the gripper-material interface.

## **4 System Description**

The SCG 1xE100 gripper used in this research (see Fig. 2) is manufactured by J. Schmalz GmbH.

**Fig. 2.** Left: SCG 1xE100 gripper with suction plate Right: cross sectional sketch of a coanda-effect based gripper based on [8]

The gripper is activated by supplying air at a positive operating pressure ே to port 1 this high pressure air fills a primary cavity, which it can escape through a thin slit leading to a central secondary cavity. The inwards moving air will tend to adhere to the slightly curved top surface. This primary flow ܳே facing upwards will have a suction effect on the air at port 2 creating a negative pressure and a secondary airflowܳ. During operation an additional suction plate is screwed onto the gripper at this interface. The combined airflow will leave the suction cavity by port 3. Another connection from the outside directly to the suction cavity is provided by port 1/2, which is usually used for a blow off functionality [5].

Initial tests probing the internal suction pressure at different positions by laterally inserting hypodermic needles into the suction cavity have shown, that even small translational variations of the probing-position can have small (varying by about ͲǤͲͷ݇ܲܽat suction pressures of ൌ ͲǤͳ݇ܲܽ) but significant effects on the resulting pressure readings. These differences are presumably caused by the non-homogenous suction plate geometry and turbulent flow regimes directly behind the it. Therefore, an optimal probing position should be situated as far away from the plate as possible. For transferability of measurements amongst different SCG- 1xE100 grippers the blow-off port 1/2 has been chosen as an ideal probing position.

#### **5 Procedure**

An adapting gripping system capable of monitoring and reacting to changing disturbance forces may be realized by using the controller architecture shown in Fig. 8.

In such a system a desired contact force ܨǡௗ௦ can be set and controlled, which will directly implement dynamical reactions to any disturbance forces acting on the handled material (see equation (2)). A PID-controller can be used to dynamically change the operating pressure of a suction gripper depending on an error measurement based on desired and estimated contact forces between the gripper and the handled textiles.

**Fig. 3.** Proposed control loop for the final gripper system. Parts analyzed in this research are highlighted

This research aims to show how a disturbance force estimation in the shape of a model ܨሺܳேǡ ሻ can be obtained by taking the following steps:

1. Recording of example pull-of-experiment data


#### **6 Experimental Setup**

In order to characterize the correlation of disturbance forces ܨே and the suction pressure an experiment as shown below is set up.

The experimental setup sketched in Fig. 3 consists of an SCG 1xE100 gripper that can be moved up and down by a distance ݏ with a lead screw driven by an electric stepper motor (not depicted in the figure). The forceܨ, at which the gripper is pulled towards the base is measured by a force sensor. The material specimen is clamped between two metal plates with a central rectangular cutout (100x100mm seen in Fig. 3.). The operating pressure ே is supplied by a proportioning valve, while the flow rate ܳேis measured by a flow meter and suction pressure being monitored by a differential pressure sensor.

**Fig. 4.** Sketch and photograph of the used test stand.

**Fig. 5.** Leakage occurring after deformation of the gripped fabric

Pull-off tests have been performed by supplying a specific operating pressure ே and moving the gripper upwards by an increasing distanceݏ, while recording ܳேǡ ܨ and. This movement leads to a gradual separation of the material from the suction plate allowing for additional air to bypass the manipulated sheet, avoiding the pressure drop across the textile, leaking into the suction cavity and thereby decreasing the measured value of. This leakage happens in the test stand as well as when gripping freely hanging sheets as depicted in Fig. 4.

## **7 Experimental Results**

#### **7.1 Model Construction**

This section discusses the aforementioned offline calibration for the carbon fiber material shown in Fig. 3.

Exemplary pull-off tests yield the results presented in Fig. 5, with pull-off forces ܨand differential pressures plotted against the traveled distance s (Fig. 5 a). and ܨ are plotted against one another from test start up until the maximum force ܨ௫ is reached (Fig. 5 b). The same plot is repeated on the right for measurements afterܨ௫ was reached (Fig. 5 c). The clusters of lines shown in Fig. 5 correlate to supply pressures of 0.5; 1; 1.5; 2; 2.5 and 3 Bar.

**Fig. 6.** Measured forces and negative pressures for an exemplary pull-off test

At the beginning of any given pull-off-test the pressure stays nearly constant, with no noticeable separation of the gripped material from the suction plate. Recorded forces are here mainly dominated by the material deformation. After 0.5-1.5 mm of travel slight separation can be observed with a decline in. The recorded forces keep rising until a maximum force ܨ௫ is reached close to an inflection point seen in the ൈ ݏplot between 1 to 4 mm of travel. At s = ~5 mm full separation can be observed with forces quickly approaching 0 N and settling at a non-zero value generated by the flow resistance of the installed suction plate.

Since the supply pressure at the gripper is set and not directly measured, the operating air flow measurementܳே, which behaves almost proportional to ே has proven to be more reliable in dynamic situations. As seen in Fig. 5 the negative pressure increases practically linearly with increasing operating pressure, while decreasing almost quadratically with increasing disturbance forces. Based on this observation a simple polynomial modeling approach ሺܨǡܳேሻ ൌ ଵܨଵܳே ଵଵܨܳே ଶܨ<sup>ଶ</sup> is chosen. A parameter optimization for the specific results shown in Fig. 5, gives a polynomial numerical model, relating pull off forces and operating pressure to the measured differential pressure (beforeܨ௫ was reached):

$$p\_{U,a}(F, Q\_N) = -1826Pa - 279.8 \, F \, \frac{Pa}{N} + \\\ 174.02 \, Q\_N \frac{Pa}{l \, ml \, n^{-1}} - 56.72 \, F^2 \frac{Pa}{N^2} - 3.168 \, F \, Q\_N \frac{Pa}{N \, l \, ml \, n^{-1}} \tag{3}$$

Here,ܳே is given in norm liters per minute.

The model represented by equation (3) reaches an adjusted ܴതଶ = 0.9930 over 224 sample points collected in 18 individual experiments.

Analogously, another model can be obtained for the part of the tests after the maximum force has been reached.

$$p\_{U,b}(F, Q\_N) = -301.3Pa - 161.5\ F\frac{Pa}{N} + \\\\ 1.24\ \,\mu\_N\frac{Pa}{l\,\,mol\,\,^{-1}} + 15.64\ \,F^2\frac{Pa}{N^2} - 0.8202\ F\,Q\_N\frac{Pa}{N\,\,l\,\,m\,\,^{-1}}\tag{4}$$

Using the same approach, similarly close approximations have been found for a variety of different materials. The two models described above can be combined using a linear separator at ൌ <sup>ଵ</sup> ଵ ொಿ షభ ڄ ͶͲͲͲܲܽ െ ͲͲܲܽ.

$$p\_U(F, Q\_N) = \begin{cases} p\_{U,a}\{F, Q\_N\} \ , \text{if } \frac{1}{160} \frac{Q\_N}{l \text{ min}^{-1}} \cdot 4000Pa - 600Pa \ < p\_U \\ p\_{U,b}\{F, Q\_N\} \ , \text{else} \end{cases} \tag{5}$$

For any combination of Ͳ and ܳே Ͳ a corresponding estimated disturbance force ܨ can be found by calculating the force, which minimizes the absolute difference between the predicted and measured negative pressure:

$$F\_{\mathbf{e}}(\mathbf{p}\_U, Q\_N) = \arg\min\_{\mathbf{F}}(|\mathbf{p}\_U\{\mathbf{F}, Q\_N\} - \mathbf{p}\_U|)\tag{6}$$

#### **7.2 Model Application**

Manually moving the gripper up and down on the test-stand allows for a comparison between measured forces and estimations made by the previously described models, solely dependent on and ܳே. Results of these simple validation tests at an operating pressure of ே ൌ ʹܾܽݎcan be seen in Fig. 6.

It is clearly visible, that significant disturbance forces and in turn separations are directly seen by the estimation. While both shown curves overlap for most parts of the diagram slight deviations can be seen, which may be explained by non-elastic behavior of the tested fabric.

**Fig. 7.** Model evaluation by manual movement of the gripper

In another test manually pulling the fabric down similar observations were made as shown in Fig. 7.

**Fig. 8.** Model evaluation by manual movement of the fabric

Measured and estimated forces closely match one another while forces are applied evenly around the gripper. Under this load case the model estimation shows an average error of about ͳܰ.

With punctual loads (as seen at ݐ ൌ ʹͲݏ in Fig. 7) forces will not be estimated as precisely. Here, the estimated force is off by almost 100%. Nevertheless, it can be concluded, that the estimation model qualitatively shows good correlation with the measured data.

#### **8 Conclusion and Outlook**

This work has introduced a differential pressure-based estimation method for disturbance forces acting on fabric being manipulated by a low-pressure suction gripper. After obtaining experimental data, polynomial models can be formulated which serve as a look-up point for the desired force estimation. It has been shown, that such estimation model can be used to deduce qualitative information about forces acting on the gripped material. While these estimates do not show perfect resemblance of the forces, especially for different load cases, the two show a strong correlation.

In future research these results will be generalized to allow for model estimation based on material properties and gripper characteristics. Which will then be used to define a control structure capable of adjusting the systems. As shown by the works of Förster such a control structure can furthermore be used for robust separation of singular sheets of air-permeable material from a stack magazine [3], paving the way for highly automated production lines in both classical and technical textile industries.

## **9 Acknowledgement**

This research was financially supported by the German Research Foundation (DFG) (funding no. 397485737)

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Integration of Collaborative Robotics in Vocational Training Using Mixed Reality**

Torsten Sebastian Sievers<sup>1</sup> , Bianca Schmitt<sup>2</sup> , Maren Petersen<sup>2</sup> and Kirsten Tracht1[0000-0001-9740-3962]

<sup>1</sup> University of Bremen, Bremen Institute for Mechanical Engineering (bime), 28334 Bremen, Germany <sup>2</sup> University of Bremen, Institute Technology and Education (ITB), 28334 Bremen, Germany sievers@bime.de

**Abstract.** The change in production towards smaller batch sizes demands a high degree of flexibility, especially in assembly. In this context, human-robot collaboration (HRC) is becoming increasingly important. The goal is not to displace the human being by the robot, but instead receiving support from it. Nevertheless, employees at all levels have to be prepared for the HRC. Trainings offered from robot manufacturers focus on robot specific operations and are not sufficient for the development of competencies for innovative and creative process planning and optimisation. Therefore, a mixed reality learning environment can be used for the implementation of such training, since real production facilities are only available to a limited extent for training purposes. In the context of the implementation of a mixed reality learning environment, the question arises as to the necessity of haptic feedback. This article presents a concept for integrating HRC in German initial and continuing vocational education and training using mixed reality.

**Keywords:** Human-Robot Collaboration, Learning Factory, Assembly.

## **1 Introduction**

The importance of flexible assembly is growing in particular due to the increased production of small series through customer-specific productions [1]. The stepwise automation of the production process enables a cost-effective production of customer-specific products. In particular, the design of human-robot collaboration (HRC) plays a central role with regard to future assembly processes, whereby the form of cooperation between human and machine is decisive. The implementation of HRC enables a high overall productivity and a better product quality. [2] However, the safety of the employees showed as a difficult problem [5]. Therefore, the planning of collaborative processes can be supported and validated in a virtual environment [6]. The virtual environment is based on a digital twin using data from new product variants and resource configuration. It allows a simulation of the collaborative processes including the human. The results show, that in order to ensure a safe use of collaborative robots and increase their potential, qualified employees are required. [7] Therefore, employees at all job levels must be qualified, because it is unlikely that HRC will result in production without employees and it does not represent a desired goal. Rather, the human being remains the most important control and decision-making authority. [3] The aim should be to place the human being at the centre of the process in the sense of designing the work processes [4].

In most cases, the system integrator trains the employees in the use of a specific robot. This training is limited to the teaching of programming knowledge and productspecific basic qualifications. Initial research, as part of the KoRA third-party funding project indicates that these training courses do not meet the expectations from employees in the company. In this context virtual environments offer a safe and reconfigurable learning environment. GAMMIERI ET AL. already published a virtual environment for HRC enhancement [8]. They integrated a forward and inverse kinematic of a KUKA collaborative robot and detected collisions. Furthermore, the digital model can be connected with a real robot. Due to the integration of several collaborative assembly scenarios, the virtual environment can be used for safety considerations as well as for training operators. Another approach form MATSAS AND VOSNIAKOS introduced a virtual reality training system for HRC [9]. The System visualises safety issues like contacts or collisions and can be used to investigate acceptability of HRC. Moreover, OYEKAN ET AL. investigate the reactions from human to movements of the robot [10]. The goal is to test collaborative processes in advance to ensure a safe process. However, none of these approaches provided neither a differentiated learning environment based on the individual qualification level, nor an appropriate teaching concept.

#### **2 Mixed Reality Learning Environment - KoRA**

In September 2019, the University of Bremen started a research project named KoRA with the aim to prepare employees at all levels for HRC. The teaching and learning concept enables employees to view complex assembly systems holistically in order to recognise the potential of HRC and use it in a targeted manner. The approach uses a learning software made in Unity in combination with learn and work assignments (see section 3) that rely on the visualisation techniques of Mixed Reality. Within the assignments, the dissolving of rigid borders between manual and automatic assembly is focused by the use of collaborative assembly.

The possibility of participation of the employees offers the potential for dismantling obstacles to collaboration with robots [9]. This is of particular important because the employees conviction about the advantages of HRC is a crucial component for the successful introduction into the factory [11]. The complex assembly scenarios are prepared within the framework of learning and work assignments. These take into account operational reality and real work processes and thus enable project-based learning in realistic scenarios. In particular, scenarios on the topics safety, collaboration suitability and technical feasibility are intended (see Fig. 1). The concept addresses trainees, skilled employees and supervisors, as well as trainers and teachers in relation to the German dual vocational training.

**Fig. 1.** Concept of the KoRA mixed reality training

#### **3 Didactical concept**

The didactical concept is based on learn and work assignments (see Fig. 2). They are particularly suitable for dual vocational training due to their occupational world relevance, the link to problematic situations and the project-based learning organisation [12]. Especially, learners benefit from the problems they face in their daily work because the relevance, i. e. the present and future significance (according to KLAFKI), becomes apparent to them [13].

The learn and work assignments are also characterised by their project form, which promotes the independent development of a problem solution, both as an individual and as a team effort. The acquired knowledge is applied in the development of solutions, the planning of the work steps as well as the implementation and adaption to the individual starting position.

A further advantage of learn and work assignments is the option of flexible design. Heterogeneous target groups identified to be taken into account and to be trained in a targeted manner. The learn and work assignments can be differentiated for the different modes of the learning application and the corresponding learning objectives as well as competence focal points. However, the focus of the gradation is not on the qualification level, but rather on the necessary characteristics of the competence profile for the employees in the collaborative robot environment in the individual company context. The different focal points are magnified in the KoRA learning software in different modes, which have to be selected by the user at the beginning.

**Fig. 2.** Principle of learn and work assignments (according to HOWE [12])

Besides the employee, the trainer, in school, company and inter-company context, represent another important target group for the concept. They are addressed in trainthe-trainer workshops and understood as a central element of the didactic formulation of objectives. In addition to the didactic concept, the trainers are equipped with prepared content and methods for their own teaching and learning context. This lowers the inhibition threshold for the use of the learning software and supports a use beyond the duration of the project.

## **4 EMMA - Experimental modular assembly plant**

The experimental modular assembly plant (EMMA) of the bime Institute consists of six autonomous sub-modules, which can be arranged flexibly (see Fig. 3). Three modules are designed for manual assembly processes and two are fully automated. A collaborative module is used to supports another manual assembly station. All assembly stations are connected by a belt conveyor for transportation. The manual stations also have a large touch screen for interactive communication with the employee. This allows, for example, a support for the assembly process to be displayed. As an example product, a ball labyrinth is produced on the plant. This ball labyrinth has been designed in such a way, that it can be dismantled non-destructively. The assembly of the product requires the use of various joining processes, from clips to screws to the knotting of a rope. Together with a variety of possible variations of the product, this results in variable assembly scenarios. The plant as a whole is available as a virtual image and supports a cyber-physical linking between real system and VR environment.

**Fig. 3.** The experimental modular assembly plant (EMMA)

The collaborative module can be used for several assembly tasks during the production of the ball labyrinth. One possibility is the support during the rack assembly. Within this task, the robot positions the profiles in such a way that the employee can carry out the required screw connections in an optimised manner. The robot is programmed offline in advance on the basis of CAD data. In practical operation, however, collisions can occur between robots and employees. This can be caused, for example by individual use of the workstation or by the size of the employee. In this case, a hand-guided teachin of the robot by the employee is necessary in order to adapt the movements of the robot to the respective situation. With the intention of avoiding downtimes in production, the adjustment process must be carried out efficiently by the employee. This scenario can be used as an initial scenario within the virtual learning environment for first investigations about haptic feedback.

## **5 Haptic Feedback within the learning environment**

For a Mixed Reality learning software it is necessary to enable a realistic representation of the virtual environment. Furthermore, the applicability of the hardware solution at the different learning locations is a requirement for the application and its technical implementation. Therefore, the current technological developments and the resulting possibilities and limits have to be considered. The integration of a haptic feedback for the user increases the individual immersion into the scene [14]. This aspect has already been tested in a number of scientific studies. In case of virtual reality training in manufacturing a multi-modal feedback enable effective transfer of virtual acquired knowledge to real-world [9]. However, a corresponding hardware for the consumer market is only available in limited cases. The consumer hardware is regularly limited to feedback gloves and trackers for tracking physical objects. For the use in the educational system, the simple purchase and a clear integration effort are decisive. The project investigates the necessity and influence of haptic feedback for and on the learning process in a virtual environment. Therefore, a stepwise integration of haptic feedback for different tool classes into the learning software is planned (see Fig. 4). A distinction is made between the following tool classes: static tools (e.g. screwdriver), dynamic tools (e.g. electric screwdriver) and kinematic tools (e.g. Cobot). Moreover, aspects such as human perception, cognitive signal processing and individual visual abilities are taken into account.

#### **Learn and Work Assignments in KoRA**

© bime, Sv-0300

**Fig. 4.** Experimental setup to evaluate the required haptic feedback

Regardless, it must be investigated under which conditions a higher immersion leads to a higher learning success. This relationship has been investigated particularly in the medical field, where the use of mixed reality is already frequently used in doctor training. RANGARAJAN ET AL. have analysed nine studies where a haptic feedback is used in surgical mixed reality education [15]. They report that six out of the nine studies come to the conclusion that a haptic feedback significantly enhance the simulation and reduces the learning curve. Two studies shows no difference and one study suggested negative effects for training due to decreased realism. RANGARAJAN ET AL. conclude that there can be no general recommendation for the use of haptic feedback. Rather, it must be decided individually for each scenario whether a haptic feedback contributes to the learning success and the additional benefit justifies the costs. [15]

Within the framework of the KoRA project, extensive studies are therefore carried out to determine the required degree of haptic feedback. For this purpose, a first test scenario will be created based on Unity and different levels of haptic feedback will be integrated. On the one hand, physical components will be tracked and integrated into the virtual environment and on the other hand, feedback gloves will be used to simulate contact situations. In order to examine the haptic feedback that is required, the test scenario is first carried out with vocational school classes. The degree of feedback is varied and the learning curve of the participants will be observed. Afterwards, the test will be repeated with employees from companies. Based on these results, the decision of which haptic feedback justifies the associated effort can be evaluated.

### **6 Summary**

This paper describes a concept for a Mixed Reality learning environment that transfers the approach of collaborative robotics into the German initial and continuing vocational education and training. Initial studies indicate that particularly in this area exists a gap in actual teaching and training concepts. The introduced concept is based on the didactical concept of learning and work assignments, which represent project-based and employment-oriented learning. All scenarios within the learning software use an experimental modular assembly plant, which exists as a virtual environment. This allows a stepwise integration of haptic feedback elements to analyse the increase of the learning curve. Afterwards, a decision for each scenario has to be made between the benefits and efforts of feedback elements.

### **Acknowledgement**

The project "KoRA" ("Kompetenzentwicklung zur Gestaltung von Mensch-Roboter-Kollaboration unter Anwendung eines Mixed-Reality-basierten Lehr-Lernkonzeptes", Grant number 01PV18009) is funded by the German Federal Ministry of Education within the "Digital Media in Vocational Training" programme.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Creation of an Experimental Engineering Toolbox for the Digital Transformation of Manual Jet Engine Assembly**

Konstantinos Spanoudakis<sup>1</sup> and Fabian Jonen<sup>2</sup> and Christian Borck<sup>3</sup>

> B-TU Cottbus-Senftenberg, 03046 Cottbus, Germany konstantinos.spanoudakis@b-tu.de fabian.jonen@b-tu.de christian.borck@b-tu.de

**Abstract.** Modern aircraft engines require appropriate care and understanding of design and manufacturing. This is even more important, as the production of aerospace engines remains a manual process in many cases with limited data sources. Quality control has to consider verification of manufacturing and assembly steps through specific checks and controls whilst implementing additional data sources.

In this article, a review of the challenges with regard to controls, automation and process, and technical understanding for aerospace engine production and repair is provided. As this requires the collaboration of many teams and partners, an improvement and step change towards deeper understanding and process efficiency is required. As many operations remain manual, innovations in the way humans interact with technology and collaborate with industrial environments are needed.

This article demonstrates the creation and usage of the proposed solutions for collaboration, troubleshooting and error correction in jet engine assembly.

**Keywords:** Industry 4.0, Autonomous Robot Programming, Collaborative Cognitive Robots, Augmented Reality, Virtual Reality.

## **1 Introduction**

If transportation is a blood-vascular system of our world, then all motions elements of this are red blood cells. The systematic prophylactic is needed.

The area of the jet aircraft is a high frontier of the new technologies. It is jet aviation that allows people to travel the fastest traveling. Unlike high-speed trains, the network of air routes covers the whole world.

It has been proven that technological processes and quality control of aircraft engine maintenance are key factors for advancing the process of continuing air-worthiness support. [7]

Unlike many other modern high-tech areas like car and combustion engine assembly lines and production lines of electric motors, the assembly and maintenance of jet engines is quite conservative. In production and in service many manual labor operations are used. This is a factor of an additional risk in spite of numerous checks and controls. Automatic assembly (e.g. with the use of industrial robots) is not applicable.

This makes cooperation difficult. But maintenance and repair of jet engines requires the coordinated work of a large number of people and factories.

To ensure the reliability of aircraft engines, it is of great importance to develop methods for complex multi-parameter diagnostics of technical condition. [15]

In the research projects Cockpit 4.0 and Insight Industrial Analytics - running until mid-2020 - Rolls-Royce Germany and the chair of automation of the Brandenburg technical university Cottbus-Senftenberg is already developing the first digital technologies in the field of assembly, logistics and the linking of development and production / assembly.

Important is not only the use of several manual operations during the maintenance and repair but the use of old communications and data recovery methods during a lot of these operations. It seems first of all about the use of "paper documentation" in the working area and the problem of its actualization.

To solve these problems, considering the restrictions of the production environment and the strict regulations in the aircraft industry, the following tools were chosen within the research project:


This is already the first step forward in product understanding and increased operational efficiency.

However, the partners expect a significant quantum leap through the far-reaching combination of collaborative cognitive robot, smart human-machine interaction, operational machine-related data, logistics information and the exploitation of the so-called digital thread from concept, development, production / assembly, testing to service.

The approach can be generally applied to all maintenance fields. In this research, the before mentioned tools are described and the way they improve manual maintenance is examined.

## **2 Object Detection**

#### **2.1 The Detector**

For the robotic object detector in the jet engine assembly, the faster R-CNN detector [17] is used in combination with the large Resnet101 network, which has a high average precision (more on that in the next section). Detectors are divided into region based and single shot detectors, where single shot detectors (SSD) have a much higher frame rate enabling to perform real time object detection at the cost of lower accuracy. Region based detectors like the mentioned faster R-CNN detector demonstrate an accuracy advantage if real-time speed is not needed.

According to the elaboration of Hui from 2018, compared to other detectors, faster R-CNN is not the fastest but one of the most accurate detectors [10].

#### **2.2 Network Architecture**

In the robotic vision application, the faster R-CNN detector sits on top of a neural network which was trained by the synthetic training data (mentioned in the next section). There are two methods to train a neural network – one where the network architecture will be assembled manually layer by layer with convolutional layers, pooling layers, fully-connected layers, hidden layers etc. It takes a lot of computing power and time to train the layers and their millions of neurons to detect low level contours, colors, shapes etc. in order to recognize objects. The other method is to use a pre-trained model which can already detect various objects like animals, furniture, humans etc. and only overwrite the classifier layers with the objects that need to be detected. However, in this method, the base layers (feature extraction layers) -where the neurons already learned how to identify contours, colors, shapes etc.- remain pre-trained. This is giving the advantage of a short training time and layers already trained by experts for state-of-theart feature extraction. Within the research project the last approach was used. As a pretrained model, the Resnet101 with the already trained coco image dataset [6] was used. Everything that needed to be done was the training of the top classifier layers with training data of objects that need to be recognized.

#### **2.3 Synthetic Training Data Generation**

To generate enough training data and increase the diversity of object appearance, we create multiple sets of rendered images. In these images, we differentiate the position, orientation, scale and material of the 3d object as well as the intensity of the light source. Two sets of 3d objects are sampled, root objects and non-root objects. For root objects, we create an animation sequence of 10.000 frames. The object is animated to rotate in Z and X axes during the frame range. This process provides images of the object at every angle with a 2 degrees threshold. Furthermore, is taken into consideration the pose of the object in the real world and no image is created where the object will not be accessible with the camera, thus, avoiding feeding the CNN with unnecessary data. For non-root objects the same process is followed, and the images created correspond to every angle of the referred object with a threshold of 4 degrees. In order to improve the perception of our CNN we use material diffuse colors as close as possible to real ones. To achieve that we use the actual camera that will be later used, to take photos of the object in different light conditions and different angles to receive realistic range of texture color. Subsequently we pick a range of RGB colors from the photos and use it as diffuse color in our 3d material. Throughout the animation mentioned above, the diffuse color of the object changes smoothly between all received colors. During scale and position transformations, we keep at least 25 by 25 pixels of the object's image inside the safe frame of the camera.

The parts to be inspected will not only be free in space but also parts of an assembly, thus the need for our CNN to be able to recognize parts fused with each other. To generate appropriate data, we integrate the assembly steps into the 3d environment. Starting with the full assembly, we dedicate one render channel per object class and setting its name to the class representative. With all but the first layer of objects, according to fitting instructions, hidden and with regard in transparent background image format and alpha channel enabled we render the entire animation sequence. In each next step of the rendering process we unhide the next visible object class in the assembly and render the animation sequence.

After completion, if the assembly consists of 10 different object classes, thus assembled in 10 steps, we have 100.000 training images.

Furthermore, with the multi-channel rendering, each object which is visible in an assembly step, also has rendered images. The 550000 images of the multi-channel renderings are used only to generate annotation data in a VOC-Pascal format for each image.

The next section describes the approach of how to automatically generate annotation data for our training images.

**Creating annotation data.** With OpenCV the bounding boxes can be obtained for each object in the assembly sequence. Therefore, multi-channel images are created from one assembly scene. The main image of a specific assembly step has a different channel for each object. These multi-channel images only display the (partial) objects, which are not concealed by other objects. From these separated objects, OpenCV can detect bounding boxes of pixel groups. The coordinates of the bounding boxes will then be written into the annotation file. All the bounding boxes of all the objects from one image (main image) of an assembly step will be gathered in one annotation file. Both the main image and the annotation file will then be fed as training data into the neural network. This method allows to automatically generate training data from 3d models in any partial assembled state, allowing the classification layers of the neural network to recognize partial concealed or assembled objects. Combining the large number of synthetic training images, with a small number of real images with ground truth human annotations, will form a training image set with high performance [8].

#### **2.4 Overfit-Resistant Image Synthesis**

We synthesize thousands of images for each 3d model. While the realistic effect considered to be better, images with simple lighting conditions and materials prevent deep CNNs from picking up unreliable patterns. We simplified the creation of clutterness embedding all important randomizing elements together. For image rendering, we explore parameters of rotation, scale, move and light intensity. For the position, scale and orientation condition, we sample values according to the original size of each object as for the deformed object to occupy enough space in the final image thus, to be exploited by the CNN. For the lighting condition, intensity of the light source is sampled. Images rendered as above have a fully transparent background, and the object boundaries are highly contrasted. To prevent classifiers from overfitting such unrealistic boundary patterns, we synthesize the background by a simple and scalable approach [19].

For each rendered image, we sample multiple backgrounds of solid colors, photos of industrial environments, nature and of same-textured-parts assemblies. We use alphacomposition to blend a rendered image as foreground and a scene image as background [19]. Images cropped by cameras safe frame and images with cropped parts by other objects from the multi-layer renders teach the CNN to recognize occluded or truncated images. Finally, we use 75% of our created images as a training data, while the rest 25% of it is used as validation data.

## **3 Collaborative Cognitive Robot (CCR)**

#### **3.1 Cognitive Vision**

With Chapter 2 we created a visual feedback, where a robot can identify objects in his workspace and calculate rough positions in his base coordinate system. The rough estimation of an object's position in the 3d workspace of the robot can be achieved by using the knowledge of the size of the detected object, the constraint that the object always stands on a known height (floor) relative to the robot and the xyz-coordinates and angles of the robotic tool center point (camera position). Since the accuracy of the calculated pose will be far from good and only represent a rough estimation, a second iteration is necessary for the robot to get the exact position of the detected object – getting into close contact with the haptic feedback.

#### **3.2 Haptic Feedback**

**Haptic calibration.** For a robot it is necessary to know the exact object position or damage might occur. Making a robot calibrate itself is great for versatile workspaces without fixtures, granting possible human collaborators freedom in their own ways of working. The CCR achieves this self-adjustment with the help of camera-based object detection and a force-torque sensor. Both can be attached to any off-the-shelf collaborative robot. In the experimental study a force-torque sensor (FT-sensor) from Robotiq (FT300) and a camera module with the google coral edge TPU dev board was used. The process of self-calibration involves calculating a rough position of the object with the camera-based object detector and finally the haptic calibration. Within the haptic calibration, at first the robot slowly moves close to the object's estimated position with a TCP speed slower than 15cm per second. Then the robot will be close to the object performing a slower tactile move (5cm per second) towards the objects surface. When the surface is reached the force applied to the sensor should exceed 2 N or 0.12 Nm, which is gentle enough to not move the object if its weight is above 2 kg. The force threshold for recognizing the touched object should not be lower, since the FT-sensor has a certain noise level. When touching the first contact point of the object, two or more additional points are required for calculating the middle point and angle of the object in the base coordinate system of the robot. After the self-calibration with haptic feedback, the object's position will be accurate within +/-5mm, depending on how the touching tip of the TCP is designed. A very small touch tip would grant the best accuracy, however, since the application is collaborative according to EN ISO/TS 15066, a surface on the robot must never be smaller than 1mm2 resulting in a larger tip and a

lower accuracy. The resulting position error of +/-5mm can be challenging for follow up assembly or pick and place operations. To compensate the error, again the FT-sensor can be used for force-controlled operations.

**Force-controlled operations.** If the position of an object is not programmed or calibrated correctly, damage can be caused to involved components. To become more flexible and versatile a robot can be equipped with a force-torque sensor. With robotic skills like "spiral search" or "linear search" the robot can use haptic feedback to identify object positions or assembly positions simply by monitoring a force drop or rise on contact or successful insertion. Sometimes, the misalignment of the case makes the first insertion attempt fail. With the spiral search algorithm, the bushing is able to be inserted in a flexible way. Each time a bushing is inserted, the correct assembly position is used to correct misalignment or calibration errors. This way the robot learns from his assembly attempts to identify the exact position of the object.

Combining visual with haptic feedback clears the path for flexible and versatile collaboration between robots and humans while using self-calibration and force-controlled operations.

## **4 Smart Assembly Model (SAM) and Autonomous Robot Programming**

With the usage of skills, workspace data and process data, robot programs can be generated and kept updated automatically. These key-components are gathered and managed in a Smart Assembly Model (SAM). SAM is a high- and low-level assembly resource planning system which integrates elements for robotics, humans and processes. It also serves as a data collector and sends out commands to robotic devices, machinehuman interaction systems and other technology in order to control and guide assembly.

As a data collector, it stores CAD-files of assemblies and parts, 3d-models of assembly sites, digitized fitting instructions, point clouds, assembly data etc.

Through its management system it is able to perform a particular set of actions that include path planning, Image processing, point cloud processing, digital fitting instruction interpretation, animations, digital twin processing etc.

These days robot programming is a manual task. The programmer needs knowledge about the assembly routine and the workspace. Adapting robot programs in a changing environment is time consuming and increases non-value-added tasks.

With skill-based programming an autonomous programming system can be implemented which reads digitized fitting instructions (FI) and assembly control records (ACR) with semantic product data (assembly routine) and overlays instructions with 3d factory model (workspace).

Subsequently, the data is processed with a neural network in order to create a robot program consisting of pre-programmed skills and path plans. This automatically generated robot program can be reviewed by a robotic specialist and used as a framework to do fine-tuning.

#### **5 Technology Enhanced Qualification (TEQ)**

Technology enhanced learning enables the worker to learn new processes and work steps independently and immersively, without having to be instructed by a supervisor. It enables the worker to access work instructions, training and additional information on various work steps. The aim is to make the knowledge about the process accessible to the employee. The field of Technology Enhanced Learning for Automation serves on the one hand as a database for knowledge about process flows and on the other hand as a platform for training courses which can be demanded independently by the worker with the right technological means (VR glasses, CAVE, tablet, etc.). Corresponding learning scenarios are automatically created from digital fitting instructions and simulated installation paths (called keyframes). Embedded in a virtual shop floor and using reduced 3D models derived from the product data, immersive learning units can be automatically generated from existing digital fitting instructions.

After the worker has internalized a process in technology enhanced learning, he can train it alone or in groups. The training scenarios are generated from the "Smart Assembly Model". The results of the training can be analyzed and compared with reference values. The deviations can be used to determine which assembly steps are mastered and to what extent (good, sufficient, insufficient). In this way, recommendations can be made for training courses.

#### **6 Smart Human Machine Interaction**

We use a smart system for the interaction between robot and human that employs Augmented, voice/gesture control etc. Using that system, the operator can interact efficiently with the robot and its digital twin. Through that interaction data flow in both directions, from the user to the robot and vice versa. With the proper processing the data is converted into information that both the interacting parts can use. That information processing and exchange leads to gradual continuous development of knowledge. Through the abovementioned interaction with the robot and its digital twin the operator can see visualized trajectories and next steps, future poses and workspace, status, errors etc. Furthermore, collaborative skills like hand over can be controlled using the Augmented reality (AR) interface. AR can be used in many fields of everyday life and the economy [3]. Reduction of error search and correction can be achieved by using AR for assistance tasks in the areas of training [5] [2], assembly, repair [9] or automation [13]. AR technologies have great potential in commissioning and maintenance. For example, when servicing equipment or complex machines, it is important for the technician to have all relevant information available. This includes, for example, information about what activities are to be performed on this system and what has already been done in the past [18].

The research on possible applications of AR has been done for many years [11] [16] [5] because the AR and AR applications are used in many fields, such as project and factory planning. Areas of mechanical engineering and manufacturing automation are interesting for AR. Primary AR is used for the visualization of assembly tasks, machine

operation, or reparation processes. With a camera (e.g., integrated in the tablet or smartphone) the system automatically recognizes the component to which need to be repaired and allows all relevant information as well as the exact assembly instructions to flow directly into the real image. E.g., the automatic fade-in of the circuit diagram while working on electrical system components is conceivable. It is much easier for the technician to record very complex system and to check the relevant measured values [18].

Despite numerous research and development projects in the field of AR technologies, it has not yet been widely used in mechanical engineering. The main reason is because of the hardware. Since the first experiments from the year 2003 up to the increasing interest since the year 2008, the hardware available for AR applications was too heavy and / or too big or too expensive.

Now the actual stage of the hardware and software make a developing and a using of an AR-application easy and possible for everybody [4]. There is a huge amount of free software that can be used easily for complex applications. Thus, it is possible to program easy applications for complex mechanical problems [4].

The task of check and control can be simplified with a using of the AR as well. A practical example of using of the AR for this, is a project for Rolls Royce Germany [14].

#### **7 Conclusion and Future Work**

The current situation in the maintenance and repair of jet engines is far from optimal. The organizational and manager solutions and methods for technical processes, control of information and other things are not optimal either. These solutions solve the problem of reliability and safety in the field of engine building and engine maintenance. But such decisions require considerable time and material costs.

This is especially evident with the introduction of new engine models. Thus, classical solutions, which imply for example a large number of paper documents, are not flexible enough. This is unacceptable in a constantly changing modern world.

This problem can be formalized as the following points:


So, there is not one, but several similar problems. It cannot be enough to use only one new method or approach to solve them all. These problems require a comprehensive solution.

This article proposes and discusses the following methods and technological solutions to overcome the problem:


The article provides a basic description of these methods and solutions. Showing their main advantages and disadvantages.

The next step of the authors of the article is the practical implementation of several methods in the framework of a small test bench. The main focus will be on the design and use of Smart Human Machine Interaction, machine learning and CCR.

#### **8 Acknowledgement**

This publication is funded by the European Union Fund for Regional Development and supported by the Federal State of Brandenburg.

The project is COCKPIT 4.0 (CustOmized Collaborative Knowledge Pilot for Industrial Technology 4.0) with EFRE project number 80170503 (www.b-tu.de/en/automation/research/projects/current-projects#c174887).

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Robot-Based Fiber Spray Process for Small Batch Production**

Edgar Schmidt<sup>1</sup>, Jonas Winkelbauer<sup>2</sup>, Georg Puchas<sup>2</sup>, Dominik Henrich<sup>1</sup>, and Walter Krenkel<sup>2</sup>

<sup>1</sup> University of Bayreuth, Chair for Robotics and Embedded Systems, D-95440 Bayreuth, Germany, edgar.schmidt@uni-bayreuth.de

<sup>2</sup> University of Bayreuth, Department for Ceramic Materials Engineering, D-95440 Bayreuth, Germany, jonas.winkelbauer@uni-bayreuth.de

**Abstract.** Fiber spraying processes are used in the production of fiberreinforced composites, but are performed mainly manual. Given various possibilities for automating such processes, only a few are suitable for the production in small and medium-sized enterprises. We present an approach on robot-based automation of fiber spraying processes. This approach is characterized by its easy programmability of the robot system using the playback programming paradigm. We simulated the reachability and accessibility of our robot-based automation approach and present our prototype system, which can be used for the production of small batches of fiber-reinforced composites.

**Keywords:** industrial robotics, automation technologies, flexible manufacturing, small batch production, intuitive robot programming, playback programming, fiber spraying

## **1 Introduction**

*Fiber spraying processes* (FSPs, also called spray lay-up processes) are one of the most commonly used production methods for short fiber-reinforced composites [1]. While often used in the production of glass and carbon fiber composites, FSPs are not commonly used for ceramic matrix composites, such as oxide fiber composites (OFC). However, recent research shows that applying an FSP in context of manufacturing of OFC can save over one third of the production costs [2]. A further reduction of the production costs and an increase of the throughput is possible with the automation of such processes.

Robot-based automation is still only rarely spread in small and medium-sized enterprises (SMEs), even though a low cost and high quality production is possible with their use [3]. The slight use of robot-based automation is largely due to the production of small batches, which is characterized by different phases of robot use with frequent reconfiguration and changeover of the robot systems [4].

**Fig. 1.** Proposed cell for robot-based fiber spraying process.

A fast reconfiguration can be achieved through a speed up of the robot programming, which is possible when using intuitive robot programming approaches like *playback programming* [5][6][7]. With the use of such techniques a cost-efficient production of small batches could be possible.

We envision a robot-based automation approach of FSPs, with the aim to perform various FSP with different fiber materials in one cell with a shorter changeover time. The resulting system should be quickly and spontaneously programmed by non-experts in robot programming, which are experts in the given process, who we call *domain experts*.

In this paper, we present a novel robot-based production system for the production of OFC, in which a robot is covering a positive or negative mould with a fiber slurry mixture (Figure 1). The system can be extended to other kinds of FSP like glass fiber spraying. The robot in the cell is further designed to be programmed with the intuitive playback programming paradigm. Section 2 gives an brief overview of the related work regarding the automation of FSPs and their programming frameworks. Then, we explain in Section 3 our robotbased automation approach in detail. The simulation and development of our prototypical system is described in Section 4. In Section 5, we summarize and conclude the paper.

### **2 Related Work**

Most of the commercially available fiber spraying systems are manual operated systems, on which the various required components are installed, so that they can be used like a paint spray gun [8][9]. When used, these manually operated systems expose the domain expert to polluted air (overspray, fibers) during the spraying. Also, the quality of the produced composite depends heavily on the expertise of the operator with every workpiece. So, the quality of produced composites is hardly reproducible and the error rate is high. An automation approach should put a stop to these disadvantages.

To the best of our knowledge, there is only one prototypical system for robotbased FSPs using off-line programming, which can be characterized by graphical programming and virtual reality [10]. Different advantages of this approach are given. One of these is, that during the production no worker is exposed to the polluted air. Also, the use of a rotary table as an external rotation axis enables the processing of complex shapes. Further, layers with uniform thickness can be applied repeatedly. However, the off-line programming approach is only of limited suitability for small batch production since such programming framework requires a special training in robot programming. A new approach should put a focus on an *intuitive* robot programming framework, which exploits the process specific knowledge of the operator direct.

Using on-line programming paradigms could allow a cost-efficient production, for example by hand guiding of the robot tool head, also called kinesthetic guiding [6][11]. One of the approaches using kinesthetic guiding is playback programming and has the advantage, that process-specific knowledge of a domain expert continues to be incorporated directly into the programming process of the robot, as already shown for other processes in [7][11][12][13]. A new approach to robot-based automation of FSP for small batch production can also take advantages from this programming method. For this reason, a novel approach on robot-based automation of FSP should integrate a playback programming framework.

### **3 Automation of Fiber Spraying**

In the following, we describe our approach in automation of FSPs. First, the process itself is explained. Second, we consider important design criteria for an automation approach. Then, we explain our system design in detail. Finally, we describe how the robot programming framework of the process is designed.

#### **3.1 Fiber Spraying Process for OFC**

In an FSP for the manufacturing of OFCs aplications, continuous oxidic fiber bundles (rovings) are chopped to uniform length by a cutting unit immediately before being ejected out of the cutting unit and into a slurry spray, by which they are entrained. Due to the angle and the distance between the tool head (cutting unit and spray gun) and the mould surface the bundles are infiltrated during flight with the slurry, as shown in Figure 2. The infiltrated fiber bundles, with slurry being inside the fiber bundles as well as between the fiber bundles reach the mould surface and are orientated randomly.

**Fig. 2.** Schematic representation of a manual fiber spraying process.

If desired, the thus prepared pre-impregnated fiber preform (prepreg) can be conditioned in a climate chamber to adjust a defined water content, stacked and laminated. In case of flat prepregs, they are stacked to a laminate and laminated or directly compressed under pressure, if they are not already corresponding to the final form. After consolidation in a drying chamber, the green bodies are sintered for two hours at 1225 ◦C. The short fiber bundle reinforcement allows more complex shapes than it is the case with fabrics, without fear of damaging the fiber bundles or getting delaminations due stresses in the fabrics and a resulting resilience.

As already mentioned this method can be used on the one hand as a primary shaping process, such as by spraying on a positive or a negative mold, or as a shaping process using the flat prepreg to shape more complex geometries, which are not accessible using fabric-based prepregs. On the other hand the spraying process build flat prepregs which can be processed like conventional prepregs. The big advantage here is the increased drapability of the short fiber prepregs compared to the fabric prepregs. Furthermore, the random arrangement of the fibres results in area-isotropic mechanical properties.

#### **3.2 Evaluation and Design Criteria**

The design of an automation approach for an FSP depends on various criteria. A very important criterion is the possibility to adjust different parameters of the process. One of these parameters is the image of the spay pattern from which the fibers are captured. This spray pattern defines the slurry spray which infiltrates the fiber bundles, entrains them towards the mould and so defines also the resulting material properties of the workpiece. An automation approach must be able to adjust the slurry spray regarding flow volume and spray pattern, according to the objectives of a workpiece. In addition to the slurry spray, the fiber flow needs also to be adjustable so that material properties can be modified. This plurality of free variables allows the production of a wide variety of workpieces with different material properties.

In addition, economic criteria have to be considered, which are directly related to the production costs. One of these criteria is the overall material consumption of fiber and slurry. Minimizing the material consumption, e.g. the amount of overspray which does not form the workpiece, will lead directly to lower production costs when using high-expensive fibers. Another criterion is the production time, whose minimization will increase the throughput of produced components. A further time related criterion is the setup time for the production of a single component. This time is required for the configuration of the system, when a new component of the same material class has to be produced. Reducing the configuration time is essential for the production of small batches as described in Section 1. Only if all of the listed criteria are minimized to its best, a cost-efficient production of small batches can be reached.

Finally, process-specific criteria must be considered. The first two criteria are to keep a fixed orientation and distance from the tool head to the mould and to guarantee the accessibility of the spraying on the mould. This results in an optimal application of the fiber slurry mixture. The accessibility is challenging due to the fact that we want to accomplish a complete coating even for moulds with complex geometries. The last criterion is the ability to generate a homogeneous coverage over the complete mould. Achieving a homogeneous coverage leads to uniformly distributed material properties (e.g. the tensile or bending strength) over the entire geometry, which can be used as an quality measure for a correctly produced workpiece.

#### **3.3 Robot-based Automation**

As automation approach, we propose a robot system, which manipulates standard components of manually operated systems for FSPs, focusing on the production of small components with a maximum size of up to 500×500×500 mm<sup>3</sup>. First, the structure of the cell is explained, then the concept of the tool head.

The boundaries of the cell are formed by a cabin of system profiles (W: 2000 mm × H: 2200 mm × D: 1800 mm). The cabin is sealed by windows and resealable doors made of acrylic glass, so that overspray and fibers are contained within the cell and the production can be monitored from outside. In addition, all control elements are mounted outside the cell so that the presence of a person inside the cell is not necessary while the FSP is running. Within the cell, an underfloor downdraft table (W: 1080 mm × H: 460 mm × D: 1195 mm) is set up, where a mould can be positioned. In the middle of the downdraft table a programmable turntable is installed allowing the rotation of the mould through

**Fig. 3.** Proposed concept of a robot-based fiber spraying cell. Left: Interior of the fiber spraying cell. Right: Tool head design with possibility of kinesthetic guiding.

this external axis. Additional degrees of freedom can be added by extending the turntable to a two-axis turntable. The downdraft table limits the intended working space of a six degree of freedom robot, which is mounted in front of the downdraft table (Figure 3, left).

The possibility of programming by kinesthetic guiding was taken into consideration in the design of the tool head (Figure 3, right). A force/torque sensor is mounted in z-direction from the flange. A handle is additionally attached on the force/torque sensor, through which the guiding of the robot arm is enabled. The automatic spray gun is installed in front of the tool head. The slurry flow volume as well as the spray pattern are adjusted by a programmable system of proportional pressure regulators. The fibers are fed via a cutter mounted over the spray gun. The fiber roving supply is mounted on the ceiling of the cell. The amount of fiber throughput is also controlled by the proportional pressure regulators. A diffractive optical element projects a pattern onto the mould allowing the estimation of the spray cone and thus the application of material without a material flow. The resulting tool head has a weight of 4 kg, which can also be mounted on lightweight robots.

Complying a fixed orientation and distance to the mould and achieving homogeneous layer thicknesses is possible due to the programmability and repeatability of robot movements. At the same time an increase in the throughput of produced ceramics can be achieved by the autonomous execution of robot programs. The accessibility of the mould is guaranteed by the use of a turntable, which also reduces the robot wear.

#### **3.4 Robot Programming**

An approach in robot-based automation for small batches requires a programming technique, which can quickly provide robot programs for covering various moulds fast. Textually created robot programs require the specialized knowledge of a robotics expert. Further, a new robot program for each workpiece or groups of workpieces in case of simple geometries (plates, tubes) is needed. For this reason, the robot has been equipped with the ability to be guided kinesthetically by a domain expert. The kinesthetically guiding allows the use of the playback programming paradigm [6]. Given sensor data from a force/torque sensor, the robot moves in answer to this input of the operator, who can be a domain expert. While the robot arm is guided by the operator through the spraying task, the system records robot configurations in a fixed cycle and generates a robot program with the demonstrated trajectory. After this programming stage, the robot program can be played back so that he can execute the intended task.

The use of the playback programming paradigm allows the quick teaching of trajectories for geometrically complex moulds, which would be time and cost consuming by a textual programming approach. Also, kinesthetic programming is intuitive and can be done by domain experts. Further, our system allows, that the playback programming can be done without material flow and is so resourcesaving. The resulting program can be used for the automatic production while the domain expert can address oneself to other tasks. Finally the reproducibility of robot movements guarantees that a fault-free program can be executed again and thus reduces the error rate of the production.

#### **4 Prototypical System Development**

In this section, we describe the development of our prototypical system. First, we describe the analysis of our approach through simulation. Then, we show how our first prototypical system was designed and give an overview about its use.

#### **4.1 Simulation**

Given geometrical data of the cell and tool head in form of CAD-models, we simulate the reachability of the robot for different poses. Our aim is to check, whether the optimal distance for covering a mould is achievable. The optimal distance is given by our first application, the process given in [2]. With the optimal distance, we generate distributed poses on a hemisphere around the position where the mould will be placed. After that, we check if the robot kinematic can reach the poses on the hemisphere. Additionally, we let the simulated robot move into the reachable positions, using a collision check between the robot and the downdraft table to guarantee the accessibility of the poses.

For simulation we use the *St¨aubli Robotic Suite* [14] (Figure 4, left). We sample 500 poses and a distance from spray jet to mould of 400 mm is used, resulting in a overall distance from flange to the hemisphere center point of 645

**Fig. 4.** Simulation of our approach on robot-based fiber spraying process. Left: Simulation setup with reachable poses marked as black dots. Right: Detailed results of our simulation with classification into not reachable poses (red), reachable poses but the movement causes a collision (blue) and collision-free reachable poses (green).

mm. 263 of the generated poses were reachable, in which 85 result in collisions with the robot cell while moving. So, the remaining 178 poses are accessible from the robot without collision. Eight collision-related poses are characterized by being in an area, which is dominated by accessible poses. The results are shown in Figure 4, right.

One of our conclusions from the simulation is, that only with the use of a turntable geometrical high complex moulds with the required maximum size can be covered complete. Another conclusion is, that for plain geometries, the reachability and accessibility for the optimal distance can be guaranteed with our concrete approach. Our approach, which uses a combination of turntable and robot, is thus sufficient for the required process and workpieces.

#### **4.2 Prototypical System**

For our first prototypical system we use a St¨aubli TX2-90L [15] as robot and an ISEL IT 116 [16] controller for the turntable. The control of the different pressures needed for the material feed is given by a system of proportional pressure regulators. Slurry output is enabled through a spray gun, which is a commercially available system suited for the use mounted on a robot. The turntable and pressure regulators can also be controlled by the manual robot control pendant. Five pressures can be configured via the regulators: two defining the slurry, one for setting the actual slurry output to ON/OFF, one for controlling the volume flow of the slurry, and the last for the rotation speed of the chopper unit. A force/torque sensor provides the input data for the playback programming. All components are connected and controlled via the controller of the robot.

**Fig. 5.** An example use of our prototypical system for robot-based fiber spraying process. The operator utilizes the manual control pendant for programming.

Given a mould to cover, the operator teaches the required trajectory for the robot by kinesthetic guiding. The playback programing without material feed is defined through the user interface of the manual pendant control before the programming stage begins. Also, the turntable and the pressure regulators are configurable with the manual pendant control, as shown in Figure 5. After teaching, the trajectory can be stored and reloaded for a later production. In the execution stage, the trajectory is replayed with temporal synchronization between the robot, the regulators and the turntable.

This prototypical system allows the playback programming of geometrical complex trajectories with a one shot demonstration, resulting in the fast production of small batches of new workpieces. The quality of the programmed trajectory depends on the skill of the system operator and is suitable for workers with expertise in the given process, but can easily used by them without special training in robot programming.

### **5 Conclusion and Future Work**

In this paper, a novel robot-based automation approach for production of fiber composites with FSPs is proposed. The developed prototypical system can be used for the production of OFCs. Reachability and accessibility is guaranteed through simulation. The presented playback programming framework allows the programming of the robot in a new way by a domain expert without special training in robot programming. However, the presented playback programming framework is still not sufficient for a small batch production of geometrical high complex workpieces. For this reason, we will use in future work a playback programming framework which allows the intuitive programming of such tasks using video editing concepts [5]. Also, this approach will be extended with functionalities for supporting the domain expert, like the simulation of the process or the optimization of the programmed trajectory before execution.

### **Acknowledgements**

This work has partly been supported by the European Regional Development Fund (ERDF) under the project "Roadmap zur flexiblen Produktion individueller Produkte" (Roadmap flexPro).

## **References**


304

**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Affordance Based Disambiguation and Validation in Human-Robot Dialogue

Kim W¨olfel and Dominik Henrich

University of Bayreuth, Chair for Applied Computer Science III, (Robotics and Embedded Systems) Universit¨atsstraße 30, 95447 Bayreuth, Germany kim.woelfel@uni-bayreuth.de, dominik.henrich@uni-bayreuth.de, http://robotics.uni-bayreuth.de/

Abstract. Speech-based robot instruction is a promising field in private households and in small and medium-sized enterprises, because it facilitates a comfortable way of communicating with robot systems, even while the users' hands are occupied. An essential problem in transforming the speech-based instructions into complex robot motions is the validation of the resulting motions regarding feasibility. This emerges from the fact that the user may not be fully aware of the capabilities of the robot including its gripper in a given working environment. We present an approach that tackles this problem by utilizing dynamic affordances for the disambiguation and elemental validation of speech-based instructions.

Keywords: Situated Dialogue, Affordances, Disambiguation

## 1 Introduction

One long term goal in current robotic research is the development of robot systems which have approximately the same cognitive, communicational, and handling abilities as humans. As part of this ongoing development, application domains for robot systems shall be expanded, from industrial settings with separated working cells, fixed object positions, and preprogrammed motions towards a flexible usage in small or medium-sized enterprises or private households.

An intuitive way of communicating with such a robot system is to instruct it via spoken commands. Advantages of this approach are that it facilitates the instruction of a robot without having to learn a robot programming language and an instruction while the users' hands are occupied. In user studies we noticed that ambiguities are a problem that occurs while instructing a robot and that in most cases ambiguities result from differences in the mental representation of the user and the robot system. For example, users omit parameters in an instruction, which seem obvious from their perspective. Informing the user about missing or incorrect parameters can only be performed well if a common ground is present. One way of doing this is by utilizing affordances which can be seen as a functionality of objects, e.g. graspability or liftability. In the case of force based object manipulation this is a helpful tool, because affordances of the workpiece, the robot arm, and the tool in use can be evaluated for a given instruction.

The main contribution of this paper is a concept that is able to disambiguate and validate instructions based on dynamic affordances regarding the robot, its tool, and the workpiece. The scientific gain of this approach is bipartite: First, affordances are used in a preprocessing step to reduce the parameter space and even empowers a system to execute motions emerging from incomplete commands. Second, by using dynamic affordances the feasability of instructions can be applied in changing working environments and for various hardware. All this is done to investigate the application of affordances for previously unknown tasks, instead of relying on plans or pre-trained models. Moreover, utilizing affordances facilitates a communication with the user on a common ground, which is helpful in the case of sensor-based robot motions.

In Section 2, an affordance overview is given, highlighting approaches regarding affordances, disambiguation, and a combination of these. In Section 3, the approach is described in detail. Section 4 contains the description and results from our user study verifying the validity and usefulness of our approach and in Section 5 a summary of the approach is given, as well as a glance at future work.

#### 2 Related Work

In this chapter, approaches are presented that deal with the definition and use of affordances, disambiguation of spoken instructions in human-robot dialogue, and concepts combining these areas. We also describe in what way our approach differs from the state of the art.

Experts and non-experts may not always be aware of the capabilities of a robot system while instructing it. This holds true especially in force-based tasks, like peg-in-hole or cutting tasks, where force parameters have to be considered as well. A well studied approach for defining and evaluating these functionalities are affordances introduced by [6], defining what can or cannot be done with an object [20]. The concept of affordances has been part of the scientific community for several decades. It leads to a lot of publications surveyed regarding psychology, neuroscience and robotics [11], human-robot interaction [21], and developmental robotics [19]. Being dissatisfied from the variety of formalizations, a computational affordance model is given in [27] which classifies approaches regarding perception (selective attention, temporality, order, level, perspective), structure (chaining, competitive, abstraction), and development (exploitation, generalization, prediction, learning, acquisition). Affordance learning has also been studied for a lot of applications, for example grasping [22], [5], traversability [2], [12], [24], or tool-use [1], [28].

Disambiguation has also been discussed a lot along the last decades [15], [14], [17], [25]. Most approaches try to solve ambiguities by using clarification questions based on yes/no questions [4], [9], listing all options [16], or generic

Fig. 1. Concept overview: Blue arrows highlight the main information flow and grey arrows the control flow between the logics and the hardware/data base .

WH-questions [23]. In search for a general approach for disambiguation, [25] suggests three design recommendations. Robots should present all options only if the number of options is less than three, otherwise robot systems should generate yes/no or WH-questions (D1). Clarification requests should be formulated in a way, that the user thinks the robot understood the goal of the instruction (D2). Robots should use pragmatically appropriate phrasings (D3).

Investigating the combination of affordances and disambiguation in the field of speech-based interaction has been poorly investigated by just a few approaches [20], [8], [10], [3]. In [10] the application of affordances for co-reference resolution in speech based commands was mentioned, but not further investigated. In [8] ambiguous commands where solved utilizing the affordance concept together with a prediction based on previous commands. In [3] a neural network learned with a huge text corpora was used to disambiguate incomplete instructions.

In difference to the introduced approaches, we present a system which applies the concept of affordances in form of pre-conditions [13] for disambiguating and validating spoken instructions without relying on previous recorded data or text corpora, but solely on dynamic affordances, which means that their state can change over time or depending on the hardware set-up. For example, the reachability of an object can change during a manipulation task (hidden behind other work pieces) or not reachable by the robot arm currently installed.

#### 3 System Overview

This section describes the main aspects of our framework. After giving a short framework overview (Section 3.1), we describe the modules in more detail, which are part of the main information flow. Besides explaining the dialogue module (Section 3.2), we present our disambiguation approach implemented in the identification and validation module (Section 3.3 and Section 3.4), and explain how and what kind of feedback is generated by the system (Section 3.5).

#### 3.1 Framework Overview

Our framework for SPeech based Instruction of RObot systems (SPIRO) is divided in static modules: dialogue-manager (DM), identification module, validation module, feedback module, and changeable modules: The active robot (R), the attached tool (T), and the knowledge data base (DB) (see Figure 1).

The information flow starts by a spoken user instruction which is recognized and semantically enriched by the DM. This information is then passed to the identification module, which has the purpose to identify the motion contained in the instruction along with the necessary parameters based on attached affordances. The validation module then checks the given input regarding the affordances in the current world state. The validated information is then passed to the feedback module which passes the appropriate motion data to the hardware and the acknowledgement to the DM. If an ambiguous or, in terms of the domain, incorrect instruction is recognized by the DM, identification or validation module, the problematic affordance is tagged as non valid or ambiguous and an appropriate clarification question is generated by the DM. In this case, no robot motion is generated.

To allow a flexible use of our system we designed it in a way, that the robot, tool, and database can be exchanged easily. Being aware that switching the robot arm does not happen as often as switching the tool or underlying database we nevertheless included it to be also able to cover even rare cases.

#### 3.2 Dialogue Manager

The DM is implemented using the ontology based dialogue platform Semvox <sup>1</sup> , because it offers a fast design and adjustment of dialogue system, combined with the Nuance<sup>2</sup> framework, because of its valuable speech-to-text and speechrecognition features. The DM has four purposes: Automatic speech recognition (ASR), semantic role labeling (SRL) [18] of incoming user instructions based on pre-defined ontologies, i.e. labeling red as a colour or shove as a verb, generating verbal phrases or starting a dialogue based on data provided by the feedback module, and returning system answers in an acoustic way (TTS). Here, an ontology is an "is a"-hierarchy which allows the user to add semantic knowledge to the spoken instructions.

The ASR module converts the spoken user instruction into word set hypothesis. The hypothesis with the highest probability is then provided to the system for further processing. For example, an instruction like: "Place the cup on the plate!" is transformed into an instruction tuple

I = (verb(Place), workpiece(cup), tool(gripper), destination(plate)).

This SRL is done automatically by Semvox as long as semantic roles like verb, tool, etc. are pre defined. The generated instruction tuple is then sent to the

<sup>1</sup> https://www.semvox.de/en/technologies/odp-s3-the-basis-for-interactive-systems/

<sup>2</sup> https://www.semvox.de/en/technologies/nuance-speech-recognition-tts/

Fig. 2. Validation Module validating incoming instructions based on affordances. If an ambiguity is perceived each task directly sends the information to the feedback module.

identification module and the validation module to compute the appropriate answer to the instruction.

The other purpose is to generate verbal feedback for the user in one of the four cases: Each parameter of the instruction was specified correctly (C1), the system estimated a disambiguation of an incomplete or ambiguous instruction (C2), the system was not able to disambiguate parameters of the instruction (C3), the system recognized incorrect instruction parameters (C4).

In case C1 an acknowledgement like "Yes!" or "Yes, of course!" is sent to the user. In case C2 the disambiguated instruction is sent to the user with the opportunity to interrupt the system by saying "No!" or "Stop!" and the motion is executed by the robot. In case C3, a clarification question based on the ambiguous parameters is generated and in case C4 the user is informed about the false parameters.

#### 3.3 Identification Module

This module has the purpose to identify the motion, i.e. finding an appropriate motion template in the DB, and accompanied parameters intended by the user instruction. Given the instruction tupel I recognized by the DM, the suitable motion template is searched and filled using the affordance filtering approach [7] performed by the two sub modules: Motion identification (T1) and parameter identification (T2) (see Figure 2).

In T1 the intended motion has to be identified. Here, two cases arise. Case 1: The verb is mentioned. In this case, the system directly picks the appropriate motion templates for this verb from the DB and continues with T2. Case 2: No verb is mentioned. In this case, we consider two possibilities: The user implicitly uses the verb from the previous instruction or he explicitly did not mention the verb because its obvious for him. The implicit case can easily be solved with Semvox, which allows an auto-filling of the verb slot if the current instruction has the same structure as the previous instruction, which then automatically leads us to Case 1. The explicit case leads to a work around. The system then has a look at the other specified parameters and tries to find a verb which maps to these parameters, both syntactically and affordance-based. If this still does not lead to a single verb candidate, the verb slot is filled with the possible candidates, tagged as ambiguous and the augmented instruction tupel is send to the validation module. Imagine the user just saying: "The big screw!". Instead of not answering at all or responding: "What do you want me to do with it?", we have a look at the valid affordance at this time step and try to disambiguate the opportunities for the verb slot. If this is not possible, we tag the verb slot as ambiguous.

After the motion is identified, the remaining parameters are filled using the motion template provided by the DB. In the simple case all the parameters are defined and we send the augmented instruction tupel to the validation module. If one of the parameters is ambiguous, a disambiguation is necessary. Here again, we are using affordances to filter ambiguous or even generate missing parameters. Imagine the instruction: "Shove to the left!" Normally, we would have to ask the user which object he wants us to shove to the left, but maybe there is only one object in our workspace that is shove able. If there is more than one option left, we tag the affected parameter slots as ambiguous before sending the augmented instruction tupel to the validation module.

#### 3.4 Validation Module

The purpose of the validation module is to evaluate the instruction parameters by evaluating the affordances for the given world state and tag them with the description: valid, invalid, ambiguous, and disambiguated. Using dynamic affordances is necessary, because in some cases valid defined instructions may not be executable, because the object is out of reach for the robot arm or the gripper is not suitable for grasping the workpiece. A parameter is tagged as valid, if it is distinct regarding the necessary affordance and has the correct type regarding the verb frame. A parameter is tagged invalid, if it does not have the correct type regarding the verb frame. A parameter is tagged ambiguous if a set of possible real world objects is possible for it and they are of the right type. A parameter is tagged disambiguated if the identification and validation module managed to disambiguate the user instruction. The tagged instructions are then send to the feedback module for further investigations.

#### 3.5 Feedback Module

The feedback module is responsible for transforming validated instructions into either a text-based response for the DM or a text-based response for the DM in combination with robot motion parameters for the motion generator. What kind of feedback is given depends on the validity of the parameters slots of the instruction. In the case that the validity slot of all parameters holds valid the feedback is given in form of an acknowledgement phrase ("Ok!", "Sure!", "Yes!") send to the DM together with motion parameters sent to the motion generator, which generates a hybrid motion based the approach presented in [26]. If the validity slots hold disambiguated as well as valid, the instructions are sent to the DM for a spoken response before the motion is executed. If the validity slot

Fig. 3. 1: Example working environment for instructing a robot arm. 2 - 4: Intermediate state of the key holder. 5: Finalized key holder.

of at least one parameter of an instruction holds I, the user is told to correct the specific parameter and no robot motion is generated. If one of the given parameter holds ambiguous, a clarification question based on D1 - D3, defined in section 2, is generated and send to the DM. This process is repeated for one property at a time until the presented instruction is valid regarding all occuring parameters.

## 4 User Study

In this section, a user study is presented which was conducted to gain information on how users react to incomplete and ambiguous instructions and why they choose their reaction. After a presentation of the user study setup (Section 4.1), the results are evaluated as well as discussed (Section 4.2).

#### 4.1 Setup

The study was set up using the online tool Survey Monkey<sup>3</sup> and was divided in two main parts.

In the first part, we wanted to gain information about the necessity of a validation component and presented the participants the working environment shown in Figure 3, 1, together with the request to instruct the shown robot to place a motor with a given weight of 4 kg from location P1 to location P2. This was asked to see, if users are aware of the capabilities of lightweight robot arms and at the same time if a validation or correction of user instructions is necessary in cases where a robot is not able to perform an action.

In the second part, the participants were put in the position of an assistant robot, which helps its co-worker to build a key holder (see Figure 3, 2 - 5). The tasks for the participants where chosen in that manner, because they are either better executable by a robot due its precision or because the user typically does

<sup>3</sup> https://www.surveymonkey.de/


Table 1. Votes for the different reaction types (RT1 - RT6) for the tasks marking, inserting, and cleaning in the second part of the user study.

not want to perform the task. For each instructed task, the participants had an image of the current state of the key holder, an instruction of his co-worker, and possible tools or objects to perform the tasks. They were then asked to choose between a set of reaction types: spoken disambiguation of three candidates and no motion generation (RT1), spoken disambiguation of two candidates and no motion execution (RT2), spoken query for the missing/ambiguous parameter and no motion execution (RT3), spoken feedback in form of the disambiguated instruction and motion execution (RT4), Spoken acknowledgement and motion execution (RT5), no spoken feedback and motion execution (RT6).

The first instruction is: "Mark the board with pattern M1!", the information what M1 looks like is given as well as the tools: pencil, text marker and an permanent marker. The participants are then told that their co-worker inserted holes at the marked spots and that the new instruction is: "Insert the hooks into the holes!", followed by a set of hooks with different lengths. After that the participants are told that their co-worker screwed in the hooks, took the key holder away, leaving a dirty workplace with the comment: "Clean the workplace!". They are also given a set of tools suitable for cleaning a workplace: sponge, cloth and hand brush.

#### 4.2 Results

Overall 15 participants joined the first part of the study. Although experts as well as non-experts where among the participants, all of them instructed the robot to move the motor from place P1 to place P2 without considering if the robot arm is even capable of doing it. This result clearly shows, that a validation even in simple tasks, like moving an object, is necessary. Without a validation, this would normally lead to an incorrect execution of the motion resulting in frustration of the user or even damage. Using our affordance based validation, we are not only able to perceive this error, but also to formulate a spoken solution to this problem, in this case: "I can not lift the motor since it is too heavy!".

Overall 16 participants joined the second part of the study. The results are shown in Table 1. In the marking task most of the participants (43%) favoured RT4, especially the use of the pencil. One participant even wrote, that mentioning the instruction he understood, enables his co-worker to intervene, if he is not pleased with the disambiguation. In the inserting task, even more (56%) participants chose the reaction RT4. Here, two participants justified their reactions by saying that the user is able to interrupt the motion, if he is no pleased with it. The remaining participants also mentioned that they made their decision based on the physical properties of the hooks and the board, thus, in a way our system would work as well. In the cleaning task, the majority of the participant chose RT5 as their favourite reaction and RT4 only got the second place. Based on the participants justifications, the brush is in this case the best tool, or at least as good as the cloth, but no participant mentioned, that either the sponge or the cloth would be the best tool. This leads us to the conclusion that a listing of opportunities is only useful, if the evaluation of affordances leads to a similar rating.

### 5 Conclusion

We present a concept for an affordance based disambiguation and validation of incomplete, incorrect, and ambiguous spoken robot instructions. By utilizing dynamic affordances the system is able to reduce the amount of candidates in the case of ambiguous instructions or even solve the ambiguity. Moreover, using dynamic affordances allows not only validating semantically correct instructions, but also contains the information of the problems' source on a human understandable level. Limits of our approach are the common problem of not being able to reduce the solution space and the complexity of computing the dynamic affordances, while the latter one is still part of current research.

To evaluate our concept, we also conducted a user study where we collected information on how and why users react to incomplete and ambiguous reaction, and if they are aware of robot capabilities even in simple tasks. On the one hand, the results correspond to the presented concept. On the other hand, we also found a way to improve our concept by using acknowledgements if the disambiguated parameters are rated high enough.

In the future, we will implement the whole concept and perform further user studies to be able to improve our dialogue system.

#### Acknowledgements

This work has partly been supported by Deutsche Forschungsgemeinschaft (DFG) under grant agreement He2696-18.

#### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Trajectory Optimization for the Handling of Elastically Coupled Objects via Reinforcement Learning and Flatness-Based Control

Daniel Kaczor, Martin Bensch, Moritz Schappler and Tobias Ortmaier

Leibniz Universit¨at Hannover, Institute of Mechatronic Systems Appelstraße 11a, 30167 Hannover daniel.kaczor@imes.uni-hannover.de

Abstract. Positioning objects in industrial handling applications is often compromised by elasticity-induced oscillations reducing the possible motion time and thereby the performance and profitability of the automation solution. Existing approaches for oscillation reduction mostly focus on the elasticity of the handling system itself, i.e. the robot structure. Depending on the task, elastic parts or elastic grippers like suction cups strongly influence the oscillation and prevent faster positioning. In this paper, the problem is investigated exemplarily with a typical handling robot and an additional end effector setup representing the elastic load. The handling object is modeled as a base-excited spring and mass, making the proposed approach independent from the robot structure. A model-based feed-forward control based on differential flatness and a machine-learning method are used to reduce oscillations solely with a modification of the end effector trajectory of the robot. Both methods achieve a reduction of oscillation amplitudes of 85 % for the test setup, promising a significant increase in performance. Further investigations on the uncertainty of the parameterization prove the applicability of the not yet widely-used learning approach in the field of oscillation reduction.

Keywords: flatness-based control, reinforcement learning, double deep Q-network, trajectory optimization, oscillation reduction

## 1 Introduction

Pick and place processes are one of the most common applications in automation technology and robotics and their extensive use in modern production enables significant benefits even by the smallest improvements in accuracy or speed of motion. For this reason, various efforts are being made to increase the performance of industrial robots and, above all, to optimize the accuracy of their end effector positioning.

Classical methods to achieve this are e.g. model-based feedforward control of actuator speed and torque for a given motion profile. The motion profile can further be optimized regarding motion time and smoothness based on the robot dynamics, as shown in [13] for high-speed pick and place applications. A generalization of feedforward control is given by flatness-based control which theoretically makes it possible to design a motion profile for complete vibration reduction [6]. The implementation for high-speed positioning is e.g. presented in [1] for a linear flexible motion system using trapezoidal motion profiles. The concept of adaptive input shaping from [14] allows a more robust parameterization than the feedforward control by using frequency parameters instead of specific physical parameters, which can be difficult to obtain. These approaches all share the necessity of a correct robot model and an identification of the required model parameters, making them partially impractical in their implementation.

The methods of machine learning on the contrary do not require system knowledge or any kind of parameter identification, since they are purely based on recorded data. An overview of different learning methods for the feedforward control of a 7-DoF (degrees of freedom) serial link robot arm and an assessment of their performance is given in [9].

The presented methods until here all have the common goal of increasing the accuracy of the robot and its end effector. The objects to be manipulated, especially without rigid coupling, are not part of the considerations. For this reason, only the oscillations occurring at the robot structure can be reduced. In [2] the reduction of shear forces between a suction pad gripper and the object is shown by adjusting the orientation of the end effector while moving. The underlying assumption of the availability of degrees of freedom in the orientation is however not always complied with by the robot or the application. Similiar to the pursued approach in this paper, [3] proofs that the oscillations of a rope-coupled mass attached to a flying drone can be reduced by methods of reinforcement learning.

To extend these results to the oscillation reduction in the positioning of elastically coupled objects with robot manipulators, the contributions of this paper are

1. a structure-independent flatness-based control for handling elastic objects,

2. the comparison of this method with state-of-the-art machine learning.

The remainder of this paper is organized as follows: A brief overview of the theoretical and mathematical background of the presented methods is given in Sec. 2 and their implementation for the specific system at hand is given in Sec. 3. The experimental setup and results are presented in Sec. 4, demonstrating the high potential of the proposed methods. Sec. 5 concludes the paper.

#### 2 Background

First, a brief summary of flatness-based control, reinforcement learning in general and Double Deep Q-Network (Double DQN), the algorithm used within this work, is given.

#### 2.1 Flatness-Based Control

Dynamic systems with the property of differential flatness are characterized by the existence of a flat output y<sup>f</sup> which in the single input single output case allows transformations

$$x = \psi\_x\left(y\_\mathbf{f}, \dot{y}\_\mathbf{f}, \dots, y\_\mathbf{f}^{(k-1)}\right) \quad \text{and} \quad u = \psi\_u\left(y\_\mathbf{f}, \dot{y}\_\mathbf{f}, \dots, y\_\mathbf{f}^{(k)}\right) \tag{1}$$

leading to a representation where the state vector x and the input u can be expressed by means of a flat output y<sup>f</sup> and a finite number of its derivatives up to the order of the the system's relative degree k [6]. While no general method for finding a flat output of a nonlinear dynamic system exists, in the case of linear controllable systems the controllable canonical form allows determining the flat output directly [6]. After a flat output is found, one can use (1) to calculate an input signal u(t) that leads to the desired output trajectory y(t) for the system.

#### 2.2 Reinforcement Learning

In the context of reinforcement learning an agent finds itself in a state S<sup>t</sup> and interacts with its environment through an action A<sup>t</sup> derived by a policy π. Acting on its environment results in a change from state S<sup>t</sup> to St+1 and an observation of a reward Rt+1, valuing the chosen action and new state. The agent's objective is to maximize the discounted return

$$G\_t = R\_{t+1} + \gamma R\_{t+2} + \gamma^2 R\_{t+3} + \dots = \sum\_{k=0}^{\infty} \gamma^k R\_{t+k+1},\tag{2}$$

where γ ∈ [0, 1) describes the importance of future rewards [11]. The true value for a specific state s and action a, under a policy π then becomes

$$Q\_{\pi}(s, a) = \mathbb{E}\_{\pi} \left[ G\_t | S\_t = s, A\_t = a \right] = \mathbb{E}\_{\pi} \left[ \sum\_{k=0}^{\infty} \gamma^k R\_{t+k+1} \Big| S\_t = s, A\_t = a \right], \tag{3}$$

where the optimal value is denoted as Q∗(s, a) = max<sup>π</sup> Qπ(s, a) and the optimal policy is made up of choosing always the highest valued action in each state [12]. In case of large or continuous state spaces, it becomes inadequate to store the Q-value for each discrete time step t separately. Instead, the objective becomes learning a parameterized value function Q(s, a; θ), were θ denotes the parameterization. The function approximator can be linear in its weights θ or nonlinear (e.g. an artificial neural network, ANN) [11]. The proposed approach uses an algorithm from the latter class of methods.

The standard update rule for the parameterized Q-learning algorithm is

$$
\theta\_{t+1} = \theta\_t + \alpha \left[ U\_t - Q(S\_t, A\_t; \theta\_t) \right] \nabla\_{\theta\_t} Q(S\_t, A\_t; \theta\_t), \tag{4}
$$

where α is the step size and the target U<sup>t</sup> is defined by

$$U\_t = R\_{t+1} + \gamma \max\_a Q(S\_{t+1}, a; \theta\_t). \tag{5}$$

In the Double DQN approach from [12], two ANNs are used for function approximation. Both networks share the same network architecture but deviate in their weight vectors, resulting in the target

$$U\_{\rm DoubleDQN,t} = R\_{t+1} + \gamma Q(S\_{t+1}, \arg\max\_a Q(S\_{t+1}, a; \theta\_t); \theta\_t^-). \tag{6}$$

Here, θ<sup>t</sup> parameterizes the online network and θ − t the target network. Inspired by Double Q-Learning [4] the online network is used to select the current action, while the target network estimates its value. Similar to [8] the weights of the target network θ − <sup>t</sup> are updated by or copied from the online network θ<sup>t</sup> every N time steps and are kept constant otherwise. To break correlations between training samples it is beneficial to use experience replay [5], where tuples of observed transitions (St, At, St+1, Rt+1,terminal) are stored in a buffer with size N<sup>B</sup> and sampled uniformly in order to train the network; where 'terminal' describes whether St+1 is a terminal state.

#### 3 Implementation

As motivated in Sec. 1, the considered scenario is a pick and place task where the object is not rigidly but elastically coupled to the tool center point (TCP), e.g. by elastic grippers like suction cups. Therefore, the coupling between the robot's TCP and the object is modeled as a base-excited mass, illustrated in Fig. 1 (a), where the robot structure is assumed to be rigid. To investigate the principal relations, a 1-DoF replacement model is considered for the robot and the TCP.

Thereby y and ˙y denote the objects Cartesian position and velocity, while u

Fig. 1. (a) Base-excited mass and (b) testbed construction.

and ˙u are the TCP position and velocity. Furthermore, c and d denote spring stiffness and damping, which altogether results in the differential equation

$$d\,m\ddot{y} + d\dot{y} + cy = d\dot{u} + cu,\tag{7}$$

with disappearing initial values for y and u and their derivatives. This model is employed for both the flatness-based control and the training of the agent.

#### 3.1 Flatness-Based Control

Equation (7) describes a linear system with a relative degree of one. A transformation into its corresponding state space representation leads to a state space model in controllable canonical form (with y<sup>f</sup> 6= y)

$$
\begin{pmatrix} \dot{y}\_{\text{f}} \\ \ddot{y}\_{\text{f}} \end{pmatrix} = \begin{pmatrix} 0 & 1 \\ -\frac{c}{m} - \frac{d}{m} \end{pmatrix} \begin{pmatrix} y\_{\text{f}} \\ \dot{y}\_{\text{f}} \end{pmatrix} + \begin{pmatrix} 0 \\ 1 \end{pmatrix} u \tag{8}
$$

$$y = \begin{pmatrix} \frac{c}{m} \ \frac{d}{m} \end{pmatrix} \begin{pmatrix} y\_{\text{f}} \\ \dot{y}\_{\text{f}} \end{pmatrix},\tag{9}$$

where y<sup>f</sup> represents a flat output of the system (7). The dependency on time t for states y<sup>f</sup> and ˙y<sup>f</sup> , input u and output y is omitted for a better readability. For the objective of a rest-to-rest motion two approaches are pursued:

## Approach 1 (Neglection of damping): Neglecting d in (7) results to my¨ + cy = cu, (10)

where the physical state y itself already is a flat output and the control input u can be obtained after dividing both sides by c.

Approach 2: Instead of transforming the whole reference trajectory yref for the physical output y into flat coordinates, only the initial and end conditions for the output are transformed. Since the goal is to achieve a rest-to-rest motion for the output y, only the dynamics at start and end of the motion have to be constraint. After transforming initial and end values of the physical trajectory into flat coordinates, a polynomial of degree 9 for the reference trajectory yf,ref in flat coordinates is used. The constraints on derivatives up to order 4 avoid exciting unmodeled robot dynamics. For the transformation of initial as well as end conditions (9) is used to obtain

$$y(T) = y\_T = \frac{c}{m}y\_l(T) + \frac{d}{m}\dot{y}\_l(T) \quad \text{and} \quad \dot{y}(T) = \dot{y}\_T = \frac{c}{m}\dot{y}\_l(T) + \frac{d}{m}\ddot{y}\_l(T) \quad (11)$$

with T denoting the end of motion. In condition (11) the second derivative of y<sup>f</sup> appears, which then can be substituted by the last row in (8)

$$
\dot{y}\_T = \frac{c}{m}\dot{y}\_\mathbf{f}(T) + \frac{d}{m}\left(-\frac{c}{m}y\_\mathbf{f}(T) - \frac{d}{m}\dot{y}\_\mathbf{f}(T) + u(T)\right).
\tag{12}
$$

At the end of motion the mass has to be in rest at a specific position, which leads to the conditions

$$y(T) = y\_T \quad \text{and} \quad \dot{y}(T) = 0 \tag{13}$$

and, in addition, the robot's TCP should simultaneously arrive at the same end position as the mass, leading to

$$u(T) = y(T) = y\_T.\tag{14}$$

Finally, the transformation for position and velocity at time T appears to be

$$y\_{\mathbf{f}}(T) = \frac{m}{c} y\_T \quad \text{and} \quad \dot{y}\_{\mathbf{f}}(T) = 0 \tag{15}$$

while initial values equal zero.

#### 3.2 Double Deep Q-Network

For the learning algorithm a Double Deep Q-Network is used where the agent is trained in a simulated environment using the same mathematical model of (7) as for the flatness-based control approach. The agent's objective is to learn a correction for the reference trajectory yref, which is again a polynomial of degree 9, but in physical coordinates, leading to the position profile

$$p(t) = y\_{\text{ref}}(t) + a(s),\tag{16}$$

where a(s) denotes the agent's action in state s. The agent selects out of 101 equally spaced positions a ranging from −0.03 m to 0.03 m. For the state representation the error in position and velocity between the mass and the reference trajectory

$$\mathbf{e} = \begin{pmatrix} y\_{\text{ref}} - y \\ \dot{y}\_{\text{ref}} - \dot{y} \end{pmatrix} = \begin{pmatrix} e \\ \dot{e} \end{pmatrix} \tag{17}$$

is used, since learning can be sped up if the ANNs inputs have a mean of zero [10]. Further, the reward signal penalizes error in distance, velocity and large changes by the agent through

$$r = -\{e\} - \{\dot{e}\} - 0.5\{a\},\tag{18}$$

where SI-units are removed with the {·} operator. The network consists of three fully connected layers, two inputs, 48 neurons in the hidden layer and 101 neurons in the output layer. Training in a simulated environment led to a discontinuous position profile, which was unsuitable for highly dynamic motions. Hence, a polynomial of degree 19 was fitted to the position profile p(t) altered by the agent. Finally, Table 1 summarizes the most important hyper parameters.


Table 1. Hyper parameters for the Double Deep Q-Network.

### 4 Experiments

The effectiveness of both presented methodologies is studied and verified using a 4-DoF delta robot. Following a short introduction of the experimental setup in Section 4.1, the experimental results and the comparison of approaches are presented in Section 4.2.

#### 4.1 Experimental Setup

Exemplarily for a typical industrial pick and place robot for highly dynamic applications, the delta robot Codian D4-1100 controlled by a standard industrial PLC and servo inverters is selected for an evaluation of the proposed methods for oscillation reduction. Fig. 2 (a) shows the experimental setup including the

Fig. 2. (a) Delta robot with attached test construction, (b) measurement setup detail.

whole robot structure. A detailed view of the TCP-mounted test construction is shown in Fig. 2 (b). To evaluate both methods without further unknown and unmodeled effects, a test construction following the mathematical model was created. It consists of a flat spring with an exchangeable additional mass, designed to be mountable to the TCP of the robot, as shown in Figure 1 (b). To identify the mechanical parameters of the construction, free oscillations test were performed, where the spring stiffness was identified as c = 702 <sup>N</sup> m and the damping as d = 0.135 Ns <sup>m</sup> with a measured mass of m = 0.281 kg. Two acceleration sensors were used: one rigidly connected to the TCP (sensor I) and the other one attached to the mass (sensor II), see Fig. 2 (b).

The influence of different speeds and different masses was investigated. The agent learned its correction a(s) for one motion time T and was also tested for slower reference trajectories. Though the agent was trained with the identified mechanical parameters, it was also tested with varying masses by changing the mass attached to the flat spring. In Summary, 10 movements per time and weight case were recorded. All following results are also transferable to other structures as the approaches are not restricted to a specific hardware or robot type.

#### 4.2 Experimental Results and Comparison of Approaches

The variations of motion time T and mass m were performed using four different methods for calculating the TCP trajectory u(t):


Fig. 3 shows exemplary acceleration signals for all three methods each compared to the solid model for an attached mass of m = 0.281 kg. The highlighted area I indicates the time of motion for the TCP and the area II the time after arriving at the goal position with remaining mass oscillations. The evaluation concentrates on the remaining accelerations after arriving the goal position. Fig. 4 shows the results for the time variation tests as box plots. The agent was trained for a motion time of 0.4 s, mass of 281 g and a linear movement of 0.55 m. The ordinate shows the maximal remaining acceleration after the TCP arrives its end position and the abscissa corresponds to the motion time. Note that the axes scales of Fig. 4 and Fig. 5 are different, since using the solid model (directly using the reference trajectory) leads to higher oscillations. The highest improvement in relation to the solid model of around 85% can be seen for the shortest motion time in Fig. 4. After investigating different motion times, the influence of incorrectly identified masses is examined, for a motion time of 0.4 s in Fig. 5. As opposed to Fig. 4, different masses attached to the bending beam are given on the ordinate. All methods were able to achieve a vibration reduction except for a mass of 98 g, for which the solid model performed better. One possible explanation for such a behavior is that a small mass tends to vibrate rather than a large one. At the same time, the flatness-based control reacts sensitive to deviations of identified parameters with respect to the real parameters [14].

The experiments indicate that both methods lead to a vibration reduction. For the present case, the effect of the spring stiffness is much greater than the damping, the flatness-based control approach with consideration of damping was just slightly superior to the one without it.

Fig. 3. Exemplary acceleration signals in comparison to solid model for (a) Flatnessbased control with damping, (b) FBC without damping and (c) Double DQN.

Fig. 4. Variation of the motion time with a constant mass; (a) FBC damping, (b) FBC, (c) Double DQN and (d) solid model. 10 test motions per box with a mass of 281 g.

Fig. 5. Variation of the mass with a constant motion time; (a) FBC with damping, (b) FBC, (c) Double DQN and (d) solid model. 10 test motions per box.

The tests with respect to incorrectly identified masses showed similar results, however, with deterioration for the lightest mass. Overall, both methods led to similar performances in vibration reduction.

#### 5 Conclusion and Future Work

This paper illustrates two methods for reducing the vibrations of not rigidly coupled objects during highly dynamic robot movements. Especially the object vibrations at the end of the movement are observed and can be reduced with both methods by up to 85 %. This results in a significant increase in placement accuracy and a time saving during object manipulation. It can thus be shown that both flatness-based control and reinforcement learning with the Double DQN are methodically suitable for the aim of vibration reduction of elastically coupled handling objects. This clear improvement was also shown with inaccurately identified object parameters. Both methods are therefore (to a certain extent) robust against parameter uncertainties. It is important to mention that both methods are completely independent of the considered robot structure, since they are based purely on the TCP position and therefore no consideration of the robot dynamics has to be carried out.

Due to the simple underlying linear model of the base-excited mass, there are clear advantages for flatness-based control in terms of implementation effort. For this reason we consider the flatness-based control approach to be superior to the Double DQN, since programming the reinforcement learning algorithm, including the time for tuning hyper parameters and training, took much more time than designing and implementing the control u(t). However, it should be noted that these described advantages decrease with increasing complexity of the model. In addition, it is not guaranteed that a flat output can be defined for more complex models. In this case, flatness-based control is no longer applicable so that the Double DQN approach would be advantageous. In addition, reinforcement learning offers the possibility to be used in combination with convolutional neural networks, enabling it to learn from raw image data [7].

Future work will focus on the comparison of the methods for more complex (non-linear) systems regarding the achievable vibration reduction and the implementation effort. Furthermore, the investigation of adapting the approaches to three-dimensional space curves is still an open question. In this paper a valuebased algorithm of reinforcement learning was used, which is limited to a discrete action space. An actor-critic algorithm with a continuous action space promises further improvements.

#### References


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Correlation between Geometric Component Properties and Physical Parameters of an Aerodynamic Feeding System**

Torge Kolditz<sup>1</sup> , Mirco Wolf and Annika Raatz<sup>1</sup>

1 Institute of Assembly Technology, Leibniz University Hannover, Garbsen 30823 Germany kolditz@match.uni-hannover.de

**Abstract.** In previous research, an aerodynamic feeding system was developed, which autonomously adapts to different components by using a genetic algorithm that controls the physical parameters of the system (e.g. angle of inclination, nozzle pressure). The algorithm starts with two individuals with random values, generated within the boundaries of the parameters set by the user. Due to this, the setting time - the time that passes until a satisfactory orientation rate is reached is hard to predict. The aim of this work is to identify basic interactions of geometric component properties with the physical parameters of the aerodynamic feeding system to determine in which areas of the workspace a satisfactory solution can be expected. By doing so, the initial population of the genetic algorithm can be generated based on certain geometric properties and would therefore no longer be random, presumably reducing setting time.

To identify interactions of component properties and system parameters, exemplary components were developed. They represent relevant single properties that have significant impact on the aerodynamic orientation process. These components were then fed into the aerodynamic orientation process and their behavior was documented. To identify correlations between certain geometric properties and physical parameters of the feeding system, the tests were planned and carried out using Design of Experiments methods. The results of the tests were also used to determine the direct interrelations of said properties and the suitability for aerodynamic orientation.

**Keywords:** Design of Experiments, Genetic Algorithm, Aerodynamic Feeding.

## **1 Introduction**

In the highly automated assembly processes seen in modern production, part feeding plays a vital role. The most common part feeding system used in the industry is the vibratory bowl feeder (VBF) [1, 2]. The essential component of a VBF is the bowl with a spiral track, which is connected to a drive unit, which creates an alternating rotational movement of the bowl using suspension springs and electromagnets [3]. This vibration is used to separate parts from the bulk at the bottom of the bowl and transport them up the spiral track, where traps are used to either reorient or reject components that are not in the desired orientation. While the bowl and the drive components have relatively low costs, the design of the traps and the bowl can be very expensive and timeconsuming [4]. In order to address this problem, a lot of research was done with the aim to decrease the cost for track and trap design using simulation tools [3-8]. However, even with simulation tools at hand, retooling of the orienting devices requires manual labor in the manufacturing and assembly of the specially designed bowls and traps.

To meet the demand for a highly flexible and efficient feeding technology, an aerodynamic feeding system has been developed at the Leibniz University Hannover [9- 12]. The system uses a constant air jet to exert force on the components passing by. Using a genetic algorithm, the system is designed to parametrize itself for an optimal output rate, so that no manual labor or manufacturing is needed for retooling. The working principle of the aerodynamic feeding system and the algorithm for self-parametrization will be outlined in the following.

### **2 The aerodynamic feeding system**

The aerodynamic feeding system presented and used in this work operates with only one air jet, which every component passes. In other work, systems have been presented that use multiple nozzles or air cushions to orient and transport parts [13, 14]. Fig. 1 shows the process of aerodynamic orientation in the described feeding system. It can be seen that the components behave differently, depending on the orientation they have when arriving at the nozzle. Should the workpiece arrive in the wrong orientation, it is turned over by the air jet, as can be seen in Fig. 1 a), whereas it keeps its orientation, if it already arrives in the correct orientation (Fig. 1 b)). The reason for the different behaviors of the component lies in the shape of the workpiece and the distribution of mass in the very same [13].

The shape of the workpiece influences the projected inflow area and the drag coefficient, which determine the lift force transmitted to the workpiece. If the shape of the component is not uniform over the entire length, the force transmission also changes depending on the position of the component above the nozzle. This results in a rotational impulse around the center of gravity of the workpiece. The location of the center of gravity is determined by the distribution of mass in the component. This also influences the dynamic behavior of the part when it is lifted and accelerated by the air flow. In addition to their properties, the behavior of the components in the aerodynamic orientation process is determined by the adjustment of the four physical system parameters *α*, *β*, *v* and *p* seen in Fig. 1. In the exemplary process in Fig. 1, the angle of inclination *α* promotes clockwise rotation of the workpiece and impedes counterclockwise rotation, which supports the successful orientation process. The angle *β* influences the force of gravity acting on the component and determines the impact of the friction between the component and the guiding plane. The nozzle pressure *p* directly affects the magnitude of drag force acting on the workpiece. If it is set too low, the component might not rotate at all, whereas a higher pressure can lead to multiple and unpredictable rotations. Lastly, the component speed *v* determines, how fast a workpiece passes the air jet and therefore how long the drag forces affect it. The parameter can be controlled by adjusting the speed of a conveyor located ahead of the nozzle.

**Fig. 1.** Illustration of the aerodynamic orientation process [9]

In order to identify the effects and interrelations of the system parameters on the orientation process, BUSCH [15] used Design of Experiments (DoE) methods. In his work, BUSCH did not take any geometric or physical properties of the workpieces into account, because only the pneumatic plug seen in Fig. 1 was used as exemplary component. Looking at the working principle of the aerodynamic orientation process, it can be seen that certain component properties like length, diameter, shape or mass and the physical parameters of the aerodynamic feeding system (nozzle pressure, inclination, velocity) influence each other. The relevance of understanding the interaction between the component properties and the system parameters becomes apparent, when looking at the functionality of the genetic algorithm that is used to find a satisfactory set of parameters for the feeding process. The goal of the genetic algorithm (GA) is to find a set of the four physical system parameters *α*, *β*, *v* and *p* that induces a satisfactory orientation rate (e.g. > 95 %). A GA was chosen for this task because the search for a good orientation rate constitutes a non-linear optimization problem with a solution space that is not necessarily continuous [9, 10, 15]. A major disadvantage regarding the convergence and predictability of the algorithm is the random initial population since there is no input correlating with the component at hand. While this assures that the whole workspace of the feeding system is searched for a solution, it also creates a huge solution space. The aim of this work is to determine in which areas of the workspace a satisfactory solution can be expected, depending on certain properties of the workpieces to feed by identifying basic interactions between geometric component properties and the physical system parameters *α*, *β*, *v* and *p*.

### **3 Exemplary components**

To identify the interactions of component properties with the physical system parameters, these properties have to be defined first. The properties are divided into physical and geometrical features. Geometrical features include length, width, height, shape, symmetry and center of gravity. Relevant physical properties are mass, moment of inertia, damping properties and friction with other materials. Since it is not possible to determine the interactions of all of those properties in this work, three geometric properties are selected, that can be varied independently and determined easily. The selected properties are:


The independence of the component properties is an important factor for the DoE method because it allows a design where each experiment represents the change of only one variable without altering the others. As aforementioned, the DoE method was already used in prior work by BUSCH [15] to identify the effects and dependencies of the system parameters on the orientation of a pneumatic plug. The geometric properties were not taken into account, because only one type of plug was used. In this work, DoE is used to evaluate the effects and dependencies of geometric components properties in order to make the results more applicable to different types of workpieces.

The exemplary part used for the trials has to represent the selected properties individually, with as little interferences as possible. The simplest part that can be described with length and diameter is a cylinder. The parameters length and diameter can be varied independently. To achieve eccentricity of the center of gravity of a cylindrical part a hole from one of the front surfaces is included in the design. The eccentricity can be varied by altering the depth and diameter of the hole. At the same time, the hole does not affect the outer shape of the cylinder. Fig. 2 shows a drawing of the component, labeled sleeve in the following, with the parameters *l* (length) and *d* (diameter).

**Fig. 2.** Sketch of the sleeve used as exemplary component

In order to set up a test plan, it is necessary to define the boundaries of the parameters *l* and *d*. Suitable values for the eccentricity *Ξ* also have to be selected. Preliminary tests show that it is very difficult to make a universal statement about the minimum and maximum values a component's property must have to be suitable for aerodynamic orientation. The reasons for this are:


An example for the interaction between separate properties are the length and diameter. The boundaries of each of those properties depend, to some extent, on each other. For example, preliminary testing showed that a sleeve with a length of 12 mm can be oriented, if the diameter is not bigger than 6 mm. Generally the ratio of length and diameter should be greater than two to allow aerodynamic orientation and to avoid uncontrolled behavior of the component (e.g. component tumbling down the guiding edge). Another limiting factor are the constructional characteristics of the feeding system. For example, the maximum flow and therefore the maximum impulse of the air jet is limited by the diameter of the air nozzle. The pressure of the air jet can only be increased until the critical pressure ratio *ß* is reached. After that, the jet is no longer usable for technical applications [13, 14]. For air, the critical ratio is 0.528, which limits the nozzle pressure *p* to 0.89 bar.

Different versions of the exemplary sleeve were printed using a 3D-printer and then fed into the orientation process. A part was evaluated as generally suitable for aerodynamic feeding, if the constant air jet could reorient it. The measurements of the resulting exemplary components for the following experiments are presented in Table 1. The 3Dprinted components are shown in Fig. 3.


**Table 1.** Measurements of exemplary components

**Fig. 3.** 3D-printed exemplary components (sleeves) with component numbers (cf. Table 1)

#### **4 Interactions of component properties and system parameters**

To investigate the interaction of component properties and system parameters, an experimental plan has to be designed. In addition, the target figures of the trials have to be defined. After that, the trials are carried out and the results are evaluated.

**Experimental plan.** To design a proper experimental plan, it is necessary to first determine the input and output parameters of the trials. In the previous section, length, diameter and eccentricity of the components used were selected as component parameters for this work. Additionally, the physical parameters *α*, *β*, *p* and *v* are defined as input parameters.

The aim of the experiments is to find out how to narrow the limits of the physical system parameters depending on the selected geometric component properties length, diameter and eccentricity in order to achieve a high orientation rate. The orientation rate is therefore defined as the only output. It is easy to quantify and has distinct limits (0 % to 100 %). Furthermore, a high orientation rate is the optimization goal of the genetic algorithm, the improvement of which is the goal of this work.

In the previous section, a full factorial test plan for the geometric properties of the components was already drawn up and corresponding values for the upper and lower limits selected through preliminary testing. In order to determine the interactions of the parameters *α*, *β*, *p* and *v* and the component properties with the orientation rate, it is necessary to include the system parameters in the test plan. Table 2 shows the values of the parameters (factors) *α*, *β*, *p* and *v* for each of the components.

The upper and lower factor settings (*xmin* / *xmax*) of the physical system parameters *α*, *β* and *v* are determined by the constructional boundaries of the feeding system. As mentioned before, the boundaries of the pressure *p* are 0 bar and 0.89 bar. Since the components have different weights, as a result of the varying dimensions, the limits for this parameter cannot be generally determined for all components. If the pressure is too low, a component will not be lifted at all, if the pressure is too high, the component will spin multiple times, uncontrollably or even be blown out of the feeding system. To avoid this, additional preliminary tests were carried out to determine the boundaries of the pressure *p*.


**Table 2.** Factor setting for the physical system parameters

The use of a full factorial test plan with four two stage factors would result in a test effort of 16 test runs per workpiece. Accordingly, a total of 128 test runs would have to be carried out. Due to the size and variety of the workpieces, the attached centrifugal feeder and the automated recirculation system cannot be used. Therefore, the components have to be applied to the conveyor (parameter *v*) manually. In order to reduce the testing effort, the full factorial test plan is reduced to a screening plan presented in Table 3. In order to reduce the influence of environmental conditions (temperature, humidity), the screening plan was run twice in the interval of two weeks. The order of the individual test runs (8 runs per workpiece) was randomized. For each test run, 100 workpieces are fed into the orientation process in alternating initial orientation. This means that if no reorientation takes place, the orientation rate is 50 %. If all workpieces that arrive at the nozzle in wrong orientation are reoriented, but none of the workpieces arriving at right orientation are reoriented, the orientation rate is 100 %.


**Table 3.** Screening test plan for each component

**Results.** The results of the two screening trials show a high consistency regarding the orientation rates measured for each parameter configuration. Since the presentation of the results of all experiments would go beyond the scope of this work, the analysis of the test results is presented in the following. In order to determine the influence of a system parameter on the orientation rate *O*, the main effects of each parameter have to be calculated. Table 4 shows the main effects of all components together with the specific properties of the components. For example, the main effect *E<sup>v</sup>* for the second component indicates that increasing the velocity *v* from 55 to 75 m/min is associated with an average increase of the orientation rate of 15.9 %. To calculate the effect of the parameter *v* for the first component, the results of the 16 test runs (two screenings with eight test runs each) done with the component are split in two halves, one representing the test runs with high velocity *v* and the other representing those with a low velocity. The effect is then calculated by subtracting the mean orientation rate of the half with lower velocity from the mean orientation rate of the half with higher velocity (equation 1). In addition to the main effects, secondary effects - interactions between the factors - play a role, but will not be discussed in more detail here.

$$\mathbf{E}\_{\rm v,comp1} = \frac{\sum \mathbf{O}\_{\rm v+}}{8} \mathbf{\cdot} \frac{\sum \mathbf{O}\_{\rm v-}}{8} = \mathbf{15.9} \text{ } \text{\textquotedblleft} \mathbf{0} \text{\textquotedblright} \tag{1}$$


**Table 4.** Main effects of the physical parameters on the orientation rate

Derived from the analysis of the main effects and the secondary effects, recommendations for the selection of the system parameters can be made. These are given in Table 5. In the table, a "+" represents a high value of the component property or system parameter and a "-"represents a low value. For example, the combination "*l* -", "*d* +" and "*Ξ +*" represents a component with a small length, a high diameter and a high eccentricity of the center of gravity. For such a component, a low inclination angle *α*, a high inclination angle *β* and a high velocity *v* should be selected to achieve higher orientation rates. It is expected that by narrowing the range of those parameters in an area, where high orientation rates are expected to be more likely, the setting time of the system can be reduced. The pressure *p* has to be considered with caution, as the upper and lower limits of the pressure are not constant between the different components in contrast to the limits of the other three system parameters.


**Table 5.** Recommendation for the selection of system parameters dependent on component properties

#### **5 Interaction of component properties and orientation rate**

In the previous section, interactions between component properties and system parameters were investigated. In the following section, the results of the tests carried out are further analyzed to find out how the properties of the components and the orientation rate are related. The results will be used to help predict the suitability of a workpiece for aerodynamic orientation.

In order to evaluate the influence of the component properties length, diameter and eccentricity, again, a quality characteristic must be selected. As described in section 4, the orientation rate is the most suitable characteristic to determine the suitability for aerodynamic orientation. In the tests, orientation rates were determined for 64 different combinations of component properties and parameter sets. From the results of these tests, different variants of the quality characteristic orientation rate *O* can be derived. On the one hand, the mean value of the orientation rate for each exemplary component *Omean* is selected as quality characteristic. The value indicates the average orientation rate that is achieved with one component. On the other hand, the maximum value, which indicates the highest orientation rate accomplished with an exemplary component *Omax* is selected. The values are given in Table 6.


**Table 6.** Mean and maximum orientation rate of 16 test runs for each component

The values in Table 6 are used to determine the effects of the individual component properties on the orientation rate. To do so, the main effects of parameters and their interrelations are calculated analogously to section 4 (cf. equation 1). These factors are calculated using the mean orientation rate of all test runs with one component as quality criteria on the one hand and the maximum orientation reached with a component on the other hand. The results are illustrated in Fig. 4, sorted by size of the effects. It can be seen, that the factors have similar values, regardless of whether they are calculated from the mean or maximum values of the orientation rate in Table 6.

**Fig. 4.** Visualization of the effects of component properties and their interrelations (factors) on the orientation rate sorted by size

The results indicate that the length of a component has a large positive impact on the orientation rate. With an effect of 7.9 % calculated from the average orientation rate and 28.8 % calculated from the maximum orientation rate, the length has the largest effect of the three properties investigated in this work. Therefore, it can be concluded that longer components are potentially more suitable for aerodynamic orientation than shorter components. This may be because, on the one hand, a longer component is in the air jet for a longer period of time, when moving at the same velocity as a smaller component and is therefore more strongly influenced by the air jet. On the other hand, the lever with which the lifting force of the air jet rotates the component around the center of gravity becomes longer. It has been observed in the preliminary tests that longer components tend to have a more stable movement behavior on the guiding plane and edge.

The second largest effect is the two-factor interaction of length and diameter, which is positive. That indicates that components with a very large or a very small length to diameter ratio are less suitable for the orientation process. The ratio of the components used in the test runs varies from *l*/*d* = 2 to *l*/*d* = 25. From the results illustrated in Fig. 4 it can be deduced that both extreme characteristics achieve a lower orientation rate in comparison to the components with ratios like *l*/*d* = 4 or *l*/*d* = 12.5.

The effect of the diameter is negative, which indicates that the orientation rate diminishes as the diameter of the component increases. This could already be observed in the preliminary testing and is attributed to the Bernoulli Effect, which reduces the lifting force applied to a component. When the sleeve passes over the nozzle, it creates a gap between itself and the nozzle guiding edge. Therefore, the flow path of the air jet is narrowed which results in a higher velocity of the fluid. This, according to Bernoulli's principle, leads to a decrease of pressure between the sleeve and the guiding edge ultimately resulting in the part being dragged down [14-16]. This effect becomes stronger with increasing component diameter.

The eccentricity of the center of gravity has a positive influence on the orientation rate. This has already been investigated by LORENZ [13] and is confirmed by the tests carried out for this work. An interesting observation concerns the interaction of eccentricity and length, which is negative. From this it can be concluded that an increase of the eccentricity has a higher positive impact on the orientation rate for short components than for long components. The two-factor interaction of eccentricity and diameter is very small and therefore not further discussed.

The three-factor interaction of length, diameter and eccentricity is positive. The Interpretation of this interaction is difficult because for some combinations of geometric properties it contradicts the other effects. For example, according to the three-factorinteraction, a long component with a high eccentricity of the center of gravity must have a high diameter to reach a high orientation rate. This contradicts the clearly negative effect of the diameter. At the same time, the three-factor interaction can be used to determine what a suitable component should look like. It is difficult to make a generalized statement about the geometric properties of a suitable workpiece, since the suitability is also influenced by factors not considered in this work, like nozzle diameter and shape. Yet, the results obtained by the analysis of the test data indicate that workpieces with a higher length (cf. *El*) and eccentricity (cf. *EΞ*) as well as a medium sized diameter (cf. *El∙d* and *El∙d∙Ξ*) are more suitable for aerodynamic orientation in the sense that a high orientation rate can be reached at different parameter configurations.

#### **6 Conclusion and Outlook**

The aim of this work was to determine the interaction between component properties and the setting of the physical parameters of the aerodynamic feeding system. In addition, it was to be found out how the change of individual component properties affects the suitability for aerodynamic orientation.

In order to achieve these objectives, relevant component properties were defined in a first step. By conducting preliminary tests, suitable sample components were designed and subsequently manufactured using a 3D printer. Based on eight sample component geometries, a screening test plan was drawn up in order to be able to represent the influencing factors well. A total of 128 test runs were carried out with 16 test runs per component geometry. In each test run, 100 components were fed to the aerodynamic orientation process and the resulting orientation rate was measured.

By evaluating the test results it was possible to create a matrix which makes it possible to generate recommendations for the setting of the physical system parameters *α*, *β*, *v* and *p,* depending on given component properties such as length, diameter and eccentricity of the center of gravity (cf. Table 5). The findings can be used to narrow the range of the system parameters in order to reduce the search space and therefore the setting time of the genetic algorithm. Furthermore, positive and negative effects of certain component properties on the orientation rate and thus on the general suitability for aerodynamic orientation could also be determined (cf. Fig. 4). A higher suitability means that a high orientation rate can be achieved easier, which in return also decreases the setting time of the genetic algorithm.

The interactions and correlations derived from the tests carried out in this work are based on the aforementioned set of 128 test runs (two screenings with 64 test runs each). The goal of future work is to increase the accuracy of the recommendation matrix by repeating the test to reduce the influence of statistical errors. Also, the tests should be carried out using a different test plan. Using the screening test plan, it was assumed, that the interrelations are linear in between the boundary values. One reason for using a genetic algorithm for setting the system parameters is the non-linearity of the optimization problem. Therefore, to better model the interrelations in the entire workspace of the aerodynamic feeding system, a new test plan with a Central-Composite-Design (CCD) will be generated and carried out. The results of the CCD test plan will also be used to refine and validate the interrelation of component properties and suitability for aerodynamic orientation as shown in section 5.

Another objective of future work will be to investigate if the results generated with the exemplary components in this work are applicable to components of entirely different geometry. The sleeves used in this work were designed to only represent the effect of an eccentric center of gravity, length and diameter on the orientation process. With the cylindrical outer contour, a very simple geometry was chosen. The aim of future work must therefore also be to reproduce the interrelationships of the component properties with the orientation process for more complex components. Two further component geometries have already been developed for this purpose and can be seen in Fig. 5. With those components, the success of the aerodynamic orientation process is not only based on the eccentricity of the center of gravity but also on the varying projected inflow surface of the components.

**Fig. 5.** 3D-printed exemplary components for further testing

### **7 Acknowledgements**

The results presented in this paper were obtained within the project "Model-based increase of the flexibility and robustness of an aerodynamic part feeding system for highperformance assembly (project number: 243351293). The authors would like to thank the German Research Foundation (DFG) for their financial support of this project.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.