Johannes Masino

**Band 82**

J. Masino

Road Condition Estimation with Vehicle Based Sensors

# **Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors**

Johannes Masino

## **Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors**

#### **Karlsruher Schriftenreihe Fahrzeugsystemtechnik Band 82**

Herausgeber

#### **FAST Institut für Fahrzeugsystemtechnik** Prof. Dr. rer. nat. Frank Gauterin Prof. Dr.-Ing. Marcus Geimer Prof. Dr.-Ing. Peter Gratzfeld Prof. Dr.-Ing. Frank Henning

Das Institut für Fahrzeugsystemtechnik besteht aus den Teilinstituten Bahnsystemtechnik, Fahrzeugtechnik, Leichtbautechnologie und Mobile Arbeitsmaschinen.

Eine Übersicht aller bisher in dieser Schriftenreihe erschienenen Bände finden Sie am Ende des Buchs.

# **Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors**

by Johannes Masino

Karlsruher Institut für Technologie Institut für Fahrzeugsystemtechnik

Road Condition Estimation with Data Mining Methods using Vehicle Based Sensors

Zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften von der KIT-Fakultät für Maschinenbau des Karlsruher Instituts für Technologie (KIT) genehmigte Dissertation

von M.Sc. Johannes Masino

Tag der mündlichen Prüfung: 15. November 2019 Referent: Prof. Dr. rer. nat. Frank Gauterin Korreferent: PD Dr.-Ing. Markus Reischl

**Impressum**

Karlsruher Institut für Technologie (KIT) KIT Scientific Publishing Straße am Forum 2 D-76131 Karlsruhe

KIT Scientific Publishing is a registered trademark of Karlsruhe Institute of Technology. Reprint using the book cover is not allowed.

www.ksp.kit.edu

*This document – excluding the cover, pictures and graphs – is licensed under a Creative Commons Attribution-Share Alike 4.0 International License (CC BY-SA 4.0): https://creativecommons.org/licenses/by-sa/4.0/deed.en*

*The cover page is licensed under a Creative Commons Attribution-No Derivatives 4.0 International License (CC BY-ND 4.0): https://creativecommons.org/licenses/by-nd/4.0/deed.en*

Print on Demand 2021 – Gedruckt auf FSC-zertifiziertem Papier

ISSN 1869-6058 ISBN 978-3-7315-1004-8 DOI 10.5445/KSP/1000104538

# **Vorwort des Herausgebers**

Die Fahrzeugtechnik ist kontinuierlich Veränderungen unterworfen. Klimawandel, die Verknappung einiger für Fahrzeugbau und –betrieb benötigter Rohstoffe, globaler Wettbewerb, gesellschaftlicher Wandel und das rapide Wachstum großer Städte erfordern neue Mobilitätslösungen, die vielfach eine Neudefinition des Fahrzeugs erforderlich machen. Die Forderungen nach Steigerung der Energieeffizienz, Emissionsreduktion, erhöhter Fahr- und Arbeitssicherheit, Benutzerfreundlichkeit und angemessenen Kosten finden ihre Antworten nicht aus der singulären Verbesserung einzelner technischer Elemente, sondern benötigen Systemverständnis und eine domänenübergreifende Optimierung der Lösungen.

Hierzu will die Karlsruher Schriftenreihe für Fahrzeugsystemtechnik einen Beitrag leisten. Für die Fahrzeuggattungen Pkw, Nfz, Mobile Arbeitsmaschinen und Bahnfahrzeuge werden Forschungsarbeiten vorgestellt, die Fahrzeugsystemtechnik auf vier Ebenen beleuchten: das Fahrzeug als komplexes mechatronisches System, die Fahrer-Fahrzeug-Interaktion, das Fahrzeug in Verkehr und Infrastruktur sowie das Fahrzeug in Gesellschaft und Umwelt.

Art und Zustand von Fahrbahnen sind bedeutsam für Verkehrssicherheit, ungehinderten Verkehrs-fluss und Energiebedarf der Fahrzeuge. Üblicherweise werden in Deutschland Straßen und Brückenbauwerke in festen Intervallen von vier bzw. sechs Jahren auf Ihren Erhaltungszustand über-prüft. Dazu werden spezielle, aufwändige und kostenintensive Prüfverfahren eingesetzt. Neben den hohen Kosten der Prüfung ist ein erheblicher Nachteil die lange Zeitdauer zwischen aufeinanderfolgenden Prüfungen, so dass das Wissen über den Fahrbahnzustand meist nicht aktuell ist. Eine flächendeckende Karte der verbauten Fahrbahntypen gibt es gegenwärtig nicht.

Die vorliegende Arbeit will hier Abhilfe schaffen. Es wird ein Verfahren vorgeschlagen, bei dem eine einfach in Kraftfahrzeuge integrierbare Messtechnik über die Erfassung von Geräuschen und Schwingungen, die durch den Reifen-Fahrbahn-Rollkontakt am Fahrzeug entstehen, Art und Zustand der Fahrbahnen ermittelt. Dazu werden aus den Signalen charakteristische Muster aus Werten geeigneter Kenngrößen mittels verschiedener Methoden ermittelt. Es findet dabei eine Fusion der Daten verschiedener Fahrzeuge statt, was die Erkennungsrate deutlich erhöht und damit den messtechnischen Aufwand am einzelnen Fahrzeug gering hält. Eine mit dem vorgeschlagenen Messsystem ausgestattete Fahrzeugflotte kann stochastisch die gesamte Oberfläche aller Fahrbahnen erfassen und liefert so über die Zeit eine vollständige Karte.

Im Ergebnis liefert dies die aktuelle Information über Fahrbahnart und –zustand sowie zu Änderungen über der Zeit. Es erlaubt ein situationsgerechtes Fahrbahnerhaltungsmanagement und die Erstellung von Zustandsprognosen, mit denen sich Instandhaltungskosten und volkswirtschaftliche Folgekosten durch Staus und Umleitungen reduzieren lassen. Die gewonnene Datenbasis erlaubt weiterhin die Verbesserung von 3D-Karten für assistiertes oder automatisches Fahren und damit eine genauere Vorhersage von Bremsweg, Kurvengrenzgeschwindigkeit, Rollwiderstand oder auch Rollgeräuschemission.

Karlsruhe, 15.11.2019 *Prof. Dr. rer. nat. Frank Gauterin*

# **Abstract**

The dissertation provides novel methods to process inertial sensor and acoustic sensor data for road condition estimation and monitoring with application in vehicles, which serve as sensor platforms. Furthermore, methods are introduced to combine the results from various vehicles for a more reliable estimation.

A systematic, continuous, and comprehensive monitoring of existing roads is becoming increasingly important. Yet, current practice is laborious and time-consuming as most steps of the process are done manually.

A novel measuring device for vehicles is developed to autonomously acquire a large set of real world data on public roads. In addition, an extended simulation approach is presented to record data under controlled conditions to investigate the impact of parameter variations. Furthermore, the methods for data processing are implemented in a user friendly Matlab toolbox to allow an automatic evaluation.

The quantitative and visualized results show that the estimation of road unevenness and roughness accord with the ground truth. The prediction accuracy of road attributes, such as potholes, and road surfaces, such as damaged concrete or asphalt, yield 81.0 % and 96.1 % on average. The investigation of combination strategies shows that multiple vehicle fusion incorporating the precision matrix from the vehicle classifiers increase the confidence level of the final estimation. The normalized mean square error of the regression function to estimate road roughness is 4.5<sup>h</sup> on average.

Overall, the developed methods can improve or partially substitute the current practice to evaluate the road infrastructure. This work contributes concepts and methods to allow an automatic and comprehensive road condition monitoring. The data processing methods can run on electronic control units of modern vehicles, which are equipped with corresponding sensors. However, the presented system consisting of the developed measuring device and Matlab toolbox can also be applied on vehicle fleets as it is.

# **Kurzfassung**

Diese Arbeit bietet neuartige Methoden zur Verarbeitung von Inertialsensorund akustischen Sensordaten für die Straßenzustandsabschätzung und überwachung mit Anwendung in Fahrzeugen, die als Sensorplattform dienen. Darüber hinaus werden Methoden eingeführt, um die Ergebnisse mehrerer Fahrzeuge für eine robuste Schätzung zu kombinieren.

Eine systematische, kontinuierliche und umfassende Überwachung bestehender Straßen wird immer wichtiger. Die derzeitige Praxis ist jedoch mühsam und zeitaufwendig, da die meisten Schritte des Prozesses manuell durchgeführt werden.

Ein neuartiges Messgerät für Fahrzeuge wurde entwickelt, um autonom eine große Menge an realen Daten auf öffentlichen Straßen zu erfassen. Darüber hinaus wird ein Simulationsansatz vorgestellt, um Daten unter kontrollierten Bedingungen aufzuzeichnen und die Auswirkungen von Parametervariationen zu untersuchen. Des Weiteren sind die Methoden zur Datenverarbeitung in einer benutzerfreundlichen Matlab-Toolbox implementiert, um eine automatische Auswertung zu ermöglichen.

Die quantitativen und visualisierten Ergebnisse zeigen, dass die Abschätzung von Straßenunebenheiten und -rauheiten mit den tatsächlichen Gegebenheiten übereinstimmen. Die Vorhersagegenauigkeit von Attributen der Straße wie Schlaglöchern und Straßenoberflächen wie beschädigtem Beton oder Asphalt liegt im Durchschnitt bei 81,0 % bzw. 96,1 %. Die Untersuchung von Kombinationsstrategien zeigt, dass die Fusion mehrerer Fahrzeuge mit der Präzisionsmatrix der Fahrzeugklassifikatoren die Sicherheit der endgültigen Schätzung erhöht. Der normalisierte mittlere quadratische Fehler der Regressionsfunktion zur Schätzung der Straßenrauhigkeit beträgt im Durchschnitt 4,5h.

Insgesamt können die entwickelten Methoden die bisherigen Verfahren zur Bewertung der Straßeninfrastruktur verbessern oder teilweise ersetzen. Die vorliegende Arbeit trägt Konzepte und Methoden bei, um eine automatische und umfassende Straßenzustandsüberwachung zu ermöglichen. Die Datenverarbeitung kann auf elektronischen Steuergeräten moderner Fahrzeuge erfolgen, die mit entsprechenden Sensoren ausgestattet sind. Das vorgestellte System, vor allem bestehend aus einem entwickelten Messgerät und einer Matlab-Toolbox, kann aber auch problemlos in seiner jetzigen Form auf Fahrzeugflotten angewendet werden.

# **Acronyms**




# **Symbols**

Data mining and processing







# **Contents**





# **1 Introduction**

*Outline:* This chapter gives the motivation in the first section. Section 1.2 explains the causes for a degradation of road condition, the effects of road defects, and shows the necessity of a novel real-time road condition monitoring. The current process for road condition monitoring is described in Section 1.3 and the following Section 1.4 reviews related literature of automated and mobile road condition monitoring. Based on these sections open problems are deduced (Section 1.5) and the aim and research question of this dissertation are given in Section 1.6

# **1.1 Motivation**

The wealth pyramids of modern economies and studies show that the foundation of prosperity, productivity, growth, and social wellbeing are infrastructure and mobility [1].

In Germany, road transport is the main source of traffic volume and the backbone of transport infrastructure for citizens and the economy [2]. However, the state of the road network deteriorates noticeably in many regions of Germany [3]. The road infrastructure is subject to permanent stress and needs to be repaired or renewed in order to ensure the substance and utility value. The degradation of the roadway condition results in a multitude of undesirable factors which burden vehicle users and third parties.

Current practice of road condition monitoring is laborious and time-consuming as most steps of the process are done manually [4]. The steps involve the collection of data, the identification and assessment of defects based on the collected data, and the calculation of a road condition index for road segments with a specific length. Few countries have expensive specialized vehicles for an automated data collection whereas the monitoring is only scheduled in fixed intervals of 1 to 4 years [5]. In most of the countries, the data is collected manually by inspectors, who pass the road infrastructure and record road defects, their location, type of distress, and severity level, among others. The evaluation of the road segments and defects is often subjective as it is based on the experience of the technician.

At current practice defects are unlikely to be comprehensively identified in early stages, when repairs are more cost-efficient. Due to the decline in building new roads and lack of financial resources, a systematic, continuous and comprehensive monitoring of existing roads is becoming increasingly important. With this approach, the functionality of the roads could be maintained in an economical and environmentally compatible way.

# **1.2 Road condition**

The road profile is mainly characterized by the texture wavelength λ [6]. The wavelengths physically illustrate the different lengths of periodically repeating structures of the profile. The road texture is categorized as microtexture with a wavelength of less than <sup>0</sup>.5 mm, as macrotexture with wavelengths from 0.5 to 50 mm, and mega texture with wavelenghts from 50 mm to 0.5 m [6]. The profile texture with wavelengths from 0.5 m to 50 m is referred to as unevenness.

The road profile is overlaid by road discontinuities, which are inadvertent, for example pavement failures, or due to pavement design, such as expansion joints, railway, and tramway crossings, manholes or speed bumps [7].

Pavement failures can be summarized as cracks, surface deformations, and surface defects [8]. Surface deformations mainly include ruts and shoving, surface defects represent bleeding, raveling, and potholes. The repair procedure is to determine the type of pavement failure, determine the cause, fix the cause and to perform the repair [8].

## **1.2.1 Causes for road defects**

The main criteria to be met by the road infrastructure are [9]


With a systematic maintenance, the life span of asphalt layers is 10 to 15 years and 15 to 20 years for binding layers. Concrete surfaces can last up to 30 years and base layers 40 to 50 years [10].

The life span of a road is limited by either structural aspects and the aging of the construction materials or by outside influences, such as the weather condition and traffic load. For example, binders naturally lose their binding and restoring force over time. Furthermore, the wear of the road occurs due to tire road contact.

One of the most important external factors influencing the state of the roads is heavy-load vehicle traffic and the axle load in particular. The axle load is the proportion of the total weight attributed to one axle of a vehicle. The total weight of the vehicle consists of its own and loaded weight. A long-term test showed that a linear increase in axle load leads to a damage of the road in fourth power [11]. Since 2002 the traffic volume of commercial road haulage has increased by more than 50 % in Germany, from 1,454 to 2,245 million tonnes annually [3].

In addition to traffic loads, the season is the main factor, which influences the road condition. Particularly long-lasting heat periods lead to damages on asphalt and concrete roads. For example, the concrete plate stretches due to solar radiation and reduces the transverse joints. The plates can break if there is not enough space and cause a ripple, which is referred to as blow up [12]. Asphalt has relaxation capacity because of bitumen and expands with constant stress and increasing temperature and softer bitumen. The results are deformations and cracks on the surface [12].

The condition of the road infrastructure can significantly deteriorate after a period of very cold weather. An aged or cracked road surface allows the entry of water into pores and cracks. The water freezes by frost into ice crystals, which enlarge the road damages. Repeated melting and solidification and traffic loads result in the formation of potholes [12].

### **1.2.2 Effects of road defects**

Many studies have investigated the influences of bad road conditions [13–15], the increase of tire road noise was proven by the experiments of the author [16]. The influences can be summarized as


These influences result in even more undesired effects. For example, a high level of tire road noise leads to general annoyance, sleep disturbances, or speech interference [17, 18]. Furthermore, it increases the costs for noise abatement measures, such as sound-absorbing barriers [19]. Tire road noise also yields to a decrease in value of houses and land close to noisy road transport infrastructure [20–22].

Road defects not only decrease the driving comfort and affect the behavior of the vehicle [23] but can also cause damage to the vehicle. A deep pothole might deform the wheel rim and constant higher load impacts the wear of the rubber mounts. Particularly bicycle riders are vulnerable from rough surfaces, which increase the danger of fall or injury.

An increase of rolling resistance leads to higher carbon dioxide emission and vehicle operational costs, such as the fuel consumption, and lower range of electric vehicles [24]. Moreover, if the road condition exceeds a critical value due to delayed detection of road defects, the erosion of the road substance accelerates. Subsequently, the road needs a complete renewal and the costs increase over the life cycle of the road [25]. A renewal often yields to a complete roadblock and to traffic jams. Overall, repairs result in high economic costs due to the time spent in traffic jams and diversions.

# **1.3 State-of-the-art of road condition monitoring**

The current applied processes for road condition monitoring were analyzed based on manuals from transportation departments around the world, e.g. [26– 32]. The condition monitoring process is often based on several steps, which involve the data collection, defect identification, defect assessment, and the calculation of a road condition index [4]. However, the execution and details of each step differ from country to country.

The road infrastructure is often scaled in segments of a specific length, which usually ranges from 10 to 100 meters among the countries.

The data collection is mainly performed either manually by accredited surveyors, who either walk or drive along the roadway, or automatically by specialized vehicles, which are equipped with various sensors. Latter vehicles create high costs of purchase and maintenance and only few countries use such vehicles [33]. In Germany for example, the Federal Highway Research Institute performs measurement campaigns with these specialized vehicles on interurban roads in fixed intervals of multiple years. However, their use is limited to the primary road network and a manual data acquisition is performed in communes.

The types of defects under consideration of most of the countries are mainly different types of cracks, rutting, shoving, distorting, potholes, raveling, and bleeding. The defects are assessed manually on site, are based on pictures and data from measurement vehicles or through an index calculated on the basis of the severity and type of defect. Finally, an overall road condition index is calculated based on the defects and their severity or density, which differs in weights and calculation across the transportation departments.

The different equations for this index from various states in the USA are evaluated in [34]. Based on the assessment of the quality of the road segments, a priority rating for repair is assigned by the inspectors.

In many countries the international roughness index (IRI), a slope statistic in m/km of road segments is calculated with road profilometers. There are different International Quality Levels (IQL-1 up to IQL-3), but even the most accurate IQL-1 systems typically report the roughness at 10 to 20 m intervals, while IQL-3 systems only report at intervals greater than 100 m. Hence, for the purpose of identifying defects the IRI is of limited use, as further information would have to be collected in some other way, before specific reconstruction planning can take place.

Interviews with experts were conducted to collect qualitative data and information about the current status and development of road condition monitoring that could not be found in the literature [198]. The focus of the interviews was on the area of Karlsruhe, Germany. An expert should have special knowledge which, in contrast to general knowledge, involves complex, integrated knowledge and is also constitutively related to the pursuit of a profession [35]. Three experts with the described special knowledge were selected<sup>1</sup> .

They stated, that the costs for a specialised measurement vehicle are approximately 120 Euro per kilometre [198]. A visual road condition assessment is conducted from inspectors every two years in Karlsruhe and the road segments are classified into five groups of quality [198]. However, the approach in small municipalities is different. Thus there are communes, which do not measure their state of the road at all [198]. There is a clear need for a continuous monitoring system, since environmental influences can cause short-term changes of the road state, especially after a strong winter or a hot summer [198]. A real time road condition monitoring system would be a huge step forward [198].

## **1.4 Survey of related research**

## **1.4.1 Overview**

Researchers have addressed the problem of a manual, expensive, and discontinuous process to monitor the road infrastructure and present concepts and methods for a mobile data acquisition and automated processing to estimate the road roughness or to detect road hazards. Figure 1.1 gives an overview and the structure of relevant research for this chapter.

<sup>1</sup> The experts are Univ.-Prof. Dr. h.c. Ralf Roos from Karlsruhe Instiute of Technology, Prof. Dr.-Ing. Markus Stöckner from Karlsruhe University of Applied Sciences, and Manfred Geiger from the civil engineering department of the city of Karlsruhe.

The measurement sensors, which have been applied to monitor the road infrastructure are mainly


RCM: road condition monitoring

RP: road profilometers, C: camera, AS: acoustic sensor, IS: inertial sensor

RI: roughness index

Figure 1.1: Automated and mobile road condition monitoring (RCM) research structure and the amount of research papers in parentheses. The first level indicates the sensor, the second level the method and the third level the output. The output is mainly either road attributes, e.g. defects, or a roughness index (RI), such as the IRI. Nodes annotated with 'other' for the data processing method level means, that the method was not clearly specified in the paper or the method can not be assigned to the classes PM or ML.

The numbers of research papers indicate, that there has been a lot of research with cameras and mainly machine learning (ML) methods were applied. There only exists few studies which investigate acoustic sensors to monitor the road infrastructure. More publications can be found for road profilometers and inertial sensors.

ML: machine learning, PM: physical model

With latter sensors, physical model or heuristic models with ML algorithms can be applied to estimate the road condition. The majority of studies, which use inertial sensors and ML, output a roughness index and only few predict defects. The literature applying inertial sensors and ML is reviewed in more detail in subsection 1.4.5 with focuses on specific subjects, such as labeling of training data or feature selection.

The level of detail of the output for an automated road condition monitoring can be distinguished into in three categories [36]. The categories are



Table 1.1: Evaluation of sensors based on previous literature.

Table 1.1 summarizes the sensors, considered defects, level of detail, investment, and operating costs and the effort to install the sensor and operate the measurement system, based on the reviewed literature.

## **1.4.2 Application of road profilometers for road condition monitoring**

Mobile road profilometers can be distinguished in tactile and laser profilometers. The longitudinal profile analyzer (LPA) serves as an example for mobile tactile profilometers [37]. The measurement system consists of one or two trailers with one suspension and wheel each, which is in continuous contact with the road. A movable transverse beam transmits the vertical movement of the wheels to the data acquisition unit. The road profile from this system is used as reference for other road condition methods, for example in [37, 38].

3D laser road profiling systems were investigated in [39–41]. They are able to identify most types of road defects and vehicles with such systems can travel with speeds of up to 100 km/h [42]. Data processing techniques are also of high standard and researchers were able to drastically reduce computation times for 3D dynamic image optimization in recent years [43]. However, the costs of equipping one vehicle with laser line profiling technology are quoted up to 220,000 USD [44] and they have to be handled by trained operators to ensure proper functionality [45]. Furthermore, objects on the road, such as sand, gravel, salt or snow negatively influence the results and lead the system to misidentify road defects.

### **1.4.3 Application of cameras for road condition monitoring**

There has been a lot of research in using cameras for an automated road condition monitoring. A recent literature review to camera road condition monitoring gives an overview of previous research focused on camera systems [36].

Most of the proposed methods operate on 2D images, e.g. [5, 46–60], while there are few 3D reconstruction methods that utilize the notion of stereovision for detecting road defects or road markings [61–67]. 3D reconstruction methods offer the possibility to reconstruct the surface using images from various video cameras. A stereo camera is utilized in the Magic Body Control from Mercedes-Benz, first introduced in 2013. The camera is located behind the windscreen just in front of the rear view mirror and scans the road surface ahead to detect changes in the road elevation. With this information, the suspension is actively adjusted to reduce vehicle body movements [68].

Research with cameras was mainly performed to detect road hazards but not to estimate the road roughness. Table 1.2 summarizes the most cited papers with the applied method, type of object recognition and level of detail, focusing on road condition monitoring with camera and an automated data processing.


Table 1.2: Overview of literature employing camera based methods.

Machine learning multiclassifier algorithms nowadays are the most common methods for object detection in images. They allow simultaneous segmentation and recognition of several objects in images [69–71]. Since the defects can be seen in the image and labeled by an annotator, mainly supervised learning with either artificial neural networks (ANNs) or support vector machines (SVMs) were applied. A semantic texton forest (STF) was applied in [36], as it can use various features for segmentation, namely texture, layout and context.

The results of the reviewed studies show, that cameras in combination with machine learning methods are able to precisely detect and mark defects within images. They are inexpensive and modern vehicles with special packages are equipped with cameras ex works so that comprehensive coverage of the road network can be achieved.

However, cameras have general features that present definite disadvantages. Firstly, their use is limited to decent weather conditions, since rain, fog, mist, or snow can make capturing images of sufficient quality impossible. A second limitation is given by the speed at which vehicles can travel without the vibration and suspension movements affecting the sharpness of the image. The authors of a recent paper conceded that they had to drive at speeds between 10 and 15 km/h to keep unexpected vibrations of their vehicle at a minimum to not affect the quality of their data [36]. One solution is to use cameras that possess high enough resolution, frame rates, and image stabilizing to warrant accurate measurement results at higher speeds. However this leads to increasing costs for hardware and reveals the trade-off between the quality of the picture and the corresponding amount of data storage required.

Throughout the corresponding literature, a general problem with ML is the algorithm training phase. Even the most recent papers have to rely on supervised training and manual labeling of training data. This is quite time-inefficient and thus limits the amount of valuable training data available. Hence ways to label data more efficiently, increasing the amount of training data, and thus improving the accuracy of the algorithms or the application of unsupervised learning should be of high interest for future research in this field.

## **1.4.4 Application of acoustic sensors for road condition monitoring**

One limitation of indirect methods to estimate the road roughness based on accelerometer measurements is the limitation in frequency range. Highfrequency excitations from the road surface, for example the macrotexture, are filtered by the tire and suspension, which act as a low-pass filter [45]. However, passing from smooth to coarse asphalt results in audible changes in level and frequency distribution of sound, which can be captured by microphones attached to the vehicle [45]. Results from previous research indicate that methods based on sound have to compensate the current vehicle velocity. Further limitation are the effect of snow and water on the road on the sound measurements.

Microphones in or outside the vehicle body have the disadvantage that noise from the environment, such as other vehicles and reflections, influence the data. An alternative method to analyze the sound pressure in the tire cavity was developed in [72]. The road surface oscillates the tire and the vibrations generate sound waves in the tire cavity, which is a reverberation room and insulated from the environment.

The measurements of the studies investigating tire road noise show a good reproducibility. Moreover, they demonstrate the potential to distinguish between road surfaces with different characteristics, especially various macrotextures [73–75]. However, an acoustic sensor in the tire cavity has only been used under specific driving conditions and no automated data processing has been developed.

## **1.4.5 Application of inertial sensors for road condition monitoring**

#### **Inertial sensors and physical models**

In computer simulation, researchers have investigated the possibility to estimate the road profile based on physical vehicle models but have not considered to detect road defects or other road features. The estimation in computer simulation is mainly based on a half-car models with inertial sensor data from the sprung and unsprung masses of the vehicle model [76, 77].

The inversion from the vehicle model, where usually the road profile serves as input and the accelerations as output, is a common method to derive the road profile [199]. The applied methods with real data are mainly based on sliding mode observers [78, 79] or filters [38, 45, 80–83], such as Kalman filters. The basis of these methods with real data is mainly a quarter car model.

Overall, the results of these reviewed studies show that the methods give a good estimation of the road profile. However, the accuracy of the estimation is prone to changes in velocity. Moreover, the transportation departments concentrate on detecting type of defects, which requires an additional step in the data processing of the methods employing inertial sensors and physical models. In contrast, methods using ML can directly output the type of defect.

#### **Inertial sensors and heuristic models**

Machine learning approaches for road condition estimation have been mainly applied for real driving rather than in computer simulation. The methods can be distinguished in their output, which is either a roughness index or defects. Table 1.3 gives an overview of recent literature employing inertial sensors and a heuristic model.

In computer simulation, studies applied ANNs to estimate the road profile with data obtained from vehicle dynamics models [84–88], mainly half-car models.

A method to detect potholes based on acceleration sensors and global position system (GPS) for localization is presented in [89]. Seven taxis were equipped with the sensor system, which managed to cover 2,492 distinct kilometers


Table 1.3: Overview of literature employing inertial sensors and heuristic models.

during ten days. Various signal processing filters were applied for data analysis a misclassification rate of 0.2 % was reported. Research in this field with similar filters was extended by [44], who equipped 100 taxis in the Shenzhen urban area with low-end accelerometers and GPS devices to obtain vibration pattern, location, and vehicle velocity. The advantages of machine learning approaches over physical models was pointed out by [85]:


A comparison of different machine learning models, namely ANN, SVM, and random forest (RF) was performed by [88].

Research in road condition monitoring with inertial sensor data is becoming increasingly popular. The main advantages are that modern vehicles are equipped with inertial sensors and in contrast to camera systems, this method is not limited to decent weather and daytime light conditions.

The limitations are that each vehicle has to be calibrated to compare the results [80]. Moreover, tire road contact on the area to be measured has to be ensured. In contrast, cameras can detect defects between or next to the two vehicle lanes. However, a large number of surveying vehicles with inertial sensors might diminish the disadvantage of the limited measured area and collect data of the most interesting area of the road. The level of detail is limited and the determination of the attributes of road defects has not been presented.

Another issue that arises is the labeling of training data acquired with inertial sensor. In contrast, labeling methods for camera data presented in research are mostly coherent and automated. The presented annotating methods of inertial sensor data range from describing the road features into a microphone while acquiring training data, or trying to match road images with the corresponding inertial sensor data, which is performed offline.

**Review of comprehensiveness and generalization** Comprehensiveness and generalization of a method are of utmost importance, as the aim of an automated road condition monitoring is to be able to work in all driving and environment conditions. To test the generalization, experiments have to be comprehensive in the variation of vehicles they use, road segments to be measured and the amount of kilometers of road network covered. Many papers only use computer simulated data to test the accuracy of their algorithm and others use only specific vehicles on fixed test tracks, a few kilometers of selected road stretches, or consider only few and specific outputs.

For example, Eriksson, Girod, Hull, *et al.* [89] equipped seven identical taxis, which covered the road infrastructure in Boston over a ten day period. Menant, Martin, Meignen, *et al.* [97] used 12 similar probe vehicles on a 1000 km road network in the north of France to measure the IRI and Chen, Lu, Tan, *et al.* [44] deployed a road surface monitoring system on 100 taxis in Shenzhen but did not provide information of the type of vehicle. Seraj, Zwaag, Dilo, *et al.* [96] used 5 different vehicle types, including hatchbacks, superminis, and an SUV and covered a total of 45.9 distinct kilometers in the testing phase. However, they did not report if and how they combined the measurement data from the different vehicles or if they trained each vehicle separately.

**Review of methods for feature selection** Supervised ML approaches mainly classify based on features rather than time series. Nitsche, Stütz, Kammer, *et al.* [88] state, that "In order to use the proposed machine learning models effectively, one of the most important steps is to extract expressive features from the collected data".

Only few papers provide insight on their feature selection. Tai, Chan, and Hsu [93] modify some of the 59 roughness parameters for measuring road surfaces presented in [98]. Finally, they use a total of six distinct features extracted from their input data, which includes the acceleration values in three axes, instantaneous speed, and position. The features are the mean speed and the mean, range, standard deviation, maximum, and minimum of the acceleration for each axis.

Perttunen, Mazhelis, Cong, *et al.* [94] use sliding windows of 2 s length with a 0.5 s slide to extract features from the acceleration signals. They utilize the backwards feature selection algorithm of PRTools [99] to select optimal speed scaled and non-speed scaled feature sets.

Nitsche, Stütz, Kammer, *et al.* [88] used a total of 35 features as input for the models, gained from the longitudinal, lateral and vertical acceleration as well as the angular speed of each wheel. Among the selected features were the mean, standard deviation, range, and the mean of the short time energy as well as the power spectral densities of the accelerations, and angular speed of both rear wheels, respectively.

To prepare data for the classification phase Seraj, Zwaag, Dilo, *et al.* [96] compute features from time domain, transformation in frequency domain and wavelet decomposition for a total of 18 different features.

Laubis, Simko, and Schuller [90] and Laubis, Simko, Schuller, *et al.* [91] use the same features based on GPS, 3-axis accelerometer, and 3-axis gyroscope measurements for both papers to estimate road roughness. They perform a continuous wavelet transformation to analyze the frequency content of the acceleration and gyroscope features. Other aggregation functions used are the mean, range from minimum to maximum, standard deviation, variance, and root mean square (RMS). Overall, most studies use 35 or even more features for finally estimating the IRI or a similar roughness index.

**Review of performance and stability of methods** Amongst the different machine learning methods, support vector machines (SVMs) have shown to be most popular recently and outperform other methods, e.g. random forest (RF) and artificial neural network (ANN) [88]. For example, Nitsche, Stütz, Kammer, *et al.* [88] state that "Choosing a set of representative features is important for each model, but the RF and SVM are found to be more stable than the ANN with respect to the quantity of features. For example, a drastic reduction of features (from 35 to as low as 5) does not significantly alter the results for the former two, while a neural network is not able to capture the structure of the data efficiently in this case.".

**Review of methods for labeling training data** An important step in ensuring the accuracy of any selected machine learning algorithm is providing sufficient amounts of correctly labeled training data. Without a large enough pool of training data to learn from, the performance of any algorithm is severely hindered.

While some papers do not describe their labeling technique for real tests and others rely entirely on computer simulations, where the data can be automatically annotated, a couple of methods for labeling real world data have been used with varying success.

Eriksson, Girod, Hull, *et al.* [89] have used a computer in the vehicle and a trained annotator, who sits in the passenger seat of the vehicle, presses keyboard keys corresponding to predefined defect classes when they occur. However, they also used a set of loosely labeled data, where the types and rough frequency is known but not the number and location of the defects.

Tai, Chan, and Hsu [93] use a voice recorder to label the training data during collection with motorcycles in Taiwan. They chose to do this to minimize the required interaction of the motorcycle driver, but had to concede high inaccuracy and subsequently had to develop a very elaborate relabeling process to make their training data usable for the algorithm they used.

Perttunen, Mazhelis, Cong, *et al.* [94] covered 25 km of road in one drive capturing video data next to the accelerometer and gyroscope readings. They preprocessed global position system (GPS) and acceleration signals and plotted a spectrogram of the sum of the band energies of accelerometer signals, superimposing speed in the same figure.

Two independent annotators then produced two label files by analyzing the video data obtained during the drive. This method of training data labeling is rather complicated and even the authors concede that labeling data from video is time consuming and error prone work.

In the work from Seraj, Zwaag, Dilo, *et al.* [96], labeling was performed by one person sitting in the passenger seat using the microphone and the camera of the smart phone. They took various drives over the same road segment at different speeds and hitting the defect in different angles, with the annotator loudly mentioning the type of defect as the vehicle approached it. A software was used to transcript all voice labels with accuracy in the order of milliseconds.

Problems arose when synchronizing the subtitle file with the smartphone data timestamp and lags between the labeled segments of data and the actual defect was noticed. As a result almost all labels had to be manually corrected.

**Review of output metric** As shown in Table 1.3, the most common output metric is the road roughness, especially the international roughness index (IRI). However, as mentioned in Section 1.3 the IRI can only give a generally assessment of the quality of a road segment but the knowledge of the severity and location of specific defects, such as potholes, is more important for a road condition monitoring.

Few publications distinguish between different types of defects and only consider three or less classes. Furthermore, no study investigates to differentiate between different types of materials, such as concrete or asphalt.

# **1.5 Open questions**

The presented literature shows, that road profilometers are too expensive to equip various vehicles with and usually have to be attached outside the vehicle. Hereby, they are prone to environmental disturbances, such as snow or water.

Camera systems for road condition monitoring have been extensively studied. Further improvements are to precisely measure the attributes of defects and to estimate the road profile. With improvements of the hardware, also this task will be solved soon. Furthermore, fundamental research in unsupervised and reinforcement learning in computer vision might replace the time consuming labeling while using supervised learning.

Acoustic sensors are rarely investigated in terms of comprehensive and automated road condition monitoring. However, they seem to be a good option to monitor road infrastructure and even to estimate the macrotexture of the road profile. Previous literature has used acoustic sensors to measure the tire road noise, which is also dependent on the macrotexture. An acoustic sensor outside of the vehicle has many disadvantages, mainly the negative influence from background noise, reflections, among others. However, a dynamic pressure sensor in the tire cavity is robust against disturbing noises. So far, research investigated the tire cavity sound on test benches or test vehicles under controlled conditions. Moreover, the focus was on the correlation with conventional tire road noise measurement procedures, such as the close-proximity method. There is no concept or method to comprehensively and automatically apply a tire cavity sound measurement system for road condition monitoring. This implies the development of a measuring device, which is inexpensive and can be installed on vehicles without a special permit. Furthermore, a method is needed, which analyzes the data from the tire cavity sound sensor in all driving and environment conditions.

Inertial sensors for road condition monitoring compensate for various disadvantages of camera systems. They are robust against the environment conditions and measure the ride comfort, which is mainly dependent on the road roughness.

Previous presented methods to estimate the road roughness, e.g. the IRI, have several disadvantages. Physical models use special vehicles with the knowledge of the suspension characteristics to acquire the vehicle dynamic data. The road roughness is then derived with an inverse vehicle model. This method is prone to a change of the vehicle vibration behavior and needs additional sensors at the unsprung masses besides an inertial sensor in the vehicle body. Recent literature presents a machine learning (ML) approach with one sensor in the vehicle body to train vehicles with ground truth data acquired with profilometers. However, this method involves a costly training process and is mostly limited to road surfaces, which have been trained. A method to estimate the road roughness based on a vehicle inertial sensor, which is not limited to specialized vehicles or does not involve a training process, needs to be investigated.

The research on road object detection with inertial sensors has still room for improvements. Studies using computer simulations mainly focus on the estimate of road profiles and do not classify defects, which is an important input for road condition monitoring. Furthermore, ML methods, which are applied for real test drives should also be tested and verified in computer simulations. A new simulation approach taking these points into account, might be the basis for further research on this topic under controlled conditions to analyze the influence of specific parameters or vehicle behavior on the performance of ML methods.

For real drive measurements, the research gap involves the acquisition of data, especially the process of labeling trainings data. Moreover, a method to investigate and evaluate the features to enable a robust and computational efficient classification has to be introduced. Previous literature has used many features, without giving more insight why they were selected. Too many features increases the risk of over-fitting and decreases the generalization ability. The classification methods so far have been trained and tested mainly under limited conditions, e.g. specific velocities, road conditions, road areas, specific vehicles. Furthermore, a method to combine the output from several vehicles has not been presented.

# **1.6 Aim of this work and research objectives**

The aim of this work is to induce methods to estimate the road infrastructure condition with vehicle based sensors and models for an efficient and automatic data processing. It presents the development of a novel measurement system for the acquisition of both tire cavity sound and inertial sensor data of the vehicle body. Furthermore, a novel simulation approach is developed, which enables to test ML methods and vehicle dynamics models with varying parameters. The first research question is addressed to the inertial sensor: *How can we process the inertial sensor data in the vehicle body to provide a road roughness index without training and special vehicles?*

The second research question takes the acoustic sensor in the vehicle tire into account: *How can we estimate the road roughness with tire cavity sound data under various driving and environment conditions from various vehicles?*

Besides estimating road roughness, the further research question arise: *How can we process inertial sensor data from the vehicle body to detect road attributes, such as potholes?*

Lastly, a research question is: *How can we combine the output from various vehicles to improve the prediction accuracy?*

To address these research questions this work aims to


# **2 Fundamentals**

*Outline:* This chapter describes and explains existing methods, which are necessary for the development of the new concept and fundamental for the development of new methods for road condition monitoring with vehicle based sensors in this work. Section 2.1 defines the coordinate axis and systems of the vehicle. Section 2.2 describes the vibration of the vehicle and the tires and is important to comprehend the selection of vehicle sensors for this work and the derivation of vehicle models. On-board vehicles sensors are discussed in Section 2.3. Vehicle and road models, which are used for the novel simulation approach, are described in Section 2.4 and data mining methods for the developed data processing in Section 2.5. Section 2.6 describes performance indices for the validation of the developed measuring devices in this work.

## **2.1 Coordinate systems**

According to [100], an axis system has three mutually orthogonal directions, to which the *X*, *Y* and *Z* axis are assigned. The standard is based on a right-hand system. A coordinate system consists of an axis system and an origin. The earth-fixed axis system *X*<sup>E</sup> , *Y*<sup>E</sup> , *Z*<sup>E</sup> is connected to the location and the plane spanned by *X*<sup>E</sup> and *Y*<sup>E</sup> is perpendicularly to the direction of gravity g. The road plane *X*R, *Y*R, representative for the road surface, is within the tire contact area. In the case of an uneven road there is a different road plane for each tire contact area with a specific road profile. The vehicle coordinate system is based on the vehicle axis system and the origin lies in the vehicle reference point, as exemplarily shown in Figure 2.1. The vehicle reference point is fixed in the frame work of the sprung mass of the vehicle. In the vehicle axis system, the *X*<sup>V</sup> -axis is directed forward and parallel to the vehicle longitudinal center plane, if the vehicle is at rest. The *Y*<sup>V</sup> -axis is perpendicular to the vehicle longitudinal center line and points to the left, the *Z*<sup>V</sup> axis points upwards. The vehicle reference point can be set at various points whereas the usual place is

Figure 2.1: Vehicle with three translational and three rotational degrees of freedom. The reference point of the vehicle axis system X<sup>V</sup> , Y<sup>V</sup> , Z<sup>V</sup> with the angles ψ<sup>V</sup> , φ<sup>V</sup> , and θ<sup>V</sup> is placed in the center of gravity of the vehicle (CG<sup>V</sup> ). One of four wheel axis systems X<sup>W</sup> <sup>4</sup>, Y<sup>W</sup> <sup>4</sup>, Z<sup>W</sup> <sup>4</sup> is defined with the angles θ<sup>W</sup> <sup>4</sup>, ψ<sup>W</sup> <sup>4</sup>, γ<sup>W</sup> <sup>4</sup>.

the center of gravity of the sprung mass (CG<sup>V</sup> ). The longitudinal acceleration *a*<sup>x</sup> is the component of the vehicle acceleration in *X*<sup>V</sup> direction, the lateral acceleration *a*<sup>y</sup> in *Y*<sup>V</sup> direction and the *a*<sup>z</sup> in *Z*<sup>V</sup> direction. The vehicle velocity v<sup>V</sup> is the velocity of the vehicle reference point, with which the vehicle moves relative to the earth-fixed coordinate system. The yaw angle <sup>ψ</sup><sup>V</sup> is around the *<sup>Z</sup>*<sup>V</sup> axis, the pitch angle <sup>θ</sup><sup>V</sup> around the *<sup>Y</sup>*<sup>V</sup> axis and the roll angle <sup>φ</sup><sup>V</sup> around the *X*<sup>V</sup> axis. The pitch and roll angle is not measured relative to the road surface but to the earth-fixed coordinate system. Accordingly, a vehicle in stand on an inclined road surface also has a non-zero pitch angle. The roll velocity <sup>ω</sup><sup>x</sup> is the x-component of the vehicle angular velocity <sup>ω</sup>, the pitch velocity <sup>ω</sup><sup>y</sup> represents the y-component and the yaw velocity <sup>ω</sup><sup>z</sup> the z-component.

The wheel axis system *X*<sup>W</sup> , *Y*<sup>W</sup> , *Z*<sup>W</sup> for each of the four wheels act in the wheel center of gravity CG<sup>W</sup> and they are defined according to the vehicle axis system. The accelerations are denoted by *a*<sup>W</sup> <sup>x</sup> , *a*<sup>W</sup> <sup>y</sup> , and *a*<sup>W</sup> <sup>z</sup> and the angles are defined as the wheel rotation angle <sup>θ</sup><sup>W</sup> , wheel position angle <sup>ψ</sup><sup>W</sup> , and wheel camber angle <sup>γ</sup><sup>W</sup> .

# **2.2 Vehicle vibration**

## **2.2.1 Overview**

Resulting forces from road irregularities induce an excitation of the vehicle oscillation system [101]. Since road irregularities occur as excitation with different amplitude and wavelength at irregular intervals, the road can be described as a stochastic vehicle excitation. Aside the road surface, dynamic roll and pitch movements during longitudinal and lateral dynamic maneuvers as well as internal excitations from power train and wheel generate forces [102].

Furthermore, environmental influences can have an impact on the vibration behavior of the vehicle. For example, the eambient temperature influences the viscoelasticity and rubber friction [103] and snow or water on the road influences the tire road contact. Moreover, the aging and chemical process of the materials of the vehicle change the vibration behavior [104, 105].

### **2.2.2 Tire road contact**

The footprint of the tires is the only direct contact of a motor vehicle to the roadway. They are responsible to carry the vehicle load, to ensure the transmission of force between the vehicle and the road and to compensate road irregularities. The two major physical effects responsible for tire road friction are adhesive friction and friction from hysteresis due to the viscoelastic properties of the tire [102]. In addition, tire characteristics should ensure safety in different weather conditions and driving situations, achieve a high degree of economic efficiency due to a low rolling resistance and long service life and be mechanically and acoustically comfortable with regard to noise, vibration, and harshness (NVH). In order to meet these requirements, the tire can be optimized through the tread pattern design, rubber compound, contour, and construction, and taking into account possible interactions of the components [106].

The tire belt behaves as a membrane and vibrates due to the excitation from the road surface. Pavement defects, pavement facilities, such as bridge joints, and rough road surfaces cause the tire profile to vibrate and subsequently the tire belt and the tire sidewall [7]. These vibrations lead to radiation of sound waves in the tire cavity and to the environment. In addition, the sound is transmitted as structure-borne noise over the rim into the interior of the vehicle. The sound emission to the environment is mainly responsible for the tire road sound, which has long been a problem for industry and research to reduce [106].

### **2.2.3 Vehicle body vibration**

The mass, stiffness and damping distribution in the structure and the vibration excitation of the structure is responsible for vibration behavior of a vehicle [107]. The vehicle is a multibody vertical dynamic system with various degrees of freedoms (DOFs). The objectives of safe vertical dynamic characteristics are mainly low vehicle body accelerations, low roll and pitch motions, low dynamic wheel load fluctuations as well as a load independent vehicle vibration behavior [108]. The vertical forces essentially consist of spring and damper forces, which ensure the support of the vehicle body relative to the suspension system and limit the movements of the vehicle relative to the roadway [102]. Today, usually coil and air springs are installed in vehicles.

Table 2.1 shows several vibration phenomena that are caused by the road surface [102]. Other vibration phenomena, such as bucking, are caused by imbalance or power train.


Table 2.1: Influences of road surface on ride comfort [102].

## **2.3 On-board vehicle sensors**

Nowadays, vehicles are equipped and delivered ex factory with various sensors [109]. Electronic control units (ECUs) and sensors exchange data through the controller area network (CAN) bus, which allows the micro-controllers and other devices to communicate in almost real-time. The purpose of the sensors and ECUs are mainly to increase the safety of the vehicle passengers, the driving comfort, to improve the in-vehicle entertainment, and to control the entire operation of the vehicle, such as engine control, transmission control. Almost all vehicles today feature safety systems, such as anti-lock braking system (ABS) and electronic stability control (ESC), which require a yaw rate sensor, steering angle sensor, longitudinal and lateral accelerometer, and wheel speed sensors. More sophisticated automotive systems, such as adaptive cruise control (ACC), vehicle dynamics control (VDC), magic body control (MBC), need information from additional sensors on-board, such as stereo camera, pitch and roll rate sensors, vertical accelerometers, and radar.

Another evolution of car sensing technology is the vehicle-to-everything (V2X) communication, which includes vehicle-to-vehicle (V2V) and vehicleto-infrastructure (V2I) communication among others. It allows applications, such as platooning, road works warning, or forward collision warning.

The availability of sensors on-board greatly depends on the automotive packages. Since various sensors are already on-board in modern vehicles to measure the operating state of the vehicle as well as its environment, novel applications arise [109]. These are, for example weather and environmental sensing to develop local micro-climate urban emission models [110, 111], driver behavior characterization [112], or road safety to identify dangerous road network portions from ABS or emergency braking or to estimate the road quality, which is discussed in this work.

While the basic function of the systems of different original equipment manufacturers (OEMs) are similar as well as the employed sensors, the exact name of the automotive packages and the utilized sensors slightly vary between OEMs. For example, the package for an active suspension is called Active Body Control by Mercedes Benz. Continuously Controlled Chassis Concept by Volvo and Dynamic Chassis Control by Volkswagen adjusts the characteristics of the damping system. Furthermore, the details of the sensors are often not publicly available due to the intellectual properties policy of the companies and needs to be gathered through various press releases about novel automotive packages.

## **2.4 Vertical vehicle dynamics and road models**

#### **2.4.1 Overview of models for vertical vehicle dynamics**

Models provide the possibility to investigate and analyze a specific behavior under defined and controlled conditions. The characteristics of multibody vertical dynamic systems can be determined by vehicle models, such as a quarter car, bicycle or full car model [107]. Thereby, the road profile serves as the input for the vehicle model and should be modeled as a real road profile.

#### **2.4.2 Vehicle models**

The mechanical vibration is a more or less periodic motion of a mass, which transforms kinetic energy to potential energy and vice versa. The Lagrange method can be applied to find the equations of motion of the vehicle model [107]. The Lagrange equation applied for vehicle models is

$$\frac{d}{dt}\frac{\delta(K-U)}{\delta\dot{q}\_i} - \frac{\delta(K-U)}{\delta q\_i} + \frac{\delta D}{\delta \dot{q}\_i} = F\_i, \qquad i = 1, 2, \dots, n \tag{2.1}$$

where *K* is the kinetic energy, *U* is the potential energy of system *i*, and *D* is the dissipation function, which incorporates the damping of the system. *F*<sup>i</sup> is the force, which is applied on the mass *m*<sup>i</sup> , *q*<sup>i</sup> are the generalized coordinates, *q*˙<sup>i</sup> are the generalized velocities and *n* the number of DOFs. The finite element method as an alternative to model a vehicle can be more accurate but needs more computation time. It is mainly applied for specific and complex parts of the vehicle, such as the tire.

A vertical dynamic model of a vehicle mainly consists of masses, springs and dampers. Thereby, the mass stores kinetic energy and the spring potential energy. The damper serves as the dissipative element. The sprung mass *m*<sup>s</sup> of the vehicle is the mass, which is supported by the suspension [100]. The suspension consists of springs, shock absorber, control arms, among others. However, in vibration models often only springs and dampers are considered. The unsprung mass *m*<sup>u</sup> is defined as the mass, which is not supported by the suspension but by the tire [100]. At a specific working point of the spring, the spring stiffness *k*<sup>s</sup> is defined as the change in force to deflect the spring divided by the change in deflection *q*. In contrast, the damping force is proportional to the velocity *q*Û.

For vehicle models usually a constant spring stiffness and damping coefficient is assumed. With these assumptions, the kinetic energy *K*, potential energy *U*, and Rayleigh dissipation function *D* are

$$K = \frac{1}{2} \sum\_{i=1}^{n} \sum\_{j=1}^{n} \dot{\mathbf{x}}\_{i} h\_{ij} \dot{\mathbf{x}}\_{j},\tag{2.2a}$$

$$U = \frac{1}{2} \sum\_{i=1}^{n} \sum\_{j=1}^{n} \mathbf{x}\_{i} k\_{ij} \mathbf{x}\_{j},\tag{2.2b}$$

$$D = \frac{1}{2} \sum\_{i=1}^{n} \sum\_{j=1}^{n} \dot{\mathbf{x}}\_{i} c\_{ij} \dot{\mathbf{x}}\_{j}. \tag{2.2c}$$

Figure 2.2 shows a quarter car model, which is most employed to investigate the vibration behavior of a vehicle [107]. Thereby, the tire stiffness is represented by a spring with spring stiffness *k*<sup>t</sup> .

Figure 2.2: Quarter car model.

With (2.2a), (2.2b), and (2.2c) and employing (2.1), we get the differential equations of the motion:

$$\begin{aligned} \ddot{z}\_s m\_s &= -c\_s(\dot{z}\_s - \dot{z}\_\mu) - k\_s(z\_s - z\_\mu), \\ \ddot{z}\_{\mu} m\_\mu &= c\_s(\dot{z}\_s - \dot{z}\_\mu) + k\_s(z\_s - z\_\mu) + k\_t(r\_R - z\_\mu). \end{aligned}$$

#### **2.4.3 Road models**

A road profile for a quarter car model or various individual road profiles for half car or full car models have to be modeled. Hereby, the road profile for simulation is supposed to represent a real road profile. In general, the unevenness density of a real road profile decreases with an increase in spatial frequency.

Figure 2.3: Power spectral density of road profiles.

The power spectral density (PSD) of a road profile is defined in [113] as

$$G\_{d}(\Omega) = \begin{cases} G\_{d}(\Omega\_{0}) \cdot \left(\frac{\Omega}{\Omega\_{0}}\right)^{-n\_{1}} & \text{if } \Omega \le \frac{1}{2\pi} \\ G\_{d}(\Omega\_{0}) \cdot \left(\frac{\Omega}{\Omega\_{0}}\right)^{-n\_{2}} & \text{if } \Omega > \frac{1}{2\pi} \end{cases},\tag{2.4}$$

where <sup>Ω</sup> <sup>=</sup> <sup>2</sup>π/*l*<sup>R</sup> is the angular spatial frequency and *<sup>l</sup>*<sup>R</sup> the wavelength of the unevenness.

Usually, *<sup>n</sup>*<sup>1</sup> <sup>=</sup> 2, *<sup>n</sup>*<sup>2</sup> <sup>=</sup> <sup>1</sup>.5, <sup>Ω</sup><sup>0</sup> <sup>=</sup> <sup>1</sup> and the degree of roughness *<sup>G</sup>*d(Ω0) can be obtained from [113] and are shown in Table 2.2 for different road classes. The unit of *<sup>G</sup>*d(Ω0) is <sup>10</sup>−6*<sup>m</sup>* 3 and the unit of Ω<sup>0</sup> is *rad*/*m*.

Figure 2.3 shows the power spectral density for four roads with different values for degree of roughness. Hereby, road class F represents a very uneven road and A a new asphalt in very high quality.

A periodic road profile can be generated with a Fourier series, the sum of individual sine waves

$$r\_R(\mathbf{x}) = \sum\_{i=0}^{N} \hat{p}\_i \cdot \sin(\frac{2\pi}{l\_R} \cdot i \cdot \mathbf{x} + e\_i),$$

where *<sup>e</sup>*<sup>i</sup> as a random phase angle uniformly distributed between <sup>0</sup> and <sup>2</sup><sup>π</sup> and *i* the running index. A random profile to fit a PSD described in (2.4) can be generated with a setting from the literature, e.g. from [114, 115].

Figure 2.4: Road profile generated with (2.5) with four different values for degrees of roughness.

With *p*ˆ<sup>i</sup> = p <sup>2</sup>π ·∆Ω · *<sup>G</sup>*d(Ω) and with the frequency resolution ∆Ω.

$$r\_R(\mathbf{x}) = \sum\_{i=0}^{N} \sqrt{2 \cdot \Delta \Omega \cdot G\_d(i \cdot \Delta \Omega)} \cdot \sin(2\pi \cdot i \cdot \mathbf{x} \cdot \Delta \Omega + e\_i). \tag{2.5}$$

Figure 2.4 shows road profiles generated with (2.5). The characteristics of the PSD of these road profiles are similar to the PSD in Figure 2.3.

#### **2.4.4 State space**

The differential equations of the motion from the vehicle model with the road model as input can be transformed into a state-space representation, which is often applied for multiple input multiple output systems. State-space equations consists of a set of input, output and state variables. Latter variables present the status of the system. The advantage is that


For a linear, time-invariant model with *n*<sup>u</sup> inputs, *n*<sup>y</sup> outputs, and *n*<sup>x</sup> states, the state-space model is

$$\begin{aligned} \dot{\mathbf{x}} &= A\_S \mathbf{x} + B\_I \boldsymbol{\mu}, \\ \mathbf{y} &= C\_O \mathbf{x} + D\_F \boldsymbol{\mu}, \end{aligned}$$

where *<sup>A</sup>*<sup>S</sup> is the state matrix with dimension *<sup>A</sup>*<sup>S</sup> <sup>∈</sup> <sup>R</sup> <sup>n</sup>x×n<sup>x</sup> and *B*<sup>I</sup> is the input matrix with *<sup>B</sup>*<sup>I</sup> <sup>∈</sup> <sup>R</sup> <sup>n</sup>x×n<sup>u</sup> , which describes the properties of the system. The output is determined by *<sup>C</sup>*<sup>O</sup> as the output matrix with *<sup>C</sup>*<sup>O</sup> <sup>∈</sup> <sup>R</sup> ny×n<sup>x</sup> , and *D*<sup>F</sup> as the feedthrough matrix with *<sup>D</sup>*<sup>F</sup> <sup>∈</sup> <sup>R</sup> <sup>n</sup>y×n<sup>u</sup> . *<sup>u</sup>* <sup>∈</sup> <sup>R</sup> <sup>n</sup><sup>u</sup> is the input vector, *<sup>x</sup>* <sup>∈</sup> <sup>R</sup> nu the state vector and <sup>y</sup> <sup>∈</sup> <sup>R</sup> ny the output vector.

*n*<sup>x</sup> is the minimum number of state variables needed to describe the system, which responses to the initial condition *<sup>x</sup>*(0) and a set of inputs *<sup>u</sup>*(*t*) for *<sup>t</sup>* <sup>∈</sup> <sup>R</sup> >0

.

For the state-space model of the quarter car model in Figure 2.2 with the input *u* = *r*<sup>R</sup> and state vector and output vector

$$\mathbf{x} = \begin{bmatrix} z\_s \\ \dot{z}\_s \\ z\_u \\ \dot{z}\_u \end{bmatrix}, \quad \mathbf{y} = \begin{bmatrix} \ddot{z}\_s \\ \ddot{z}\_u \end{bmatrix},$$

the corresponding matrices are

$$A\_S = \begin{bmatrix} 0 & 1 & 0 & 0\\ -\frac{k\_s}{m\_s} & -\frac{c\_s}{m\_s} & \frac{k\_s}{m\_s} & \frac{c\_s}{m\_s} \\ 0 & 0 & 0 & 1 \\ \frac{k\_s}{m\_u} & \frac{c\_s}{m\_u} & -\frac{k\_t + k\_s}{m\_u} & -\frac{c\_s}{m\_u} \end{bmatrix}, \quad B\_I = \begin{bmatrix} 0 \\ 0 \\ 0 \\ 0 \\ \frac{k\_L}{m\_u} \end{bmatrix},$$

$$C\_O = \begin{bmatrix} -\frac{k\_s}{m\_s} & -\frac{c\_s}{m\_s} & \frac{k\_s}{m\_s} & \frac{c\_s}{m\_s} \\ \frac{k\_s}{m\_u} & \frac{c\_s}{m\_u} & -\frac{k\_t + k\_s}{m\_u} & -\frac{c\_s}{m\_u} \end{bmatrix}, \quad D\_F = \begin{bmatrix} 0 \\ \frac{k\_L}{m\_u} \end{bmatrix}.$$

## **2.5 Data Mining**

#### **2.5.1 Overview**

There are mainly three types of machine learning tasks, which are supervised learning, unsupervised learning and reinforcement learning [116]. There is a lot of research and there are several approaches and algorithms of all the three machine learning tasks. A detailed discussion and weighting of all possible methods of machine learning and the combination of those is beyond this work.

However, the main goal of this study is to classify and estimate the road condition based on features extracted from sensor data, which are physically comprehensible. Furthermore, it should be possible to evaluate the accuracy of the classifier. These points lead us to apply a data mining method, which is based on supervised learning. The advantages of supervised learning are that less training data is needed and the output is comprehensible. The main disadvantage is the costly labeling of data to train and test the models. Latter drawback can be narrowed down by using data, which are automatically acquired and labeled in a simulation.

The steps of the methods based on supervised learning, which are applied in this work, are shown in Figure 2.5.

Figure 2.5: Overview of methods to predict classes or a metric based on time series.

Features are extracted from time series acquired from sensors to reduce the dimension. The dimension of the feature vector can be reduced through feature selection (subsection 2.5.4) and feature aggregation (subsection 2.5.5). The classification (subsection 2.5.2) and regression model (subsection 2.5.3) needs a training phase to find a function to separate the classes or to find the relationship between the dependent variable and the explanatory variables. From testing and training phases, performance measures can be calculated to estimate the generalization (subsection 2.5.6).

The input matrix for the classification or regression problem is matrix *X* with *<sup>X</sup>* <sup>∈</sup> <sup>R</sup> h×n , which consists of *n* various features and *h* feature vectors x<sup>i</sup> with <sup>x</sup><sup>i</sup> <sup>∈</sup> <sup>R</sup> n . The input matrix *X* is defined as

$$X = \begin{bmatrix} \boldsymbol{\chi}\_{11} & \boldsymbol{\chi}\_{12} & \cdots & \boldsymbol{\chi}\_{1n} \\ \boldsymbol{\chi}\_{21} & \boldsymbol{\chi}\_{22} & \cdots & \boldsymbol{\chi}\_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ \boldsymbol{\chi}\_{h1} & \boldsymbol{\chi}\_{h2} & \cdots & \boldsymbol{\chi}\_{hn} \end{bmatrix} = \begin{bmatrix} \mathbf{x}\_1^T \\ \mathbf{x}\_2^T \\ \vdots \\ \mathbf{x}\_h^T \end{bmatrix},$$

and the output vector <sup>y</sup> with <sup>y</sup> <sup>∈</sup> <sup>R</sup> h as

$$\mathbf{y} = (\mathbf{y}\_1, \mathbf{y}\_2, \dots, \mathbf{y}\_h)^T.$$

For regression, y<sup>i</sup> is the observed value, for classification, y<sup>i</sup> ∈ *K*<sup>j</sup> , *<sup>j</sup>* <sup>=</sup> <sup>1</sup>,. . .,*<sup>l</sup>* is the class corresponding to x<sup>i</sup> .

An example of an input matrix to classify road attributes in this study is shown in Table 2.3.


Table 2.3: Example of an input matrix and output for classification.

A road segment annotated with a specific class is described by various features, e.g. standard deviation from vertical acceleration, roll rate, and pitch rate. In case of a regression problem, the output is a metric instead of a class.

#### **2.5.2 Classification**

The goal of a classification problem is to find a function *f*X, which returns a class based on features extracted from the sensor data. The classification function in this study, estimated with annotated data, is used to predict road attributes based on vehicle inertial sensor data (Section 3.1). After a training process, the model can be tested for generalization in a testing process and afterwords the function can predict classes based on new measurement data.

A support vector machine (SVM) is a model to find such a function [117]. It is known as a top performer and it finds a global optimum, maximizes the generalization ability, is robust to outliers, and is geometrically explicable [118]. The disadvantages are, that a SVM needs a training process like all supervised learning methods and that it has to be extended for a multiclass problem, since it uses a direct decision function [118]. The one-against-one method can be used to classify various outputs, which was introduced in [119] and firstly applied on SVMs in [120].

For a multiclass problem, *l*(*l* −1)/2 classifiers are constructed to find hyperplanes between each class. Each classifier is trained on data from two classes *<sup>p</sup>* and *<sup>q</sup>*, notated as (*p*,*q*), which leads to the following classification problem:

$$\begin{split} \min\_{\mathbf{w}^{(p,q)}, \boldsymbol{\xi}\_{i}^{(p,q)}} \quad & \frac{1}{2} (\boldsymbol{\Psi}^{(p,q)})^T \boldsymbol{\Psi}^{(p,q)} + C \sum\_{i} \boldsymbol{\xi}\_{i}^{(p,q)} \\ \text{subject to} \quad & (\boldsymbol{\Psi}^{(p,q)})^T \boldsymbol{\phi}(\mathbf{x}\_{i}) + b^{(p,q)} \geq 1 - \boldsymbol{\xi}\_{i}^{(p,q)}, \text{if } \mathbf{y}\_{i} = p \\ & (\boldsymbol{\Psi}^{(p,q)})^T \boldsymbol{\phi}(\mathbf{x}\_{i}) + b^{(p,q)} \leq -1 + \boldsymbol{\xi}\_{i}^{(p,q)}, \text{if } \mathbf{y}\_{i} = q \\ & \boldsymbol{\xi}\_{i}^{(p,q)} \geq 0. \end{split} \tag{2.6}$$

*C* Í i ξi (p,q) is the penalty term with the penalty parameter *C* and slack variables <sup>ξ</sup><sup>i</sup> . The goal is to find a balance between the training errors and the regularization term <sup>1</sup> 2 (w (p,q) ) <sup>T</sup> w (p,q) , where w is the orthogonal vector of the hyperplane to separate the classes. *b* is the shift of the hyperplane.

The input x<sup>i</sup> can be mapped into a higher dimensional space with the feature map φ. The minimization problem (2.6) can be transformed into a dual problem and the classification rule for a binary classification problem is then

$$f(\mathbf{x}\_i) = \text{sgn}((\boldsymbol{\omega}^{(p,q)})^T \boldsymbol{\phi}(\mathbf{x}\_i) + b^{(p,q)}).$$

The instance is finally classified with a voting strategy, which is called 'Max Wins' strategy [121]. The vote for class *p* is increased by one, if x<sup>i</sup> is predicted as class *p*. Otherwise one vote is added to class *q*. Finally the class with the most votes determines the instance classification.

#### **2.5.3 Regression**

A linear regression function is defined as

$$f(\mathbf{x}\_i) = \mathbf{x}\_i^T \boldsymbol{\mathcal{B}} + b\_\*$$

where x<sup>i</sup> is the input, β the regression coefficients and *<sup>b</sup>* the error variable. The regression coefficients can be estimated by minimizing the distance of the ground truth y<sup>i</sup> and *f* (xi). Common estimation techniques for linear regression are for example linear least squares.

A version of SVM was introduced in [122] to find such a function with a set of training data. For this study, the regression function, estimated with annotated data, is applied on acoustic sensor data acquired in the tire cavity to estimate the road roughness. The goal of the *SV M* regression is to minimize the distance of the upper and lower bounds to the true value while being as flat as possible. Considering slack variables, ξ and ξ ∗ , to allow regression errors with cost *C*, the optimization problem can be formulated as:

$$\begin{aligned} \min\_{\boldsymbol{\beta}, \boldsymbol{\xi}, \boldsymbol{\xi}^\*} \quad & \frac{1}{2} \boldsymbol{\beta}^T \boldsymbol{\beta} + \boldsymbol{C} \sum\_{i} \left( \boldsymbol{\xi}\_i + \boldsymbol{\xi}\_i^\* \right) \\ \text{subject to} \quad & \boldsymbol{y}\_i - \left( \boldsymbol{x}\_i^T \boldsymbol{\beta} + \boldsymbol{b} \right) \le \boldsymbol{\epsilon} + \boldsymbol{\xi}\_i \\ & \left( \boldsymbol{x}\_i^T \boldsymbol{\beta} + \boldsymbol{b} \right) - \boldsymbol{y}\_i \le \boldsymbol{\epsilon} + \boldsymbol{\xi}\_i^\* \\ & \boldsymbol{\xi}\_i, \boldsymbol{\xi}\_i^\* \ge 0. \end{aligned}$$

The -intensive loss function describes the idea behind the SVM regression:

$$L\_{\epsilon} = \begin{cases} 0 & \text{if } |\mathbf{y}\_{i} - f(\mathbf{x}\_{i})| \le \epsilon \\ |\mathbf{y}\_{i} - f(\mathbf{x}\_{i})| - \epsilon & \text{otherwise} \end{cases}$$

Only estimated values with a distance greater to the observed value are penalized.

Analogous to subsection 2.5.2, the input x<sup>i</sup> can be nonlinearly transformed with the function φ. The optimization problem can be computationally simpler solved in its Lagrange dual formation.

#### **2.5.4 Feature selection**

In order to evaluate the features and to select a number *n*˜ of the most important ones, the goal is to find a function *Q*, which describes the feasibility of a combination of features to separate the classes [123]. The function should find the index set *<sup>I</sup>* <sup>=</sup> {*i*1,· · · ,*i*n˜ } from all possible index sets *<sup>I</sup>*cand with all feature candidates, at which the function values of the measure *Q* are maximized:

$$I = \underset{I\_{\text{cand}}}{\text{arg}\,\text{max}}\,Q(I\_{\text{cand}}) \qquad \text{with} \qquad |I\_{\text{cand}}| = \tilde{n}.$$

The feature selection improves the model in terms of simplification for an easier interpretation, shorter training times and avoids redundancy and overfitting. Overfitting is a problem occurring when the statistical model is too complex. It tends to describe random errors and noise instead of the underlying relationships. If too many parameters are taken into account during the training phase, the classifier will learn all data by heart and will show a very good error on training data. On the other hand, the predictive performance and the generalization usually degrade and the prediction error increases.

A multivariate analysis of variances (MANOVA) [124] can find the objective function *Q*. For the MANOVA method, each class of the output is approximately described with a *n*-dimensional normal distribution of the features x [125].

feature 1 Figure 2.6: Illustration of MANOVA.

The goal of the method is to find a group *I* out of *n* features, in which the means of feature values of the same class are concentrated and in which the means of feature values of different classes are diffuse. In other words, the means of the feature values of different classes are supposed to be apart as much as possible, which is illustrated in Figure 2.6.

To find a measure of the described requirement, only the features from the feature set *I* are selected from the input matrix *X* and the new input matrix *X*˜ has the dimension (*h*,*n*˜).

The MANOVA method evaluates the feature sets based on the between classes variance matrix *V*<sup>B</sup> and the within classes variance matrix *V*<sup>W</sup> with *l* classes, which are defined as

$$\begin{aligned} V\_B &= \sum\_{j=1}^{l} h\_j \cdot \left(\mu\_j - \mu\right) \cdot \left(\mu\_j - \mu\right)^T, \\ V\_W &= \sum\_{j=1}^{l} h\_j \cdot \hat{\Sigma}\_j, \end{aligned}$$

with

$$\begin{aligned} \boldsymbol{\mu}^T &= \frac{1}{h} \cdot \boldsymbol{J}\_{1,h} \cdot \check{\mathbf{X}}, \\ \boldsymbol{\mu}\_j^T &= \frac{1}{h\_j} \cdot \boldsymbol{J}\_{1,h\_j} \cdot \check{\mathbf{X}}\_j, \\ \boldsymbol{\Sigma}\_j &= \frac{1}{h\_j} \cdot \left( \boldsymbol{\mathcal{X}}\_j - \frac{1}{h\_j} \cdot \boldsymbol{J}\_{h\_j,h\_j} \cdot \check{\mathbf{X}}\_l \right)^T \cdot \left( \check{\mathbf{X}}\_j - \frac{1}{h\_j} \cdot \boldsymbol{J}\_{h\_j,h\_j} \cdot \check{\mathbf{X}}\_j \right), \end{aligned}$$

where µ T is the mean of all features of all classes, µ T j is the mean of features only describing the *j*th class, and Σˆ j is the estimated covariance matrix of the *j*th class. The total variance matrix *V*<sup>T</sup> is defined as

$$V\_T = V\_W + V\_B \dots$$

The analysis of variance implies to maximize the between groups variance *V*<sup>B</sup> and to minimize the group variance *V*<sup>W</sup> . The solution is an eigenvalue problem, which can be converted into an ordinary eigenproblem

$$(V\_B - \lambda\_i V\_W)\boldsymbol{\nu}\_i = \mathbf{0} \qquad (V\_W^{-1} V\_B - \lambda\_i E)\boldsymbol{\nu}\_i = \mathbf{0},\tag{2.7}$$

with the identity matrix *E* and eigenvectors v<sup>i</sup> .

With the sorted eigenvalues (λ<sup>1</sup> <sup>≥</sup> <sup>λ</sup><sup>2</sup> ≥ · · · ≥ <sup>λ</sup>n˜) from (2.7), different statistics can be applied to find the objective function *Q* , such as the Hotelling-Lawley Trace, Pillai-Bartlett Trace, Roy's Largest Roots, Wilk's Lambda [126]. However, the quality and the question, which of the statistics is the best one, is still in discussion, e.g. [127, 128]. Here, the Wilk's Lambda or likelihood ratio test is applied, which is widely used in recent literature, e.g. [129, 130] and defined as

$$
\Lambda\_I = \prod\_{I=1}^h \frac{1}{1 + \lambda\_I}.
$$

For a better interpretation of the test statistics, the test statistics is defined as

$$Q\_I = 1 - \Lambda\_I = 1 - \prod\_{i=1}^h \frac{1}{1 + \Lambda\_i} = 1 - \frac{\det(V\_W)}{\det(V\_B + V\_W)},$$

where the best value is *Q* = 1, which indicates a good evaluation of the feature set *I* and the worst value is *Q* = 0.

The number of features to be used is set by the user, and every possible feature combination is tested. The best combination is selected by choosing the quality criterion *Q* closest to one.

#### **2.5.5 Feature aggregation**

The feature aggregation is applied to reduce the size of the feature space in order to compute the classification rules in a lower-dimensional space and to save computation time. In opposition to feature selection, no features are disregarded. There are several methods to perform feature aggregation as described in e.g. [131, 132], one of them is the linear discriminant analysis (LDA) [133].

The basic principle behind the linear aggregation is to use a projection matrix *<sup>A</sup>* <sup>∈</sup> <sup>R</sup> n˜×n˚ to reduce the size of the feature space from the original feature matrix *X*˜ into the aggregated feature matrix *X*˚.

$$\pounds = \pounds\_{\mathcal{A}}$$

The goal of LDA is to maximize the ratio between the within classes variance *V*<sup>W</sup> and the between classes variance *V*<sup>B</sup> after the projection using the transformation matrix *A*.

The maximization problem can be written as the eigenproblem

$$(V\_B - \lambda\_i V\_W)\boldsymbol{\nu}\_i = \mathbf{0} \qquad (V\_W^{-1} V\_B - \lambda\_i E)\boldsymbol{\nu}\_i = \mathbf{0}.\tag{2.8}$$

The solution of (2.8) is

$$A = (a\_1, \dots, a\_{\vec{n}}) \quad \text{with} \quad a\_{\vec{i}} = \nu\_{\vec{i}},$$

where the eigenvectors <sup>v</sup><sup>i</sup> belong to the *<sup>n</sup>*˚ largest eigenvalues (λ<sup>1</sup> <sup>≥</sup> <sup>λ</sup> ≥ · · · ≥ <sup>λ</sup>n˚ ≥ · · · ≥ <sup>λ</sup>n˜) from *<sup>V</sup>*<sup>W</sup> <sup>−</sup>1*V*B.

#### **2.5.6 Performance measures**

The output of a classifier from the training and testing process is a confusion matrix *<sup>M</sup>* <sup>=</sup> (*m*i j) ∈ <sup>N</sup> l×l for classes *K*<sup>i</sup> ,*<sup>i</sup>* <sup>=</sup> <sup>1</sup>,. . .,*l*, as shown in Table 2.4.

Table 2.4: Confusion matrix of a classifier from the training or testing process.


In the confusion matrix, *m*i j presents the *true positives* for class *i*. The other elements in column *j* are called *false negatives*, in row *i false positives* and in the diagonal *true negatives*.

An overview of various performance measures for different classification problems to evaluate a classifier and to give proof for generalization is given in [134]. The most important performance measures are precision, recall and accuracy, which are derived from the confusion matrix from training or testing and calculated as follows:

$$\begin{aligned} \text{Precision}\_{i} &= \frac{m\_{ii}}{\sum\_{j=1}^{n} m\_{ji}},\\ \text{Recall}\_{i} &= \frac{m\_{ii}}{\sum\_{j=1}^{n} m\_{ij}},\\ \text{Accuracy} &= \frac{\sum\_{i=1}^{n} m\_{ii}}{\sum\_{i=1}^{n} \sum\_{j=1}^{n} m\_{ij}}. \end{aligned}$$

## **2.6 Performance indices for measurement validation**

The goodness of fit between estimated values *Y*ˆ and reference values *Y* can be quantified using the coefficient of determination or *R* 2 and the normalized root mean square error (NRMSE).

The coefficient of determination *R* 2 is defined as the square of the Pearson correlation coefficient ρ [135].

$$\begin{aligned} \rho(Y,\hat{Y}) &= \frac{\text{cov}(Y,\hat{Y})}{\sigma(Y)\sigma(\hat{Y})},\\ R^2 &= \rho^2. \end{aligned}$$

*R* 2 ranges from 0.0 to 1.0, with higher values indicating better goodness of fit. However, this performance index only evaluates linear relationships between the estimated and reference observations. Therefore, it is insensitive to proportional differences of the two data sets and an additional performance index to determine the true error is needed.

The root mean square error (RMSE) is a commonly used measure to determine the goodness of fit [136] for *n* observations. The measure can be normalized with the range of the reference data, which is referred to as NRMSE. The values of these measures vary from 0.0 as perfect goodness of fit to ∞ as bad goodness of fit.

$$\begin{aligned} \text{RMSE} &= \frac{1}{\sqrt{n}} \| Y - \hat{Y} \|, \\ \text{NRMSE} &= \frac{1}{\sqrt{n}} \frac{\| Y - \hat{Y} \|}{Y\_{\max} - Y\_{\min}}. \end{aligned}$$

Since *Y* max and *Y* min are singe values, the measure is not robust for signals with outliers. A more robust version is the division with the expected value of *Y*:

.

Figure 2.7: Performance indices for measurement validation.

Figure 2.7 shows the function *sin*(*x*) as reference data and variations of the function as estimated observations. If the reference data and estimation are identical, *R* 2 and NRMSE show the best possible values (prettyreffig:performanceindicesa). As mentioned above, *R* 2 indicates a perfect goodness of fit for a proportional variation of the two data series (Figure 2.7b). The two time series in Figure 2.7c are very similar with a harmonic of the reference observations. NRMSE indicate an almost perfect goodness of fit. However, the variance of the two time series are identified by *R* 2 . The values for Figure 2.7d are poor for both indices.

# **3 Road condition monitoring with vehicle based sensors**

*Outline:* This chapter presents a novel concept to monitor the road infrastructure automatically, comprehensively, inexpensively, and almost in real-time. Section 3.1 gives an overview of the concept, which is mainly based on vehicle sensors. The following Section 3.2 analyzes the behavior of the vehicle driving over road features and defects. Based on these theoretical considerations, Section 3.3 evaluates available on-board vehicle sensors and introduces the selected sensors for this work. Besides the selected on-board sensors, this work also considers an acoustic sensor in the tire cavity to estimate the road condition. Section 3.4 describes a novel road condition monitoring system, which consists of an automated data acquisition with the selected sensors and data transmission to a central database. Section 3.5 introduces a novel extended simulation approach to analyze the vehicle behavior and to investigate new data mining methods under controlled conditions. Section 3.6 concludes this chapter.

## **3.1 Concept of a novel road condition monitoring procedure**

Figure 3.1 gives an overview of the proposed novel concept to monitor the road infrastructure. The vehicle sensors measure the road infrastructure directly, e.g. with cameras, or indirectly through the vehicle vibration due to unevenness of the road infrastructure. Furthermore, they measure the environment, such as the weather condition, which influences the sensors itself, the vibration behavior of the vehicle and the road infrastructure. Methods are developed and implemented to combine the collected sensor data from various vehicles and to gain information of the road infrastructure. The information, such as the position of road hazards or dangerous road segments, is provided for advanced

Figure 3.1: Overview of the data flow and participants of a novel concept for road condition monitoring with on-board vehicle sensors.

driver assistance system (ADAS), e.g. to warn the driver of road hazards or suggest an alternative route with a smoother road surface. They are also visualized to allow road engineers to structure and prioritize their maintenance and quickly react to dangerous defects.

The novel concept allows to monitor the road infrastructure comprehensively in short intervals and the road transportation departments to give an overview of the recent road condition. It is mainly based on standard on-board vehicle sensors to avoid the investment of additional sensors, which often need a special permit, especially if they are mounted on the outside of the vehicle. The data are available from the controller area network (CAN) bus and they can be preprocessed with electronic control units (ECUs). The required pre-processed data for road condition monitoring or already extracted information can be sent to other vehicles or to a central data bank via vehicular communication systems, which is referred to vehicle-to-everything (V2X) communication.

The fundamental parts of the implementation of this concept are the selection of the appropriate vehicle sensors, the development and application of data mining methods to extract the favored information and to combine the output of various vehicles as well as the output of each vehicle under different operating conditions, e.g. speed, wheel load, drive, or braking torque, which overruns the same or an adjacent location. The following sections in this chapter prepare for the presentation of the novel methods in Chapter 4 and the implementation of fundamental parts of the concept in Chapter 5.

## **3.2 Vehicle motion from road irregularities**

Figure 3.2 shows the principal movement of a vehicle due to obstacles on the road. The red colored coordinate system with its origin as the center of gravity describes the orientation of the vehicle at rest, which changes its position during passing the obstacle, indicated by the black colored coordinate system. An extended obstacle on only one vehicle track mainly causes a roll motion of the vehicle Figure 3.2a and the change of position of the center of gravity in z and y direction. Examples for such obstacles are ruts on only one side of the road or potholes. A wide obstacles over two tracks of the vehicle are for example concrete joints, transverse cracks, or level crossings, which mainly induce a pitch motion of the vehicle Figure 3.2b. Furthermore, the original center of mass moves in x and z direction. The two effects can overlap, for example when the right front wheel of a vehicle firstly passes a pothole or the vehicle drives over beveled level crossings or cracks.

Figure 3.2: Principal movement of vehicles due to road obstacles.

The vibrations of the vehicle body are damped due to the suspension system (Section 2.2). The suspension compresses as the wheel moves up and extends as the wheel moves down to limit the movement of the vehicle body. Since the tire and the suspension work as a low-pass filter, vibrations due to very small obstacles or the macro texture of the road might not pass through to the vehicle body. However, the road profile deforms the tire structure, which creates sound waves in the tire cavity, which is illustrated in Figure 3.3.

Figure 3.3: Vibration of the tire due to road irregularities.

## **3.3 Evaluation of on-board sensors**

Vehicles today have various sensors on-board, which also measure the environment of the vehicle and the vehicle vibration and movement (Section 2.3). For example, a stereo camera system is mounted to the inside of the windshield to classify objects and determine distance mainly for ADASs and autonomous driving [137]. The video frames can also be used to derive objects on the road infrastructure (Section 1.4). However, the stereo camera is currently not available with standard automotive packages and only with a predictive active suspension system, such as the Magic Body Control from Mercedes-Benz. A mono camera is available with less sophisticated packages but the level of detail to detect road defects is lower compared to the stereo camera (Section 1.4).

The motion of the vehicle due to road irregularities (Figure 3.2) can be determined with an inertial sensor. Accelerometers in longitudinal and lateral direction as well as a yaw rate sensor is available with standard packages, such as electronic stability control (ESC) and anti-lock braking system (ABS). The sensors, which are important to estimate the road quality, pitch and roll rate sensor and vertical accelerometers are available with vehicle dynamics control or active suspension packages. Hereby, the vertical accelerometer can be placed in the vehicle body or at the front wheels. Newer vehicles, such as the new Audi A7 is even equipped with an inertial sensor with higher sample rate of up to 100 Hz as standard, whereas older vehicles only have an longitudinal and lateral acceleration sensor with lower sample rates to measure the driving dynamics.

The motion of the suspension can be mainly measured with a ride height position sensor in chassis systems.

The steering angle sensor and lateral accelerometer might be helpful to recognize evasive maneuver due to potholes.

The position of the vehicle is necessary to map road defects. Commonly, vehicles have a navigation system, which provides the vehicle position as well as the velocity. In modern vehicles, the GPS sensor signal is combined with inertial sensor data to estimate the precise position with an accuracy of less than 1 m through data fusion using Kalman filtering.


Table 3.1: Relevance of on-board vehicle sensors for road infrastructure monitoring.

<sup>1</sup> ++ highly relevant

+ relevant

◦ relevant as covariate

– not relevant

2 at vehicle body and/or suspension system

\* considered in this work Latter can also be acquired through wheel speed sensors. An on-board sensor to estimate the macro texture and to measure the vibration of the tire (Figure 3.2b) is not delivered ex factory.

Table 3.1 summarizes selected available on-board sensors, the required automotive package, the cost of the sensor, and their relevance to detect road irregularities based on previous and current research (Section 1.4).

# **3.4 Novel road condition monitoring system**

## **3.4.1 Sensors**

The survey of related research (Section 1.4) shows that the stereo camera has been extensively studied in previous literature and has several drawbacks. Road profilometers are too expensive for data acquisition with a vehicle fleet.

An inertial sensor in the vehicle body represents a promising sensor to estimate the road quality, as it measures the movements of the vehicle due to obstacles. An inertial sensor placed at the wheel carrier results in higher measurable frequencies due to lower inertia. However, since series vehicles are hardly equipped with inertial sensors at the wheel carrier, the focus lies on inertial sensors positioned in the vehicle body. Section 1.4 showed that research on inertial sensor for road infrastructure monitoring still has room for improvements, especially in data processing and combination of the output of various vehicles.

Acoustic sensors outside of the vehicle have various disadvantages. They are prone to environmental influences, such as water, and suffer from acoustical reflections from the environment, e.g. from noise protection walls or houses, or noise, e.g. from passing vehicles. Acoustic sensors inside the vehicle body suffer from noise inside the cabin, e.g. from radio and conversations of the passengers.

An acoustic sensor in the tire cavity for broad application has been rarely studied (Section 1.4). So far, only prototypes have been developed, which are too expensive for fleet tests and which need a specific vehicle permit due to their large constructed size and the modification of the rim for lead-through of the sensors. Furthermore, the tire cavity acoustic sensor has only been researched in the context of tire road noise. Therefore, this work considers a tire cavity acoustic sensor with the aim to develop a new measuring device for fleet tests and to estimate the road roughness or macro texture.

The position of the vehicle and therefore global position system (GPS) is necessary to visualize and map the estimated road quality or defects. The accuracy of the positioning can be improved by sensor fusion of GPS and inertial sensors, as shown in, which is implemented in modern series vehicles.

Overall, based on the evaluation of previous research (Section 1.4) and onboard sensors (Section 3.3), this work considers the following sensors for road condition monitoring:


The following sections investigate


#### **Inertial sensor in the vehicle body**

A cleat test has been performed to investigate whether information from inertial sensors in the vehicle body can be extracted to detect rough road obstacles. For this purpose, a test vehicle BMW 116d (E87) was equipped with an inertial sensor in the trunk above the center of the rear axle. In addition, two inertial sensors were placed on the lower wishbone of the left and right wheel to investigate the acceleration at the unsprung masses for further research.

Four cleats with heights of 5, 10.5, 13.0 and 19.5 mm were rolled over, with both vehicle lanes and with either one. The test drives were repeated three times with velocities of 20 and 40 km/h. The performance indices with *R* <sup>2</sup> of 0.9210 and NRMSE of 0.04275 on average indicate a good reproducibility of the test drives [200].

Figure 3.4 and Figure 3.5 show the vertical accelerations of the sprung and unsprung masses as well as the roll and pitch velocity of the vehicle body for the overrun of two cleats with different heights.

Figure 3.4: Vehicle and wheel response to cleat on both vehicle lanes with height 0.5 cm and with velocity 20 km/h. The relevant section is grayed.

The accelerometer measures the gravity of approximately 9.81 m/s<sup>2</sup> at rest. The overrun for the rear axle of the cleat takes place at 2 s. The grayed area for pitch and roll velocity of the body starts at 1.3 s, since the front axle runs over the cleat at 1.5 s. Figure 3.4 indicates, that the cleat with a height of 5 mm has a low impact on the acceleration of the body. The standard deviation of the signal during the overrun of the cleat is 0.062 m/s 2 compared to a standard deviation without the cleat of 0.016 m/s 2 . The SNR defined as the ratio of the variance of the acceleration at the cleat and without the cleat is 5.69. No significant change of the amplitude of the roll rate can be detected. The SNR of the pitch rate is 6.76, of the unsprung masses 6.36 on average.

Figure 3.5 shows the sensor signals for the overrun of a higher cleat of 13 mm with only the left vehicle lane. The SNR of the acceleration of the sprung mass is higher with 9.95 as well as the acceleration of the rear left unsprung mass with 30.76. Furthermore, the roll motion of the vehicle is reflected by a SNR of 5.76 for the roll rate. The pitch rate shows a SNR of 4.84. Since the

Figure 3.5: Vehicle and wheel response to cleat on left vehicle lane with height 1.3 cm and with velocity 20 km/h. The relevant section is grayed.

first vehicle axle passes the cleat at 1.5 s, the pitch and roll rate show a higher amplitude at this point.

The analysis of all data during this test under controlled conditions with simple road obstacles and two different velocities allow the following conclusion:


#### **Acoustic sensor in the tire cavity**

The time series of the tire cavity sound under vehicle operation with constant velocity of 60 km/h is shown in Figure 3.6. The data were acquired with a MB S-Class W220 model as test vehicle, a 255/60R16 tire, and the acoustic measurement system described in [74], where the microphone is placed inside the tire and rotates with the wheel. The tire is excited by the surface texture, which is shown in the high-frequency wave representing the first circumferential tire cavity mode.

Figure 3.7 shows the power spectral density (PSD) of the tire cavity sound under vehicle operation with the same vehicle (BMW 116d (E87) with 205/55R16 tire) on different surfaces (Figure 3.7a) and with different vehicles and different tire sizes, a BMW 116d (E87) as vehicle 1 and MB S-Class W220 with 255/60R16 tire as vehicle 2 (Figure 3.7b). The power spectral density (PSD) was calculated with the Welch's power spectral density estimate. The peaks in Figure 3.7a represent the tire cavity resonance frequency and its harmonics (1 to 4), which can be calculated with

$$f(i) = \frac{i \cdot c\_{sound}}{2\pi \cdot r\_{TC}},$$

where, *i* is the number of the harmonic, *c*sound the sound velocity and *r*TC the radius of the tire cavity on average.

Furthermore, the PSD level of both curves underline the theory that a greater road surface roughness leads to greater tyre deformation and thus to greater excitation of cavity modes within the tyre torus. The sound pressure level (SPL) of the tire cavity sound for the rough surface is over 20 dB higher than for the smooth surface.

Figure 3.6: Time series of tire cavity sound.

The difference of the resonance frequency of different vehicles is illustrated in Figure 3.7b. The peaks are not at the same frequency due to the different radii of the tire cavity, whereas vehicle 2 had tires with a greater radius. Moreover, vehicle 2 had a higher velocity and a higher SPL compared to vehicle 1.

From the analysis of tire cavity sound pretests the following points can be conducted:


### **3.4.2 Data acquisition**

To acquire data comprehensively on roads with various vehicles and to enable a road infrastructure monitoring in short time intervals, a measuring system must meet the following requirements:


Figure 3.8 shows the concept of a novel road condition monitoring system based on the selected sensors (Section 3.4). The basis of the measuring system is a data logger, which stores the data from various sensors during the drive. The sensors are an inertial sensor and GPS receiver placed in the vehicle body and directly connected to the data logger. A tire pressure monitoring system (TPMS) controls the tire temperature and pressure and transmits the data wireless to the data logger during vehicle operation. A telemetry attached to the rim acquires the data from the tire cavity sound sensor and transmits the data to the data logger as well.

Figure 3.8: Concept of a novel road condition monitoring system.

The data stored on the data logger are transferred to a central database on a server via an access point. Access points can be placed close to the parking area of the vehicle. The upload starts, after the data logger is connected to the access point. The data from the various sensors can then be fused on the server, also with additional data, such as weather data. Afterwards, the data is processed with methods developed to provide a comprehensive and up to date overview of the condition of the infrastructure.

The data logger and sensors in the vehicle are activated and deactivated with the ignition of the vehicle. The TPMS, telemetry, and tire cavity sound sensor is activated or set in stand by, dependent on the centrifugal force of the wheel.

### **3.4.3 Demonstration of novel measuring device**

Modern vehicles are equipped with many sensors (Section 2.3) and the respective data can be sampled from the CAN bus, especially the acceleration of the vehicle. However, the data is encrypted and can usually only be accessed by original equipment manufacturers (OEMs) or automotive suppliers. Researchers have used dedicated inertial measuring units (IMUs) [138] or smartphones as multi sensor platforms [139, 140]. However, dedicated IMUs are expensive and sophisticated to use and therefore not applicable for automated data acquisition with various vehicles. It is difficult to equip smartphones with additional sensors or to simultaneously acquire data from different positions within or outside the vehicle. Furthermore, the hardware can usually not be extended, e.g. with an antenna to improve the wireless connection to sensors outside the vehicle, such as the acoustic sensor in the tire cavity.

Therefore, a new measurement device is developed to meet the requirements of the novel road condition monitoring system, which is defined in Section 3.4. The goal is to acquire data from various sensors, to enable an automatic data acquisition and data transmission to a central server, and to have low cost of investments per unit for fleet use.

The core of the novel data acquisition method is a data logger in the vehicle, which consists of a single-board computer (SBC), namely a Raspberry Pi. The Raspberry Pi has input/outputs (I/Os) to connect additional sensors. Moreover, WLAN and Bluetooth adapters can be plugged in to receive or send data, as well as a memory card to store a large amount of data. Periphery, e.g. buttons, can be installed to label the data live with the desired output. The software can be self developed as open-source and the processor can merge and preprocess the data from various sensors. Furthermore, the SBC can be extended by additional boards, such as GPS or a uninterruptible power supply (UPS).

Besides the data logger, a new telemetry system at the vehicle wheel has to be developed to transmit the acoustic data of the tire cavity to the data logger.

The sensors of the novel road condition system are selected and briefly described in the following paragraphs. Furthermore, the new developed telemetry system and the database to centrally store the data from all vehicles are specified.

The accuracy of the selected sensors for the novel measurement device are validated and the results are shown in Section 6.2. The signals of the selected inertial sensor and acoustic sensor are compared with signals from reference sensors, which are accurate but too expensive for fleet use. The accuracy of the velocity from GPS is evaluated by comparing the data with velocity acquired with on-board diagnostics (OBD).

#### **Inertial sensor**

Microelectromechanical systems (MEMS) sensors have a good performance, and have advantages in size and power consumption compared to other technologies. Therefore, a MEMS inertial sensor (LSM9DS1) is selected to measure the motion of the vehicles. Furthermore, the sensors measures all six directions of movements.

#### **Acoustic sensor**

In contrast to an inertial sensor, which is either available on-board via CAN bus, in a smartphone or dedicated IMUs, an automated and inexpensive data acquisition of tire cavity sound data has not yet been developed. The most recently developed prototype, presented in [74], to sample tire cavity sound data is too expensive, the battery lasts only 10 hours, the vehicle needs a special permit to participate in public road traffic, and it needs manual interaction to start and stop the measurement.

The sensor in the tire cavity is exposed to extreme conditions. The SPL in the tire cavity to be measured exceeds 150 dB and the static pressure is approximately 2 bar above atmospheric pressure. The temperature in the tire cavity varies between -10 and 70 ◦C and the weight of the sensor influences the imbalance of the wheel. The frequency range to be analyzed is 50 to 5,000 Hz.

A conventional electret microphone with low investment costs per unit does normally not meet the described requirements, especially to measure the high SPL in the tire cavity. However, a novel modification of conventional electret microphones is proposed to measure higher SPL with less total harmonic distortion (THD). For this purpose, the field effect transistor (FET) is rewired from its common source configuration into a source follower configuration [141]. With the new configuration, the microphone is supposed to have a reduced sensitivity but an increased maximum output voltage.

Different types of microphones were experimentally analyzed to identify the sensor with the best ability to measure the tire cavity sound. The following microphones were considered:


To evaluate the acoustic sensors, the following experements were performed:


A novel telemetry device at the wheel, which transmits the sound data to the data logger in the vehicle, has been developed to start and end the measurement automatically [142]. A 3-axis ultra-low power MEMS accelerometer is located on the battery-power telemetry and tracks the centrifugal acceleration of the wheel. As soon as a predefined threshold of the acceleration is exceeded, the accelerometer sends an interrupt to a micro-controller, which then enables the power supply for the telemetry and sends the telemetry back to stand-by if no acceleration is detected.

#### **GPS**

As GPS sensor the MTK3339 chipset is employed and connected to the data logger via UART to acquire the position and velocity of the vehicles.

#### **TPMS**

The static pressure and temperature in the tire torus influence the tire torus sound and needs to be recorded. Modern vehicles have a TPMS on-board and direct TPMSs have a sensor usually at the valve in each tire to measure the pressure and the temperature [201]. Similarly to the telemetry of the acoustic sensor, the TPMS sensor is activated by an accelerometer. Moreover, a diagnostic device or the receiver of a high-line TPMS can request the sensor for data. The sensor is calibrated under ambient air pressure to take advantage of the better range of values for the analog digital converter (ADC). The information of the sensor, the pressure, temperature, sensor ID and received signal strength indicator (RSSI), is transmitted to the receiver via the carrier frequency of the ISM band, which corresponds to 433 MHz in Europe. The signal can be sampled via the CAN bus or intercepted and decoded by an external data logger in vehicle, which is implemented in this work. Figure 3.9 shows the time history of the tire temperature and pressure acquired with a TPMS. Naturally, the first signal of the TPMS is transmitted a few minutes after starting the drive. The initial conditions were approximately 25 ◦C for the temperature outside and <sup>2</sup>.4 bar for the tire pressure.

Figure 3.9: Tire temperature and pressure acquired with TPMS.

#### **Database**

The data transmission is realized via WLAN to avoid costs for mobile data plans, which is usually implemented when using a smart phone as data logger [139, 140]. The data logger connects to an access point, which is for example close to the parking area of the vehicle, and uploads the data to a server, where it is stored, merged and processed. Furthermore, the software of the data logger can be automated updated, by pulling the software from a web-based hosting service and version control system, where the source code is managed.

Figure 3.10: Functional diagram of the developed data acquisition method.

Figure 3.10 summarizes the developed method to acquire and centrally store the data from the sensors in the vehicle body and tire cavity.

# **3.5 Extended simulation approach for road condition estimation**

### **3.5.1 Overview**

Simulation has several advantages compared to real measuring drives, e.g. a perfect reproducibility, low cost and effort, among others [143]. Furthermore, it is difficult to investigate a single influencing factor during real test drives, while keeping all other dependent parameters constant. In a vehicle simulation, the physical behavior of the vehicle dynamics can be precisely investigated. Moreover, it is costly and time consuming to acquire and label real measurement data to apply supervised learning. Therefore, this section describes the development of an extended simulation approach for road condition estimation. Especially for the purpose of using the angular rates and acceleration of the vehicle body to estimate the road condition, the following examinations are considered:


Figure 3.11: Full car model for vertical dynamics simulation.

The consideration of all influences, parameters and constructional elements in the simulation is not possible due to computational costs or because the values for the parameters are unknown and can not be determined. Therefore, the vehicle model for this study sets various simplifications, which are discussed in the following sections. A common full car vehicle model, shown in Figure 3.11, is extended to sample the vehicle body motion from different positions and different set-ups, such as an active suspension or anti-roll bars. Furthermore, the road profile as input of the simulation is extended compared to previous literature.

Previous studies have focused on the behavior of the vehicle on roads with different levels of roughness with road models presented in Section 2.4, e.g. [144, 145]. However, these simulation approaches do not represent a real road infrastructure. This study extends previous simulation approaches not only by the vehicle model but also by the road model, e.g. by the implementation of road features and obstacles.

The models are transferred into a state-space model as introduced in Section 2.4 to be simulated in Matlab. The simulation can run on any modern computer without special hardware. The user can set the different parameters, which are discussed in the following sections. The simulation with a road length of 10 km takes around 5 min with standard processor (Intel Core i5) and main memory (8 GB). The corresponding data file then has a size of approximately 20 MB.

## **3.5.2 Extended road model**

The developed novel simulation considers not only different degrees of surface roughness, such as the international roughness index (IRI) as introduced in subsection 2.4.3. Also the following road features are implemented, such as defects and constructional obstacles:


The basic road surface and different levels of roughness can be modeled with (2.5). The implemented road features are described by continuously differentiable functions with varies parameters for height, depth, and length to avoid numerical issues in the time step integration process of the simulation itself [146]. Thus, the transition part of the road features are smooth. The implemented road model can be found in [202, 147].

#### **3.5.3 Extended full car model**

The basic full car model is extended by an anti-roll bar and an active suspension. The anti-roll bar is a part of many automobile suspensions and helps to reduce the body roll of a vehicle when the deflection of body and tire between left and right side are different [144]. The implemented model of anti-roll bar can be found in [144].

In addition to the passive suspension with basic springs and dampers, active suspensions use actuators between the chassis and wheel assembly, which can exert an independent force on the suspension in order to improve the driving comfort. One quarter of the full car model with active suspension is shown in Figure 3.12. The force *f*<sup>i</sup> is applied between the body and wheel assembly. It is controlled by feedback and represents the active component of the suspension system.

Figure 3.12: One quarter of the full car model with active suspension exemplarily for wheel i. The force f<sup>i</sup> is controlled by feedback and represents the active component of the vehicle model.

With this model the mathematical modeling for bouncing (3.1), pitching (3.2), rolling (3.3) of the vehicle body and vertical motion of each wheel (3.4) is derived as follows:

$$m\_{s}\ddot{z}\_{s} = \sum\_{i=1}^{4} -k\_{s\dot{i}}(z\_{s\dot{i}} - z\_{u\dot{i}}) - c\_{s\dot{i}}(\dot{z}\_{s\dot{i}} - \dot{z}\_{u\dot{i}}) + f\_{\dot{i}},\tag{3.1}$$

$$\begin{split} J\_{\mathcal{Y}\mathcal{Y}}\ddot{\theta}V &= \sum\_{i=1,2} -d\_{f}(k\_{si}(z\_{si} - z\_{ui}) + c\_{si}(\dot{z}\_{si} - \dot{z}\_{ui}) - f\_{i}) \\ &+ \sum\_{i=3,4} d\_{r}(k\_{si}(z\_{si} - z\_{ui}) + c\_{si}(\dot{z}\_{si} - \dot{z}\_{ui}) - f\_{i}), \\ J\_{\text{xx}}\ddot{\theta}V &= \sum\_{i=1,3} d\_{rl}(k\_{si}(z\_{si} - z\_{ui}) + c\_{si}(\dot{z}\_{si} - \dot{z}\_{ui}) - f\_{i}) \\ &+ \sum\_{i=2,4} -d\_{l\varepsilon}(k\_{si}(z\_{si} - z\_{ui}) + c\_{si}(\dot{z}\_{si} - \dot{z}\_{ui}) - f\_{i}), \\ m\_{ui}\ddot{z}\_{ul} &= k\_{si}(z\_{si} - z\_{ui}) + c\_{si}(\dot{z}\_{si} - \dot{z}\_{ui}) - k\_{li}(z\_{ui} - r\_{Ri}) - f\_{i}. \end{split} \tag{3.3}$$

There has been a lot of research in the design of a suitable control strategy for the active suspension. [148–150] compared and investigated different advanced controllers, such as linear quadratic regulator (LQR), Fuzzy, Skyhook and H∞. Most often the H∞ based method achieves the best results with a good efficiency. Furthermore, it is widely used in the automotive industry because of its low cost and simplicity [151]. Therefore, a H∞ controller is implemented for this extended simulation approach. The main control objectives are formulated in terms of passenger comfort and road handling, which relates to body acceleration, pitch acceleration, roll acceleration and suspension deflections *z*de f i = *z*s i − *z*ui . Hence, the feedback is

$$f = \left(\ddot{z}\_s, \ddot{\theta}\_V, \ddot{\phi}\_V, \mathbb{Z}\_{def1}, \mathbb{Z}\_{def2}, \mathbb{Z}\_{def3}, \mathbb{Z}\_{def4}\right)^T.$$

Other factors that influence the control design include the characteristics of the road disturbance, the quality of the sensor measurements for feedback and the characteristics of the available control force actuator.

#### **Measurement position of vehicle body motion**

The value of the acceleration is dependent on the measurement position in the vehicle body [143]. It increases with a larger distance from the vehicle center of gravity (CG<sup>V</sup> ) along the x-axis. In simulation, the accelerations and rotation rates of the vehicle body are mainly measured in the CG<sup>V</sup> of the vehicle body. To investigate, if the measurement position of the sensor has in impact on the accuracy to classify the road condition, the simulation is adapted. Specifically, different measurement positions at (*x*O, <sup>y</sup>O,*z*O) of the vehicle body motion are implemented

The relation between vertical acceleration at the new measurement position *a* 0 z and the acceleration at CG<sup>V</sup> *a*<sup>z</sup> is approximately

$$a\_{\underline{z}}' = a\_{\underline{z}} + a\_{\underline{z}}'' = a\_{\underline{z}} - \not\!\! \mathbf{\bar{\theta}}\_{V} \cdot \mathbf{x}\_{O} + \not\!\! \mathbf{\bar{\theta}}\_{V} \cdot \mathbf{y}\_{O},\tag{3.5}$$

for small vertical distances from the CG<sup>V</sup> .

The vehicle body is regarded as a rigid body, hence the angle and motion is shared among the entire rigid body

$$
\begin{aligned}
\ddot{\theta}\_V{}' &= \ddot{\theta}\_V, \\
\ddot{\phi}\_V{}' &= \ddot{\phi}\_V.
\end{aligned}
$$

With (3.5) the output of any position in the vehicle coordinate system can be approximately derived and the influence of the position of the interial sensor on the classification result can be analyzed.

### **3.5.4 Comparison of computer-simulated and real measurement data**

The data acquired in computer simulation with the presented vehicle model are compared to real measurement data to prove the usability of the simulation to predict road obstacles.

Firstly, the general behavior of the vehicle model is analyzed by overrunning a cleat in simulation equivalent to the real world cleat test described in subsection 3.4.1. Hereby, the vehicle, a BMW 116d, overruns a cleat in computersimulation and real world, which is 13 mm high and 70 mm wide at the speed of 20 km/h. The inertial sensor is placed in the glove compartment to be approximately vertical above the axle. In addition, inertial sensors are placed at the control arms of the front axle to measure the acceleration of the unsprung masses. These measurement points for vertical acceleration and pitch and roll rate are identically reproduced in simulation. Figure 3.13 shows the result of this test and indicates that the general behavior of the simulation is correct, although the data from the real measurement shows more noise.

Figure 3.13: Comparison of the simulated and real vehicle response while passing a cleat.

Secondly, the vehicle model in simulation is validated by comparing the vehicle vibration with real world measurement while traversing different obstacles with various velocities. During the real test drive six different attributes at four different velocities are measured. All of the following signals are acquired at the center of gravity. For each velocity and each attribute measure, four runs are conducted and then the averaged value of the parameter is reported. The error bars show the standard deviation of the obtained values. The result of the measured data and the simulation are shown in Figure 3.14. It is observed that the graph of the standard deviation of vertical acceleration and roll rate for simulation follow the graph for real measurements.

Figure 3.14: Comparison of the standard deviation (SD) of vertical acceleration and roll rate.

Figure 3.15: Relative differences of real and simulation measurements.

The relative differences of the standard deviation values from real measurements and simulation are shown in Figure 3.15. For most of the values, the differences are below 25 % for the SD of vertical acceleration and for the SD of roll rate. The values for SD of vertical acceleration are higher for real measurements for *asphalt* and *cobbled road*. The values for SD of roll rate is higher in simulation for *manhole cover* for higher velocities. Overall, the computer simulation describes the vibration of the vehicle well, even for various velocities and road attributes. There are specific outliers in the relative differences (Figure 3.15). However, the order of the absolute values for the different road attributes of simulation corresponds with the real measurements (Figure 3.15). Therefore, the simulation is suitable for the purpose of this study (subsection 3.5.1), e.g. to test various parameters of data processing methods and investigate influences of various vehicle settings and parameters on the classification accuracy. However, the simulation does not perfectly replicate the vibration of a vehicle under all conditions. Therefore, the classifiers for real vehicles still have to be trained with real world measurement data.

## **3.5.5 Variation of parameters for vehicle model data processing**

There are many parameters influencing the classification results, including the size and physical parameters of the vehicle, different loads on the vehicle, and the type of suspension e.g. passive suspension, passive suspension with anti-roll bar, and active suspension. This subsection describes the variation of parameters, which are considered for the simulation. The results of the investigations of varied parameters are shown in Section 6.4.

To determine whether a classifier can be generally used on another vehicle, the tests of the classification with different variations are simulated. Therefore, a BMW 116d is simulated with additional load of 200 kg and 400 kg. Furthermore, a vehicle model with parameters compared to a S-Class W220 with active suspension is tested as well as with parameters compared to a Sprinter.

Besides the variation of parameters for the vehicle model, the influence of the position of the inertial sensor in the vehicle body is analyzed. The measurement position of the vehicle body motion in the simulation is the center, axle, side, and corner of the vehicle, defined as S1, S2, S3, and S4. The coordinates (*x*O, <sup>y</sup>O,0) of the four positions in the coordinate system of the full car model are presented in Table 3.2.


Table 3.2: Parameters for the different positions of outputs.

# **3.6 Conclusion of novel concepts for road condition monitoring**

This chapter introduces a novel concept to monitor the road infrastructure, based on vehicle crowds and on-board sensors (Section 3.1). On-board sensors, which measure the environment of the vehicle sensor (e.g. cameras) or the vehicle motion due to road irregularities are evaluated (Section 3.2). An inertial sensor in the vehicle body and an acoustic sensor in the tire cavity are selected to be considered in this work (Section 3.3) and first data of the sensors are discussed. A concept of a novel road condition monitoring system and a simulation approach are introduced to acquire data automatically and to investigate specific influencing factors (Section 3.4 and 3.5).

The concepts are the basis for novel methods to acquire sensor data, which includes the selection and improvement of appropriate sensor models, acquisition devices, and methods to transmit and store the data in a central data base. As pointed out in Section 3.1, the core of the monitoring concept is the development and application of novel methods to process the data to estimate the condition of the road infrastructure. Furthermore, methods have to be developed to combine the output of various vehicles to improve the accuracy of the prediction of the ground truth. Finally, the developed methods and novel measuring devices have to be implemented and verified.

# **4 Novel methods for road condition estimation**

*Outline:* This chapter presents methods to process data, which are acquired with the measuring device described in Section 3.4.3. First, the raw data is preprocessed (Section 4.1). Section 4.2 describes methods how to process data, which are either annotated with the desired output variable or without a label. The application of the introduced supervised learning methods for labeled data is presented in Section 4.3. For this application, a benchmark data set is acquired under controlled conditions, e.g. one specific vehicle, four different velocities, and three different road attributes or obstacles. Finally, methods are developed to find road sections, which are passed through by various vehicles. Moreover, methods are presented to combine the prediction of each vehicle with confidence π to gain a more robust overall estimation of the road condition (Section 4.4). This procedure is only possible for the output of labeled data processing, since prediction confidence π cannot be calculated from non-labeled data processing results. The flow-chart of the data processing and methods are illustrated in Figure 4.1. Section 4.5 concludes this chapter.

Figure 4.1: Overview of data flow.

### **4.1 Sensor data preprocessing**

Since supervised classification is applied in this study, time series needs to be labeled with the desired output, namely the ground truth on the road. This is implemented by two buttons, which annotate the data while pressing. In contrast to previous studies [89, 93, 94, 96], which for example label the data ex post with video material, the data can be annotated live. However, if the annotator is not experienced, the label might not be aligned with the data corresponding to the ground truth. Therefore, the first step of the preprocessing is to align the label, which needs to be done manually but can be performed fast since the original label gives the rough position of the related time series. Figure 4.2 exemplarily shows the described step.

Figure 4.2: Aligning of the original label.

The button was pressed later than the attribute started, which leads to a delay of the label. This was manually adjusted and shifted to the correct position.

The vehicle vibration is dependent on the velocity [152]. The velocity dependency at least for the frequency shift can be minimized by transforming the time series into space series [153]. Therefore, the distance increments are calculated with the velocity and the time vector:

$$
\Delta \mathbf{s} = \boldsymbol{\nu} \cdot \boldsymbol{\Delta t}.
$$

Afterwards, the space domain data is resampled to a new fixed rate, set to 100 m−<sup>1</sup> , with a linear interpolation. The result of the resampling with linear

Figure 4.3: Application of different methods for resampling.

interpolation is shown in Figure 4.3a. The figure indicates, that the new resampled signal follows the original one. However, overfitting can arise with different parameters for resampling, which is shown in Figure 4.3b. Here, the new signal is interpolated with splines and removes peaks of the original signal.

Furthermore, Halfmann and Holzmann [154] suggest to filter signals from on-board sensors, which are subject to noise. For example, Figure 4.4a shows the resampled pitch rate from the gyroscope sensor. Despite the noise of a MEMS gyroscope, which is visible as spikes in the signal, it is well known for its good accuracy in short term [155]. There are various smoothing filters for different purposes, and the discussion of all possible filters is beyond this study. However, a suitable filter for this purpose is the median filter, which is robust against outliers and removes noise while preserving high frequency content. A *n*th order one-dimensional median filter for the signal *x* is defined as

$$\mathbf{y}(t) = \text{med}\left(\mathbf{x}(t - \frac{n-1}{2}) : \mathbf{x}(t + \frac{n-1}{2})\right), \text{ for } n = 2 \cdot k + 1, k \in \mathbb{N}.$$

Figure 4.4b shows the filtered signal with *n* = 50, which retained the original shape without the spikes.

Since there are no such spikes in the data from the microelectromechanical systems (MEMS) accelerometer but more noise [155], a Savitzky-Golay FIR

Figure 4.4: Application of median filter.

smoothing filter is applied. The filter with the frame length *<sup>n</sup>* <sup>=</sup> <sup>2</sup> · *<sup>k</sup>* <sup>+</sup>1, *<sup>k</sup>* <sup>≥</sup> <sup>0</sup> and order *<sup>o</sup>* < *<sup>n</sup>* is defined as

$$y(t) = \sum\_{i=-\frac{n-1}{2}}^{\frac{n-1}{2}} C\_i x(t+i),$$

with *n* convolution coefficients *C*<sup>i</sup> , which are dependent on the frame length and the order *o*. The derivation of the convolution coefficients is beyond this study. However, corresponding coefficients can be looked up in tables, e.g. in [156, 157]. The filter fits a polynomial of a specified degree to frames of noisy data and minimizes the least-squares error [158]. Therefore, the filter outperforms standard averaging FIR filters, which might remove high frequency content with the noise. Figure 4.5 exemplarily shows the unfiltered and filtered vertical acceleration with *n* = 135 and *o* = 3.

Figure 4.5: Application of Savitzky-Golay filter.

## **4.2 Sensor data processing**

### **4.2.1 Overview**

The goal of the applied methods is to process the sensor data to estimate the road condition while incorporating effects, which cannot be explained by the output variable. For example, the tire vibration does not only increase with the roughness of the road but also with the vehicle velocity. The advantage of supervised learning methods is that various classes, such as road features or defects, can be defined and predicted by the model. Based on the predictions from the training or testing phase of the supervised model, a confusion matrix can be derived to estimate the accuracy and generalization of the approach. The combination of several predictions at one location is then possible based on the precision or confidence in the model. The disadvantages are the computing time of the training and manually annotation of the data.

Therefore, in addition to the supervised learning model an approach is presented, which does not involve training and manual data labeling. This section discusses the influencing effects and proposes methods to process non-labeled data as well as labeled data.

### **4.2.2 Influencing effects**

First, the principal influencing effects of the vehicle body vibration is discussed followed by the determination of important influencing effects of the sound pressure level of tire cavity sound (SPL(TCS)).

Figure 4.6 shows the standard deviation of the vertical acceleration and the roll rate of the vehicle body driving over various road features with different velocities. The standard deviation is calculated for a frame length of 5 m. The experiment was conducted four times for each velocity and road feature. The marker in the error bar plot represents the mean of the measurement values and the error bars show the standard deviation of the four observations. The standard deviation for the figure is multiplied by 10 for a better visualization of the error bars. The error bars remain small except for observations for pothole with velocities 40 and 50 km/h.

The trend of the observations indicate, that the vibration of the vehicle is dependent on the velocity. The vibration due to a pothole remains constant from 50 to 60 km/h since the wheel does not fully rebound. In contrast, a manhole cover above surface level increases the vibration of the vehicle body with higher velocities. Furthermore, the vibration from the smooth surface slightly increases with the velocity. Further influencing effects, such as load or type of vehicle, are investigated with the extended simulation approach in Section 3.5.

Figure 4.6: Vehicle vibration (standard deviation (SD)) due to road features for various velocities.

To determine and quantify the influence of the various variables on the SPL(TCS), data from an experimental study [159] are analyzed. The experiments were performed under controlled conditions with the internal drum test bench from the Institute of Vehicle System Technology at Karlsruhe Institute of Technology, which is described in various dissertations and publications, recently in [160]. Table 4.1 shows the full factorial variation of the parameters of the considered variables for the experimental study.

The mean profile depth (MPD) of the selected asphalt is 0.56 mm and of concrete 1.38 mm. The sound pressure level (SPL) of the tire cavity sound ranges from 100 to 200 dB. The tire temperature was measured with a sensor placed in the tire torus.

Regression analysis is performed to estimate the relationship of the independent variable (Table 4.1) and the dependent variable, SPL(TCS). Firstly, simple regressions are conducted and afterwards a multiple regression with all considered independent variables.


Table 4.1: Considered variables and variation of their parameters.

Since the data is from experiments and subject to outliers, a robust regression instead of a standard least squares regression is applied. An overview of robust estimators can be found in [161].

Here, a bisquare weight function with default tuning constant of <sup>4</sup>.<sup>685</sup> is performed [162]. A *t*-test statistics is used to test that the coefficient is not equal to zero but has significant influence on the dependent variable.

Figure 4.7: Exemplary results from the investigation of the influence of various variables on the SPL of the tire cavity sound. Y¯<sup>0</sup> represents the average regression line. (\* p < 0.05, \*\* p < 0.01, \*\*\* p < 0.001)

Figure 4.7 shows the influence of various variables on the SPL of the tire cavity sound and Figure 4.7a indicates that speed has the major impact on the SPL. The regression lines indicate the influence of the variables, whereas the data points in Figure 4.7b, c, and d are colored corresponding to the velocity. The captions represent the average regression line. The figure suggests that the surface in this experiment has the second strongest impact.

Table 4.2 shows the results from a multiple robust regression. For the regression the values of the features are scaled into the range of 0 to 1 for a better comparison of the regression coefficients.


Table 4.2: Variable and corresponding regression coefficient.

A closer investigation of the influence of speed controlled for all other described variables show, that it has a non-linear incluence on SPL(TCS). Figure 4.8 shows the observations of the experiment [159] as well as the lines of a robust linear and robust non-linear regression.

The non linear regression model better fits the observations, which confirms the mean square error (MSE) of 2.04 for the residuals of the non linear and 2.85 of the linear model.

Figure 4.8: Observations and lines of a linear and non linear regression model for SPL as dependent and velocity as independent variable with all other variables constant.

This finding corresponds to the results of many experimental investigations of the dependence of the rolling noise on the rolling speed and on the temperature. These have even found their way into the standard describing the legal noise release testing of tires (ISO 13325:2003).

The best fit of the measured values results from a linear regression of the logarithm of the speed. The non linear dependency needs to be kept in mind for the development of a method to process the tire cavity sound data (Section 4.2).

## **4.2.3 Road unevenness and roughness estimation with non-labeled data**

This section presents the developed and applied methods to process data of inertial and acoustic sensor without a training process. The advantage is that no labeling process is needed, which is time consuming and expensive. So far, no approach has been presented, which does not involve a training process or calibration of the vehicle. However, an absolute value for the road condition, such as the international roughness index (IRI), can not be estimated, and specific attributes, such as potholes, can not be detected without a training process.

The aim of the methods is to estimate a road condition score based on the intensity of the vehicle vibration and tire vibration. Moreover, influencing effects on inertial sensor and acoustic sensor measurements, identified in subsection 4.2.2, should be incorporated. The score of various vehicles for identical road segments should be merged to reduce outliers of single vehicles. Lastly, the results, namely the road condition score, is to be mapped and potential road hazards are to be automatically labeled. Herefore, the following data processing, summarized in Figure 4.9, is developed.

After preprocessing of data (Section 4.1), features are extracted for specific framelengths of the vehicle trajectory from the space series, which represent the vibration of the vehicle or tire mainly due to road unevenness or roughness. Based on these features, a road condition score is calculated for each frame of all vehicles and all trajectories are divided into road segments. The scores from various vehicles in each road segment are merged to one score. Afterwards, a double thresholding algorithm of the road segments is applied to suppress road segments in good road condition and to combine road segments in bad road condition. In the next, the connected-component labeling algorithm automatically detects and labels areas of the map in bad road condition or with obstacles. Lastly, these areas are exported with various information, such as the severity and the size of the obstacle, and the overall calculated scores, representing the road condition, are visualized on a map.

Figure 4.9: Overview of methods to process non-labeled data.

#### **Inertial sensor data**

To describe and evaluate the vibration of the vehicle due to road obstacles, the weighted root mean square (RMS) acceleration is extracted from the inertial sensor data. The RMS acceleration is described in [163]. It is expressed in m/s 2 for translational vibration and rad/s 2 for rotational vibration. The running weighted RMS takes occasional shocks and transient vibration into account, e.g. through potholes, and is in space domain defined by

$$a\_{WRMS}(\mathbf{s}\_0) = \sqrt{\frac{1}{\tau} \int\_{s\_0-\tau}^{s\_0} a\_W(\mathbf{s})^2 \mathbf{d} \mathbf{s}},$$

where *<sup>a</sup>*<sup>W</sup> (*s*) is the instantaneous frequency-weighted acceleration and <sup>τ</sup> is the integration space for sliding average. Since the aim of this study is to identify defects in the road surface, the acceleration is not weighted to keep all information of the signal. However, for the identification of subjectively disturbing defects, the signal can be multiplied with a weighting function. In discrete form, the RMS is calculated with

$$a\_{WRMS}(\mathbf{s}\_0) = \sqrt{\frac{1}{k} \sum\_{i=0}^{k} a\_W \mathbf{2}[(\mathbf{s}\_0 - i)]},$$

with *k* as the numbers of data points in one frame.

To consider vibration due to overall road unevenness and single lane obstacles like potholes, the sliding weighted RMS acceleration is iteratively calculated from the vertical acceleration and the roll acceleration of the vehicle body.

To reduce the influence of the velocity on the vibration of the vehicle, a statistical data binning is applied and the running weighted RMS acceleration is grouped into bins of velocity intervals. The velocity bins can be freely defined. However, a good option is to define velocity bins representing velocity intervals for 30 km/h zones, inner-city roads, rural roads, and freeways.

Firstly, the maximum and minimum of both the RMS of the vertical and roll acceleration are determined for each velocity bin *i* and vehicle *j* (minj,i(*a*zRMS),maxj,i(*a*zRMS),minj,i(ω˙ xRMS),maxj,i(ω˙ xRMS)). After having acquired a sufficient amount of data, reasonable values for the maxima and minima are identified. However, the values of the extrema are updated with new acquired data.

Secondly, the scaled RMS for vertical and roll acceleration for small segments of the trajectory and for each vehicle *j* is calculated with

$$\begin{split} \text{scaled } a\_{\mathbb{z}\_{j}\bar{\jmath}\_{RMSS}} &= \frac{a\_{\mathbb{z}\_{r}RMSS} - \min\_{j,i}(a\_{\mathbb{z}\_{r}RMSS})}{\max\_{j,i}(a\_{\mathbb{z}\_{r}RMSS}) - \min\_{j,i}(a\_{\mathbb{z}\_{r}RMSS})},\\ \text{scaled } \dot{\omega}\_{\boldsymbol{x}j\_{RMSS}} &= \frac{\dot{\omega}\_{\boldsymbol{x}\_{r}RMS} - \min\_{j,i}(\dot{\omega}\_{\boldsymbol{x}\_{r}RMSS})}{\max\_{j,i}(\dot{\omega}\_{\boldsymbol{x}\_{r}RMSS}) - \min\_{j,i}(\dot{\omega}\_{\boldsymbol{x}\_{r}RMSS})}. \end{split}$$

Both scaled values are in the range of 0 to 1. To include the strongest shock or vibration of the vehicle due to road obstacles, the maximum of the scaled RMS for vertical and the RMS roll acceleration is calculated, which is referred to as road unevenness index (RUI):

$$RUI\_{f,i} = \max\left(\frac{a\_{\mathbb{Z}RMS} - \min\_{f,i}(a\_{\mathbb{Z}RMS})}{\max\_{f,i}(a\_{\mathbb{Z}RMS}) - \min\_{f,i}(a\_{\mathbb{Z}RMS})}, \\ \begin{aligned} &\frac{\dot{\omega}\_{\times RMS} - \min\_{f,i}(\dot{\omega}\_{\times RMS})}{\max\_{f,i}(\dot{\omega}\_{\times RMS}) - \min\_{f,i}(\dot{\omega}\_{\times RMS})} \end{aligned} \right).$$

The final score RUI for vehicle *j* ranges from 0 to 1 and represents the road condition, whereas 1 is bad condition.

To reduce outliers of single vehicles, the RUIs of various vehicles for the same road segment are merged. For this purpose, the road infrastructure area to be analyzed is divided into regular quadrilaterals, as exemplary illustrated in Figure 4.10, and the frames of the vehicle trajectories are assigned to the corresponding quadrilaterals. The resolution of the quadrilaterals, referred to as road segments, can be individually defined. Lastly, the RUIs of various vehicles for each road segment are combined by the arithmetic mean or maximum. Hereby, the arithmetic mean reduces outliers and the maximum takes the worst calculated RUI of one vehicle into account. Furthermore, the standard deviation of the RUIs can be calculated to evaluate the variation of the score among vehicles or drives.

Figure 4.10: Road infrastructure divided into segments.

To highlight road segments with a high excitement on the vehicle, which indicates a road hazard, a double threshold algorithm and a connected component labeling algorithm are applied.

The double threshold algorithm accounts for noise and variation of the score and merges road segments with similar RUIs. The algorithm suppresses road segments with a score below a low threshold *T*1. Road segments with a score above the high threshold *T*<sup>2</sup> are marked as relevant segments. If the score of road segments with a threshold above the low threshold and below the high threshold is close to relevant segments, it is connected to those segments, and otherwise suppressed.

The procedure is illustrated in Figure 4.11 and described with the following assignment procedure (4.1) for RUI of road segment *<sup>S</sup>*(v,*u*), where (v,*u*) is the location expressed by the longitude and latitude.

*<sup>S</sup>*(v,*u*) is assigned to a region *<sup>R</sup>*0, which is of no interest, or a region *<sup>R</sup>*2, which is of interest and contains only road segments with a high RUI. *S* 0 (v 0 ,*u* 0 ) is any connected road segment to *<sup>S</sup>*(v,*u*). The assignment procedure (4.1) ends if no more road segments can be assigned. *A* is the the identifier of the attribute.

$$\begin{aligned} A\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} &:= R\_0, \text{ if } \mathrm{RUI}\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} \le T\_1, \\ A\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} &:= R\_2, \text{ if } T\_1 < \mathrm{RUI}\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} \le T\_2 \text{ and } \mathrm{RUI}\_{S'(\boldsymbol{\nu}',\boldsymbol{\mu}')} > T\_2, \\ A\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} &:= R\_2, \text{ if } \mathrm{RUI}\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} > T\_2. \end{aligned} \tag{4.1}$$

Figure 4.11: Double thresholding method. Black segments ≤ T1, T<sup>1</sup> < red segments ≤ T2, blue segments > T2.

The connected component labeling algorithm (Algorithm 4.1), firstly presented by Rosenfeld and Pfaltz [164], clusters and labels the connected road segments with a high score obtained from double thresholding. Finally, the scores of the road segments can be visualized and the clusters from connected component labeling represent the road hazards.


The identified areas with road hazards can be summarized in a list with the position of the center of the cluster, the radius of the cluster and additional statistics, such as the minimum, average, maximum and standard deviation of the score over the trajectories, the number of drives through the road hazard, and the mean and standard deviation of the velocity.

#### **Acoustic sensor data**

The data from the acoustic sensor in the tire cavity is processed with the same method as described for the inertial sensor except for the feature extraction. Hereby, the SPL of the tire cavity sound is calculated to estimate the roughness of the road. The estimation is referred to as road roughness index (RRI).

The SPL feature is more sensitive to the velocity and therefore finer velocity intervals with <sup>7</sup>.2 km/<sup>h</sup> steps are used. Furthermore, the data set to be investigated is reduced by data, for which the tire temperature, tire pressure and load is out of a specific defined range to have comparable data and to avoid data acquired under extreme conditions. For example, during parking of the vehicle, the tire cavity adopts a similar temperature of the environment, which can range from -10 ◦C in the winter to 40 ◦C in the summer. However, various drives in different environment conditions have shown that the temperature stays approximately constant at 35 to 40 ◦C after 10 min drive. Consequently, the beginning of a drive and longer stops when tire temperature decreases again are filtered out analogous to the tire pressure, which approximately follows the tire temperature (Figure 3.9). The remaining data is then processed according to the flow chart in Figure 4.9.

#### **Trend identification**

A trend over time of the estimated RUI or RRI can be identified by applying a regression over time. A robust regression is applied with the same parameters as described in subsection 4.2.2 for the RUI or RRI merged over all vehicles for each road segment. Figure 4.12 shows two scatter plots of the score of RUI over time for two road segments from the same vehicle. On the x-axis, day 0 corresponds to 8th December 2016. Figure 4.12a indicates, that the score for the road segment increases, which suggests a degradation the road condition.

This indication is underlined by the estimated regression function with a highly significant positive regressor. Figure 4.12b shows a slightly decrease of the score. This method presented here can be applied to a large data set. The trend can be visualized on a map and the road segments can be colored accordingly to the value of the calculates regressor.

Figure 4.12: Examples for trend identification. (\* p < 0.05, \*\* p < 0.01, \*\*\* p < 0.001)

## **4.2.4 Road attribute and surface estimation with labeled data**

Labeled data allow the detection of specific objects or road surfaces with specific characteristics via supervised classification. The road roughness is estimated with supervised regression. The most important part of the processing of labeled data is to find features, which can separate the desired classes in the feature space. Firstly, candidates of features are extracted based on theoretical and physical considerations or the analysis of data from pretests, which is referred to as feature engineering. Another option is to use feature learning techniques, to automatically find representations of the raw data for classification [165, 166]. However, the latter approach is not considered in this work.

The feature candidates are automatically evaluated with multivariate analysis of variances (MANOVA) (subsection 2.5.4) and the best features to separate the classes are selected to reduce the dimension of the feature vector and to avoid overfitting. Overfitted models have a poor generalization with too many parameters compared to the number of observations. Then, the model rather describes random effects than the underlying relationship.

It is difficult to illustrate the high dimensional feature space. Therefore, the dimensions can be further reduced by feature aggregation, for example, with discriminant function analysis (subsection 2.5.5). Therewith, the aggregated feature space with for example two dimensions, can be visualized. The disadvantage of this method is a decrease of accuracy of the classification and the difficult interpretation of the aggregated features. Finally, the regression or classification is performed with a support vector machine (SVM) (subsection 2.5.2 and 2.5.3).

This work differentiates between the classification of road attributes, such as potholes, and the estimation of the road roughness within the macro texture. The object detection is based on the vibration of the vehicle body measured by the inertial sensor while the road roughness estimation is based on the tire vibration captured by the tire cavity sound (Section 3.4). Therefore, the first data set consists of the acceleration and angular rates of the vehicle in three directions and the second of sound pressure in the tire cavity. Both data sets are extended by influencing parameters, such as the vehicle velocity (subsection 4.2.2).

The methods to process labeled data are summarized in Figure 4.13. The raw data is preprocessed and transformed into space domain to extract suitable features.

Figure 4.13: Overview of methods to process labeled data.

Afterwards, the best features are selected with MANOVA and aggregated with discriminant function analysis. To finally estimate the road type or attributes, a SVM classification is applied and a regression to estimate the road roughness.

#### **Inertial sensor data**

For the feature extraction of inertial sensor data the following data streams are considered:


Furthermore, the roll and pitch acceleration as well as the jerk are deviated in time domain of the above data streams. The jerk *a*Û<sup>z</sup> for step *t* is calculated with

$$\dot{a}\_{\varepsilon}(t) = \frac{a\_{\varepsilon}(t) - a\_{\varepsilon}(t-1)}{t}.$$

The time series is transformed into space domain, as discussed in subsection 4.2.2. Furthermore, the data is transformed into spatial frequency domain, which contains the short-term, space-localized frequency content of the signal [167]. The spectrogram for a signal *x* with window function w(*s*), spatial frequency ω and space index τ is expressed in the discrete path case as

$$\operatorname{spec}\{\mathfrak{x}(s)\}(\tau,\omega) = \sum\_{k=-\infty}^{\infty} \mathfrak{x}[k]\mathfrak{w}[k-\tau]e^{-j\omega k}.$$

Hereby, features based on specific frequency bands can be investigated. In addition, the spectral centroid from the spectrograms of signal *x* for each time index τ is calculated with

$$\text{centr}\{\boldsymbol{x}(\boldsymbol{s})\}(\boldsymbol{\tau}) = \frac{\sum\_{k=1}^{N} m(k)\omega(k)}{\sum\_{k=1}^{K} m(k)},$$

where *<sup>m</sup>*(*k*) is the magnitude of frequency bin number *<sup>k</sup>*, ω(*n*) the representing frequency, and *K* the number of frequency bins [168]. The spectral centroid is a quantification of the power distribution of the signal [169]. Overall, the three data streams are extended by the following data streams, which leads us to 10 data streams in total:


The features are calculated for frames with a specific length in spatial domain and a specific overlap. If a longer frame length is chosen, short peaks for example due to potholes have a weaker impact on the value of features, which incorporate the overall signal, such as standard deviation. These short amplitudes can be captured by shortening the frame length or using features, which calculate extrema.

From the data streams (1) - (4), the following features are calculated


with

$$\begin{aligned} \bar{\boldsymbol{x}} &= \frac{1}{N} \sum\_{i=1}^{N} \boldsymbol{x}[i], \\ \sigma(\boldsymbol{x}) &= \sqrt{\frac{\sum\_{i=1}^{N} \left(\boldsymbol{x}[i] - \bar{\boldsymbol{x}}\right)^{2}}{N-1}}, \\ p(\boldsymbol{x}) &= \max\left(\boldsymbol{x}\right) - \min\left(\boldsymbol{x}\right), \end{aligned}$$

where *x* is a data frame of the corresponding data stream (1) - (4) and *N* the framelength.

The effective value or RMS (*x*¯RMS(spec)) is calculated for the short-time Fourier transformed data streams (5) - (7) for the following spatial frequency bands (1/m)

[0.1, <sup>0</sup>.5] [0.5, <sup>15</sup>] [15, <sup>20</sup>] [0.1, <sup>25</sup>] [0.1, <sup>50</sup>].

Afterwards, the following features are extracted for each frame of the vehicle trajectory from *x*¯RMS(spec) of each frequency band and from the spectral centroids (8) - (10).


Subsection 4.2.2 discusses the strong influence of velocity on the vehicle vibration. Previous research suggests to perform a linear regression with each feature as the dependent variable and the velocity as the independent variable [94].

The velocity dependency is then reduced by subtracting the estimated linear equation from the corresponding feature. However, the vehicle vibration and the extracted features are not linear dependent on the velocity (subsection 4.2.2).

In this work, the dependent parameters are incorporated in the classification and therefore, mean velocity is calculated for each window as additional feature. To allow non-linear relationships a polynomial kernel function of higher order is applied on the classification (subsection 2.5.2), which is defined as

$$k(\mathbf{x}, \mathbf{y}) = (\mathbf{x}^T \mathbf{y})^d,\tag{4.2}$$

where *x* and y are features and *d* the degree of the polynomial.

#### **Acoustic sensor data**

The described data processing method for the inertial sensor data is analogous applied for the tire cavity sound data. Road surfaces with various characteristics especially with different grain size distributions can be classified based on power features of different frequency bands, as we have already shown in [170, 171].

This work pursues to estimate the road roughness based on the tire vibration represented by the SPL of the tire cavity sound and applies a SVM regression. Subsection 4.2.2 shows that the velocity has also a strong influence on the tire cavity sound pressure and is incorporated as a feature for this regression.

Furthermore, the tire pressure, tire temperature, and vehicle load is included although the influence is weaker compared to the velocity.

Overall, the RRI can be estimated for each window with the following regression function

$$RRI = \beta\_1 \cdot SPL\_{(TCS)} + \beta\_2 \cdot \vec{\nu} + \beta\_3 \cdot \vec{p} + \beta\_4 \cdot \vec{T} + \beta\_5 \cdot \vec{P} + \beta\_0,$$

with the following features:


The features are scaled to the range of 0 to 1 with

$$
\alpha' = \frac{x - \min(\alpha)}{\max(x) - \min(\alpha)},
$$

where *x* is the feature and *x* 0 the scaled feature. Furthermore, a polynomial kernel function of higher order as presented in (4.2) accounts for the non-linear relationship of velocity and SPL(TCS). Further features can be integrated, such as the type of tire, temperature of the environment, if water or snow is on the road, among others.

# **4.3 Exemplarily application of methods on benchmark data set**

A benchmark data set is used to exemplarily show the data processing with the introduced supervised learning methods (Section 4.2) with inertial sensor data of the vehicle body. Moreover, the influence of classification parameters, such as the kernel trick or feature aggregation, and the performance measures are explained with this data set.

The benchmark data set is designed for good illustration and less complexity. Therefore, only three features are extracted to separate three classes, which are *potholes*, *surface with light damages*, and *smooth surface*. The extracted features are standard deviation of the vertical acceleration, roll and pitch rate, which represent the principle vibration of the vehicle.

The data was acquired on different road segments consisting of the mentioned road features with a BMW 116d. To further reduce the complexity of the benchmark data set, the road features were overrun with specific velocities,

Figure 4.14: Distribution of feature and velocity values of benchmark data set.

which are 30, 40, 50, and 60 km/h. The distribution of the values of these features and the velocity for the acquired data is shown in Figure 4.14.

Figure 4.15: Single feature space of benchmark data set.

The histograms indicate, that the number of data points per velocity is approximately uniformly distributed (Figure 4.14a). The values for data points corresponding to potholes are mainly larger over all features compared to other data points (Figure 4.14b, c, and d).

The standard deviations for damaged and smooth surface overlap for some data points. Therefore, we can expect wrong predictions from the classification of these features.

Figure 4.15 illustrates the classes in features space. Potholes are separable from the other classes, whereas damaged and smooth surface overlap.

Table 4.3 shows the confusion matrix of the classifier, a standard linear SVM with penalty term *<sup>C</sup>* <sup>=</sup> <sup>1000</sup> and <sup>=</sup> 1. Wrong predictions occur only between the classes damaged and smooth surface.


Table 4.3: Confusion matrix from single feature classification.

Performance measures of the training process and 5-fold cross-validation, derived from the confusion matrix as explained in subsection 2.5.6 are listed in Table 4.4. Overall the accuracy of the cross-validation is 99.0 %.


Table 4.4: Performance measures of single feature classification.

The high dimensional feature space is reduced to two dimensions with two features aggregated with linear discriminant analysis (LDA) function (subsection 2.5.5). Figure 4.16 shows the two-dimensional feature space. The solid lines represent the hyperplane to separate the classes and the dashed line shows exemplarily the maximum margin of the support vectors and the hyperplane. Figure 4.16a shows the hyperplane for the linear SVM and Figure 4.16b for the nonlinear SVM with polynomial kernel for degree-2 polynomials. For this example, the nonlinear SVM creates no big advantage. However, for a larger data set with various classes and features, it might be necessary to increase the accuracy of the classifier.

Figure 4.16: Aggregated feature space of benchmark data set.


Table 4.5: Confusion matrix from aggregated feature classification.


Table 4.6: Performance measures of aggregated feature classification.

Table 4.5 and 4.6 indicate that the values of the performance measures decrease with the aggregation of the features. Therefore, if computation time of classification is not critical, a feature aggregation should be avoided.

## **4.4 Multiple vehicle fusion**

#### **4.4.1 Overview**

The vibration behavior of various vehicles differ greatly due to the structure, suspension system characteristics of the vehicle, and the position of the measuring system. Each vehicle might have an individual classifier based on different feature spaces. This section addresses two problems, which are published in [172, 173].

Firstly, trajectories or trajectory frames from various vehicles through the identical road segments are found based on distance algorithms. Four distance algorithms are presented in subsection 4.4.2 and the results and computing time of the algorithms are compared. In contrast to other methods, such as *R* 2 , the distance algorithms find similar trajectory frames more precisely. Hereby, trajectories pointing in other directions are not considered as similar. This is essential to ensure accurate training data for new vehicles, which can be acquired automatically by labeling the data based on the label of the road segment from ground truth data of other vehicles.

Secondly, the classification results from different vehicles for one identical road segment can be merged with different methods to achieve a combination with the highest confidence. For this purpose, different combination strategies are presented and investigated in various scenarios, which are outlined in subsection 4.4.3.

### **4.4.2 Identification of road segments with trajectories from different vehicles**

To find identical road segments, which are overrun by various vehicles, the trajectories from those vehicles are compared. Firstly, a range search algorithm finds the trajectories from vehicles, which are close to a reference trajectory. Trajectories *A*<sup>T</sup> and *B*<sup>T</sup> of the same length are time series of global position system (GPS) coordinates in the form [174]

$$\begin{aligned} A\_T &= ((a\_1^\chi, a\_1^\circ), (a\_2^\chi, a\_2^\circ), \dots, (a\_n^\chi, a\_n^\circ)), \\ B\_T &= ((b\_1^\chi, b\_1^\circ), (b\_2^\chi, b\_2^\circ), \dots, (b\_n^\chi, b\_n^\circ)), \end{aligned}$$

where *a* x and *b* x represents the longitude coordinate, *a* y and *b* y the latitude coordinate and *n* the number of GPS coordinates representing the trajectory.

To find trajectories, which are close to each other, a range search algorithm is applied to reduce the computing time of the overall procedure. The range search algorithm, which is based on a k-nearest-neighbor (knn) algorithm, is to find all points *P*<sup>q</sup> within a radius around a point *P*<sup>p</sup> [175]. A k-dimensional (kd) tree is constructed out of all points with Algorithm 4.2, which assigns each point to a node to have an equal distribution of number of points in each node, d represents the depth of the tree. Figure 4.17 exemplary shows a 2d tree with 100 points and 8 nodes, each consists of 12 to 13 points on average and the application on trajectories, where the points are represented by GPS coordinates.

After the construction of the tree, the points can be identified, that lie within a circle of radius *r* around a new point *P*p. The corresponding algorithms are explained in more detail in our paper [172]. The basic idea is to find all nodes, which share an area within the circle around point *P*p. In the second step, only these nodes are searched for points, that have an Euclidean distance smaller

Figure 4.17: Illustration and application of the range search algorithm.

than *r* to the point *P*p. With this method the time complexity in big *O* notation can be reduced from *O*(*n*) for common search algorithms to *O*(log*n*).

However, as Figure 4.17b indicates, also coordinates from trajectories that run in a different direction as the reference trajectory are recognized as close points. The transferring of labels or fusion of classification outputs lead to wrong results. Therefore, further algorithms must process the trajectories to narrow the candidates of GPS coordinates belonging to the identical road segment and road lane.

Methods, which can follow the range search algorithm and eliminate wrong points are the Euclidean distance, principal component analysis (PCA), hausdorff distance, and dynamic time warping (DTW) distance.

The Euclidean distance is defined as

$$D\_1 = \frac{1}{N} \sum\_{n=1}^{N} ((a\_n^\ge - b\_n^\ge)^2 + (a\_n^\ge - b\_n^\ge)^2)^{\frac{1}{2}},$$

where the length of the compared segments of trajectories *A*<sup>T</sup> and *B*<sup>T</sup> needs to be equal. With the Euclidean distance, only segments of trajectories with the same driving direction and therefore on the same driving lane are determined, as shown in Figure 4.18a.


The PCA converts the points into a set of points of linearly uncorrelated principal components *a* c k and *b* c k of trajectory *A*<sup>T</sup> and *B*<sup>T</sup> respectively. Afterwards, the Euclidean distance of the coefficients is calculated, whereas a smaller distance indicates a greater similarity of the two trajectories [176]:

$$D\_2 = \left(\sum\_{k=1}^2 (a\_k^c - b\_k^c)^2\right)^{\frac{1}{2}}$$

Figure 4.18b shows the green colored trajectory segments, which were identified by the *D*<sup>2</sup> function as close to the red trajectory.

Figure 4.18: Application of range search algorithm with Euclidean distance and PCA algorithm.

After the application of the PCA algorithm, the trajectory segments that turn on the interaction and point into the orthogonal direction of the red trajectory are not detected as close. However, the trajectory segments with an opposite direction are incorrectly detected.

To address this problem Algorithm 4.3 is applied to consider the driving direction of trajectory *A*<sup>T</sup> and trajectory *B*<sup>T</sup> and to get same results as for function *D*1.


The Hausdorff distance calculates the spatial distance between two trajectories as [177]

$$D\_3 = \max\{d(A\_T, B\_T), d(B\_T, A\_T)\},$$

where *<sup>d</sup>*(*A*T,*B*<sup>T</sup> ) <sup>=</sup> maxa∈A<sup>T</sup> minb∈B<sup>T</sup> ||*a*− *b*||.

The application of the Hausdorff distance shows similar results to the PCA distance, where trajectory segments with an other direction could not be filtered (Figure 4.18b).

The DTW algorithm finds the minimum comprehensive path between two trajectories, which minimizes the cost of the warping. The distance is defined as [178]

$$D\_4 = \min\{\frac{1}{N}(\sum\_{n=1}^N w\_n)^{\frac{1}{2}}\},$$

where w<sup>n</sup> is the *n*th element of the warping path. The DTW algorithm returns similar results as the Euclidean distance (Figure 4.18b).

#### **Calculation time of distance algorithms**

Since all distance algorithms, partly by applying the direction Algorithm 4.3, produce similar results, the algorithm is selected based on runtime. Table 4.7 shows a comparison of the algorithms with respect to the measured time per 1,000 calculations. The Euclidean distance is outperforming the other algorithms due to its simple computation.


Table 4.7: Average run time of the algorithms for 1,000 trajectories with a length of 50 or 200 m.

#### **Calculation times with and without range search algorithm**

The range search algorithm function is intended to accelerate the algorithm, since all nearby points can also be found using the algorithms discussed in subsection 4.4.2. A test is performed with a data set of 215 km travel distance and 251,177 data points that are up to 70 km apart from each other. The method to find close trajectories is applied with and without the range search algorithm. The results in Table 4.8 show, that the use of the range search algorithm is indispensable. The calculation time with this algorithm is almost 500 times faster than the Euclidean distance algorithm alone.

Table 4.8: Run times with and without range search algorithm (RSA) for a traveled distance of 215 km.


The performance of the Euclidean distance algorithm with and without range search algorithm is also tested with a high density of trajectories in a small area. Therefore, the intersection data, which has been used in Subsection 4.4.2, is multiplied 50 times. Consequently, there are 1,050 trajectories with a total of 48,350 data points in a small area. A test with many vehicles can therefore be simulated. If each car passes an intersection twice a day, the data set corresponds to approximately 18 vessels, which navigate the crossing in 30 days. The results in Table 4.9 show that the Euclidean distance algorithm performs well even at a high density of trajectories. The calculation time is even smaller including range search algorithm in the pre-processing.


#### **Accuracy of the transfer of ground truth data**

The goal is to communicate locations together with the corresponding road attribute, where an other vehicle has recognized road attributes. Thus, the second vehicle has training data to learn a correct classification of the characteristics based on its own measurements. To test the method to transfer ground truth data to train a new vehicle, an intersection with trajectories from the BMW 116d test vehicle is used. A single road infrastructure attribute, namely a *railway crossing*, is approximately at position (8.4465, 49.0365). The corresponding data from the BMW test vehicle are annotated with *railway crossing*. To show the functionality and accuracy of the method to transfer ground truth data, *railway crossing* is overrun multiple times with a new test vehicle, a Smart.

Figure 4.19 shows the result of this test. The green sections are the actual *railway crossings* from the ride with the Smart and the blue sections represent data with the automatically transferred label *railway crossings* from already existing BMW 116d data. The figure shows that the label *railway crossing* is successfully transferred except for one case. The unsuccessful transfer is due to bad GPS signal and consequently the route is too far away from the previously traveled routes. However, this underlines the motivation to only transfer ground truth data, if the conditions, e.g. the GPS signal, are well and if there actually was an anomaly in vibration of the vehicle.

Figure 4.19: The result of the test to transfer ground truth data to new vehicles to train a new classifier.

With the described method in subsection 4.4.2, the label of already existing training data can be transferred to data from a new vehicle with a different suspension system to develop a new individual classifier. Labels are only transferred if a certain amount of labels from different vehicles and classifiers exist. Furthermore, labels of road hazards, e.g. potholes, are only transferred if the signal of the new vehicle actually shows an anomaly. Therefore, the amount of labels, which are wrongly transferred, decreases. Without the mentioned conditions, the transferred label might not be correct in the following situations:


#### **4.4.3 Multiple vehicle combination**

Since functions *D*<sup>1</sup> to *D*<sup>4</sup> find similar trajectories on the same road lane, the output for a specific road segment form various vehicles can be combined with the following method. The precision presents the fraction of retrieved instances that are relevant and can be seen as the probability <sup>π</sup>i j of the classifier to predict class *<sup>i</sup>* as class *<sup>j</sup>* for *<sup>i</sup>*, *<sup>j</sup>* <sup>=</sup> <sup>1</sup>,. . .,*l*.

A precision matrix Π can be derived from the confusion matrix for each vehicle *m* with *k* overruns as shown in Table 4.10.

The precision vector for class *K*<sup>j</sup> for vehicle *m* is then defined as

$$
\pi\_f^{(k)} = (\pi\_{1j}, \dots, \pi\_{lj})^{(m)} \cdot
$$

The approach, illustrated in Figure 4.20, is to combine the prediction from various vehicles or overruns with prediction probability <sup>π</sup><sup>j</sup> for a specific road


Table 4.10: Precision matrix derived from the confusion matrix.

segment with the median position (long., lat.). The goal is to get a robust and more precise estimation of the corresponding ground truth, in contrast to consider the output of only one vehicle or a simple maximum vote.

Figure 4.20: Approach to combine the predictions of the condition of road segments from various vehicles. The final prediction has the highest confidence by incorporating the confidence π of the predictions of each vehicle.

Vehicles might have different classifiers with different precision vectors. Also, the actual precision vector of a vehicle might be unknown but can be approximately estimated through the testing process of the classifier.

#### **Combination strategies**

Since the prediction of a vehicle usually has an uncertainty, multiple vehicles can be combined to increase the confidence of the prediction compared to only one vehicle, which is regarded as multiple expert problem. The final decision for each road segment can then be made by the combination of the individual predictions [179]. The classifier predicts one out of *l* possible classes and from the precision matrix π, the confidence of the corresponding classification be be derived.

The collected predictions for one road segment, named observation *<sup>o</sup>* <sup>∈</sup> <sup>R</sup> l , are distributed according to a multinomial distribution. The observation *<sup>o</sup>* <sup>∈</sup> <sup>R</sup> l counts the number of predictions per class.

The following methods provide a final estimation of the condition of one road segment. One possibility is to count how often each attribute is predicted, which is referred to as *majority vote*. The one with most votes is assumed to be the ground truth

$$V\_{\text{majority}} = \arg\max\_{i \in \{1, \dots, l\}} o\_i \dots$$

An extension to the *majority vote* approach is the vote with the diagonal vector from the precision matrix, which is referred to as *vote with precision diagonal matrix*. Here, the counts for the predictions are weighted with the precision vector:

$$
\omega\_{\pi} = (\pi\_{11}, \pi\_{22}, \dots, \pi\_{ll}).
$$

That leads to an adapted observation *o* 0 :

$$o' = o w\_{\pi}{}^T \, .$$

As before, the final decision is class with the most votes:

$$V\_{\text{diagonal}} = \arg\max\_{i \in \{1, \dots, l\}} o'.$$

Each of the *n* vehicles might use an individually optimized classifier. This leads to a precision matrix Π (k) and observation *o* (k) for vehicle *k*. Due to the assumption of independent predictions, the probability that class *i* is the ground truth, is calculated as

$$p\_{O|i} = \frac{\prod\_{k=1}^{n} \prod\_{j=1}^{l} (\pi\_{ij}^{(k)})^{o\_j^{(k)}}}{\sum\_{q=1}^{l} \prod\_{k=1}^{n} \prod\_{j=1}^{l} (\pi\_{qj}^{(k)})^{o\_j^{(k)}}},$$

with *O* = {*o* (k) } for *<sup>k</sup>* <sup>=</sup> <sup>1</sup>,. . .,*<sup>n</sup>* [179].

As final estimate, the class with highest probability is selected with

$$V\_{\text{matrix}} = \arg\max\_{i \in \{1, \dots, l\}} p\_{O \mid i \sim l}$$

The latter vote is referred to as *vote with precision matrix*.

#### **Scenarios**

The presented methods to estimate the condition of a road segment are evaluated by simulating different scenarios. The proposed scenarios considers various situations, e.g. that the road condition changes after maintenance or that a classifier of one vehicle is over-fitted and outputs random predictions. The point of interest is how many vehicles or overruns are necessary to reach a very small proportion of misclassifications for the prediction of the road segment and how to react to latter described scenarios.

**Scenario 1: Classification with one global precision matrix** In Scenario 1, one global precision matrix Π<sup>b</sup> for all vehicles is considered. For simulation, a road with a specific number of segments, each segment having a different attribute, is assumed. The outputs of various vehicles lead to the evaluation of the different fusion strategies. Additionally, the prediction quality of the different classes is analyzed.

**Scenario 2: Classification with unknown vehicle precision matrices** In reality, not all vehicles use the same distribution for prediction and the real precision matrix is unknown, since it might be different to the precision matrix calculated during training or testing. Therefore, in the second scenario each vehicle classifies the road segments by its own classifier while the final decision is made based on the global precision matrix πb. Hereby the vehicle precision matrix π (k) v is an slightly adapted version of the global one. In the simulation, vehicle *k* generates a random number to classify the road segments. For segment having property *i*, it follows the multinomial distribution described by the entries πi j(*k*), *<sup>j</sup>* <sup>=</sup> <sup>1</sup>,. . .,*<sup>l</sup>* from the adapted precision matrix π (k) v .

**Scenario 3: Classification with equally distributed precision matrix** Another scenario under consideration is the classification by a vehicle totally independent of the ground truth. An example in real life is a vehicle with a classifier, which is over-fitted, a change of the vehicle vibration behavior or a broken sensor. For this scenario, the effect on the fusion strategies, the possibilities to identify the vehicle making trouble, and to react with adjustments of the combination method are analyzed.

**Scenario 4: Change of the road condition** As last scenario the change of the ground truth from one time to another is simulated. It characterizes situations like a pothole that gets repaired, or a hard winter that destructs the road. It is investigated if rules of forgetting or less weighting the past can improve the outcomes.

The calculations and investigation of multiple vehicle fusion with 10 different vehicles are implemented in Matlab. In the calculation, the road consists of *s* segments and each segment represents one of *l* classes as ground truth with *l* = 5 for all scenarios. In reality, theses classes could correspond to types of damages, such as potholes. Therefore, the number of wrong predictions ranges from 0 to *s* for the created road. The values of the global precision matrix are set as shown in Table 4.11. This precision matrix is calculated from a confusion matrix based on a testing process from a real vehicle.

Each vehicle drives over the road and classifies the segments with its individual precision matrix, which is unknown to the back-end.


Table 4.11: Global precision matrix for simulation.

For **Scenario 1 to 3**, 5 segments are simulated and 10,000 iterations of the drives are performed to reduce stochastic effects. In **Scenario 1** the precision matrix of the vehicles correspond to the global precision matrix. The total number of wrong predictions are counted and the proportion of wrong classifications of all classes are analyzed as well as the wrong predictions of each class and number of vehicles. The wrong predictions in scenario 1 are calculated with the combination method based on *simple vote*, *vote with precision diagonal matrix*, and *vote with precision matrix* (subsection 4.4.3).

In **Scenario 2**, each vehicle has an unknown precision matrix, which simulates e.g. different vibration characteristics. However, the combination of the predictions made by all vehicles is still performed with precision matrix used for scenario 1. The difference of the vehicle and global precision matrix could correspond to differences in the vehicle mass or various environment conditions and therefore a slightly different vibration behavior of the vehicles.

For the simulation of the further scenarios the output fusion is performed only with the method *vote with precision matrix* since the results of the previous scenarios have demonstrated the highest accuracy with this method.

In **Scenario 3**, one vehicle with a problem of the classifier is simulated, which is referred to vehicle *with broken sensor*. It classifies the segments totally random and the question is, how much impact it has on the final prediction. To decrease the negative impact of such a scenario, the following method is applied.

After 5,000 overrun segments and at least 3 vehicles, the simulation compares the output of one vehicle with the output of the other vehicles, which are fused with the system precision matrix as explained in earlier scenarios. Lines 8 to 16 in Algorithm 4.4 show the pseudo-code of this method, which is embedded in the simulation algorithm. Based on these comparisons, the precision matrix of the one vehicle is estimated and checked if the values differ greatly from the back-end precision matrix.

If the difference exceeds a value, which is set to 0.15 in this simulation, the vehicle with the equally distributed precision matrix and wrong predictions is identified. Furthermore, the estimated vehicle precision matrix is used for further calculations to reduce the confidence of the output of this vehicle and therefore reduce the overall number of misclassifications.

For the simulation of this method and for **Scenario 4**, the number of segments is set to *<sup>s</sup>* <sup>=</sup> <sup>10</sup>,<sup>000</sup> and one run of the simulation per vehicle is performed. The properties of the segments, representing one of the five classes, are nearly uniformly distributed.

In the simulation, the ground truth changes after 5 runs for every third road segment. It depicts situations like the fix of road damages or the formation of new ones over time. The scenario should clarify that the back-end needs a forgetfulness of previous classifications to guarantee a fast reaction to the change. As benchmark it is counted how many of the *<sup>s</sup>* <sup>=</sup> <sup>10</sup>,<sup>000</sup> segments are wrongly classified when using the *vote with precision matrix* without forgetting. Additionally, a modified version is implemented, which has the ability to forget. Therefore, the vehicle observations *o* (k) , with *<sup>k</sup>* <sup>=</sup> <sup>1</sup>,. . .,3, are scaled down. E.g. the classification of vehicle *k* from time step *t* −*i* has only the weight *f* i in time step *<sup>t</sup>* with *<sup>f</sup>* ∈ [0,1]. A forgetting factor of *<sup>f</sup>* <sup>=</sup> <sup>0</sup>.<sup>5</sup> is used for the simulation. A higher factor means less forgetting because the observations are weighted stronger.


## **4.5 Conclusion of novel methods for road condition estimation**

This chapters describes the developed and applied methods to process inertial and acoustic sensor data for road condition estimation. Firstly, preprocessing is applied to align the labels, the desired output, with the raw data. Furthermore, the data in time domain is transformed in space domain by velocity and time vector. The data is linearly resampled to a uniform sample rate of 100 m−<sup>1</sup> . The rate sensor data is smoothed with a median filter and the vehicle velocity and acceleration data with a Savitzky-Golay filter.

Before presenting methods to process the data, influencing effects on the sensor data or the output variable are experimentally identified. Besides the road condition, the velocity is the main influencing effect of vehicle body vibration. The main parameter influencing the sound pressure in the tire cavity is also velocity besides the road roughness or obstacles. The type of tire, surface temperature, tire temperature, tire pressure, and load have minor impact on the sound pressure.

Incorporating these effects, methods to estimate the road unevenness and roughness with non-labeled data are presented. The output of the road condition estimation is a colored map. Moreover, obstacles are determined, highlighted, and listed in a table with information, such as number of drivers, size, average velocity. Furthermore, a trend identification is developed based on regression of the road condition over time. For data, which are labeled with the desired output variable, methods are presented to predict road attributes based on inertial sensor data. Moreover, a regression of sound pressure level in the tire cavity and influencing factors, such as velocity, is introduced to estimate the road roughness. The classification process, possible parameter variations, and performance measures are exemplarily demonstrated and discussed with a benchmark data set, which incorporates three classes and three features, representing the vehicle vibration.

To combine the prediction from the classification of various vehicles, the following procedure is developed. Firstly, similar trajectories are identified with range search and distance algorithms to combine the classification results of only those trajectories. Secondly, methods to combine these classification results are presented, which are referred to as *majority vote*, *vote with precision diagonal*, and *vote with precision matrix*.

Overall, the presented methods cover the whole data processing chain to estimate the road condition with vehicles as sensor platform, from preprocessing to visualization and multiple vehicle combination.

# **5 Implementation**

*Outline:* This chapter describes the developed measuring devices as well as the software to acquire, transmit, and centrally store the data (Section 5.2). The presented methods to estimate the road condition are implemented in a Matlab toolbox, which is introduced in Section 5.3. Overall, the measuring devices and the software to acquire and analyze the data are developed and implemented under the following aspects, also to allow the application of a large number of measuring devices: automation, low-cost, no manual intervention, only few different computing environments and programming languages to allow a continuous data processing flow. The software for the measuring device to acquire the data is written in Python with bash scripts for routines, the data processing methods are implemented in Matlab.

## **5.1 Overview**

The vertical vehicle dynamics simulation to vary parameters and to investigate their influence on road condition estimation with data mining, described in Section 3.5 is implemented in Matlab. The corresponding code and description to perform the analysis can be found in [147].

The methods to combine the classifications derived by various vehicles of an identical road segment are also implemented in Matlab. The developed toolbox is published in [180].

The following sections focus on the implementation of data acquisition and a Matlab toolbox to process the acquired sensor data. The complexity of the selection and combination of hardware, especially the sensors and data logger in the vehicle and the acoustic sensor in the tire cavity, as well as the requirements of the measuring device are outlined in Section 3.4. This chapter mainly presents the final result and application of the implementations and describes the most important steps of the set-up and operation of the measuring devices. The detailed instructions for setting up and operating the measuring device to acquire data can be found in [181]. The instruction includes the self-developed software and installation file, files to 3D-print the cases, files for the self-developed printed circuit board (PCB) and software for the microcontroller. The developed toolbox in Matlab, named Vehicle Learner Toolbox, consisting of the implementation of the data processing methods, presented in Section 4.1 and 4.2 is published in [182].

Overall, the measuring devices and the toolbox are developed to enable an automatic data acquisition as well as processing. For the set-up and installation of the measuring devices, only few manual interventions are needed. The measurements run automatically and the toolbox only needs the desired parameters as input.

# **5.2 Data acquisition**

The developed methods and described sensors for acquiring sensor data automatically (Section 3.4.3) are the basis for the measuring devices developed in this work. The measurement system for acquiring inertial sensor data, vehicle position, and velocity and to transmit the data to a central data base is shown in Figure 5.1. To annotate the raw data with the label of the ground truth, two buttons are installed. For every measurement drive, data describing a certain type of surface (e.g. if asphalt 1, otherwise 0) and an attribute (e.g. if pothole 1, otherwise 0) are recorded. After the measurement drives, binary coding of the data of the respective files is transformed into the coding given in Table 5.1. The unix time *t*, *ID* for the sensor, speed v<sup>V</sup> , position and time stamp of the GPS *lat*, *lon*, and *<sup>t</sup>*GPS, accelerations *<sup>a</sup>* and rotation rates ω along all three axes, and the two labels for the attribute *e* and surface type *m* are recorded and stored in a csv-table on the measurement system (Figure 5.1).

The sound pressure in the tire cavity is acquired with a self-developed telemetry system incorporating the novel methods described in Section 3.4.3, which is attached to the wheel (Figure 5.2). The audio data are transmitted via Bluetooth to the measurement system in the vehicle, along with the state of charge (SOC) of the battery, and saved as a wav-file. The tire pressure monitoring system

Figure 5.1: Measurement unit, consisting of Raspberry Pi, inertial sensor, Adafruit ultimate GPS Hat, UPS Hat, and two buttons for labeling the data. In service, the inertial sensor is fixed under the Raspberry Pi on the bottom of the case. The case is cut open for visualization purposes.


Table 5.1: Label and name of the classes for road surface and road attribute.

(TPMS) data is transmitted on the radio frequency 433 MHz and fused with the inertial sensor data among others in the csv-table.

To enable an automatic installation of the self-developed software, the SD card for the Raspberry Pi needs to be written with a self-developed image analogous to a normal Raspian OS image. The basis as Raspian OS of the self-developed image is Jessies. Alternatively, a new self-developed image can be created with the script *kernel\_img\_push-to-SDCard.sh* on Linux. To download and install the software on the Raspberry Pi, it needs to be connected to the internet

(a) telemetry system without wheel (b) telemetry system attached to the wheel Figure 5.2: Telemetry system, consisting of an acoustic sensor and TPMS in the tire cavity, a self-developed y-valve for cable entry and air, and the wireless transmitter in a case (without cover for visualization purposes).

via LAN cable to automatically download the self-developed software from a Git-repository on a web-based hosting service. For the installation process, there is no manual interaction needed. If the Raspberry Pi is to be coupled with the telemetry system and the software is to be installed on the Bluetooth module of the board, the Bluetooth antenna needs to be plugged in and the Raspberry Pi needs to be connected to the telemetry via UART. The credentials for the access point and the remote data storage to transmit the data can be adjusted in the repository in the file *rpi\_config.cfg* in the sections 'WLAN' and 'Samba' or directly on the Raspberry Pi.

The micro-controller of the telemetry system, which is responsible for activating the PCB by rotating vehicle wheels, needs to be flashed with the self-developed software. Furthermore, the firmware of the Bluetooth module on the board needs to be updated. The software of the Bluetooth module is automatically installed during the installation of the Raspbeery Pi.

The software for the measurement system in the vehicle is mainly developed in Python, routines for automation are implemented as bash scripts. The *sensorrecorder.py* is the main measurement collection service of the software. The collection service runs the processing task of every sensor in a separate thread, as illustrated in Figure 5.3. The threads are responsible to acquire data from global position system (GPS), inertial sensor referred to as IMU, TPMS, and on-board diagnostics (OBD) and are divided into initializing and running. On program start, the threads initialize the required libraries for the sensors, which are initialized and set up. For GPS the gps3 and for IMU the rtimu library are imported. Furthermore, the settings, such as orientation, and the poll interval are set for IMU. For TPMS and OBD, the serial connection to the receivers are started. Moreover, the pin configuration for the buttons is initialized. Afterwards, the sensor threads get data. In addition, the GPS thread generates data dictionary and the IMU thread log the sample rate. OBD and TPMS thread decode the received data. In this work, only the data for vehicle velocity and wheel revolutions per minute are acquired. However, it is possible to sample other sensor data from OBD. The button thread checks the states of the pins. While running, the service collects the data from all sensors, combines and tags them with a timestamp and queues the data for writing them to an output file. The write-out task then will write the measurement data from all tasks into a file, creates a new one after five minutes and compresses the old one. If a sensor or receiver is not connected to the measurement system, the data from the others are still acquired.

The *audiorecorder.py* to acquire the audio data from the telemetry system at wheel runs separately from the main collection service. It starts automatically triggered by userspace /dev (udev), a device manager for the Linux kernel, as soon as a bluetooth connection between the Raspberry Pi and the telemetry system is established. The telemetry system is configured as master and actively tries to connect to the Raspberry Pi. If the connection is successful, the recorder starts the audio transmission via a remote console on the bluetooth module of the telemetry system and the audio data is acquired and saved as wav-file.

New files are created every 5 minutes to avoid large data files for the upload to a remote file storage. The file names contain an ascending ID and the timestamp to allow merging the audio data with data from the other sensors.

The *wifiupload.py* script is triggered by the uninterruptible power supply (UPS) module, which manages the power outages and shutdown of the measurement system. The script assumes, that the vehicle's engine is powered off and a

Figure 5.3: Structure of the main da ta collection service *sensorrecorder.py* with processing tasks for each sensor as separate, independent threads. The data is combined and tagged with a timestamp in a csv-table.

properWIFI link is available. If these conditions are valid, the script will update the Git repository and transmit the files to a remote file storage. Afterward, it will switch off the measurement system or restart the recording software, if power is available again.

During the data recording, one can connect to the measurement system via LAN to monitor the recording and, if necessary, to save the data files. On a Windows PC, the programs 'Putty' and 'WinSCP' are suitable for this purpose. The IP address of the measurement system is set to 192.168.111.3 and the data is located in the folder /home/pi/logging\_data.

Overall, the measurement device runs mainly autonomously. The measurements of the sensors in the vehicle body start and stop with the ignition of the vehicle and the measurements of the sensors at the wheel with its rotation. After the measurement is finished and the data logger is connected to a known access point, all data is automatically transferred to server. From there, the data can be processed with the toolbox, which is presented in the following section. The only manual intervention needed is to charge the battery of the telemetry system at the wheel via mini USB with output voltage of 5V after more than 150 h of measurement time..

# **5.3 Data processing**

## **5.3.1 Overview and import**

To facilitate operation by non-experts, the methods presented in this work to preprocess and process the data (Section 4.1 and 4.2) are implemented in a graphical user interface called *Vehicle Learner Toolbox*. It is based on Matlab and implements several machine learning operations of the freely available toolbox SciXMiner [183] (formerly, Gait-CAD [184]). The *Vehicle Learner Toolbox* provides the possibility to


A project folder can be selected and sensor data can be imported in the corresponding menu element *Data* (Figure 5.4). There is the option to assign the sensor data to specific vehicles, since they vary in suspensions, damping, and other parameters, which has an impact on the vibration behavior. Therefore, *supervised learning* data processing, described in subsection 5.3.3, can be performed for data dependent on the vehicle. The data from various vehicles can be merged for the *unsupervised learning* data processing. The import allows .csv and .xlsx file format with the following column headers:


#### 5 Implementation


Figure 5.4: The import data frame of the *Vehicle Learner Toolbox*. Different vehicles can be chosen and the data type can be set.


Furthermore, csv files along with wav files can be imported, if tire cavity sound pressure was acquired. The sensor data from the csv and wav files are aligned and merged based on the unix timestamp.

The data is imported and preprocessed with the methods presented in Section 4.1. Moreover, the imported data set can be categorized as training, testing, or unlabeled data.

Furthermore, the parameters for the length of road segments and overlapping of these segments as well as the resampling frequency can be determined. The standard window profiles are *surface type* with a window length of 50 m and *attribute* with a window length of 5 m.

#### **5.3.2 Non-labeled data processing**

The sections of the toolbox for the non-labeled data processing for inertial sensor and acoustic sensor data automatically apply the methods presented in subsection 4.2.3. Figure 5.5 shows the frame of the toolbox for the processing of inertial sensor data. The frame for the process of acoustic data is similar except for the possibility to filter the data for the range of the tire pressure and tire temperature. The data of various vehicles can be combined and the time interval as well as the area of interest can be chosen. Furthermore, the lower threshold and upper threshold as well as the resolution for the algorithms needs to be selected. Furthermore, the maximum, mean or the standard deviation of the score for each road segment can be calculated. The output of this data processing can be a map based on open street map (OSM) format with the road state scores, a csv file with the most damaged road segments and an OSM colored with the trend of the road state score.

#### 5 Implementation


Figure 5.5: The non-labeled data processing frame of the toolbox.

### **5.3.3 Labeled data processing**

After the import and preprocess of the data, new data streams are calculated and features are automatically extracted, as proposed in subsection 4.2.4. Any code to calculate new data series or features can be easily added in the corresponding Matlab function. The in Section 4.2 proposed data mining methods are implemented in the toolbox. The user can select and use the methods under the menu element *Supervised Learning* (Figure 5.6). Furthermore, the toolbox contains additional data mining methods from the SciXMiner toolbox, introduced in subsection 5.3.1, which the user can select to process data. However, this work concentrates on the data mining methods described in Section 4.2. These methods showed the best performance compared to other methods from SciXMiner, as investigated in [203].


Figure 5.6: The train classifier frame of the *Vehicle Learner Toolbox*. Various classifiers can be trained with a wide range of options.

From the training or testing data set, labels are encoded according to Table 5.1 and data annotated with specific labels can be excluded from the classification. Another option is to thin out classes with significant more data points than other classes to allow an approximately uniform distribution of data points among the classes to prevent over-fitting of specific classes. Furthermore, systematic errors during labeling the data can be removed, e.g. if the button to annotate the data live was activated too early or too late the annotation can be shifted or data points with the wrong annotation can be excluded. After the generation of the data set to be processed, the settings for the classifier can be determined under the tab *Train Classifier*, shown inFigure 5.6.

A new classifier model can be created with a training data set or an existing model can be selected to test on new data. The next part contains the settings of feature selection (e.g. multivariate analysis of variances (MANOVA)) and aggregation (e.g. discriminant analysis), as proposed in subsection 4.2.4. Reducing the amount of features highly influences the classification result by reducing the chances for over-fitting. It is possible to cross-validate the training process by setting the k-fold-cross-validation value to higher than 1. The last section offers a variety of settings for the classifier, e.g. for a support vector machine (SVM), including the kernel function and penalty term. Afterwards, the classifier can be trained and data can be plotted on OSMs. Furthermore, the confusion matrix and the total loss is shown in the Matlab console.


Figure 5.7: The train regression frame of the *Vehicle Learner Toolbox*. Various regressions can be trained with a wide range of options.

Similar to training the classifier with features from inertial sensor data, a new regression model can be trained with various variations under the menu *Supervised Learning* and *Train Regression* (Figure 5.7). For example, the vehicle can be chosen and the time interval and area to be investigated can be set. Furthermore, the independent variables of the regression as well as the dependent variable have to be determined. Lastly, the size of the dataset can be automatically reduced to avoid long calculation time and the number of folders for the cross-validation can be entered. The output is a plot with the regression function and the measured observations as well as performance measures. Analogous to the classification, new data can be tested as well as plotted on OSMs.

For testing new data, a data set with modifications in time range and area to be analyzed can be generated as described for training, and a trained classifier needs to be selected. If the test data set is labeled, the output of the prediction is again a confusion matrix and the classification error. Moreover, the results can be visualized and plotted on OSMs. The trajectories will be cut into segments of different color referring to the corresponding classes, which are predicted.

# **5.4 Conclusion of implementation**

This chapter demonstrates the developed measuring device to acquire data from inertial sensors, acoustic sensors in the tire cavity, GPS, OBD, and TPMS. The instructions for setting up and operating the measuring devices, including the software, technical drawings of the cases, the developed PCB, and a list of the required hardware, is available in [181]. The measuring device is designed for an automated installation and autonomous operation and therefore for mass production and comprehensive data acquisition with various vehicles.

The methods to process the acquired data are implemented in a user friendly Matlab toolbox, which is available in [182]. With this toolbox, the data processing is automated, from importing and storing data to combination and visualization of the road condition estimates. The user only has to set the parameters for the data processing methods.

# **6 Results and discussion**

*Outline:* This chapter presents the results of the validation of the developed measurement device (Section 6.2) and the estimation of the road condition with non-labeled and labeled data from inertial and acoustic sensor (Section 6.3). The visualization of the road condition estimation is implemented in the developed Matlab toolbox. Matlab provides maps based on open street map (OSM), on which the estimations can be visualized. These maps are only available colored and the investigation of the results works on monitors. However, for this work, the estimations are mapped with the open-source application QGIS, which provides Stamen's toner maps as basemap layer that are black and white and more suitable to read. The results of the investigation of the influence of different parameters and vehicles on the classification accuracy in simulation are presented in Section 6.4. Furthermore, the methods to combine the prediction of various vehicles are evaluated in Section 6.5. Each presentation of results is followed by a discussion. In particular, the developed measuring device is discussed and contrasted with vehicle on-board sensors. Moreover, the results of the road condition estimation (subsection 6.3.3) and the implementation of the presented methods on an electronic control unit (ECU) of the vehicle to run online (subsection 6.3.4) are discussed. Furthermore, the extended simulation approach is concluded and the application and results are discussed (subsection 6.4.3). Lastly, each section in this chapter includes open problems and suggestions for succeeding works with the topic of road condition estimation using vehicles as sensor platforms.

### **6.1 Overview**

For the validation of the development measurement device, which is presented in Section 3.4, data were acquired with a BMW 116d driving specific maneuvers. In addition to data from the developed measurement device, reference data from established measurement devices, which are too expensive for fleet use, were simultaneously acquired. Based on the comparisons of the data, the developed measurement device is evaluated in Section 6.2.

Data for road condition estimation were acquired over a period of three years with different vehicles, different drives and under different conditions, such as temperature and weather. The measurement device in the vehicle body with the GPS and inertial sensor was always placed between the driver and co-driver seat, glued on the armrest behind the hand brake. The acoustic sensor was attached to a free-rolling wheel.

For the estimation of road unevenness and roughness with the methods presented in subsection 4.2.3, the time consuming process of data labeling is not necessary. Therefore, data were constantly acquired over the period of three years, mainly with a Ford Galaxy. In addition, a BMW 116d, VW Golf, Smart and Mercedes-Benz S-Class were used to acquire data to test the methods for different vehicles. Overall, over 300 h of data were gathered for the estimation of road unevenness and roughness with non-labeled data. Subsection 6.3.1 shows results of the application of the methods from subsection 4.2.3 on the mentioned data set. Furthermore, the estimation accuracy is compared for two different vehicles on the same track and the influence of the combination of multiple estimation results is identified.

To apply the supervised learning methods, explained in subsection 4.2.4, the data need to be manually annotated with the desired output variable, which is time consuming. Therefore, less data were acquired compared to the data set for non-labeled data processing. To ensure a generalization of the presented methods, measurement data were recorded on randomly selected roads in the Karlsruhe area. The velocity, the road conditions, and the environmental conditions of the data set vary in a wide range. In total, reference data are recorded over a distance of 250 km over a period of 6 months. The data were recorded on 14 days by three different drivers. The acquisition of reference data is a lengthy process, as individual attributes, such as potholes, need to be overrun several times with varying approach angles, vehicle lanes and velocities. The results of applying the methods from subsection 4.2.3 on the described labeled data set are shown in subsection 6.3.2.

The variation of vehicle parameters and the influence of this variation is investigated in the extended simulation approach, presented in Section 3.5. Vehicles used for real measurement data acquisition, are replicated in simulation to generate a larger data set. The results and the robustness of the application of the methods from subsection 4.2.3 on the data set generated with simulation and varied paramaters are presented in Section 6.4.

The basis for the multiple vehicle combination, such as the precision matrix, is derived from real measurement data. However, the calculations to combine the classification results are performed offline and results and evaluation of the combination strategies are shown in Section 6.5.

## **6.2 Verification of the developed measurement device**

#### **Inertial sensor**

The performance of the microelectromechanical systems (MEMS) inertial sensor is compared to a dedicated inertial measuring unit (IMU) (OXTS-RT2502 from Oxford Technologies Soluations Ltd.), which is accurate but too expensive for fleet use. Figure 6.1 shows the comparison of the signal of the MEMS inertial sensor and from OXTS-RT2502, which is referred to as reference, for various maneuvers, including slalom, acceleration and deceleration, and a cleat test. The filtered signals are smoothed with a Savitzky-Golay filter with parameters *k* = 7 and *f* = 21 for the vertical acceleration and with *k* = 3 and *f* = 51 for others. Although the vertical acceleration show higher noise (0.1921 m/s<sup>2</sup> ) compared to the reference sensor (0.1428 m/s<sup>2</sup> ), Figure 6.1 indicates that the MEMS sensor is suitable to measure the vehicle motion with a similar accuracy like the dedicated data logger. Table 6.1 shows performance measures for signals of the maneuvers and proves this suggestion.

#### **Acoustic sensor**

Figure 6.2a exemplarily shows the power spectrum from the CMA-4544pf-4w std. from the THD test, which is the basis to calculate the THD value of the sensor. Figure 6.2b shows the SPL for the microphone INMP411 from the excitation of the tire with an impulse hammer. The maximum SPL in the tire cavity was 165 dB. However, the microphone INMP411 shows a plateau with (a) unfiltered signals from longitudinal acceleration of a start and braking maneuver

(b) filtered signals of start and braking maneuver

(c) filteres signals from vertical acceleration of a cleat test maneuver

(d) filtered signals from yaw rate of a slalom maneuver

Figure 6.1: Comparison of signals from MEMS and reference inertial sensor.


Table 6.1: Summary of statistics for reference and MEMS inertial sensor signal.

a SPL of 132 dB, which corresponds to the acoustic overload point of 131 dB specified by the manufacturer.

Figure 6.2: Results from THD and SPL test.

Figure 6.3 shows the frequency response of all microphones during vehicle operation. The figure indicates that the frequency response of the modified conventional electret microphone (CMA-4544pf-w mod.) approximates the response from the reference sensor (PCB 103B02) best.

Figure 6.3: SPL with a 1/12 octave band smoothing of all microphones during vehicle operation with constant speed of 60 km/h.

Table 6.2 lists quantitative criteria of all sensors and tests. The table underlines, that the modified electret microphone measures almost the same sound pressure level (SPL) as the reference microphone. Furthermore, it has a similar total harmonic distortion (THD) value and power spectral density (PSD) values for the test runs with 30 and 60 km/h compared to the reference microphone. Lastly, *R* <sup>2</sup> between the reference and each microphone is highest for the modified electret microphone.

The results of the experiments suggest, that the microphone CMA4544PF-w a modified configuration is a good alternative to the expensive PCB 103B02 to perform acoustic measurements of the tire cavity sound.


Table 6.2: Summary of acoustic and statistics of the experimental examinations.

#### **GPS**

The accuracy of the GPS was determined by stopping a test vehicle five times at the same position. The orthodromic distance, the shortest distance between two points in the surface of a sphere, from the five acquired positions is illustrated in Figure 6.4a and the standard deviation of the distances is 1.12 m. Figure 6.4b shows the comparison of the vehicle velocity obtained from onboard diagnostics (OBD) and global position system (GPS). The signals have a similar graph, the *R* 2 is 1.00 and NRMSE is 1.90. Overall, the GPS sensor has a good accuracy in velocity as well as in position.

#### **Discussion**

A novel measurement device for acquiring GPS, inertial sensor, and acoustic sensor data is presented in this work. It works autonomously and any vehicle can be equipped with this device.

However, inertial sensors are even partly integrated in the standard equipment of new vehicles for fusion with GPS for a more precise position determination. The data from the measuring device used in this work are compared with

Figure 6.4: Orthodromic distance from five GPS signals at the same position and velocity from GPS and OBD .

inertial sensor data from an Audi A7. The sliding standard deviation of the rolling rate, pitch rate and vertical acceleration of both sensors proved to be almost identical with coefficient of determination of 0.9577 on average and a normalized root mean square error (NRMSE) of 0.0416 on average, which we have showed in [185]. Therefore, the developed methods for the inertial sensor can be applied on modern vehicles without the employment of an external measuring device. The advantage of the inertial sensor is that it is inexpensive and already on-board in many new vehicles. In comparison to a camera, inertial sensors can only be used to evaluate the directly traversed lane area. However, data of the inertial sensor already contain information about minor unevenness which causes the vehicle to vibrate, which might be mistaken by the camera. Moreover, road condition estimation with cameras is problematic under varying conditions, such as darkness and moist on the road surface. All in all, a system consisting of inertial sensor is a cost-effective and computationally efficient alternative to cameras to monitor the condition of the road infrastructure. The fusion of the two systems represents a possibility to optimize mobile condition estimation.

# **6.3 Road condition estimation**

## **6.3.1 Road unevenness estimation and trend identification with non-labeled data**

Figure 6.5 shows the estimation of the road unevenness index (RUI) based on the data processing methods presented in subsection 4.2.3. The frame length for the road unevenness index (RUI) estimation is 5 m. The road condition estimation for the highway *B10* in Niefern, Germany, in the figure are based on inertial sensor data acquired with a VW Golf. The estimation shows areas with severe unevenness, colored red. These locations are automatically detected and tagged with a circle with the double thresholding and connected-component labeling algorithms (subsection 4.2.3). The tags are shown in Figure 6.5 and labeled with number 1 to 4. The estimations are compared to the officially determined condition assessment and evaluation 2015 (ZEB).

Figure 6.5: Continous road uneveness estimation in Niefern, Germany.

Label 1 and 2 are bridge gaps, which have a strong excitation of vibration of the vehicle, and Label 3 and 4 are severe road surface defects. Overall, the coefficient of determination for the estimated RUI compared to the ground truth for the whole test drive around Niefern is 54 % and for the unweighted aggregation of various drives 63 %, which we have showed in [185]. The coefficient of determination for a different vehicle, a BMW 116d, for the same test track is 50 % [185]. The coefficient of determinations show, that the estimation can explain a large portion of the ground truth and that the aggregation of various drives improves the estimation result. Furthermore, the estimation seems to be stable for different types of vehicles. The *R* <sup>2</sup> of RUI estimation of the autobahn (A6) close to Niefern with the VW Golf yields 45 %, which corresponds to the estimation accuracies for highway B10. The estimation accuracy based on a camera is higher for B10 with 76 % and similar for A6 with 47 %[185].


Table 6.3: Automatically detected hotspots with severe unevenness.

Table 6.3 exemplarily shows the output of the toolbox with the automatically detected severe unevenness hotspots. In addition to maximum and on average estimated RUI, max(RUI) and µ(RUI), of the corresponding hotspot, the coordinates of the midpoint of the hotspot, long. and lat., are displayed. Furthermore, the radius of the hotspots*r* and the average and standard deviation of the velocity of vehicles driven over the hotspots, µ(velocity) and σ(velocity), and the amount of drives are automatically calculated.

Figure 6.6 exemplarily shows the estimation of the RUI for a greater road network based on data acquired with Ford Galaxy between 2016 and 2017. The Labels 1 to 3 show road segments with severe unevenness, which are confirmed by the driver.

The former examples show the advantage of the developed methods and toolbox. The data are automatically acquired, processed, and visualized. The maps with the colored estimation of the road condition and the table with hotspots give a comprehensive overview of the road condition. However, the output is only an unevenness index, without knowledge of the actual attribute on the road. The results of the prediction of such attributes are shown in subsection 6.3.2.

Figure 6.6: Continous road uneveness estimation in Karlsruhe, Germany.

Figure 6.7 shows the results of the trend identification method for RUI for Karlsruhe, Germany, and surrounding area. The trend identification method, which is presented in Subsection 11, estimates if the condition of a road segment decreased or increased over time. The figure indicates, that the quality of the road infrastructure remained constant except for small road segments in the city and larger areas marked with Label 1 and 2. With such an output, road infrastructure engineers can identify road segments, where the condition greatly decreases in short time intervals to identify the reason and to take countermeasures. The reasons for such decreases in road condition might be a high portion of heavy goods vehicles or mistakes during the construction of the road.

Figure 6.7: Trend identification in and around Karlsruhe, Germany.

## **6.3.2 Road attributes and surface estimation with labeled data**

This work distinguishes between the classification of attributes and road surfaces. The classification of road surfaces aims to give a rough overview of the condition of the road network. Hereby, the length of each road segment to be classified is longer, 50 to 100 m, compared to the segment length for attribute classification, where it is 5 to 10 m long.

Road surface classification separates the classes *smooth asphalt and concrete*, *damaged asphalt*, *damaged concrete*, *cobblestones/off-road*. Road attribute classification incorporates the classes *potholes*, *manhole covers*, *railways crossings*, *speed bumps*, *light damages*, *good condition*.

Figure 6.8 illustrates the classification results of both approaches in aggregated features space. Table 6.4 shows the performance measures (subsection 2.5.6) of the classification.

Figure 6.8: Classification results with two aggregated features and borders of the classifier in black.

The results, including Figure 6.8, and visualized examples for both examples are explained in more detail in Subsection 6.3.2, as well as the regression and visualization of the roughness estimation of road surfaces based on sound data of tire cavity.

#### **Attribute classification**

The cross-validated accuracy of classifying attributes yields 81 % on average without feature aggregation. The most important features to separate the classes, determined with multivariate analysis of variances (MANOVA) are:


Table 6.4: Results of single features classification.


Previous analysis have shown that the vehicle velocity has a high influence on the sensor data and is therefore selected as an important feature for the classification. The aggregated feature space, aggregated with linear discriminant analysis, and the lines of the function to classify the attributes are shown in Figure 6.8b.

The illustration of the classification shows, that road segments in *good condition*, with *light damages*, *speed bumps*, and *potholes* can be separated well. This indication is proofed by the quantitative results, listed in Table 6.4. The precision and recall for *good condition*, *light damages*, *speed bumps*, and *potholes* is above 70 %, whereas the performance measures for *manhole cover* and *railway crossing* is below 62 % on average.

By comparing each class with each other, it emerges that the peak-to-peak values of pitch and roll acceleration are mainly responsible to separate attributes, which occur on


In addition, the average RMS of the vertical acceleration is important to separate light damages and road segments in good condition. Furthermore, *potholes* and *manhole covers* are dividable by the maximum RMS of the roll acceleration in the frequency range 15 to 25 m−<sup>1</sup> . However, latter attributes are often misclassified as segments in *good condition* or *light damages*. *Speed bumps* and *railways crossings* are separable by the value of the peek-to-peek of the pitch rate, whereas *railways crossings* are also often misclassified as light damages.

Figure 6.9: Attribute classification results of two high speed roads in the south west of Karlsruhe, Germany.

To test the classifier, a data set of more than 200 km of real data is classified and plotted on maps based on open street map (OSM). The results are promising and represent the actual street condition in many occasions. A few examples of classified areas are shown below.

The first example shows the attribute classification results on two different high speed roads (Figure 6.9). The upper one with Label 1 is a freshly renovated asphalt highway with close to no damages and the lower one with Label 2 is a poorly patched asphalt road with a lot of medium and severe damages. The classification successfully predicted the upper roadway as *good street*. Most parts of the lower street were predicted as *light damage* and some points even as *potholes*. The results conform to the actual road condition.

Figure 6.10: Attribute classification results for road segments in the city of Karlsruhe, Germany.

The second example presents data acquired in an urban area in Karlsruhe, the predictions are shown in Figure 6.10. The roads in this area are poorly preserved and there is a *speed bump* at a pedestrian crossing (Label 1). The classification model correctly predicts the *speed bump* (Label 1) for all overruns and a *pothole* (Label 2).

The third interesting sector is shown in Figure 6.11. *Potholes* (Label 2 and 3), which were at the edge of the driving line, were overrun multiple times and the classifier predicts the severe damage accordingly. Sometimes the output at the road segments is not *pothole* but *light damages* or even *good road condition*. The reason might be that the pothole was avoided by the driver.

Figure 6.11: Attribute classification results of road segments outside of Karlsruhe, Germany, for the events *potholes* and *railways crossing*.

The *railway crossing* (Label 1) is more elevated than other crossings and misclassified as *speed bump* in few cases.

#### **Road surface classification**

When classifying road surfaces, cross-validation yields 96.1 % accuracy on average without aggregation of features. The three best individual features for the classification of road surface according to MANOVA are:


The aggregated feature space is shown in Figure 6.8a. The figure indicates, that the biggest portion of misclassifications results from asphalt classified as *damaged asphalt* or *damaged concrete* and vice versa. The illustrated results are underlined by Table 6.4, where the precision and recall for *cobblestone* is above 99.0 %, whereas the performance measures for *asphalt*, *damaged asphalt*, and *damaged concrete* are between 92.0 and 97.6 %.

The values of RMS of the roll acceleration and vertical acceleration separate the classes *smooth surfaces*, *damaged asphalt* and *cobblestone*. The values are highest for *cobblestone* and low for *smooth surface*. Standard deviation of pitch rate separates the classes *damaged concrete* from all other classes. The reason is probably poor and aged concrete joints.

The surface classifier was applied on the same data set described for attribute classification. The classifier was able to reflect the road surface precisely. The following figures display the performance on different surfaces. Analogous to the attribute classification shown in Figure 6.9, the surface classifier could distinguish between both pavements and correctly classified them as *smooth surface* and *damaged asphalt*, respectively.

Figure 6.12: Road surface classification of road segments in urban area.

The classification results from data acquired in the urban area of Karlsruhe (Figure 6.12) show two correctly predicted areas of cobblestone (Label 1 and 2). The remaining road segments are correctly classified as segments with *light damages* or in *good condition*. Especially the latter class was correctly predicted for a road segment, which was recently renewed (Label 3). One misclassification of *cobblestone* can be found close to Label 1. However, this road segment is highly damaged with various potholes, which have a high impact on the vehicle vibration similar to cobblestone.

Figure 6.13 shows highway segments with aged concrete and distinctive concrete joints (Label 1 and 2), which have to be maintained shortly. Except for one short segment, which was classified as *smooth surface*, the road state was correctly predicted.

Figure 6.13: Road surface classification of Karlsruhe, Germany, and highways with different surface types.

#### **Road roughness estimation**

Figure 6.14: Estimated regression function for road roughness index (RRI).

Figure 6.14 shows the observations and estimated regression function for a portion of the acquired data. The regression error, expressed as normalized mean square error, of a five fold cross-validation of all acquired data is 4.54<sup>h</sup> on average and the standard deviation is 0.15h.The figure and the calculated error indicate that the regression function is able to reproduce the actual observations very well and it can also be used for the estimation of road surface roughness for future measurements.

For example, Figure 6.15 and 6.16 show the estimated roughness based on acoustic data acquired with BMW 116d with varied velocities. Figure 6.15 shows the roughness on a highway from Karlsruhe to Niefern, Germany. Changes of road roughness (Label 1 and 2) are visible along the highway, which correspond to the actual roughness condition.

Figure 6.16 indicates, that the roughness at the crossover in Niefern (Label 1), at which traffic is very heavy, is coarser compared to other road segments in Niefern, Germany. Furthermore, the highway leading to Niefern (Label 2) is new, which is represented by the results of this analysis.

Figure 6.15: Road roughness estimation on highway from Karlsruhe to Niefern, Germany.

Figure 6.16: Road roughness estimation in Niefern, Germany.

### **6.3.3 Discussion of road condition estimation**

The results (Section 6.3) of the non-labeled data processing of the vehicle body vibration and the tire vibration give a good estimation of the degree of the road unevenness and road roughness. The achieved coefficient of determination for two different vehicles is even comparable to results from recent studies, but those involve a training process, e.g. [90]. The output of the data processing, e.g. the maps or tables with hotspots, provide a comprehensive overview of the condition of the road infrastructure. Road inspectors can identify road segments in poor conditions and can structure and prioritize the maintenance of the infrastructure.

In contrast to supervised learning methods, a costly labeling process for this approach is not needed. However, the method can not differentiate, if the vibration of the vehicle or the tire originate from road defects or from structural obstacles. But if the vibration of the vehicle is high and the RUI accordingly close to 1 due to a striking structural obstacle, a maintenance might be necessary even it is not a defect. Examples are railway crossings or bridge expansion joints, which are corrupted.

Moreover, the thresholds for the algorithm to identify road segments with severe excitation on the vehicle are adjustable based on the purpose of the road inspector. The estimator of the RUI is dependent on the vehicle, since different vehicles have various suspension systems and vibration characteristics. However, the estimates can be merged across different vehicles and trajectories.

An extension or specification dependent on the vehicle of the feature set might improve the estimator. For example, the features based on the pitch rate can be considered. Moreover, the lower and higher frequency thresholds for the calculation of the RMS with can be specified for vehicles to decrease the portion of frequencies not relevant of road unevenness. However, this requires an extensive study of the vehicle vibration characteristic due to obstacles and a defined test track with various defects.

The results of the data processing for the inertial sensor data show that both, road materials and attributes can be classified. The features automatically selected by MANOVA are in agreement with the theory of vehicle excitation. Surface type classification performs well according to the results of the crossvalidation and the test data.

There are various misclassifications among the prediction of attributes, especially for structural obstacles, such as *manhole covers* and *railway crossing*. However, these attributes might be marked on a map and excluded from classification and investigation, as the main objective is to detect road damage.

One reason for misclassifications of the attributes *good condition*, *light damages*, and *potholes* might be false manual annotating, since there is sometimes only a fine line between the degree of damages, or the events were not fully overrun, especially for *potholes*.

The inertial sensor and the developed data processing flow can be adapted to detect other road features, which might be interesting for transportation authorities. Road features, which might be classified with a good accuracy, are wavelike deformations in longitudinal direction.

Other road infrastructure information, such as road slope or radius of curves, can also be estimated solely based on inertial sensor data, as we showed in [112, 186]. For this purpose, features from other time series, such as yaw rate, longitudinal and lateral acceleration of the vehicle, are more important than time series used for the estimation of road condition.

The data processing of acoustic sensor data can be extended to estimate the tire road noise. In general, the degree of road roughness is positive correlated with tire road noise. However, there are exceptions, such as a open-pored road surface, which absorbs sound and reduces the tire road noise. Moreover, the microphone inside the tire cavity might also be useful to detect extended water residues on the road surface.

Overall, the methods can be tuned and the results can be improved with novel demonstration, examination, and reference areas, such as *duraBASt*. Ground truth data for different surface textures for the estimation of the RRI and of different longitudinal unevenness for the estimation of RUI can be obtained with lower effort under controlled conditions. Moreover, ground truth data for defined road defects, which can be modified over time, can be acquired under defined environment conditions, such as rain and temperature.

### **6.3.4 Discussion of online estimation**

The data processing presented in this work can probably run on electronic control units (ECUs) of modern vehicles. However, the ECU has limits in computational resources, such as working memory. The results of MANOVA of the labeled inertial sensor data processing show that the most important features are peak-to-peak or standard deviation of specific space series data. The major task is to avoid the calculation of features for windows, since it is costly to keep all values. There exist online algorithms, especially for the calculation of mean and variances [187, 188], which can be applied to run the data processing on ECUs. Algorithms for the online calculation of other features such as peaks, can be adjusted from ECG signal analysis, e.g. [189]. Furthermore, there exist approaches for real-time frequency and harmonic evaluation, e.g. [190, 191].

## **6.4 Influences on classification performance**

### **6.4.1 Variation of vehicle parameters**

The classifier is trained with data, which is simulated from the BMW 116d passive full car model driving on a 4 *km*-long road with fixed size of attributes. Figure 6.17 shows the classes in feature space, described by two aggregated features from 15 single features. The class *pothole* has two clusters because two different sizes of potholes, each with smaller variations in dimensions, are simulated, which have different impacts on the vehicle vibration. The figure indicates, that the road features *pothole* and *cobblestone* are clearly separable to the other classes. *Unevenness* is close to the road features *railway crossing* and *cobblestone* but still good dividable. However, *railway crossing*, *manhole cover* and *smooth surface* are very close and show some misclassifications. The reason is that the vehicle vibrations due to these road features in simulation are similar.

The results of testing the classifier for different types of vehicles and vehicle set-ups are shown in Table 6.5. Overall, the generalization error remain good with a maximum of 10.2 % for S-Class.

The classifier from BMW 116d is used to predict unseen data, which are simulated from models of different vehicles, loads and suspensions driving over a 2 *km*-long road with random size of attributes. The testing results are shown in Table 6.6. It is obvious that the classifier performs good with the same vehicle model. However, the accuracy of the classifier declines from 90% to 81% with the increasing load on the vehicle. For other vehicles and set-ups, the performance measures decline greatly, which means the mass, size and

Figure 6.17: Classes separated by two aggregated features from 15 single features.


Table 6.5: Performance of classifiers for different vehicles.

parameters of spring and damping have an individual effect on the accuracy of the classifier. The results indicate, that the classifier, which is trained with BMW 116d, is not robust and the performance for the application on other vehicle set-ups greatly declines.

The performance measures improve, if the original classifier is trained incorporating the parameters and set-ups of the vehicles as features. Therefore, three vehicle model variations are considered as features, the load of the vehicle, the model of the vehicle and the type of suspension. The new classifier is trained on a road of length 180 km with a wide variation of vehicle loads, models, and type of suspension. Analogous to above, this classifier is tested on the presented vehicle models. The results (Table 6.7) suggest that this classifier is more robust against different vehicle models compared to the classifier solely


Table 6.6: Performance of the classifier for BMW 116d applied on different vehicle set-ups and vehicles.

based on data from one specific vehicle. However, the performance measures for slightly changes of the set-up, e.g. the increase of load, decrease compared to results shown in Table 6.6. Overall, the lowest value for performance measures is 73.0 and the highest value for generalization error is 24.5 %.


Table 6.7: Performance of the classifier for various vehicle models applied on different vehicle set-ups and vehicles.

### **6.4.2 Variation of classification parameters**

The influence of the position of the inertial sensor in the vehicle body in the simulation is shown in Figure 6.18. The best accuracy can be achieved with output obtaining from position S2, namely in the middle above the front or rear axle.

Moreover, the number of selected features and the order of the polynomial kernel also have an effect on the classification. In the practical application

Figure 6.18: Influence of the sensor position on the accuracy of the classification.

on the vehicle, the classifier should at least maintain at a high level no matter how the load of the vehicle changes. Thus each data flow contains the output simulated by a selected vehicle model driving across different road attributes with a series of loads from 0 kg to 400 kg. The features are selected with MANOVA and the number of features is set from 10 to 30. The classification is performed with support vector machine (SVM) with a polynomial kernel with orders 1 to 5. The results, respectively for the BMW 116d with anti-roll bar, are shown in Table 6.8. The results suggest that the maximum accuracy can be achieved with a kernel order of 2 and 25 road features, which leads to 13.85 % misclassification. Using 25 features and kernel order 3 is the second best option with 14.38 % misclassification. The results indicate that the classes are better separable with nonlinear classification functions. Furthermore, a higher amount of features, e.g. 30, seems to lead to over-fitting.


Table 6.8: Misclassifications in % of classifier with varying kernel order and number of features.

### **6.4.3 Discussion of extended simulation approach**

The proposed extended simulation approach enables to investigate the influence of parameter variation of vehicles on the classification performance. The simulation consists of a full car model with 10 degrees of freedom (DOF), various extensions, such as active suspension, a modified point contact tire model, dependent on vehicle velocity and length of obstacles, and road models including road features and defects. It is shown that it matches actual measured data, and therefore is a good representation of the real-world. Supervised machine learning techniques involve a costly and laboriously collection of labeled data for training and testing. The simulation reduces these costs, and it can be used to investigate and quantify the impact of specific vehicle parameters and settings on the classification results. Furthermore, the best parameters for the data processing and classification can be determined or tested.

Further improvements are to incorporate additional elements of a real vehicle into the vehicle model to the point of a commercial complex simulation, such as CarMaker [192]. With the exact set up and behavior of real vehicles in the simulation, classifiers can be trained in simulation before assigning the classifier to real vehicles.

## **6.5 Multiple vehicle combination**

The performance results of the proposed combination strategies for different scenarios, which are described in subsection 4.4.3, are shown and discussed in this section. Figure 6.19a shows the difference between the estimators for **Scenario 1**, where the precision matrix for the combination strategy and for all vehicles is identical. The *vote with precision matrix*, which incorporates the precision derived from the confusion matrix for all classes, is the one with fewest wrong predictions. Already with one vehicle, the proportion of wrong predictions is below 20 %, which means that each segment is classified correctly on average. With increasing number of vehicles, the number of wrong predictions for *vote with precision matrix* decreases faster compared to *majority vote* or *diagonal vote*. Between latter two estimates no big difference can be identified. Figure 6.19b represents the results of the prediction based on the vote with precision matrix split into the classes. The result shows a high impact of the class 5, *C*5, on the number of wrong predictions because the precision of the vehicles to predict class 5, π55, is only 0.73 and lower compared to other classes (Table 4.11).

Figure 6.19: Results of vehicle combination: Scenario 1. Each vehicle predicts the ground truth based on the global precision matrix.

In **Scenario 2**, each vehicle has an unknown precision matrix but the combination of the classifications is still performed with the global precision matrix (Table 4.11). As shown in Figure 6.20, again *vote with precision matrix* is the most promising estimate. When comparing one vehicle, which drives multiple times over the road and classifies the segments (Figure 6.20a), and multiple vehicles (Figure 6.20b), the latter one achieves better results. The reason is that the fixed precision matrix of only one vehicle can not compensate the difference to the global one and the proportion of wrong predictions converges towards 10 %. However, five or more different vehicles perform better and surpass 10 % (Figure 6.20b).

Figure 6.21 shows the influence of a vehicle with *broken sensor* from **Scenario 3** on the number of wrong predictions. In Figure 6.21a, the first vehicle has or has not an equally distributed precision matrix and therefore randomly predicts the classes. Above five vehicles, in average less than 10 % of the road is classified wrongly. Above ten vehicles the impact of the bad precision matrix can be ignored. Figure 6.21b shows the results of a simulation, in which the third vehicle has an equally distributed precision matrix, which is referred to as *broken sensor*. The blue line shows the proportion of misclassifications with the third vehicle with a *broken sensor* and no adjustments. As soon as the output of the third vehicle is included in the simulation, the number of wrong

Figure 6.20: Results of vehicle combination: Scenario 2. Unknown distribution of classification due to different precision matrices for vehicles.

classifications increases greatly. With increasing the number of vehicles with good precision matrices, the number of wrong predictions converges again to 0. The green line shows the results of the simulation with the identification of the vehicle with the bad precision matrix and the adjustment of the precision matrix in the back-end for this vehicle with the proposed Algorithm 4.4 in subsection 4.4.3. With this adjustment, the increase of number of wrong predictions is not as high as without adjustment and the number of misclassifications faster converges to 0. Summarized, this method improves the results of our multiple vehicle fusion and can identify vehicles with bad classification outputs.

Figure 6.21: Results of vehicle combination: Scenario 3. Classification with and without broken sensor. a: 1st vehicle with and without broken sensor. b: 3rd vehicle with broken sensor with and without automated adjusting precision matrix for this vehicle.

The results of **Scenario 4**, where the ground truth changes at one time sample, can be seen in Figure 6.22. After the change of the ground truth at time sample *t* = 5, the *vote with precision matrix* without forgetting needs 4 more time stamps to reduce the proportion of wrong classified segments below 10 %. When using the forget factor, already in the next time stamp an error rate below 2 % can be reached. In contrast, also correct classifications and good results are forgotten by the algorithm. However, the impact is small, which shows the results at time sample *<sup>t</sup>* <sup>=</sup> <sup>2</sup>,3,4, where the green line is only slightly above the blue line.

Figure 6.22: Results of vehicle combination: Scenario 4. Change of the road conditions and simulated with and without forgetting of previous classifications.

#### **Conclusion and discussion**

With the presented methods to identify trajectories from different vehicles for each road road segment, an automatic annotating of sensor data with the label of the ground truth is possible. Hereby, the label of road segments, known or predicted from already trained vehicles, is transferred to new and untrained vehicles. For this purpose, the trajectories of multiple vehicles are compared and road segments, driven by various vehicles, are identified. Problems can occur with poor GPS signal. However, this can be avoided by applying an integrated navigation system, which is based on the sensor data fusion of GPS and MEMS-IMU to provide an accurate position of the vehicle [193, 194]. Modern vehicles are already equipped with integrated navigation systems. Furthermore, the difference of the time duration for obtaining the ground truth of a specific road segment and the time when a new vehicle drives this road segments and receives the label should be small. In the meantime, the ground truth could be changed, e.g. the condition of the road segment is significantly decreased or maintained. The investigation of methods to combine the prediction of various vehicles for specific road segments show, that the combination based on the precision matrix of the classifier outperforms the combination based on *majority vote*. Various extreme scenarios are simulated, e.g. a sudden change of the ground truth or wrong predictions of a vehicle, to quantify the impact on the classification results. Moreover, methods to overcome these extreme situations are presented. The combined processing is mainly performed in simulation. The next steps are to apply these methods on a fleet of vehicles.

# **7 Conclusion**

This work presents novel methods to comprehensively and automatically estimate the road condition with vehicle based sensors. To acquire the data from the sensors, a novel measurement system was developed [195]. The installation and operating process of the measurement system requires almost no manual intervention and the data can be automatically transferred to a central data base. A road unevenness index (RUI) is estimated with an inertial sensor in the vehicle body and road roughness index (RRI) with an acoustic sensor in the tire cavity. The estimation of the indices are based on the measurement of the vibration of the vehicle and tire with the mentioned sensors. It was shown from experiments under controlled conditions, that the vibrations are not only dependent on road roughness or unevenness but also on influences from the environment or vehicle operation, especially velocity. To decrease the effect of these influences, the indices are estimated for specific bins or intervals, which are constructed based on the main influences. The comparison with ground truth data suggest that the estimators conform with the actual road condition. The estimators of various vehicles can be combined and visualized on maps. Furthermore, the trend of the road condition can be identified with a robust regression over time for each road segment to be investigated.

Supervised learning data processing is introduced for the inertial and acoustic sensor to classify road features and road roughness. The major novelties include a feature selection with multivariate analysis of variances (MANOVA) to reduce the complexity and avoid over-fitting of the classifier. Two classifiers for the inertial sensor data are developed to estimate both, road surface type and road attributes. The prediction of the methods were observed and the results suggest a good performance.

Moreover, methods are introduced to realize an automatic transmission of labels to new measurement vehicles to train their classifier. With the same methods, road segments and lanes can be identified which are driven by various vehicles and methods to combine the output of these vehicles are investigated to improve the prediction rate. Overall, the methods are combined in a Matlab toolbox and graphic user interface for evaluation of data sets with alternative parametrization by non-experts.

Summarized, this work achieves the following goals


The developed methods can improve or partially substitute the current practice for road infrastructure evaluation to increase the automation rate and to reduce the costs and response rate. The results of the developed methods show that the inertial sensor and acoustic sensor represent a very good option to collect information of the tire road contact at low costs and over wide area. The data processing methods can run on electronic control units (ECUs) of modern vehicles, which serve as sensor platforms. Moreover, as the developed system is of modular design, the number and type of sensors and sensor modality can be varied. Lastly, a vehicle fleet can be equipped with the presented system as it is to estimate the condition of the road infrastructure. For example, the German Federal Environment Agency currently deploys seven measuring devices, developed in this work, and applies the presented methods, to automatically and comprehensively monitor the road condition.

# **List of Figures**




# **List of Tables**



# **List of Algorithms**


# **Bibliography**


## **Supervised seminar, bachelor, and master thesis**


# **Publications**

## **Peer-reviewed journal papers**


<sup>\*</sup> contributed equally


## **Peer-reviewed conference papers**


[195] J. Masino, M. Frey, F. Gauterin, and R. Sharma, "Development of a highly accurate and low cost measurement device for Field Operational Tests", in *IEEE International Symposium on Inertial Sensors and Systems*, Piscataway, New Jersey, USA: IEEE, 2016, pp. 74–77. doi: 10.1109/ISISS.2016.7435548.

## **Other**


## **Karlsruher Schriftenreihe Fahrzeugsystemtechnik (ISSN 1869-6058)**

Herausgeber: FAST Institut für Fahrzeugsystemtechnik


Die Bände sind unter www.ksp.kit.edu als PDF frei verfügbar oder als Druckausgabe bestellbar.


Die Bände sind unter www.ksp.kit.edu als PDF frei verfügbar oder als Druckausgabe bestellbar.









ISBN 978-3-7315-1004-8

## Karlsruher Schriftenreihe Fahrzeugsystemtechnik

The work provides novel methods to process inertial sensor and acoustic sensor data for road condition estimation and monitoring with application in vehicles, which serve as sensor platforms. Furthermore, methods are introduced to combine the results from various vehicles for a more reliable estimation.

A systematic, continuous, and comprehensive monitoring of existing roads is becoming increasingly important. Yet, current practice is laborious and time-consuming as most steps of the process are done manually. A novel measuring device for vehicles is developed to autonomously acquire a large set of real world data on public roads. In addition, an extended simulation approach is presented to record data under controlled conditions to investigate the impact of parameter variations. Furthermore, the methods for data processing are implemented in a user friendly Matlab toolbox to allow an automatic evaluation.

Overall, the developed methods can improve or partially substitute the current practice to evaluate the road infrastructure. This work contributes concepts and methods to allow an automatic and comprehensive road condition monitoring. The data processing methods can run on electronic control units of modern vehicles, which are equipped with corresponding sensors. However, the presented system consisting of the developed measuring device and Matlab toolbox can also be applied on vehicle fleets as it is.

ISSN 1869-6058 ISBN 978-3-7315-1004-8 J. Masino

Road Condition Estimation with Vehicle Based Sensors

**Band 82**