**Hamdi Aloulou · Bessam Abdulrazak · Antoine de Marassé-Enouf · Mounir Mokhtari (Eds.)**

# **Participative Urban Health and Healthy Aging in the Age of AI**

**19th International Conference, ICOST 2022 Paris, France, June 27–30, 2022 Proceedings**

# **Lecture Notes in Computer Science 13287**

#### Founding Editors

Gerhard Goos *Karlsruhe Institute of Technology, Karlsruhe, Germany*

Juris Hartmanis *Cornell University, Ithaca, NY, USA*

#### Editorial Board Members

Elisa Bertino *Purdue University, West Lafayette, IN, USA*

Wen Gao *Peking University, Beijing, China*

Bernhard Steffen *TU Dortmund University, Dortmund, Germany*

Moti Yung *Columbia University, New York, NY, USA* More information about this series at https://link.springer.com/bookseries/558

Hamdi Aloulou · Bessam Abdulrazak · Antoine de Marassé-Enouf · Mounir Mokhtari (Eds.)

# Participative Urban Health and Healthy Aging in the Age of AI

19th International Conference, ICOST 2022 Paris, France, June 27–30, 2022 Proceedings

*Editors* Hamdi Aloulou Digital Research Centre of Sfax Sfax, Tunisia

Antoine de Marassé-Enouf Agence du Numérique en Santé Paris, France

Bessam Abdulrazak Université de Sherbrooke Sherbrooke, QC, Canada

Mounir Mokhtari Recherche et de l'Innovation Institut Mines-Télécom Paris, France

ISSN 0302-9743 ISSN 1611-3349 (electronic) Lecture Notes in Computer Science ISBN 978-3-031-09592-4 ISBN 978-3-031-09593-1 (eBook) https://doi.org/10.1007/978-3-031-09593-1

© The Editor(s) (if applicable) and The Author(s) 2022. This book is an open access publication.

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

### **Preface**

This year we organized the 19th edition of the International Conference on Smart Living and Public Health (ICOST 2022), a series which has succeeded in bringing together a community from different continents for almost 20 years and raised awareness about frail and dependent people's quality of life in our societies.

After 18 very successful conferences held in France (2003, 2009, 2017), Singapore (2004, 2013, 2018), Canada (2005, 2011), Northern Ireland (2006), Japan (2007), the USA (2008, 2014, 2019), South Korea (2010), Italy (2012), Switzerland (2015), China (2016), and Tunisia (2020), this 19th edition of the conference was organized by the Institut Mines-Télécom (IMT), Paris, France, and held during June 27–28, 2022. The conference was organized as an official event of the French Presidency of the Council of the European Union. The theme of the conference was "Participative Urban Health and Ageing Well in the Age of AI."

ICOST 2022 provided a premier venue for the presentation and discussion of research on the design, development, deployment, and evaluation of AI for health, smart urban environments, assistive technologies, chronic disease management, and coaching and health telematics systems. ICOST 2022 aimed to understand and assess the diverse and disparate impact of digital technologies on public health in developing and developed countries. It brought together stakeholders from health care, public health, academia, and industry along with end users and family caregivers to explore how to utilize technologies to foster health, disease prevention, and independent living and offer an enhanced quality of life. The ICOST 2022 conference featured a dynamic program incorporating a range of oral and poster presentations along with panel sessions.

ICOST 2022 was proud to extend its hospitality to an international community consisting of researchers from major universities and research centers, representatives from industry, and users from 17 different countries. We would like to thank the authors for submitting their current research work and the Program Committee members for their commitment to reviewing submitted papers. The ICOST proceedings and chapters have now reached over 400,000 downloads and are in the top 25% of downloads of Springer LNCS. We are extremely thankful to our sponsors for their commitment and support to the vision and mission of ICOST.

June 2022 Hamdi Aloulou Bessam Abdulrazak Antoine de Marassé-Enouf Mounir Mokhtari

### **Organization**

### **General Chair**

Mounir Mokhtari Institut Mines-Télécom, France

### **General Co-chair**

Antoine de Marassé-Enouf Agence du Numérique en Santé (French eHealth Agency), France

### **Scientific Advisory Board**


### **Program Committee Chairs**


### **Program Committee**

Afef Mdhaffar University of Sfax, Tunisia David Menga EDF R&D, France

Hisato Kobayashi Hosei University, Japan Houssem Aloulou University of Sfax, Tunisia Jean Marie Bonnin IMT Atlantique, France

Aitor Almeida University of Deusto, Bilbao, Spain Armel Ayimdji Tekemetieu Université de Sherbrooke, Canada Belkacem Chikhaoui University of Quebec, Canada Boussada Rihab University of Manouba, Tunisia Elyes Lamine IMT Mines Albi and ISIS Castres, France Eric Campo LAAS-CNRS, Université Toulouse-Jean Jaurès, France Hassan Mostafa Ahmed Fahmi Université de Sherbrooke, Canada Hongbo Ni Northwestern Polytechnical University, China Hui Chen Université de Sherbrooke, Canada Jeffrey Soar University of Southern Queensland, Australia Jérôme Boudy Télécom SudParis and Institut Polytechnique de Paris, France Laurent Billonnet University of Limoges, France Laurent Clavier IMT Nord Europe, France Manfred Wojciechowski University of Applied Sciences Dusseldorf, Germany Meriem Zerkouk University of Science and Technology of Oran, Algeria Salim Hima ESME-Sudria, France Sergio Copelli MultiMed Engineers srls, Italy Stefanos Kollias University of Lincoln, UK Valérie Gay University of Technology Sydney, Australia Vigouroux Nadine Institut de Recherche en Informatique de Toulouse, France Wael Sellami University of Sfax, Tunisia Xiaolan Xie École des Mines de Saint Étienne, France Zuraimi Sultan Berkeley Education Alliance for Research in Singapore (BEARS), Singapore

### **Organizers**

Institut Mines Télécom, France Université de Sherbrooke, Canada Digital Research Center of Sfax, Tunisia National University of Singapore, Singapore Nanyang Technological University, Singapore New York University, USA

### **Sponsor**

Agence du Numérique en Santé (French eHealth Agency), France

### **Contents**

#### **IoT and AI solutions for E-health**



#### **E-health Solutions for COVID 19**



# **IoT and AI solutions for E-health**

# **Self-healing Approach for IoT Architecture: AMI Platform**

Bessam Abdulrazak, Josué Ayi Codjo, and Suvrojoti Paul(B)

AMI-Lab, Dept. Infromatique, Faculty of Sciences, Université de Sherbrooke Research Center On Aging, Sherbooke, Canada {Bessam.Abdulrazak,Josue.Ayi.Codjo, Suvrojoti.Paul}@usherbrooke.ca

**Abstract.** The fast growth of the IoT and the unlimited possibilities in terms of applications and processing brought forth by the 5G which is around the corner is making IoT an active part of the activity of daily living. Those massive architectures become then the target of security issues, broken services, broken hardware or malfunctioning of applications. Moreover, in a system of million connected devices, operating each one of them is impossible making the platform unmaintainable. In this study, we present our attempt to achieve the autonomy of IoT infrastructure and we present some of the existing and recurrent issues undermining the IoT architecture. Then we review existing self-healing techniques that enable a system to be autonomous and solve issues. We also described our IoT platform that targets the self-healing concern. Finally, we point out some recommendations to make an overall reliable, resilient IoT system with cognitive entities for self-management.

**Keywords:** IoT · Responsibility · Self-healing · Automation · Reliability · Networking

### **1 Introduction**

An extensive growth in the development of the Internet of Things (IoT) in recent times has enabled the possibility of gathering a large amount of data from every aspect of life. This rapid growth in every sector imposes several challenging operational issues. It is possible to deal with these issues for a limited number of IoT devices however extremely difficult in case of a large number of devices. We argue that the self-healing approach can efficiently handle operational issues generated when various IoT devices are incorporated into an architecture.

The global IoT market size is forecasted to grow around 1.6 trillion by 2025 [1]. These millions of connected devices create a massive amount of data and unlimited possibilities. Foreseeing that infinite connection allows IoT to bring its benefits in every aspect of life (e.g., in domains such as healthcare and industrialization). IoT-based systems have an important role in improving health-related quality of life. Still, they introduce various issues that require immediate attention from developers before considering a large-scale deployment. IoT solutions, e.g., used in activity trackers, and sleep trackers aim at abstracting daily routines to offer better services. Albeit such solutions promote people to improve their typical daily activities, they are recurrently exposed to security breaches due to exposing acquired data to online services [2]. The IoT security aspect in today's world is the number one operational issue addressed in literature [3]. Researchers proposed various self-protection solutions as countermeasures for reverting a system to a normal state [3]. Still, managing and handling a large number of devices impose challenging operational issues, e.g., energy depletion (battery saving), broken communication (network connectivity), operating environment effects (overheating, cybercrime, storage management), discontinuity in services. Thus, these issues need to be addressed for every single device within an IoT infrastructure. Self-healing is an optimal approach for mitigating such issues with the ultimate goal to build reliable systems. It is related to a set of concepts (e.g., resilient systems [4], cognitive systems [5], responsible and autonomous systems [6]). It is a property of the system to identify and diagnose (i.e., independently) breakdowns and to autonomously determine and implement proper mitigation strategies. More specifically, self-healing confers system reliability through responsibility and awareness of the environment, issues, and expected countermeasures during unanticipated situations. Therefore, a self-healed IoT system should incorporate monitoring, awareness, and knowledge for detecting unforeseen states. Upon detecting a conceivable issue, the system initiates optimized planning for executing proper actions.

The contribution of this study is twofold. First, highlighting the self-healing concerns of IoT solutions. Second, providing an insight into how a self-healing system should behave. We strive to achieve a self-healing system capable of reasoning over security, software, hardware, networking concerns and reacting in a proper way to keep the overall system on tracks without much human intervention. We consider, in our research, the capability of the system to sense, be aware of its environment and take proper actions for overcoming a problem.

The rest of the paper is organized as follows: Sect. 2 introduces the IoT architecture and existing issues. Section 3, present existing self-healing solutions. Section 4 depicts an IoT based case study. Section 5 describes the results of the experiments and evaluations conducted in order to validate the functionality of the system. Section 6 highlights our discussions for a reliable system. We finally outline the conclusion in Sect. 7.

### **2 IoT Architecture and Existing Issues**

We introduce in this section an IoT architecture to better illustrate the operational issues and the required intervention at diverse levels (based on our AMI-Architecture [7]) (Fig. 1). **\*** The Edge (or Device) layer (i.e., the closest layer to end-users/devices) consists of several devices, e.g., sensors, actuators, and smart devices. Geographically distributed, those devices are responsible for sensing environmental information and the edge layer is tasked with acquiring this data. **\*** The fog layer (i.e., the middle layer or the bridge between the cloud layer and the latest layer) encompasses a large number of fog nodes, e.g., routers and gateways. They oversee scheduling, storing, data transmission and managing distributed computation. **\*** The cloud layer is responsible for permanent data storage and extensive computational analysis of data. \* The business layer is the data analytics and visualization layer which tends to user needs and specifications.

**Fig. 1.** Repartition of the components and existing risks based on the common Fog computing architecture.

We can encounter various issues at the diverse layers of the architecture, which in turn prevent IoT solutions to be fully operational. Issues are considered as faults, where faults are unintended defects that channelize to the cause of an error [8]. Hence, an error indicates the incorrect state of a system. Eventually, errors lead to the ineffectiveness of IoT systems. To describe a well-suited self-healing system, we construe the variety and nature of issues. Following IoT architecture depicted in Fig. 1, issues can be categorized into five major tracks as follows:


be mistaken in libraries, the direction of changes and version control [8]. Methods can be changed over version update and versions of components or libraries can be no longer compatible. **\* c)** Faults in source code [8] which went unchecked in any prior patches often generate system/service/application failures. **\* d)** Services and applications are also other kinds of the issue [10, 11]. The applications and services built upon the sensors can cause failures when they are in an incorrect state (e.g., on/off/crashed).


We present in the next section attempts that have been conducted to solve the aforementioned issues.

#### **3 Existing Self-healing Solutions**

Faults and failures are inevitable issues in computer-based systems, due to the inhospitable environment, unattended deployment, or unforeseen situations. Thus, researchers proposed numerous solutions to address these issues. Autonomic computing is the common approach used in IoT to detect and resolve issues in an autonomous manner. This model regroups the capabilities of a self-healing system as follows; monitoring, analyzing, planning, execution, and building knowledge. Following our review of IoT selfhealing solutions: **\*** Wallgren et al. [18] designed, implemented and evaluated a system called 'SVELTE'. It is a lightweight intrusion detection system for the IoT. The workflow of this system is as follows. First, it acts as a listener to gather the information around. Then, an intrusion detection component analyzes the mapped data and detects an intrusion. At last, a firewall was implemented to reduce the node overload by filtering unwanted traffic. **\*** Dai et al. [19] proposed a system based on acquisition, detection, and reaction. It's a self-protected system based on feature recognition using virtual neurons. They proposed five self-protecting mechanisms each one of them equipped with an algorithm to sense, detect, and prevent possible attacks. **\*** Mendonça et al. [3] proposed an architecture based on MAPE-K [20] loop, which consisted of a sensing phase present on all the nodes and a monitoring phase driven by an Artificial Neural Network Multi-layer perceptron (MLP). Information gathered by the latter phase was processed and analyzed to classify the information as safe, danger or inflammatory signals. Apart from security categories, hardware, software, and networking categories have also been studied. They include hardware failures, energy depletion or disrupted communications due to environmental conditions wind, rain, likewise communication link failure between the gateway and the sensors. **\*** G. Gupta et al. [4] proposed an architecture for the runtime recovery of the sensors from the clusters in which the gateway has experienced faults. It's a detection and recovery architecture centered around communicating gateways. It can detect whenever one of the gateways suffers a fault and allocate the sensors to a new cluster where the gateway is functional. **\*** Nguyen et al. [21] integrated the MAPE-K to a self-healing system for cyber-physical systems in smart buildings and cities. They emphasize building services for monitoring and processing data and the planning and execution of actions. **\*** Angarita et al. [6, 22] proposed a self-healing system based on transactional web services. They worked on the recovery of web services upon failure, while Al-Dahoud et al. [15] emphasis on fault detection which is the first step to achieving self-healing. **\*** Another effective way to self-healing is IoT virtualization. It's easy to deal with services and software than hardware. Hence the capacity of a virtualized system becomes easily manageable. e.g., the concept of virtual objects has been invoked to overcome battery consumption and high-processing capabilities that lie in hardware and software categories [23, 24]. The virtual objects replace physical objects of IoT, consequently provide a lot of possibilities in terms of computation, storage and recovery over a fault or a failure [25]. Indeed, services can be handled more flexibly than hardware and can be monitored in different ways improving the capability of the system to recover from an unexpected state.

We present in the next section our AMI IoT case-based system and how the concept of self-healing of IoT is implemented.

#### **4 Self-healing Approach for AMI IoT Architecture**

The AMI-lab has been developing several IoT architectures for the past decade as infrastructure to provide adaptable services for aging in place [7]. Based on our experience in developing and deploying IoT solutions, we decided to incorporate a self-healing solution that is not limited to security issues, but also targets hardware, software, human interaction, networking issues as well.

The simplified presentation of *AMI*-IoT platform, as depicted in Fig. 2 includes four main layers (i.e., end-user environment, network, and cloud) which build the chain of communication. **\* End-user environment:** includes all the environmental components such as sensors (e.g., physical nodes and sensors such as doors, motion, oxygen, bedembedded sensor), actuators and nodes gathering the data. This component is the key point of the *AMI*-Platform, being the starting gathering point. \* Fog Layer: includes the gateways which are tasked with the data modeling and preprocessing of data before it is transmitted to the cloud. **\* Network**: includes all the components enabling the link between the end-user layer and the cloud layer and responsible for establishing secure communication. **\* Cloud layer:** includes all the components enabling processes and store data (i.e., cloud nodes, database, computational nodes). This layer embeds the virtualization concept by replacing IoT physical objects with virtual objects.

We adopted the autonomic computing approach, that is based on the three main phases (monitoring, analyzing, planning, execution, and knowledge), in implementing self-healing in *AMI*-Platform. Therefore, we hypothesized that the three elements, Sensing Approach (**SenS**), Awareness of Issues (**AwaR**) and Responsibility & Actions (R**eAct**), are prerequisites to address the IoT self-healing concerns in the three aforementioned layers. Additionally, the virtualization technique has been used and a set of self-healing agents/listeners are spread through the AMI-Platform (Fig. 3).

#### **4.1 Sensing Approach (SenS)**

The SenS element is the entry point of the architecture, and it is considered as the most important element. It ensures that the design and the goal of the architecture are both respected. If any unexpected event happens, it takes the responsibility to report the abnormality. The SenS can be regarded as the listening state of the AMI architecture. It is the foundation of the self-healing concept, enabling the power of monitoring and covering the behavior of the architecture.

The AMI-Platform is composed of a set of components and the SenS is responsible for monitoring the whole chain from data detection (i.e.: by sensors/actuators) to its migration from environmental nodes (i.e., sensors to gateway) to the database (i.e., cloud nodes). To fully incorporate the SenS element, the cloud servers monitor all the components (i.e., from the end-user environment to the cloud architecture), but the components of the end-user environment should also monitor themselves. Knowledge is required to be gathered regarding the components of the AMI-Platform to be able to

**Fig. 2.** Self-Healing components in AMI-IoT architecture.

**Fig. 3.** AMI-IoT architecture with its device, fog and cloud layers.

sense accurately and efficiently the components. This knowledge will serve as a mark for the working state of these components. Following SenS in the AMI architecture.

• **SenS in the end-user environment:**It includes detection of battery level of all devices and sensors, data detection coming from nodes to the gateway, security level, network stability, connection to the cloud nodes, the gateway's processor state, and the services' state.


#### **4.2 Awareness of Issues (AwaR)**

The awareness element intends to make the system aware of abnormalities, which can occur and put the overall system in a disabled state. On this note, a system cannot be aware of failure state if the knowledge around the component is not accurate or well defined. Hence, though the SenS is important for gathering all the components information through monitoring, the AwaR plays its role by providing a parameter (i.e., what is an abnormality based on the sensed information and the knowledge). As a result, AwaR is related to the the layers of AMI-Platform as follows.


this point and battery issue is nearly nonexistent. On the security level, listeners are developed and applied to the firewall to notify at the slightest change in the firewall table and an intrusion that occurred.

Regarding the security of the data in the database, everyone who is not allowed to access the data will be reported as soon as an attempt is made. Since the peer (i.e., cloud nodes getting the data from the gateway in the end-user environment) is virtualized, when the peer system encounters a high-level failure, the system should also notify the situation in an optimal amount of time. Another critical point is the storage of the data. AMI-Platform deployed listeners to notify when the storage capacity on the cloud servers reached a certain point so that servers will not be overloaded before actions are taken.

Regarding human interaction, a pipeline has been deployed to notify all the programmers when a code has been uploaded. Validation of the code is then processed to check the continuity with the overall code. In this way, there will be fewer co-programming issues and likewise fewer failures upon deployment.

• **AwaR in Network:** Named in IoT architecture, the weak link, due to its public nature, it can be subject to many issues mentioned in the previous subsection. Listeners have been improved and applied to the security level for the firewall table. Notifications will be sent upon addition, subtraction or change in the table and for intrusion detection in the system. The system will notify as well when there is a missing link between the end-user node and the peer in the cloud. Services that are monitored by the SenS will be applied listeners as well to make the system aware of the availability of the services.

#### **4.3 Responsibility and Actions (ReAct)**

The ReAct element confers the "responsibility" feature to the AMI-Platform, which enables the autonomy in decision-making (Angarita and Kelaidonis et al. work [6, 23, 24]). A responsible component is achievable only because the SenS and AwaR elements exist. The more elaborated knowledge, the more accurate data and the more efficient the method to deal with an unexpected event. Hence, making a system able to take actions, depending on the outcome of a situation is the key role of this part. Toward a meticulous work, we strive to achieve a self-healing architecture. Since "responsibility" is the way, a component autonomously manages itself, we target to spread the concept through the all the layers of AMI-Platform. Following concrete examples of ReAct implementation in AMI-Platform.

• **ReAct in the End-user environment** Concerning the battery level, actions will be taken dynamically on the sampling ratio or the encoding format of the data. Therefore, the battery consumption will drop, and the battery life will be maintained for a longer period. As for services that are monitored, they can be restarted upon failure. Regarding the network state, the link can be reestablished after an idle time. Afterward, notification is sent upon resolution. As for security, AMI-lab deployed a set of agents responsible for reconfiguring the firewall table upon a change detection and logging out a user upon an intrusion detection and reconfiguring the whole gateway environment to ensure the same level of security is respected. Other agents are deployed for data detection and transmission and the communication with the other peer in the cloud.


### **5 Testbed Implementation**

To evaluate the reliability of the proposed model, in this section we first focus on the implementation of each component used, located at different layers of the system, then we present and discuss the obtained results.

### **5.1 System Architecture**

A bed-embedded sensor, a motion sensor and a smart IoT gateway were used in the IoT end-user layer, whereas and an IoT virtual gateway and servers were used in the IoT Cloud layer. **Bed-embedded sensor**: The sensor used in our system, monitors and collect ambient temperature, vital features such as heart rate, wake-up time, sleep time, total time of sleep, bed interruptions and the out of bedtime. The sensor used is a fiber optic mattress. **Motion sensor**: The sensor used in our system, is an event sensor reacting to movement in a specific area, collecting upon detection.

Both sensors send the collected information to an IoT smart gateway which will process the information. The motion sensor is connected to the gateway through Zwave protocol which is a low power wireless protocol. However, the bed-embedded sensor sends information through a serial connection which is physical and wired.

*Smart IoT Gateway:* We used a Raspberry Pi 3 model B with a 1.2 GHz Quad-Core ARM Cortex processor, 1 GB of RAM, and which is permanently connected to an electrical power supply and located in the person room. To allow interoperability of the heterogeneous sensors, Raspberry Pi is equipped with several communication modules. Therefore, a Zwave controller was used to establish a communication with the motion sensor, while a serial connection was used to interact with the bed-embedded sensor. The overall information is then processed and sent over MQTT protocol. An MQTT broker has been implemented on the Raspberry Pi to enhance light communication between two entities i.e., one publishing and the other one subscribing.

*IoT Virtual Gateway:* It's a virtual smart gateway based on python programming language and virtualization technology. It leverages virtualization and through an MQTT client (paho-mqtt) subscribes to the information published by the MQTT broker. The received information is processed and then stored inside a database (Elasticsearch).

*Servers:* The servers are built through a virtualization platform (Vsphere). Regarding the database, we used Elasticsearch coupled with Kibana to enable full-text search and process the information in different ways.

#### **5.2 Results and Analysis**

The experimentation has been conducted over 4 days. We compared the results with an architecture without self-healing components and an architecture with self-healing components. Upon analysis, several components revealed to be important for the purpose of the architecture. They are mainly:


On a prediction base, the amount of data which is supposed to be gathered by the end of the 4th day is the Awaited Data (AWD). The acquired data will be titled as ACD, the number of actions that have been taken and fixed the issue will be titled NAS, while the number of actions that failed is titled NAF. Other metrics such as the Data Coverage (DC), Failed Self-Healing Rate (FSHR), the Successful Self-Healing Rate (SSHR) and the Self-Healing Model Accuracy (SHMA) are described to measure the performance of our system.



**Table 1.** Metrics before self-healing.


**Table 2.** Metrics after self-healing.

Finally, over the four days, several values were recorded. Looking at the system without the self-healing components, the awaited data was 1774080 hits whereas the acquired data was 73920, giving a DC of 4,2%. Compared with the system coupled with the Self-Healing model the values are different. They were for the awaited data 1774080 hits, the acquired data 1331057 producing a DC of 75%. From those result, we noticed that the implementation of a Self-Healing model, makes the system reliable. The results can be seen through Tables 1 and 2.

### **6 Discussion**

Apart from the solutions described in Sect. 3, we argue that the self-healing in IoT can benefit from the virtualization and the test coverage technics. To start with, Applications/Firmwares that are developed in a way to incorporate unit tests and log functions (i.e., test coverage) make the system alive and make the sensing phase more complete. Additionally, developers that feed IoT applications with the possible issues that can arise and the way to solve them, from the start (at the development and deployment phases), re-enforce the self-healing concept. Hence, the system is already prepared to sense (SenS) the environment, detect the identified issue (AwaR) and react accordingly (ReAct). Furthermore, from the real-world applications of IoT experiments, information and issues shall be gathered more to make an efficient aware system. Consequently, the more knowledge gathered the more efficient the self-healing system.

Bringing forth the virtualization concept in the process, the AMI-Platform emphasized replacing the physical objects by a virtual object. The platform handles more flexibly the process of recovery and embeds the concept of microservices. Thus applications/services are made modular to enable significant adaptability and scalability of the platform offering numerous possibilities Kelaidonis et al. [26].

#### **7 Conclusion**

Failures are bound to happen in today IoT solutions. Thus, we focus in this paper on the approach to address these issues. We start in this paper by reviewing the progress of the research works related to IoT issues and classify them into five categories (i.e., Networking, Software, Hardware, Human interactions and Security issues). Moreover, to mitigate such issues, proposed solutions from researchers are analyzed. We found that most of the efforts focus only on one issue, none focuses on a comprehensive approach that addresses all the five.

We reflected in this paper on our approach that covers most of the issues. It used the concept of self-healing agents that act in listener, detector and healer modes. The listener mode allows the agents to sense the environment in the main layers of IoT architecture (i.e., Device, Fog and Cloud layers). This mode enables the collection of data from vital components (e.g., network, security, services). Through the knowledge and the sensed information, the detector mode crosschecks the information to get an accurate detection of an abnormality in the system (i.e., network state down, services down, security lacking). Finally, the healer mode defines the proper course of action to take to revert the system to a normal state.

Moreover, instead of a centralized monitoring/self-healing system, we opted for a agent-based distributed self-healing system which focus on spreading the agents through all the layers. This system builds an optimal and autonomous way to recover from a failure right after an unexpected situation occurs.

We also discussed in this paper the prerequisites (i.e., knowledge, listeners, detectors, healers, virtualization and test coverage technics) to deploy a scalable and resilient IoT infrastructure in order to ensure a self-healing mode. We target in our future work a more efficient IoT system with Quality of Service (QoS) integration via developing an advanced monitoring approaches and fault tolerating models which are to be based on data-analytics and reasoning algorithms over the large amount of data collected.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Digital Twin Driven Smart Home: A Feasibility Study**

Alireza Asvadi(B) , Andrei Mitriakov, Christophe Lohr, and Panagiotis Papadakis

IMT Atlantique, Lab-STICC, UMR CNRS 6285, Team RAMBO, 29238 Brest, France *{*alireza.asvadi,andrei.mitriakov,christophe.lohr, panagiotis.papadakis*}*@imt-atlantique.fr

**Abstract.** We aim to facilitate the daily-life activities of frail or elderly people in collaboration with mobile assistive robots through the means of a digital twin-powered smart home. Being able to quickly and efficiently produce a digital twin of the human user's environment, can help to further develop personalized assistive solutions. As our first investigation toward this goal, we describe our proof-of-concept "digital twindriven smart home" implementation. It consists of a virtual representation, robot navigation and environment semantics using open-source software. The initial obtained results on the building process of the digital twin are encouraging and suggest the possibility of integration of digital twin for smart spaces.

**Keywords:** Ambient assisted living *·* Cyber physical system *·* Living-lab *·* Semantics

### **1 Introduction**

In recent years, the availability of low-cost sensors and open-source middle-ware software have opened up interesting research areas in the robotics field. In particular, the next generation of simulation models called Digital Twin (DT), which represents a continuous virtual replica of a physical system, has gained increasing attention. It can be used to create a simulation of a smart home including assistive/service robots and human users. It has applications in the optimization of robots and smart home settings. For example, finding the optimal number and configuration of sensors especially when new robots or users are introduced. As another examples, monitoring in real-time, further analysis and learning of edge cases and rare situations in DT for the safety of human users, *e.g.*, by pushing those events from simulation to real-world and vice-versa (thanks to DT that enables a bi-directional link between simulation and real-world). In practice, such optimization should be carried out over a variety of houses and users.

In this context, this paper aims to put forward the use of DT technology in the development of a smart home system. The main contributions are:


The paper is organized as follows. Section 2 gives a definition and recent related works. Section 3 details the proposed DT development process. Experimental results are described in Sect. 4, and Sect. 5 brings some concluding remarks and proposes directions for further research.

### **2 Background**

With the progress of the internet of things and artificial intelligence, several strategies were developed to advance smart technologies, including Industry 4.0 [11] and Society 5.0 [4]. At the core, these technologies are based on a cyber– physical system (CPS) that integrates physical and digital entities. In this context, DT is an approach to implement cyber–physical integration. The DT is a digital representation of a physical system that precisely models and can predict how the actual system will perform. In this respect, it serves as a tool to bridge the gap between physical and digital worlds.

**Definition.** Some of the commonly accepted concepts in the definition of the DT are the following: it is a digital representation of a physical entity (including geometric, functional, and usage models), being ultra-realistic and fully consistent with the physical entity, continuous and automatic update to be able to adapt to changes, semantic content to support intelligent decision making, and implementation in a decentralized structure to collaborate with other DTs [17].

**DT for Robotics.** While there is a rich literature on DT concept [5,8], there have been few reports on the 3D modeling aspect in robotics and smart home applications. Kousi et al. [9] combined multi-sensor data and CAD models to create a virtual representation (including semantic and dynamic update components) of a production environment. Havard et al. [7] proposed a co-simulation approach between DT and virtual reality in a human-robot collaborative workplace. Another notable work is Habitat 2.0 [16] which is a platform for training simulated robots in interactive environments. Phanden et al. [13] recently reviewed existing simulation software for DT development. For robotics, they found Gazebo and V-REP [15] for modeling sensors and image-based systems, ROBOTRAN to model multi-body systems, and ROBCAD for multi-device robotic and automated manufacturing processes.

**DT for Smart Home.** As to digital models of buildings, To et al. [18] performed drone-based reconstruction for DT augmentation of buildings. Calderita et al. [1] proposed a CPS for ambient assisted living including humans and assistive robots in a care-giving center. A related term to DT is Building Information Modeling (BIM). Czerniawski and Leite [2] reviewed approaches for BIM. They identified the following steps for model creation: collecting sensor data and 3D reconstruction, semantic and geometric modeling for the recognition of semantic classes in sensor data and describing the instances' shapes, and finally the creation of BIM that integrates semantic and geometric components. In comparison with BIM (which is one-way modeling), in a DT there is a bi-directional data flow between the virtual and physical entities.

**Fig. 1.** The conceptual digital twin of the smart home. The dashed-red arrows mark the process that are not implemented in a fully automated way. The white box shows the component that is developed primarily for a feasibility study. (Color figure online)

There are a number of other works and projects that discuss related concepts on ambient assisted living and the synergy of ambient intelligence with robots. Examples are GIRAFF+ that consists of a network of smart home sensors and a telepresence robot to monitor elderly people, and STRANDS that explores adaptation to changes over time for mobile robots in 3D dynamic human environments. Previous work of the authors concentrated on the potential services that could be offered from such systems [12].

#### **3 DT-Powered Smart Home**

The DT of a smart home is defined as: using a priori information (*e.g.*, architectural plans, input from inhabitants) and given input sensory data (*e.g.*, home automation and robot sensors), build a continuous model including robots and user models. As a first step toward this goal, a prototype DT model of the smart home has been developed. Figure 1 demonstrates an abstraction of our understanding of what a DT workflow should consist of. Ideally, such a workflow has limited dependence on human supervision so as to reduce errors and favor portability and generalization. The steps are defined as follows. The smart home is scanned using a hand-held RGB-D camera and a SLAM algorithm to build a reconstruction of it. A modeling software is used to construct the CAD models of the home and objects inside it from the scan. The CAD models and a robot model are imported into a robotic simulator. A proof-of-concept is developed to enable semantic understanding of objects.

The DT can be conceptualized in **data**, **algorithmic** and **model**. The data layer represents the communicated data including point cloud, navigation (velocity and pose), etc. The algorithmic layer includes software to develop the modules. The model layer comprises the geometric, functional and semantics including the CAD models, a TurtleBot<sup>1</sup>, and a pre-trained object detection model. More information can be found on the project website<sup>2</sup>.

#### **3.1 3D Reconstruction**

To have a metric measurement of the actual environment a hand-held scanning is performed. The Orbbec Astra Pro RGB-D camera is used which is shown to be one of the best low-cost sensors for the reconstruction of indoor spaces [3]. The RTAB-Map (Real-Time Appearance-Based Mapping) [10] algorithm is used. It consists of loop closure and proximity detection (using a bag of words), graph optimization (to decrease odometry drift) and global map assembling (using local occupancy grid). It requires an odometry estimation, *i.e.*, motion estimation between consecutive scans. This is performed by computing visual odometry using RTAB-Map and the camera. The process outputs a dense point cloud. The CAD models are created from point cloud data. A modeling software is used to craft the CAD model of the home structure (*e.g.*, walls) from the point cloud, also online public repositories are used to obtain the models of the real home items. It should be noted that an automatic transformation of 3D scans into CAD is an open question (see *e.g.*, Scan2CAD project) that will not be addressed in our work.

#### **3.2 Physics-Based Simulation**

The Gazebo is used to simulate the dynamics of the system. A digital 3D model is developed to represent the smart home environment and the objects in the simulator. The CAD models are used to define the visual and collision properties of the items in this model. Other physical properties can be defined, for example pose, mass, being static (*e.g.*, walls) and dynamic (*e.g.*, objects). A TurtleBot model is spawned in the simulated system. The use of Gazebo with ROS enables communication between the simulated and the real system.

#### **3.3 Semantics**

The detection of the categories of objects of interest (*e.g.*, tables and chairs) and their position can have several benefits in a DT framework. The You Only Look Once (YOLO)-v3 algorithm [14] is used for object detection. It reads camera image data and returns the 2D bounding boxes. The pre-trained model is able to detect 80 classes including chair, sofa, table, bottle, microwave, person, etc. The *darknet ros 3d* package<sup>3</sup> is used to add the 3D bounding boxes of objects to the YOLO. It combines the detected 2D bounding boxes with point cloud data to calculate the 3D bounding boxes.

<sup>1</sup> https://www.turtlebot.com/turtlebot2/.

<sup>2</sup> https://sites.google.com/view/heron-project.

<sup>3</sup> https://github.com/IntelligentRoboticsLabs/gb visual detection 3d.

**Fig. 2.** Top-left: Top panoramic view of the smart home. top-right: Illustration of an area in the reconstructed map (from the view shown by a red arrow in the left image). Bottom: the full reconstructed map (from top view).

### **4 Results and Discussion**

Sections 4.1 and 4.2 describe the process results for the creation of the digital 3D model and a comparison of the DT and the actual system. Section 4.3 provides discussion on the semantics aspect. The approach is evaluated in a small experimental flat including a kitchen, a bathroom and a living room with furniture (see Fig. 2).

#### **4.1 Creation of the Digital 3D Model**

**3D Reconstruction.** A detailed colored 3D point cloud of the flat is achieved which has about 700k points (see Fig. 2). The following are some of the observed limitations.


*•* The textureless walls, ground and ceiling must be avoided or rectified (*e.g.*, by adding picture frames to plain walls). Because the visual odometry in RTAB-Map is based on extracted features from the RGB-D images. Therefore, in environments without enough features, the odometry cannot be computed.

**Digital 3D Model.** An idealistic 3D model of the flat was made by a team of students. The model was built by first measuring different dimensions of the flat and then recreating it using a modeling software. A quantitative evaluation is performed with a comparison of our CAD model against this idealistic model (see Fig. 3-Top). To perform the evaluation the models are converted to point cloud data using ray tracing operations. The root mean squared error (RMSE) and Hausdorff distance metrics (which are common general-purpose metrics for comparing 3D shapes) are used to estimate the differences (see Table 1). The accuracy of about 9 cm average RMSE is obtained which is encouraging and demonstrates an interesting perspective for an automated reconstruction-based model creation. As it can be seen in Fig. 3-Top, the perpendicularity of the walls in our developed model (shown in red) are preserved well with only a slight slope on the right side. Figure 3-Bottom shows the furnished smart home with a simulated TurtleBot in the Gazebo.

**Table 1.** A comparison of the quality of the developed smart home's CAD model against the idealistic model.


#### **4.2 Robot Navigation**

Experiments were performed to compare the traversed trajectory and the generated 2D maps in the actual and simulated conditions.

**Trajectory Evaluation.** A velocity message with the linear and angular values of +0.25 and *−*0*.*1 is published (with the frequency 10 Hz for 4 s) to control the translational and rotational speed of the robot. The objective is to perform the navigation in the DT and reality, and to compare the trajectories.

To obtain the trajectory of the actual Turtlebot, a visual marker was used. We position the marker on the upper part of the Turtlebot in the field of view of a high-resolution camera (see Fig. 4). The robot poses can be retrieved by detecting the marker in the camera image. A toolkit tracks marker poses which enables tracing of the Turtlebot trajectory in the camera coordinate frame. The experiments were performed in 5 iterations. The average and standard deviation of the trajectories in both real and simulated cases were reported in Fig. 5. In

**Fig. 3.** Top: Illustration of the representations. Superimposed simplified mesh (obtained from the point cloud), the model created from point cloud (red), and the idealistic model (green) for a visual inspection. Bottom: The simulated smart home with a Turtlebot model. (Color figure online)

addition a simplified analytical solution for the expected trajectory of a robot performing uniform circular motion is computed. It is equal to traverse of 0.4 rad on the perimeter of a circle with radius of 2.5 m. It should be mentioned that the complete Turtlebot model is a differential drive consisting of two wheels and two caster wheels which was not considered here. It was noticed that the turtlebot in the real condition traversed a shorter distance in the trajectory (about 25 cm difference with the simulated one). This was probably because of the uncertainties on the friction parameters between the robot and the ground surface. Due to

**Fig. 4.** Left: The trajectory evaluation experiment. Right: Left and right show the Turtlebot at the start and end of the experiment. Top and bottom show the simulated and actual setup.

**Fig. 5.** The average and standard deviation of the obtained trajectories in the real and simulated situations (the starting point is on the left-side of the curves).

**Fig. 6.** Illustration of the 2D occupancy grid maps. From left to right: the generated grid map in the simulated environment, the created map in the actual home, the obtained map after the additional rotation in the real case.

**Fig. 7.** 2D Object detection using YOLO-v3. The images show the detection results in different areas in the smart home.

using a simplified model, there were also differences in the recorded trajectories in comparison with the analytical solution.

Understanding the difference between the virtual model and the real world and adjusting such parameters can improve the robot's behavior in reality. It helps to accurately estimate and plan the robot trajectory and in the application level to provide both reliable and consistent service for elderly users in the smart home.

**Mapping Experiment.** A 2D mapping process is performed and compared in the DT and reality. The turtlebot rotates 360◦ (with the angular speed of 10◦ per second) to cover the whole space. The *gmapping*<sup>4</sup> package from ROS is used [6]. It is a laser-based SLAM. To be able to work with a RGB-D camera, it converts a depth image to a fake laser scan. It uses particle filters (each particle carries an individual map) and several adaptive techniques to learn the occupancy grid map of the environment. Figure 6 shows examples of the generated maps. It was observed that the actual turtlebot could not cover the full profile of the home using the rotation message of 360◦. The rotation of 540◦ was published to the actual turtlebot to fulfill the coverage of the whole area.

#### **4.3 Semantics**

Object detection can be used as a coupling mechanism of the real and the digital worlds. As a feasibility study, 2D and 3D versions of YOLO object detection were employed to detect the objects in the smart home.

<sup>4</sup> http://wiki.ros.org/gmapping.

**Fig. 8.** 3D Object detection using YOLO-v3 and Darknet ROS 3D. Left: shows a person standing and moving in the smart home. Right: shows results in another setting with different furniture.

**Quantitative Analysis of 2D Object Detection.** Figure 7 shows 2D object detection results in different areas of the smart home. An evaluation is performed to quantitatively measure the object detection performance in the smart home. A number of 30 detection frames were selected randomly and the precision, recall, and F1 scores computed for the two most frequent items seen in the smart home: the chairs and TV monitors, with the total number of 50 chairs and 41 monitors observed in all images. As it can be seen in the Table 2, the detection rates were different for each item. For example, the TV monitors were detected with higher precision whereas the detector has a better recall value on chairs.

**Table 2.** A comparison of 2D object detection in the smart home.


**Qualitative Analysis of 3D Object Detection.** Figure 8 shows an example of the 3D detection results. The top shows the 2D detections. The yellow boxes in the bottom show projected detections into the point cloud in the 3D space.

### **5 Conclusions**

In this paper, we presented an approach for the generation of a digital 3D model of a smart home using open-source software and a low-cost sensor. The methodology and initial results (including the obtained models) are promising and appear to have a potential for the design of DT-driven smart homes. However, further research is needed to explore the integration of the digital profiles of the connected home automation sensors (*e.g.*, thermostat and pressure sensors) into the DT. The next challenge is to extend the developed DT to include other entities: sophisticated robots (than a simple TurtleBot) and human users (*e.g.*, to sense where the user is and what is doing). Another interesting directions are benchmarking, further analysing and quantifying the different steps of DT creation.

**Acknowledgments.** The authors would like to thank Maroua El Houicha, Zhi Zhang, Jeanne Th´evenin, and Damien Bouchabou, students of IMT Atlantique – Brest, for providing the idealistic 3D smart home model. The authors would also like to thank Fabien Dagnat and Jean-Christophe Bach for helpful discussions on the digital twin architecture. The work has been performed in the context of project HERON - Habitat intelligent Et RObotique d'assistance bas´es sur le jumeau Num´erique - financed by the region of Brittany (R´egion Bretagne) and the department of Finistere (CD29).

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Modeling IoT Design Patterns Proven Correct by Construction

Imen Tounsi(B) , Najeh Khalfi, Abdessamad Saidi, and Mohamed Hadj Kacem

> ReDCAD Laboratory, University of Sfax, Sfax, Tunisia imen.tounsi@isims.usf.tn

Abstract. Formal method techniques are used to model complex systems as mathematical entities. By building mathematical rigorous models of IoT design patterns, it is possible to verify their properties in a thorough fashion. In this paper, we propose a refinement-based approach for modeling IoT design patterns. It allows the modeling of correct by construction IoT design patterns. It takes advantage of formal methods by the specification of design pattern models with the Event-B method and checking the design correctness. Our goal is to design IoT patterns proven correct by construction to successfully apply them and promote their reuse. Our approach is experimented through pattern examples and we illustrate it with a case study in the health care domain.

Keywords: IoT design patterns · Software architecture modeling · Pattern modeling · Formal specification

### 1 Introduction

System architectures are becoming more complex in their understanding and analysis. Among these architectures, we find the architecture of Internet of Things (IoT). The IoT is a recent concept. It is a variety of smart objects interconnected via the internet. This variety of objects faces the problem of complexity of the system. This complexity opens the way to the use of a variety of IoT design patterns to address many issues [10]. They propose solutions for common and recurring problems to architects and designers in the IoT domain. Most of these patterns are presented visually and informally, there is no formal semantics associated with them. Hence, their meanings may be imprecise. They can lead to their misunderstanding and use.

Faced by this situation, and to remedy the problem, we propose an approach that allows to model and specify these patterns with a formal notation that allows to reuse them correctly. Our objective is to prove the relevance of these patterns. We propose an approach, allowing the formal modeling of these patterns in order to describe both their structural and behavioral features. Structural features are generally specified by the types of entities. The configuration of the entities is also described in terms of static relationships between them. Behavioral features describe successive interactions between the different entities of the IoT application. The proposed formal specification of IoT patterns is generic and it is based on the formal Event-B method. We apply our approach on different pattern examples and we illustrate it with a case study in the health care domain.

The rest of this paper is organized as follows. Section 2 discusses related works. Section 3 describes how to formally specify IoT design patterns with the Event-B method. In Sect. 4, we present an application to a case study of our approach. Section 5 concludes and gives future work directions.

### 2 Literature Review

Most of the proposed design patterns (Object-Oriented design patterns, Enterprise Application Integration (EAI) design patterns, Service-Oriented (SOA) design patterns, and Connected Object (IoT) design patterns) are described by a combination of a text description and a graphical representation sometimes using a proprietary notation in the aim of making them easy to understand like SOA design patterns [5] and Object Oriented design patterns [6]. However, these descriptions make patterns ambiguous and may lack details. Some work so have proposed the semi-formal representations of these patterns using modeling languages [4]. Some other works use or provide formal languages based on mathematical notation for a precise pattern specification [16]. However, these approaches require knowledge of mathematics and first order logic to use them. Some research has chosen to combine the semi-formal and formal representations of patterns. This representation ensures a better understanding and precision of patterns. Generally speaking, there is a consensus on the elements that make up and define a design pattern. However, there is no consensus on the specification of the patterns. Tounsi et al. [12,13,15] focused on both the modeling, the formal specification and the composition of SOA design patterns [14] and established the link between them with an automatic transformation. They used the SoaML language for the pattern modeling that ease the understanding of pattern models. For the pattern specification, they used the Event-B formal method in order to attribute formal notations to SOA design patterns for the purpose of checking their design correctness.

In this work, we are interested with the IoT design patterns. In this context we find several researchers who proposed a set of IoT design patterns in various categories. Eloranta et al. [3] proposed patterns for the construction of distributed control systems. Qanbari et al. [9] presented four patterns for the supply, deployment, orchestration and monitoring shipboard applications. Lukas et al. [10,11] have published patterns for device power supply, operation and communication modes and a number of IoT design models. All these patterns are described with a visual and informal notation. There is no formal semantics associated. There are few numbers of research work that deals with the modeling of IoT patterns. Borelli et al. [2] proposed a language called BIoTA (Buildout IoT Application Language), to assist and streamline the building of software architectures for IoT. The specification and implementation of the BIoTA language involve a grammar and a compiler, responsible for syntax and semantic analysis, as well as code generation. This work facilitate the formalization of software architectures using automata, which would otherwise be created in an informal and ad-hoc manner but it addresses specific IoT scenarios. In this paper, we present the formal modeling of IoT patterns proposed by Reinfurt et al. [10].

### 3 Pattern Specification

Ensuring the reliability and the correctness of IoT design patterns is a goal that we have fixed. For this, we propose an approach to formally specify IoT design patterns by using the formal method Event-B that is well suited to our needs and goals. This approach allows the validation of the modeling part and ensures the verification of the relevant properties of design patterns.

Event-B is a formal method for system-level modelling and analysis [1]. It is well-suited for specifying IoT design patterns because: The primary concept in doing formal developments in Event-B is that of a model. It is made of several components of two kinds: machines and contexts. Machines contain the dynamic parts of a model, whereas contexts contain the static parts of a model. Thanks to this classification, Event-B allows the specification of structural and behavioral features of design patterns. Refinement techniques proposed by this method allow us to build patterns gradually and at different abstraction levels. Mathematical proofs allow verifying model consistency and consistency between refinement levels. The most important reason to use Event-B method is the availability of a supporting tool called the Rodin platform for modeling and automated proof [1]. The platform is open source and is further extendable with plug-ins. A range of plug-ins have already been developed including ones that support animation and model checking like the Prob plug-in [8] that we used.

There are two general pattern models as instances of the proposed metatamodel depending on the location of the device. If the device is placed locally, we use the "Medium Based Bootstrap Pattern" as a solution to configure the new device. If it is placed at a distance, we use the "Remote Bootstrap Pattern" as a solution.

In the specification, we concentrate on two categories of design patterns; "Bootstrapping Design Patterns" and "Registration Design Patterns". "Bootstrapping Design Patterns" allow configuring new devices. They are composed of "Medium Based Bootstrap Pattern" and "Remote Bootstrap Pattern". "Medium Based Bootstrap Pattern" allows to configure a new device on-site through a removable storage medium inserted in the device. This support contains the necessary information for configuration. "Remote Bootstrap Pattern" is a configuration pattern used in case that a device is placed far away and is difficult to reach. The configuration in this case is done by downloading configuration information from a bootstrap server.

"Registration Design Patterns" allow to register the attributes and the features of a new device on the Back-end server. The registration is used to facilitate the communication and the interrogation with other connected objects. There are many registration patterns. In this work, we present two patterns. So "Registration Design Patterns" are composed of "Automatic Client Driven Registration Pattern" and "Server Driven Model Pattern". The "Automatic Client Driven Registration Pattern" allows the device to register on the Back-end server via an API call. The "Server Driven Model Pattern" is used to create a device model that includes its description and functionality.

We model structural features of design patterns with contexts in the Event-B method. They describe by a set of concepts, the structure of an IoT architecture. We use it to describe the architecture of IoT design patterns. More specifically, it is to define the entities that can be involved in the pattern, their types and their dependencies (connections). We model behavioral features of design patterns with machines in the Event-B method. We describe through this Event-B component successive interactions between the different entities of the IoT application in order to represent the two categories of the design patterns.

#### 3.1 Contexts and Machines Relationships

We present a specification of two categories of patterns. So, the specification will be too complicated and error prone if it is done in one shot. To reduce this complexity, we define specification levels. At the first level, we create a very abstract model (a context *C*0 and a machine *M*0). At the next levels, we use refinement techniques to gradually introduce detail and complexity into our model until obtaining the final pattern's specifications. Our refinement strategy is explained in Fig. 1. First, we create a generic context *C0*. This generic context, presented in follows, plays the role of a metamodel to define types of IoT design pattern elements.

At level 1, we present contexts of the pattern category in general with their properties. These contexts extend the generic context. We specify in the context "BootstrappingPatterns", which extends the "C0" context, the set of configuration patterns contained in the "MeduimBasedBootstrap" pattern and the "Remote Bootstrap" pattern. We also declare the location to specify the position of a device (*Device*) if it is locally located or remotely. In the context "RegistrationPatterns" which extends the context "C0", we specify that in the "RegistrationDesignPatterns" category, we find the "AutomaticClientDrivenRegistration" and "ServerDrivenModel" patterns. At level 2, we present contexts of the different patterns belonging to these categories as an extension of their pattern category. These contexts are used by the corresponding machines in order to specify their appropriate behaviors.

#### 3.2 Structural Features

There are two major types of entities in the Internet of Things. The first is an «*Object*» or a "thing" which you intend to make smart by providing connectivity. The other is the «*Component*» or embedded system which provides this connectivity. Sensors, Actuators, SmartThings, and Devices are examples of Objects. Using Event-B, we specify in the context *C*0 the two entities as constants. The

Fig. 1. Contexts and machines relationships

set *Entity* is composed of the set of all *Components* and the set of all *Objects* (*Entity* <sup>=</sup> *Component*∪*Object* <sup>∧</sup> *Component*∩*Object* <sup>=</sup> <sup>∅</sup>). This is specified by using a partition in the AXIOMS clause (*Entity*\_*partition*).

Components name C*<sup>i</sup>* are specified as constants in the CONSTANTS clause. Formally, we specify {C1,...,C*n*} ∈ *Component* ∧ C<sup>1</sup> = C<sup>2</sup> ∧...∧ C*<sup>n</sup>*−<sup>1</sup> = C*n*.

Objects name O*<sup>i</sup>* are also specified as constants. Formally, we specify {O1,...,O*n*} ∈ *Object* ∧ O<sup>1</sup> = O<sup>2</sup> ∧...∧ O*<sup>n</sup>*−<sup>1</sup> = O*n*.


CONSTANTS *O*<sup>1</sup> *... O<sup>n</sup>* AXIOMS Object\_TYPE : {*O*1*, ..., On*} ∈ *Object* axm 1 : *O*<sup>1</sup> = *O*<sup>2</sup> ∧ *...*∧ *On−*<sup>1</sup> = *On.*

The communication path between Entities within an architecture is called a «*Connector*». It ensures the link between a «*Provided*» port and a «*Required*» port to form a complete and coherent system. «*Ports*» constitute the interaction points with entities environment. A required interface on a port specifies one or more operations required by behaviors of the object. A provided interface on a port specifies one or more operations that an object must provide.

A «Connector» PushE*i*E*<sup>j</sup>* is a link between two entities. It can be between two components (PushC*i*C*<sup>j</sup>* ), two objects (PushO*i*O*<sup>j</sup>* ) or between a component and an object. When the direction of the connection is from a component to an object, it is named PushC*i*O*<sup>j</sup>* and if it is from an object to a component, it is named PushO*i*C*<sup>j</sup>* . Formally, *Connectors* are specified with an Event-B relation between two entities. *Connector* 's name PushE*i*E*<sup>j</sup>* are specified with constants in the CONSTANTS clause. The set of *Connectors* is composed of all *Connectors* name. This is specified formally with a partition (*Connector*\_*partition*).


To define the source and the target of a connector, two axioms must be added, namely the domain and the range.

> PushE*i*E*j*\_Domain : *dom*({*P ushEiE<sup>j</sup>* }) = {*Ei*} PushE*i*E*j*\_Range : *ran*({*P ushEiE<sup>j</sup>* }) = {*E<sup>j</sup>* }

Internet of Things (IoT) architecture requires a message-based communication. Formally, «MessageType» is the type of messages exchanged between different entities, it is declared in the SETS clause. Messages name M*<sup>i</sup>* are specified in the CONSTANTS clause. They are attributed with their type with a partition in the AXIOMS clause (*Message*\_*partition*).

```
SETS
 MessageT ype
CONSTANTS
 M1
 ...
 Mn
AXIOMS
 Message_partition : partition(MessageT ype, {M1}, ..., {Mn})
```
#### 3.3 Behavioral Features

A machine of a pattern specification *M i* has a state defined by means of a number of variables and invariants. Some variables can be general as the variable *Send*, which denotes the sent message. The state can be modified by means of events. Events are instantaneous and their effect can cause the occurrence of other events. This copes well with the behavior of IoT design patterns. The sending of a message is instantaneous and thus can lead to the sending of other messages.

The structure of a machine *PMi* is presented in follows. Machine *PMi* sees context *PCi*. Variables *V1* are defined to know the availability of entities, sent messages, processed messages, etc. Invariants INV1(V1, S, X, Y) are defined to specify the various predicates which the variables must obey. Events (Evt1.1, Evt1.2, ...) are the various events of the machine.

```
MACHINE PMi
  Sees
    PCi
  Variables
    V1
  Invariants
    INV 1(V 1, S, X, Y )
  Events
    Evt1.1, Evt1.2, ...
  END
```
Entity activation is specified with a variable *Dispo* in the VARIABLES clause. It is used in order to know the availability of each entity (available or not). Thus, it is declared with a partial function between the Entity type and a the Boolean type. This is expressed with the invariant *Dispo\_Function*.

> VARIABLES *Dispo* INVARIANTS Dispo\_Function : *Dispo* ∈ *Entity* → *BOOL*

Internet of Things (IoT) architecture requires a different kind of message queue based communication than other types of software systems or big data solutions. With IoT, the messaging needs are more complex since IoT requires two way message communication between the server-side or cloud components and the IoT hardware devices. Formally, the variable *Send* is defined with the invariant *Send\_Relation* which specify that *Send* is a relation between a *Connectors* and a *MessageT ype*. In this way, we know how is the sender, the receiver and the sent message.

```
VARIABLES
 Send
INVARIANTS
 Send_Relation : Send ∈ Connector ↔ MessageT ype
```
Each pattern has its own behavior but some events can be general like the event of sending a message *Sending*\_*Mi*.

```
Event Sending_Mi
  when
    grd : G(v)
  then
    act : Send := Send ∪ {P ushEiEj → Mi}
  end
```
If the transition is reflexive, the event includes an action of the form:

```
V ar_Name := V ar_Name -
                       − {Entity_Name → Mi}
```
For example, the transition of message processing is a reflexive transition. Then the event *P rocessing*\_*M<sup>i</sup>* includes the following action: Process := Process -− {*Oi*→ *Mi*}. That is, object *O<sup>i</sup>* processes message *Mi*.

#### 3.4 Formal Verification

During our development, we use refinement techniques, they consist on developing a series of more and more accurate models of the pattern we build [1]. By using refinement techniques, we enrich the pattern model, and we make sure that refined models are not contradictory. As a result, when the last model is finished, we will be able to say that this model is *correct by construction* [1].

Four formal verification steps have been developed for checking design patterns; *type checking*, *model checking*, *animation* and *theorem proving*. Type checking is a technique controlling low level properties of variables in a specification. It is achieved within the compiler. Model checking and animation are two techniques used to show the dynamic behavior of a model, and they allow one to systematically exploring all its reachable states. We use them to check the correctness of the behavior of the pattern. Some temporal/behavioral properties are verified like liveness (no deadlocks present in the model) and reachability (prove that an event whose guard is not necessarily true now will nevertheless certainly occur within a certain finite time) properties. This is achieved by the model checker ProB [8]. Theorem proving technique allows checking properties which can be experimented either as predicates (INVARIANTS, AXIOMS, THEOREMS) or with guards in the events. It is also ensured by proof obligations generated by the Rodin platform. They define what is to be proved to ensure the consistency of an Event-B pattern model. As example of consistency constraints, we check that no entity can send a message if it is not authorised. This is controlled by an invariant called Can\_Send\_INV. For sequence diagrams, we require that every message must start activation.

$$\forall z, x, y \cdot z \in Entropy \land \{x \mapsto y\} \in Conmentor \leftrightarrow MessageType \land \land dom(\{x\}) = \{z\} \land x \mapsto y \in send \Rightarrow z \mapsto y \in Can\\_Send$$

Our approach ensures that the specified IoT design patterns are correct by construction. It offers architects the ability to reuse *correct* pattern models, which saves their efforts to prove the correctness of the patterns.

### 4 Case Study: Smart Hospital

Smart hospitals are hospitals which build new infrastructure. In this infrastructure, all are enabled by underlying digitized networking infrastructure of interconnected assets and remote-controlled automated components which can be medical sensors, medical devices, etc. It can include several other components that can be monitored and controlled remotely to offer providers a continuous stream of real-time health data such as heart rate, blood pressure, oxygen concentration level, and glucose monitoring. Most of these components can be controlled by a mobile device or a computer. In our case study, we add a new device (a camera) to a smart hospital patient room. This device makes it possible to control the various rooms of the hospital for safety issues. For example, the camera can recognize when a patient sits up or even has a restless night's sleep. Then it informs the nurse immediately through a notification sent to their smart phone. It can also send alerts when it detects big problems. First, the camera is

Fig. 2. Smart hospital case study

added without any information to initiate its first connection. We then apply the Medium Based Bootstrap design pattern to have its configuration information. This information is inserted into the device through a memory card. Second, we go through the registration procedure on the main server (BackEndServer). This procedure is done through the use of the two patterns of the Registration Design Patterns category which are the automatic client driven registration pattern and the server driven model pattern which allow registering the device on the main server. Finally, the camera became able to communicate and create connection links with its communication partners. The camera communicates with the nurse's smartphone to notify him of what is happening in real-time.

We model this application through the use of the model shown in Fig. 2. The camera is associated with an object of type "Device", the memory card is defined as an object of type "Storage Medium" and the Smart phone is associated as an object of type "SmartThing". The propagation of events between objects is done through a *DeviceGateway*.

In Fig. 3, we specify the static part of the design patterns of the model applied in the Smart-Hospital case study using contexts. We first specify the generic context C0. We change the generic concepts previously defined by the concepts of our case. A Camera is a Device of type object. A Smart phone is a Smart Thing of type object and a Memory card is a Storage Medium of type object. Then we specify the other contexts belonging to the Bootstrapping and the Registration patterns. In Fig. 4, we specify the dynamic aspect in the machine part of the

Fig. 3. Contexts belonging to the case study

Event-B method. We specify a part of the behavior of objects constituting the smart hospital model of our case. We describe this behavior of the configuration in the machine named *BootstrappingPatterns*. At the end of the specification Event-B models are proved. This case study shows that our approach is able to be applied while also taking the quality of results and performance into account.

Fig. 4. Machine BootstrappingPatterns belonging to the case study

### 5 Conclusions

In this paper, we presented an approach that allows to formally specify connected object architecture design patterns. In particular, specifying the "Bootstrapping Design Patterns" category and the "Registration Design Patterns" category. We formally specified these design patterns using the formal Event-B method and we described their structural and behavioral features. In our stage, we are working at a high level of abstraction, so there is no connection between the theoretical model and physical devices for machine learning. In our current work, we are working in it. The generalization of our approach is an objective that we have fixed. In previous work [7] our approach was validated with a Smart Home case study. In this paper, we applied our approach in a health care case study. As future work, we are working on applying our approach on other categories of IoT design patterns.

### References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **IoT Architecture with Plug and Play for Fast Deployment and System Reliability: AMI Platform**

Bessam Abdulrazak, Suvrojoti Paul(B) , Souhail Maraoui, Amin Rezaei, and Tianqi Xiao

AMI-Lab, Dept. Infromatique, Faculty of Sciences, Research Center On Aging, Université de Sherbrooke, Quebec, Canada

{Bessam.Abdulrazak,Suvrojoti.Paul,Souhail.Maraoui,Amin.Rezaei, Tianqi.Xiao}@usherbrooke.ca

**Abstract.** The rapid advancement of the Internet of Things (IoT) has reshaped the industrial system, agricultural system, healthcare systems, and even our daily livelihoods, as the number of IoT applications is surging in these fields. Still, numerous challenges are imposed when putting in place such technology at large scale. In a system of millions of connected devices, operating each one of them manually is impossible, making IoT platforms unmaintainable. In this study, we present our attempt to achieve the autonomy of IoT infrastructure by building a platform that targets a dynamic and quick Plug and Play (PnP) deployment of the system at any given location, using predefined pipelines. The platform also supports real-time data processing, which enables the users to have reliable and real-time data visualization in a dynamic dashboard.

**Keywords:** IoT · Dynamic deployment · Plug and play · Real-time processing

### **1 Introduction**

Enterprise architectures with a large number of the Internet of Things (IoT) devices impose challenging operational issues. It is possible to deal with these issues for a limited number of IoT devices, yet it is almost impossible to do likewise for many devices. We argue that a faster approach to tackle the operational issues of IoT by adopting the Plug and Play (PnP) concept in IoT architecture. IoT is a concept that builds a connection between all devices to the internet and facilitates communication with each other over the internet, with over 10 billion IoT devices wirelessly connected as of recently [1, 2]. In today's IoT technology, the major requirement is an architecture which is capable of supporting a large number of devices and still provides reliable performance. With the increasing number of devices, it becomes exceedingly complex to add a new one to an existing platform, and to reduce the integration phase of the system, therefore the concept of PnP in a smart environment was first proposed in 2006 [2]. The main objectives of PnP architecture in IoT systems are to reduce user intervention for system integration, and to have a real time solution from the system services. As an example, Helal et al. proposed a platform based on the PnP principle that enables a seamless integration of devices to the back-end systems without exterior interventions from integrators or engineers [3]. Although the concept of PnP is very intriguing, the PnP based IoT systems' reliability is the next big challenge. The reliability of a system can be evaluated by: the ability to perform a required task, ability to perform without failures, ability to perform under stated conditions, and ability to perform for a specified period [4].

The rest of the paper is organized as follows. Section 1 is the literature review of the existing architectures, implementation of IoT platforms, and visualization techniques. In Sect. 2, we propose our AMI IoT platform, wherein we give an overview of our architecture and implementations. In Sect. 3, we move into the details of our architecture and implementations of different components of the PnP base IoT architecture. In Sect. 4, we have the results and validation of our platform. We finally conclude the paper with a summary of our platform and its uniqueness compared to the other existing solutions in Sect. 5.

### **2 Literature Review**

The reliability of PnP in IoT systems is linked to the diverse components of the IoT architecture. Therefore, there is a need to understand the different architectural implementations for IoT platforms that cover the existing domains. In the following, we discuss the existing systems and their architectural implementations.

*IoT Platforms and Architectures* The IoT architectures that exist today have an evolving pattern, each new one overcomes the major challenges faced by the previous one. The initial approach was the three-layered architecture (Fig. 1.a) [5] with a limitation of having only the cloud layer for data processing, then came the Fog layer architecture, which resolved the limitations of cloud computing, by extending cloud-based services and bringing them closer to the IoT end-user environment, but it still had many issues on every layer, each distinguished by specific faults and errors that lead to an overall infective IoT system: **\*** *First, starting* from the end-user layer, where the sensors and/or actuators recording environmental data could have reading errors [6]. **\*** *Second*, the fog layer, where the gateways face a lack of memory, and processor overloading and overheating, which leads to serious issues in IoT systems [6, 7]. **\*** *Third*, the network layer where communication links in the IoT system are frequently prone to breaking, malfunctioning (e.g., incorrect state), or unavailability of the medium [7]. **\*** *Fourth*, the cloud layer with the challenge of sustaining all the different connected gateways, on both the computation and maintenance level.

We run a generalized analysis on the existing IoT platforms based on the following metrics- the deployment duration, the real-time capability, the reliability of the system, the business layer (mainly the data visualization aspect), and finally the cost factor, and following are the results. The existing platforms can be classified into two major groups **\*** *Consumer IoT* (e.g., OpenHAB [8], SmartThings [9], HomeKit [10]) and **\*** *Enterprise IoT* (e.g., AWS IoT [11], Watson IoT [12]), the dependencies of this classification include functionality and contextual information, interoperability, scalability, security, and costs [13]. We can categorize the existing platforms into three sets. **\*** The *first set* of

**Fig. 1.** IoT (a) three-layered, (b) five-layered, and (c) Fog/IoT architectures [5]

the platform involves a centralized hub (e.g., OpenHAB or SmartThings). The advantage is that it provides data processing and storage in the local settings meaning fast data processing and real-time visualization along with offline capabilities. The disadvantages include higher resource requirements, higher cost of the entire setup. This implementation also involves a lot of user intervention for the configuration of the system which hinders its scalability factor and in turn does not support the PnP aspect of IoT systems. **\*** *The second set,* which involves the cloud implementations, avoids the initial involvement of the users to a greater extent along with no requirements of any specific hubs which reduces the cost (e.g., AWS IoT, Watson IoT). The major drawback is the real-time performance or the reliability of the system as it depends on the connectivity of the systems. **\****The third set* is the hybrid approach which uses most of the hub implementations and the cloud services. Watson IoT or AWS IoT or SmartThings have tried optimizing the entire platform pipeline by segregating them into the local edge processing and the cloud processing, but we still find issues with deployments or installations due to the complexities of having to configure the environmental settings. The other major issue is the cost of the systems, which are often not affordable to all users.

*Plug and Play in IoT Platforms* In order to overcome the abovementioned issues, several studies draw their attention to bring the idea of PnP into IoT platforms. Kim et al. [14] proposed using web semantics to manage IoT devices. In their approach all sensors and actuators are connected to Arduino boards as embedded IoT devices. Each Arduino can accept configuration from central server and adapt their functionality to the new configuration. Meanwhile, they developed a Knowledge Base which stores the configuration of all supported devices and once a new device is connected, appropriate configuration is shared with the device so that they can start collecting data or triggering actions to control the environment. As a result, they were able to reduce the complexity of deployments and provide more dynamicity in IoT environments. However, they only focused on the perception layer and did not address the business and application layer, and how to visualize the result dynamically. **\*)** Kim et al. [15] target a framework dynamic enough to integrate new sensors, devices or services to an environment. They proposed a device discovery service that (1) automatically detect and generate a description of the device and its functionalities, and (2) integrating that same device seamlessly in the reasoning process without causing any interruption to platform or the framework functionality. **\*)** Kesavan et al., bring forward the scalability and dynamic approach which is an essential aspect for remote monitoring of patient data with real time analysis [16]. In their proposed method they focused on the scalability of the system in accordance with the PnP approach. The gateway is capable of handling dynamic addition of sensors and is capable of handling data transmission in a seamless manner. Another aspect of this method is the edge analytics part which informs the users of the battery level status of the sensors used so that necessary actions can be taken when required.

### **3 AMI Platform**

To address all the drawbacks of the implementations described above, we built AMI platform, based on the fog layered architecture (Fig. 1.c). AMI platform architecture has five main layers (Fig. 2), the edge layer which includes all the sensing devices with different communication protocols, the fog layer which is tasked with preprocessing the acquired data before sending it to the above layers, then comes the network layer which deals with the communication protocols and the security and privacy of the data being transmitted to the cloud. The fourth layer is the cloud layer, where the data received is decoded and processed in specific models which are then stored in storage hubs. The final layer is the business layer where the real-time data is visualized in a dynamic pattern through dashboards based on the user specification. Our platform targets supporting PnP features in all the presented layers, and as well as the link between them.

**Fig. 2.** AMI IoT architecture

#### **3.1 Architecture Overview**

The AMI Platform involves a set of physical (i.e., sensing devices and computing hardware) and logical (i.e., software applications and virtual computing resources) components that collect, process, transmit, store, and analyze data. These components are layered following Fog Computing architecture.

The physical gateway is situated in the end-user environment. It represents the fog layer and thus the bridge connecting the sensor devices in the end-user layer with the network layer. Additionally, a virtual gateway is located inside the cloud infrastructure as an extension of the network layer, to receive the data transmitted through the network layer, and push it to the cloud storage. The benefit of the separation is to utilize the computing resources on the gateways for pre-processing and ensuring the quality of service. A common challenge with the physical gateways is the deployment and update of the numerous services that are running on them that handle the communication and acquisition of data from sensors. Our focus while designing and building the physical gateway was to ensure the ease of PnP integration of a "Pipeline" that automatically or semi-automatically handles the deployment of said services. The data flows through the platform as follows: First, at the end-user layer, a cluster of heterogeneous sensors collects both environmental (e.g., temperature, motion, location) and biomedical (e.g., heart rate, breathing rate, sleep pattern) data. Next, the data is sent to the physical gateway for pre-processing and pushed to the cloud layer via a secure channel created by the network layer. Upon arriving at the cloud layer, the data will be decoded, processed in specific formats, and stored in databases. The business layer is for the user interface which also has models for dynamic graphs and charts creation which helps for a better adaptation of data visualization for any kind of user need. We also propose a different approach to resolve the existing challenges of scalability and maintainability based on microservices, where the different functional components of the system are split into independent applications [17].

#### **3.2 Edge Layer**

In the context of IoT, the end-user layer is located in the monitored person's living environment, and it is considered the entry point of the platform. It consists of a set of sensing devices of heterogeneous nature that observe key parameters of both the living environment and the person in multiple dimensions. **\*)** *Wearable Sensors* (e.g., Apple Watch, Garmin Watch, Fitbit Watch, Mi Bands, Pulse Oximeters), which are installed and/or implanted on participants' bodies (e.g., wrist, fingers) to monitor their everyday activities, provide various measurements, including motion (e.g., acceleration, step count, distance walking), location (e.g., attitude, longitude, altitude), biomedical information (e.g., heart rate, heart rate viability, breathing rate, and oxygen saturation). One major advantage of wearable sensors is their portability: the participants can easily carry the devices either at their home, doing outdoor activities, or during clinical assessment. However, this type of sensor also shows limitations under specific circumstances. Because the patient's expertise in advanced technologies cannot be guaranteed, ensuring proper charging, usage, and maintenance of the devices are challenging tasks. **\*)** To overcome these limitations, there is the second type of sensing devices, *Environmental Sensors* (e.g., door/window, motion, temperature, illuminance, and humidity sensors) that are designed to track indoor environmental parameters and patients' ADL continuously while maintaining their autonomy and independence [18]. These sensors are often installed in fixed positions (e.g., wall, door, and window) inside a subjective room. **\*)** Additionally, other sensors can be placed or connected to furniture such as the smart mat device, a highly sensitive micro-bend fiber optic sensor (MFOS) often placed on top of the bed (under the mattress or bed sheet) or on a char non-intrusively to monitor location information and vital signs (e.g., heart rate (HR), breathing rate (BR), Ballistocardiograph (BCG)), which enables the possibility of detecting sleep cycles and sleep disorders, such as Apnea [19].

#### **3.3 Fog Layer**

Our implementation of the fog layer is represented as a physical gateway, that acts as the hub for multiple devices with different communication protocols. The physical gateway has the following major functionalities: **\*)***Data acquisition*: the gateway supports diverse communication protocols, e.g., Z-Wave, Insteon, Bluetooth, Serial line communications, and other similar protocols. Each adapter is an abstract layer for the hardware and constantly listens to its corresponding sensor. It also enables easy and reliable integration of new devices (PnP concept in sensor installation [20]) with the same set of protocols. **\*)** *Preprocessing of data*: the raw data obtained from heterogeneous sensors suffers from poor interoperability, so the data format varies between different types of sensors, leading to difficulties in data interpretation and transmission. Therefore, we developed a data model library to overcome this issue. The data model library enables the gateway to pre-process the data by adding semantic information and populating its values into one of the compatible formats. **\*)** *Data Transfer*: The next step is the transfer of data from the edge layer to the network layer, the processed data is sent through network protocols, for example, the messaging protocol called Message Queue Telemetry Transport (MQTT) or the hypertext transfer protocol (HTTP) protocol. Currently, in our implementation, we are using MQTT as it has a publish-and-subscribe architecture where the client devices and applications publish and subscribe to topics handled by a broker. **\*)** *Status Updates*: The other service offered by the gateway is the system or status updates. All the above-mentioned services require system resources and to avoid failures the system check services provide live status updates of the resources, for example, CPU, memory, and internet connectivity.

#### **3.4 Network Layer**

Sending confidential data and information over the internet is vulnerable to attacks, so it is critical to create a secure channel for connections, which is taken care of by the network layer, the subspace between the physical gateway (at the edge of the end-user layer) and the virtual gateway (at the edge of the cloud layer). A great amount of effort is put into securing the network connection. **\*)** Firstly, a virtual network is created to establish a peer-to-peer secure channel between the two gateways, adding an extra layer of security to our protocol of communication. The advantages of the software network solution are efficiency and quick adaptability. It takes advantage of the existing internet connection instead of setting up a new one. In our platform, we use OpenVPN to implement this secure channel. OpenVPN is an open-source virtual private network (VPN) system that creates routing and bridging connections between entities. An OpenVPN server automatically issues certificates for our physical and virtual gateways, triggered by their respective deployment pipelines, and then stores them in a secure storage hub. This approach limits the risks of transporting the certificates/keys from one entity to another one. **\*)** lastly, enforce the firewall rule by allowing only a small number of essential connections and blocking all others. All communication between the physical gateways and virtual gateway goes through this channel, securing the confidential data transmitted between them.

#### **3.5 Cloud Layer**

The cloud layer is responsible for the management, storage of the collected data. Considering the essential need for a real-time monitoring solution, the cloud infrastructure needs to have minimal delay and downtime, to allow for a reliable and real-time visualization and analysis. Hence, the agent-based microservices approach is adopted on the cloud layer. Each component on the cloud layer is containerized using Docker daemon. This allows easy deployment of components anywhere without having to worry about the specifics of the underlying platform.

**Virtual Gateway:** Being the contact point of the cloud layer, the virtual gateway is created from a pre-built docker image and plays a crucial part in the platform. It connects with its peer physical gateway through the VPN in the network layer, receives data with the configured communication protocols (e.g., MQTT, REST), and reconstructs the data message using a mapping function to meet the requirement of the data storage. However, managing on the physical gateway the rapid evolution of sensors' data models and numerous deployments of physical gateways requires a manual configuration and intervention before every deployment and after every update. This challenge pushed us to build the gateway in a way that will allow for a PnP integration seamlessly with a pipeline that automates its deployments, and updates.

**Data Storage:** IoT data representation and storage are an ever-changing and expanding field, with this in mind we have opted to use a non-relation database, or more specifically document based. The advantages of this are: 1**)** *first* the dynamicity and flexibility of document-based databases, which is particularly useful for our platform due to the evolution of data generated by IoT devices over time, and by eliminating the time required to adapt or update the structure of tables or other entities to allow inserting new fields or data models; *2***)***second* the horizontal scalability of non-relational databases, as opposed to vertical scalability of relational databases [21]. As consequence, keeping up with the increase in either the volume or velocity is only a question of adding more nodes to the database cluster. This architecture also ensures higher availability because, if any node fails, all its functions will be delegated to other nodes in that cluster. Our choice of data storage is Elasticsearch, an open-source distributed search and analytics engine at the center of the Elastic Stack [22]. It supports various commonly used data types, including numerical, textual, keywords, state, and multi-dimensional.

#### **3.6 Business Layer**

**Data Visualization**: Environmental and wearable sensors acquire a massive amount of data, e.g., on average, 1.2 to 1.8 million entries per day during the course of our first deployment. With such amount of data constantly generated, it is crucial to develop an appropriate visualization tool that can present collected information in real-time to provide better assistance to the end-users. However, the variety of needs, roles, and metrics in different applications along with changes in needs over time make it difficult to have only one dashboard to fit all the needs. Our adaptive solution enables even users without technical knowledge to create a dashboard adapted to their needs in minimal time.

**Fig. 3.** Approach for adaptive data visualization

In our architecture, a unique approach is taken to ensure PnP by visualizing data adaptively (Fig. 3). We first choose web-based technologies to create our system due to their accessibility. Unlike traditional software that is tied to a specific system, web applications are reachable from multiple platforms, including desktop, mobile, tablets, and smart TV. Through these web-based solutions, we enable users to dynamically define the environment and add sensors to it. When the environment model is defined, they can design the dashboard based on their needs which will be adapted to the environment automatically. This system consists of three major components:

**1)** *Chart Template Designer*: Empower users to dynamically design charts and save them as templates specific to each type of sensors. Users can choose the data source and visually generate queries to filter data for charts. Once the user finalizes the chart design, the configuration of the designed chart is stored in our data storage, and the defined query is parsed to accept parameters in order to enable linking the designed charts to external parameters. The first step to design charts is to retrieve required data from the data source. We used a query builder tool to allow users to define their intended criteria to filter fetched data. We added the concept of chart template to our platform in which when a request is sent to view a chart, the configuration of the chart along with other information will be loaded. Thus, the designed system gives the user flexibility to reuse designed charts in many places by only passing the required parameters (e.g., a device id).

**2)** *Environment Configuration Tool:* This tool enables non-technical users to create a 3D model of the deployment environment, input the address and occupants, and place items, furniture, and predefined sensors using simple drag-and-drop actions [Fig. 4]. The output of this component is used by Dashboard Designer and Reasoner Engine to generate the intended design and knowledge for each space in this environment. We defined the concept of "Space" in our platform as a group of sensors, associated with a place. The output generated by this tool is a number of "Spaces," each of them includes several sensors and descriptive information about the area. This form of output enables all other consumer components to adapt themselves to the different Spaces.

**Fig. 4.** 3D representation of the environment

**3)** *Dashboard Designer:* Allows users to dynamically design their desired pages according to the needs. This designer incorporates configuration tool data and designed chart templates to visualize sensors information on the pages automatically. Users can drag and drop necessary components to the page and easily move the elements and change components' size to achieve the desired design. A rich set of components are added to the designer which enables users to add and customize pages and link them to environmental sensors. Once the design is finished, the design will be applied to the all provided "Spaces," received from configuration tool and the data sources for elements in each page will be adapted to the related environment automatically. e.g., a user may design a chart to visualize room temperature, they can design it for one room and the dashboard engine will adapt the chart for other rooms provided by configuration tool and replace all the required parameters to visualize data related to each room. Another layer of flexibility is provided by allowing users to customize the elements for one specific environment so that they can fully adjust the design and deliver the data collected from sensors instantly to intended users. To manage the access level of designed pages, we built an integrated authentication system based on JWT tokens, each token contains information about the user and their assigned roles. Once a user logs into the system, a token is automatically added to the requests which allows the server to authenticate users. This authentication service is integrated with our designer to allow specific groups of users to have access to certain pages.

### **4 Pipelines**

To speed up the process of PnP deploying and running the system, we have created a set of pipelines to automate the deployment of all services on every layer of the Fog Computing Architecture. It is a three steps process that starts from a sensor-gateway connection and ends with a running, real-time dashboard.

#### **4.1 Gateway**

The gateway acting as the hub is capable of handling multiple sensors simultaneously. For the installation of the services to provide the functionalities described in the previous section, we use *Ansible* to automatize the process. The ansible scripts are divided into three sections, first group of scripts are tasked with installation of all the required dependencies that our services would be needing to run properly in the system, along with this the setting up of firewall rules and other security aspects are also taken care of by these group of scripts, the second section includes the installation scripts for all the services provided by our platform, from cloning the repositories to providing them the execution rights to act as a system service is being handled in this section. The third section is the self-healing installation phase, it installs services that are used for monitoring the system resources, and report back for cases of abnormalities. This approach results in a significant decrease in user complexity and errors in the installation phase. Once the installation is done the gateway nodes are ready to plug and play for any deployment. The time taken for this installation phase of the deployment process is approximately less than thirty minutes. The gateway nodes have OpenVPN installed during this process, which gives them a secure connection to the servers to communicate with.

### **4.2 Server**

The server pipeline consists of two main steps, both of which are automated with a script as well and require only the identifier of the gateway being deployed for an easy PnP solution. Each step corresponds to a layer in the Fog Computing Architecture. The **first step** is the configuration of the Network Layer by registering the gateway certificates in the VPN server and then setting up firewall rules to only accommodate the ports used by our services, thus protecting the connection between the server and the gateway by filtering and blocking all other traffic from gaining unauthorized access. The **second step** is to configure the node microservice that is responsible for receiving and pushing the data into our storage hub. As we support multiple communication protocols, the microservices need to be configured to a specific protocol before their deployment. Once the execution of the deployment script finishes, the gateway would be able to connect to our VPN server and start sending the data collected from the sensors and actuators to our node microservice, which will then be pushed to our data store.

#### **4.3 Dashboard**

This step completes the IoT cycle by offering a plug-and-lay solution to design a dashboard where we can monitor the collected data in real-time. The first step is to design the Environment in our 3D configuration tool where we would create the 3D space where we have deployed our gateway and communicate with it to collect the list of sensors and actuators that have been deployed in the environment. Right after we feed the generated Environment Model to our Dashboard Designer to automatically generate a view for all the spaces in the environment and the devices that have been deployed in each space. The user has full flexibility to design the required pages.

### **5 Results and Analysis/Validation**

Our platform was deployed for a project which validates the working of our architectural metrics. The project was conducted for a span of six months at an apartment in AMI. The main goal of the project was to monitor the behavior change of a person based on the type of medication received at different stages of medical intervention. For the platform analysis, the metrics we targeted in this deployment are as follows:

#### **5.1 Fast Deployment**

We deployed in an apartment to monitor the behavior of a person and to mark the changes with the change of medication provided to that person. In this deployment, we deployed two nodes (Raspberry PIs), five door sensors, and seven motion sensors, along with a sleep mat. The steps followed in the deployment of our platform in the apartment are as follows: scouting of the area to have a better understanding of the environment, start connecting our nodes to the power outlets and configuring them with the network available in the apartment, once the node configuration is done with the network we start attaching the sensors at different places in the apartment like the walls and doors, with the completion of our entire setup the final step is to check the system performance and the data being transmitted to our servers. This entire deployment, from attaching the sensors to the walls and doors in every bedrooms and kitchen and living room, to deploying the nodes to set up the network for the nodes, took us less than two hours in the apartment.

#### **5.2 System Reliability**

The key challenge in any IoT system is the reliability of the platform. Reliability in this context refers to system failures and the efficiency of the platform in providing the decided upon services through various technical and non-technical challenges. The technical challenges include the resources of the system, the CPU and memory capacities might fail or overflow with time resulting in an unreliable system. The non-technical challenges include internet failures, the offline capability of a system is a big challenge in the development of any IoT system nowadays. In our deployment, we received certain reporting on our system status (Fig. 5) which highlighted the warnings related to CPU usage, physical memory usage, internet connectivity, and service resource consumption.

**Fig. 5.** Treemap of all the system status.

#### **5.3 Real-Time Data Processing**

In the healthcare system, a real-time platform is essential because of the impact each data might contain. For example, heart rate monitoring systems cannot have a huge delay for patients with cardiac issues. In IoT, there are very few systems in the market that can provide a real-time data processing platform. The deployment of our platform was at an apartment in Sherbrooke, Canada, which is around 10 kms (about 6.21 mi) from our servers. Our architecture has proven to be able to provide a real-time data processing and visualization technique with an average delay of less than one second (Fig. 6). We have a specific model for the data encapsulation in which we have two timestamps, the first timestamp (phenomenonTime.instant in Fig. 6) is recorded at the moment the data is received from the sensors and the other timestamp (resultTime in Fig. 6) is noted after the data is processed in the gateway and is ready to be transmitted. Based on these two timestamps we see that there is very little to no delay being added in our data processing period.


**Fig. 6.** A JSON of stored sensor reading

#### **6 Discussion: Lessons Learned**

The results obtained with the deployment of our platform showcase our real-time success however we faced a few major challenges throughout the period of deployment. The first major challenge was the sensors or device malfunctioning. Now our platform is dependent on third-party manufacturers for the devices and the sensors which hinder the performance of our system. One of the deployed sensors stopped working due to some manufacturing issues in the very first week of deployment. We quickly resolved the issue, by changing the sensor with a new one, as our system has a continuous monitoring aspect which informed us about the failure in a short time interval. Using trusted manufacturers for the devices might lead to higher prices but comes with greater reliability. The second challenge was with memory leaks. Currently, the physical gateways (in our case, the raspberry pi) have a limited memory resource. Due to the explicit number of services running there was a memory leak concerning the logging of issues. This issue was quickly resolved with the introduction of another self-monitoring service which monitored the overall memory usage of the system along with specific usages by each service. If the usage surpasses a specific threshold the logs are rotated and the unwanted system logs are deleted. Along with this our visualization platform provided a great level of flexibility and enabled us to design desired content for our deployments. However, there are still spaces left for improvement, especially when users add a large number of visualizations and charts. The high-frequency data generated by some devices (namely smart mats) was another challenge for visualization which we were able to overcome in the new versions of our platform. For our future works, we have already advanced on some new self-healing concepts [23] and distributed architecture to provide better performance while maintaining the quality of service (QoS) of our IoT platform.

#### **7 Conclusion**

Faster PnP deployments of IoT platforms with reliable performance, especially in tackling real-time data processing, is a challenge that hinders wider implementation of most IoT solutions. Thus, we focus in this paper on the approach to addressing these key issues. We start this paper by reviewing the progress of the research works related to IoT architectures and platforms that are in the market. After analyzing the existing solution, we come across the definitive drawbacks, and we present our architecture and IoT platform which addresses these drawbacks. We also detailed in this paper each of the components of the architecture and described how components are connected to overcome the said drawbacks. The logic in our business layer provides for a unique approach to handling user specifications by creating dynamic dashboards. We highlighted the success and the challenges faced during the six months of deployment of our system in a residence which acts as proof of concept for our main objectives. It also opened the possibility of future developments and improvements. We target in our future work to focus more on the quality-of-service (QoS) nature of the platform and to have a distributed architecture to enhance our scalability aspect.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Annotation Systems in the Medical Domain: A Literature Review**

Zayneb Mannai1(B) , Anis Kalboussi2,3, and Ahmed Hadj Kacem1

<sup>1</sup> ReDCAD Research Laboratory and Faculty of Economics and Management, University of Sfax, Sfax, Tunisia

mannaizayneb889@gmail.com, ahmed.hadjkacem@fsegs.rnu.tn <sup>2</sup> Higher Institute of Computer Science and Management, University of Kairouan, Kairouan,

Tunisia

anis.kalboussi@isigk.rnu.tn

<sup>3</sup> ReDCAD Research Laboratory, University of Sfax, Sfax, Tunisia

**Abstract.** In the literature, a wide number of annotation systems in the e-health sector have been implemented. These systems are distinguished by a number of aspects. In fact, each of these systems is based on a different paradigm, resulting in a jumbled and confused vision. The purpose of this study is to categorize medical annotation systems in order to provide a standardized overview. To accomplish this, we combed through twenty years' worth of scientific literature on annotation systems. Then, we utilized five filters to determine which systems would proceed to the classification phase. The following filters have been chosen: accessible, free, web-based or stand-alone, easily installable, functional, availability of documentation. The classification step is performed on systems that evaluate "true" for all of these filters. This classification is based on three modules: the publication module, the general information module and the functional module. This research gave us the chance to draw attention to the issues that healthcare professionals may face when using these systems in their regular work.

**Keywords:** Annotation · System · e-health · Healthcare professional · Patient · Interoperability · Partnership

### **1 Introduction**

Annotative activity had previously occurred in medieval Latin society. The examination of palimpsests discovered throughout history shows that, at that time, many people annotated the same manuscript, which resulted in the accumulation of several layers of annotations. So, we can say that annotation was a necessary and crucial activity that allowed messages to be passed down to future generations.

Since the invention of printing, each person has had their own annotation support where they can write private notes. In this way, only the owner of the annotated support has been able to read the annotation [1].

Annotative activity is now widely used in almost every aspect of life. In this regard, consider the case of a student who highlights essential passages in a text and makes notes in the margin. Also, we can cite the example of a teacher who deciphers a number next to a word and then consults the bottom of the page to discover the associated note. Annotation types and functions differ widely, and they are classified depending on a lot of factors, such as the annotator's type, the goal, the spatiotemporal frame, and so on [2].

Traditionally, annotation is done on material supports. The researchers were interested in this approach; therefore, they took the required measures to help it evolve by implementing computer systems to manipulate annotations. These systems offer new functionalities that facilitate the tasks of the annotator. Such as: storing the annotations in files or databases separate from the original document, combining annotated documents, exchanging data, etc. [3].

When it comes to the medical field, annotation is a critical skill. Therefore, healthcare practitioners annotate medical records that are both written on paper and stored electronically. The purpose of this research is to offer some insight into the annotation systems used in the medical field. Therefore, we will try to obtain a standardized and clear overview of the various medical annotation systems that have been developed in the literature. To achieve our goal, we will identify a strategy that brings together the filtration and classification of these systems. In the literature there is little research on the classification of medical annotation systems. However, no work has been done to implement a filtration strategy for these systems. Our method enables health professionals to select the appropriate annotation system for their specialties while also ensuring that the system is operational and available.

This strategy has the following form: To begin, we will filter the annotation systems that were chosen throughout the search phase. This filtering is based on five filters: {F1: accessible, F2: web-based or quick installation, F3: functional, F4: free, F5: availability of documentation}. To be elected for the phase of classification, an annotation system must have the value 'true' for all filters. The phase of classification enables the elected systems to be classified according to three classification modules: publication module, general information module and functional module.

The following sections comprise this article:

The second section is labeled "Annotation Systems Filtering". This section includes a strategy for filtering annotation systems. This part will aid us in weeding out systems that do not meet our needs.

The third section, "Annotation Systems Classification" will classify annotation systems using three classification modules.

In the fourth section, "Observations", we will discuss our observations regarding the study of these systems.

Finally, we will conclude our paper and give various viewpoints to round out our study.

#### **2 Annotation Systems Filtering**

Due to the important number of annotation systems, we discovered, we omitted plugins and standalone software that require the user much effort to install them locally. Indeed, our main attention should be the annotation activity., avoiding all installation and configuration complications.

An annotation system must be accessible, i.e., the scientific publication should contain a functional link that allows access to the tool.

It must also be functional, in the sense that he completes the duties that have been given to him appropriately.

Elected systems must be also free [4].

An annotation system must have a documentation in order to study its internal structure.

The following is a list of the election filters that we have fixed:


A system must have the value' true' for all filters in order to be elected. Elected systems will be thoroughly investigated in the classification stage.

Election Decision = {F1 and F2 and F3 and F4 and F5}

The following figure (Fig. 1) shows the process of filtering annotation systems.

**Fig. 1.** Filtering process

### **3 Annotation Systems Classification**

#### **3.1 Publication Module**

We cannot examine an annotation system in isolation from the facts surrounding its publishing. This module describes elements of both the tools and its scientific publication. These are crucial aspects to evaluate the tool's originality. The following are the publication criteria that we have selected [4, 5].


The following table (Table 1) summarizes the publication criteria for each paper.


**Table 1.** Publication module


#### **Table 1.** (*continued*)

#### **3.2 General Information Module**

[3, 27] The annotation activity begins by selecting the anchor and the shape of the annotation from the software toolbar. Then, the annotation must be attached to a welldefined target in order to satisfy all of the annotation's requirements. This annotation activity can be classified into three broad types.

	- **Automatic format:** the machine is programmed to carry out the entire annotation process without human intervention.
	- **Semi-automatic format:** in this situation, the user initiates the process. The algorithm eventually learns and understands how the user annotates. It then suggests automated annotations based on an annotation model developed with rules in development. When no adjustments are made and the suggested rules are fully accepted, human intervention is cancelled and the process becomes fully automated.

Annotation can be classified into two types.


The annotation has two types of structures:


The information in the table below (Table 2) highlights a classification of annotation systems based on the following characteristics:



#### **Table 2.** General information module

#### **3.3 Functional Module**

Each annotation system has a particular variety of features [3, 28].

#### **Sharing Features:**


#### **Memorization Features:**


#### **Reuse Features:**

	- Manual: Depending on the user's preferences, the health care provider can choose to see only a subset of annotations.
	- Automatic: Only annotations that have been granted permission to be seen by the healthcare professional are visible to him.

The following table (Table 3) includes a classification of annotation systems according to functionalities.

**Table 3.** Functional module

### **4 Observations**

This survey allowed us to address the research question posed in the introduction of the article, which serves to obtain a uniform and clear overview of the different exciting annotation systems in the literature.

By finding the answer to the research question, we now have a better understanding of the issues that the healthcare providers may run into when utilizing these tools.

**Interoperability Problem:** each of the systems examined above is based on a set of standards and formats unique to that system. This results in an issue with the integration of heterogeneous programs, and as a result, there is a communication blockage. there is no longer any exchange of data between the apps. A health professional cannot send or receive data from his colleagues. This creates a problem for both patients and health professionals because it restricts their mobility. Additionally, the patient is unable to communicate his personal data to his doctor, resulting in data loss. To circumvent this issue, harmonizing annotation systems' models and standards is required.

**Problem of Patient Integration:** because the patient is unable to access the content of the annotation systems, he can no longer communicate his ideas in a meaningful way. In addition, he finds himself in a state of uncertainty as to his understanding of the notes that have been placed in his medical file. Indeed, he does not master the scientific language and medical terminology provided by his doctor, which leads him to ignore his current state of health, his medications, and his entire treatment protocol, which can sometimes make the recovery more difficult. In this way, the annotation is unable to promote collaboration between different stakeholders. But, on the contrary, it leads to a breakdown in communication between health professionals and patients.

**No Development of a Partnership Cycle:** with the integration of this cycle, the annotation will ensure sharing and communication between all stakeholders. This must be done to reach a consensus on a decision.

**Problem of Deciphering the Structure of Annotation Systems:** several of the systems reviewed are open sources, which allows for the examination of their documentation and source code in order to gain a better understanding of their structure. However, decoding the structures of other systems is difficult, if not impossible, because we no longer have access to their source codes.

**Hardware restrictions:** the only data entering materials are a keyboard and a mouse. Improved data entry assistance devices, such as the stylus, are required. The stylus is a tool that can make healthcare professionals' annotation activities easier and more versatile.

### **5 Conclusion**

Towards the end of this article, we will briefly say that we have proposed a strategy for the filtering and classification of annotation systems in the medical profession. This has allowed us to draw attention to the problems healthcare professionals may encounter when using these systems in their daily activities.

As part of our future study, we want to focus on establishing an annotation modeling standard that allows both healthcare professionals and patients to annotate electronic medical records.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

**Wellbeing Technology**

# **SAATHI: An Urdu Virtual Assistant for Elderly Aging in Place**

Anand Kumar(B) , Ghani Haider(B) , Maheen Khan(B) , Rida Zahid Khan(B) , and Syeda Saleha Raza(B)

Dhanani School of Science and Engineering, Habib University, Karachi, Pakistan *{*ak05173,gh05177,mk04389,rk04364*}*@st.habib.edu.pk, saleha.raza@sse.habib.edu.pk

**Abstract.** With the rise of the digital age, life has become a lot easier for the vast majority of the population. However, the ever-increasing elderly population has suffered, especially in countries like Pakistan, where limited accessibility to technology, often due to language barriers, hinders elderly from reaping technological benefits. In this paper, an Urdu virtual assistant application is proposed which provides an intuitive and empathetic platform for the elderly in Pakistan that helps them perform essential tasks such as reminding them of their medications, organising their work, getting daily news highlights, and connecting them with their loved ones. It also provides entertainment in the form of user-specified video playlists or by positively engaging them in conversations on various topics.

**Keywords:** Elderly care *·* Virtual assistant *·* Conversational agent *·* Urdu natural language understanding (NLU)

### **1 Introduction**

According to the World Health Organization (WHO), life expectancy is increasing due to advancement in science, strong economies, and healthy behaviours [10]. Due to this increased life expectancy, most people are expected to live beyond their sixties, allowing them to contribute to society as mentors, innovators, and in a variety of other ways. The latest census of Pakistan states that the current population of people over the age of 60 (old-age) is estimated to be 5.54% [2] and this number is set to increase. However, due to the increasing normalisation of a nuclear family structure [20] in Pakistani society and the hectic lifestyles of modern-day individuals, oftentimes there are not enough people to take care of older family members. As a result, the elderly might end up isolated and confined to their homes. This can negatively impact their emotional and physical well-being. Consequently, there is a growing demand for technology that supports the elderly at home [11] and can assist them in their daily tasks helping them stay connected to their loved ones.

To address the aforementioned problems, a solution called "Saathi" is proposed. Saathi intends to offer the elderly a supportive environment by allowing them to use an Urdu mobile virtual assistant to perform everyday tasks such as setting medicine and appointment reminders, calling their loved ones, navigating through different apps in their phones using visual icons or voice commands in Urdu to improve their everyday life. In addition to that, it provides them with entertainment, prompts them to take their meals on time, and generates empathetic conversation or actions accordingly. While an abundance of such applications exists in languages like English, there are currently no assistive applications for the elderly that support the Urdu language. Saathi acts as a bridge that allows the elderly to use their smartphone by making it more accessible through a voice-enabled Urdu virtual assistant as well as a consolidated and user-friendly user interface.

The rest of the paper is organized as follows. In Sect. 2, the existing work is reviewed in the domain of Socially Assistive Robots (SAR) and virtual assistants (VA) for elderly. Section 3 explains the proposed solution which includes the overall architecture and features of the application. Section 4 presents the application prototype and the results related to testing performed on the conversational agent. Section 5 discusses the obtained results and mentions the future scope of the project and Sect. 6 concludes the paper.

#### **2 Related Work**

Much research and development has been done in the domain of Socially Assistive Robotics (SAR) to develop companion robots to assist older people in a variety of tasks such as monitoring and promoting physical health in the elderly at home [3,11], helping them by autonomously navigating the user's home, reminding them of their medications, and providing entertainment [1]. Other than that, technologies such as Internet of Things (IoT) and wearable technologies could offer promising solutions for elderly care. Wearable devices are a part of IoT systems which can be worn by the people for monitoring the physical activities and physiological data [19]. These devices are embedded with sensors and algorithms that allow them to track, analyse and guide users' behaviour [17], vital signs or movement. Wearable tech has the potential to assist with several scenarios in elderly healthcare and can improve the quality of life for the elderly and allow them to maintain their independent lifestyle [19]. However, one of the major issues with SAR systems and wearable technology is the lack of social component and interaction, which hinders their acceptance among older people. Another problem with these assistive robots is their significant cost, which is not within everyone's budget, especially in low income countries like Pakistan.

Like other technological devices, virtual assistants have also become an element of great help to integrate and keep people active, in addition to facilitating the different everyday tasks that need to be performed [13]. A virtual assistant is an application that recognises voice instructions and does tasks on the user's behalf.

There has been significant work done to develop user-centered virtual assistants that feature different functionalities to assist elderly people. Researchers have found that spoken language seems to be the most preferred mode of interaction for elderly people, they prefer to interact with virtual assistants in a simple, hassle free way [18]. Beskow et al. [6] designed a user-centric virtual assistant that could be used to set medicine reminders after surgery and had a virtual character with whom users could communicate and set reminders, among other things. Lesin [16] developed a virtual assistant solely for mobile platforms, which aimed to assist diabetic patients with their daily routines. In addition to providing reminders, it also possessed the option to reach out to doctors/medical staff. Bickmore et al. [7] developed a virtual assistant to foster fitness and exercises in elderly people. Mival et al. [12] developed a virtual assistant that resembled a dog so that people may have virtual pets and communicate with them. Klein [15] created a virtual assistant that could aid a user in managing stress and anxiety by offering a variety of solutions for reducing stress and anxiety. Kasap et al. [14] created a virtual assistant that could change its facial expressions depending on how the user was feeling. Yasuda [21] created a virtual assistant with a human-like aspect to encourage user attention in a purely conversational medium.

The aforementioned works are in English and hence would not be helpful in the local context of Pakistan as majority of the population does not converse in English. Little work has been done in the field of virtual assistants in Urdu. In 2019, C-Square collaborated with Genesys to launch Pakistan's first AI enabled Urdu voice recognition bot named RUBA (Real Urdu Bot Automation) [4] at the Smart CX conference. RUBA stands for Real Urdu Bot Automation and is a Siri-styled virtual voice-enabled personal assistant. The bot can converse with the user in Urdu and react in Urdu while performing simple activities like as checking the amount of the user's bank account, sending text messages, and so on [4]. However, the domain of this app is very limited and does not focus on solving problems specific to any particular group in the Pakistani society.

There are a few other mobile applications available such as Carer, Elderly Care and BaldPhone which are designed for elderly people, however all of these apps are solely in English and not voice activated hence they would be of little use to the majority of Pakistani elderly individuals who do not communicate in English.

#### **3 Proposed Solution**

Saathi is an Urdu virtual companion application for elderly people in Pakistan. It facilitates them in their daily tasks and provides them with a supportive environment to keep them engaged and entertained through technology. Saathi is built using Flutter for the frontend, Firebase for the backend and the conversational agent has been designed using the RASA Framework. Figure 1 represents a high-level architecture of Saathi.

**Fig. 1.** System architecture diagram

#### **3.1 Conversational Agent**

The Urdu conversational agent is based on the RASA framework. RASA is a Python based, machine learning framework for conversational AI [8]. Since RASA is an open-source conversational AI framework, its code is readily available to use for free. It is an accessible, flexible, robust, and transparent framework. A benefit that RASA provides is that it follows a modular, extensible, micro-services architecture that fits well in any typical software development scenario. It is easy to integrate and customize to fit the needs of our application. Most importantly, RASA allows to develop conversational agents that leverage the power of NLP to determine user intents and mimic human-like conversation, as compared to chatbots or conversational agents with hardcoded logic and no capacity of learning.

Saathi uses RASA to engage the elderly in chit-chat about subjects that are most common among Pakistan's senior citizens. These topics of interest could include food-related discussions, asking them to describe their past experiences, or conversing with them about their favourite activities. Along with daily conversations, the conversational agent can be used to respond to queries about the weather, news, and time among other things.

RASA framework breaks down the processing of user queries and returning of responses into two main components, namely, Natural Language Understanding (NLU) and Core Dialogue Management [8]. Figure 2 shows the complete datapipeline for the conversational agent module.

**Natural Language Understanding (NLU) Component:** The NLU component of the RASA framework is further divided into two components:

1. **Natural Language Processing (NLP) Component:** The raw Urdu text is first passed through the Urdu Natural Language Processing pipeline built using the SpaCy library. The Urdu NLP pipeline consists of a tokenizer, a parts-of-speech tagger, a parser and a named entity recognition module.

**Fig. 2.** Data pipeline for RASA framework

The tokenizer breaks the input sentence into smaller pieces called tokens. These tokens are tagged and categorized in correspondence with a particular Urdu part of speech by the tagger. The parser then extracts the syntactic structure of the input text by analyzing the words based on the grammar of the language. Lastly, the named entity recognition module identifies and classifies the named entities in the text. These named entities refer to the proper nouns found in the text.

2. **Intent Classifier and Entity Extractor:** The processed query from the NLP component is passed on to the intent classifier and the entity extractor module. Entities in RASA are structured pieces of information inside a user message [8]. The module maps the user query to a pre-defined intent and extracts important entities from it. RASA uses DIET (Dual Intent and Entity Transformer) architecture as part of its NLU component. DIET is a multi-task transformer architecture that handles both intent classification and entity recognition together [9]. The DIET architecture comes with the ability to plug-and-play various pre-trained embeddings and support for custom components and pipelines to use any other ML model.

**Core Dialogue Management Component:** The extracted intents and entities from the NLU component are then passed on to the second component of the framework called RASA core dialogue management. RASA core uses a classifier as a response selector. The response selector finds and outputs the best response to the user input query. In the case where the RASA module cannot map a query to a pre-defined intent, a fallback policy is activated by the RASA module which sends a fallback response to user and asks the user to rephrase their query.

**Custom Actions:** Sometimes, the queries by users might require the execution of custom code to process the response required by the user. Such queries are handled by the custom actions module in RASA. RASA's custom actions module allows running custom code which can be used to perform database queries, make API calls, etc. Saathi uses custom actions to add events to reminders, open mobile applications, dial calls, send messages, etc. Following are a few custom actions that the user can perform using Saathi conversational agent:

1. **Feeling connected:** Saathi allows the elderly to open any application that might be installed in their smartphones through Urdu voice commands. It also helps them to dial a call or send a message to their loved ones by using either their name or phone number. These actions are performed by using an android intents plugin supported by Flutter. Such features help keep the elderly feel connected and give them a sense of independence.


#### **3.2 Empathetic Responses**

It is important to emphasize that all intents and responses in the conversational agent have been pre-defined in Urdu. This means that the scope of the application has been set up during the development phase. The responses and interactions are tailored in such a way that they are as empathetic and human-like as possible. Firstly there are intents, which are used to categorize user messages. For example, for an intent called "greet" all the different kinds of greetings in Urdu are defined. Whereas, responses are what the chatbot sends to the user. After defining all the anticipated messages from the user and the replies from the chatbot, multiple intents and responses are tied together to form stories, which act as templates for possible conversations. These templates are provided as training data for the chatbot's dialogue management model [8] which in turn helps generalize the model to deal with unseen conversation paths. In Fig. 3, the story shows a happy path where the user might say hello and the chatbot greets them in return. It then asks them about their mood and depending on the mood, different story paths are chosen. In this particular example, the user's mood is great and so the chatbot replies with an appropriate response.

The RASA conversational agent requires datasets in the form of intents and responses. For our application, we have defined 28 intents in total for now, covering a range of queries, concerns and actions that the elderly might require. However, this set of intents can further be extended depending on the future needs of our application. Each of these intents have 10–15 Urdu examples on which our model has been trained. These example sentences have been crowdsourced from native Urdu speakers to train the model to best identify each requisite intent. Each intent is associated with a response or custom action that

**Fig. 3.** RASA file format

is sent as output to a triggered intent to help the elderly. RASA allows making the conversational agent's replies more interesting if multiple response variations are provided to choose from for a given response name. Where possible, we have added variability in responses by providing multiple possible response utterances mapping to specific intents. RASA then selects one of the provided utterances at random and sends it to the user.

In contrast to question-answer bots, Saathi is categorized as a companion or a conversational agent. Therefore, special care is taken in appropriately responding to different scenarios. One such example is when the elderly user mentions that they are feeling sad or anxious, an empathetic response is sent, which asks them if anything can be done to help improve their mood. In some cases, a YouTube video of narration of an Urdu short story is also sent to cheer them up.

#### **3.3 User Interface**

To use Saathi, the user first needs to sign up for the application and fill in the required information such as their email, password, medicine timings, and preferences. These preferences can be related to their meal, medicine or appointment reminders, specifying a Youtube playlist they like to watch, etc. The registration process also requires the user to provide information of at least one caretaker, so that they can be alerted in cases of emergency such as when the elderly might be feeling unwell. After signing up, the elderly can interact with the application and use features such as the visual assistant, which allows them to navigate through their phones easily, the scheduler, where they can see their reminders and events, and the virtual assistant which can be used by their voice in the Urdu language.

To use the virtual assistant, the user can give their queries in the form of speech or voice commands in Urdu. The Urdu voice data is first converted to text using the speech to text feature of the underlying mobile platform and then passed on to the conversational agent module in the form of a text query. On receiving the text response from the conversational agent, it is converted to voice using the text to speech feature and sent to the user.

Saathi relieves the caretakers - people in charge of looking after the elderly the burden of constantly being worried about the well-being of their elderly loved ones. This is done by keeping them up to date on the elderly person's medicine and meal intake, the status of their appointments and other such events. To update the caretaker, a daily log of above-mentioned activities is maintained and shared with them on WhatsApp using the Twilio API.

### **4 Prototype and Results**

This section presents the application prototype to validate the proposed solution and the results obtained while testing the conversational agent.

### **4.1 Application Prototype**

The mobile application is installed on the elderly's smartphone. It is used to communicate with the conversational agent which is deployed on the cloud. Through the application, the elderly can navigate to different screens and choose the activities that they are interested in performing. The application allows them to do the following tasks:


### **4.2 Prototype Validation**

In order to test the application, unit and integration tests were written for different components and modules of the application. Usability testing was also performed to evaluate the accessibility and ease of use of the application from the user's perspective. The testing was carried out on 10 users over the age of 60. One of the feedbacks received from the users was that they faced difficulty in navigating to different screens of the application. Another issue pointed out was the limitation in the type of conversations with the conversational agent.

After getting the feedback, the user interface was made more intuitive and easy to use for the elderly by increasing the font, icon and image sizes. To provide ease in navigation, we provided a visual screen where all the basic features are consolidated on a single screen. For the conversational agent, more data was added around different topics such as checking elderly mood, discussing their interests, likes and dislike to keep the elderly engaged.


**Fig. 4.** Registration screens

#### **4.3 Results for Conversational Agent**

Since the conversational agent will be directly interacting with elderly people it must be as accurate and well-performing as possible. To do this, the model was tested through the RASA test module. The cross-validation method was used to test the chatbot, where the dataset is split into k number of groups, each of equal size, in this case, *k* = 5. One of these *k* splits is then chosen for testing and the rest of the splits are used for training the chatbot. This process is repeated *k* number of times until each split is used for testing. Finally, a weighted average of all k iterations is taken to validate the model's performance. Figure 6 shows the confusion matrix of intents with the cross-validation approach. It shows that even though the diagonal values are high, there are several misclassifications of the intents. The ratio of misclassifications to correct classifications is 0*.*247. While the model performs well on a majority of the intents even with cross-validation, there is still room for improvement. Due to these misclassifications, a threshold of 0*.*7 is set as the confidence level of an intent classification. This implies that if the confidence score of any intent goes below the defined threshold value, a fallback policy is triggered and the user is asked to rephrase the sentence. The overall goal is to ensure that no inconvenience or discomfort is caused to the user by any incorrect or hurtful responses.

#### **5 Discussion**

From the results, it is apparent that the conversations are not strictly domainspecific and can instead be more open-ended. As the conversations are open-

**Fig. 5.** Application prototype

ended, it is difficult to account for all possible conversational intents and their relevant replies. This is where a need for an Urdu conversation generation model arises.

However, to our knowledge, no such model exists to perform this task in the Urdu language. Hence, this area can potentially be explored further.

Urdu, being Pakistan's national language is understood by almost everyone irrespective of their ethnic identity. For a language so widely spoken, there need to be technological advancements so applications like Saathi can be developed with ease. The features that makeup Saathi are usually only provided by applications that do not support the Urdu language and most elderly people in Pakistan are not fluent English speakers. This huge gap makes technology inaccessible for around 15 million [2] elderly people of Pakistan. Additionally, Urdu being a resource-poor language with respect to the NLP resources currently available means that it is very difficult for researchers and developers to effectively solve Urdu NLP related problems like Saathi.

**Fig. 6.** Confusion matrix of intents with cross-validation

#### **6 Conclusion**

There are many virtual assistants that exist in English and other commonly used languages specifically designed for the elderly population. However, no such application currently exists in the Urdu language as per our knowledge. This paper presented a one of its kind Urdu language based virtual assistant named "Saathi", in the form of a mobile application. It can help the Urduspeaking elderly individuals navigate through their smartphones with ease and provide them with a supportive environment to stay entertained and engaged in daily activities. Saathi serves as a companion to the elderly and helps them stay connected with their loved ones.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Smart Technology in the Home for People Living in the Community with Mental Illness and Physical Comorbidities**

Cheryl Forchuk1,2(B) , Abraham Rudnick3,4, Deborah Corring1,2, Daniel Lizotte2, Jeffrey S. Hoch5, Richard Booth2, Barbara Frampton6, Rupinder Mann1, and Jonathan Serrato1

> <sup>1</sup> Lawson Health Research Institute, London, ON, Canada cforchuk@uwo.ca

<sup>2</sup> Western University, London, ON, Canada

<sup>3</sup> Dalhousie University, Halifax, NS, Canada

<sup>4</sup> Nova Scotia Operational Stress Injury Clinic, Nova Scotia Health Authority, Dartmouth, NS,

Canada

<sup>5</sup> University of California Davis, Davis, CA, USA

<sup>6</sup> Ontario Peer Development Initiative, Toronto, ON, Canada

**Abstract.** This study evaluated a smart technology intervention in the home as a support for individuals with severe mental illness. This study recruited 13 participants in a variety of community-based homes. Participants were offered a smartphone, a touchscreen monitor and health devices such as smartwatches, weigh-scales, and automated medication dispensers. Data was exported to the Lawson Integrated DataBase for care providers to monitor/track. Interviews with participants and focus groups with participants and care providers were conducted at baseline, 6-months and 12-months, and survey instruments were used to collect quantitative data about different dimensions of health and social determinants. Descriptive statistics from these outcome measures are presented as the sample size was too small for meaningful statistical inference. Qualitative analyses revealed a high degree of acceptability of the devices and motivation for healthy living, communication and mental health. Health Care Providers also noted improvements to client health. This study proves the feasibility of deploying smart technologies to support individuals with severe mental illness. Future scale-up would further our understanding of their impacts.

**Keywords:** Smart technology · Smart homes · Mental health · eHealth · Physical health

### **1 Introduction**

Smart homes to support mental illness are an appealing new concept but have not yet been implemented at the community level as a form of mental health care or support. Literature reviews have found that hospitalizations, burden on care providers, consultation and wait times, and health care costs can all be reduced through health monitoring technology [1]. Although new technologies are rapidly developed for the consumer market, many are not stringently tested in a mental health care setting; leading to a lack of robust literature. This rapid pace of development also means that new technological advances become obsolete quickly over time [2].

Mobile technologies such as smartphones have been utilized to support mental illnesses such as bipolar disorder [3] and schizophrenia [4]. Patient beneficence and wellbeing can also be supported using smart technology by providing greater access to information and resources, as well as through symptom tracking and monitoring features [5]. A systematic review of smart technology for mental illness detailed that smartphone applications were the most studied, but monitoring and adherence supports were lacking significantly [6]. A more recent systematic review also revealed that research into technological interventions based on the Internet of Things model is still lacking, with mental health data scattered and segregated depending on the vendor/platform for these devices [7]. A smart technology intervention should require minimal input and provide continuous monitoring passively that does not interfere with daily activities.

Physical activity can also be an indicator for changes to mental health and/or indicative of a crisis. Including physical activity into daily routines can be difficult, particularly as symptomology, motivation, experience, fatigue and poorer access to resources are common considerations for people with mental illness [8]. This is particularly pertinent during the current time with physical activity and mean peak heart rate readings decreasing significantly during COVID-19 lockdowns [9]. As such, the use of smart technology to maintain a healthy lifestyle is even more needed, particularly among vulnerable individuals who may experience barriers due to their health status, lack of accessibility or socioeconomic status.

Early inceptions of the current study, each lasting 12 months, attempted to provide supportive systems within an individual's environment in hospital settings and transitional hospital apartments [10]. The prototype intervention provided in the hospital setting was largely successful. This study set out to establish the use of smart technology in assisting individuals with mental illness and physical comorbidities living in housing provided by the Canadian Mental Health Association (CMHA) and London-Middlesex Community Housing (LMCH). The objective of this study was to establish and evaluate smart home technology in the community to assist people with mood and psychotic disorders. We hypothesized that the smart technology would: a) increase levels of community integration; b) increase housing stability; c) decrease health, justice, and social service utilization; and d) support mental and physical health.

The study also sought to answer the following research questions: a) What are client and staff experiences of smart mental health homes? b) How do clients and staff perceive the utility of smart technologies in the home? c) What improvements to the technologies do they suggest? d) What ethical issues are identified with the use of smart mental health homes? e) What policy issues are identified with the use of smart homes? f) What commercialization issues are identified by key stakeholders in relation to smart homes?

### **2 Method**

#### **Design**

This study used a within-group, mixed-methods, repeated-measures design. Interviews were conducted over three assessment timepoints conducted at baseline (Time 1), 6 month (Time 2) and 12-month (Time 3) follow-ups. The assessments included an individual interview with each participant. However, due to the COVID-19 pandemic, interviews and focus groups were switched to virtual and telephone formats, and focus groups were conducted via a one-to-one discussion instead to maintain safety as well as convenience for participants. Ethical approval was obtained through Western University's Research Ethics Board.

#### **Sample**

The research team first recruited health care providers who then referred potential participants for the study. Participants from a range of housing types were eligible to take part. The inclusion criteria for participants were as follows: 1) Must be on a caseload of a participating health care provider; 2) Able to understand English to the degree necessary to participate; 3) Living in housing provided by the CMHA or LMCH; 4) Diagnosed with a psychotic or major mood disorder; 5) Must be between the ages of 18–85 years old; 6) Able to provide informed consent.

#### **Intervention**

This study consisted of two interactive platforms, the Lawson Integrated Database (LIDB) and the Collaborative Health Record (CHR). The LIDB is an information management platform that collates and manages health data and is protected behind the St.Joseph's Health Care hospital firewall [11]. It is a web-based application that can be accessed by health care professionals and research staff via secure log-in. The LIDB connects to the clouds of health monitoring devices and is able to automatically export data from each cloud encrypted authentication keys and SSL connectivity. The CHR allows health care providers to send questionnaires ("Qnaires®"), both standardized or customized, as an SMS text message, email or both to participants. The link then opens to a custom webpage with the questionnaire for participants to complete. The platform has the additional feature of videoconferencing and instant messaging. Care providers were able to securely log-in to the LIDB and CHR to view data from their participant's health monitoring devices and Qnaires® respectively.

Screen devices offered included smartphones (Samsung J3®) and wall-mounted touch-screen computer monitors powered by a Raspberry Pi-3 B+® mini-computer. The latter were developed by the research team programmer who also designed and customized the interface. These devices received and responded to prompts from the LIDB to assist participants experiencing cognitive deficits and to facilitate self-care. Participants were able to "acknowledge" prompts and reminders on the touchscreen monitors by pushing a "Got It" button on screen which sent an automated email to the care providers who set that reminder. Additionally, a button which says "Please get in touch with me" was included, which sent a message to the participant's care team requesting them to schedule a meeting.

Health monitoring devices that were offered to participants included a Withings Nokia Body+® smart weigh scale, a Fitbit Charge 3® activity tracker and a Karie® automated medication dispenser developed by Ace Age. Apps for each of these devices were added to the participants' smartphones so that they could also track and monitor their own data. Although activity logs for device usage were not recorded, health care providers could observe the measurements provided by the health devices through the LIDB thereby providing an idea as to frequency of device usage.

#### **Procedure**

The research team met with prospective participants and after obtaining informed consent, participants selected the equipment they felt they needed and completed the baseline interview. Interviews were conducted one-to-one with a member of the research team either in-person or virtually. Equipment selections were then verified and approved by the participant's health care provider before being installed by the research team's programmer. Training on the equipment was provided by the research team programmer as well as the research coordinator. Technological literacy was measured at baseline through the demographics measure of the interview. Health care providers also received training on how to use the devices, CHR and LIDB by the research coordinator. The research coordinator contacted the participants monthly to check in on any potential technical difficulties and offered assistance if issues arose. Calls and emails were made to health care providers to offer support and troubleshooting with the LIDB, CHR and any general concerns regarding the devices.

Focus groups with participants and health care providers were conducted separately to avoid one group from influencing the other. These were conducted in a group format prior to the COVID-19 pandemic but then switched to virtual groups or a one-to-one discussion via telephone or teleconferencing software.

#### **Instruments**

The following assessments tools were utilized for the semi-structured interviews conducted with participants: Community Integration Questionnaire - Revised (CIQ-R) [12], Short-Form 36 (SF36) [13], the Health, Social and Justice Service Utilization (HSJSU) questionnaire [14], EuroQol-5 Dimension-3 Level (EQ-5D3L) [15], and a Perception of Smart Technology Questionnaire. The latter was a researcher-developed measure that assessed participants' attitudes and opinions of the devices provided. Demographic and housing history data were also collected during each interview. The CIQ-R was the primary outcome measure. Open-ended qualitative questions regarding the use of the devices and software platforms were utilized during the focus groups and one-to-one discussions.

#### **Data Analysis**

The participants' health, housing, service utilization and community integration were assessed using the instruments. Descriptive statistics are provided for the outcome measures. Due to the small sample size, comparative analyses across timepoints were not possible, and generalization beyond the sample is limited by its size. Qualitative data was analyzed using a thematic ethnographic approach [16] to ascertain collective experiences and implications. Data collected from the health devices and the Qnaires® were not obtained for analysis by the research team in order to maintain participant privacy and not to interfere with the patient-provider therapeutic relationship.

### **3 Results**

A total of 13 participants were interviewed at baseline. Table 1 summarizes the demographic characteristics of the sample. Nine participants completed the full 12-month study; two passed away due to natural causes before Time 2. At Time 3, one declined to complete the interview while the other participant could not be reached. No participants experienced housing instability throughout the course of the study. Two participants were eventually moved from group homes to independent apartments. The vast majority of participants were single or never married (n = 10), indicating that these participants may need additional support if they are living alone.


In terms of technological literacy, the participants were asked at baseline to score their experience with devices (see Fig. 1). The sample overall indicated they were comfortable with technology. Participants largely rated that they were comfortable with technology in general. Three participants reported they were "extremely comfortable", six reported they were "comfortable", and two participants each reported they were "slightly comfortable" and "slightly uncomfortable" respectively.

#### **Quantitative Findings**

**Community Integration Questionnaire - Revised.** The mean scores on all CIQ-R subscales and CIQ-R Total score across all 3 timepoints are reported in Table 2. The CIQ-R asked participants how often they completed certain activities and whether someone else assisted or completed these activities.


**Table 2.** Community Integration Mean Scores

**Short Form 36.** The mean scores on all SF-36 subscales across all 3 timepoints are reported in Table 3. Subscales are scored on a scale of 0–100, with higher scores representing better health outcomes.

**Health, Social and Justice Service Utilization.** A variety of health and social services were accessed throughout the study. Table 4 maps out the utilization of these services.

**EQ-5D-3L.** The Visual Analogue Scale revealed participant's mean self-report ratings of their overall health, physical health and mental health from Time 1 to 3 on a scale of 0 (lowest health) to 100 (best health) (see Table 5).


**Table 3.** Short form 36 mean scores

**Perception of Smart Technology.** Three key questions from the questionnaire are reported below. The first inquired as to whether participants felt the technologies improved their health care (see Fig. 1). This was asked at Times 2 and 3 after the technologies had been implemented. The second focused on acceptability of the technologies in the home. The majority of participants responded favourably at Time 2 – Mixed (1), Mostly Satisfied (2), Pleased (4) and Delighted (4). At Time 3 there was a slight increase in positive ratings – Mostly Satisfied (1), Pleased (3), Delighted (5).

The third key question focused on recommendations for what devices should be added to future interventions. The suggestions were evenly spread with a blood pressure cuff receiving the most recommendations (2), followed by a tablet, a smart television, smart glucometer, a Google Home® device and a newer model of Fitbit® (all 1). However, the most frequent answer was "Nothing else/None" (3) as participants felt the devices offered as part of this study provided suitable coverage for their needs.

#### **Qualitative Findings**

Several themes pertaining to healthy living were discussed by the participants. Participants noted that they were motivated to be healthier through exercising and maintaining a healthy weight. The health data on the apps on their smartphones allowed them to track their progress which provided a level of accountability.

*The scale especially um it, it, it allowed me to keep track of what was happening as far as weight and things like that went and um yea it didn't, it has motivated me um to start watching my diet and things like that.*

Health care providers also reported that their clients were becoming more motivated to use the devices provided to lead healthier lifestyles and, in some cases, assist with symptoms of pre-existing physical conditions.

*Uh, I definitely noticed, um, that she was able to maintain a healthy lifestyle, um, in term, or I guess a healthy weight. There was, you know, only small fluctuations in their weight, but it was nice to see that. Cause I know they, they talk about that a lot. Um, just wanting to, because they have another, um, illness related to their weight, like polycystic ovarian syndrome. So they were, it was more of like an important thing to focus on*


#### **Table 4.** Services accessed at all timepoints

**Table 5.** EQ-5D-3L visual analogue scale scores


**Fig. 1.** Perception of improvements to health care following the use of smart technologies.

*managing or getting, you know, getting to be a healthy weight so that the symptoms of that are minimized.*

Other participants noted benefits to their mental health through this new ability to be healthier and through the cognitive supports the technology offered.

*I have issues with memory and being able to have reminders to drink water. Um, how, cause instead of feeling more lethargic and fatigued from not drinking water, um, I felt less lethargic, less fatigued. Therefore it increased my mood because I wasn't sleeping all the time. And for me, a trigger for depression is sleeping all the time.*

It was noted that the technologies were also able to support mental health through a biofeedback approach with the devices monitoring physical activity.

*...and like as I say the, the Fitbit, especially the pulse um the heart rate um is, is really helpful because um if I'm having problems with my mental state sometimes, I need to look at, especially with anxiety, I need to look at my pulse and, and sort of be aware of it and help, it can help me bring it down...*

Enhanced communication was also seen as a major benefit of the study. Participants noted they were able to maintain communication with their friends, family, and health care providers during the pandemic.

*Um, well being connected with your family definitely helps you with your mental health and like being able to follow-up with appointments and stuff like that. Um, yeah and I also have been able to like, find resources about my mental health like on CAMH and stuff like that and um meditation resources as well.*

Many health care providers also spoke of the social benefits of providing participants with a smartphone but also noted that the technology enhanced communication with the care providers as well as flexibility in communicating with them.

*...Like some clients especially like with mental health, they find it easier for them to, instead of talking to someone over the phone, to text and communicate that way.*

There were numerous instances of participants noting that the reminders and prompts from the technology provided them with the support needed to maintain healthy lifestyles and live safely in their own homes. In particular, participants with the medication dispenser noted that they no longer missed doses.

*Having the ability to monitor um the, the medication was a problem and the technology just sort of took that problem away because it replaced the need for me to keep track of things.*

With technological interventions, there are also some limitations and barriers to overcome. One such issue is device reliability. There were some issues regarding the accuracy of their readings and difficulties with connectivity.

*Yeah, um, but when I switched phones it was very difficult to get um, to get my Fitbit to sync over Bluetooth.*

A precipitating factor for technical difficulties could be the lack of technological literacy and understanding how specific devices worked. Although monthly refreshers were provided by the research team, health care providers highlighted that frequent retraining would be recommended.

*I think there was, um, a few items that like the clients weren't really familiar with or like, knew how to use our troubleshoot, even though like, I don't know, maybe I'm just the younger generation. I know how to use it. Like the Fitbit. They don't really know how to like sync it or look at their data on their, on the app, on the phones and stuff.*

It is also recommended that alternative devices are available for participants who require additional assistance with using technology.

*Like I said, I have hand tremors, so takes me awhile to get it to work.*

In terms of future improvements for commercialization, the health care providers suggested simplifying the approach to a single integrated database or as an app.

*You'll have to be streamlined through a maybe single-handed maybe app? ... that will probably be uh better, for future reference, and it could be uh quite heavily used then.*

**Ethical Analyses.** This project's findings suggest that the use of the technologies advances equity and fairness by increasing access to care for people experiencing mental illness and physical comorbidities. During the COVID-19 pandemic, participants were able to access care from the safety of their own home using the technologies provided by the study. Another ethical advantage of this project was that it enhanced autonomy of participants, reducing their dependence on some health services such as by using the medication dispenser device. Overall, this project's findings are promising from an ethical perspective and suggest the need for larger scale research on such technological interventions.

**Policy Analyses.** It was felt that it would be purposeful to propose an amendment to the Assistive Devices Act to incorporate smart technologies as assistive devices for mental health. Devices aimed at supporting mental health are not currently covered by the Assistive Devices Act. Individuals with mental illness or agencies supporting this population would have to purchase the devices themselves. For individuals in other jurisdictions, amending current policy to provide funding for technological innovations in the home to support mental illness would be advantageous.

#### **4 Discussion**

There is a lack of literature on community-integrated research that utilizes a systemslevel novel approach to health and smart technology. Smart technology interventions, such as this study, should be designed to provide support a range of physical and mental health conditions [17]. This study positions itself as a foundation for future research among individuals with mental illness to build upon. As there was a small sample size, it would be difficult to argue that this intervention represents a definitive approach to mental health care, but provides an impetus for new research to cover some of the gaps currently in the literature. Data from the Perception of Smart Technology questionnaire supported that the technologies were well-accepted and used by participants of different age groups. As the majority of participants were single or never married, this intervention may be a needed support for individuals living along without immediate assistance. The connectivity of the intervention with care providers could help to mitigate potential risks that may arise due to isolation or a lack of communication.

A key strength for this intervention was the use of non-clinical devices that are accessible to the public. Data from the clouds of these devices were able to be transmitted and collated in one database, the LIDB. Health care providers were able to access all the data in one location as opposed to individual databases or independent datasets. This allowed for a tailored approach where each participant could have a customized intervention. Participants were also able to connect their smartphone (either their own, or one provided as part of the study) to their devices and access the data themselves using the devices' apps. An implication for future research is to measure the effects of smart technology among a larger sample with other mental and physical health conditions. It would be highly beneficial for potential researchers to track health data (i.e. weight, steps, activity, heart rate, etc.) empirically to assess the impact on health behaviours and physical condition.

There is also the opportunity to utilize this intervention to support physical conditions that may benefit from activity and weight tracking and notifications from a health care team. This study found that smart technology was able to promote healthy lifestyle choices. Based on the qualitative findings, the devices acted as an accountability tool which provided encouragement and motivation for healthier living. Frequent observation and self-monitoring of health data using personal digital assistants and daily feedback messages have been linked with weight loss [18].

A systematic review by Liu et al. [19] reported there is no evidence that technology tracking and monitoring biometric data resulted in improvements to quality of life or disability. However, the qualitative findings of this study suggest that the feedback from the health adjunct devices and their respective apps on the smartphones can be helpful. The use of an activity tracker to encourage physical activity and exercise may have been the most contributory piece of equipment in this regard. Participants reported how the technologies supported mental health such as mood and anxiety through greater communication and the ability to monitor physical health more easily. Cognitive support was also described in assisting participants to remember tasks such as taking and tracking medications. There was some discussion among participants that the weigh scale and the activity tracker may have occasionally given inaccurate readings. Future studies would be advised to frequently check devices for accuracy during the course of the study and allay participant concerns.

The COVID-19 pandemic meant lockdowns were enforced during the course of the study. Individuals with mental illness and physical conditions are especially vulnerable to the negative outcomes of isolation and distancing [20]. The COVID-19 pandemic meant that virtual and telephone interviews were conducted instead. Although participants were satisfied with these arrangements, some participants were difficult to contact and required multiple sessions to complete the interview. The technician for this study provided technical support where possible but opportunities to visit the participant and check the devices were limited.

Due to the small sample in this study, regression analyses were not conducted; thus, findings should be interpreted with consideration to this. However, the data collected from questionnaires are pertinent as it provides better understanding of the characteristics of individuals with mental and physical health diagnoses who may benefit from smart technology use. Notably, it was observed through questionnaire scores that there was high heterogeneity in our sample and in participants responses (e.g., frequency of service utilization). Further research into the use of smart technology with individuals who have mental illness with a larger sample is warranted. All participants from this study were recruited from the same, moderate-sized, city. A larger study with a variety of locations including more rural-based individuals may reveal different experiences and learnings.

#### **5 Conclusion**

This study establishes that a smart home technological intervention is a feasible, reliable, and safe way to provide additional support in the home. The provision of commercially available devices may provide a suitable alternative or an additional option for in-home support. These technologies can supplement healthy living behaviours which could lead to other health benefits such as weight loss, as supported by the qualitative analyses. Participants noted a self-reported increase in physical activity, diet tracking and greater access to mental health support. The COVID-19 pandemic likely impacted health, social and justice service utilization therefore studies exploring this, including this study, needs to be interpreted with caution. This study can act as a foundation for future research to build upon by exploring other applications and populations with this intervention. Future studies and technological innovations would be advised to offer in-home technologies that can be easily implemented into the living environment and to address gaps currently in the literature.

**Acknowledgments.** We would like to acknowledge Lawson Health Research Institute, St. Joseph's Health Care, London Health Sciences Centre, Canadian Mental Health Association and London Middlesex Community Housing for facilitating the research environment. We would like to thank the participants and care providers for their voluntary participation. Finally, we would also like to acknowledge the granting agency, Canada Mortgage and Housing Corporation, for funding this project.

**Conflicts of Interest.** The authors declare no conflict of interest.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Toward a Trip Planner Adapted to Older Adults Context: Mobilaînés Project**

Bessam Abdulrazak1,2,3, Sahar Tahir1,2,3(B) , Souhail Maraoui1,2,3, Véronique Provencher3,4, and Dany Baillargeon3,5

 AMI-Lab, Université de Sherbrooke, Sherbrooke, Canada {Bessam.Abdulrazak,Sahar.Tahir,Souhail.Maraoui}@usherbrooke.ca Dept. Informatique, Faculty of Sciences, Université de Sherbrooke, Sherbrooke, Canada Research Center on Aging, Université de Sherbrooke, Sherbrooke, Canada {Veronique.Provencher,Dany.Baillargeon}@usherbrooke.ca School of Rehabilitation, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Canada Dept. Communication, Faculty of Literature and Humanities, Université de Sherbrooke,

Sherbrooke, Canada

**Abstract.** Mobility is essential for older adults to keep a good level of socialization, health and well-being. Still, aging is often accompanied by multiple mobilityrelated challenges. Hence, these mobility limitations make travel and use of public transport a big challenge. Numerous mobility trip planner tools can be found now days. However, they are not adapted to older adults' needs and preferences. We discuss in this paper our attempt to develop a one-stop platform to help older adults in their mobility where, when, and how they want. We introduce our co-creation approach, our software architecture, as well as our first prototype.

**Keywords:** Mobility · Older adults · Trip planner · Web application · Age-friendly trip planning

### **1 Introduction**

The proportion of older adults has continuously growth over the years, as stated by World Health Organization [1]. A variety of difficulties and barriers related to mobility frequently appear with aging including: **1**) Walking aspect: walking limitations, lower walking speed, loss of balance and incapacity to pass by a high slope road [2]; **2**) Fears aspects: fears often arise with advancing age [3], including the fear of not getting off at the right bus stop, not getting on the right bus, making the use of public transportation a real challenge [4]; **3**) Cognitive and Sensory abilities: vision and hearing deteriorate with aging [3] which may interfere with safe driving; **4**) Financial aspects: advancing age is also accompanied by life transitions, including retirement, which leads to diminishing financial resources [3] that do not favor the use of taxis by older adults. Consequently, these difficulties and barriers affect older adults' ability to move around safely and independently, prevent them from maintaining a social life as active as they wish. While aging, promoting the autonomy represents an important vector for social integration of older adults and their participation in the activities of daily life. In this context, various planning transit tools around the world were developed to help people to get easily to their destination and provide information about various modes of transport. These transit tools are helpful for general public and basic use. However, they are not tailored to all older adults' needs since they consider only a few aspects important to this population. We argue that a trip-planning tool providing personalized maps in terms of textual and graphical presentation and paths adapted to older adults' physical and sensory impairments and cognitive capacities is much needed [3]. In this context, we introduce Mobilaînés, Mobility as a Service (MaaS) platform which incorporates different modes and forms of transportation services, to help older adults move around where, when, and how they wish.

The rest of the paper is structured as follows. Section 2 presents Related work. Mobilaînés approach and mobility key needs are presented in Sects. 3 and 4. Section 5 introduces Mobilaînés architecture. Preliminary results are presented in Sect. 6. Lastly, the paper is concluded in Sect. 7.

#### **2 Related Work**

Trip planning tools were developed to help people reach their destinations through different modes of transport (e.g., car, taxi, bus, car sharing, bicycle, walking), give them details about the trip (e.g., directions, which bus to take, which station) and provide users the option of planning their trips whenever they want. Based on our analysis, most of the existing tools address the shortest/fastest path functionalities considering only one criterion (i.e., distance, time, or number of transfers), which make them not tailored to older adults' needs and preferences, except for a set of tools that provide some options and functionalities that might be useful for older adults. A few number of tools address the walkability either by minimizing walking (transit1, moovit2), or by considering the maximal tolerated walking distance and speed of the walk (Mobilitéit3, alternéo4). The prototype mPASS [5] customizes maps and paths to each user's needs and preferences, based on the user's profile and location. mPASS [5] provides users with accessible paths taking into consideration accessibility barriers and facilities like stairs and zebra crossing. Path2.0 [6] suggests accessible paths to users with disabilities by providing users with the shortest/fastest path as well as the path that was frequently utilized by other users with the same disabilities. *Transit* (See footnote 1) is a mobile application proposing paths without stairs and avoiding slopes; while ViaNavigo5, RATP6 and Google maps<sup>7</sup> suggest

<sup>1</sup> Transit trip planning tool:" https://transitapp.com/ (accessed May 16, 2022).

<sup>2</sup> Moovit Urban Mobility App in Sherbrooke, QC," https://moovitapp.com/sherbrooke\_qc-3064/ poi/en (accessed May 16, 2022).

<sup>3</sup> Mobilitéit trip planning tool: https://www.mobiliteit.lu/fr/ (accessed Oct. 19, 2021).

<sup>4</sup> Alternéo trip planning tool: https://www.alterneo.re/ (accessed Oct. 19, 2021).

<sup>5</sup> Vianavigo trip planning tool: https://me-deplacer.iledefrance-mobilites.fr/itineraire (accessed Oct. 19, 2021).

<sup>6</sup> "RATPtrip planning tool for Paris-France: https://www.ratp.fr/ (accessed Feb. 08, 2021).

<sup>7</sup> Google Maps App: https://www.google.ca/maps (accessed Oct. 19, 2021).

accessible paths for people with reduced mobility using a wheelchair. Furthermore, few tools give information about the crowd, in other words, road traffic (Google maps (See footnote 7)) and bus crowd (Google maps (See footnote 7) and RATP (See footnote 6)) that remains to be a useful information especially for older adults who want to avoid crowded places and roads. In the aim of encouraging active and ecofriendly paths, Citymapper<sup>8</sup> calculates respectively the burned calories and the amount of CO2 emission. Additionally, information about weather was given by Mobilitéit (See footnote 3), OC transpo9. To conclude, the presented trip planning tools don't consider the totality of older adults' needs and preferences. Therefore, there is a need for an approach that takes the diversity of older adult needs in consideration for building the trip planner, which was the first motivation behind our Mobilaînés project.

### **3 Mobilaînés Approach**

Mobilaînés is Mobility as a Service (MaaS) e-tool, in other words, a one-stop platform transport service combining different modes of transport and various forms of transport services to help older adults move where, when, and how they want. In order to meet older adults' needs, Mobilaînés is based on a living lab co–design approach, which facilitates the collaboration between stakeholders from various sectors to create, validate and test new technologies, services, products and systems in real-life contexts, and makes the interaction with older adults easier [7–9]. For this purpose, a steering committee composed of 8 stakeholders from the public, scientific, and private sectors as well as older citizens (n = 8) was formed. Like any other living-lab approach, Mobilaînés project comprises three phases, each with several iterative subphases (More details on [10]).


<sup>8</sup> Citymapper trip planning App. https://citymapper.com/ (accessed Oct. 19, 2021).

<sup>9</sup> OC Transpo trip planning tool: https://www.octranspo.com/en/ (accessed Feb. 08, 2021).

input the departure, destination address, trip time and the considered criteria to give as output the adapted path. *3) User interface:* The user interface implementation is done using an iterative process. Therefore, usage scenarios and mock-ups (static designs of the web application) were produced to present potential functionalities (content and interface).

3) **Evaluation**: Older adults with various physical and cognitive conditions and multiple mobility preferences will participate in the usability tests of Mobilaînés platform.

### **4 Key Needs to be Addressed**

The Mobilaînés team run a survey about older adults' mobility needs and preferences, to better address the requirement of the older adults in the development of the trip planner. Following is a summary of the retrieved six key values that highly impact older adults' mobility and needed to be included in the trip planner (more details can be found on [10]), with examples of how we can address them in Mobilaînés:

	- Example on how we address Health is by providing adapted walking options, e.g., paths that avoid slopes, recommend paths passing by benches to have a rest, consider walking speed in path planning.
	- Example on how we address Safety is by helping users plan their return before sunset (so they won't be out in darkness), and by consider less crowded roads.
	- Example on how we address QoL is by including agreeability for leisure trips, e.g., by recommending paths passing by parks.
	- Example on how we address Autonomy is by simplifying the user interface, and by providing various alternatives of paths so that the user can choose freely the path he wants to follow.
	- Example on how we address Eco responsibility is by providing older adults with information about CO2 emissions of each proposed path, and by promoting eco-friendly options, e.g., bike, public transportation.

### **5 Mobilaînés Architecture**

**Fig. 1.** Mobilaînés architecture

Creating an adapted trip planner tailored to older adult needs and preferences (that considers the six described key needs) requires 3 key components /Layers: **1) Data layer:** where we basically get data from various data sources to consider the older adults needs, **2) An age-friendly user interface:** older adults are not familiar with new technologies, therefore it's important to provide an easy to use and adapted user interface, **3) Mobilaînés engine:** an engine that is based on two main components: **3.2)** *Profiling***:** a key element for a better personalization and adaptation of a trip planner is to better creating an adequate profiles, and *3.2) Planification engine:* this component is responsible of a multicriteria algorithm optimization that considers various and multiple criteria and provide the best compromise of all (Fig. 1).

#### **5.1 User Interface Layer**

As people age, they inevitably experience certain physiological and cognitive changes that make the use of technology challenging [18]. Nevertheless, there are also older adults who aren't as comfortable or familiar with technology [19] therefore Mobilaînés platform should be designed in a way that makes of it accessible and adaptable to older adults. Before designing Mobilaînés, we first analyzed the existing Trip planning tools and tested trip planning mobile applications by 6 older adults aged over 65 years via a workshop. Results of the workshop confirmed that the existent tools have strengths and weaknesses regarding user interface adaptability to older adults. Most of the evaluated trip planning tools lack users' support and do not have indications where the user is in the process of planning his trip, most of them require multitasking what makes them not adapted. To better accommodate the six key needs (Sect. 4) we opted for building user interface by respecting the following four design principles: *1) Easy to use:* The website's text is adjustable, and it provides large buttons [20, 21]. Our user interface provides an easy navigation by having a flowchart that shows the step the user is in within his path planification. Furthermore, each page contains navigation buttons to go to the next step and go back to the previous step [22]. *2) Consistency:* The designed user interface uses icons used by other trip planning tools for the same purposes [22] (e.g., favorite icon, icons for the walking, bus). *3) Help and support:* To support older adults with digital literacy in using our platform, we also provide a contact phone call to reach an agent who is there to help in planning the trip [23]. **4**) *Minimizing memory load* the designed interface provides a summary page of validation to make sure that the older adult is aware of her/his inputs. It also shows recent trips for quick trip planning and give her/him the option to save a trip that can be reused.

#### **5.2 Mobilaînés Engine Layer**

Mobilaînés engine is a server application that exposes numerous methods for routing, signing up users, updating profiles, etc. The Mobilaînés engine is based on two main components:

#### **5.2.1 Profiling**

Mobility needs and preferences are diverse from an older adult to another they depend on the individual's health, psychological and financial conditions. Therefore, it's necessary to create a profile that captures this diversity and includes the user characteristics, physical capacities and limitations, mobility behavior, etc. By adding a profile component to Mobilaînés trip planner, we can include more information on user needs and preferences in the planning of the trip, and hence propose personalized and adapted paths that are as close as possible to what older adults is looking for. Adding information on Mobilaînés trip planner use will also allow to learn from user mobility behavior to improve Mobilaînés and propose better itineraries in a potential second version.

#### **5.2.2 Planification Engine**

The planification engine is a major component of our proposed Mobilaînés engine. Indeed, there are multiple open-source software tools and APIs that were developed in the purpose of trip planning and that includes routing engine. The two most prominent open-source solutions are NAVITIA10 and OpenTripPlanner<sup>11</sup> (OTP). We opted for OTP due to its powerful trip planners and the parameters taken into consideration in path planning (e.g., stairs reluctance, preferred routes, and banned routes). OTP includes a graph builder and a basic routing engine based on A-star and Range Raptor algorithms. Our additional layer focuses mainly on considering multiple criteria and consequently build a multi-criteria route plan algorithms.

#### **5.2.3 Data Layer**

The data layer is the architectural unit that provides our routing engine with all the necessary information and data to return the optimal route to the user. Data layer is linked to the platform users' profile, preferences, route history, and any other piece of information that proves useful to our routing algorithm. Various data sources are needed for a trip planer adapted to older adults' needs, including: 1) *Geolocalization data* to collect data about streets and amenities; 2) *City data* (of Sherbrooke in our case)12. More cities now have various open-source databases with rich information that can be beneficial for older adults' trip planning, e.g., information on snow removal that can help plan a safe trip; some that enables avoiding icy sidewalk in winter; and some that makes possible considering road to avoid road segments with affected by the work; 3) *Transit data*: Public transit agencies provide their transit data including trip updates, service alerts and vehicle positions and bus traffic in the general Transit Feed Specification – Realtime (GTFS-RT) specification [46]; and 4) *Weather data* (OpenWeather in our case) are used to consider sunset time, temperature and snowstorms predictions to propose safe trips.

### **6 Preliminary Results - Moblaines First Prototype**

To demonstrate that the architecture discussed having a practical potential, we developed the first prototype for Mobilaînés project, with the following components:

<sup>10</sup> Navitia trip planner platform: https://navitia.io/ (accessed Apr. 19, 2022).

<sup>11</sup> "OpenTripPlanner: http://docs.opentripplanner.org/en/latest/ (accessed Jun. 02, 2021).

<sup>12</sup> "Sherbrooke City Open Data." https://donneesouvertes-sherbrooke.opendata.arcgis.com/ (accessed Apr. 19, 2022).

#### **6.1 Data Acquisition**

We use in our first prototype: **1**) OpenStreetMap data for geographical data, to collect street data, data about public toilets, benches, etc., and **2**) GTFS data for transit, including trip updates, service alerts and vehicle positions, supplied by the bus company of the city of Sherbrooke.

### **6.2 Engine**

We use in our first prototype OTP engine, for non-transit trip (walking, bike, car) planning. The results generate a list of itineraries using A-star search, an algorithm that extends the Dijkstra's algorithm for better computation complexity. It uses heuristic evaluation function (in this first prototype we use Tung-Chew heuristic [24]**)** to calculate the costs of each neighboring node. However, for transit paths, the path is segmented into three segments [25]: **1**) Access from the origin to transit stops, **2**) egress from transit stops to the destination where we use. A star search algorithm (same as non-transit trips), and **3**) transit service connecting the two using the Multi-criteria Range Raptor algorithm [26]. For two given bus stations, it computes all Pareto-optimal journeys minimizing the arrival time and the number of transfers made—between them. RAPTOR [26] is round based, in other words, it operates in rounds and processes each route of the network at most once per round.

#### **6.3 User Interface**

We built our first prototype of the user interface for trip planning according to the result of our Exploration co-creation phase, putting in mind the usability of the interface by our older adults. The first prototype of the user interface is composed of four pages:


**Fig. 2.** Path definition page

**1**) Path definition (Fig. 2. Path definition page), where the user adds his departure and destination, he can do that either by typing an address or by selecting an area in the map. There is also an option to add addresses to favorites. Finally, the user sets the timeframe of the trip, to either leave now, arrive by, or depart at. This page also keeps track of the user's recent planned paths so they can be accessed directly. **2**) User options (Fig. 3. User options page.), on this page the user selects his needs and preferences (this page can be filled in automatically by the information stored in the user profile). **3**) the third page (Fig. 4) shows all itineraries returned by the routing engine that meets the user's selected needs and preferences so that he can select the path to follow. **4**) The fourth page (Fig. 5) is for validation, we make sure that the user is aware of what he chose as a departure/destination address, timeframe and selected needs and preferences. Finally (Fig. 6) the user can visualize the chosen path on a map, with directions, and information about this path.


**Fig. 3.** User options page.


**Fig. 4.** Available paths page


**Fig. 5.** Validation page

**Fig. 6.** Path's details page

### **7 Conclusion**

We introduced in this paper Mobilaînés, a trip-planning tool adapted to older adults needs that consist of one-stop platform transport service combining different modes of transport and various forms of transport services to help older adults move where, when, and how they want. We also introduced our adopted a co-creation process in Mobilaînés project. The process allowed to highlight the six key needs in mobility (i.e., Health, Safety, Quality of life (QoL), Equality, Autonomy, and Eco responsibility) and the technological services to address these needs. We also presented in the paper the Mobilaînés software architecture to implement the desired services, as well as the first prototype. Usability tests are planned to evaluate the prototype and identify gaps to be filled in the next version (that includes a multi-criterion optimization algorithms).

**Acknowledgement.** The authors wish to acknowledge the valuable contribution to this study made by the older adults and Mobilaînés team especially Catherine Girard and Sara Bahrampoor Givi, as well as all the partner-members of the steering committee and our collaborators. We also thank the Fonds de recherche du Québec for its financial support.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Data-Driven Smart Medical Rehabilitation Exercise and Sports Program Using a Living Lab Platform to Promote Community Participation of Individuals with a Disability: A Research and Development Pilot Program**

Seungbok Lee1,2, Yim-Taek Oh2, Hogene Kim3, and Jongbae Kim1,2(B)

<sup>1</sup> Yonsei Enabling Science and Technology Research Center, Yonsei University, Seoul, Republic of Korea

jongbae@yonsei.ac.kr

<sup>2</sup> Korea Wheelchair Rugby Association, Seoul, Republic of Korea

<sup>3</sup> Ministry of Health and Welfare, National Rehabilitation Center, Seoul, Republic of Korea

**Abstract.** Patients discharged from hospitals following the onset of an acute illness or injury rendered with disabling conditions require systematic medicalbased and rehabilitation-focused sports and exercise programs accessible in their communities. This proposal aims to build a data-driven smart health system that allows people with disabilities to continuously improve their health by alleviating modifiable factors, including architectural barriers and related challenges following discharge from an inpatient hospital or rehabilitation course. Our goal is to promote a multi-ministerial data-driven innovative medical exercise system using a digital living lab platform as a testbed program to provide lifestyle exercise and physical education for community-dwelling individuals with disabilities. The pilot program of services will be rendered at the living lab center of the National Rehabilitation Center, equipped with data servers for storing accumulated pertinent information and continuous data acquisition. We envision an encrypted data collection and acquisition system, whereby newly acquired data will be merged with data information from original records of individuals generated during the inpatient hospital course.

**Keywords:** Smart medical · Rehabilitation sports and exercise · Data · Living lab · Disability

### **1 Introduction**

Upon completing an inpatient rehabilitation course, most patients discharged from the hospital face medical complications, which places them at high risk for readmission [1, 2]. Without proper exercise programs, equipment, and guidance by trained exercise professionals, to promote their medically proven health care, they are falling into the abyssal valley of death between hospital care and community sports. This is especially true for individuals with a spinal cord injury or stroke, whereby the rendered paralysis usually results in varying degrees of impaired mobility [3, 4].

In the absence of an appropriately equipped environment conducive to promoting self-directed care in health and medical management, individuals with disabilities are denied equitable participation in many aspects of community life. And thus, after receiving rehabilitation treatment from an inpatient hospital setting due to an illness or accident that results in disabling conditions, patients require a systematic medical-based and rehab-focused sports and exercise program within their dwelling communities.

These programs should be comparable to or mimic the services rendered from the perspective of an institutional setting. It should also serve as a vital component in the overall health management and continuity of care within a community environment. However, local communities lack the necessary infrastructure—workforce, space, equipment, and devices— encompassing a program of services that focuses explicitly on smart medical-based and rehab-focused exercise and sports activities catered to these individuals [5].

Furthermore, from an economic perspective, studies have substantiated cost-benefit aspects based on total annual medical and healthcare expenses for individuals with disabilities by comparative studies analyzing expenditure outcomes between individuals with disabilities who engage in some form of regular exercise activity as opposed to those who did not regularly participate in exercise programs [6, 7].

And thus, as the marginalization of the disabled population—inequity of essential healthcare amid a public health crisis—continues to be the social norm globally [8], such a provision for this population undoubtedly yields equitable opportunity in promoting their health and well-being as that of the non-disabled members of their communities at large [4].

We intend to build a system that allows people with disabilities to continuously improve their health by alleviating modifiable factors and related challenges after being discharged from the hospital. This study aims to promote a multi-ministerial data-driven smart health system using a digital health living lab platform as a testbed program for providing lifestyle exercise and sports-related physical education for communitydwelling individuals with disabilities.

#### **2 Paper Preparation**

The common aim was to promote the activities and exercises of people with disabilities in the community through joint efforts of multi-ministries to integrate resources and infrastructure from the medical, technological, economic, and social sectors. Through a governmental competition among 108 candidates, strategic planning with a large grant was awarded to the National Rehabilitation Center (NRC) in the Ministry of Health & Welfare by joint efforts from the three related ministries: Ministry of Culture-Sports & Tourism, Ministry of Science & Technology, Information & Communication, and Forest administration. To fulfill the tasks, appropriate requests for proposals (RFP) were designed by each ministry.

The collaboration of tasks and assignments span broadly under the tutelage of four governmental ministries: 1) the Ministry of Health and Welfare, which oversees medical exercise services and its distribution as a leading role; 2) the Ministry of Culture, Sports and Tourism, which oversees exercise programs in the community; 3) the Ministry of Science, Technology Information and Communication (ICT), which oversees exerciserelated data service and information technology.

The Ministry of Health & Welfare, NRC, has announced 21 projects as a leading role. The individual projects are categorized into four disciplines: 1) "Partnership and Collaboration," 2) "Equipment and Biometric Technology," 3) "Data Integration," and 4) "Exercise Programs and Services." The assigned projects by recipients of various academic institutions and business corporations are summarized as a figure which is not included herein but is available as a supplement. Each task consists of a consortium of universities and business corporations collaborating on delegated sub-tasks, ultimately establishing the NRC 'Smart Rehabilitation Exercise Living Lab Center'.

#### **2.1 Roadmap and Plan of Action**

The defined roadmap for this multi-faceted endeavor of building the smart rehabilitation exercise living lab center requires integration with collaborative efforts from multi-ministerial forces. The collaborating entities will work toward data acquisition and service connection using input from the public and private organizational resources. Field application within the designed living lab center will also require expert guidance from clinicians, therapists, and allied health professionals from disability sports and physical education arenas. The action plan spans a three-year timeline focused on accomplishing three broad aspects.

The first year (2021) was dedicated to the detailed design of the smart medical living lab platform. This was achieved by collecting opinions from the multi-ministerial council and advice from expert advisory groups. Participating entities successfully received institutional review board approval for clinical trials involving human participants. Platform integration was also accomplished by connecting with the Ministry of Health and Welfare representatives to oversee medical services distribution, the Ministry of Culture, Sports and Tourism to supervise the exercise and sports activities, and the Korea Forest Service.

In 2022, Year 2, the focus was to set up and execute operation plans for the smart medical living lab. The center will be set up and housed at Korea National Rehabilitation Center. The prototype module development and integration of various established platforms will be connected under the multi-ministerial linkage efforts. The recruitment efforts and enrollment of study participants for the preclinical trials with testbed services of the program activities will get underway.

The focus in the final period, Year 3, is to verify and stabilize operations efforts of the disability exercise center via expansion establishment of the model nationwide. The final evaluation and assessment will be performed along with data linkage with the multi-ministerial entities.

The described pilot services in physical education and program of activities will be rendered at the living lab center of KNRC. The center will be equipped with data servers to store continuously acquired data and accumulate pertinent information. We envision an encrypted data collection and acquisition system, whereby newly acquired data will be merged with data information from original records of individuals generated during the inpatient hospital course.

Twenty-one tasks have been delegated across various institutions and corporations. These include four assignments for systems, seven for equipment and device development projects, five studies in physical education and exercise programs, and five assignments in data linkage.

#### **2.2 Pilot Test Run**

We acquired preliminary data using the above data set and conducted a testbed run. The study sample (n = 7) comprised five athletes and two coaching staff from the Korea Wheelchair Rugby Association. All participants were of male gender and had a varying cervical level of neurological injury (C5-C6-C7); one with thoracic level injury (T3). The para-athletes were initially examined by a staff physiatrist trained in spinal cord injury medicine. The basic clinical characteristics of the participants were collected followed by an assessment of 'functioning status' using the 6-min Walk Test (6-MWT) via wheelchair propulsion.

### **3 Results**

The dataset we envision collecting via purchased equipment, manufactured devices, apparatus, and survey questionnaires from personal interviews comprises 208 pieces of information. This dataset is broadly categorized into three sub-datasets: 1) Personal information (PI), 2) Evaluation Data, and 3) Exercise Data. In the Evaluation Data category, it consists of four sub-categories. These include medical and community records about general health conditions, which were evaluated, 1) during hospitalization (dH), 2) in clinics after hospitalization (iC), 3) from a personal trainer (PT), and 4) National Insurance in South Korea (NI). These periodical evaluations are able to determine each individual's health condition at the time of clinical evaluations, which would be compared to community exercise intervention programs. The Exercise Data category consists of two sub-categories. These include 1) the biometrics of exercise (BE) using physiological sensors, e.g., heart rate, oxygen saturation during the exercise, and 2) the amounts of exercise (AE), frequently described by parameters of Frequency (# per a week), Intensity (Exertion and speed), Time (mins per one time), Type (Aerobic/Anaerobic, Cardiac, Strengthening, Resistance, etc.). If there exist intersection parameters in evaluation/exercise sub-categories, each recorded time would be different to discriminate the health conditions.

These evaluation data can be further categorized into six sub-datasets in terms of data characteristics. Table 1 summarizes the demographic information (#1–10); Table 2 shows (1) Body composition (#11–30), (2) Administrative medical and health data (#31–40: medical and surgical histories; lab draws; diagnoses), (3) Functioning status (#41–90: physiology; biomechanics; body function test), (4) Quality of Life and physical assessment (#91–101: social; mental; general). Table 3 shows the exercise data (#102–130) summarizing the exercise performance in the program of activities. These data were examples of parameters collected from the manufactured devices: balance & coordination exercise; anaerobic exercise; aerobic exercise, which is generated by the exercise equipment manufactured by the participating companies in current projects.


**Table 1.** Clinical characteristics (Personal Information (PI))

#### **3.1 Results of Pilot Study**

The participants were all male and had a mean age of 29.7, a range of 19 to 46, and a median age of 25. One of the participants had a thoracic level injury (T3), while the remaining six were cervical level injuries. These and other basic characteristics are summarized in Table 4. The parameters measured in the 6-MWT included heart rate (HR), rate of perceived exercise (RPE), respiratory failure (RF), metabolic equivalence in exercise test (METs), maximal oxygen consumption (VO2/kg(ml)), minute ventilation (VE), respiratory exchange ratio (RER(R)), distance propelled in meters, and reason for discontinuing when applicable. The measurement also included resting BP and HR and post-exercise BP and HR. The acquired data are reported in Table 4 (Personal Information as "Clinical Characteristics"), however Table 3A (Functioning Status as measured by the "6-MWT") data are not reported herein but are available as a supplement.








**Table 3.** Exercise data



**Table 3.** (*continued*)

**Table 4.** Clinical characteristics


#### **4 Discussion**

Upon the onset of an illness or injury requiring medical treatment, patients are usually admitted to a hospital facility to receive clinical care. Herein, patients undergo appropriate medical and or surgical treatment for their illnesses. The primary focus is strictly medical stabilization and health maintenance. During this inpatient hospital phase- toward a path of recovery--medical professionals of varying disciplines manage and provide essential care for patients, including physicians and allied health professionals.

Under appropriate settings, a medically prescribed rehabilitation course may also be indicated in many situations, especially following the onset of a disabling injury. The goal from a rehabilitation perspective is to enhance patients' functioning ability, thereby ultimately ensuring a safe and stable transition phase in their recovery toward community re-integration. This is achieved through a prescribed rehabilitation program consisting of conventional occupational (OT) and physiotherapies (PT). Depending on the facility, this program may include a specific training period focusing on activities of daily living (ADL) within a simulated home and community environment before discharge.

This simulated home and community environment—a data-driven smart medical living lab center housed at the National Rehabilitation Center of Korea—serves as the fulcrum. The linkage of pilot services' multi-ministerial rehabilitation program will be linked to the disabled population throughout the local communities. Immediately after discharge from the hospital, patients will transition to a "smart medical healthy" living lab environment before returning to their respective domiciles in the community. Patients will spend several days in a simulated home and community setting to undergo training in various activities of daily living. The medical-based program focuses on rehabilitation exercise and sports activities specifically designed to aid these individuals in developing a habit of management in self-directed care and health maintenance. Unlike the inpatient acute hospital course, the setting herein does not require licensed clinicians to provide these services. And thus, participating individuals will be instructed and trained by non-clinical professionals with expertise in disability sports and physical education to ultimately ensure safe functioning status toward a stable community re-integration.

#### **4.1 Linkage of Multi-ministerial Rehabilitation Platforms**

The testbed participants will be recruited from the NRC inpatient population as part of their discharge planning from acute inpatient rehabilitation and individuals with chronic disabilities from the local community. The collected hospital data will be merged with newly acquired information under the tutelage of the four governmental ministries to accumulate a data bank using the onsite server.

A living lab testbed built and operated by the NRC is currently in the development and establishment phase. It is expected to open and get underway by recruiting participants from the National Rehabilitation Center's inpatient population and local communities. A smart health medical rehabilitation-based and data-driven living lab will render the pilot service R&D efforts designed to provide rehabilitation and physical education programs for individuals with a disability in the local community.

#### **4.2 Effects of a Collaborative Research & Development**

The overarching benefits of this integrative research and development (R&D) are multifold. The many advantages are realistic and expected to infiltrate many sectors of our society.

Medical benefits will improve healthcare services through an integrated rehabilitation model with increased therapeutic efficacy expected to result via the rehabilitation exercise guidelines applied to various disease models.

The benefits from a 'technology' perspective are also multi-fold. We expect that there will be further advanced development of technology for enhanced data acquisition that will trigger commercialization with the expansion of various contents applicable to other research and development areas.

The economy can also benefit by an overall reduction in the cost of healthcare for all ages and social classes, especially for the elderly and marginalized population. This will likely be achieved by providing welfare services for the financially destitute and socially disadvantaged. Industries can also work hand-in-hand to benefit from the expansion of the technology development into overseas markets, fueling the creation of job opportunities for the unemployed.

Social advantages include reduced healthcare costs and mortality by promoting a healthy lifestyle and self-directed care in health management. This will mainly apply to the general population as people are aging and living with more chronic disabilities. The creation of more social platforms is also foreseeable in advantageous ways yielding an artificial intelligence and smart device-driven society.

#### **4.3 Pilot Study of Para-Athletes**

Our initial testbed run yielded results comparable to output under the circumstances from individuals with functioning disabilities stemming from spinal cord injury. The acquired data were not statistically analyzed in any specific manner or for comparison purposes. The raw data are provided to convey the intent of pursuing the upcoming complete study applying "Clinical characteristics" (Personal information; Table 4) and "Functioning status" (6-MWT; Table 4A) from the proposed data set. Our goal is to substantiate the effects of the smart-medical living lab simulated herein, which will apply to and ultimately lead to a governmental establishment of these fitness program centers specifically designed for individuals living with disabilities nationwide.

### **5 Conclusion**

Patients with disabling conditions due to an acute onset of illness or devastating injury are discharged without proper training and education for safe return to and participation in their communities. They require a systematic medical-based and rehab-focused sports and exercise program accessible within their respective communities. A smart medical rehabilitation-based and data-driven living lab center can provide the necessary platform for successful community re-integration.

**Acknowledgments.** We thank Dr. Junghwan Kim and Dr. Hyosun Kwon at National Rehabilitation Center for their helpful comments and great support on this project. We also extend our appreciation to Mr. Yunho Kim and the pilot test participating athletes of the Korea Wheelchair Rugby Team.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Real-Time Human Activity Recognition in Smart Home on Embedded Equipment: New Challenges**

Houda Najeh1,2(B), Christophe Lohr<sup>1</sup>, and Benoit Leduc<sup>2</sup>

<sup>1</sup> IMT Atlantique, Lab-STICC, Brest, France *{*houda.najeh,christophe.lohr*}*@imt-atlantique.fr <sup>2</sup> Delta Dore Company, Bonnemain, France bleduc@deltadore.com

**Abstract.** Building Energy Management (BEM) and monitoring systems should not only consider HVAC systems and building physics but also human behaviors. These systems could provide information and advice to occupants about the significance of their practices with regard to the current state of a dwelling. It is also possible to provide services such as assistance to the elderly, comfort and health monitoring. For this, an intelligent building must know the daily activities of its residents and the algorithms of the smart environment must track and recognize the activities that the occupants normally perform as part of their daily routine. In the literature, deep learning is one of effective supervised learning model and cost-efficient for real-time HAR, but it still struggles with the quality of training data (missing values in time series and non-annotated event), the variability of data, the data segmentation and the ontology of activities. In this work, recent research works, existing algorithms and related challenges in this field are firstly highlighted. Then, new research directions and solutions (performing fault detection and diagnosis for drift detection, multi-label classification modeling for multi-occupant classification, new indicators for training data quality, new metrics weighted by the number of representations in dataset to handle the issue of missing data and finally language processing for complex activity recognition) are suggested to solve them respectively and to improve this field.

**Keywords:** Real-time human activity recognition *·* Deep learning *·* Smart home *·* Sensors *·* Embedded equipments

### **1 Introduction**

Smart home services are an important provider of systems of technology. They focus on constructing, operating and maintaining buildings in the most cost and effective efficient manner. At the most primary level, smart homes deliver functional building services for occupants by providing them security, air quality, thermal comfort, appropriate illumination, at the environmental impact as well as the lowest cost and over the building life cycle [20]. Reaching this vision needs adding intelligence from the design phase's beginning throughout to the end of the buildings useful life. They use information technology during operation to connect different subsystems, which operate independently, so that these systems can share information to optimize the performance of the building. They look beyond the building equipment within their four walls. They are connected to the smart power grid, and they interact with building operators and occupants to empower them with practical information and new visibility levels.

Building system is decomposed into two parts: the physics with an important number of appliances and building envelope, controllers for energy management, as well as a human part associated to building occupants. These two parts interchange: the physical part is modeled and controlled by the occupants, while the occupants waiting for new services from the physical part. Research works about buildings mostly focuses on physical part and disregards the human one. However, the elaborate hardware and the most advanced software in the real applications be nothing but transistors and wires and without the people that use them to work more functionally. In that sense, the occupants that run a smart home are a crucial component of its intelligence. Consequently, in order to define the state of the buildings, non-measured values should be known not only in the physical part (i.e. air flow) but also human (i.e. occupancy and activities). Household occupants carry out different activities each day as part of their normal quotidian behavior. These include getting up, taking a shower, switching on lights and preparing breakfast. While such activities are common to us all, there will be surely differences [11]. For example, one household may use the shower at different times of the day, while others might prefer to take a bath.

In the literature, an important number of HAR techniques are available. They have significantly improved the use of deep learning techniques, and they could be classified into two categories. The first one is real-time. The HAR systems are a reactive system that depend on their environments and are in permanent interaction with them. They must respond to the latter's stimuli while respecting certain constraints, the most important of which is the time constraint. These systems use real-time data (i.e. the activities are detected while they are being performed) and in which the data describing activities are partially observed. The second is offline using historical data (i.e. the activities are recognized when they are completed) and in which the data describing activities are completely observed [3]. By way of explanation, regardless of the rate of sampling, the data captured from uncompleted activities contain fewer details about the activities compared to the data captured from completed activities, and this makes realtime HAR a more challenging problem compared to the offline HAR.

This work presents an overview about the latest advances in the field of realtime HAR in smart home to obtain a better understanding of the difficulties, challenges and opportunities. The main contribution is to highlight the trends regarding the real-time HAR task from pattern classification (Sect. 2) and to propose new solutions for data segmentation (Sect. 3), data variability (Sect. 4), datasets pre-processing issues (Sect. 5) and complex real-time HAR (Sect. 6). A discussion and a conclusion are provided respectively in Sects. 7 and 8.

#### **2 Pattern Classification**

Various deep learning models have frequently been used by researchers in the HAR field. The learning generally involves creating a statistical or probabilistic activity model that is augmented with large training data. The task of the model is to learn and recognize patterns that differentiate various classes in the training data, and apply this knowledge for the classification of test data.

The HAR task proceeds from pattern recognition, where the techniques are decomposed on two categories.

#### **2.1 Ontology Based Approaches**

Various techniques have been proposed in the literature to tackle the HAR task, and most of the basic ones focus on location because it is a crucial part of the context. They assume that the semantics of a spatial position are static, that's why they focus on how to specify the semantics of a spatial position and also focus on how to identify the spatial position of occupants [31]. Theses techniques work well to some scope. However, they are unable to recognize human activities with enough accuracy. For example, if the occupant stays at a kitchen, she or he may be cooking food or putting the dishes away; the efficacy of this method is limited because a location offer various activities. As an example, a living room can support various activities such as eating, playing TV games or reading. For now, research works fails to handle these changes in semantics.

Another type of ontology-based approaches is thing-based techniques that identify dynamically the change in activities by using remote sensors to identify the objects that the occupant is interacting with [28]. For example, to recognize an activity, [32] developed a system of semantics identification that identified the activity space from objects forming the immediate environment of the occupant. These techniques try to overcome the diversity of activities in the most comprehensive way possible. However, they require a complete knowledge of the domain to elaborate activities models. They are not robust to handle settings change and uncertainty, and they have implementation problems regarding the recognition accuracy of the used sensors depending on their cost. In addition, they propose to follow expert knowledge to model activities, which is time-consuming and difficult to maintain in case of environment evolution. In this paper, we propose modeling this type of activity as a multi-label classification problem. We could use a semantic segmentation of sensors and activities and inspire from techniques developed in the literature such as [23] for example. Another solution is to exploit CNN architectures. This allows the model to relate the possibility that different activities could be distinguished for the same occupant.

#### **2.2 Data Driven Approaches**

Another approach to recognize human activities in real-time is to monitor the status of different appliances to understand the behavior of the different loads in the household. In the literature, several load monitoring techniques can be implemented. They can be divided into two categories [7,27]. The Intrusive Load Monitoring (ILM) is a data-collection method where the devices are installed at each appliance node to detect the sensor events and therefore characterize in detail the occupant activities using deep learning techniques, for example.

The databases generated by these systems can be labeled manually (i.e. the user label the monitored appliance), or automatically (i.e. the system is trained with examples from distinctive appliances and then recognizes the appliance that is being used). Generally, manual setup ILM systems master automatic setup ILM systems. Result's accuracy is the main profit of this method, but it requires expensive and complex installation systems.

Non-Intrusive Load Monitoring technique (NILM) is an alternative process, in which one single monitoring device is installed at the main distribution board at the household, and an algorithm is applied to determine the state of operation for each individual appliance. The main advantage of NILM is the fact that only one single monitoring device is needed. Therefore, it lowers significantly the cost and the intrusion at the household level. The main inconvenience is the lower accuracy compared to ILM systems, in particular, those with manual labeling.

In general, the appliance of a household can be categorized in the following classes: (i) finite-state appliances such as dishwashers or fridges, which have different states, each one had its duration (cyclic or fixed) and its own power demand; (ii) continuously varying appliances, such as computers, which behavior is not periodic and have different states; (iii) on/off appliances such as light bulbs, which have only two states: on and off, with a fix power demand. The duration depends on the user; (iv) permanent demand appliances, such as alarm-clocks, which are always ON with a fixed power request.

The appliances can be identified by "event-based" techniques that detect the On/Off changes, or by "non-event-based" methods that detect whether an appliance is ON during the sampled duration. These techniques can use different sensor measurements and features data, such as current and voltage measurements and signal waveform. The sampling rate is an important parameter in the complexity level of the methods of discretion. It affects not only the type of feature that can be measured, but also the type of algorithm that can be used. A detailed review about the features and algorithms is discussed in [22].

A high frequency sample data rate (1 s–1 min) permits more detailed analysis and accuracy in the detection of appliances loads. However, the large amount of data needs higher quality hardware and requires a capacity of processing and storage (locally or in the cloud) to run the algorithms. Another important issue in data driven approaches is related to the quality of data. In this work, we are interested in dealing ambiguous outliers detections in both training and testing data and insufficiency of labeled data. Outliers could be detected using a multiclass deep autoencoding model for example. Some autoencoder variants, such as a variational autoencoder, can be used to better extract high-level features for the estimation network. We are interested also to perform experimental results to prove the outstanding performance of the proposed solutions and compare them with state-of-the-art rival methods.

### **3 Data Segmentation**

In HAR task, each sensor data need to be divided into chunks, in a windowed manner. On each window, the features are computed, and then are used as an instance for learning phase. This task is difficult since occupants perform activities continuously, and successive activities can not be clearly distinguished. This section details the most used data segmentation techniques in real-time HAR context.

#### **3.1 Time Windows**

Time windowing (TW) techniques consists on dividing the data stream into time segments with a regular time interval. The selection of optimal duration of the time interval is the biggest challenge for these techniques. In fact, if the time interval is very large, the events could be common to many activities and consequently the predominant activity in the time window will have the more influence in the label's choice [4]. TW techniques are commonly used in the segmentation of sensor event for real-time activity recognition. This technique is more favorable to the sensor time series with regular or continuous time sampling. However, in the smart home context, sensor data are often generated in a discrete form along the timeline where a fixed size time window is not suitable.

#### **3.2 Sensor Events Window**

Sensor events windowing technique consists on dividing the whole sequence into a set of sliding windows with an equal number of sensor events [16]. Each window is labeled with the last event's label in the window, and the sensor events that precede the last event in the window define the last event's context.

It is easier to implement; however, the challenge consists on the fact that the actual occurrence of the activities is not intuitively reflected [18]. In fact, one sliding window could cover two or more activities, or sensor events belonging to one activity can be bust into several sliding windows. Furthermore, if several activities occur simultaneously, this technique is unable to segment sensor events. In addition, this window type differs in terms of duration, and it is consequently impossible to interpret the time between events.

To solve this problem, a method that segment the sensor events into portions that are consistent with the incidence of each activity is proposed by [15]. This approach can correctly outline the extremities of the activity, but it could take a longer time until sufficient information to define one segmentation is received. Moreover, the question that arise is how to determine if two sequential sensor events belong to similar activities or not? This is a challenge.

#### **3.3 Dynamic Window**

Unlike the previous techniques, dynamic windowing uses a non-fixed window size. In the literature, [26] proposed a dynamic segmentation model where the window is reduced and expanded based on the use of the current state of activity recognition, the sensor data and the temporal activity information. It is a twostage methodology [21]. An offline phase consists in splitting the data stream into events windows, then extracting the "best-fit sensor group" from the event window. In the real-time phase, the dataset is streamed to the classification algorithm. When the "best-fit sensor group" is identified in the stream, the corresponding label is associated with the given segment's input by the classifier. A non properly annotated dataset is the most challenging for the use of the technique. Also, it is not able to handle complex HAR in real-time [1].

#### **3.4 Fuzzy Time Window**

Another important type of data segmentation is fuzzy time windowing (FTW) [14]. The objective of this window's type is the generation of features for each sensor time series according to its evolution for a given interval of time. It is created for encoding multi-varied sequences of binary sensors.

To predict the right activity label, one should not rely solely on the past because in some cases, delaying the recognition time allows a better decision to be made. Consider the following example, if a binary door contact sensor is activated, the activation can be associated with the following activity "the resident has left the house". However, it can happen that the inhabitant only opens the front door to speak to another person at the entrance of the house and returns home without going out. To improve the precision, the use of the activation of the following sensors is a good alternative, and it will be useful for example to introduce a delay in the decision-making. The longer the delay, the greater the accuracy. However, a problem may arise if this delay is too long and, in effect, the delay prevents real time recognition. While a long delay may be acceptable for some types of activities, others require very short decision time in the event of an emergency, such as a resident falling. Furthermore, this method is applicable only to binary sensors' data, although this is not always the case because a smart building also contains non-binary sensors such as *CO*<sup>2</sup> concentrations sensor.

#### **3.5 Outline**

Many applications implement real-time activity recognition using wearable sensors such as accelerometers. HAR using these devices is less challenging because the data are generated continuously with fixed frequency, allowing data to be segmented based on the number of sensor signals or time intervals, and they are limited only sample on simple activities such as walking, cycling. In contrast, smart home ambient sensors are normally generating data in a discrete manner; so, this still remains a challenge for dynamic segmentation. Besides, the classified activities are often complicated, composed of a number of sub-activities where the duration of sensor segmentation and the boundary are difficult to determine.

In this work, we aim to develop a new dynamic real-time sensors' event segmentation approach to sensor data analysis which incorporates two components: sensor correlation computation, and time correlation calculations.

### **4 Data Variability**

In the case of HAR, most variations occur in settings and data. The following subsections detail the variability in settings and data issues.

#### **4.1 Variability of Settings**

Involving inhabitants from residential buildings is still a key challenge because each smart home is unique due to its architecture, configuration of sensors and equipment, but mostly because of its inhabitants' different profiles. The site models, the sensor locations and human activities/preferences are mostly not available, however, these are required for mass deployment solutions [27].

Some may be small, such as a single apartment, where the sensors may be fewer and have more overlap and noisy footage. Others may have multiple rooms and contain many sensors and devices. Indeed, the performance of an activity recognition system can be influenced by the number, type and location of smart home sensors. Therefore, a model optimized in one house may not work well in another because of this difference in architecture.

In this work, we are interested to test the transfer learning methods for solving this problem [10]. These techniques allow the use of pre-trained deep learning models with different data distributions. We are also motivated to analyze the different types of knowledge that can be updated with deep learning algorithms and taking advantage of recent advances in transfer learning for deep learning.

#### **4.2 Temporal Drift**

Thanks to the large number of sensors as well as the interaction between the occupants in a smart home, the initial training data constitutes a mirror of the activities carried out at the time of registration. A model is generated and trained using this data.

Habits and behavior of occupants may change over time. The data collected in real time is no longer the same as the data used in the training phase. This is translated by the concept of temporal drift as presented in [24], and means a change in the distribution between the training data and the test data. Let's consider the following example. A primary approach, which is prevalent in many buildings, is to use passive infrared (PIR) sensors for activity estimation. However, motion detectors fail to detect activity when the occupants remain relatively still, which is quite common during activities like regular desk work. Furthermore, drifts of warm or cold air on objects can be interpreted as motion, leading to false positive detection.

The statistical properties of a variable that the model attempts to predict change over time in unexpected ways. Taking this drift into account requires realtime adaptation to changes in human activities using new HAR algorithm data in smart homes [19]. Based on real-time monitoring of human activities, we are interest to perform Fault Detection and Diagnostics (FDD) but also predictive maintenance. It is indeed possible by artificial intelligence algorithms to detect drifts and intervene before it is too late. Numerous research studies still make it possible to develop these techniques to make them truly operational, in particular to adapt easily to changes in piloting mode and to different configurations of the building [29].

### **5 Datasets Pre-processing**

Following a data collection, the performance of activity recognition is evaluated with datasets generated through different sensors.

In a dataset, most of the activation data of sensors are not annotated because the dataset has been built with predefined activity sets to be labeled. Consequently, the activation of sensors linked to other activities are annotated under the "Other" label. This special class represents more than 50% of the datasets studied, provided by the Center for Advanced Studies in Adaptive Systems (CASAS) [8]. Event sequences of this class can be similar to event sequences of normal classes, as they can be quite different from each other. The difficulty of classification increases in proportion to this growth of the "Other" class. This leads to class confusion. In addition, each sensor activation gives only some information about the current activity. For instance, the activation of the kitchen motion sensor could indicate different activities like cooking, housekeeping or washing dishes. Therefore, the information offered by this sensor is exploitable only in coincidence with neighboring sensor activation. Thus, the time series is irregular and sparse, and it contains highly variable data and unbalanced classes.

Labeling is also challenging. In fact, in supervised learning techniques, which are extensively used in a lot of HAR applications, the problem of the required target arises in the determination of the activities of occupants i.e. the labeling issue is usually addresses using video cameras. In order to accept the privacy of occupants, the use of cameras is generally not acceptable in many places.

Also, the labeling is time-consuming. In general, the datasets for an actual building are labeled by the residents themselves using a graphical user interface and then used post-processed for the validation of research works. Missing labels are the most challenging problem in datasets. Let's consider the example of CASAS dataset developed by Washington State University, the activation of sensors may not be correctly ordered in a chronological order of timestamps. Also, duplication of days and events (i.e. same activity label, timestamp, sensor and value) could arise in a dataset and a cleaning step should be considered before the training process. Figure 1 presents an extract from the Milan dataset where labels are missing.


**Fig. 1.** CASAS dataset anomaly

The contribution in this work is to take into account the annotation of events in the judge of performance using evaluation metrics weighted by the number of representations in a dataset, F1 weighted score or balanced accuracy when the dataset classes are unbalanced [12].

#### **6 Complex Human Activities**

An activity consists of a pattern of multiple actions over time. Typical examples include reading, coffee time and cooking. The occupant's activities could be classified into two categories. Simple activity refers to primitives that realize a simple purpose or function, while complex activity consists of the temporal combinations of multiple simple activities over time.

In the literature, an important number of research works have been developed in the framework of real-time HAR. However, they focused mostly on simplified scenarios involving simple activity recognition and a single occupant, and there is no much attention on the real-time complex HAR.

In a house setting, occupants perform frequently different actions simultaneously in a variety of temporal combinations. That's why, activities are much more complex than actions, but they represent more the real life of residents.

Occupant activities are frequently carried out in a complex manner. Activities can be carried out in an interleaved or concurrent manner. An occupant may alternately wash dishes and cook, or listen to music and cook at the same time, but could just as easily wash dishes and cook, alternately, while listening to music. The possibilities are limitless in terms of activity planing. However, some activities could not appear in the dataset and could be also abnormal, such as cooking while the individual sleeps in his room. Modeling this type of activity is a challenge and only few research works are presented in the scientific literature. However, it could be modeled as a problem of multi-label classification. For example, in [23] a semantic segmentation of sensors and activities has been investigated and this allows the model to relate the possibility that certain activities may or may not occur for the same resident simultaneously.

Multi-user activity is another challenge. Besides, while the monitoring of a daily single resident activities is already a hard task. The complexity increases more with many residents. The same activities become more difficult to recognize for the following reasons. Firstly, in a group, a resident may interact to do common activities. In this case, the sensor activation reflects the same activity for each resident in the group. Secondly, everyone can perform simultaneously different activities. This produces a concurrent sensors' activation for different activities. These activations are then merged in the activity sequences, where an activity performed by one occupant is a noise for the activities of another occupant. Some researchers are interested in this problem [17]. However, most of the existing methods are of the offline type and there is no interest to date for the recognition of complex activities in real time.

The real-time activities that a smart homes want to recognize can be on the contrary seen as sequences of micro actions. These sequences generally follow a certain pattern, but there are no strict constraints on their compositions or the order of micro actions. To solve this problem, an important number of research works are present. A two-layer based LSTM technique is investigated to address the diversified composition of actions for HAR based on wearable sensors in [30] and based on videos in [9]. Another idea consists by exploiting the techniques developed by the natural language processing field, where the word's context vary, the words vary and the texts have a multilevel hierarchical structure.

Embedding techniques, such as BERT [5] and ELMo [6] have been investigated to operate sequential data. These techniques help the processing of long sequences. However, all these techniques are only experimenting with offline HAR. The study presented by [5] is limited by a certain size of possible sensor activations and vocabulary, and it is consequently impossible to obtain a representation of sensor values that have never been observed. In this work, we are motivated to investigate methods that could capture more semantics in realtime, i.e. that could split words and represent sensor activation into sub-words such as byte pair encoding (BPE) [25].

### **7 Discussion**

In this survey, we have highlighted the key challenges in real-time HAR in smart buildings. A taxonomy of the main components of a real-time HAR algorithm (time series analysis, data segmentation, data variability, data pre-processing and classification) has been introduced.

The data variability is still an unsolved problem because sensor data are very sensitive to the localization of sensors as well as the house configuration. We are interested in (i) performing Fault Detection and Diagnostics (FDD) to detect drifts using artificial intelligence algorithms to solve the temporal drift, and (ii) the use of transfer learning algorithms to solve the variability of settings.

Multi-occupant and concurrent activities' recognition are the most challenged problems in human activity recognition task using ambient sensors. We are motivated to work on this issue by modeling this type of activity as a multi-label classification problem or using CNN architectures.

Taking into account the quality of the training data was a problem rarely discussed in the literature. In fact, as the available information changes over time, the structure of the training data should also be readjusted to deal with such dynamic aspects. In [13], the authors have evaluated big data quality using different indicators: precision, accuracy, completeness, volume, timeliness and consistency. In this work, we are motivated by using other indicators to test the data quality, such as spread rate technique proposed in [2] which considers the global space of the data and does not look at each class alone.

The unavailability of sensor data over long periods of time, missing data as well as non-annotated events are also challenging ones. In this direction, we are interested to (i) take into account the missing data by adding a penalization function in the optimization algorithm when the data are not available, and (ii) to judge the performance of algorithms using evaluation metrics weighted by the number of representations in a dataset, F1 weighted score or balanced accuracy when the events are not annotated.

Finally, pattern recognition analysis and feature extraction challenges are solved by deep learning algorithms, such as RNN, CNN, LSTM, or hybrid architectures (RNN and LSTM for example). Both approaches based on CNN and LSTM would give equivalent performance levels, but CNNs are faster in the training phase and therefore more suitable for real-time HAR. However, challenges related to sequence analysis still remain largely unresolved. We argue that the application of language processing techniques can bring advances in solving some of these challenges as they deploy sequence analysis methods. In this context, our contribution to investigate byte pair encoding (BPE) methods [25] able to split words and represent sensor activation into sub-words and therefore capture more semantics in real-time.

#### **8 Conclusion**

This work provides an insight of new challenges for real-time HAR in smart buildings. Based on the existing methods and applications, different trends are analyzed and deals with pattern classification, data segmentation, data variability data sets pre-processing and complex human activities. Under these issues, various opportunities and solutions are analyzed along with its recent applicationbased approaches. Based on the observations, the findings of the survey are summarized to improve the field of real-time HAR. In this work point of view, it is recommended to improve deep learning techniques by developing neural network architectures that takes into account the quality of training data (missing values in time series and non-annotated event), the variability of data, the data segmentation and ontology of activities for better performance improvement.

Future work will be around the analysis of different software/hardware architectures for real-time HAR in smart buildings.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **E-health Solutions for COVID 19**

# **Design COVID-19 Ontology: A Healthcare and Safety Perspective**

Hamid Mcheick1(B) , Youmna Nasser2, Farah Al Wardani2, and Batoul Msheik2

<sup>1</sup> Department of Computer Science and Mathematics, University of Québec at Chicoutimi, Chicoutimi, QC G7H 2B1, Canada Hamid\_mcheick@uqac.ca <sup>2</sup> Department of Applied Mathematics and Ecole Doctorale, Lebanese University, Beirut, Lebanon {youmna.nasser,f.alwardani}@st.ul.edu.lb, batoul.msheik@hotmail.com

**Abstract.** The COVID-19 pandemic has flooded a vast amount of information into the world. To help control this situation, good utilization of the overflow in data is required. However, data come in different forms, posing numerous challenges in subsequent processing. Therefore, a uniform knowledge representation of COVID-19 information is needed, and ontology can play a role. The ontology will model patient healthcare-related data, ranging from symptoms to side effects and medical conditions, and the necessary precautions, especially for healthcare workers, to obtain protection from the COVID-19 virus. We followed Sánchez's methodology to build the vocabularies, which include current ontology concepts, W3C standards RDF, OWL and SWRL. This work shows promising results that can be applied by different organizations.

**Keywords:** Design ontology · COVID-19 · Healthcare · Knowledge uniform methodology · Knowledge representation collaboration

### **1 Introduction**

The World Health Organization (WHO) [1] defines COVID-19 as an infectious disease that emerged as a new strain of coronavirus. WHO [1] adds that approximately 405 million people were infected with this virus, killing almost six million from the beginning of the pandemic till February 2022. The scientific community is being tasked with controlling the spread of COVID-19, and this paper is a contributor. We aim to provide a knowledge representation ontology for the collection and analysis of data related to COVID-19 to track relevant data and eventually help control this pandemic.

To respond to this health emergency, information-sharing across different platforms and systems is demanded. However, data can be heterogeneous. Systems collect information based on discipline-specific terminologies, which can restrain them from sharing the information between platforms. Hence, ontologies can offer uniform ways of representing knowledge. Data publishers and collectors can use a shared vocabulary to collect COVID-19 data. An ontology, as presented by Tudorache et al. [2], defines a common jargon for sharing data amongst researchers and incorporates machine-interpretable definitions of essential concepts and relationships in this field. Knowledge represents the foundation of creating an artificial intelligent model. Dou et al. [29] state how research has shown that the frequent integration of semantics presents improved outcomes in data mining and deep learning.

The ontology presented in this paper provides a knowledge representation model of COVID-19 from a healthcare perspective, demonstrating every patient's case and recommending how people, especially healthcare workers, can protect themselves and follow appropriate safety and protection measures to decrease the risk of contracting the virus.

This article is organized as follows. Section 2 presents an overview of the existing COVID-19 ontologies. We classify our ontology in Sect. 3, and then we build a COVID-19 ontology and describe how it differs from other ontologies in Sect. 4. Section 5 specifies a few reasoning rules applied in this ontology. Finally, conclusions and future works are given in Sect. 6.

### **2 Literature Review**

This global pandemic has given rise to several COVID-19–related ontologies in order to cope and take control. Each ontology models this virus from a different perspective. This section discusses some of these ontologies, shedding light on how our ontology differs.

The COVID-19 Surveillance Ontology, presented by de Lusignan et al. [4], is intended to help surveillance in primary care. The fundamental objective of this ontology is to monitor COVID-19 cases and related respiratory conditions using information from various clinical record frameworks. It was built as a taxonomy with classes including exposure to COVID-19, knowledge related to COVID-19, definite and indefinite contraction of COVID-19. However, this ontology does not comprise any property, hence lessening its semantic expressivity.

Another ontology is CODO, presented by Dutta and DeBellis [5], which is the most similar to our work. This ontology was designed for cases and patient information representation to help in publishing COVID-19 data using Findability, Accessibility, Interoperability, and Reuse (FAIR) standards. It was built to facilitate the organization and illustration of daily-produced COVID-19 data, the relationships between the datasets, and the surrounding factors, for the further analysis of data. Our work differs in that it also covers data related to the safety and protection measures that should be applied.

Many ontologies cover the medical perspective of COVID-19. He et al. [3] presented the well-known CIDO, which is a disease ontology that presents the etiology, transmission, pathogenesis, diagnosis, prevention and treatment of this virus. The information includes the nature of the virus, means of transmission, common symptoms, and medical treatments. In contrast, our approach looks to this disease from a healthcare perspective.

### **3 Classification of Ontologies**

Ontologies can be classified based on content, goals, application technique and timing, domain representation, reusability, and field of application. Many ontology classification methods have been defined over the decades that vary in their perspectives. Ajami and Mcheick [7] expanded on OntoCube and added more performance standards, including machine-readability, i.e., whether the ontology could be easily understood and processed by the computer; reusability of concepts and classes to accomplish an objective; and complexity, i.e., measures of time and resources needed to achieve a certain task. Applying their classification technique, our ontology is considered formal as it employs the Web Ontology Language (OWL). Also, it is domain-specific since it describes terminology in healthcare, particularly COVID-19, allowing it to be semi-reusable. Finally, ours is a heavyweight ontology containing classes and relations, in addition to axioms and rules.

#### **4 Designing a COVID-19 Ontology**

Studer et al. [6] define an ontology as a "formal, explicit specification of a shared conceptualization" that is a clearly defined, simple, machine-readable interpretation of real-world concepts and their interrelationships, providing shared knowledge for the target community. In this paper, we adopt this definition of ontology.

This section provides a description of the design and development methodology of COVID-19 ontology. Well-known methods of designing an ontology include METHON-TOLOGY [8], TOVE [9], Cyc 101 [10] and YAMO [11]. We follow Sánchez's methodology [25] for building a medical ontology, also used by Ajami and Mcheick [7], since our ontology focuses on the medical field of the COVID-19 disease. This methodology combines both METHONTOLOGY [8] and Cyc 101 [10] and consists of five main steps that we follow to build our ontology, that are as follows: determine the domain scope, reuse the ontology, develop the conceptual model, implement the ontology, and evaluate it.

#### **4.1 Determine the Domain and Scope of the Ontology**

The domain and scope refer to the main field this ontology covers, putting boundaries around the conceptualization, and the ontology's purpose. According to Ajami and Mcheick [7], researchers pose a set of questions to determine the domain of the ontology, which we answer:

• *What is the domain that the ontology will cover?*

The COVID-19 is the main domain this ontology covers.

• *What is the purpose of this ontology?*

The purpose of this ontology is to facilitate the gathering and publication of COVID-19–related data as semantic services. Our ontology tracks COVID-19 patient's medical status and predicts their severity level for a better understanding of the nature of the virus and how patients could be treated. It also tries to reduce the risk of contracting COVID-19 by tracking the essential safety and control measures taken by people.

• *Who will use the ontology?*

This ontology can be used by organizations willing to collect COVID-19–related data to help control this pandemic and know more about this disease, such as hospitals, government agencies, health organizations and researchers.

	- 1) What are the most common symptoms of patients with COVID-19?
	- 2) How severe is a patient's case?
	- 3) Are the safety measures effective in protecting against contracting COVID-19?
	- 4) Is there a relationship between COVID-19 and a certain disease?

#### **4.2 Reuse the Ontology**

To the best of our knowledge, the ontological representation of COVID-19 patient clinical healthcare and safety measures is poor since many current COVID-19–related ontologies tackle the disease from a medical standpoint. However, we do integrate some medical terms that are used to represent health data. In our ontology, we integrate concepts from: Schema.org, Friend of a Friend (FOAF), and SNOMED CT.

#### **4.3 Develop a Conceptual Model**

The following is a set of guidelines for the development of a conceptual model, as suggested by [7].

• *Enumerate key terms in the ontology.*

The crucial terms that describe a context need to be defined. These terms include nouns that represent a specific concept (e.g., a patient is described by the noun "Patient"), attributes that describe the type and value of what is being modelled (e.g., the value of temperature is a float), verbs that describe the relationships between nouns or between nouns and attributes (e.g., a patient "is a" Person). Since standard terminology shall be used t model medical terms, we used SNOMED CT ontology in building our ontology to model the concepts of drugs and symptoms.

• *Define classes and class hierarchy.*

This step starts by defining the classes used in the ontology, then defining the taxonomy of these classes by matching subclasses to classes.

• *Define class properties.*

The two main class properties are object and datatype properties. These are used to model the relationships among different elements of the ontology, as classes alone do not provide enough information to represent the context behind this ontology. Object properties build relationships between classes by specifying the class domain of the relationship and its class range. The datatype property models the value and type of the concept, such as string, integer, and boolean.

• *Define the facet of slots.*

According to [7], a slot shall be assigned different kinds of facets that frame its value type, allowed values and cardinalities, to be added as required. They are mostly represented as string, integer, and float in our ontology.

• *Create instances.*

Individuals of a certain class are created by choosing a class, then filling the value slots. An example is creating Steve as an instance of Patient.

• *Develop our ontology domain.*

We model our COVID-19 medical ontology with respect to the information and guidelines provided byWHO. It contains information related to a patient with COVID-19, including symptoms and treatment, the patient's medical history and the safety measures to control this virus by healthy people, patients, or healthcare workers. Our ontology consists of four main sub-ontologies, as presented in Fig. 1.

In developing our ontology, we used OWL-DL, a descriptive logic ontology language. The Protégé ontology editor developed at Stanford University with a Pellet reasoner plugin was employed.

We divided our ontology into four sub-ontologies based on the recursive algorithm by Le Pham et al. [15], using Even's algorithm from Amir and McIlraith [16].

**Fig. 1.** COVID-19 sub-ontologies

In our ontology, "Patient" is the minimum vertex separator; hence, our ontology is divided based on this. The first subgraph includes patient personal information and physical factors, in addition to the precautions to follow, while the other subgraph accounts for the patient's clinical status in addition to the COVID-19 disease ontology that include the common symptoms and treatments of COVID-19 patients. Each sub-ontology can be further divided into two sub-ontologies following the same reverse algorithm suggested by Le Pham et al. [15]. The first subgraph can be divided through the "Person", which is the minimum vertex separator, thus separating our ontology into two, one dealing with the precautions a person should follow and the second one deals with the physical state of a person. The second ontology can also be divided into two sub-ontologies following the same methodology, with the "Patient" as a minimum vertex separator; one concerns the patient's clinical status and the other with the COVID-19 disease. Descriptions of these ontologies follow.

**Fig. 2.** Part of the patient ontology

**Patient Ontology.** As shown in Fig. 2, it includes a person's personal information and the relevant physical vital signs that are alerted when a person contracts COVID-19, as suggested by [1, 12, 13]. We use SNOMED CT to represent classes such as the Patient and some physical attributes.

**Fig. 3.** Part of the patient's clinical status ontology

**Patient's Clinical Status Ontology.** The clinical status ontology demonstrates a patient's medical history, past diseases, current diseases, hospitalizations, examinations, and current medications, as illustrated in Fig. 3. According to [14], people with some medical conditions may have worse cases of COVID-19 and require more care and attention.

**Disease Ontology.** The disease ontology mainly represents COVID-19, as shown in Fig. 4, modelling its variant type, symptoms, current stage, severity, potential risk factors, the location where the patient is monitored, and the patient's medication for COVID-19 treatment. SNOMED CT is used to model the symptoms and treatments of a patient with COVID-19.

**Fig. 4.** Part of the disease ontology

**Safety Measures Ontology.** The precautions ontology, shown in Fig. 5, models the protection mechanisms needed to follow to protect oneself against contracting COVID-19. These precautions are for healthy people, patients and healthcare workers, especially those working on the frontline fighting this pandemic. All these mechanisms are modelled in accordance with WHO's instructions [17, 18].

**Fig. 5.** Part of the safety measures ontology

#### **4.4 Implement the Ontology**

As mentioned before, we used the Protégé tool to build our ontology. We formalized our ontology in OWL-DL so that it can be highly expressive, and hence, it enables us to apply appropriate reasoning techniques. We chose the Pellet reasoner for our ontology as it ensures that an ontology does not contain any contradictory facts, checks if any instances of a class are possible, computes the subclass relations between every named class to create the complete class hierarchy and computes the direct types for each of the individuals, as stated by Sirin et al. [24].

#### **4.5 Evaluate the Ontology**

Several ontology evaluation measures have been proposed. Evaluation measures the quality of an ontology and if the constraints and requirements have been met. This section covers some of the proposed evaluation metrics and assesses our COVID-19 ontology accordingly.

Yu [19] suggested the following ontology evaluation criteria:


**Ontology-Level Evaluation.** Srinivasulu et al. [20] set four ontology-level metrics to measure the complication of an ontology's purpose. We compare our ontology to the metrics against the gold metric specified in Ajami and Mcheick [7].


**Class-Level Evaluation.** Brewster et al. [21] suggested class-level metrics to evaluate the complexity of the ontology, and we use four for our ontology:


In addition, a dataset from Carbon Health data lab [28] that included cases from people who contracted COVID-19 was used to validate our ontology. We were able to partially test our reasoning rules and have gotten promising results in terms of consistency and accuracy. For future work, we're working on fully validating our ontology with a complete dataset and with different scenarios.

Overall, the evaluations show that our ontology is adequate and can be reused or expanded.

### **5 COVID-19 Reasoning Rules**

We use the Semantic Web Rule Language (SWRL) to formulate those rules, combined with an OWL knowledge base, which, according to Horrocks et al. [22], extends the OWL abstract syntax to a high level.

### **5.1 Reasoning in Our Ontology**

**First Case.** Used to determine the severity level of a patient's condition and the place where they shall be monitored in. The National Institutes of Health (NIH) [23] specifies the severity level of a patient diagnosed with COVID-19 based on symptoms, physiological signs, and hospitalization requirements. [23] classifies the severity level of a patient into these four categories: asymptotic, mild, moderate and severe. An asymptotic person doesn't exhibit any symptoms though having a positive polymerase chain reaction (PCR) test. A patient with mild severity level is one that exhibits symptoms but doesn't have shortness of breath, dyspnea, or abnormal chest imaging. A patient with moderate severity level has an oxygen saturation (SpO2) percentage above or equal to 94%, while a severe case is when the patient's respiratory rate is above 30 breaths/min. Nonetheless, patients with asymptotic or mild severity level shall be monitored at home, while those with moderate and severe cases shall be hospitalized.

**Second Case.** The second case determines whether the safety measures imposed by the European Centre for Disease Control [26] are being applied, such as wearing a mask, sanitizing, and social distancing. These rules also aid in showing the effect of these measures on the health and lives of people not diagnosed with COVID-19 in terms of contracting the virus, especially healthcare workers who need to be more cautious when working with COVID-19 patients. Our ontology can be used in different frameworks, such as an Internet-of-Things (IoT) system that detects if people are actually following the COVID-19 safety precautions, and evaluating the effectiveness of such precautions.

#### **5.2 Defined Classes**

We built axioms in OWL that define the necessary information to declare an individual as a member of a class. Reasoners use these axioms and build the class hierarchy in accordance while adding further reasoning to these individuals. Most of our axioms were built inside the EquivalentTo field in Protégé. Hence, whenever an individual fulfills these conditions, the person is considered an instance of the equivalent class. An example in our case is a patient – being a patient is equivalent to being a person who has undergone a PCR test and received a positive result.

### **6 Conclusion**

All the data generated from the pandemic can be rendered useless if not organized and translated into meaningful information. An ontology is a key element to extract the concepts from emerging COVID-19 data to help control the pandemic. We designed this ontology to model concepts related to patient healthcare data and help with the collection and analysis of the symptoms and effects on patients with COVID-19, in addition to reducing the risk of contracting the virus by setting well-defined precautionary measures. The built ontology was compared against evaluation metrics and shown to be of good quality and ready to be used or expanded. With the onset of vaccinations and new vaccination data emerging daily, future work can include extending our ontology with a detailed vaccine sub-ontology to cover the assessment and side effects of each vaccine. Nonetheless, we are in the works of validating our ontology with a complete dataset that tests all the reasoning rules in to fully prove the ontology's effectiveness in the healthcare and safety domain.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Social Response to COVID-19 SMART Dashboard: Proposal for Case Study**

Karenina Zaballa1,2(B) , Gabriela Fernandez1,2, Carol Maione1,2,3, Norbert Bonnici1,4, Jarai Carter1,5,6, Domenico Vito1, and Ming-Hsiang Tsou1,2

<sup>1</sup> Metabolism of Cities Living Lab, Center for Human Dynamics in the Mobile Age, San Diego State University, 5500 Campanile Drive, San Diego, CA 92182-4493, USA nikazaballa@gmail.com, gfernandez2@sdsu.edu

<sup>2</sup> Department of Geography, San Diego State University, 5500 Campanile Drive, San Diego, CA 92182-4493, USA

<sup>3</sup> Department of Management, Economics and Industrial Engineering, Politecnico di Milano, via Lambruschini 4b, 20156 Milan, Italy

<sup>4</sup> Institute of Space Sciences and Astronomy (ISSA), University of Malta, Msida, Malta

<sup>5</sup> Smart Lab, Procter and Gamble, Champaign, IL, USA

<sup>6</sup> Columbia University, New York, NY, USA

**Abstract.** The COVID-19 pandemic took a toll on the world's healthcare infrastructure as well as its social, economic, and psychological well-being. In particular, Italy's unexpectedly high COVID-19 case and death rate from March to June, 2020, captured headlines due to its speed and virulence. Many governments are currently implementing measures to help contain and slow down the spread of COVID-19. The Social Response to Covid-19 Smart Dashboard was built by researchers at the Metabolism of Cities Living Lab, Center for Human Dynamics in the Mobile Age at San Diego State University and Politecnico di Milano. This dashboard provides an aggregated view of what people in 10 Italian metropolitan cities (Milan, Venice, Turin, Bologna, Florence, Rome, Naples, Bari, Palermo, and Cagliari) tweet during the pandemic by monitoring social media behaviors in the north, center, south, and islands. Moreover, the dashboard is a geo-targeted search tool for Twitter messages to monitor the diffusion of information and social behavior changes which provides an automatic procedure to help researchers to: associate tweets based on geography differences, filter noises such as removing redundant retweets and using machine learning methods to improve precisions, analyze social media data from a spatiotemporal perspective, and visualize social media data in various aspects such as weekly trends, top urls, top retweets, top mentions, and top hashtags. The Social Response to Covid-19 SMART Dashboard provides a useful tool for policy makers, city planners, research organizations, and health officials to monitor real-time societal perceptions using social media.

**Keywords:** Smart dashboard · COVID-19 · Social media · Italy · Monitoring and tracking · Twitter

### **1 Introduction**

Social media is more ubiquitous than ever, enabling it to be a good tool to keep connected during the pandemic. Using automatic data processing for Twitter messages, the Social Response to COVID-19 SMART (Social Media Analytic and Research Testbed) Dashboard helps researchers search Tweets in different cities, filter noise (such as removing redundant retweets and using machine learning methods to improve precision), analyze social media data from a spatiotemporal perspective, and visualize social media data in various aspects (such as weekly and monthly trends, top URLs, top retweets, top mentions or top hashtags). The Social Response to COVID-19 SMART Dashboard uses multiple data mining programs, GIS methods, and advanced geo-targeted social media API's to track selected topics in space and over time. There are multiple components to searching, processing, and visualizing social media messages from the Twitter Standard Search application programming interfaces or API's. The filtered statistics of the focus topics and geo-targeted cities are visually represented in the SMART Dashboard.

The daily and almost live monitoring capability of the Dashboard has great potential for local, state, public health agencies, and practitioners to integrate real-time information to investigate large-scale disease outbreaks. For example, the Social Response to Covid-19 SMART Dashboard can be used to study sentiments on COVID-19 and vaccines in Italian cities based on new policy mandates and curfews. Because of the Dashboard's unique capability to capture the temporal and spatial nature of COVID-related policies, behaviors, beliefs, and sentiments through Twitter content revealing various trends in diverse geographic areas, community leaders can use this tool to closely connect to their constituents and mitigate social issues before they become full-blown movements. Another potential use is to monitor public opinion towards crisis events such as the SARS COVID-19 outbreak. The Dashboard visualizes the most popular media shared in Twitter based on the COVID-19 pandemic in real-time.

### **2 Literature Review**

To provide more background on the Dashboard, the following areas will be discussed in detail: the impact of COVID-19 on the 10 metropolitan cities in Italy to understand the geographical and temporal constraints, social media analytics to delve into their use cases, and SMART Dashboard 2.0 to delve into the history of the dashboard.

#### **2.1 Impact of COVID-19 on the 10 Italian Cities**

The COVID-19 pandemic has turned the once tourist-filled cities of Italy to ghost towns due to quarantine measures. The SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2) is caused by a coronavirus, and it presents itself with symptoms that include "fever or chills, cough, shortness of breath, difficulty in breathing, fatigue or tiredness, muscle or body aches, headaches, new loss of taste or smell, sore throat, congestion or runny notes, nausea or vomiting, and diarrhea" [12]. During the beginning of March 2021, Italy came into the forefront of world health news due to its rapidly rising COVID-19 cases and deaths as well as for being the first country outside of Asia to have such high cases and deaths.

To provide more background, Italy's first confirmed COVID-19 case was reported in the Province of Lodi, Lombardy region on February 20th, 2020 [11]. The next day, Italy and all of Europe the first COVID-19-related death was announced in the province of Padova and Veneto region [1]. Due to the increasingly older residents who have a larger likelihood of comorbidities in Italy, the majority are at risk for the disease [2].

Other Twitter dashboard studies have focused on identifying real-time Twitter trend analysis using big data analytics and machine learning techniques [3]. For instance, Garg and Kaur [4] have explained the analysis of Twitter data using components of Cloudera distribution of Hadoop. In fact, the study's objective assigned polarity to each tweet. Map reduce and Apache SPARK frameworks were used for sentiment analysis. The result showed that Apache SPARK is better than MapReduce. Saad and Yang [5] have performed sentiment analysis of Twitter data using ordinal regression. While, Ahmed and Rodriguez-Diaz [6] have performed sentiment analysis on online customer reviews as a form of visualization. Finally, Rathod and Barot [7] researched the same field to predic public opinion on ongoing events by analyzing tweet sentiments using machine learning classifiers like SVM, Naive Bayes, logistic classifier, and KNN classifier. SVM was found to be the best classifier with the least mean square error for the classifications. Garg et al. [8] have identified the trending pattern in Twitter using SPARK. These patterns were obtained by collecting tweets on a real-time basis and identifying trending hashtags at the same time. It was implemented using a big data technology SPARK streaming. This type of technique can help governments or companies know about more about the behaviors/trends of their given campaign/program and/or brand/product awareness and customer needs.

The time frame chosen to provide a proof of concept for the Dashboard is from March 3rd to June 25th, 2020. This period is divided into Phases 0, 1, and 2. Phase 0 started when the first case was reported until before the lockdown. Phase I, or the lockdown phase, started on March 11th, 2020 and ended on May 4th, 2020 [9]. Phase 2 lasted from May 4th, 2020 until June 3rd, 2020 [2].

Phase I is marked by increased restrictions in Milan in response to the pandemic. Specifically, educational institutions, religious events, cultural centers, and all events and places that required gathering were prohibited [9]. This included professional sporting events. Visits to family and relatives were prohibited as well as patronizing bars and restaurants. Dining institutions were allowed takeout with limited hours. Face masks were required in all public spaces indoors and outdoors. In addition, there was a selfcertification form that the government required the residents to fill out and keep on their person whenever they left their homes that enabled contact tracing measures [9]. The lockdown was exacerbated when military force was ordered to keep lockdown measures in place. Due to travel restrictions, no airports were open for use. The only travel of any kind allowed was to the grocery store, pharmacy, or the hospital. Next, Phase II marked the easing of restrictions in Phase I. Businesses opened without limits to their hours of operation [9]. Some airports opened enabling reduced international travel. Public parks also opened as well as public transportation with reduced capacity [9].

#### **2.2 Metropolitan Italian Cities**

For this study, Italian major metropolitan cities were explored to understand the interconnections between geographical location, number of COVID-19 cases, social response to the pandemic and locally-enforced measures based on Twitter data. Table 1 shows the 10 cities that were selected across Italy. These cities were chosen by the Crowdfight International Team, a multidisciplinary research group, based on economic and cultural factors. Since the outbreak started in the North, the team decided to start there while other cities were added over time. Milan and Venice were chosen to represent the Northwestern region. Turin and Bologna were chosen for the Northeastern region. Florence and Rome represented the Central region. Naples and Bari claimed the Southern region. Palermo and Cagliari represented the Islands.


**Table 1.** Socio-demographics for the 10 Italian metropolitan cities [13]

#### **2.3 SMART Dashboard**

The idea of the "prototype created by the Center for Human Dynamics in the Mobile Age at SDSU was to facilitate the rapid dissemination of official alerts and warnings notifications from OES during disaster events via multiple social media channels to targeted demographics" [15]. The platform can identify and recruit top 1000 social media volunteers based on their social network influence factors and can aid government agencies to communicate more effectively to the public [14].

In our study, this same Dashboard was refitted to 10 metropolitan cities in Italy. More specifically, the north, center, south and island cities of Italy [14]. The backend was improved and mounted on larger servers.

### **3 Methodology**

#### **3.1 Data Collection**

To provide the analysis, the team began by collecting Twitter data through the Twitter Standard Search API. This involved making a Twitter Developer account, requesting access tokens and keys followed by authentication of said keys. The API allows for collecting specific metadata, so the researchers had freedom to choose which ones to use for the study. In addition, Table 2 shows the keywords that were used to harvest the Tweets. These were chosen by the Crowdfight International Team in partnership with the Metabolism of Cities Living Lab under the Center for Human Dynamics in the Mobile Age (HDMA), after discussions with Italian colleagues as well as medical professionals. Keywords were selected based on popularity based on hashtag and word of mouth.


**Table 2.** Social response to Covid-19 smart dashboard selected keywords


**Table 2.** (*continued*)

#### **3.2 Data Collection**

In order to understand how the data is analyzed it is important to understand the client and server framework in Fig. 1 below.

**Fig. 1.** Data framework [4]

The server side for the Dashboard is explained below. For the database, the social media data tends to be more unstructured, so a NOSQL database, specifically MongoDB was used [10]. The Twitter Search Engine, coded in Python, was used to specify keywords, time period, and automate collection [10]. The web server used is written in NodeJS so that there would not be a need to switch to other server-side languages to implement the server [10]. This was specifically written so that JavaScript and node modules can be utilized to expand the functionality. Having NodeJS for the server also enabled for easier REST API creation, since the API is also built with NodeJS [10]. The client side of the framework is built upon HTML5 (HyperText Markup Language 5), JavaScript (JS), and CSS3 (Cascading Style Sheets, Version 3) as the base. On top of which are various JavaScript libraries to be discussed in Table 3.


**Table 3.** JavaScript libraries in the client side [10]

#### **3.3 Dashboard Features**

Due to the flexibility of the original SMART Dashboard 2.0, the Social Response to COVID-19 Dashboard was created by first changing the geo-tagged tweets during data collection then changing the keywords and filtering out specific links that may be deemed inappropriate or unrelated to the cause on the SMART Dashboard. Each section of the COVID Dashboard is discussed below.

The first few components that the user sees is the screen in Fig. 2 below, containing the Dashboard Toolbar on the far left, the SMART index at the top, and the Trend and Top Media sections below the SMART index. It also houses the "Stop Auto Refresh" button in order to enable researchers to stop the feed and conduct analyses.

Dashboard was created by first changing the geo-tagged tweets during data collection then changing the keywords and filtering out specific links that may be deemed inappropriate or unrelated to the cause on the SMART Dashboard. Each section of the

**Fig. 2.** Initial interface of social response to Covid-19 smart dashboard

COVID Dashboard is discussed below. The SMART Dashboard 2.0 Toolbar, on the far left, contains the shortcuts of each component on the Dashboard. It also houses the keywords used to extract the Tweets. In addition, it contains the "Download" button to gain access from the data in the dashboard, the Privacy Policy, and Feedback buttons. The "Home" button enables the selection of keywords and filtration of certain Tweets that may be inappropriate or that adds noise to the findings. The toolbar also enables the selection of keywords simply by checking and unchecking the keywords desired.

The SMART Index, which consists of the four multi-colored blocks across the top, shows the most current metrics from the last 10 min it refreshed. The blocks will be discussed from left to right. The first block (blue) from top to bottom shows the number of Tweets harvested within the past hour, the date they were extracted, and the distribution of the time that each tweet was extracted. The second block (green) shows the number of Tweets extracted in the past 24 h, current date, and distribution of the Tweets over time. The third block (yellow) contains the number of Tweets since the day before the current date. It also contains the distribution of the number of Tweets from the day before and the current date. The fourth block (pink/salmon) shows the number of Tweets since the beginning of collection and the distribution of Tweets from the beginning of the Tweet harvest until the current date The Trend Section shows the frequency of Tweets generated by the keywords over time through a series of line graphs. Users can hover over any section of the graph and it will show the Tweets, both filtered and unfiltered, in the time frame. Any point in the line can be clicked to show the Tweets at the selected timeframe within the point selected. In addition, the tabs on the top can change said time frame. In Fig. 2, the graph shows how users can visualize Twitter metrics from the past 10 min, hour, daily, weekly, and monthly, therefore shrinking the graph towards the left. The bottom sliding scale can also change the distribution of the timeline of the graph.

The Top Media Section on the lower left of Fig. 2, shows the most shared images posted within the timeframe. The user has options to change the time frame, whether to show all media from the beginning of extraction, a week of current date, a day from current date, and from the current date.

The Top URL Section shows the most posted links or web pages within the timeframe. The user has options to change the time frame, whether to show all URL's from the beginning of extraction, a week of current date, a day from current date, and from the current date. Figure 4 shows it all. A unique feature in the Social Response to COVID-19 Dashboard is the Word Cloud Section in Fig. 3. It includes a word cloud and most frequent vocabulary words table within the selected time period. The word cloud function contains the most frequent vocabulary words within the corpus at any chosen time period. The size of the words indicate a higher frequency, while words with smaller fonts are less frequent. Word clouds are an intuitive, decorative, and convenient way to see most common keywords in a corpus. Future developments for certain word clouds can include using stopwords, or words that are used so commonly that they provide little to no value to the visualization. For example, in English, this could include articles and prepositions, like "the" and "into." This would naturally mean selecting a particular language, which when harvesting geo-tagged Tweets, do not guarantee one specific language.

**Fig. 3.** Word cloud

In addition, the Vocabulary Frequency Table shows the most frequent words in the corpus in the selected time frame. The information is presented in bar chart form arranged from most frequent to least frequent. Another unique feature of the Dashboard is the Tweets in Cities section shows the normalized tweeting rates by city population within the certain time period selected. Basemaps can be changed to the user's preference. In our example in Fig. 4, the map is based in Milan, Italy and the selected time period is all Tweets since the beginning of the extraction. Other options include a week from current date, a day from current date, and the current date. What may also be notable for researchers is the geographic visualization of where the tweets were collected as well as the collection radius and other useful statistics like the total number of Tweets collected in the selected time period and the latest population information that the API can find.

The most common Retweets from the selected time period are displayed in the Top Retweets section. Like the other sections, users can select which time period they want: all Tweets since the beginning of extraction, a week from the current date, the day before the current date, and the current date. Each Retweet has its frequency next to it. Retweets are important because they are quantifiable measures of influence. They also heavily affect a corpus if the study does not require original Tweets.The Top Mentions section shows the most frequent user references (beginning with '@') in the selected time period. This section is notable because mentions are quantifiable measures of reference. It shows the frequency of interaction between the Twitter users within the collected corpus. Their corresponding frequency is displayed next to each user that was mentioned. Users can

**Fig. 4.** Tweets in cities

select which time period they want: all Tweets since the beginning of extraction, a week from the current date, the day before the current date, and the current date. The Top Hashtags, which refer to an idea or theme of a tweet, are shown below the Top Mentions section. Users create this hashtag to refer to certain movements, using the pound sign ('#'). Like mentions, these are also quantifiable measures of reference and levels of interaction between users and hashtags. The corresponding frequency is displayed next to each frequent hashtag in the time period. Users can select which time period they want: all Tweets since the beginning of extraction, a week from the current date, the day before the current date, and the current date. The last shows the Geocode Status of the Tweets collected in the selected time period. This is meaningful for the researchers because it gives context to the successfully geocoded tweets in the corpus. It can give insight into error rates, so future experiment parameters can be adjusted accordingly. Corresponding counts and percentages are displayed next to each status (Fig. 5).


**Fig. 5.** Top hashtags

### **4 Discussion and Future Work**

This type of dashboard is successful in filtering certain websites and content and the unique combination of visualizations increases the potential of the tool to be used in many different settings. For the purposes of social response to COVID-19, it allows policymakers to understand the current behaviors of society and can be used to observe public opinion during and after crisis events or disease outbreaks. The SMART Dashboard is available for use to assist response and assistance efforts during the pandemic. Real-time public health information and major events captured using social media are now at the forefront of behavioral measurement, disease surveillance, health promotion, and more. Different cities and regions may reveal different patterns of social media messages and trends. By analyzing the context of social media messages, linking place and time together we can discover more meaningful patterns and insights depending on the goals of the study of disease outbreaks and social media activities. Having expounded on the Dashboard's capabilities, it is useful to note that the limitations of this study are dependent on the Twitter Standard Search API, the capacity of the server to store data, the extraction parameters in data collection, and the specific keywords used in the study. With the constant sharing of ideas online, it is impossible to capture the totality of themes online. In addition, certain natural language processing techniques for the word cloud can be improved by implementing specific stopwords in order to see specific keywords rather than articles and prepositions. Work can be done to make the techniques agnostic to language including stopword adjustments.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Adopting the Internet of Things Technology to Remotely Monitor COVID-19 Patients**

Abdessamad Saidi(B) , Mohamed Hadj Kacem, Imen Tounsi, and Ahmed Hadj Kacem

ReDCAD Laboratory, ENIS, University of Sfax, Sfax, Tunisia abdessamadsaidi2@gmail.com https://www.redcad.tn

**Abstract.** The coronavirus known as COVID-19 is the topic of the hour all over the world. This virus has invaded the world with its invariants, which are characterized by their rapid spread. COVID-19 has impacted the health of people and the economy of countries. For that, laboratories, researchers, and doctors are in a race against time to find a cure for this pandemic. To combat this virus, cutting-edge technologies such as artificial intelligence, cloud computing, and big data have been put in place. In our work, we use Internet of Things (IoT) technology. The use of IoT in an efficient way can lead to detecting infected people and avoiding being contaminated. In this paper, we are interested in the remote medical monitoring of patients who have tested positive for COVID-19. We propose a meta-modeling technique to model the IoT architecture. Then we implement two IoT solutions that permit the remote medical monitoring of patients infected with COVID-19 and the respect of social distancing by instantiating correct models that conform to the proposed meta-model in order to mitigate the COVID-19 outbreak.

**Keywords:** Internet of Things *·* Meta-model *·* Covid-19 *·* Cloud *·* Android *·* Sensor *·* Actuator *·* Iot application

### **1 Introduction**

Since 2019, the world has been living with the pandemic of coronavirus known as COVID-19, and it has impacted many people around the world. COVID-19 has a lot of variants, such as the Delta and Omicron variants, which are characterized by a high propagation speed in a short period of time. Generally, the virus causes respiratory sickness because it attacks the lungs. COVID-19 has three categories of symptoms: most common symptoms such as fever, less common symptoms like loss of taste or smell, and severe symptoms such as difficulty breathing. After much research, some institutes are able to produce vaccines against COVID-19. They are approved by the World Health Organization (WHO). Globally, as of February 18th, 2022, there have been 418,50,474 confirmed cases of COVID-19, including 5,856,224 deaths reported to WHO. As of February 15th, 2022, a total of 10,279,668,555 vaccines have been administered [25]. Despite the vaccines discovered, the battle is not over. Not only because the demand for the purchase of the vaccine is greater than the production and poor countries do not have the financial means to buy it, but also because of the mutations of the virus. For that, we need to reinforce protection by using emerging technologies. The Internet of Things can play an important role in the fight against COVID-19 and can be considered as an enabler by providing smart and innovative solutions. Collecting the data, transmitting data, reacting after the processing of data, and visualizing the data are exactly what the IoT provides to us and what is needed to deal with the virus. IoT, can be applied in different domains like transportation, agriculture, and, in our case, healthcare. It helps to prevent and detect diseases and react at the right time. In the case of COVID-19, we can use an IoT-based wearable body sensor to monitor patients remotely. But, creating applications in this domain is too hard due to the complexity of IoT systems. In addition to the storage constraint, we have a huge number of connected devices, communication, and processing infrastructures. The design of architectural software systems enables architects to grasp the construction of complex software systems. However, their colloquial description may cause confusion, resulting in erroneous software system implementation. In this paper, to address this issue, we propose a meta-model and implement two scenarios using an IoT system to fight COVID-19 using some sensors and actuators in order to test the proposed meta-model. The first one is to monitor contaminated people and alert the concerned authorities. And the second one is to apply social distancing when people wait their turn in supermarkets for payment, in administration, etc.

The remainder of the paper is organized as follows. Section 2 outlines related work. Section 3 presents the proposed meta-model. Section 4 introduces our use cases and their modeling. The necessary equipment, the implementation details, and results are discussed in Sect. 5. The conclusion is reported in Sect. 6.

#### **2 Literature Review**

The Internet of Things refers to the process of connecting the physical world to the Internet, including things such as light bulbs, medical devices, and traffic lights via a network. Process the data collected by sensors in the cloud. The result of the processing phase is actions in order to react and provide the user with useful information through applications. For this, several business sectors can benefit from IoT such as home automation, health, transport, industry and agriculture. Some researchers propose methods and techniques to assist developers and architects from the modeling phase to the final IoT product. Previous works are classified into two categories: those with a high level of abstraction and others with a low level of abstraction. In the second category, we notice different application domains. In the first category, in [5], authors propose a new domainspecific language called BIoT. This makes the creation of software architectures for IoT applications easier for software developers and permits them to avoid errors in the design step. Kallel et al. [9] use a business process model (BPM) and integrate the IoT-fog architecture with the cloud. Heterogeneous and nonheterogeneous IoT resources, resource metrics, and QoS constraints have all been gained by the architecture. Authors in [21] propose a profile called UML4IoT that enables us to exploit IoT in manufacturing systems. It enables the designer to create a cyber-physical component utilizing software and system standards. For the modeling and specification of design patterns, authors in [22,23] propose an approach to designing the SOA design patterns using the SoaML standard. Next, they attribute to them a formal semantic using the Event-B method. In past work [19], we applied our methodology of modeling IoT systems to a smart home system. In the other category, we find several application domains. In the agriculture domain, authors present an IoT architecture for smart farming [6] based on wireless sensor networks and a plug-and-play approach for standalone nodes. The used algorithm increases the network's lifetime. In smart home field, paper [12] presents a smart home system that improves the quality of life of a home owner using IoT techniques. The authors implement a server to interconnect things in the home and a web application to control these things in real time. In the healthcare sector, the study [20] designed a system that detects the quality of sleep in a patient by collecting some data with sensors. These are transmitted to a local server in order to use the random forest classification method for predictions and classification. Authors in [13], add a new technology to the IoT, which is Deep Learning. An optimized neural network with an accuracy of 97% can be used in an IoT system to detect fall. Authors in [24], propose an approach that combines IoT, fog, and cloud computing technologies in a healthcare monitoring system for healthy aging in a familiar setting to improve quality of life. In [15], authors predict and analyze indoor air quality, which is important in the case of infected people in quarantine using IoT technology and a machine learning model with high accuracy. Baskaran et al. [3], developed a system that permits the detection of COVID-19 infection in a work environment by detecting if a person is wearing a mask and thermal image. In [8], the authors propose a healthcare system based on the IoT using Neural Networks. The system processes data and makes decisions using the fuzzy logic system. One of the problems caused by COVID-19 is breathing. The lack of oxygen and oxygen concentrators in developed areas prompted the authors in [14] to propose a low-cost device that is easy to build. In the public safety and environmental monitoring domain, papers [1,2] analyze the air quality in open areas using air quality sensors and air pollutant data. This analytic permit to classify regions if they are good to live in or unhealthy. In [26], the authors propose a framework that integrate a Fog technology. The framework minimized the delay to 8% and had an accuracy rate of 95%. In the transportation field, a previous study [17] proposed a system that would permit monitoring of the platform automatically with the arriving trains using IoT sensors and actuators. Finally, any IoT system needs to be secure (unauthorized persons). For that, authors in [10] propose an intrusion detection system for the smart IoT environment. A system was developed to remotely manage IoT devices using 3G connectivity technology in [16]. We have a vertical layer that across the horizontal layers of IoT which is security layer. For that, a profile called IoTsec is discussed in [18]. This UML/SysML extension claims to describe IoT security knowledge, and it can be considered as a first step to building a robust modelling language for IoT systems in terms of


**Table 1.** Comparison between previous work

**Fig. 1.** The proposed meta-model

security and hence user safety. In [7] authors give a survey on how to deal with privacy and security while using IoT applications.

By comparing previous works, we find some gabs in the conceptual part as well as the implementation part. However, most of the previous studies do not take into account how the components of an IoT system are interconnected in their models, as well as the nature of the data exchanged in the IoT network. Furthermore, some meta-models or profiles are specific to a particular domain application. For the proposed IoT applications, the major drawback of some applications is that they do not respect the general architecture of the IoT. In addition, they don't provide the system modeling, which is an important step before the implementation phase. Our work belongs to the two categories mentioned before; we describe our system with a meta-model at a high level of abstraction and we implement it.

#### **3 The Proposed Meta-model**

We propose a meta-model to describe the Internet of Things architecture as shown in Fig. 1. The proposed meta-model takes into consideration all the components that we can find in an IoT architecture, the connections between these components, and the type of the exchanged messages. The following are the fundamental components of the meta-model:


We have added design patterns to ensure some quality attributes. These design patterns are represented as an interface to be implemented. For example, certain devices can't connect directly to a network because they don't support the appropriate communication methods. A DEVICE GATEWAY design pattern proposes a layer in the IoT gateway that translates the communication technology to the right one. Connectivity, cost, and reusability are quality attributes guaranteed with this pattern.

### **4 Case Study**

In this section, we present our use cases. Figure 2 shows the general structure of our IoT system. We are dealing with two scenarios. The first one is to capture the corporal temperature (the most common symptom of COVID-19) of contaminated people through a sensor. Also, their heart beat and oxygen saturation. Next, the doctor can visualize the value of the collected data via a website or Android application. If the temperature is high or if there is a breathing problem, the doctor will react immediately by generating an alert in order to avoid any further complications. The second scenario is to apply social distancing in shopping queues or offices in order to avoid the spread of COVID-19 by using sensors and actuators.

**Fig. 2.** General architecture of the IoT system

#### **4.1 Modeling the System**

Based on the meta-model and the use of the diagrams of the Unified Modeling Language (UML) version 2.5, it is possible to model the structural and behavioral views of systems. We use the component diagram from UML to model the structural features of our system. It is represented in Fig. 3.


To model the behavioral features of our application, we use the sequence diagram of UML. Figure 4 shows the sequence diagram of the first scenario, in which sensors send the collected data to the IoT Cloud Platform via the home router. After that, the doctor subscribes to the Firebase Cloud Platform in order to visualize the data of patients. Firebase offers a real-time, reliable, and extensive database, provides APIs to developers (ease of integration), and allows the management of users who have access to the collected data and its community. Figure 5 represents the sequence diagram of the social distancing scenario. The ultrasonic sensor measures the distance between two people and sends the value to Firebase. If the distance is less than 1.5 m, Firebase sends action (turn ON) to the buzzer. This last will make a sound, which means that social distancing is not respected. Through the Android application, we can see the status of buzzers (ON or OFF) and even whether social distancing is respected or not.

#### **5 Implementation and Results**

In this section, we present the materials used (software and hardware) in the development. A typical IoT architecture is composed of four layers:

1. **Physical layer:** is composed of sensors and actuators.

**Fig. 3.** Structure of the IoT system


#### **5.1 Hardware**

In order to develop our system, we need some materials, such as sensors and actuators. In our case, we need micro-controllers, three sensors, and an actuator. The materials that we found in the perception layer are listed below:

	- 1. **MLX90614 Sensor:** is a non-contact temperature sensor with a high accuracy, it's used for human body temperature measurement.
	- 2. **MAX30100 Sensor:** used to measure heart rate using the photoelectric method and oxygen saturation.
	- 3. **Ultrasonic Sensor:** it measures the sensor's distance from the obstacle.

#### **5.2 Software**

We have the Arduino IDE, which is used to program the ESP8266. The card supports many programming languages. Among them, we chose embedded C,

**Fig. 4.** The Sequence diagram of the remote healthcare monitoring

which is an optimised language dedicated to embedded systems. We need to install a module in order to detect the ESP8266 through the board manager. As an IoT platform, we chose Firebase because it is secure with free multi-platform authentication, scalable API's, and contains a real-time database. To visualize the data, we developed an Android application using Android Studio and the Java programming language.

#### **5.3 A Part of Code Sources**

For the embedded part, we first need to install some libraries.


#### **Listing 1.1.** Import libraries and declarations

```
1 #include ''FirebaseESP8266.h"
```
2 #include *<*ESP8266WiFi.h*>*

```
3 #include <Adafruit MLX90614.h>
```

**Fig. 5.** The Sequence diagram of the social distancing application

After that, we need to share the link of the real time database as well as the key with certain devices (for security) in order to be able to read value from the database and write collected data into it. Both the database and the key are needed in the physical layer (ESP 8266) and the visualization layer (Android application). Finally, we used WiFi as a communication technology because it is supported by most devices. The Listing 1.1 shows the declaration of two constants in lines 6 and 7, which refer to the SSID and password of the IoT Gateway. Listing 1.2 presents two predefined methods that permit to capturing ambient temperature and the temperature of objects without contact. We calibrate the object temperature value by comparing the obtained values with those of a real thermometer and the sensor.

**Listing 1.2.** Functions to manipulate the MLX sensor

1 void loop()*{* 2 temp amb = mlx.readAmbientTempC(); 3 temp obj = mlx.readObjectTempC() + calibration; *}*

The first line in Listing 1.3, is an instruction that permits storing the temperature value in the real-time database. The second line is used to read data from the real time database of Firebase.

**Listing 1.3.** Methods to read and write data from Firebase

1 void loop()*{* 2 Firebase.setFloat(firebaseData, ''Patient 1/Temperature", T); 3 T1 = Firebase.getFloat(fbdo, ''Patient 1/Temperature"); *}*

For the second scenario of social distancing, while waiting turn in line, people should wear a device like a medal that contains an ultrasonic sensor and a buzzer. First we declare the type of pin if it is used as an input (to collect data) or output (to react) as mentioned in the three first lines of Listing 1.4. Next, we calculate the distance between the two people (line 8 of Listing 1.4). After that, if the social distancing is not respected, the buzzer will turn ON to alert the person and update the state of social distancing in the application to **NOT RESPECTED**. Otherwise, the buzzer will stay off and the status of social distancing will be **RESPECTED**.

1 void setup() *{* 2 pinMode(trigPin,OUTPUT); 3 pinMode(echoPin,INPUT); 4 pinMode (Buzzer, OUTPUT) *}* 5 void loop() *{* 6 digitalWrite(trigPin,HIGH); 7 delayMicroseconds(10); 8 digitalWrite(trigPin,LOW); 9 duration = pulseIn(echoPin, HIGH); 10 distance = duration∗0.034/2; 11 if (distance*>*D min) *{* 12 digitalWrite (Buzzer, LOW); 13 rtdb.setString(''Ultrasonic/SD status", ''Respected");*}* 14 else *{*digitalWrite (Buzzer, HIGH); 15 rtdb.setString(''Ultrasonic/SD status", ''Not Respected");*} }*

#### **5.4 Results and Discussion**

Our system is operational after interconnecting all hardware and deploying software. Figure 6a represents our real-time database. It is composed of two parts. The first part represents the real collected data of COVID-19 patients. Patient 01 body temperature reached 35.1◦, with a heart rate of 75 per minute and 98% blood oxygen saturation. The second part contains all the users who have access to the database. A user is identified by his email address, full name, and phone number. For security reasons, the Firebase IoT Cloud platform offers the ability to manage users. As an example, we can delete a user who doesn't have the permission to consult the data of a patient. And the most important part of our application is that the doctor can visualize the collected data via the Android application, as shown in Fig. 6b. Table 2 shows the status of a patient depending on his measured oxygen saturation. Table 3 summarizes the variation of heart rate by age. By comparing the collected data with the reference values in Table 2 and Table 3, the doctor can generate an alert such as the need for oxygen. Figure 7 illustrates real-time sensor data monitoring for a smart system

#### 176 A. Saidi et al.

**Fig. 6.** System implementation results

**Fig. 7.** IoT monitoring every hour (a): Patient 1 (b): Patient 2 (c): Patient 3

that remotely monitors COVID-19 patients. Curves a, b, and c in Fig. 7 show the history of real-time measurements of oxygen saturation and body temperature of three COVID-19 patients with different ages, every hour from 8:00 a.m. to 21:00 p.m.. In every plot, the X-axis refers to the time and the Y1-axis, Y2-axis to the corresponding temperature and oxygen saturation respectively of the patient. We choose three patients to validate our system by comparing our collected data with the data of a commercial device, we find an acceptable divergence of *±* 0.3 ◦C for temperature and *±* 1 for *SpO*2. With this accuracy, our medical kit is incurring a total cost of 13 c, whereas, on the other hand, a commercial device is incurring costs of 25 c. Our application respects the general architecture and the three dimensions of the IoT:





80–99 73 *±* 10

**Table 3.** Heart rate reference value

The originality of our solution lies in the fact that it resolves some problems that we found in previous studies. For example, the authors in [3] propose an IoT solution to COVID-19, which is not practical in all situations because they developed an application that works only in India. The main disadvantage of the system developed by [11], is that authors use a local server instead of the cloud, which is not recommended in an IoT system (storage and processing constraints). In a recent paper by [4], authors merge the application layer with the processing layer by exploiting the dashboard (data visualization) offered by the Thing Speak Cloud platform. The first problem is that the system does not respect the general architecture of the IoT. The second one is that all users need to have access to the cloud platform in order to visualize the collected data. For that, users will have full control over data (see all data even if they don't have permission, alter data, etc.), and this is not good for the privacy of patient data. Also, we can't manage users. However, most of the previous studies do not take into account the modeling part of the proposed system.

A part of security is guarantees using the blacklist and whitelist patterns. This will lead to establishing a trusted communication partner. Also, we share the key and the link of the database with trusted devices. We have complete control over the users who have access to patient data to maintain confidentiality.

#### **6 Conclusion**

In this paper, we presented our proposed meta-model that permits modeling the general architecture of the IoT. We implemented two functional applications to help the world in this pandemic and outbreak the spreading of COVID-19. Firstly, we presented the necessary equipment in both scenarios. Next, we designed correct models by construction using UML diagrams to describe the behavioral and structural views of our system. After that, we implemented the system using the Embedded C (to program the Node MCU) and Java (to program an Android application) languages. Finally, we presented and evaluated our application. The application has shown that it is effective in fighting the coronavirus with the obtained results.

In the short term, we will strengthen our application by adding more sensors to offer more services and actuators to have a good preventive and alerting system.

In the long term, we will integrate new technologies such as artificial intelligence to get the most out of the huge amounts of data that sensors collect. This will allow us to make highly precise diagnoses and reactions.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Biomedical and Health Informatics**

# **Tree-Based Models for Pain Detection from Biomedical Signals**

Heng Shi<sup>1</sup>, Belkacem Chikhaoui<sup>2</sup>, and Shengrui Wang1(B)

<sup>1</sup> Department of Computer Science, Universit´e de Sherbrooke, Sherbrooke, Qc, Canada

*{*heng.shi,shengrui.wang*}*@usherbrooke.ca <sup>2</sup> Department of Science and Technology, Universit´e TELUQ, ´ Quebec City, Qc, Canada Belkacem.Chikhaoui@teluq.ca

**Abstract.** For medical treatments, pain is often measured by selfreport. However, the current subjective pain assessment highly depends on the patient's response and is therefore unreliable. In this paper, we propose a physiological-signals-based objective pain recognition method that can extract new features, which have never been discovered in pain detection, from electrodermal activity (EDA) and electrocardiogram (ECG) signals. To discriminate the absence and presence of pain, we establish four classification tasks and build four tree-based classifiers, including Random Forest, Adaptive Boosting (AdaBoost), eXtreme Gradient Boosting (XGBoost), and TabNet. The comparative experiments demonstrate that our method using the EDA and ECG features yields accurate classification results. Furthermore, the TabNet achieves a large accuracy improvement using our ECG features and a classification accuracy of 94.51% using the features selected from the fusion of the two signals.

**Keywords:** Pain detection · Physiological signals · Classifier · TabNet

### **1 Introduction**

Pain is a precious tool for medical attention. A widely accepted definition considers pain as an unpleasant sensory and emotional experience associated with actual or potential tissue damage, or described in terms of such damage [1]. Pain helps identify harmful situations, avoid tissue damage, and promote healing [2]. While each individual learns pain through previous experiences related to injury, which can confound subjective pain ratings [3]. The criterion for evaluating pain can be different for each person. Consequently, accurate assessment of pain is challenging due to the subjectivity of pain.

As reported in [4], pain is a complex and not entirely understood phenomenon. Self-report is used as the golden measurement, which is however not always reliable and valid [5]. For instance, assessing pain from children depends on the cognitive development, clinical context, and pain typology [6]. Therefore, the awareness of pain may vary among different ages of children. For the elderly, effective pain evaluation can be a challenge [7]. Moreover, pain may not be self-reported in some situations. For example, neonates cannot utter words to describe their pain [8]. Individuals who have no clear consciousness of their feelings, like dementia [9], cannot report their pain neither. An objective system for automatically recognizing pain can contribute to detecting pain in such situations. This in turn helps improve people's physical health and mental happiness.

A bunch of machine learning methods have been developed for pain recognition, such as random forest [11] and support vector machines (SVM) [10]. However, these classifiers rely on the selected features and empirical knowledge and therefore suffer from the complexity of the tasks. Nowadays, it is well known that deep learning can provide accurate recognition or prediction for complex data science problems [12] as it allows discovering hidden patterns in data and modelling complex relationships between variables.

In light of the above statement, we explore four tree-based classification models, i.e., eXtreme Gradient Boosting (XGBoost) [13], random forest, Adaptive Boosting (AdaBoost) [14], and TabNet (which is the recently proposed deeplearning-based ensemble decision trees) [15]. To improve the classification accuracy, we extract new electrodermal activity (EDA) features and electrocardiogram (ECG) features of physiological signals. The main contributions of this work include: 1) the extracted novel features from biomedical signals, which can boost the current models in pain assessment; 2) the application of TabNet model, which is the first attempt in pain detection; 3) the outstanding performance of TabNet for most detection tasks, compared to the existing models.

#### **2 Related Work**

In this section, we will introduce the previous studies that are related to this work. All the related work mentioned below with better performance will be compared to our method in Sect. 4.2.

Early research in pain assessment was mainly focused on fusion of multimodel signals, involving facial expression and biomedical signals based on BioVid Heat Pain Database (BVDB) (Part-A) [16]. For example, Werner et al. [17] applied a random forest classifier on multi-model signals to detect pain level. K¨achele et al. [18,19] conducted random forest classifier for continuous prediction of pain intensity. However, facial expression-based pain recognition needs tracking of the facial regions, which can be cumbersome and complex in clinical application. Pain research indicates that the presence of pain significantly interacts with autonomic nervous system and thus leads to changes in EDA and heart rate [20]. The bio-physiological signals are recorded in BVDB as follows: 1) EDA, which is also referred to as skin conductance (SC) or galvanic skin response (GSR), measures the changes in the electrical properties of the skin and shows the strong association to the emotional arousal [16]; 2) ECG records the electrical activity of the heart and supplies a significant amount of information about heart function [21]; 3) Electromyogram (EMG) records the electrical activity of muscles. The activity of the trapezius is a hint of a high stress level which is to be expected during pain stimulation [16].

Given the correlation between physiological signals and unpleasant pain, nowadays, pain assessment researchers have shifted to focusing on merely physiological signals. Gruss et al. [22] extracted a total of 159 features and proposed SVM classifiers for binary pain classification. Deep learning models on physiological signals have shown promising results in pain recognition. In Lopez-Martinez et al. [23], EDA and ECG signals features were introduced into multi-task neural networks which have two hidden layers, one shared and one person-specific, and proved to perform better than single-task neural networks. Wang et al. [24] proposed deep Recurrent Neural Network based hybrid classifiers to classify the pain intensity. They applied a bidirectional Long Short-Term Memory (LSTM) network to learn temporal dynamic characteristics of physiological signals and fused them with handcrafted features. Thiam et al. [25] proposed a multi-modal information aggregation approach based on Deep Denoising Convolutional Auto-Encoders for pain assessment on two different pain databases including BioVid Heat Pain Database. Thaim et al. [26] designed Convolutional Neural Network (CNN) based on physiological signals (EDA, ECG, and EMG) for pain classification. Pouromran et al. [27] computed features from EDA, ECG and EMG signals and trained machine learning models—including Linear Regression, Support Vector Regression, Neural Network and XGBoost—on these features for pain intensity estimation. Subramaniam et al. [28] proposed a hybrid CNN-LSTM classification based on ECG and EDA signals for binary pain detection.

In [26], the pain classification with EDA signals was much higher than other signals and the fusion of EDA and ECG signals. However in this work with our new model, we observed different results. For ECG signals, our extracted features provide additional information allowing to improve pain detection. For instance, by comparing the results of the classification task *B*<sup>0</sup> vs *P*<sup>4</sup> using random forest classifier, we notice that K¨achele et al. [18,19] provided an accuracy of 53.90%, Werner et al. [17] 62%, while we were able to generate an average accuracy of 67.18% using our new ECG features. Furthermore, using the TabNet model with ECG features, we were able to improve the accuracy to 81.12%.

#### **3 Proposed Model**

#### **3.1 Biovid Heat Pain Database (BVDB)**

The BioVid Heat Pain Database [16] contains multidimensional datasets, both video signals and biopotentials, which provide potential to advance an automated pain recognition system. The data was collected on 90 subjects from 18 to 65 years of age, using a thermode at right arm for the pain elicitation. The experimenters randomly heated the participants with four calibrated intensities (*T*1*, T*2*, T*3*, T*4) and each stimulation was held up to 4s (Fig. 1a). Thus, it generates four pain intensity levels (i.e., *P*1*, P*2*, P*3*, P*4). 20 times of each pain level were given. The pauses between the stimuli were kept at baseline temperature (*T*0) and around 8 s-12 s, which was no pain (*B*0). Each dataset was extracted with

**Fig. 1.** (a) Pain stimulation, represented by the heat levels from *T*<sup>0</sup> to *T*4. The features are extracted from the blue window of length 5.5 s. (b) The standard ECG signal with P, Q, R points and amplitudes.

the length of 5.5 s starting 1 s after the stimulation was given, where 2816 data points were collected. Consequently, the experimenters retained 20 samples/class × 5 classes = 100 samples for per subject.

#### **3.2 Feature Extraction**

BVDB includes five parts, Part A-E [29]. Besides Part-A, we also explored our methods on Part-B with EDA and ECG signals. Before extracting features from ECG signals, a Butterworth bandpass filter with frequency range of [0.1, 250] was introduced to remove the noise of the muscles, baseline wander and other interference [30]. In Part-B, 6 samples contain a quantity of nulls with EDA signals. Their corresponding ECG signals are also invalid. Then we removed the 6 samples in our experiment, which are all from no pain stimulation (*B*0).

In addition to utilizing EDA and ECG features proposed in [23], we explored novel features. We found that some EDA signals are like the sine function and some are not, so we investigated the numbers of crests and troughs, and the relations between the start and end states, minimum and maximum. These four SC features were appended. On the other hand, ECG signals are mainly reflected on the characteristics of P, R and T points. ECG signals consist of the P-QRS-T waves and P, R, T points are the peaks of P, R, T waves (Fig. 1b) [31]. We detected false R points with visualization and removed them via their amplitudes because these points may seriously impact on the preciseness of further feature extraction. We extract the information of R, P, and T points as ECG features.

**Skin Conductance (SC).** We applied 12 SC features in [23]. (1) maximum; (2) range; (3) standard deviation; (4) interquartile range; (5) root mean square; (6) mean; (7) mean absolute value of the first differences (Mav1d): 1 *N*−1 -*N*−1 *<sup>i</sup>*=1 |*x<sup>i</sup>*+1 − *xi*|; (8) means absolute value of the second differences:

1 *N*−2 -*N*−2 *<sup>i</sup>*=1 |*xi*+2 − *xi*|; (9) mean absolute value of the first differences of the standardized signal: *<sup>x</sup>*−*mean*(*x*) *std*(*x*) ; (10) mean absolute value of the second differences of the standardized signal; (11) skewness; (12) kurtosis were computed. In addition, we appended the novel features: (13) differences among maximum, first state and last state (Max1n): *maximum* − *x*<sup>1</sup> − *x<sup>N</sup>* ; (14) the number of troughs; (15) differences between minimum, first state and last state (N1Min): *x<sup>N</sup>* − *x*<sup>1</sup> − *minimum* and (16) the number of crests. The abbreviation of SC features are summarized in Table 1.

**Electrocardiogram.** First, we located P, R, T points. Their amplitudes and positions were kept. The R points were detected by Pan-Tompkins algorithm [32]. After locating the R peaks, the area of T wave was determined from *R*(*i*) + 0*.*16×*RR*(*i*+ 1) to *R*(*i*)+0*.*57×*RR*(*i*+ 1) [21], where *R*(*i*) is the *i th* position of R point; *RR* represents RR interval, *RR*(*i* + 1) is achieved from *R*(*i* + 1) − *R*(*i*). The peak of phase signal within the segment was considered as the T points. Finally, in the similar way, P waves were demarcated from *R*(*i*)+0*.*7×*RR*(*i*+ 1) to *R*(*i*)+0*.*07×*RR*(*i*+ 1) [21], where the range was modified to make it suitable for datasets from BVDB. The P, R, T points were correctly located and a sample of ECG signals is shown in Fig. 2.

Second, we calculated the following ECG features. The 4 of 5 ECG features proposed in [23] are included in our work. They are (17) the mean of the IBIs; (18) the root mean square of the successive differences (RMSSD); (19) the mean of the standard deviations of the IBIs (SDNN); (20) the mean of the slope of the liner regression of IBIs in its time series; (21) the ratio of SDNN to RMSSD. New features that we computed are (22) the number of R peaks in the windows of 5.5 s (Fig. 1a); (23) the range of R amplitudes; (24) the standard deviation of R amplitudes; (25) the mean of the duration of PT (Fig 1b): <sup>1</sup> *M* -*M*−1 *<sup>i</sup>*=1 (*T*(*i*)−*P*(*i*)), where *M* is the minimum of the number of P points and T points, *T*(*i*) is the *i th*

**Fig. 2.** Raw ECG signal of a sample with detected T, P, Q, R, S points.


**Table 1.** Extracted features

position of T point, *P*(*i*) is the *i th* position of P point; (26) the root mean square of the successive differences of the duration of PT (PTxRMSSD); (27) the mean of the standard deviation of the duration of PT (PTxSDNN); (28) the ration of PTxSDNN to PTxRMSSD; (29) the mean of the difference of amplitude of PT: <sup>1</sup> *M* -*M*−1 *<sup>i</sup>*=1 (*Amplitude*(*Ti*) − *Amplitude*(*Pi*)); (30) the root mean square of the successive differences of the amplitude of PT (PTyRMSSD); (31) the mean of the standard deviation of the amplitude of PT (PTySDNN); (32) the ration of PTySDNN to PTyRMSSD. They are summarized in Table 1.

#### **3.3 Methodology**

The main goal is to discriminate between no pain (*B*0) and the presence of pain (*P*1, *P*2, *P*3, *P*4) based on physiological signals. For this, we establish four binary classification tasks. In the tasks, we extract features from EDA and ECG signals – i.e., create tabular EDA and ECG features. Based on these features, we explore the decision trees (DTs) based classifiers, i.e., random forest, AdaBoost, XGBoost, and TabNet. In general, ensemble methods tend to yield better results than standard DTs. For this reason, we choose tree-based tabular data learning methods. Random forest [11] is an ensemble of decorrelated decision trees. Adaboost is a classic type of Boosting [14]. Boosting is a general ensemble method that produces a strong classifier from an ensemble of weak learners. XGBoost [13] is an ensemble DT approaches that follows the principle of boosting. TabNet [15] is the integration of deep neural network into DTs. The architecture of TabNet contains a Batch Normalisation layer to filter the raw data, and several transformer blocks to learn relevant features. It also consists of a sequential attention mechanism and learnable masks to choose which feature to process at each decision step. This characteristics enables efficient learning as the learning capacity is used for the most salient features. Finally, we can select the best combination of signals, features, and models based on their performance.

#### **4 Experiments**

#### **4.1 Setup**

On the Part-A and Part-B datasets, we conducted four binary classification tasks of discriminating subjects with no pain (*B*0) versus pain (*P*1*, P*2*, P*3*, P*4) condition, i.e., *B*<sup>0</sup> vs. *P*1, *B*<sup>0</sup> vs. *P*2, *B*<sup>0</sup> vs. *P*3, and *B*<sup>0</sup> vs. *P*4. Since Part-A includes 87 subjects and Part-B 86 subjects, we have 87×20×2 = 3480 samples for each of the four Part-A tasks and (86 × 20 − 6) + 86 × 20 = 3434 samples for each Part-B task. We applied the PyTorch implementation of TabNet [15], which is available at [33]. We evaluated the model's performance via the stratified 10-fold cross-validation because the Part-B data is imbalanced due to 6 invalid samples in no pain level (*B*0). To improve the models' performance, we employ the grid search optimization method to select the best hyperparameters from the following hyperparameter candidates:


#### **4.2 Results and Discussion**

We present and discuss classification results with respect to Part-A and Part-B in this section. Since it's the first attempt to investigate machine learning algorithms on Part-B, we only compare our work with previous works involving physiological signals of Part-A.



**Table 2.** Pain recognition with physiological signals of **Part-A**. Mean accuracies are reported for stratified 10-fold cross-validation, for representative models and binary classification tasks (Mean% *±* Standard Deviation%).

The best accuray is 69.19%, 74.34%, 76.82%, and 82.85% with ECG signals, and 75.24%, 78.74%, 82.44% and 88.44% with the fusion of EDA and ECG. It shows the same trend as Part-A.

In Table 4, we compare our results with previous studies on single signals of Part-A for classifying no pain and highest pain levels. For EDA signal, Thiam et al. [26] applied CNN algorithm with an average accuracy of 84.57% and achieved the highest accuracy among previous work. For XGBoost, we achieved an accuracy of 85.23%. TabNet outperforms the most previous work as well. For ECG signal, TabNet with our selected features yields 72.18%, 71.81%, 77.04% and 81.12% accuracy for the four tasks, outperforming the currently best model [28] (with an accuracy of 68.7%, 62.61%, 67.86%, and 75.21%). The performance with ECG signals has been significantly improved. From these results, we found that ECG signals are essential for pain detection, which are comparable to EDA signals; this finding has never been previously reported.

Table 5 shows the comparison between the best binary classification accuracy reported by the previous studies and ours, for discriminating no pain (*B*0) and pain tolerance (*P*4) on Part-A. The facial expression (Video) turns out to be non-promotional for pain detection, according to results shown in [17–19]. On the contrary, using physiological signals, especially EDA and ECG signals, can boost the pain detection. Additionally, using the fusion of EDA and ECG signal, TabNet achieves the best pain detection results, with an accuracy of 94.51%.

**Table 3.** Pain recognition with physiological signals of **Part-B**. Mean accuracies are reported for stratified 10-fold cross-validation, for representative models and binary classification tasks (Mean% *±* Standard Deviation%).


**Table 4.** Performance comparison of the binary classification task *B*<sup>0</sup> vs. *P*<sup>4</sup>


In order to investigate the contribution of EDA and ECG features for the classification of no pain and pain tolerance, we plot the significance of the features extracted from the fusion of EDA and ECG in using XGBoost (Fig. 3a) and TabNet (Fig. 3b). Based on the XGBoost's feature importances, we find that the extracted EDA features contribute 65% to the classification, while ECG contributes 35%. (The computation is not explicitly presented here due to the space limit.) Moreover, the combination of the 4 features from EDA (i.e., the features #13–16 in Table 1) and the 10 from ECG (i.e., features #23–32 in Table 1) con-


**Table 5.** Comparing the best binary classification accuracy of each study on Part-A

tribute 37.48% to the classification. For TabNet, the extracted EDA and ECG features make a contribution of 61.15% and 38.85%, respectively, while our 4 new EDA features and 10 ECG features contribute 39.76% to the classification. The comparison reveals that 1) besides EDA, ECG is significant for the pain detection; 2) important features have been overlooked by previous researches.

**Fig. 3.** The importance of features (EDA and ECG) determined by XGBoost(a) and TabNet(b) for the classification of *B*<sup>0</sup> and *P*4.

#### **5 Conclusion**

We proposed in this paper an automatic pain detection method for the classification of subjects with no pain and pain condition. For each of the four classification tasks, we explored the performance of four tree-based models that are learned based on the features extracted from a single physiological signal (EDA and ECG) and from their fusion (EDA + ECG). We removed the noise of ECG signals and appended corresponding P-QRS-T wave information to ECG features. Our method can close the gap to previous work and discover significance of ECG in pain detection. The experimental results demonstrate that our method achieves the highest classification accuracy when using a single signal (EDA or ECG). The 94.51% accuracy on the data of physiological signals reveals the promising of our method in practical use – e.g., detecting pain based on the real-time data collected from wearable devices.

**Acknowledgement.** This work was supported by the Natural Sciences and Engineering Research Council of Canada under Discovery Grant RGPIN-2020-07110 and Discovery Accelerator Supplements Grant RGPAS-2020-00089 to Shengrui Wang. Thanks Jianfei Zhang, Jean Marie Tshimula, and Patrick Asante Owusu for their suggestions that have helped greatly improve this paper.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Stress Prediction Using Per-Activity Biometric Data to Improve QoL in the Elderly**

Kanta Matsumoto1(B) , Tomokazu Matsui<sup>1</sup> , Hirohiko Suwa1,2 , and Keiichi Yasumoto<sup>1</sup>

<sup>1</sup> Nara Institute of Science and Technology, 8916–5 Takayama-cho, Ikoma-shi Nara 630-0192, Japan matsumoto.kanta.me3@is.naist.jp <sup>2</sup> RIKEN Center for Advanced Intelligence Project AIP, Tokyo, Japan http://www.naist.jp/

**Abstract.** To improve the QoL of the elderly, it is essential to predict their stress states. In general, the stress state varies from day to day or time to time depending on what activities are performed and how long/strong. However, most existing studies predict the stress state using biometric data and specific activities (e.g., sleep time, exercise time and amount) as explanatory variables, but do not consider all daily living activities. Therefore, it is necessary to predict the stress state by linking various daily living activities and biometric information. In this paper, we propose a method to improve the prediction accuracy of stress estimation by linking daily living activities data and biometric data. Specifically, we construct a machine learning model in which the objective variable is the result of a stress status questionnaire obtained every morning and evening, and the explanatory variables are the types of daily living activities performed in the 24 h prior to the questionnaire and the feature values calculated from the biometric data during each of the performed activities. The results of the evaluation experiments using the one month data collected from five elderly households, show that the proposed method (using per-activity biometric features) improves the prediction accuracy by more than 10% from the baseline methods (with biometric features without considering activities).

**Keywords:** Health care *·* Smart home *·* Wearable sensors *·* Stress prediction

### **1 Introduction**

With the aging of the population in developed countries, there is an urgent need for effective measures to promote the care of the elderly and the extension of their healthy life expectancy. However, there are many elderly people who wish to continue to live independently at home. To grant this wish, it is important to build an environment in which elderly people can understand their daily activities and manage their health conditions by themselves. There have been many studies on activities sensing/recognition technologies in the home and health condition monitoring using these technologies to build an environment for self health management and improve lifestyle habits of the elderly [1].

In order to provide monitoring services and improve the lifestyle of the elderly, it is necessary to develop indicators of health status and investigate the factors that cause these indicators to change. Currently, the World Health Organization (WHO) has developed a questionnaire index called WHOQOL-100 [2] as a representative indicator of health status. However, the method of measurement using such a questionnaire index places a heavy burden on respondents due to a large number of questions. For this reason Amenomori et al. [3] conducted a study to measure HRQOL (Health Related Quality of Life) with less burden by using devices such as smartphones and smartwatches. They aimed to improve HRQOL which is strongly related to physical and mental stress states, by continuously measuring HRQOL and detecting signs of stress early to prevent it.

Stress estimation has also been studied using devices rather than questionnaires to reduce the burden of estimation. Stress estimation using devices has been applied to many situations such as daily life [4–6]. Fukuda et al. [4] collected wake-up times and sleep data obtained from wearable devices and answers of occupational health and safety questionnaire called the Depression and Anxiety Mood Scale (DAMS) in office workers for two to three weeks at a general company, and constructed a machine learning model to estimate the questionnaire answer. In addition, Natasha et al. [5] proposed a method to quantitatively measure and predict the health, stress, and happiness of the next day using smart devices, respectively, in order to understand the stress state at work.

In this paper, as a method to improve the prediction accuracy of stress estimation, we propose a method to predict stress by linking daily living activities data and biometric data. Specifically, we construct a machine learning model with the results of a stress status questionnaire obtained every morning and evening as the objective variable, and the types of daily activities performed in the 24 h before the questionnaire and the features calculated from biometric data during each daily activity as the explanatory variables. As biometric data, we use the Lorenz plot area calculated from the heart rate data collected by a smartwatch (Fitbit). The Lorenz plot area is known to be useful for stress estimation because it can visualize the activity level of the parasympathetic nervous system [7].

To evaluate the effectiveness of the proposed method, we applied the method to the dataset [1] consisting of daily living data, biometric data, and stress status questionnaires (two questions in the morning and two questions in the evening) collected from five households of elderly people over 60 years old for one month. To confirm the validity of the per-activity biometric features, we compared the baseline method 1 (using 24-h RRI variance and Lorenz plot area [7,8] as features), the baseline method 2 (adding sleep time, which has been validated in the previous study [4], as a feature to the baseline method 1), and the proposed method (using RRI variance and Lorenz plot area for each activity type as features). For three class classification, the accuracy of the four questionnaires on average was 34.5% for the baseline method 1 and 47.25% for the baseline method 2, while the average accuracy of the proposed method was 58.25%, confirming the effectiveness of considering the per-activity biological features.

### **2 Related Research**

This section surveys existing studies on QoL estimation, stress estimation and health management using smart home technologies.

#### **2.1 QOL Estimation**

Quality of life (QoL) is a measure of satisfaction and quality in the daily life. The QoL [9] research originally started as a concept to discuss the quality of life after treatment in the medical field, but it is now used not only in the medical field but also as a concept related to the quality of life in general, such as work-life balance and happiness.

In particular, QoL, which is directly related to human health, is called HRQOL (Health Related Quality of Life), and is evaluated by categorizing it into various domains such as physical, psychological, social interaction, economic and occupational, and religious and spiritual states. The World Health Organization (WHO) has developed various indicators to quantitatively assess HRQOL, such as WHOQOL [2] and Short Form [10]. These indicators are assessed using paper questionnaires. However, the WHOQOL-100 [2] requires 100 items in 6 domains, while the SF-36 [10] requires 36 items in 8 domains. The labor to answer these questions make it difficult to assess the quality of life on a daily basis.

Amenomori [3] et al. have proposed a method to continuously measure HRQOL using mobility and biometric information obtained from smartphones and smartwatches, and have shown that HRQOL can be predicted using a small number of questionnaires and information from smart devices. They aimed to improve HRQOL by continuously measuring HRQOL, which is strongly related to physical and mental stress states, to detect early signs of stress and to prevent it.

Unlike Amenomori's work, the goal of our work in this paper is to detect and prevent causes of daily living stress in the elderly at an early stage. For this goal, we try to use per-activity biometric information to predict the stress state so as to understand which activity causes the stress.

#### **2.2 Stress Estimation**

There have been many studies on stress estimation using devices in many situations. There are two groups of stress estimation. The first group is the studies conducted in a controlled laboratory environment [11]. In these studies, the researchers intentionally generated stress using some kind of stress test, where the researcher has complete control over the stress level, and high-stress detection accuracy (80–97%) is usually reported.

The second group is studies that analyze stress in real life [4,5,12,13] which have reported low accuracy [14]. Therefore, various systems are being researched to improve the prediction accuracy. To analyze stress in workers, not only smartwatches but also chest-mounted heart rate sensors (e.g., WHS-3 [15]) have been used [16]. However, systems that use multiple wearable sensors place a high burden on the subject and are not suitable for real-time stress detection in daily living. Therefore, it is desirable to use wearable devices as minimum and unobtrusive as possible.

#### **2.3 Health Management in Smart Home**

Since the elderly spend more time at home than out of the house, there has been development of systems to manage health conditions using activities data in the home, such as smart homes. Some smart home research has attempted to correlate activities and predict the well-being of occupants within a living space. The goal of Intel Research's Computer-Supported Coordinated Care (CSCC) project [17] is to identify the care network characteristics and needs of older adults who wish to live at home. In addition, Jakkula et al. [6] aimed to identify trends in health status over time, with the goal of creating a smart home system focused on next-generation health care capable of home health care. However, most existing studies related to stress estimation in the smart home have been conducted in a controlled smart home, not in the subject's home or other ordinary residence. Therefore, there are few studies that analyze stress in the home in real life.

In addition, in most of the related studies described above, the sensor data and feature values used do not take into account all aspects of the subject's home life. Therefore, if there is no information related to activities of daily living, it is not possible to understand the activities that causes stress, and it may not be possible to improve the stress state. Therefore, it is necessary to incorporate data related to activities and biometric information in the home.

#### **3 Proposed Method**

In this study, we propose a method for estimating the stress state at the end of a day (or the beginning of the next day) from the values of the stress index for each of daily living activities which are performed in a day. In the following subsections, we describe the method of obtaining daily living activities and stress indicators used in the proposed method, the definition of the stress state to be predicted, and the method of constructing the model.

#### **3.1 Acquisition of Daily Living Activities**

In this study, we assume an environment in which the activities of residents can be automatically obtained by using an in-home daily living activities recognition

**Fig. 1.** Components of SALON system.

system (such as SALON [18]). We target primary activities such as cook, meal, rest, work, clean, wash, go out, bath, and sleep. We assume that the in-home activity estimation system records the time of day when each activity is performed. Table 1 shows an example of activities log. The SALON system does the following to stabilize the annotation accuracy. The SALON system places annotation buttons near the activity location with an aim to improving annotation accuracy. When a user forgets to press the annotation button, annotation data is corrected by the user.

#### **3.2 Acquisition of Stress Index**

In this study, we assume that residents wear a wearable biometric sensor (ECG) that can measure heart rate and heart rate variability. In the proposed method, we focus on the RRI variance and Lorenz plot area as stress indicators that can be calculated from heart rate variability data [7,8].

In a Lorenz plot (LP), denoting RR interval at time t by RRI(t), a dot is plotted at the coordinate (RRI(t), RRI(t + 1)) in a two dimensional plane. The area of the ellipse that covers the plotted dots is calculated as the Lorenz plot area as shown in Fig. 3.

On the y = x axis, let m denote the mean of the distance from the origin (0, 0) (on the y = x axis), δ*<sup>x</sup>* the standard deviation from the origin. On the y = *−*x axis, let δ(−*x*) denote the standard deviation from the origin (0, 0). In this case, the area of the ellipse with major axis δ*<sup>x</sup>* and minor axis δ(−*x*) is calculated as follows:

$$S = \pi \cdot \delta\_x \cdot \delta\_{(-x)} \tag{1}$$

#### **3.3 Association of Activities with Stress Indicators**

In the proposed method, for each activity obtained in Sect. 3.1, the RRI variance and Lorenzplot area are calculated from the heart rate data while the activity is performed by the method described in Sect. 3.2. If the same activity is performed repeatedly in different time intervals (e.g., Cook, Meal and Rest in Table 1), the biometric data measured during those intervals are integrated and stress indicators are calculated for the integrated data. Finally, per-activity stress indicators values are calculated as shown in Table 2.

**Fig. 2.** Tabulation procedure for morning and evening questionnaires.

#### **3.4 Prediction Model Construction**

In this study, we construct a machine learning model to predict the stress state where we used as groundtruth the answers for the questions that residents give at the end of the day, from per-activity stress indicators values explained in Sect. 3.3.

### **4 Evaluation Experiment**

For the evaluation experiment, we collected a dataset consisting of activities data, biometric data and questionnaire answers. In each home of the elderly participants, we set up sensors (up to 10 motion and ambient sensors and a few door sensors), five annotation buttons (corresponding to activities) and a server of a SALON shown in Fig. 1. In the experiment, first we extract features from the dataset, and then construct a random forest classifier to predict the stress level.

#### **4.1 Dataset**

We collected the dataset from five general elderly households (one single, four married couples, all in 60's), which consists of daily living activities data, biometric data, sensor data, and stress state data (obtained through questionnaire). The data collection period is one month for each household. The details of each data are described below.

**Fig. 3.** How to calculate the Lorenz plot area.

**Daily Living Activities Data.** We collected data on five typical daily living activities (bathing, cooking, eating, going out, sleeping, and other). Residents record the start and end of each activity by pressing the annotation buttons Shown in Fig. 1 installed at the locations where the activities are performed.

**Biometric Data.** We use the heart rate data collected by a smartwatch (Fitbit Alta HR) as biometric data. Fitbit collects the heart rate [bpm] once every 15 s for one minute.

We convert the heart rate to RRI [ms] and generate features for stress estimation, i.e., the variance of RRI and Lorenz plot area. The conversion equation for RRI is shown in Eq. (2).

$$RRI = \frac{60}{Hearrate \times 1000} \tag{2}$$

where RRI is defined as the heartbeat interval in ms, and Heartrate is the number of heartbeats per minute.

**Stress State Data.** Stress state data is collected by asking residents to fill out questionnaires immediately after waking up and before going to bed each day. We adopted four questionnaire items related to mental stress and physical stress. In the questionnaire, a five-point Likert scale was used. The specific questionnaire items are shown below.

#### **Questionnaire Items Immediately After Waking Up**

**M1.** Did you feel physically refreshed this morning? **M2.** Did you feel mentally refreshed this morning?




#### **Table 2.** Per-activity stress indicators

#### **Questionnaire Items Before Bedtime**

**N1.** Do you experience any physical stress due to physical pain or discomfort? **N2.** Do you experience mental stress?

#### **4.2 Selection of Features and Baseline Methods**

The proposed method uses per-activity biometric features of all activities performed in 24 h before the questionnaire. In this evaluation experiment, two other methods (baseline 1 and baseline 2) are set as baselines.

The common basic features used for all three methods are as follows:


The specific features used for each method are as follows:

**Baseline 1** Lorenz plot area in the last 24 h

**Baseline 2** Lorenz plot area and sleep time in the last 24 h

**Proposed method** Lorenz plot area for each activity (up to 6 types), Lorenz plot area for each activity in 3 h after waking up (up to 6 types)

#### **4.3 Model Construction and Validation**

We use random forests to construct stress level prediction. This time, we divided the data into three subsets and used a three-fold cross-validation method for model training and validation. In constructing models, the answers (in 5 level

**Fig. 4.** Confusion matrix for the N1 (physical stress) questionnaire at night.

likert scale) of the questionnaire were reorganized into three levels: good, bad, and neither good nor bad.

To confirm the effectiveness of the per-activity biometric features, we compared the baseline method 1 (using 24-h RRI variance and Lorenz plot area as features), the baseline method 2 (adding sleep time as a feature to the baseline method 1), and the proposed method (using RRI variance and Lorenz plot area for each type of activities performed in 24 h as features).

### **5 Results**

#### **5.1 Effects of Different Features**

The results of predicting the questionnaire results of three methods are shown in Table 3. The proposed method achieved prediction accuracy of 0.57, 0.56, 0.63, 0.57 for questions M1, M2, N1, and N2, respectively while baseline 1/2 did 0.32/0.47, 0.35/0.49, 0.33/0.54, and 0.38/0.39, respectively. This suggests that adding sleep time features (baseline 2) improves accuracy so some extent but adding per-activity biometric features (the proposed method) greatly improves the accuracy compared to baseline 1.

#### **5.2 Stress Estimation Model Construction**

The stress level prediction model is constructed using random forest using features described in Sect. 4.2. Each question's answer obtained as stress state is


**Table 3.** Prediction accuracy of three methods for each question

**Table 4.** Evaluation of recall and f-measure for questionnaire N1


**Table 5.** Importance of nighttime questionnaire N1 (physical stress).


used as the ground truth data. The stress state data is rated on a 5-point scale, but since it is only necessary to judge whether the stress state is good or bad, the scale is reorganized to 3 levels: good, neutral, and bad. A three-fold crossvalidation method is used to validate the model.

We show more detailed results of N1 prediction in Table 4 and the confusion matrix in Fig. 4.

#### **5.3 Evaluation of Feature Importance**

In order to identify the features that contribute to the estimation, we investigated their importance. Table 5 shows the contribution of features in the model to predict N1's answer. In Table 5, the mean value and standard deviation of RRI had the high contribution (1st and 4th rank). Lorenz plot areas for some activities also show high contribution. Especially, Lorenz plot area for bathing activity is ranked in the second. We also confirmed that the Lorenz plot area of bathing often ranked high in other questionnaires. Therefore, it is suggested that the stress prediction accuracy could be improved with per-activity stress indicators more finely than just using biometric features. Also, it is suggested that the Lorenz plot during bathing is somehow related to stress.

### **6 Conclusion**

In this paper, to improve the prediction accuracy of stress estimation, we proposed a method to predict stress level by linking daily living activities data and biometric data. Through the experiment using the dataset collected from elderly households, it is shown that the proposed method with per-activity stress indicators features in the last 24 h improves the prediction accuracy to a great extent compared to the baseline methods using only stress indicators without distinguishing activities.

As future work, we plan to apply the results of stress estimation obtained in this study to create a system that can predict QoL with high frequency and provide feedback on daily living activities based on the results of QOL estimation, aiming to build a system that encourages changes of activities to improve health state.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Short Contributions: Medical Systems and E-health Solutions**

# **An Exploratory Study on Development Smart Cradle for Women with Spinal Cord Injury: Focus Group Interview**

Jae-nam Kim1, Ha-yeon Yang2, Min-kyung Kim2, Hyun-kyung Kim2, Sun-hwa Shim3, Eun-joo Kim3, Wan-ho Jang3(B) , and Sun-young Jo4

<sup>1</sup> Health Science Research Institute, Jeonju University, Jeonju 55069, South Korea

<sup>2</sup> Department of Rehabilitation Science, Jeonju University, Jeonju 55069, South Korea

<sup>3</sup> Department of Occupational Therapy, Jeonju University, Jeonju 55069, South Korea

otjang@jj.ac.kr

<sup>4</sup> Department of Rehabilitation Medicine, Jeonbuk National University Hospital, Jeonju 54907, South Korea

**Abstract.** This study is preliminary research to develop a smart cradle for women with spinal cord injury. The purpose of this study was to investigate the needs for improvement of the product and important factors related to product development. A focus group interview was conducted with a total of 5 women with spinal cord injury who had experienced parenting after spinal cord injury. After recording all of the focus group interviews, researchers individually analyzed the content and integrated the results. Easy access cradle design for wheelchair users, attachment of wheelchair and cradle when moving at home, an open and lockable door one side of the cradle were required in cradle structures. Electronic height adjustment, bounce mode, children's motion sensor, and function linked with a smartphone should be reflected in the development of the cradle. This result is meaningful in that it suggests points to be considered in the process of developing an assistive device by reflecting the desire to understand the grievance women with spinal cord injury when parenting.

**Keywords:** Smart cradle · Spinal cord injury · Child care · Parenting · Focus group interview

### **1 Introduction**

Parenting is one of the greatest challenges and joys adults can experience, and adults with spinal cord injury (SCI) who are physiologically capable of having children are no exception [1]. Women with SCI face various challenges in the process of conceiving, giving birth, and raising a child due to physical limitations [2, 3]. Mothers with SCI say they often see themselves as superhuman by performing parenting tasks that others would not be able to do because of their disability, or performing childcare tasks that are considered inappropriate compared to mothers with physical abilities [3–6]. Fears of being branded as inappropriate parents or of 'parenting' their children put some parents under pressure to handle all parenting activities on their own, without the help of family or other support providers [7–11]. The use of assistive devices is a representative method that can help people with physical disabilities to solve difficulties and problems in the process of parenting. Assistive technology approaches such as assistive devices and social support play an important role in independently performing daily activities related to parenting. For example, a cradle designed for the physically disabled people can be used to care for child in the early stage of parenting, and has a great impact on the safety of a child. Therefore, the need for research on the development and effectiveness of assistive tools that are more ergonomic and functional and incorporate the latest technologies such as IoT systems has been raised. Along with this trend, the need for research that can elicit social support including financial support so that people with disabilities can practically use cutting-edge tools is also being emphasized [12]. The purpose of this qualitative study is to identify the needs related to the cradle of women with spinal cord injury, and to develop a smart cradle by reflecting the research results. Furthermore, it is expected to provide data that can elicit social support for assistive technology through the results of tracking research on the effectiveness, importance, and user satisfaction of the developed smart cradle.

### **2 Method**

#### **2.1 Focus Group Interview (FGI)**

In this study, a Focus Group Interview (FGI) was conducted to in-depth exploration of users' needs. FGI was developed based on the method described by Krueger and Casey [13]. It is a qualitative research method that allows you to better understand the feelings and thoughts of the participants by forming a focus group with common characteristics, such as a study and procedure [14]. This study was conducted to reflect the considerations when developing the cradle by analyzing the experiences of women who have raised a child after spinal cord injury.

### **2.2 Participants**

For one month in June 2021, research participants were recruited with help of the spinal cord injury association. The criteria for selection of participants in this study are as follows. First, women with spinal cord disorders who had experience raising a child after childbirth, second, those who were able to express their opinions sufficiently, and third, those who agreed to participate in the study after hearing the explanation of the purpose and method of the study. Total 5 subjects who met the selection criteria participated in this study (Table 1).

#### **2.3 Research Questions**

The focus group interview questionnaire used in this study was composed of open, introductory, key, and closing questions based on the method suggested by Kruger and Casey. The key questions of this interview were as follows: (Q1) have you ever used


**Table 1.** General characteristics of participants (n = 5)

a cradle? (Q2) what were the inconveniences when using a regular cradle? (Q3) how did you put your child on the bed?, how did you hold your child out of bed? (Q4) what aspects of the cradle would you like to improve considering your physical disabilities?

#### **2.4 Data Collection & Analysis**

We informed study participants about the interview contents and process before FGI. Interview was conducted through videoconference and participants feely expressed their experiences and thoughts for 2 h. All data were collected by recording interview video. Researchers categorized the recorded contents according to common themes using the technique suggested by Krueger [15]. Similar sentences, experiences were classified, and data were analyzed by grouping important themes and categories through continuous review. This qualitative research method focuses on revealing the essence of experience in the context of the participant's situations.

### **3 Results**

#### **3.1 Needs for Cradle Structures**

Various needs were investigated for the structural design of the cradle desired by the participants in order to consider the efficiency of the cradle development. SCI patients spend most of their time at home in wheelchairs. Therefore, in order to take care of a child, lower part of the cradle must be manufactured so that there is a space for a wheelchair to enter. They also said that wheelchair should be able to move with a cradle attached (Fig. 1), there should be an open and lockable door one side of the cradle to pull a child up (Fig. 2).

*"A wheelchair must be able to fit under the cradle because we need to take care of a child as close as possible" (Participant E).*

*"It would be nice if the wheelchair could fit in the cradle and move together" (Participant B)*.

*"Cradle should have an open and lockable door for a child's safety and ease of use" (Participant A)*.

*"Need a storage space around the cradle to put children's items" (Participant C).*

**Fig. 1.** Attachment of cradle & wheelchair **Fig. 2.** Open and lockable door on cradle

#### **3.2 Needs for Cradle Functions**

Participants said that they needed an electronic height adjustment function to change the position of the cradle. Motion sensor and a bouncing mode are needed to detect child's movements (Fig. 3). Finally, they argued that the development of a smart cradle based on artificial intelligence that can be linked with a smartphone is necessary to monitor child's health and surrounding environment (Fig. 4).

*"Need a lift function that can adjust height of the cradle by pressing switch" (Participant C).*

*"It is necessary to be able to control the cradle and check the condition of a child with a smart phone." (Participant D)*.

*"The electric bouncing function seems to be useful for caring for a child" (Participant B)*.

**Fig. 3.** Bouncing function using smartphone **Fig. 4.** Monitoring child's health &

environment

### **4 Conclusion**

We tried to find out important factors related to product development. A focus group interview was conducted with 5 women with spinal cord injury, and various opinions were derived. In terms of design and structure, it is important to easily access the cradle in a wheelchair and attach the cradle to the wheelchair when moving. they also wanted to have a door that could be opened and locked on one side. Electric height adjustment and bouncing, as well as checking the health and environment of the child with a smart phone are necessary for the function. This study is meaningful in that it presented the difficulties that disabled with SCI may experience during childrearing as a factor to be considered in the development of assistive devices. It is hoped that it will be helpful in the development of parenting devices for the disabled people.

**Acknowledgments.** The research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HJ21C0014).

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **ICT-Based Customized Off-Loading Cushion to Prevent Pressure Ulcers for People with Spinal Cord Injury: A Pilot Study**

Yun-hwan Lee1(B) , Kwang-tae Moon1, Dong-wan Kim1, and Jongbae Kim2(B)

<sup>1</sup> Department of Occupational Thearpy, The Graduate School, Yonsei University, Seoul, South Korea enn1210@gmail.com, otbuilder@yonsei.ac.kr, dwan3303@naver.com <sup>2</sup> Department of Occupational Thearpy, College of S/W Digital Healthcare Convergence, Yonsei University, Wonju, South Korea

jongbae@yonsei.ac.kr

**Abstract.** The wheelchair cushion is one of the intervention methods for preventing pressure ulcers in people with spinal cord injuries. Recently, a customized wheelchair cushion along with off-loading technology that distributes pressure by removing the seat surface in contact with the bony protrusion of the buttocks in the sitting position is attracting attention. In spite of this, they are exposed to the risk of secondary pressure ulcers because they cannot recognize their body pressure distribution and unintended posture changes due to sensory dysfunction. Accordingly, we developed an ICT-based off-loading cushion that can monitor the pressure distribution and pelvis alignment in a sitting position in real-time. People with spinal cord injuries who participated in the pilot study were satisfied with the device and service, and the ICT-based customized off-loading cushion had a positive psychosocial effect on them. We hope that this will contribute to improving the quality of life by preventing pressure ulcers in people with spinal cord injuries.

**Keywords:** ICT-based off-loading cushion · Pressure ulcers · Spinal cord injury

### **1 Introduction**

Pressure ulcers is local injury that occur to the skin and internal tissues of bony protrusions due to excessive pressure and shear force [1]. It requires lots of cost and time for treatment and negatively affects a wide range of areas such as functional activities, psychosocial factors, participation, and quality of life of people with spinal cord injuries [2, 3].

The wheelchair cushion is representative intervention for preventing pressure ulcers in people with spinal cord injuries. The wheelchair cushion can prevent skin damage by dispersing pressure of buttocks in sitting position [4]. It is largely divided into traditional technologies in the form of air, gel, and foam, and new technologies that combined various materials and off-loading technologies [5].

People with spinal cord injuries are restricted from sensory function below the damaged neurologic segment [6, 7]. So, even though the wheelchair cushion is used, excessive pressure and shear force in the buttocks caused by using powered seating functions and outdoor activities are not recognized by themselves. Due to this, people with spinal cord disabilities are exposed risk of secondary pressure ulcers.

Therefore, this study aims to develop an ICT-based system that can monitor pressure distribution and posture changes in real-time when people with spinal cord disabilities use wheelchairs, and to investigate their satisfaction and psychosocial effects.

### **2 Method**

#### **2.1 Participants**

The participants of this study were three people with spinal cord injuries living in the community. All of the participants were male, and the average age was 43.7 years. They are complete cervical level injuries (C3,4 and 6) and the average duration of the onset is 15.3 years. Two participants use power wheelchairs and the other uses manual wheelchair. Their average daily use of wheelchairs is 8.3 h (Table 1).


**Table 1.** Characteristics of participants

#### **2.2 ICT-Based Customized Off-Loading Cushion**

#### **2.2.1 Customized Off-Loading Cushion**

The off-loading cushion is a wheelchair cushion with off-loading technology to prevent pressure ulcers. Off-loading technology is distributing the pressure by removing the seat surface and adjusting the height in contact with the main bony protrusion of the buttocks (Ischial Tuberosity, Coccyx and Greater Trochanter) in a sitting position [8]. It is contoured to fit the shape of the buttocks, so it is possible to maintain stability when seated in a wheelchair and minimize shear force (Fig. 1).

The off-loading cushion was customized through clinical and manufacturing phase. In clinical phase, occupational therapist evaluates the clinical feature and the wheelchair

**Fig. 1.** Off-loading cushion

being used. After the evaluation, we captured and scanned a sitting posture using a vacuum bag and 3D scanner. Next, we performed 3D modeling through CAD (Computer Aided Design) programs to fit each user's buttocks. And we adjusted the seat surface to minimize the pressure of the buttocks based on the result of clinical evaluation. In manufacturing phase, we performed a simulation process using the CAM (Computer Aided Manufacturing) software. After that, EVA (Ethylene-Vinyl Acetate) foam was processed into a cushion using a CNC (Computer Numerical Control) machine (Fig. 2).

**Fig. 2.** Manufacturing process of customized off-loading cushion

#### **2.2.2 Pressure Distribution and Posture Change Monitoring System**

Monitoring system is divided into hardware and software. The hardware of system consists of sensors, BLE (Bluetooth Low Energy), and a multiplexer. A total of 10 sensors were attached based on the coccyx, greater trochanter, and ischial tuberosity, which are the most common areas for pressure ulcers in sitting position (Fig. 3).

Three sensors were attached 2–3 cm below the coccyx and four on the slope between the ischial tuberosity and the thigh, and one on the ischial tuberosity area to detect the pelvis forward sliding. In addition, two sensors were attached to the greater trochanter

**Fig. 3.** The hardware of monitoring system

to detect the sideways sliding of the pelvis. The hardware was placed on the bottom of the off-loading cushion using the case.

The data input through the sensor is transmitted to the monitoring application, which is the software of the system. The application can be linked with the user's tablet or smartphone. Users can intuitively check the pressure distribution and the alignment of pelvis through changes in pressure values and colors. The degree of abnormal posture can be known through the icons. It also provides immediate feedback on posture status to users by providing notifications through banners on their smartphones or tablets when abnormal posture is detected (Fig. 4).

**Fig. 4.** The software of monitoring system (monitoring app.)

#### **2.3 Study Protocol**

Participants performed activities of daily living and community activities for a week with ICT-base customized off-loading cushions. After that, we were measured satisfaction and psychosocial effect on ICT-based customized off-loading cushions.

#### **2.3.1 Satisfaction**

Satisfaction on ICT-based customized off-loading cushions was measured through K-QUEST 2.0 (Korea-Quebec User Evaluation of Satisfaction Assistive Technology 2.0). This is a 12-item related to the satisfaction on assistive technology device (8 items) and assistive technology service (4 items). Each item is scored on a 5-point Likert scale ranging from 0 to 5, and a higher score means higher satisfaction [9].

#### **2.3.2 Psychosocial Impact**

The psychosocial impact of ICT-based customized off-loading cushions was measured using K-PIADS (Korea-Psychosocial Impact of Assistive Devices Scales). This is a 26 item, self-report questionnaire and consist of three sub-scale: Competence (12 items), Adaptability (6 items), and Self-esteem (8 items). Each item is scored on a 7-point Likert scale ranging from −3 to 3. A positive score mean that assistive technology device has a positive psychosocial impact on user [10].

### **3 Result**

### **3.1 Satisfaction**

The average for the mean score of the assistive technology device was 4.6. The mean score of comfort and effectiveness was the highest, and the mean score of weight and durability was the lowest (Table 2).


**Table 2.** The result of satisfaction on assistive technology device

The average for the mean score of the assistive technology service was 4.5. The mean score of service delivery and professional service was the highest, and the mean score of repair and follow-up service was the lowest (Table 3).


**Table 3.** The result of satisfaction on assistive technology service

#### **3.2 Psychosocial Impact**

The mean scores for competence, adaptability, and self-esteem were 2.39, 2.61, and 2.42, respectively. it means that the ICT-based customized off-loading cushion had a positive to very positive impact on the participants. (Table 4).


**Table 4.** The result of psychosocial impact

### **4 Conclusion**

People with spinal cord injuries suffer from permanent physical and sensory dysfunction due to trauma or disease. Therefore, intervention strategies should focus on participation and improvement of quality of life rather than functional recovery. For a healthy life for people with spinal cord disabilities, it is necessary to actively cope with complications such as pressure ulcers that reduce the quality of life. A wheelchair cushion is a technical intervention to prevent pressure ulcers. In this study, we developed a wheelchair cushion applying customized off-loading technology and ICT technology that can monitor body pressure distribution and posture change. As a result of applying it to three people with spinal cord disabilities living in the local community for a week, satisfaction with the ICT-based off-loading cushion was appropriate, and it had a positive psychosocial effect. Through this pilot study, we derived the need for a covering to protect hardware from urinary incontinence and improvement in applications such as colors that can intuitively recognize pressure distribution during outdoor activities. Currently, we are supplementing software and hardware based on the results derived. In the future, we intend to investigate the effects of ICT-based off-loading cushions on the clinical effectiveness, quality of life and participation through a randomized controlled trial study for people with spinal cord disabilities living in local communities. This study has a limitation in that it was conducted with only three male subjects. Therefore, we intend to include balanced gender subjects with an appropriate number of sampling to test for statistical significance in future study. We expect this to be useful in improving the quality of life for people with spinal cord injuries as a health care system.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Autism Spectrum Disorder (ASD) Detection Using Machine Learning Algorithms**

Naouel Boughattas(B) and Hanen Jabnoun(B)

ESPRIT, ESPRIT School of Engineering, Tunis, Tunisia {naouel.boughattas,hanene.jabnoun}@esprit.tn

**Abstract.** Some diseases are characterized by persistent deficits in brain activity. Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder. It appears in early childhood and evolves throughout life and needs to be detected early to accelerate the treatment and recovery process. These deficits may be detected using medical imaging techniques. In this paper, we present machine learning algorithms allowing to detect peoples with ASD from normal peoples. We used data from the ABIDE dataset. We tested 3 algorithms: Support Vector Machines (SVM), Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN). The best result was obtained using CNN algorithm with an accuracy equal to 95%.

**Keywords:** Autism · Machine learning · CNN · LSTM · SVM · ABIDE · fMRI

### **1 Introduction**

In the recent decades, the medicine industry has adopted numerous approaches and methods to discover or predict many diseases [1, 2]. In this context, researches are focusing on developing machine learning algorithms in order to improve their accuracy [3].

Brain diseases such as Autism Spectrum Disorder (ASD) have been of increasing interest to researchers over the past few years. ASD is known as a cerebral disease that involves impairments in cognitive functions, communication/social interactive, cognitive and adaptive skills. Studies from the neuroscience domain indicate that the biomarkers of ASD are still unknown however the corpus callosum and intracranial brain volume holds significant information for its detection. Based on these conclusions, we suggested machine learning models for automatic ASD detection. The proposed algorithms were tested and evaluated on the ABIDE<sup>1</sup> dataset. Since autism is a brain dysfunction disorder, we will use functional Magnetic Resonance Images (fMRI) which better describe this disorder.

In this paper three algorithms were presented: Support Vector Machine (SVM), Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN). The comparison of results showed that CNN performs better than SVM and LSTM with an accuracy equal to 95%.

<sup>1</sup> ABIDE dataset: https://fcon\_1000.projects.nitrc.org/indi/abide/.

<sup>©</sup> The Author(s) 2022

H. Aloulou et al. (Eds.): ICOST 2022, LNCS 13287, pp. 225–233, 2022. https://doi.org/10.1007/978-3-031-09593-1\_18

#### **2 State of Art**

ASD is a brain-based disorder marked by social deficits and repetitive behaviors. The main aim of psychiatric Neuroimaging research is to identify objective bio-markers that may inform the diagnosis and treatment of brain-based disorders. ASD is associated with a range of phenotype that vary in severity of social, communicative and sensory motor deficits. Recently, Machine Learning (ML) algorithms have been applied to brain imaging data to extract brain function patterns. These algorithms can retrieve robust neural patterns from brain imaging data of psychiatric disorder patients.

In their work, Omar et al. [4] proposed an effective prediction model based on ML technics in order to develop a mobile application for predicting ASD for people of any age. The model was developed by merging Random Forest-CART and Random Forest-Iterative Dichotomiser3. The proposed model was evaluated with AQ-10 dataset and 250 real dataset collected from people with and without autistic traits [4].

In another work, Kuper et al. [5] focused on adolescents and adults. They used SVM to examine whether ASD detection can be improved by identifying a subset of behavioral features from the ADOS Module 4 in a routine clinical sample. They identified reduced subsets of 5 behavioral features for the whole sample as well as age subgroups. The results of evaluation may help to improve the complicated diagnostic process of ASD by encouraging future efforts to develop novel diagnostic instruments for ASD detection based on the identified constructs as well as aiding clinicians in the difficult question of differential diagnosis [5]. In the work of [6], authors explore the possibility to test Naïve Bayes, Support Vector Machine, Logistic Regression, KNN, Neural Network and Convolutional Neural Network for predicting and analysis of ASD problems. They evaluate the proposed methods on publicly available three different non-clinically ASD datasets for children, adolescents and adults. The results suggest that CNN based prediction models work better on all these datasets.

Knowing that the Functional Magnetic Resonance Imaging (fMRI) helped to identify and detect the ASD, Dvornek et al. [7] proposed a model based on Recurrent Neural Networks with Long Short-Term Memory for classification of individuals with ASD and typical controls from the resting-state fMRI time-series. They used the ABIDE-I dataset for training and testing the LSTM models. Results with a cross-validation framework showed an accuracy of 68.5% on the whole ABIDE cohort.

The primary goal of the current study is to classify ASD patients and control participants based on their neural patterns of functional connectivity using resting state functional magnetic resonance imaging (rs-fMRI) data. Supervised ML algorithms were used and applied to a large population sample of brain imaging data extracted from the ABIDE dataset.

### **3 Methods**

#### **3.1 Convolutional Neural Network (CNN)**

The Convolutional Neural Network (CNN) is a deep learning neural network that has shown excellent performance in many computer vision and machine learning problems. It is designed to learn automatically spatial hierarchies of features through backpropagation. It uses the multiple building blocks, such as the convolution layers, the pooling layers and the fully connected layers [8]:

**Convolution Layer.** This layer is a fundamental component of the CNN architecture. It performs feature extraction using a combination of linear and nonlinear operations, i.e. convolution operation and activation function.

**Pooling Layer.** This layer provides a down sampling operation which reduces the dimensional of the feature maps and identify invariance to translation, shift and distortions. The filter size, stride and padding are the hyper-parameters in pooling operations, similar to convolution operations.

**Fully Connected Layer.** The features extracted by the convolution layers and then downsampled by the pooling layers are mapped using a subset of fully connected layers to the final outputs of the network. The fully connected layer is followed by a nonlinear function, such as ReLU. When the input data are transformed into output through the different layers, it is called the forward propagation. We build a CNN network with 8 convolution layers, 4 pooling layers and a fully connected layer.

### **3.2 Long Short Term Memory (LSTM)**

Long Short Term Memory (LSTM) network is a special kind of Recurrent Neural Networks that is capable of learning long-term dependencies. LSTM can be used to model univariate time series forecasting problems. These problems are comprised of a single series of observations and a model is required to learn from the series of past observations to predict the next value in the sequence. The LSTM model learn a function that maps a sequence of past observations as input to an output observation. As such, the sequence of observations must be transformed into multiple examples from which the LSTM can learn [7].

In our paper, we extracted mean time-series from regions of interest defined by several atlases. Each time course was normalized to represent percent change from the average signal for that region of interest.

### **3.3 Support Vector Machines (SVM)**

Support Vector Machine (SVM) is a classical binary classification algorithm, particularly adapted for big data. The SVM method was proposed by "Vapnik" [9] and has been widely used for medical image processing.

Given a set of training examples features {xi} l. <sup>i</sup>=1, and their labels {yi} l. <sup>i</sup>=1, a Support Vector Machine (SVM) builds a prediction model that assigns new examples into one class or the other depending on the side of the learned separating hyperplan on which they stand.

The decision function of the SVM is given in (1) where k is a given positive kernel.

$$f(\mathbf{x}) = \sum\_{i=1}^{l} \alpha\_i^\* k(\mathbf{x}, \mathbf{x}\_i). \tag{1}$$

The dual learning optimization problem form is given in (2). C>= 0 is the regularization parameter related to the misclassified samples.

$$\begin{aligned} \max\_{\alpha\_l} & \sum\_{l=1}^{l} \alpha\_l - \frac{1}{2} \sum\_{l,j=1}^{l} \alpha\_l \alpha\_j \mathbf{y}\_l \mathbf{y}\_j k(\mathbf{x}\_l, \mathbf{x}\_j),\\ & \text{s.t.} \quad 0 \le \alpha\_l \le C \end{aligned} \tag{2}$$

The linear kernel was used and the one-vs-the-rest scheme was adopted in our work. For dimensionality reduction we used PCA to keep 99% of variance then we transformed the phenotypic data into a dataset and determined the target.

#### **4 Material and Results**

#### **4.1 Material**

This study is performed using the ABIDE dataset which is an online sharing consortium that provides imaging data of ASD and control participants with their phenotypic information. The ABIDE dataset includes two large-scale collections: ABIDE I and ABIDE II. ABIDE I includes 1112 subjects aged between 7 and 64: 539 subjects with ASD and 573 subjects with typical controls.

The ABIDE dataset includes functional and structural brain imaging data collected from laboratories around the world to ensure data diversity and to understand the complexity of the disease to be able to make the diagnosis at earlier ages, to select optimal treatments and to predict outcomes.

Structural imaging [9] is used to visualize and analyze anatomical properties of the brain and are particularly useful for detecting brain damage and abnormalities. Functional imaging [9–11] is used to identify brain areas and underlying brain processes that are associated with performing a particular cognitive or behavioral task. It can be used to investigate the functional anatomy of the brain by identifying the parts of the brain with critical functions, to evaluate the effect of many diseases such as stroke and to brain treatment.

Our imaging data is composed of functional Magnetic Resonance Imaging from ABIDE I that allows a better description of the brain-related dysfunction. We extracted features using each correlation matrix extracted from the dataset. For dimensionality reduction, we used a PCA to keep 99% of variance.

An example of F-MRI images is shown in Fig. 1.

**Fig. 1.** Example of F-MRI images

Our proposed algorithms have been tested on a dataset of 500 patients. 51.2% of the base are subjects with ASD while 48.8% are subjects with typical controls (Fig. 2). We can say that our dataset is balanced. We assign 70% of our modeling dataset to the train and the remaining 30% to the test.

**Fig. 2.** Diagram of ASD distribution in the ABIDE dataset

For gender distribution (Fig. 3), we can see that our dataset is mainly composed of males with a rate = 84%. To understand the reason behind this, we need to know that ASD affects females less frequently than males. Of course several sex-differential genetic and hormonal factors may contribute in the appearance and the development of this disease.

While visualizing the age distribution (Fig. 4), it's clear that the most of patients are very young (aged from 5 to 15 years). This may be justified by the fact that this disease is more frequent among children than among adults and emphasizes the importance of early ASD diagnosis for children to accelerate treatment and recovery.

**Fig. 3.** Diagram of gender distribution in the ABIDE dataset

**Fig. 4.** Age distribution in ABIDE dataset

#### **4.2 Results and Discussion**

The performance of the proposed methods has been evaluated using the accuracy score.

**LSTM.** We used the binary cross-entropy loss function and the Adadelta optimizer with the default parameter values. During the training, we fixed the dropout rate during training to 0.4. The impact of parameters and variations of the proposed architecture were explored as well as training conditions. We tested the data while varying the number of hidden nodes (8, 16, 32, or 64) in the LSTM, and removing dropout. We also tested variations on the base network: connecting only the final LSTM cell's output to a single dense node and stacking LSTM layers. The variation of Loss and Accuracy rates are presented in Fig. 5. We got an accuracy equal to 76.25%.

**CNN.** CNN is typically a repetitions of a stack of several convolution layers and pooling layers, followed by one or more fully connected layers. The CNN that we have worked

**Fig. 5.** Accuracy and loss depending on the number of epoch for LSTM model

on has 8 convolutional layers, 4 max pooling layers and one sigmoid output layer. The input consists of three (316, 70, 1) patches from axial, sagittal and coronal image slices centered on the target voxel. We used Adam as optimizer algorithm. We fit the model and set the number of epochs at 60, so we got an accuracy value equal to 95% (Fig. 6).

**Fig. 6.** Accuracy and loss depending on the number of epoch for CNN model

**SVM.** The obtained results show good performance of our SVM based method with an accuracy rate equal to 88%. The feature reduction step based on PCA allowed to keep 99% of variance. Our results exceed those of Eslami et al. [12] who combined SVM with a deep learning algorithm. We can say also that our SVM based method performs better than the method presented in [13] for male subjects since our base is male oriented. The Linear SVM method presented in [14] reaches an accuracy equal to 69% and the Gaussian SVM method presented in [15] reached 66%. These two methods are less efficient than our method.

In the Table 1, we summarize the obtained results for the trained algorithms. We conclude that CNN gives the best accuracy rate.


**Table 1.** Accuracy rate of ASD detection in ABIDE dataset using different algorithms.

### **5 Conclusion**

ASD is a brain disease that it is notoriously difficult to diagnose, especially for children. It is associated with significant brain structure changes, which can be measured by magnetic resonance imaging (MRI) scan.

In this paper, we presented three algorithms to generate a model for ASD detection using image classification tools: Convolutional Neural Network, Support Vector Machines and Long Short Term Memory.

These methods for automatic ASD detection were tested on the ABIDE dataset. The performance of algorithms was evaluated throw the accuracy. The best result was obtained using the CNN algorithm with a rate equal to 95%.

In order to achieve better recognition of ASD, other modalities i.e. EEG, speech or kinesthetic modalities can be analyzed simultaneously.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Ant Colony Optimization with BrainSeg3D Protocol for Multiple Sclerosis Lesion Detection**

Dalenda Bouzidi1,3(B) , Fahmi Ghozzi2,3(B) , and Ahmed Fakhfakh2,3(B)

<sup>1</sup> National Engineering School of Sfax (ENIS), Sfax University, Sfax, Tunisia dalendabouzidi91@gmail.com

<sup>2</sup> National School of Electronics and Telecommunications of Sfax (ENET'COM), Sfax University, Sfax, Tunisia

{fahmi.ghozzi,ahmed.fakhfakh}@enetcom.usf.tn

<sup>3</sup> Digital Research Center of Sfax (CRNS), Laboratory of Signals, Systems, Artificial Intelligence and Networks (SM@RTS), Sfax, Tunisia

**Abstract.** Magnetic resonance imaging (MRI) has quickly established itself as the reference imaging tool for the management of patients suffering from multiple sclerosis (MS), both for the diagnosis and the follow-up of the evolution and evaluation of the impact of new therapies.

The treatment of multiple sclerosis does not cure the disease, but it slows its progression and can help to space out attacks. In this paper, tumor segmentation is treated as a problem of classification using the Ant Colony optimization algorithm (ACO) combined with a proposed protocol based on BrainSeg3D tools. Many studies and many existing approaches tend the multiple sclerosis (MS) which is a chronic inflammatory anomaly of the central nervous system.

The aim of this work is to evaluate and to verify the effectiveness of the proposed protocol on a public longitudinal database which contains 20 MS patients. This study is concerned with comparing these results against the ground truth performed by two experts and against other methods namely Dissimilarity Map (DM) creation and segmentation in terms of Dice Similarity Coefficient (DSC).

**Keywords:** Multiple sclerosis · Segmentation · ACO\_BrainSeg3D · Longitudinal dataset · Ground truth

### **1 Introduction**

During the research work, we focused on brain MRI analysis, particularly in the context of monitoring the condition of patients with multiple sclerosis. MRI is considered one of the complementary examinations in the diagnostic approach of this disease since it plays a key role in monitoring the evolutionary state of the disease in patients.

The major problem lies in the difficulty of analyzing and reading these MRI images because of the variability in the size, the contrast and the difficulty of locating the lesions, and the automatic segmentation of the lesions. Several recent studies have worked on the automatic segmentation of MS lesions based on several approaches to follow the evolutionary state of this disease which can provide information on therapeutic efficacy. To detect lesions from an MRI image that carries this disease, it is necessary to adopt an efficient algorithm that can easily and efficiently detect the lesions.

To solve the problem of image segmentation and in particular to improve the performance of segmentation methods by applying them for different types of images. The objectives we have set ourselves are to contribute to the development and optimization of new segmentation criteria. We adapted the metaheuristic methods by simultaneously applying several criteria to segment the images. As we have created a new protocol based on filters and automatic tools that allow us to facilitate the task of segmentation and detection of MS lesions. Our motivation is to detect MS lesions for different patients included in novel longitudinal MRI datasets.

The residual of this study is orderly as follows: we briefly present in Sect. 2 some previous works; in Sect. 3 presents the proposed algorithm ACO\_BrainSeg3D. Then we will detail the public longitudinal database we used for the test and the validation of our algorithm. Then Sect. 4 presents some results and discussion. Finally, we would end up with a conclusion and submit some perspectives.

#### **2 Previous Works**

The segmentation techniques can be classified according to different criteria and they can be presented according to a classical methodological angle. These approaches are then grouped into parametric, non-parametric and geometric approaches. Other states of the art present the different techniques by defining the classical approaches among recent approaches.

For the past twenty years, and despite the appearance of other new, more sensitive and more advanced techniques, nuclear magnetic resonance imaging has presented the tool frequently used for the diagnosis of MS since it makes it possible to detect this pathology at an earlier stage than in the past. To solve the posed problems for diagnosis, segmentation of MRI images is posed as an important task for detecting MS lesions.

In parallel with our research work for segmentation and the detection of SEP lesions, there are other similar works in progress essentially for the detection and quantification of the atrophy of the spinal cord SC in the MRI images are the key to the key. In 2018, [1] presented a fully automatic method "OPTIC" to detect the median cord line in the MRI volumes and locate the Spinal Cord (SC) in the MRI images. The authors have validated their method in healthy subjects and patients with very variable spatial resolutions and including some artifacts. The interest of this method is to detect the median line of the SC that can be used as initialization for other SC segmentation methods for pathological cases. In 2019, the authors [2] develop a fully automatic framework for segmenting SC and intra medullary lesions on a variety of MRI contrasts. The proposed approach is based on a sequence of two convening neuron networks (CNN):


In the same year, McCoy [3] used 2D convoluting neuron networks for the automatic segmentation of the SC and traumatic axial MRI contusion in T2 in a cohort of patients with acute SC injury. They have developed a image analysis pipeline integrating 2D convolutionary neuron networks for the segmentation of the entire and intramedullary spinal cord lesions.

In Article of [4] to segment the lesions in the pathology of multiple sclerosis, they studied two stages of pretreatment based on skull stripping (SS) and the improvement of the contrast which are two important steps to improve the quality rate of MS lesion segmentation. After the pretreatment step, a segmentation approach based on the method of maximizing expectations (EM) has been applied to extract SEP lesions. Again study of [5] for the contrast improvement of MRI images for a better appearance of normal tissues and sick tissues affected by the pathology of MS.

The important recent segmentation study presents a new protocol to develop benchmark segmentations of white matter tumors based on multi-rater consensus using a new MR database of 30 patients with multiple sclerosis [6]. They have developed advanced BrainSeg3Dsoftware which allows their correct and effective delineation in 3D MR images.

From this study, we carried out work based on the tools of the BrainSeg3D software and this new database: the first work is to create a novel protocol applied the tools of the BrainSeg3D software applied on this new database which contains 30 MS patients.

The new database is tested by the metaheuristic ant colony algorithm used by [7, 8] helping to improve the parameters of this algorithm [9, 10].

The research [10] manages to realize a new ACO\_BrainSeg3D segmentation algorithm that combines the ACO algorithm and the protocol based on BrainSeg3D tools, this approach is tested on the novel longitudinal dataset with reference.

Another recent study [11] objectively validate and compare several intensity-based methods for detecting white matter tumor changes using longitudinal MR image datasets with accurate ground truth.

The Validation of change detection focused on evaluating the two main steps of change detection, namely DissimilarityMap (DM) creation and segmentation. Therefore, combinations of three methods for DM creation (Subtraction of intensity images (STI), Generalized Like Lihood Ratio (GLR) and Logistic Regression Model (LRM)) and three for DM segmentation (Confidence level thresholding (CLT), Change vector angular histogram thresholding (CVAHT) and optimal threshold) were tested on 20 MR image datasets with a common pre-processing and post-processing.

It was created by two specialist evaluators who subtracted and segmented of white matter tumor of two studies: baseline and follow-up T1w, T2w and FLAIR MR Images. The reference segmentations were cooperatively treated by the evaluators till they heard on what was inspected the reference segmentation. This publicly dataset and the BrainSeg3D software are available on site web http://lit.fe.uni-lj.si/tools.

In this study, the objective is to employ the proposed algorithm ACO\_BrainSeg3D on a new longitudinal dataset containing 20 MS patients in order to evaluate the performance and reliability of this algorithm against the ground truth and other tested methods on the same database.

The aim of the actual work was to supply a literal test and to compare the results of propound algorithm with the three intensity based approaches Dissimilarity Map creation and segmentation combinations (STI/CLT, STI/CVAHT and STI/Optimal threshold), (GLR/CLT, GLR/CVAHT and GLR/Optimal threshold) and (LRM/CLT, LRM/CVAHT and LRM/Optimal threshold) [11].

We can thus note that almost all the outliers are identified by the ACO\_BrainSeg3D protocol in front of the reference, and compared to three methods for DM creation (STI, GLR and LRM) combined with three for DM segmentation (CLT, CVAHT and optimal threshold) in terms of the median coefficient of similarity.

We have tested the proposed algorithm using public longitudinal available dataset with brain MR Images containing two studies: baseline and Follow-up studies with three MRI sequences T1-w, T2-w and Flair. In all cases, our algorithm outperformed competing approaches. The obtained evaluations reported good results, where all the outliers have emerged in spite of there being falsely spotted outliers.

#### **3 The Proposed ACO\_BrainSeg3D Algorithm**

In this paper we expose the suggested method which used the combination of ACO algorithm with a propound protocol by using existing and new semi-automated outliers segmentation tools of BrainSeg3D software. So as to facilitate the tumors segmentation, we employed the MATLAB software and the specialized BrainSeg3D software. Our proposed contribution is named ACO\_BrainSeg3D.

The global function of the propound contribution is detailed on Fig. 1.

The proposed algorithm is essentially based on two algorithms as detailed below:

All MRI images from the database used are first processed by the ACO metaheuristic algorithm. Also in the studies [9, 12, 13] which are based on ACO algorithm with the same database, when we obtained good results compared to reference of the novel MR Datasets.

Concerning this algorithm, our contribution is mainly focused on the variation of the parameters empirically in order to have good results for the segmentation of the images of the new longitudinal database and a good detection of MS lesions. The latter consists of three parameters to better characterize good quality segmentation. Then, the segmentation thresholds maximizing the criteria are sought using the ACO algorithm.

The settings and the essential equations of ACO algorithm are detailed in our studies [9, 10]. Then the major tested values to do all simulations are presented in table 1.

Our important contribution is the testing of parameters in order to obtain good results. This task is done in [12, 13] and it evaluated according to existing research and the consensus of segmentation.

#### **4 Validation Dataset**

The validated dataset contains two studies baseline and follow-up MR images for 20 subjects with multiple sclerosis as illustrated in Fig. 2.

The images were acquired on a 1.5 T Philips MRI machine at the University Medical Centre Ljubljana (UMCL).


**Table 1.** ACO algorithm setting.

**Fig. 1.** Workflow of ACO\_BrainSeg3D.

**Fig. 2.** Longitudinal database (a) patient 1 to patient 6, (b) patient 7 to patient 12, (c) patient 13 to patient 18 (d) patient 19 to patient 20.

Every subjects of this dataset contains a 2D T1-weighted, 2D T2-weighted and a 2D FLAIR image with different setting indicated in the Table 2.

The reference of MR images was proposed by two expert evaluators for evaluation purposes. The 20 MS patients with baseline and follow-up studies with respectively T1-W, T2-W and Flair images are available in the site http://lit.fe.uni-lj.si/tools.


**Table 2.** MRI image of database parameters.

#### **5 Results and Discussion**

In order to validate the capability of our algorithm to detect MS tumor, we calculated three essential terms of validation: the coefficient DSC, TPR (True positive Rate) and robustness of FPR (False Positive Rate).

The True Positive Rate corresponds to sensitivity:

$$\text{True Positive Rate} = \text{TP}/(\text{TP} + \text{FN}) \tag{1}$$

The False Positive Rate corresponds to precision that explains the capability to select all existing outliers corresponds to the value:

$$\text{False Positive Rate} = \text{TP/(TP} + \text{FP)}\tag{2}$$

The Dice Similarity Coefficient also corresponds to the Kappa Index (or Dice coefficient) and measures the overlap between the test of segmentation obtained and the consensus. The Dice Similarity Coefficient (DSC) [15] can therefore be calculated with respect to consensus segmentation:

$$\text{DSC} = (\text{2} \ast \text{TP})/((\text{2} \ast \text{TP}) + \text{FP} + \text{FN})\tag{3}$$

Our important goal is to detect all outliers, True positive (TP) presents a detected region if at least one voxel overlaps the reference, otherwise it is considered a false positive (FP). Any labeling not spotted by our approach is considered as a false negative (FN) [16]. Whose values fluctuate in [0, 1] and a value superior 0.6 presented a good conformity with the ground truth.

We see here in Fig. 5 that the proposed algorithm manages to detect almost all the lesions in the axial slices of the original image as for patients 1, 3, 6, 8 and 11. For patients 7 and 19, there are slightly visible lesions not detected. And for the other patients, they

**Fig. 3.** Minimum and maximum values of DSC, FPR and TPR of longitudinal database.

are considered as healthy patients due to the absence of lesions; we can only discover this for a single patient 17 in which the proposed algorithm detects a small, slightly visible lesion.

The results of segmentation by the suggested algorithmACO\_BrainSeg3D are exposed in Fig. 5. This algorithm yields good results, where all the outliers have arisen in spite of there being falsely spotted lesions.

The respective values procured with our proposed ACO\_BrainSeg3D algorithm of DSC are between 0.44 and 1, the values of FPR are in the interval [0.5 1] and the values of TPR are between 0.33 and 1(Fig. 3). We mark that the validation values obtained in this test are lightly elevated than these reported on every outliers segmentation publicized in the literature [11].

We managed in this paper to detect several lesions according to the number and volume in the first phase of advancement of the lesions it is the relapsing remittent form RR for all the patients of the database with a percentage of 70% of rate of advancement, as we detected lesions in the secondary progressive form SP with a portion of 5% and a percentage of 25% of other patients not specified or we cannot identify whether these patients are really healthy or there are very small lesions in the early phase that we couldn't detect (Fig. 4).

This analysis has been processed for some patients in the database for example for the axial sections of the flair image for patients 2, 4, 5, 20 … as illustrated in Fig. 5.

This is due to the fact that we can see the positions of further outliers in link with the consensus, and we can spot the missed outliers, which could otherwise have a strong impact on DSC.

We objectively evaluated in this paper the proposed algorithm which combine ACO algorithm with the protocol using the BrainSeg3D tools on longitudinal MRI database of 20 MS subjects.

The aim of the actual work was to supply a literal test and to compare the results of propound algorithm with the three intensity based approaches Dissimilarity Map creation and segmentation combinations (STI/CLT, STI/CVAHT and STI/Optimal threshold), (GLR/CLT, GLR/CVAHT and GLR/Optimal threshold) and (LRM/CLT, LRM/CVAHT and LRM/Optimal threshold).

**Fig. 4.** Advancement rate of 20 patients in the database.

To explain the impact on the median DSC, Table 3 reports corresponding median values of the proposed algorithm and the combinations methods based on segmentation and creation DM after the post-processing.

The median DSC after post-processing of three combinations of three change segmentations DM and three creation DM approaches was consistently reduced in accordance of our suggested algorithm where the value DSC is 0.76.

The middle DSC value for the ACO\_BrainSeg3D algorithm is studied as follows:

$$\text{DSCmoy} = 1/20 \sum\_{i=1}^{20} \text{DSCi} \tag{4}$$

Aside from various fusions of the DM creation and segmentation approaches were tested by using the same quantitative metrics. Comparative test according to three DM alternatives (Table 3) illustrated an elevated average DSC (76%).

This result reports that the used outliers segmentation algorithm marked very substantial and reliable tumor segmentations.

That our segmentation algorithm of segmentation applied on the novel longitudinal database is significantly higher compared to the DSC of DM creation combined with three DM segmentation. So the ACO\_BrainSeg3D algorithm detects more lesions compared to ground truth.

**Fig. 5.** Axial sections of corresponding FLAIR images of 20 patients (1st column), the results of our proposed Algorithm (2nd column) and the ground truth segmentation (3rd column) for each block. (a): patients 1 to 7; (b): patients 8 to 14 and (c): patients 15 to 20.


#### **Table 3.** Values of median DSC

The advantage of proposed algorithm ACO\_BrainSeg3D applied on this longitudinal database is the detection of almost all existing lesions in the ground truth thanks to the values carried out within the acceptable measurement interval. Our algorithm needs more run time but we don't have a real time constraint. The only drawback is the non-detection of small lesions which are not very visible, which encourages the researchers to improve their techniques until they detect small lesions of a small volume.

We can thus note that almost all the outliers are identified by the ACO\_BrainSeg3D protocol in front of the reference, and compared to three methods for DM creation (STI, GLR and LRM) combined with three for DM segmentation (CLT, CVAHT and optimal threshold) in terms of the median coefficient of similarity.

This validation is carried out first based on the DSC coefficient largely which indicates similarity and second concerning the ability and precision to detect the number of tumors existing in the ground truth based on the TPR and the FPR.

In order to mark this object and allow other researchers to replicate the test of this work, we wish to extend and publicly propagate the encouraging results of our proposed algorithm using the longitudinal database available on the site: http://lit.fe.uni-lj.si/tools.

We consider this study as the first but important step towards the realization of a protocol based on BrainSeg3D software tools combined with an improved metaheuristic algorithm applied to different public databases for validation and comparison of detection methods with longitudinal and other changes.

#### **6 Conclusion**

In this paper, we propounded an automatic protocol for cerebral tumors segmentation in longitudinal MRI images. The idea was evaluated the ACO algorithm by using the MAT-LAB software and protocol based on semi-automated tools using BrainSeg3Dsoftware with a for longitudinal MRI image dataset which contains 20 patients with multiple sclerosis with reference agreed by two domain specialist.

We have tested the proposed algorithm using public longitudinal available dataset with brain MR Images containing two studies: baseline and Follow-up studies with three MRI sequences T1-w, T2-w and Flair. In all cases, our algorithm outperformed competing approaches. The obtained evaluations reported good results, where all the outliers have emerged in spite of there being falsely spotted outliers.

A short-term perspective is to make an interface that implements our proposed algorithm ACO\_Brainseg3D on computer. This can facilitate the use of the method and gain more time. It is necessary to test this interface on larger databases to clear the strengths and limitations. This step is essential before exporting our algorithm in a clinical context.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **A Systematic Review on the Development of Clothing for People with Disability in Korea**

Ha-yeon Yang1, Hyun-kyung Kim1, Min-kyung Kim2, Sun-hwa Shim2, Eun-ju Kim2, Jae-nam Kim3, Sun-young Jo4, and Wan-ho Jang2(B)

<sup>1</sup> Department of Rehabilitation Science, Jeonju University, Jeonju 55069, South Korea <sup>2</sup> Department of Occupational Therapy, Jeonju University, Jeonju 55069, South Korea otjang@jj.ac.kr

<sup>3</sup> Health Science Research Institute, Jeonju University, Jeonju 55069, South Korea <sup>4</sup> Department of Rehabilitation Medicine, Jeonbuk National University Hospital, Jeonju 54907, South Korea

**Abstract.** As the number of people with disabilities continues to increase, difficulties in dressing activities for people with disabilities are also increasing. However, the currently available clothing for the disabled do not satisfy their functional and aesthetic needs. Therefore, this study aims to contribute to vitalizing the development of clothing for the disabled in Korea by collecting domestic study on the development of clothing for the disabled and analyzing them in various areas. Finally, 5 studies were selected. As a result of classifying into 4 areas of body type, considerations, points for improvement, and design, it can be used as basic data for study related to the development of clothing for the disabled.

**Keywords:** People with disability · Wheelchair users · Clothes · Development · Design

### **1 Introduction**

According to the Ministry of Health and Welfare data, as of the end of 2021, the number of registered disabled people was 2.645,000 (5.1% of the total population), an increase of 12,000 compared to the end of last year. As the number of people with disabilities continues to increase in Korea, the difficulty of wearing clothes for the disabled is also increasing [1]. Clothing for the disabled is clothing that helps disabled people to wear clothes conveniently [2]. For example, it should be able to move more freely when pushing a wheelchair, and comfort should be ensured even when sitting for long periods of time while wearing clothing [3]. In addition, it is necessary to develop clothes that allow people with disabilities to wear clothes with a minimum of help from others, and to minimize the inconvenience of putting on and taking off themselves, so that the inconvenience of clothes felt by people with disabilities should be resolved. However, the currently available clothes for the disabled do not satisfy their functional needs as well as their aesthetic satisfaction [4]. And compared to developed countries, research and development related to clothing for the disabled in Korea is very insufficient, and the distribution of clothing considering the inconveniences of the disabled is also very limited [5].

Disabled people who use wheelchairs spend most of their time in a seated position in a wheelchair as they become dependent on a wheelchair due to lower extremity amputation or paraplegia due to congenital or acquired diseases and accidents [6]. Because of this, physical changes appear, and it is difficult for the disabled to purchase clothes suitable for their body type, so there is a limitation in that the types of clothes that are purchased and worn are items that are convenient to put on and take off [7]. In addition, people with disabilities in wheelchairs want to increase their opportunities for activities and social participation by wearing clothes that are indistinguishable from non-disabled people in appearance because their body image is distorted or their self-esteem is relatively low [8].

Therefore, this study aims to contribute to vitalizing the development of clothing for the disabled in Korea by collecting domestic study on the development of clothing for the disabled and analyzing them in various areas.

### **2 Method**

#### **2.1 Collecting Data**

This study was conducted as a literature search method. In order to investigate studies related to the development of clothing for the disabled in Korea, the DBpia, KISS, RISS, and KCI databases were used to collect documents published in domestic journals. Keywords used in the search included search terms such as 'disabled person', 'wheelchair user', 'clothes', 'development', and 'design'. In addition, missing articles were included by re-searching the references of the collected articles.

**Fig. 1.** Study process

#### **2.2 Data Analysis**

A total of 36 researches were published from 2012 to December 2021, and DBpia(4), KISS(11), RISS(14), and KCI(7) were searched for collected through literature search. After that, five studies were finally selected through a study process (Fig. 1). Analysis standards and methods were configured to be suitable for consideration on research on the development of clothes for the disabled in Korea by identifying the contents of the collected studies. The research subjects were divided into four areas: body shape, considerations, points for improvement, and design. In order to understand the contents of each item, each research study was read and detailed contents were grasped.

### **3 Results**

#### **3.1 Year of Publication**

There are a total of 5 studies on research related to the development of clothes for the disabled in Korea published from January 2012 to December 2022, 3 in 2013 and 2 in 2014.

#### **3.2 Subjects**

The age of subjects related to the development of clothes for the disabled in Korea was 40–49 years old the most at 43%, followed by 30–39 years old, 50–59 years old, 20–29 years old, and 60 years old or older.

Looking at the types of disabilities of the subjects, spinal cord injury accounted for the most at 48%, followed by amputation, polio, and brain lesion disorders.

The subjects' use of assistive devices was frequent use of wheelchairs the most at 44%, followed by walking aids, crutches, no use, and canes.

#### **3.3 Clothing Body Type**

The body type of the wheelchair disabled person's clothing is largely divided into 4 in the upper extremity and 2 in the lower extremity. The upper extremity body type is divided into a normal body type (A), a thick waist type (B), a large belly type (BB), and an inverted triangle (Y). And the lower extremity body type is divided into a normal body type (A), a thick waist type (B). The distribution according to the subject's clothing body type is shown in Figs. 2 and 3. Looking at the clothing body type, the distribution of upper extremities was the most common a normal body type (A) with 34%, followed by a body with a large belly (BB), a body with a thick waist (B), and an inverted triangle (Y). And the distribution of the lower extremity body type was 70% for those with a thick waist (B), more than those for the normal body type (A).

This result can be explained by the fact that the upper extremity exercise increases because the upper extremities must be continuously pushed while moving the wheelchair, and the lower extremity exercise decreases by relying on the wheelchair.

**Fig. 3.** Lower extremity body type

#### **3.4 Considerations When Purchasing Clothing**

Considerations when purchasing clothes are divided into aspects of functionality and practicality and aesthetic. Table 1 shows the distribution of considerations when purchasing clothes for subjects. First, in terms of functionality and practicality, it appears that 'very much considered' considers the convenience of putting on and taking off, which is the most distributed. Next, 'very much considered' and 'much considered' were widely distributed in the order of dimensional conformity, usability of clothes, and ease of repair. It can be seen that the low price is not a major consideration when purchasing clothes, as it is widely distributed in 'much considered' and 'so-so considered'.

In terms of esthetic, it is shown that 'very much considered' considers the coverage of the disability area, which is the most distributed. Next was the palatability in which a 'very much consideration' and 'much consideration' were widely distributed. Fashion trends are widely distributed in 'so-so considered', so it can be seen that this is not a major consideration when purchasing clothes.


**Table 1.** Considerations when buying clothing

#### **3.5 Clothing Improvements**

The points for improvement of clothes were divided discomforts and repair parts for clothes. Among them, the discomforts were divided into upper and lower discomforts, as shown in Table 2. First, looking at the discomforts of upper, the most common discomfort about jacket length was 31%, followed by sleeve length, extra space, upper length, and sleeve circumference. Looking at the discomforts about the lower, the most common discomfort about the length of the pants was 27%, followed by the rise length, waist circumference, hip circumference, and knee circumference.

The repair part was divided into the upper and the lower part similarly to the discomfort, and it is shown in Table 3. When looking at the repair part of the top, 38% of the respondents said that the sleeve length needs to be repaired, followed by fastening, jacket length, and armpit slack. When looking at the repair part of the bottom, the opinion that the trouser length needs to be repaired was the most at 48%, followed by the circumference of the lower, the length of the rise, the closure, and the circumference of the hip.

This result can be explained by the fact that if the jacket length is too long, the jacket may get caught on the wheelchair wheels when the wheelchair is pushed. And if the pants length is too long, this can be explained by the fact that the pants may drag on the floor when moving in a wheelchair.


**Table 2.** Upper and lower discomfort

#### **3.6 Clothing Design**

Clothing design was largely divided into color, adjustment location, fastening, and finishing, as shown in Table 4. Looking at the color design, most preferred achromatic colors at 58%, followed by colored colors and pastel colors. Looking at the adjustment location design, most preferred the front adjustment at 74%, followed by no adjustment, shoulder adjustment, and armpit adjustment. Looking at the design of the fastening, the button was the most preferred at 51%, followed by the zipper and Velcro. Looking at the finished design, 37% of the respondents preferred to do nothing like general clothes, followed by buttons, rubber bands, and Velcro.

In the case of color design and finishing design, it can be explained that the disabled people want to wear clothes that are indistinguishable from non-disabled people in


**Table 3.** Repair part of upper and lower

appearance. And it can be explained that the reason for preferring the front adjustment is that it is convenient when going to the bathroom because the study subjects are all male.

#### **Table 4.** Clothing design


### **4 Conclusion**

Through the results of the study, it was confirmed that the development of clothes for the disabled in Korea was carried out in terms of body type, considerations, improvement points, and design. However, there are limitations in that the subjects are all male and the number of studies is small. Therefore, it is necessary to collect studies with a wider period for follow-up studies, and the gender of the subjects should be included in a balanced way. And if additional research on clothing for people with disability, such as additional research on body type according to the type of disability, is conducted in the follow-up study, more systematic research data can be established. This study is meaningful in that it was able to analyze and grasp studies related to the development of clothing for the disabled, and it will be used as basic data for related research. In the future, research related to the development of clothes for the disabled will be conducted in various fields, and it is expected that the results will increase the convenience of the disabled and activate the social participation of the disabled.

**Acknowledgments.** The research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HJ21C0014).

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Short Contributions: Wellbeing Technology**

# **Empowering Well-Being Through Conversational Coaching for Active and Healthy Ageing**

Michael McTear1(B) , Kristiina Jokinen2, Mohnish Dubey3, Gérard Chollet4, Jérôme Boudy4, Christophe Lohr7, Sonja Dana Roelen5, Wanja Mössing5, and Rainer Wieching6

<sup>1</sup> Ulster University, Belfast, UK mf.mctear@ulster.ac.uk <sup>2</sup> Artificial Intelligence Research Center, (AIRC/AIST), National Institute of Advanced Industrial Science and Technology, Tokyo, Japan kristiina.jokinen@aist.go.jp <sup>3</sup> Institute für Angewandte Informatik, Germany (INFAI), Leipzig, Germany dubey@infai.org <sup>4</sup> Institut Mines-Télécom (IMT), Palaiseau, France {gerard.chollet,jerome.boudy}@telecom-sudparis.eu <sup>5</sup> Institut für Experimentelle Psychophysiologie Gmbh (IXP), Düsseldorf, Germany {s.roelen,w.moessing}@ixp-duesseldorf.de <sup>6</sup> Universität Siegen (USI), Siegen, Germany rainer.wieching@uni-siegen.de <sup>7</sup> Institut Mines-Télécom (IMT), Brest, France christophe.lohr@imt-atlantique.fr

**Abstract.** With life expectancy growing rapidly over the past century, societies are being increasingly faced with a need to find smart living solutions for elderly care and active ageing. The e-VITA project, which is a joint European (H2020) and Japanese (MIC) funded project, is based on an innovative approach to virtual coaching that addresses the crucial domains of active and healthy ageing. In this paper we describe the role of spoken dialogue technology in the project. Requirements for the virtual coach were elicited through a process of participatory design in workshops, focus groups, and living labs, and a number of use cases were identified for development using the open-source RASA framework. Knowledge Graphs are used as a shared representation within the system, enabling an integration of multimodal data, context, and domain knowledge.

**Keywords:** Active and healthy ageing · Dialogue system · Knowledge graphs · Participatory design

### **1 Introduction**

With life expectancy growing rapidly over the past century, societies are being increasingly faced with new challenges related to an ageing population and the need to find smart living solutions for elderly care and active ageing [1]. This paper presents an innovative approach involving virtual coaching that is currently being developed in a 3 year joint European (H2020) and Japanese (MIC) funded research project1,2. The virtual coach addresses the crucial domains of active and healthy ageing in cognition, physical activity, mobility, mood, social interaction, leisure, and spirituality, with the aim of empowering older adults to better manage their own health and daily activities, resulting in improved well-being and improved stakeholder collaboration. The virtual coach provides individualized profiling and personalized recommendations based on big data analytics and social-emotional computing, detecting risks in the user's daily living environment by collecting data from external sources and non-intrusive sensors and providing support through natural interactions with 3D-holograms, emotional objects, or robotic technologies using multimodal and spoken dialogue technology, advanced knowledge graph representations, and data fusion. [2] provides a general overview of the project.

The remainder of the paper is structured as follows. Section 2 reviews related work in the area of active and healthy ageing, with particular reference to the use of dialogue systems to provide a virtual coaching application. Section 3 presents the technologies being employed in the e-VITA virtual coach, looking at multimodal data fusion from sensors, emotion detection, knowledge graphs, and dialogue technology. Section 4 describes the development of an initial prototype system, including a process of participatory design resulting in use cases and content development, some of which has been implemented in the Wave 1 prototype. Section 5 concludes by outlining the next steps in the project.

### **2 Related Work**

There has been increasing interest in recent years in exploring how new developments in conversational AI and socially assistive robots can be applied to support active and healthy ageing. [3] presents a comprehensive state-of-the-art review in which several future research directions are identified, including: the need for unconstrained natural language processing and conversational strategies to enable robust and meaningful twoway conversations; the ability to interpret affective modalities in order to enhance user engagement and trust; and challenges in deployment of ensuring user adherence and data privacy. Looking at some individual relevant projects, [4] reports a month-long study of a virtual agent and robot that could interact with older adults in their homes through dialogue and gestures with the aim of providing companionship and reducing isolation. The importance of dialogue capability to enable social robot agents to provide natural interaction is also emphasized in [5], while [6] describes how information about the older adult's emotional status was extracted from an analysis of their verbal and non-verbal communication. [7] is an example of an ongoing project in the Netherlands with the aim of improving the lives of older adults through the use of voice technology. The e-VITA project also focusses on the issues addressed in these examples of related work but in

<sup>1</sup> https://www.e-vita.coach/.

<sup>2</sup> The e-VITA project has received funding from the European Union H2020 Programme under grant agreement no. 101016453. The Japanese consortium received funding from the Japanese Ministry of Internal Affairs and Communication (MIC), Grant no. JPJ000595.

addition extends the research to examine how data from sensors and emotion analysis can be used within dialogues with the older users, and how knowledge graphs can be used to store data relevant to active healthy ageing as well as personal data about the user to enable the virtual coach to offer personalized information and recommendations.

### **3 Technologies and Architecture of the Virtual Coach**

The complete e-VITA platform is based on the Digital Enabler platform provided by the project partner Engineering (Italy)3. The platform is designed to include multiple devices and software components from across Europe and Japan that are based on different technologies and standards. The platform supports communication and integration among different smart devices such as sensors and robots, as well as the collection and management of data to provide coaching functionalities.

#### **3.1 Multimodal Data Fusion from Sensors**

In a smart home, sensors can be used to capture and monitor data in order to provide the user with assisted, safe and comfortable living. In the e-VITA project three types of sensors are used: holter-based sensors worn by the user (or possibly his/her smartphone) that sense physiological and actimetric parameters; environmental sensors that measure physical data to assess the level of comfort and the quality of the indoor environment; and home-based sensors to monitor user behaviour and activities. Data from these sensors is combined (or fused) to make inferences about users in their environment, for instance, postures, localisation in the home, and the users' physiological states. Contextual information extracted from these sensors can be stored in knowledge graphs and exploited by the interactive voice-based coaching system.

Various challenges being addressed include interoperability between the sensors involving a combination of multiple different data that may operate at different sampling frequencies and with different representation formats or scales for the targeted e-VITA applications, e.g., physical exercises, activities of daily living (ADL), or fall prevention [8]. Another goal is to perform Data Minimization for transmission of information into the Cloud.

To date software modules have been implemented that deliver data fusion to assess the user's situation (location, activity, vital state) and environmental conditions (temperature, humidity). The delivered data fusion software uses the Digital Enabler framework, aiming to provide semantical situation and environment labels that can be used by the knowledge graphs and the dialogue system. The Data Fusion algorithms constitute a first step for processing heterogeneous sensor data and signals in a multimodal way. Future work will address other modalities in order to obtain a complete set of data about the user's situation and environment.

<sup>3</sup> https://www.eng.it/en/our-platforms-solutions/digital-enabler.

#### **3.2 Emotion Detection System**

A range of emotional cues can be detected in acoustic parameters in the user's voice, including parameters in the Frequency domain (e.g., fundamental frequency, jitter, or pitch), in the Time domain (e.g., speech rate, speech pauses and syllable rates), in the Amplitude domain (e.g., intensity or energy), and in the Spectral Energy domain (e.g., relative energy in different frequency bands) [9]. The Emotion Detection System (EDS) that is employed in the e-VITA platform enables the detection and classification of frequently used basic emotions such as fear, anger, disgust, happiness, surprise, and sadness during interactions between the coaching system and older adults [10]. The detected emotions can then be used by the dialogue system to provide appropriate interventions [11].

The exact classes detected by the e-VITA EDS may vary between the target languages in the e-VITA consortium depending on the availability of training data. In the current release, the German variant of the EDS can detect anger, disgust, fear, joy, neutral, and sadness. The Japanese variant can detect anger, contempt, disgust, fear, joy, neutral, sadness, surprise, and trust.

Within the e-VITA project, the EDS may need to be executed in a range of environments depending on upload bandwidth, data minimization, privacy concerns, and prototype environments. Thus the EDS may be implemented as a component within the Digital Enabler cloud environment, but could alternatively also be implemented on a local edge-computing device with yet-to-be-defined hard-/software specifications.

#### **3.3 Knowledge Graphs**

Knowledge Graphs are used to represent the knowledge that is used by intelligent AI systems [12]. A Knowledge Graph (KG) stores data about real-world things, events, and concepts. For example, an entity in a KG contains a semantic description and many characteristics that relate it to other entities, concepts, or events. Entities in a KG are represented as nodes and the properties of the entities are represented as edges in the graph that relate the nodes, resulting in a network of enhanced knowledge for a specific genre or topic.

In the current project the KG stores data about the user and the environment that is relevant to active healthy ageing and that enables the virtual coach to offer personalized information and recommendations. A localized KG stores personal information and is connected to the central KG. The central KG stores information required for various functions that are be used across the platform. The central KG is connected to a database that stores procedural details that cannot be harnessed through KG triples. The local KG is on-premise and keeps the personal data safe, while being able to offer assistance and a personalized user experience to an individual user. For instance, the fact that a user is male is stored in the local KG enabling the entire system to recommend activities suitable for male users. Figure 1 presents an excerpt of the graph that describes the prevention domain and the medical examinations that a male or female can take.

**Fig. 1.** Excerpt from the knowledge graph for the medical prevention domain

#### **3.4 Dialogue System**

The Dialogue System is the key component that enables interactions with the user. Its functions include understanding the user's input messages and providing an appropriate responses in the given context (including the previous dialogue context and environmental state) [13]. In the e-VITA project the Dialogue Manager is implemented using the RASA Open Source Framework that will be integrated with speech technology components to enable spoken interaction. RASA supports a machine learning-based Natural Language Understanding pipeline for the interpretation of the user's inputs and a combination of rule-based and machine learning-based dialogue policies to determine the system's actions [14]. Knowledge Graph technology is integrated via specific knowledge-based actions supported by RASA to supply content for the system responses by querying the domain knowledge base [15].

### **4 Prototype Virtual Coach**

A prototype version of the virtual coach was delivered at month 15 of the project. This section outlines how the requirements for the prototype were gathered and analysed, followed by a discussion of the use cases addressed in the Wave 1 prototype and an example of an interaction with the virtual coach.

#### **4.1 Requirements Gathering and Analysis**

Requirements for the virtual coach were gathered in interviews with older communitydwelling adults in Germany, Italy, and Japan, all of whom described themselves as regular users of smartphones and personal computers. In the first phase users were invited to living labs, while in the second phase studies were conducted in the homes of users. One of the studies conducted in Germany involved interactions with a nutrition chatbot and with the Nao robot in which a range of scenarios was explored including reminder functions, news, stories, and jokes, and general companionship. The end-user studies in Japan included interactions with real devices in Living Labs. One of these studies investigated users interacting with the Nao robot and a RASA dialogue system that included a general conversation of 10 min with the coach about daily living and a further 10 min conversation about food and general question-answering about the news. The users were instructed to engage in several scenarios, for example:


Results from the studies included comments that the users would like to be able to engage in longer conversations and not just receive short answers to their questions. In some cases, the system received user responses that were not included in the original design and that would have to be included in the machine-learning models of the next version of the system. Also required was the implementation of information about the user's daily routines, events in the user's calendar, and user preferences about topics such as music, and reminders.

#### **4.2 Interactions with the Prototype Dialogue System**

Based on the interviews with users the following use cases were identified for the prototype system to be developed and evaluated in Wave 1 of the project:


The following interaction is based on the RASA story represented in Table 1 which is activated following the recognition of the user's utterance "what preventive examination should I have performed by my medical doctor" as the intent **ask\_examination**. The system consults the knowledge graph shown in Fig. 1 and as ithe system does not know the user's gender in this use case, it asks the user to which the user replies "male". The system follows the **has\_right-to** link from the **male** node in the knowledge graph, retrieves the required information, and outputs it to the user. A screenshot of the dialogue produced by this story is shown in Fig. 2.

**Table 1.** RASA story: preventive examinations.




**Fig. 2.** Screenshot of dialogue in the medical prevention domain

### **5 Next Steps**

In the next phase of the project the dialogue system will be tested and evaluated in the living labs and in the homes of the older adults. For the home-based trials the devices will be prepared by the study centres and installed in the homes of the users. A diary and end-user manuals will be provided and the system will be explained and demonstrated. A human coach from the community will also visit once a week during the trials which will last for one month.

Future technical work with involve more complex knowledge modelling and knowledge graph creation and integration of the dialogue system with new knowledge graphs, sensor data, and data from the emotion detection system.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Smart Home-Based Home Modification Program for Persons with Disabilities: A Pilot Study**

KwangTae Moon1, YunHwan Lee1, Dongwan Kim1, and Jongbae Kim2(B)

<sup>1</sup> Department of Occupational Therapy, The Graduate School, Yonsei University, Seoul, South Korea otbuilder@yonsei.ac.kr, dwan3303@naver.com <sup>2</sup> Department of Occupational Therapy, College of Software Digital Healthcare Convergence, Yonsei University, Seoul, South Korea Jongbae@yonsei.ac.kr

**Abstract.** Smart Home Technology (SHT) as assistive technology (AT) is becoming an important active research field in the field of rehabilitation. For this purpose, Home Modification (HM) is one of the most common ways to improve the quality of life of the Persons with physical disabilities (PwPD). In this context, we propose a new Smart Home-based Home Modification Program (SHbHM) to improve the quality of life for PwPD. Our method simply uses Bluetooth or extends to Wi-Fi and Zigbee networks. A pilot study was conducted with five PwPD at home to investigate the effectiveness of the program. The reported results show a high quality of life, and the occupational performance and satisfaction are greatly improved, indicating that it is an efficient alternative.

**Keywords:** Smart home · Home modification · Quality of life · Disabled persons

### **1 Introduction**

Persons with physical disabilities (PwPD) often have difficulty managing and accessing their homes, and they must ask their caregiver for help to complete most of their life. Traditional home modification designed for PwPD such as Environmental Control Systems (ECS) can increase PwPD's independence and quality of life (QOL). However, several barriers limited the use and effectiveness of ECS including complex usage, their high-cost, and lack of institutional arrangements and recognition that it is not necessary [1]. smart home technology is rapidly evolving, resulting in dynamic and affordable devices. In their study, smart home played a major role in improving PwPD's quality of life, occupational performance, and home management [2]. However, in Korea, there are only studies composed of low technology for fall prevention purposes or applied to test beds, but clinical studies applying smart home to homes are insufficient [4–7]. This pilot study aims to gather information on applicability, effectiveness, and limitations to provide smart home-based home modification (SHbHM) to community practice areas as a first step in developing programs that promote quality of life, occupational performance, and home management.

### **2 Method**

This study was applied the SHbHM program to PwPD using one group pretest-posttest design and investigated the applicability, effectiveness, and limitations of the SHbHM program.

#### **2.1 Participants**

This study was selected by applying the objective sampling method. Through deliberate sampling, the study participants' intentional selection can obtain a lot of information about the selection of topics and phenomena suitable for the study topic [8]. The selection criteria are as follows. First, those who perceived that the living environment was inappropriate, second, those who could communicate, third, those with physical disabilities registered in the Disabled Welfare Act, fourth, those who did not plan to move within three months, fifth, understood the study and agreed to participate. From February 2022 to April 2022, we visited five participants at home and provided SHbHM. The characteristics of the participants were 4 men and 1 woman, with an average age of 66.6 years, 3 strokes, and 2 spinal cord injuries.

#### **2.2 Independent Variable (Smart Home-based Home Modification; SHbHM)**

This study used the SHbHM program to improve the occupational performance and quality of life of PwPD. We did the occupational profile to the participants. Beginner kits or expert kits were provided according to the occupation required through the occupation profile. After the kit is provided, the usage log for the kit is automatically logged. Occupational therapists monitor usability through participant logs, provide taskoriented and simplify occupational performance skills. In the SHBHM, the occupational therapist visited the subjects a total of eight times, including one initial assessment, one occupational profile, one kit installation and usability assessment, five task-oriented and simplify occupational performance skills, and one follow-up assessment (Fig. 1).

**Fig. 1.** Smart home-based home modification

**Occupational Profile.** The occupational profile is summary of a participant's occupational history and experiences, patterns of daily living, interests, values, needs, and Smart Home relevant contexts [9]. Using a client-centered approach, the occupational therapist collected information to understand what participants want and need through SHT and to identify past experiences and interests that can help them understand current problems. The researcher evaluated the participants' occupational performance, satisfaction, and health-related quality of life before and after SHbHM. Canadian Occupational Performance Measure (COPM) was used for occupational performance and satisfaction [10], and EuroQol-5 dimension (EQ-5D) was used for health-related quality of life [11].

**Beginner Kit.** PwPD has restrictions on hand use and indoor walking. It is difficult to control the lighting and unlock the door in life. Therefore, SHT is required. Even without smartphones and Wi-Fi, Participants can apply The Beginner kits at low cost without continuous cost through Bluetooth. The researcher monitored the participants' use logs for 4 weeks.

**Expert Kit.** The expert kit consists of door sensors, motion sensors, smart switches, Zigbee hubs, electric curtains, smart bulbs, AI speakers, smart plugs, and smart remote controls. It was automated according to the participants' daily lives. The use of Wi-Fi and smartphones was essential to use expert kit. The researcher monitored the participants' use logs for 4 weeks.

**Task-oriented and Simplify Occupational Performance Skills.** As The Task-Oriented focused on special functional tasks that incorporate musculoskeletal or nervous systems [12], we provided task-oriented routines rather than normal pattern repetitive exercises participants learned and learned skills by simplifying occupational performance skills [13].

### **2.3 Dependent Variable**

**Changes in the Quality of Occupational Performance.** We measured the quality of occupational performance evaluating the canadian occupational performance measure (COPM). Originally published in 1991, COPM is a customer-centric, individualized measure of results. Importance, performance, and satisfaction are evaluated on a 10 point scale according to occupation, and changes in scores after pre-score can be seen as therapeutic effects, which are clinically useful if there is a difference of more than two points [10]. According to the COPM guidelines, test-retest reliability has been reported in three studies, with performance reliability of 0.65, 7.99, 0.80, and satisfaction reliability of 0.84, 7.75, and 0.89, respectively [14] The evaluation was conducted before and after SHbHM.

**Changes in the Quality of Life.** Participants' quality of life was measured through EuroQol-5 Dimension (EQ-5D). EQ-5D, developed by EuroQol Group, is an indicator of health-related quality of life and can be used to evaluate health-related quality of life for people with disabilities [11]. EQ-5D is a total of five areas: mobility, self-management, daily activities, pain/comfort, and anxiety/depression. It is composed of 1 to −1 by applying weights to each measurement, and the overall health condition is −1 which is worse than 1 point death. The health-related quality of life score was calculated by applying the weight calculation equation according to Lee et al. [15]. The evaluation was conducted before and after SHbHM.

### **3 Results**

Our experiment used the SwitchBot platform for the Beginner Kit, and the Expert Kit used SmartThings, SwitchBot, AI power manager platform for log verification. The participants' usage log was checked by the researcher through IDE (https://graph.api. smartthings.com/) and application every day and recorded in Excel for 30 days (Fig. 2). As can be seen in Fig. 2, Expert Kit showed an increase of 500% from a 7 use per day to 35 use per 30 days after SHbHM was applied.

**Fig. 2.** SHbHM usage logs

We evaluated participants' occupational performance, satisfaction, and healthrelated quality of life before and after SHbHM. The results of the five are as follows (Table 1). As can be seen in Table 1, Participants increased occupational performance 6.2 (SD 1.7), occupational satisfaction 5.3 (SD 2.5), and quality of life 0.302 (SD 0.075).

**Table 1.** COPM, EQ-5D change scores for participants who had a SHbHM.


### **4 Conclusion**

This study bridged the gap in a pilot study on the service delivery model used to provide SHbHM as a home modification and confirmed the need for future studies. In addition, the purpose of this pilot study was to develop a new program and provide it to PwPD to find out the applicability of the technology. However, there is a limitation in that the sample size is small to verify the effect of the applied SHbHM. In future studies, it plans to verify the effectiveness by expanding the number of subjects. although the number of research subjects was small, it was significant in that it was a study that applied smart home technology to the homes of the PwPD in Korea.

The results of this study were intervention and home modification that enable home management for PwPD. The provision of SHbHM has the potential to inform the development of cost-effective service delivery models. The use of SHbHM as home modification to improve PwPD's occupational performance and health-related quality of life is extending area of research and intervention used in practice which is eventually tailored in community for occupational therapist.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Mask Detection Using IoT - A Comparative Study of Various Learning Models**

Mohamed Amine Meddaoui<sup>1</sup>, Mohammed Erritali1(B), Youness Madani<sup>1</sup>, and Fran¸coise Sailhan<sup>2</sup>

<sup>1</sup> Data4Earth Laboratory, Sultan Moulay Slimane University, Beni Mellal, Morocco meddaoui.med@gmail.com, m.erritali@usms.ma, younessmadani9@gmail.com <sup>2</sup> Cedric Laboratory, CNAM Paris., Paris, France francoise.sailhan@cnam.fr

**Abstract.** Wearing a mask is an effective measure that prevents the spread of respiratory droplets into the air and thereby curtails the dissemination of coronavirus. Unfortunately, despite the proven effectiveness, the idea of wearing a face mask has difficulty being accepted by part of the population. To address this significant health concern, we present a monitoring system that automatically detects whether a mask is put appropriately over a face. The system annotates the videos that are provided by cameras. In this article, we present a comparative study of machine learning models (i.e., SVM, RNN, LSTM, CNN, auto-encoder, MobileNetV2, Net-B3, VGG-16, VGG-19, Resnet-152).

**Keywords:** COVID-19 *·* Face mask detection *·* IoT

### **1 Introduction**

While there are multiple ways to fight the pandemic, a face mask remains a cost-effective measure that is widely practiced to prevent the spread of the virus. Nonetheless, the adoption of face-mask measures remains controversial. The ability to detect violations in public and work spaces is of utmost importance for organizations that host a large population and wish to monitor exposure to adopt precautionary measures. In this article, we propose a detection system that determines whether a person is appropriately wearing a mask, using the information provided by some cameras. Cameras may be disseminated in a building to provide some videos (i.e., image sequences) that are further used to support face mask recognition and ultimately issue proper directives. Still, detecting face masks in any situation is challenging because there are many variations in the appearance of the image that may alter the detection (Fig. 1): variations can be due to (i) a disruptive context (e.g. occlusion of the person, low light intensity), (ii) a rotation or an poor orientation of the person that hinders the detection, (iii) camera parameters (low resolution, excessive focus, noise) or poor positioning of the camera (misorientation, large distance to the scene). We

(a) One person of the three wears a mask (b) Two persons wear a mask ; one per-

son has sun glasses

(c) one mask is incorrectly put, a person has glasses (d) 2 persons in profile ; 2 persons wear a mask and one in front wears a cap

propose a comparative study of different ML models and evaluate their ability to handle the above situations and their suitability for deployment on IoT devices or cloud servers. Overall, our key contribution includes:


The paper is organized as follows. We present the related work (Sect. 2) and we introduce our comparative study (Sect. 3), which is further evaluated (Sect. 4).

### **2 Related Work**

Several detection systems [5–7,10,11,14,16,17] have been proposed and evaluated using some purposely-built datasets (Table 1) that contain some pictures of people with facemask, without facemask or with facemask put incorrectly. The pictures of masked people either (i) correspond to real-world person that are pictured (*real-world dataset*) or (ii) result from the addition of a mask picture on an existing facial image (*simulated dataset*). Leveraging existing datasets, detection system detects facemasks on pictures (Sect. 2.1) or videos (Sect. 2.2).



#### **2.1 Facemask Detection in Pictures**

The facemask detection system introduced in [11], consists of a feature extractor that uses convolutional neural network (Resnet50<sup>1</sup>) and a classifier that implements Support Vector Machine (SVM) and ensemble algorithm. Evaluations based on the RMFD, SMFD, CMCD datasets, show that the SVM classifier involves the fastest training and achieves the highest accuracy: 99.64% testing accuracy with RMFRD, 99.49% with SMFD, and, 100% with CMCD. In [10], the solution localizes medical face masks and annotates accordingly those images. Feature extraction process relies on the ResNet50 deep transfer learning model while the mask detection process uses YOLO v2 [12]. Following, authors rely on the Adam optimization algorithm [8] to improve the performance of the detector. Empirical evaluation shows that the Adam optimizer achieves the highest average precision percentage of 81%. In [14], face mask detection uses the faceNet image classifier [15] that implements a Convolutional Neural Network (CNN). This image classifier is trained using a purposely-built dataset including 4K images with half of the dataset containing some pictures of people wearing mask in public places (e.g. shops) while the rest concerns people without mask. Empirical results show that people wearing (or not) a face mask are detected with an accuracy of 96,85%. Arjya et al.[6] detect the facemask on image using a pre-trained CNN containing two 2D convolution layers connected to layers of dense neurons. The proposed method attains an accuracy up to 95.77% with SFC dataset and 94.58% respectively FMD dataset. In [7], a facial categorization system determines whether a person is wearing a mask or not. Face recognition is performed by a deep C2D CNN (Colour 2-Dimensional principal component analysis - Convolutional Neural Network) ; mask detection relies on special convolutional architecture that is best suited for the classification of RGB images.

<sup>1</sup> https://keras.io/api/applications/resnet.

The training relies on the RMFRD and Celeb Faces Attributes<sup>2</sup> dataset. In [5], the face mask detection system captures image, extracts features from image based on Principal Component Analysis (PCA), detects the human face using viola zones method and further uses the K-Nearest Neighbor (KNN) classifier. Experiments are based on the ORL<sup>3</sup> database in which the lower portion of detected face images is covered with black or white. Preliminary performance evaluation shows that the accuracy is around 98% with a principal component of two. Additionally, face recognition accuracy with face masks has been extensively investigated: interest reader may refer to the masked face recognition workshop and challenge<sup>4</sup>.

#### **2.2 Face Mask Detection in Video**

Another line of research aims at providing a surveillance system [17] that identifies whether a person is wearing a mask using real-time videos. Mask detection is done by MobileNetV2 [13] that achieves high accuracy of 99.98% on training data, 99.56% on validation data, and 99.75% on testing data. In [16], a mobile robot automatically detects unmasked personnel in public spaces and provides a surgical mask to them to promptly remedy the situation. The mobile robot integrates deep residual learning (ResNet-50) with Feature Pyramid Network (FPN) to detect the existence of human subjects in video (feeds). Then, Multi-Task Convolutional Neural Networks (MT-CNN) detect and extract human faces from these videos. Ultimately, a convolutional neural network classifier detects (un)masked human subjects. Training leverages four publicly available datasets: Microsoft Common Objects in Context (COCO)[9], the CelebFaces Attributes Dataset<sup>5</sup> (CelebA), WIDER FACE dataset<sup>6</sup>, CMCD. The proposed surveillance system is further evaluated using a dataset of videos collected by the robot in an educational institute. Results show a mask detection accuracy of 81.3% with a very high recall of 99.2%. While many detectors rely on pictures, only two approaches support real-time facemask detection leveraging videos. In this paper, we introduce a video-based system that incorporates several ML models and we provide a comprehensive comparison with the state of the art.

#### **3 Mask Detection**

Leveraging the videos delivered by the camera, our application detects the presence of any nearby person and determines whether the person has a mask and if the mask is correctly put. Then, the application labels the corresponding image. This detection requires converting the videos into an appropriate format, locating people face(s) (Sect. 3.1) and determining whether people wears mask (Sect. 3.2).

<sup>2</sup> https://www.kaggle.com/jessicali9530/celeba-dataset.

<sup>3</sup> https://cam-orl.co.uk/facedatabase.html.

<sup>4</sup> https://tinyurl.com/4xzjzvat.

<sup>5</sup> https://www.kaggle.com/jessicali9530/celeba-dataset.

<sup>6</sup> https://paperswithcode.com/dataset/wider-face-1.

#### **3.1 Face Detection in Pre-processed Video**

Face mask detection starts with the capture of a video partitioned in successive series of color pictures. Color pictures are further converted into RGB pictures, which render the process of discovering the face less complex comparing to color picture. For each picture, mean subtraction is applied to prevent illumination: the average intensity is computed accross all the images for each of the Red, Green, and Blue channel ; then, the mean is substracted per channel for each image. Following, each image is divided into *n × n* squares where the value of *n* depends of the object of interest (e.g. main feature of the face). Within each square, the face detection algorithm passes through each pixel of the image in addition to the adjacent pixels (i.e. the pixels located at the top - bottom - left - right - top right - top left - bottom right - and bottom left). This process is intended to store key facial features (e.g. eyes) that help in detecting face while irrelevant data that are located in the background (e.g. a car, tree, traffic light). Finally, our system determines the location of each face. Our approach consists in classifying into two classes, whether it is wearing appropriately a facemask or not. Note that the facemask is appropriately wear if the mouth, chin and part of the nose are well covered with the mask.

#### **3.2 Classification**

For facemask detection, we built some models using SVM, RNN, LSTM, Auto-Encoder, which are briefly presented, starting with SVM.

*Support Vector Machines* SVM separates the input data within the space by a hyperplane that linearly separates the data into classes (with and without mask). Input data typically refers to small training dataset made of support vectors. Herein we use a linear kernel. The hyperplane best separates the support vectors, by means of maximization of the distance between these vectors and the hyperplane. As shown in Sect. 4, the SVM remains efficient with little training data.

In addition, we consider three types of Convolutional Neural Networks (CNN) referring to Recurrent Neural Networks (RNN) and Long Short Term Memory Networks (LSTM) - that are multi-layered neural networks made of several hidden layers of neurons wherein the output of a neuron in a layer becomes the input of a neuron of the next layer. These networks have adopted diverse structures that meet different expectations. Convolutional Neural Network (CNN) is a neural made of two distinct parts: (i) the convolution layers extracts valuable features from the input (image); in practice, kernels automatically extract the relevant features based on the convolution operation, (ii) the fully connected layers leverage the data from convolution layer to generate the result.

*Recurrent Neural Network (RNN)* are a class of neural networks that differs from others in that they maintain internal hidden states and have cyclic/recurrent connections, which allow them (i) to capture the sequential information (i.e. dependencies) in the input data and (ii) information to persist. Still, RNN traditionally suffers from what is known as the problem of vanishing and exploding gradient in which the network either stops learning (vanishing gradient) or never converges to the point of minimum cost (exploding gradient). LSTM are designed to remedy both problems and thereby have become popular in modelling complex sequential data.

*Long Short Term Memory Networks* consists of a set of recurrently connected subnetworks (also coined as memory blocks). Any block contains one or more self-connected memory cells storing historical states, as well as gates that control the flow of information through the cells. Thus, LSTM may store and access information over long period of time, which prevents the vanishing gradient problem. LSTM contains four layers of neural networks.

*Auto Encoder* is a specific type of neural network in which the encoder represents the input into a compressed and meaningful representation so that the decoder has the most relevant information to reconstruct the image. In particular, the encoder learns the most important components of an input and thereby gets the best possible compression. The error made by the encoder is established based on the differences between the reconstructed data and the initial data. The training consists in modifying the parameters of the auto-encoder so as to reduce the reconstruction error measured on the different samples of the dataset. While various neural network topologies exist (e.g., vanilla, convolutional, regularized, multi-layer), we used a multi-layer auto-encoder and we encoded in an unsupervised way. The encoder contains three hidden layers: the first one is four times larger than the input, and the second one is two times larger than the input, and the size of the third one is equal to the input size. Following, we optimize the model using adam optimiser [8].

### **4 Performance Evaluation**

We assess the effectiveness of our detector relying on the following two training datasets: the Real World Masked Face Recognision Dataset [4] and the Face Mask Dataset [2] (FMD) that include some color pictures of different sizes. Together these two datasets include some pictures of people of different nationalities/ages, with/without facemask, with mask put incorrectly, with e.g., glasses, hat. As detailed in Sect. 3.1, pictures are normalized and expressed into a common format. We performed cross-validation, using scikit-learn platform<sup>7</sup>, which splits the dataset into a training (80% of the original dataset) and test dataset. Unless explicitly mentioned, 100 epochs are for building models. In the following, we evaluate the performances associated with the training and the detection. In both cases, the experiments are run either on a IoT device (Raspberry Pi 4 with 1,6 GHZ and 2 GB of RAM) with 3,1 GHZ of CPU, and Machine ASUS with 8 GB in RAM, used to perform the training and the mask detection. In order to evaluate the detection of facemask, we also conducted series of experiments using dataset and a camera of 48 Mega pixel.

<sup>7</sup> https://scikit-learn.org.


**Table 2.** Size of dataset used to train and test the model

*Delay associated with the training and detection.* In Table 3, we evaluate the time associated with the learning process, using various sizes of dataset (Table 2). As expected, the larger the dataset, the longer the learning process. The device capacity also has a significant impact on its ability to learn (quickly). Contrary to the server, the IoT device can only handle small or medium datasets, regardless of the learning model. The learning process is on average 2.631 times longer with the IoT device whose capabilities are limited, compared to the server. On the other hand, detection is much faster than learning (Table 4). In particular, the time associated with detection is completely decoupled from that associated with learning. The detection with the Raspberry takes a little longer than with a server. For both (IoT device an server), CNN is the fastest, followed by autoencoder, LSTM, RNN whose results are close while SVM is the slowest.

**Table 3. Training Delay** associated with a dataset of varying size, using an IOT device and a server


**Table 4.** Delay associated with detection


*Facemask Detection Efficiency* We compare the effectiveness of four strategies used to detect face masks (as defined in Sect. 3). Experiments were run on the IoT device and on the server; results reported bellow are the same for both. Figure 2 provides the accuracy and the loss functions associated with the training (*accuracy* and *loss*) and the validation (*val accuracy*, *val loss*) sets; the training set is used to build the model while the validation set supports the

**Fig. 2.** Accuracy and Loss associated with the training and validation with large dataset

**Table 5.** Dataset size


fine tuning of the parameters and of the model structure. Loss corresponds to binary-cross entropy. For the RNN, LSTM and CNN models, accuracy and loss appear to be inversely proportional: after few iterations of optimization, the loss reduces drastically while the accuracy greatly increases. As intended, the shape of accuracy and loss functions are quite similar with the training and validation sets. At first sight, CNN becomes quickly accurate (accuracy is high and loss is small after only 40 epochs). With LSTM (and resp. RNN), the accuracy and loss stabilize after 60 epoch (resp. 80 epochs). The auto-encoder takes much more time (400 epochs) comparing to the other methods: the loss curve still fluctuates for epoch ranges of [200,250] and [300,350]. In addition, we evaluate the performances associated with the detection in terms of precision, recall, F1 score and accuracy. Relying on these performance metrics, we evaluate in Tables 6, 7 and 8 the efficiency of the detection models considering a small, medium and large dataset (as defined in Table 5). As expected, the larger the dataset, the


**Table 6.** Models with small dataset




**Table 8.** Efficiency of models with large dataset

**Table 9.** Efficiency of state of the art models using dataset3


more effective the training and subsequent detection, as shown by the increase in precision, recall, F1 score and accuracy for any model. Note that data augmentation may be relevant to increase the dataset size (and thereby the efficiency of the ML models) by creating modified versions of images. Regardless of the dataset size and of the model, the accuracy associated with the detection of people with mask is better than that of people without mask. Regardless of the dataset size, CNN always gives the best result in terms of precision, recall, F1 score, accuracy. Then, LSTM and auto-encoder provide lower but high efficiency, followed by RNN and SVM. Table 9 compares various models. CNN gives the best results in terms of accuracy while ResNet-152 is characterised by a slighly lower accuracy but a quite similar precision, recall and F1-score.

### **5 Conclusion**

We introduce a new detection system that automatically determines whether a person wears facemask, which is put appropriately. In practice, the system detects and determines the position of the face(s) in the videos provided by cameras. For each detected face, the system determines whether a facemask is put appropriately, leveraging some machine learning models. Experimental results show that classification may be performed by a server or IoT device. Empirical results demonstrated that CNN gives is the most accurate (99.8% with a large training dataset). Future work involves improving the detection in presence of low quality picture (e.g., low light level, presence of obstacles) and evaluating the energy associated with the detection.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Understanding the Knowledge, Perception and Uptake of Contraception in Nigeria: A Case Study of Saye-Zaria**

Ayandunmola Folake Oyegoke1,2(B) and Aisha Abubakar<sup>3</sup>

<sup>1</sup> Salama Infirmary (Hospital and Maternity), Sabon-gari LGA, Zaria, Nigeria ayandunmolaonilude@gmail.com <sup>2</sup> Human Kinetic and Health Education Department, Ahmadu Bello University, Samaru Zaria,

Nigeria

<sup>3</sup> Community Medicine Department, Ahmadu Bello University, Samaru Zaria, Nigeria

**Abstract.** An integral part in the comprehensive care of HIV and a significant health service is contraception, however research carried out to evaluate the perception and utilization of contraception among HIV positive men are few. This research aim to determine the knowledge, perception and uptake of contraception among HIV positive male patients at Saye-Zaria. This was a descriptive, crosssectional study with collection of quantitative data through questionnaire and qualitative data with Focused group discussion (FGD). The majority (85.1%) of respondents have heard about contraception, most had good perception, and only 61.9% of the respondents have ever used contraception. There was a significant association between level of education and perception. In conclusion there was high knowledge, low usage and poor acceptance of contraception. Therefore, the government should put adequate policies in place to encourage male involvement in the utilization of contraception.

**Keywords:** Male patient · HIV · Nigeria · Contraception · Perception

#### **1 Introduction**

Nigeria is second to South Africa in the number of people living with HIV/AIDS worldwide, representing 9% of the global burden of the disease [1] while men account for 58% of adult AIDS related deaths [2]. HIV is mainly transmitted through heterosexual contact and new infections in the country heighten due to reduced perceived personal risk, multiple sexual partners, inefficient and inadequate treatment of sexually transmitted infections (STIs) and poor quality service delivery [1].

In Nigeria, the total contraceptive prevalence rate (CPR) indicates wide state variations, ranging from 0.3% in Jigawa to 41.6% in Lagos state, as well as zonal variations ranging from 2.7% in the North West to 28.5% in the South West [3]. In a study done in 2003 among monogamous men in Ondo State Nigeria, the study revealed that 81.0% of the respondents knew of condom, only 31.2% had ever used a condom and 15.0% reported currently using condoms at the time of interview [4]. This trend was also seen in a study done in 2006 on male knowledge, attitudes, and family planning practices in northern Nigeria where the results suggested that there is high knowledge of contraceptives and consequently low rates of contraceptive use [5] of which the reasons might range from cultural to religious misconceptions about men using contraception. There are overwhelming previous studies carried out to assess the uptake, utilization and factors affecting contraceptive use among HIV positive women and prevention of mother to child transmission of HIV (PMTCT) in both urban and low resource settings. However, research carried out to evaluate the perception and utilization of HIV positive men are few [5, 6] although it is the men that are usually the decision makers on important household issues like household purchase, family size, health of household members and timing of pregnancy in this country [7]. The uptake of contraceptives and the type of contraceptive use is heavily influenced by the male/husband dominance in the society/family. Therefore, there is a need to integrate and involve the male in the contraception policies [8].

In this study, we evaluated the knowledge, perception and uptake of contraception among HIV positive male patients in Nigeria. We majorly focused our study on patients attending ART clinic of National Tuberculosis and Leprosy Training Center (NTBLTC) Saye-Zaria. So, we provided answers to the following questions: (1) What is the knowledge of contraception among HIV positive male patients? (2) What is the perception of contraception use among HIV positive male patients? (3)What is the use of contraception among HIV positive male patients? (4) What are the factors influencing contraceptive uptake among HIV positive male patients?

#### **2 Methodology**

The study was a descriptive, Cross-sectional study collecting both qualitative and quantitative data. The study population was HIV positive male patients receiving treatment at the ART clinic at NTBLTC Saye-Zaria, Nigeria. Inclusion criteria was all HIV positive male patients attending ART clinic, HIV positive male patients who are ≥18 years and HIV positive male patients who are taking ART at the time of the interview. The sample size was determined using the formula [9]

$$n = \frac{Z^2pq}{d^2} \tag{1}$$

A sample size of 285 was used as the minimum sample size for this study. A simple random sampling technique was used to collect information from eligible participants. The data was collected using adapted, semi-structured and pre-coded questionnaires which were collected via an android device using KoBo kollect software and were administered to each respondent. The questionnaire contained information on respondent's sociodemographic status, knowledge of contraception, perception of contraception use, uptake of contraception and factors influencing the use of contraception. The knowledge of contraception was based on those that have ever heard of contraception before. The perception on contraception was scored on a 2-point perception scale for each question, positive perception was 2 point, negative perception was 0 points and undecided was 1 point. Those that score less than or equal to one (≤1) point had negative perception and those that score greater than one (*>*1) point had positive perception. The quantitative data collected was entered and analyzed using STATA, univariate analysis was done using proportions, measures of dispersion, measures of central tendency and percentage while bivariate analysis using chi square was done. A confidence interval of 95% was used and a p value of *<*0.05 was considered statistically significant.

### **3 Results and Discussion**

#### **3.1 Socio-Demographic Characteristics**

The study (in Fig. 1) showed that 84 (31.3%) of the respondents were within the 38–47 age group, the mean age of respondents was 45.6 ± 11.7. Majority of the respondents (81.0%) were Muslims, 50 (18.7%) were Christians, most of the respondents were of Hausa tribe 194 (72.4%), Fulani/Ibo (7.8%), Yoruba/Others (6%). Although, 79.1% of the respondents were married, 147 (69.3%) are monogamous and 30.0% have more than one sexual partner.

**Fig. 1.** The socio-demographic characteristics of respondents.

Some (27.2%) of the respondents have at least secondary education, 25% have Quranic education, 20.2% have tertiary education, 14.9% post-secondary and 3% have no formal education. 57 (21.3%) of the respondents are traders and self-employed, 54 (20.2%) are civil servants, 47 (17.5%) were farmers and only 6 (2.2%) are unemployed. More than half of the respondents 146 (54.5%) reside in urban areas while 122 (45.5%) reside in rural areas.

#### **3.2 Knowledge of Contraception Among HIV Positive Male Patients**

Majority 228 (85.1%) of the respondents have heard about contraception before which showed that there is good knowledge about contraception and almost all 206 (90.4%) of the respondents knew that condoms are a type of contraception, as shown in Fig. 2. It is also similar to a study done in Uganda Rhoda and a study done in Zimbabwe where more than 98% of the men reported that they have heard of at least one method of contraception [10, 11]. However, it is in contrast to a study done in Osogbo-Nigeria on male involvement in family planning where only 57.0% had a good knowledge of FP [3].

**Fig. 2.** Knowledge of respondents about contraception.

Other types of contraception identified by the respondents were Pills 48 (21.1%), Injectable 41(18%), Withdrawal 24 (10.5%), only 15 (6.6%) and 10 (4.4%) of the respondents knew Male sterilization and Female sterilization respectively. The least known methods were Foam or Jelly (0.4%), Emergency contraception (0.4%), LAM 2 (0.9%), Diaphragm 3 (1.3%), IUD 6(2.6%) and Traditional method 8 (3.5%). Mostly half of the respondents knew that contraception is used for Prevention of pregnancy and Preventing the transmission of HIV/STIs, while 28 (12.3%) of the respondents think that contraception is neither useful for both.

Half (50.4%) of the respondents in this study got their information about contraception from ART clinic/Hospital, while 30.7% got their information from radio. This is followed by 7% of respondents that got their information from television and friends/family.

#### **3.3 Perception of Contraception Among HIV Positive Male Patients**

The overall mean perception score (in Table 1) of the respondents was 1.38 ± 0.52 and most of the respondents 192 (72%) had good perception while 76 (28%) had poor perception. 22.0% were of the perception that it is only promiscuous women that use contraception without their husband's consent, 23.5% of the respondents were of the perception that they will not allow their spouse to use contraception, the perception of 15.7% of the respondents was that it is only women that are meant to use contraception and not the men.


**Table 1.** Perception of respondents on contraception.

\*Positive responses

About 16.4% and 22.0% perceived that there is no need for HIV positive men to use contraception and contraception should not be promoted among people living with HIV, while the perception of 19.4% of the respondents was that men should not assist their women in obtaining contraception this is in contrast with the study done at secondary health facility in North-Central Nigeria where 97.8% of the respondents said condom use should be promoted among people living with HIV/AIDS [12].

#### **3.4 Uptake of Contraception Among HIV Positive Male Patients**

Out of the 268 respondents (in Table 2), many of the respondents 166 (61.9%) reported that they have ever used contraception and 91 (34%) reported that their spouse has ever used contraception before while only 152 (56.7%) are currently using contraception which is almost synonymous to a study done in Nairobi, Kenya where 58.8% of the male respondents had used contraception [13].

Out of 166 respondents that reported that they had ever used contraception, 153 (92.2%) chose condom as one of the contraceptive methods that they have ever used, 17 (10.2%) withdrawal method, 7 (4.2%) abstinence, 4 (2.4%) traditional method, 3 (1.8%) chose pills while only 2 (1.2%) chose male sterilization. 152 respondents reported that they are currently using contraception of which condom still ranked highest contraceptive method used at 138 (90.8%), the second was withdrawal 14 (9.2%), followed by injectable at 8 (5.3%) and male sterilization at 3 (2.0%). None of the respondents chose IUD, LAM, foam/jelly, diaphragm and emergency contraception as a method they have used or currently using.

About 65 (39%) of the 166 respondents that have ever used contraception reported that the reason for their contraceptive use was to prevent pregnancy, 59 (36%) stated prevention of re-infection as the reason while 42 (25%) stated prevention of re-infection and pregnancy as the reason for contraceptive use.


**Table 2.** Contraceptive methods ever used and currently using by respondents.

\*Multiple responses

#### **3.5 Factors Affecting Uptake of Contraception Among HIV Positive Male Patient**

30% of the respondents choose that the reason for not using contraception was because they were married, 29% said it is because they were both HIV positive, 28% said it reduces sexual pleasure, 18% acknowledge that their religion does not permit them to use it also that they don't know where to obtain contraception from and lack of discussion by their health care provider. 15% reported that it was because of the side effects, 14% due to the attitude of the health workers, 7% reported that it was because contraception is too expensive. Out of 10% of the respondents that were not using contraception because of other reasons, most (13) of the respondents reported that it was because they are not married.

#### **3.6 Sociodemographic Characteristics, Knowledge, Perception and Uptake of Contraception Relationship**

The relationship between the demographic characteristics and the knowledge, perception and uptake of the patients were evaluated as shown in Table 3. Findings from the study indicated all demographic characteristics were found to be insignificant except for marital status which shows good relation with the knowledge (P*<*0.001) and uptake (P*<*0.001); including other characteristic like Level of education which was shows a significant relationship with their perception (P = 0.005) and uptake (P = 0.004) while age was found to have shown as significant influence (P = 0.001) on their use of contraception.

The study further indicates that the most influential demographic characteristic which significantly contributes to the level of the patients' perception, knowledge and uptake


**Table 3.** Relationship between sociodemographic characteristics and other studied criteria.

of contraception was found to be marital status, education level and age. Which were found to be evident in the nature of the responses where the married class and aged class of people (from 38 years and above) were found to have shown significant knowledge of contraception compared to other categories. The influence of education was equally seen to have enabled the patients' gain better perception of the contraception use compared to the class of uneducated patients.

### **4 Conclusion and Recommendations**

The study carried out in NTBLTC Saye-Zaria to determine the knowledge of contraception among HIV positive male patients showed that there was high knowledge of contraception among HIV positive men, good perception, low usage of contraception and poor acceptance of contraception due to religious reasons. Marital status, level of education, and age were socio-demographic characteristics that affected the uptake perception and knowledge of contraception. Therefore, it is necessary for religious organization to enlighten men on importance of contraception. The Government should put in place adequate policies to encourage male involvement in the utilization of contraception as some respondents in the study thought contraception is basically a problem the women have to deal with alone.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **In-Air Handwriting Recognition Using Acoustic Impulse Signals**

Kai Niu1,2, Fusang Zhang3(B), Xiaolai Fu<sup>4</sup>, and Beihong Jin3,5(B)

<sup>1</sup> Peking University, Beijing, China

<sup>2</sup> Beijing Xiaomi Mobile Software Co., Ltd., Beijing, China

<sup>3</sup> Institute of Software, Chinese Academy of Science, Beijing, China *{*fusang,beihong*}*@iscas.ac.cn <sup>4</sup> Beijing University of Posts and Telecommunications, Beijing, China

<sup>5</sup> Beijing Key Laboratory on Integration and Analysis of Large-Scale Stream Data, Beijing, China

**Abstract.** This paper presents AcousticPAD, a contactless and robust handwriting recognition system that extends the input and interactions beyond the touchscreen using acoustic signals, thus very useful under the impact of the COVID-19 epidemic. To achieve this, we carefully exploit acoustic pulse signals with high accuracy of time of fight (ToF) measurements. Then we employ trilateration localization method to capture the trajectory of handwriting in air. After that, we incorporate a data augmentation module to enhance the handwriting recognition performance. Finally, we customize a back propagation neural network that leverages augmented image dataset to train a model and recognize the acoustic system generated handwriting characters. We implement AcousticPAD prototype using cheap commodity acoustic sensors, and conduct extensive real environment experiments to evaluate its performance. The results validate the robustness of AcousticPAD, and show that it supports 10 digits and 26 English letters recognition at high accuracies.

**Keywords:** Acoustic sensing *·* Gesture recognition *·* Wireless sensing

### **1 Introduction**

Nowadays, touchscreen technology [1] has been widely used as a way to interact with computer systems. For example, we often use the self-ordering screen in the KFC and McDonald as shown in Fig. 1(a). In Automated Teller Machines (ATMs) information kiosks, we use touchscreen to input password and withdraw

Supported by the Project funded by China Postdoctoral Science Foundation (No. 2021TQ0048), the National Natural Science Foundation of China under Grants (No. 62172394, No. 61802373), Open Research Fund Program of Beijing Key Laboratory on Integration and Analysis of Large-scale Stream Data, and the Youth Innovation Promotion Association, Chinese Academy of Sciences (No. 2020109).

cash. However, due to the world is being affected by COVID-19, and this contacttype interaction method will increase the spread of the disease [2]. Imagine that if there is a contact-free interaction method, it will greatly reduce the spread of infections. In this paper, we aim to design a robust contact-free sensing system that can leverage cheap acoustic sensors to achieve accurate and robust input with handwriting in the air.

(a) Traditional touchscreen (b) Contactless handwriting

**Fig. 1.** The evolution from traditional touchscreen to our proposed contactless acousticbased handwriting system

Some existing contact-free handwriting systems require extra customized devices with high cost (e.g., FMCW radar [3,4] and Lidar [5,6]). Some work [7– 10] utilizes WiFi signal which has been proved with severe location-dependent issues. Recent work [11–13] utilizes acoustic signals embedded in smart devices to enable gesture tracking. FingerIO [11] can accurately track waving hand by transmitting OFDM (Orthogonal Frequency Division Multiplexing) modulated acoustic signals. CAT [14] utilizes acoustic FMCW (Frequency Modulated Continuous Waveform) to develop fine-grained motion tracking systems. However, these systems should tackle complex system delay and sampling frequency offset before use. And the distance measurements are difficult to obtain accurately.

In this paper, we propose AcousticPAD, a contact-free handwriting recognition system based on cheap commodity acoustic sensors. In our solution, we place two acoustic sensors on the corner of a surface and transmit acoustic pulse signals (Fig. 1(b)). By simply setting the signal voltage, the system can accurately measure the echo pulse reflected from user's hand and further estimate the flight time of the signal. Combined with ToF from two acoustic devices, we track the hand trajectory using the trilateration localization method. To recognize the content of handwriting input, we need to have the classification method and address the challenges of labor-intensive data collection and handwriting at differen locations and orientations in the air. Therefore, we borrow the existing dataset (MINIST) [15] from image recognition field and design a data augmentation technique to enhance the data. Such a well-designed recognition process can not only reduce the time and effort required to manually collect data but also achieve the location and orientation independent handwriting recognition. Experiment results demonstrate our system is able to recognize handwriting of 10 numbers and 26 English letters with high accuracy and robustness. Please find our demo video at the link: https://youtu.be/sCZvK2rUzEU.

The main contributions of the paper are summarized as follows:


### **2 System Design of AcousticPAD**

#### **2.1 System Overview**

Figure 2 illustrates the overview about the design of the proposed *AcousticPAD* system that leverages commercial acoustic sensors to transmit/receive pulse signal. *AcousticPAD* mainly consists of two modules: *Real-Time Handwriting Acquisition* and *Position-Independent Handwriting Recognition*. In *Real-Time Handwriting Acquisition* module, *AcousticPAD* collects sound signal from acoustic sensor and recovers the trajectory of handwriting. Then *Position-Independent Handwriting Recognition* module leverages the augmented MNIST/EMNIST dataset [15] to train a Back Propagation (BP) neural network and recognize the handwritings including digits and letters position-independently<sup>1</sup>.

**Fig. 2.** System overview of AcousticPAD.

**Fig. 3.** The trilateration localization method for handwriting.

<sup>1</sup> Position independent refers to the user can perform the handwriting without location and orientation dependency.

#### **2.2 Real-Time Handwriting Acquisition**

In this subsection, we first introduce the generation of acoustic pulse signal in our system. Then we acquire the handwriting characters in a contactless manner following steps of: distance measurement, character segmentation and hand tracking.

**Trigger to Transmit and Receive Sound Pulse Signal.** In *AcousticPAD*, two HC-SR04 acoustic sensors [17] are employed to transmit and receive sound pulse signal. The sensor contains four pins, i.e., VCC, TRIG, ECHO and GND. All the pins are connected to the Raspberry Pi [18]. When *AcousticPAD* works, Raspberry Pi supplies 5 V and 0 V voltage to the VCC and GND pins of sensor, respectively. Once a TTL (Transistor-Transistor Logic) pulse signal that lasts at least 10µs is sent to the TRIG pin, the sensor automatically transmits 8 pulses signal at 40 KHz frequency and raises ECHO pin from low-level voltage to highlevel voltage. Then the sensor monitors the echo-back signal. If the echo-back signal is received, the sensor converts the ECHO pin from high-level voltage to low-level voltage. Thus the time duration of high-level voltage in ECHO pin is the time of flight (ToF) for ultrasonic signal.

**Distance Measurement.** Suppose that the time for transition of the ECHO pin voltage is T*<sup>s</sup>* (from low-level to high-level) and T*<sup>f</sup>* (from high-level to lowlevel), the ToF of ultrasonic signal is the difference between T*<sup>s</sup>* and T*<sup>f</sup>* . Thus the distance from the sensor to user's hand can be denoted as:

$$d = \frac{c \cdot (T\_f - T\_s)}{2} \tag{1}$$

where c is the speed of ultrasound signal in the air. For both acoustic sensors, the distances can be measured as the input of subsequent steps.

**Real-Time Character Segmentation.** With accurate distance measurements, we segment the consecutive handwriting inputs in realtime. As shown in Fig. 3, when the user's hand appears in the sensing area, the measured distance from two sensors denoted as d<sup>1</sup> and d<sup>2</sup> will decrease. This is because without sensing target, the sensors measure a long distance in the environment that are outside of the sensing area. While completing the character input, the measured distance increases after user's hand gets away from the sensing area. Thus we can set a threshold based on the sensing area to detect whether user's hand in the sensing area to segment the handwriting in realtime. When the distances of both acoustic sensors are less then a specific threshold, *AcousticPAD* starts to localize user's hand and track handwriting trajectory. Otherwise, the user finishes his input and *AcousticPAD* outputs the tracking results and feeds it to the recognition module.

**Hand Tracking.** We design a trilateration localization approach to localize user's hand and track handwriting trajectory. As is shown in Fig. 3, two acoustic sensors are deployed at the corners of the input panel (e.g., table surface). The distance s between two sensors is set in advance. With distance measurements from hand to sensors, the three edges of triangle OAB constituted by sensors and user's hand are known. According to the law of cosines [19], the angle of user's hand θ with respect to OA satisfies:

$$\theta = \arccos \frac{s^2 + d\_1^2 - d\_2^2}{2sd\_1} \tag{2}$$

Thus the postiion of user's hand is:

$$\begin{cases} x = d\_1 \cos \theta \\ y = d\_1 \sin \theta \end{cases} \tag{3}$$

For each distance measurement, *AcousticPAD* can obtain the position of user's hand. After successively connecting all the discrete positions during input, we are able to aquire handwriting performed by the user. Before feeding the handwriting into recognition module, we employs Savitzky-Golay filter [20] to smooth the handwriting. The filter utilizes linear least square method to fit successive subset of adjacent positions with a polynomial. After filtering out the noise, we obtain the final handwriting trajectory as the input of recognition module.

#### **2.3 Position-Independent Handwriting Recognition**

To achieve the position-independent handwriting recognition, we divide the recognition module into offline phase to train a cross-domain model with datasets from exisiting imaging dataset and online recognition with contactless acoutic system generated handwriting dataset.

**Offline Phase:** Different from existing work that needs to collect a large number of samples to build dataset and train a classification model, *AcousticPAD* leverages existing handwriting datasets, i.e., MNIST and EMNIST [15], as the training datasets without using acoustic sensing system generated data, which requires zero data collection effort. We notice that the data samples in MNIST are specific with certain style. If we directly utilize the original datasets to train a model, the model is position dependent with low generalization ability. To solve this problem, we propose a data augmentation approach to enhance the original datasets with wide applicability. Specifically, we first transform the character image samples with different rotation angles. Assume that the orientation of original image samples is 0◦, the images are duplicated, converted from 0◦ to 90◦ (anticlockwise) and *−*90◦ (clockwise) with a step size of 15◦. Secondly, we move the image samples in both vertical and horizontal directions with a step size of 14 pixels. After these operations, the augmented dataset is 65*×* larger than the original ones and contains image samples in different positions.

As shown in Fig. 2, we utilize the augmented dataset to train a three-layer BP neural network, which includes a input layer, a hidden layer and an output layer. Due to the size of images is 28*×*28 pixels, we convert them into a vector 1*×*784 as input. Thus the number of nodes in the input layer is 784. And we set the number of nodes in the hidden layer as 500. The number of nodes in the output layer is 10/26 for digit/letter recognition, respectively. During the training process, the image data are fed in the forward direction by propagating from the input layer through the hidden layer to the output layer, we can calculate the error between output value and expected value of output layer. Then the backward propagation of errors are applied to modify the connected weight values with learning rate 0.1. In our cases, all the samples in the augmented datasets are used to train the model, while the data collected by *AcousticPAD* system are for testing. Benefiting from the data augmentation, the trained model is robust against position changes and can achieve position-independent handwriting recognition.

**Online Phase:** For online phase, *AcousticPAD* takes the acoustic sensing system generated handwriting trajectory as input and leverages the model trained in offline phase to classify the handwriting characters. *AcousticPAD* employs a camera to record the real-time video of user's handwriting as ground truth. We develop a web based front end. After the user performs handwriting, the corresponding recognition result is displayed on the web page. *AcousticPAD* automatically segments the successive characters so that the user can continuously input numbers or letters using this system.

### **3 Evaluation**

### **3.1 Experiment Setup**

We have implemented AcousticPAD with two commodity cheap acoustic sensors (i.e., HC-SR04) [17] (5\$ per unit), and a Raspberry Pi 4B module [18]. The acoustic sensors transmit/receive acoustic pulse signals with 40 KHz. The operation voltage and current are DC 5 V and 15 mA, respectively. The trigger input signal is 10µs TTL pulse. We develop a web-based user interface to demonstrate the handwriting recognition in real time, as shown in Fig. 4. The demo video can be found at: https://youtu.be/sCZvK2rUzEU.

**Fig. 4.** Testbed and experiment scenario. **Fig. 5.** Hand tracking results of 10

digits and 26 letters.

#### **3.2 Performance Evaluation**

**Hand Tracking.** We first evaluate the hand tracking results. The volunteer performs hand gestures including 10 digits (0–9) and 26 letters (A-Z) in front of a table, and two small acoustic devices are placed at the table corners. As shown in Fig. 5, we show examples of all the digit and letter trajectories captured by our contactless handwriting system. It can be seen that our system is able to track the details of hand positions for writing all the digits and letters. With different angles of straight-line strokes and curves, the obtained trajectories can perfectly match the shapes of the characters.

**Character Recognition Accuracy.** Then we evaluate the character recognition accuracy. We let 10 participants perform handwriting experiments. For each character, the participants repeat 3 times with 5 different positions and 5 orientations, which construct a sample set with 27,000 samples in total (10 participants and 36 characters). We use the data augmented MNISIT dataset to train the BP neural network. All the acoustic sensing system collected data are used to test the recognition performance. Figure 6 illustrates the confusion matrix achieved by AcousticPAD for each character. The overall recognition accuracy for digits and letters are 92% and 90.3%, respectively. While the accuracy of some characters is relatively low (about 87%), for example, '3', 'Z', this might be because the users perform '3' and 'Z' with some irregular handwriting style which makes easier to confuse with other characters.

**Fig. 6.** Confusion matrix of recognition accuracy

**Position Independent Recognition.** To demonstrate the position independent capability, we conduct the handwriting of all characters at 5 different locations and 5 orientations, and compare the performance with BP neutral model [21] without data augmentation. Figure 7 shows the comparison results of accuracy for two methods. Our methods achieve above 86% accuracy for both five locations and orientations. Compared with the model without data augmentation, our method improves accuracy for more than 32% on average, which also demonstrates the robustness of AcousticPAD system.

**Fig. 7.** Recognition accuracy at different positions

### **4 Conclusion**

In this paper, we propose AcousticPAD, a contactless handwriting recognition system that employs acoustic pulse signals to enable input interactions with cheap acoustic sensors. AcousticPAD captures acoustic signals when hand moves on the table surface, and then track the handwriting trajectory to recognize the characters. We have implemented AcousticPAD as a real-time recognition system and conducted comprehensive experiments to validate the effectiveness and robustness of the system. Our results demonstrate high input recognition accuracies across different users with position-independent ability.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Novel Interactive BRAINTEASER Tools for Amyotrophic Lateral Sclerosis (ALS) and Multiple Sclerosis (MS) Management**

Sergio Gonzalez-Martinez1, María Fernanda Cabrera-Umpiérrez1, Manuel Ottaviano1, Vladimir Uroševi´c2(B) , Nikola Vojiˇci´c2, Stefan Spasojevi´c2, and Ognjen Mili´cevi´c2

<sup>1</sup> Life Supporting Technologies Group, Universidad Politecnica de Madrid, Av. Complutense 30, 28040 Madrid, Spain

<sup>2</sup> Belit d.o.o. (Ltd.) Research and Development Department, Trg Nikole Paši´ca 9, 11000 Belgrade, Serbia vladimir.urosevic@belit.co.rs

**Abstract.** The presented demonstrated working tools in the initial version constitute the foundation of the novel ALS and MS management and monitoring, leveraging extended IoT sensing and emerging instruments infrastructure, and a basis for integration of more advanced and effective AI models (in development) for disease progression prediction, patient stratification and ambiental exposure assessment.

**Keywords:** Amyotrophic lateral sclerosis · ALS · Multiple sclerosis · MS · REST services design · Architecture · Optimization · AI models · Rich interactive web applications · Disease progression · Relapse · Big Data · Environment · Ambient · Daily living activities · Exposure · Exploratory analytics · Neurodegenerative · Smartwatches · Well-being · Wearable sensors · IoT integration

### **1 Introduction and Scientific/Technological Background**

The BRAINTEASER Project (**BR**inging **A**rtificial **INTE**lligence home for a better care of **A**myotrophic lateral **S**clerosis and multiple scl**ER**osis), funded from the European Commission Horizon 2020 programme grant number GA101017598 until 2024, aims to integrate societal, environmental and health data to develop patient stratification and disease progression models for Amyotrophic Lateral Sclerosis (ALS) andMultiple Sclerosis (MS). ALS and MS are both very complex chronic progressively degenerative neurological diseases significantly disrupting the quality of life of the patients and their families, but with very different clinical picture, evolution, prognosis and therapies.

BRAINTEASER integrates detailed retrospective and prospective clinical datasets with comprehensive heterogeneous personal health, activity, lifestyle, habitual/behavioural, and environmental data collected using commonly available sensing/IoT devices and the demonstrated first release of developed interactive tools for disease monitoring and management.

The collected data drive the development of Artificial Intelligence (AI) models able to address the needs of precision medicine, enabling early risk prediction of disease fast progression and adverse events, and the next planned releases of the tools currently in development will focus on the integration of the AI models and their outputs into generated specific highly complex feedbacks to the users (digital gamified coaching and recommendations to improve the patient's and caregiver's daily routines and overall well-being, and the analytics suite for full patient cohorts management according to stratifications output of the AI models and for interactive visual data exploration and detailed understanding of the disease progress and evolution of patient's conditions).

The demonstrated tools are being devised and developed embracing an agile usercentered multidisciplinary co-design approach, accounting for the specific technical, medical (including psychological/cognitive) and societal needs of the users. Proof-ofconcept of their validated use in real clinical settings in 4 clinical study validation sites (Lisbon, Madrid, Pavia & Turin) aims to provide quantitative evidence of benefits and effectiveness of leveraging AI models in healthcare pathways for dire neurodegenerative diseases, and a set of recommendations for the health authorities to support the transition of the current approach to healthcare from reactive to predictive, paving the path towards a healthier and more fulfilling life as long as possible.

**Fig. 1.** Main data streams across the high-level BRAINTEASER ecosystem organization.

The presented tools are primary end-user-oriented components of the overall BRAINTEASER platform ecosystem, supported by the underlying unified Data Platform, Semantic Cloud, middle tier of integration service APIs, and others as presented on Fig. 1 above with relevant main data streams and the involved external relations, including Project Open Science and Evaluation Challenge actions already underway. All together in turn provide support for the identified processes enhancing the management of ALS and MS both throughout the Project demonstration study and the care provision practice beyond, differentiated and structured through comprehensive analysis and co-design with relevant stakeholders as presented on Fig. 2 below, and progressing beyond the thoroughly analyzed key features of current relevant state-of-the-art ALS & MS management tools and applications like Emilyn1, Cleo2, and over 50 others relevant analyzed and compared in terms of specific features and functionalities.

**Fig. 2.** Overall BRAINTEASER process flows enhancing disease management of ALS and MS.

Some of the platform functionalities and architecture build and extend upon preceding and parallel related projects and initiatives tackling common issues and exploiting applicable methods in neurodegenerative progression management [7], well-being [8], functional decline prevention [9], and public health [4, 5], such as PULSE3, City4Age4, NEVERMIND5, or VERITAS6.

One of the set out goals of BRAINTEASER is also to analyze the possible effects and influences of environmental and daily living ambient factors (primarily air pollution) on the aetiology and progression of neurodegenerative sclerotic diseases like MS, as has been intensely studied in recent years [1–3] along with water and soil pollution, and the role of these factors on progression and relapse of MS and ALS, feeding the relevant collected environmental data into the dedicated developed AI models for patient exposure to the ambiental factors, complementing the models for disease progression and stratification. The cities with clinical trial sites involved in the project (Lisbon, Madrid, Pavia,

<sup>1</sup> https://play.google.com/store/apps/details?id=com.bxh\_mvp&hl=es.

<sup>2</sup> https://www.cleo-app.es/index.html.

<sup>3</sup> https://www.project-pulse.eu.

<sup>4</sup> https://cordis.europa.eu/project/id/689731.

<sup>5</sup> https://www.nevermindproject.org.

<sup>6</sup> https://cordis.europa.eu/project/id/247765.

Torino) have established extensive environmental sensing and measuring infrastructure, ingesting data both from public environmental measuring stations wherever open or available, as well as from connected IoT sensing devices deployed by BRAINTEASER and previous or parallel relevant projects like (again) PULSE, eMOTIONAL Cities7, Sharing Cities8, ACROSSING9 and others [4–6], and complemented also with data from portable/wearable compact sensing devices (carried on or with patients) deployed by the project, mostly Atmotube PRO monitors10.

### **2 Interactive Disease Management and Monitoring Tools Features**

Three main interactive tools are being developed to support the processes illustrated on Fig. 2 above, differentiated and tailored towards specific target end-user roles and distinctly siloed functionality groups required for each group of roles:


In this initial release, the mobile app. for patients/caregivers and the Clinical web Tool are primary, to support the key required functionalities of the Project study, enrolling the patients and:


as diagrammed on Fig. 3 below.

Next subsequent version in current development is focused on key "interventionrelated" features listed on bottom left and bottom right of Fig. 3, providing support to

<sup>7</sup> https://emotionalcities-h2020.eu.

<sup>8</sup> www.sharingcities.eu

<sup>9</sup> www.acrossing-itn.eu

<sup>10</sup> https://atmotube.com/atmotube-pro.

patients, caregivers and clinicians to fully manage, prevent and mitigate disease progression, from basic advisory guidance and instructions to advanced cognitive games and exercises, or voice/speech assessment in evolution of bulbar symptoms.

**Fig. 3.** Conceptual schema of primary functionalities of the BRAINTEASER Patient App. and Clinical Tool, with key data flows.

Example screen form for support service configuration in the MS-specific variant of the Clinical Tool is provided on Fig. 4 below and the detailed breakdown of UML use cases notation of basic functionalities of the Patient App. on Fig. 5 below.


**Fig. 4.** Clinical Tool screen example for MS service configuration.

**Fig. 5.** UML notation of use cases covering key functionalities of the Patient/caregiver app.

The third listed Interactive Data Management and Exploration (dashboard-based) Tool is in initial version development as it significantly depends on and is intended to integrate substantial output from the AI models which are still in development and testing, but at this stage provides at least rich interactive visual exploration of arbitrary selection of collected temporal and heterogeneous data variables (including categorized and continuous), with values evaluated or compared combined together in multimodal and multidimensional composite 2D and 3D visualizations, as exemplified on Fig. 6 below.

**Fig. 6.** Multivariate composite 3D interactive exploratory data visualization of 7 different combined continuous and categorized variables, additionally stratified per gender and marital status demographics.

### **3 Key Architectural and Service Design Challenges**

The tools are designed to be modular, dynamically adaptable, personalized and supporting alternative user interaction modalities (with significantly higher accessibility than common for "ordinary" apps, adapting to the increasing manual dexterity impairments of the patients), with simple and scalable navigation structure, optimal balance between menu structure and dynamic interaction flows, and designed for long-term usage, in the UI/UX aspects.

**Fig. 7.** BRAINTEASER ecosystem architecture with main data sources, consumers and flows, and detailed service tier breakdown.

In the overall architecture of the underlying supporting BRAINTEASER data collection, management, and provision ecosystem (Fig. 7 above), similar microservice modularity methodological principles have been followed, including the separation between domain-specific and infrastructural/utility orthogonal logic, and taking into account the system and service performance and security aspects built into the design and architecture from the start, employing loose coupling and service orchestration patterns like the baseline *API Gateway* facade pattern that may evolve into more comprehensive orchestration pattern (like ESB) if complexity or performance criticality increases in the ongoing iterative further advancement of the architecture.

### **4 Conclusion**

The presented demonstrated working tools in the initial version constitute the foundation of the novel ALS and MS management and monitoring leveraging extended IoT sensing and emerging instruments infrastructure, and a basis for integration of more advanced and effective AI models (in development) for disease progression prediction, patient stratification and ambiental exposure assessment.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Author Index**

Abdulrazak, Bessam 3, 43, 100 Abubakar, Aisha 284 Al Wardani, Farah 141 Asvadi, Alireza 18 Baillargeon, Dany 100 Bonnici, Norbert 154 Booth, Richard 86 Boudy, Jérôme 257 Boughattas, Naouel 225 Bouzidi, Dalenda 234 Cabrera-Umpiérrez, María Fernanda 302 Carter, Jarai 154 Chikhaoui, Belkacem 183 Chollet, Gérard 257 Codjo, Josué Ayi 3 Corring, Deborah 86 Dubey, Mohnish 257 Erritali, Mohammed 272 Fakhfakh, Ahmed 234 Fernandez, Gabriela 154 Forchuk, Cheryl 86 Frampton, Barbara 86 Fu, Xiaolai 293 Ghozzi, Fahmi 234 Gonzalez-Martinez, Sergio 302 Hadj Kacem, Ahmed 166 Hadj Kacem, Mohamed 30, 166 Haider, Ghani 73 Hoch, Jeffrey S. 86 Jabnoun, Hanen 225 Jang, Wan-ho 211, 246 Jin, Beihong 293 Jo, Sun-young 211, 246 Jokinen, Kristiina 257

Kacem, Ahmed Hadj 58 Kalboussi, Anis 58 Khalfi, Najeh 30 Khan, Maheen 73 Khan, Rida Zahid 73 Kim, Dong-wan 217 Kim, Dongwan 266 Kim, Eun-joo 211 Kim, Eun-ju 246 Kim, Hogene 112 Kim, Hyun-kyung 211, 246 Kim, Jae-nam 211, 246 Kim, Jongbae 112, 217, 266 Kim, Min-kyung 211, 246 Kumar, Anand 73 Leduc, Benoit 125 Lee, Seungbok 112 Lee, Yun-hwan 217 Lee, YunHwan 266 Lizotte, Daniel 86 Lohr, Christophe 18, 125, 257 Madani, Youness 272 Maione, Carol 154 Mann, Rupinder 86 Mannai, Zayneb 58 Maraoui, Souhail 43, 100 Matsui, Tomokazu 196 Matsumoto, Kanta 196 Mcheick, Hamid 141 McTear, Michael 257 Meddaoui, Mohamed Amine 272 Mili´cevi´c, Ognjen 302 Mitriakov, Andrei 18 Moon, Kwang-tae 217 Moon, KwangTae 266 Mössing, Wanja 257 Msheik, Batoul 141

Najeh, Houda 125 Nasser, Youmna 141 Niu, Kai 293

Oh, Yim-Taek 112 Ottaviano, Manuel 302 Oyegoke, Ayandunmola Folake 284

Papadakis, Panagiotis 18 Paul, Suvrojoti 3, 43 Provencher, Véronique 100

Raza, Syeda Saleha 73 Rezaei, Amin 43 Roelen, Sonja Dana 257 Rudnick, Abraham 86

Saidi, Abdessamad 30, 166 Sailhan, Françoise 272 Serrato, Jonathan 86 Shi, Heng 183 Shim, Sun-hwa 211, 246

Spasojevi´c, Stefan 302 Suwa, Hirohiko 196 Tahir, Sahar 100 Tounsi, Imen 30, 166 Tsou, Ming-Hsiang 154 Uroševi´c, Vladimir 302 Vito, Domenico 154 Vojiˇci´c, Nikola 302 Wang, Shengrui 183 Wieching, Rainer 257 Xiao, Tianqi 43 Yang, Ha-yeon 211, 246 Yasumoto, Keiichi 196 Zaballa, Karenina 154

Zhang, Fusang 293