**Kim Jongbae · Mounir Mokhtari · Hamdi Aloulou · Bessam Abdulrazak · Lee Seungbok (Eds.)**

# **Digital Health Transformation, Smart Ageing, and Managing Disability**

**20th International Conference, ICOST 2023 Wonju, South Korea, July 7–8, 2023 Proceedings**

# **Lecture Notes in Computer Science 14237**

Founding Editors

Gerhard Goos Juris Hartmanis

## Editorial Board Members

Elisa Bertino, *Purdue University, West Lafayette, IN, USA* Wen Gao, *Peking University, Beijing, China* Bernhard Steffen , *TU Dortmund University, Dortmund, Germany* Moti Yung , *Columbia University, New York, NY, USA*

The series Lecture Notes in Computer Science (LNCS), including its subseries Lecture Notes in Artificial Intelligence (LNAI) and Lecture Notes in Bioinformatics (LNBI), has established itself as a medium for the publication of new developments in computer science and information technology research, teaching, and education.

LNCS enjoys close cooperation with the computer science R & D community, the series counts many renowned academics among its volume editors and paper authors, and collaborates with prestigious societies. Its mission is to serve this international community by providing an invaluable service, mainly focused on the publication of conference and workshop proceedings and postproceedings. LNCS commenced publication in 1973.

Kim Jongbae · Mounir Mokhtari · Hamdi Aloulou · Bessam Abdulrazak · Lee Seungbok Editors

# Digital Health Transformation, Smart Ageing, and Managing Disability

20th International Conference, ICOST 2023 Wonju, South Korea, July 7–8, 2023 Proceedings

*Editors* Kim Jongbae Yonsei University Wonju, Korea (Republic of)

Hamdi Aloulou Digital Research Centre of Sfax Sfax, Tunisia

Lee Seungbok Yonsei University Wonju, Korea (Republic of) Mounir Mokhtari Institut Mines Télécom Paris, France

Bessam Abdulrazak Université de Sherbrooke Sherbrooke, QC, Canada

ISSN 0302-9743 ISSN 1611-3349 (electronic) Lecture Notes in Computer Science ISBN 978-3-031-43949-0 ISBN 978-3-031-43950-6 (eBook) https://doi.org/10.1007/978-3-031-43950-6

© The Editor(s) (if applicable) and The Author(s) 2023. This book is an open access publication.

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Paper in this product is recyclable.

## **Preface**

This year we organized the 20th edition of the International Conference on Smart Living and Public Health (ICOST 2023), a series that has succeeded in bringing together a community from different continents for 20 years and raised awareness about frail and dependent people's quality of life in our societies.

After 19 very successful conferences held in France (2003, 2009, 2017, 2022), Singapore (2004, 2013, 2018), Canada (2005, 2011), the UK (2006), Japan (2007), the USA (2008, 2014, 2019), South Korea (2010), Italy (2012), Switzerland (2015), China (2016), and Tunisia (2020), this 20th edition of the conference was organized by Yonsei University, Wonju, Republic of Korea, and was held during July 7–8, 2023. The theme of the conference was "Digital Health Transformation and Smart Ageing in place with & to disability."

ICOST 2023 provided a premier venue for the presentation and discussion of research on the design, development, deployment, and evaluation of AI for health, smart urban environments, assistive technologies, chronic disease management, and coaching and health telematics systems. ICOST 2023 aimed to understand and assess the diverse and disparate impact of digital technologies on public health in developing and developed countries. It brought together stakeholders from health care, public health, academia, and industry along with end users and family caregivers to explore how to utilize technologies to foster health, disease prevention, and independent living, and offer enhanced quality of life. The ICOST 2023 conference featured a dynamic program incorporating a range of oral and poster presentations, along with panel sessions.

ICOST 2023 was proud to extend its hospitality to an international community consisting of researchers from major universities and research centers, representatives from industry, and users from 17 different countries. We would like to thank the authors for submitting their current research work and the Program Committee members for their commitment to 41 submissions were received and were sent for review. All papers have undergo a double-blind review process with an average of 3 reviewers per paper. 18 papers have been accepted as full papers with an acceptance rate of around 44%. The ICOST proceedings and chapters have now reached over 400,000 downloads and are in the top 25% of downloads of Springer LNCS. We are extremely thankful to our sponsors for their commitment and support to the vision and mission of ICOST.

July 2023 Kim Jongbae Mounir Mokhtari Hamdi Aloulou Bessam Abdulrazak Lee Seungbok

## **Organization**

## **General Chair**

Jongbae Kim Yonsei University, Republic of Korea

## **Conference Co-chairs**


## **Scientific Advisory Board**


## **Program Committee**

#### **Chairs**


#### **Members**

Aitor Almeida University of Deusto, Spain Christophe Lohr IMT Atlantique, France Fulvio Mastrogiovanni University of Genoa, Italy Hisato Kobayashi Hosei University, Japan Houssem Aloulou University of Sfax, Tunisia

Sahar Tahir Université de Sherbrooke, Canada

Afef Mdhaffar University of Sfax, Tunisia Anthony Fleury Université Lille Nord de France, France Anuragini Shirish Institut Mines-Télécom - Business School, France Armel Ayimdji Tekemetieu Université de Sherbrooke, Canada Belkacem Chikhaoui University of Quebec, Canada Charles Gouin-Vallerand Université de Sherbrooke, Canada Christina Khnaisser Université de Sherbrooke, Canada Eric Campo CNRS LAAS Toulouse, France Hamid Mcheick University of Quebec at Chicoutimi, Canada Hongbo Ni Northwestern Polytechnical University, China Hubert Kenfack Ngankam Université de Sherbrooke, Canada Hui Chen Université de Sherbrooke, Canada Ickpyo Hong Yonsei University, Republic of Korea Iyad Abuhadrous Palestine Technical College, Palestine Jeffrey Soar University of Southern Queensland, Australia Laurent Billonnet University of Limoges, France Manfred Wojciechowski University of Applied Sciences Düsseldorf, Germany Meriem Zerkouk Université de Sherbrooke, Canada Muhammed Abdelnaby-Sadek Université de Sherbrooke, Canada Nadine Vigouroux Institut de Recherche en Informatique de Toulouse, France Naser Dehghan Université de Sherbrooke, Canada

Vladimir Uroševi´c Belit Ltd., Serbia

Salim Hima ESME-SUDRIA, France Sergio Copelli MultiMed Engineers SRLS, Italy Sha Zhao Zhejiang University, China Stefanos Kollias University of Lincoln, UK Tayeb Lemlouma Rennes I University, France Wael Sellami University of Sfax, Tunisia Walid Hassairi University of Monastir, Tunisia Won-Kyung Song National Rehabilitation Center, Republic of Korea Yves Demazeau CNRS – LIG, France Zuraimi Sultan Berkeley Education Alliance for Research in Singapore (BEARS), Singapore

## **Organizers**

Yonsei University, Republic of Korea Gangwon Regional Innovation Platform, Republic of Korea Make Level Paths, Republic of Korea Wheelopia co., Ltd., Republic of Korea Institut Mines-Télécom, France Université de Sherbrooke, Cananda Digital Research Center of Sfax, Tunisia National University of Singapore, Singapore

## **Sponsors**

Digital Healthcare Center, Yonsei University, Republic of Korea h.well National Health Insurance Service, Republic of Korea Wonju Medical Industry Technovalley, Republic of Korea Emma Healthcare, Republic of Korea InBody, Republic of Korea Office of Research Affairs/Industry-Academic Cooperation Foundation, Republic of Korea

## **Contents**

#### **IoT and AI solutions for E-Health**

**Biomedical and Health Informatics**


#### Towards a Medical Test Results Management System Based on Blockchain, Smart Contracts, and NFT: A Case Study in Vietnam . . . . . . . . . . . . . . . . . . . . . . . 79 *N. T. Anh, V. H. Khanh, N. T. Phuc, T. D. Khoa, H. G. Khiem, N. D. P. Trong, V. C. P. Loc, N. T. Q. Duy, T. Q. Bao, D. M. Hieu, H. T. Nghia, N. T. K. Ngan, and L. H. Huong*


#### **Wellbeing Technologies**


Contents xiii


#### **Short Contributions: Wellbeing Technologies**


# **IoT and AI solutions for E-Health**

# **Real-time Pilates Posture Recognition System Using Deep Learning Model**

Hayoung Kim, Kyeong Teak Oh, Jaesuk Kim, Oyun Kwon, Junhwan Kwon, Jiwon Choi, and Sun K. Yoo(B)

Department of Medical Engineering, Yonsei University, Seoul, South Korea sunkyoo@yuhs.ac

**Abstract.** As the pandemic situation continues, many people exercise at home. Mat Pilates is a popular workout and effective core strengthening. Although many researchers have conducted pose recognition studies for exercise posture correction, the study on Pilates exercise is only one case on static images. Therefore, for the purpose of exercise monitoring, we propose a real-time Pilates posture recognition system on a smartphone for exercise monitoring. We aimed to recognize 8 Pilates exercises—Bridge, Head roll-up, Hundred, Roll-up, Teaser, Plank, Thigh stretch, and Swan. First, the Blazepose model is used to extract body joint features. Then, we designed a deep neural network model that recognizes Pilates based on the extracted body features. It also measures the number of workouts, duration, and similarity to experts in video sequences. The precision, recall, and f1-score of the posture recognition model are 0.90, 0.87, and 0.88, respectively. The introduced application is expected to be used for exercise management at home.

**Keywords:** Pilates · Pose recognition · Pose correction · Deep neural network · Real-time · Smartphone

## **1 Introduction**

The interest in digital home training programs has increased since the outbreak of COVID-19 as they are accessible and cost-effective to exercise [1]. Among the home exercises, Pilates is a popular exercise that has recently become increasingly widespread in rehabilitation therapy [2]. Pilates was developed by Joseph Pilates of Germany and is a full-body exercise made for rehabilitating patients during World War I [3]. The Pilates mat exercise program is effective in treating chronic low back pain by evaluating core muscle thickness [4]. In addition, Pilates exercise without using any equipment is useful for improving respiratory functions and disease-related symptoms [5]. However, an injury may occur by repeating the wrong motion without experts. Therefore, a Pilates posture recognition system for home training is required.

In computer vision, exercise training systems through human posture estimation are being developed to increase exercise effectiveness and prevent injury in various sports fields [6]. Dittakavi, B. [7] conducted a study to classify postures from fixed images using probability techniques for Yoga, Pilates, and Kung Fu exercises and to explain which joint motion was incorrect. Wu, Y. [8] classified 45 yoga movements and created a model that gave points to the exercise movements without an expert. They extracted image features using a convolutional neural network (CNN) and trained the model using a unique contrastive loss that combines the L2 norm with cosine similarity. Li, J. [9] presented a model for classifying and evaluating 117 yoga movements. Yoga movements from 22 subjects were measured with an RGB-D camera. At this time, 3D coordinates were corrected by taking pictures from both the front and side. In addition, a new Cascade 2S-AGGN (Cascade graph convolutional neural network for yoga pose classification and assessment) model was presented by constructing a graph convolutional neural network (GCN) in a hierarchical structure. Zhao, Z [10] also used GCN to classify three types of motion and presented a model for correcting posture. The expert's motion and the subject's motion were trained separately and evaluated the subject's motion. Dynamic time warping is used to compensate for the difference in each person's movement speed. In previous studies, there is one case of Pilates exercise [7] correction of static images. However, Pilates pose correction studies on video sequences are necessary since Pilates involves segmental movements of the spine. Therefore, we studied Pilates exercise correction on video sequences.

The purpose of this study is to develop a real-time Pilates mat exercise recognition system on a smartphone for exercise management at home. We developed a Pilates exercise posture classification model that automatically recognizes the 8 Pilates exercises. We even added a parameter measurement function for exercise monitoring. We finally developed a real-time posture recognition system on the smartphone for user convenience.

"Methods" describe a data set of Pilates exercises, a posture recognition method, and real-time exercise monitoring capabilities. In the "Results", the results of the Pilates posture recognition model and the real-time monitoring system are mentioned. In the "Discussion", the results are discussed. In "Conclusion", the conclusion is described.

## **2 Methods**

#### **2.1 Data Collection**

We have selected 8 Pilates exercises as follows: Bridge, Head roll-up, Hundred, Roll-up, Teaser, Thigh stretch, Plank, and Swan. The example of the selected Pilates exercises is shown in Fig. 1.

The Pilates data were acquired with the camera facing the exercise mat and at a distance of 2.5m. We used the front camera of the Galaxy 22 smartphone. The 8 movements are repeated about 5 times and recorded as videos. The image acquisition sampling rate is 30 fps. A total of 15 subjects of videos were acquired, and the sex ratio was 6 males and 9 females.

We extracted features of 33 joints from the acquired videos using the Blazepose model [11] to train and test the Pilates recognition model. In many posture recognition studies, Blazepose was used because of its high accuracy and performance [12–14]. The 33 joint features are the location of the x, y, and z coordinates and the visibility indicating whether the joint is visible. 9 target postures were manually labeled for each image of the video sequences. It includes 8 Pilates postures and an "unknown" class to distinguish it from the prescribed exercises. The entire movement from the initial to the final position is trained, and the information of one frame is input into the model. The 15 subjects' videos are split into 11 and 4 for training and test data. The total number of Pilates classification data is 247,203. The training data is 179,446, and the test data is 67,757. The number of samples for each 8 Pilates postures is shown in Table 1.

**Fig. 1.** Example of Pilates poses. (a) Bridge, (b) Head roll up, (c) Hundred, (d) Roll up, (e) Teaser, (f) Thigh stretch, (g) Plank, (h) Swan.


**Table 1.** The number of Pilates postures recognition samples.

## **2.2 Pose Recognition Model**

The Pilates posture recognition model is designed to automatical classify 8 Pilates and unknown postures. We designed a simple deep neural network to deploy on an Android phone. The input size of the model is 1 × 132 because the x, y, z, and visibility values of 33 joint points from on frame are entered. Three fully connected layers with 128, 64, and 16 neurons are stacked. A dropout rate of 0.2 was used between each fully connected layer to prevent overfitting. Finally, a soft-max layer for classifying multiple classes was connected. The output of the model is probabilities of 9 poses.

## **2.3 Real-Time Exercise Monitoring System**

We propose a Pilates posture recognition system to operate in real-time on the smartphone. The sequence was designed to extract the number of exercises, exercise duration, and similarity with expert posture to monitor Pilates exercise. To operate the system on the smartphone, Mediapipe [15] framework was used.

**Real-Time System Architecture.** The architecture of the posture recognition system is shown in Fig. 2. First, in the pose estimation section, the features of one person's 33 joints are estimated by the Blazepose model when the input image comes through the camera. In the next pose recognition section, the pose recognition model classifies 8 Pilates exercises using the pose features. After then, since there is flickering in which the model predicts a different posture while predicting posture, a moving average filter is added that uses the average value of the last 10. Then, the pose counter & time measure section counts the number of exercises and measures the exercise duration. In the pose correction section, the similarity of the expert and current posture is calculated using joint features and recognition results. Finally, these exercise measurement parameters are combined in the draw overlay image section to generate an output image, and the result is displayed on the screen.

**Fig. 2.** Architecture of the Real-time Pose Monitoring System

**Recognition of the Up-and-down Movement.** The up-and-down for one posture was recognized using the Euclidean distance differential between two specific joints. It is used for the following function, posture counting and correction. In the Bridge pose, the hips move away from the ankles as they lift up and come closer as they lower down, recognized as Bridge-up by ankle-to-hip distance increase. Similarly, Head roll-up, Roll-up, and Teaser are identified by ear-to-knee distance decrease. Thigh stretch-up is recognized by hip-to-foot index distance increase. Plank and Swan up-and-down movements are detected using hip-to-elbow and ear-to-elbow distance differentials, respectively.

**Exercise Count and Time Measurement.** Exercise parameters for Pilates postures are shown on the screen for users to monitor exercise counts and duration. The number of exercises and duration were determined through pose recognition results and recognition of up-and-down posture. The Hundred Posture, a sustained position, was counted independently without distance comparison. For other postures, counts were measured when transitioning from up to down or vice versa. Every exercise time was measured by starting when each exercise was recognized and ending when the motion was not recognized.

**Pose Correction.** Joint angles were used as feedback parameters for Pilates postures. To enhance workout effectiveness, it is needed to pay attention to correct posture. The similarity between the current and the expert's posture was assessed by comparing angles. The method for calculating the angle of each joint is as follows. As shown in Fig. 3 (a), when there are three joint points X1, X2, and X3, the radian angle was obtained by Eq. (1). And radian angles were converted to degree units.

$$\theta = \cos^{-1}\frac{\mathbf{x}\_1 - \mathbf{x}\_2}{\mathbf{y}\_1 - \mathbf{y}\_2} - \cos^{-1}\frac{\mathbf{x}\_3 - \mathbf{x}\_2}{\mathbf{y}\_3 - \mathbf{y}\_2}. \tag{1}$$

Angles were compared to reference angles using 4 specific pairs, as depicted in Fig. 3 (b). These pairs include the shoulder angle between the ears and hip, hip angle between shoulder and knee, knee angle between hip and ankle, and ankle angle between knee and foot index. Pilates is primarily a core strengthening exercise, and the arms play an auxiliary role in balancing, so the joint angles around the torso were selected.

A weighted joint angle difference method was used to compare postures with experts. The expert posture that the reference for posture comparison was acquired from YouTube videos. One cycle of exercise motions to be corrected is each of the previously recognized up posture and down posture. The one-cycle of exercise motion was compared with the expert by calculating each section's average joint angle difference divided into 10 sections. Then the angle difference was multiplied by each joint's confidence score. Calculate the weighted joint angle difference for all 10 cycles and 8 joints as Eq. (2). Joints with high confidence scores were weighted to have more influence on the similarity measurement, the idea of the weighted distance method [16]. Finally, the angle difference was converted to a similarity score by normalizing 180 degrees as Eq. (3).

$$A\_{\rm diff} = \frac{1}{P} \cdot \frac{1}{J} \sum\_{l=1}^{P} \sum\_{i=1}^{J} V\_{l,j} \left( A\_{l,j}^{s} - A\_{l,j}^{r} \right), i \in (1, 10), j \in (1, 8). \tag{2}$$

$$Score = \frac{180 - A\_{diff}}{180}.\tag{3}$$

where *Adiff* denotes a weighted joint angle difference. The i and j denote period and joint index, respectively. *Vi*,*<sup>j</sup>* is average visibility of the i-th period and j-th joint. *A<sup>s</sup> <sup>i</sup>*,*<sup>j</sup>* denotes subject's average angle of the i-th period and j-th joint. *A<sup>r</sup> <sup>i</sup>*,*<sup>j</sup>* denotes reference average angle of the i-th period and j-th joint.

**Fig. 3.** Pose Angle. (a) Joint angle calculation using two coordinates, (b) main angle used for posture comparison.

## **3 Results**

#### **3.1 Result of Pose Recognition Model**

We conducted an experiment with 67,757 test samples from 4 subjects. Performance metrics for evaluating posture classification models used Precision, Recall, and F1-score [17], which are widely used in the performance evaluation of classification models. The formulas are (4), (5), and (6), respectively.

$$Precision = \frac{TP}{TP + FP}.\tag{4}$$

$$Recall = \frac{TP}{TP + FN}.\tag{5}$$

*<sup>F</sup>*<sup>1</sup> <sup>−</sup> *score* <sup>=</sup> <sup>2</sup> · *recall* · *precision recall* <sup>+</sup> *precision*. (6)

The precision, recall, and f1-score of the recognition model are 0.90, 0.87, and 0.84, respectively. The results of three metrics for 9 classes are shown in Table 2.

#### **3.2 Results of Real-Time Exercise Monitoring System**

**Results of ExerciseMonitoring System on Test Data.** The exercise monitoring system was tested with videos of 4 subjects on the desktop CPU. These videos are the same data used to test the pose recognition model. Table 3 shows the exercise counts for each subject in the 4 test videos. The last image of the Pilates monitoring system for each subject video is shown in Fig. 4. In Fig. 4, the first line indicates the current posture's


**Table 2.** Result of Pilates postures recognition model.

count, duration, and the previous posture's similarity score. Every 8 exercise's count, duration, and similarity score were also displayed on the screen. Figure 5 shows an example of similarity score comparison in the Roll-up posture. In the Roll-up posture, subject 2's score, who stood with his legs stable on the floor (Fig. 5 (a)), was 0.81 (Fig. 5 (b)), but subject 3, who came up using the recoil of bending his knees (Fig. 5 (c)), scored 0.77 (Fig. 5 (d)).


**Table 3.** The number of Pilates exercises for test data.

**Results of Real-time system on a smartphone.** We checked the operation of the realtime Pilates exercise monitoring system on the Galaxy 22 smartphone. Model inference ran on the GPU, and the rest of the calculations ran on the CPU. A total of 8 exercises were performed twice, and the screen of the app was recorded. Table 5 shows the number of each Pilates exercises performed for test 1 and test 2. The last image of the application for each test is shown in Fig. 6. The number of exercises, time, and similarity score for

**Fig. 4.** The result of the Pilates posture monitoring system's count, time, and similarity score on test data. (a) subject 1, (b) subject 2, (c) subject 3, (d) subject 4.

**Fig. 5.** Example of Roll-down posture score comparison. (a) Subject 2's roll-up posture. The score is the previous roll-down posture, (b) Subject 2's roll-down posture. The score is the previous rollup posture, (c) Subject 3's roll-up posture. The score is the previous roll-down posture, (d) Subject 3's roll-down posture. The score is the previous roll-up posture.

each motion are displayed. Figure 7 shows examples of scores and count results for the Teaser and Plank postures in the two tests (Table 4).




**Fig. 6.** The result of the real-time Pilates exercise monitoring application on a smartphone. (a) test 1, (b) test 2.

#### **4 Discussion**

We conducted a pose recognition and correction on video sequences for home Pilates exercise monitoring. Since there is no open data set for Pilates exercise, datasets were newly acquired. 8 Pilates exercises were selected as easy for beginners to follow and good for back pain prevention, abdominal exercise, and stretching.

**Fig. 7.** Example results of Teaser and Plank in Tests 1 and 2 of the real-time Pilates exercise application. (a) Teaser up of test 1, (b) Teaser down of test 1, (c) Plank down of test 1, (d) Plank up of test 2, (e) Teaser up of test 2, (f) Teaser down of test 2, (g) Plank down of test 2, (h) Plank up of test 2.

Using the Blazepose model and a simple neural network, it showed recognition of 8 Pilates exercises. Most errors in the pose recognition models occurred between the 8 target exercises and the unknown class. There were also recognition errors in lying postures such as Hundred, Roll-up, Teaser, and prostrating postures between Swan and Plank since the entire movement was trained. Therefore, a moving average filter was used to prevent the flickering of class prediction in the middle of the video sequences. The number of exercises, exercise duration, and similarity score with experts were calculated using the pose recognition model's prediction. Most count errors in the test videos are due to recognition errors in rollups and teasers (Fig. 4). We confirmed that all exercises counted well except head up, teaser, and plank on the smartphone. In both test1 and test2, the 2 Head-ups were not counted because the head was not raised enough in the head-up posture (Fig. 6). In test1, 1 Teaser was not counted (Fig. 6 (a)) because the subject fell so quickly as not recognized up-down (Fig. 7 (a), (b)). The remaining 4 Teasers are counted, but the score is 0 because they did not exercise enough to compare the movements (Fig. 6 (a)). However, in test 2, all the Teasers are counted and scored as enough exercise (Fig. 7 (e), (f)). Meanwhile, In Test 1, all the Planks were counted (Fig. 6 (a)) as the postures were well recognized (Fig. 7 (c), (d)). In test 2, 5 Planks were not counted (Fig. 6 (b)) because they were recognized as unknown due to the wrong motion of lifting the hip (Fig. 7 (g), (h)). With the development of our proposed exercise monitoring app, we expect that users will be able to receive Pilates exercise feedback regardless of location [16].

**Limitations and Future Work.** Although there were some technical limitations, we have plans for improvement. Since the data was acquired on only direction, there is a limitation to posture recognition. It is necessary to acquire data from various angles. The errors in posture recognition not only affect count and time measurements but also pose correction. Therefore, our future plans involve developing recognition models using time series data to enhance performance. Additionally, we aim to implement Devanne, M. [18]'s method to provide detailed feedback on specific body parts and overall posture.

## **5 Conclusion**

In this paper, we propose a Pilates exercises monitoring system on a smartphone in realtime. First, we acquired video sequences for 8 Pilates postures. Pilates postures were recognized using a deep learning model on video sequences. In addition, exercise count and time measurement function for measuring exercise volume were added. We also proposed a weighted joint angle distance method that measures the angle and movement of major joints and compares posture with experts. It is expected that Pilates exercises can be corrected at home without experts.

**Acknowledgments.** This work was supported by the Industrial Technology Innovation Program (No. 20012603, Development of Emotional Cognitive and Sympathetic AI Service Technology for Remote (Non-face-to-face) Learning and Industrial Sites) funded By the Ministry of Trade, Industry and Energy (MOTIE, Korea).

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **BRAINTEASER Architecture for Integration of AI Models and Interactive Tools for Amyotrophic Lateral Sclerosis (ALS) and Multiple Sclerosis (MS) Progression Prediction and Management**

Vladimir Uroševi´c1(B) , Nikola Vojiˇci´c1, Aleksandar Jovanovi´c1, Borko Kosti´c1, Sergio Gonzalez-Martinez2, María Fernanda Cabrera-Umpiérrez2, Manuel Ottaviano2, Luca Cossu3, Andrea Facchinetti3, and Giacomo Cappon3

<sup>1</sup> Research and Development Department, Belit d.o.o. (Ltd.), Trg Nikole Paši´ca 9, 11000 Belgrade, Serbia vladimir.urosevic@belit.co.rs

<sup>2</sup> Life Supporting Technologies Group, Universidad Politecnica de Madrid, Avenue Complutense 30, 28040 Madrid, Spain

<sup>3</sup> Department of Information Engineering, Università Degli Studi di Padova, Via Gradenigo 6/b, 35131 Padova, Italy

**Abstract.** The presented platform architecture and deployed implementation in real-life clinical and home care settings on four Amyotrophic Lateral Sclerosis (ALS) and Multiple Sclerosis (MS) study sites, integrates the novel working tools for improved disease management with the initial releases of the AI models for disease monitoring. The described robust industry-standard scalable platform is to be a referent example of the integration approach based on loose coupling APIs and industry open standard human-readable and language-independent interface specifications, and its successful baseline implementation for further upcoming releases of additional and more advanced AI models and supporting pipelines (such as for ALS and MS progression prediction, patient stratification, and ambiental exposure modelling) in the following development.

**Keywords:** amyotrophic lateral sclerosis · ALS · multiple sclerosis · MS · clinical disability function · ALSFRS · extended disability status · EDSS · REST services · architecture · AI models · disease progression · prediction · patient stratification · relapse · Big Data · pre-processing · regularization · daily living activities · exploratory analytics · neurodegenerative · wearable sensors · IoT integration

## **1 Introduction and Scientific/Technological Background**

The BRAINTEASER Project (**BR**inging **A**rtificial **INTE**lligence home for a better care of **A**myotrophic lateral **S**clerosis and multiple scl**ER**osis), funded from the European Commission Horizon 2020 programme grant until the end of 2024, integrates heterogeneous societal, environmental, health, and lifestyle/habitual data from diverse sources, developing patient stratification and disease progression AI models and applicative tools for improving disease management and ubiquitous monitoring and care delivery for ALS and MS patients (and assistance to informal caregivers).

Both are very complex chronic progressively degenerative fatal neurological diseases significantly disrupting the quality of life of the patients and their families, with notable differences in clinical picture, evolution, prognosis and therapies, but also many similarities in modelling and care/intervention delivery contexts (both in clinical and outpatient settings).

The initial releases of the novel interactive applicative tools for disease management and monitoring, currently in use in real-life settings in four clinical study validation sites (Lisbon, Madrid, Pavia and Turin) of the Project, are described with main features and functionalities in [1], including also some of the basics of the underlying back-end platform architecture and implementation. These tools and platform middleware services are enabling and supporting the first crucial step in the main overall identified and modelled BRAINTEASER data and process flows – continuous acquisition, ingestion, integration and storage of:

	- digitalized instruments and questionnaires for ALS and MS (and comorbidities) clinical evaluation and remote disease progress assessment, both standardized and in common practice (like ALSFRS-R or EDSS), as well as some innovative and evolving ones (like Awaji-Shima Consensus or Gold Coast diagnostic criteria for ALS), and
	- commonly available sensing/IoT devices, mainly Garmin smartwatches, and portable and fixed air quality and atmospheric para-meters sensing devices (like Atmotube PRO1 and PurpleAir (Classic) PA-II2).

The collected data have driven the development of Artificial Intelligence (AI) models able to address the needs of precision medicine, enabling early risk prediction of disease fast progression and adverse events. During the previous yearly period the intense development and evaluation of AI models in the Project (including relevant Open Science efforts co-organized by the Project3,4 [2]) have further resulted with the first releases of the model routines, tested and delivered ready for integration.

These generate not only the main envisioned final model outputs - such as predictions of probabilities or timeframes of occurrence of key disease progression events, like MS relapse or introduction of NIMV (Non-Invasive Mechanical Ventilation) or PEG (Percutaneous Endoscopic Gastrostomy) treatment for a patient - but also include

<sup>1</sup> https://atmotube.com/atmotube-pro.

<sup>2</sup> https://www2.purpleair.com/products/purpleair-pa-ii.

<sup>3</sup> https://brainteaser.health/open-evaluation-challenges.

<sup>4</sup> https://brainteaser.health/open-evaluation-challenges/idpp-2022.

the pre-processing or re-calibration model routines within the models, generating intermediate summary aggregated or transformed values from raw data inputs fetched from the unified platform Data Store, and passing the results back again to be persisted in the Data Store and re-used mostly for provision to the consuming applications exposed to the targeted end users, and as inputs to other following routines further in the disease progression prediction and patient stratification model pipelines. Such pre-processing routines, having been released first as initial steps in the ad-hoc AI processing pipelines, are actually turning out to be the most demanding ones from the integration perspective, as they require more frequent periodic invocation and execution (up to several times daily), and generate much higher throughput and consumption of data exchanged with the core platform services than the routines for actual disease progression prediction and patient stratification (invoked once in weeks or even months, and relying on already preprocessed results as more compacted inputs). Main algorithm types exploited in these pre-processing, feature extraction/selection and dimensionality reduction routines are Bayesian filtering and smoothing, retiming, or oximetry digital biomarkers evaluation (*pobm*<sup>5</sup> package), while the further AI models for continuous disease monitoring and progression prediction yet to be deployed exploit survival analysis algorithms like Cox proportional hazards, supported by methods like forward-recursive feature selection, and others elaborated fully and in detail in related publications like [10].

This paper describes architecture and implementation of the core BRAINTEASER platform back end and services tier integrating and supporting the mentioned released AI model routines in operation (as well as other recently developed or advanced supported features of the data feeding and consuming applications or modules coupled to the platform, presented summarily as a reminder on the general ecosystem and flows overview on Fig. 1 below). The general integration approach, as described in the following sections, is the same for both abovementioned types of AI model routines for now, with eventual specific critically performance-dependent alternative pathways and design patterns supporting tighter coupling or more extensive query stream parallelism having also been developed in reserve, and used for some scenarios of environmental/ambiental data processing, as described in [8]. Pilot demonstration and validation phase of the Project (initially focused on ubiquitous personal data collection, cleaning and integration at this stage, as mentioned) has just started a couple of months ago at the time of writing of this paper, with still a limited number of recruited study subjects and collected feedback and data on the usage. After at least a further semester of increased recruitment and more intensive continuous usage of the BRAINTEASER platform applications deployed towards the end-users, there will hopefully be sufficient data and statistics on the usage of the platform for at least an elementary sound and substantial analysis of the overall results and performance of the deployed implementation of the architecture to complement and expand on the work reported here, requiring more space than available in this short conference publication format, possibly in an evolved derived journal article.

<sup>5</sup> https://pypi.org/project/pobm.

## **2 Architectural Overview**

The diagram on Fig. 1 below provides a broad overview of the overall envisaged BRAIN-TEASER Data and ICT Tools ecosystem with main data flows and structural breakdown of data consuming or sourcing components and modules (more detailed in particular of the services tier in the biggest frame top right, crucial for integration), and the logo icons next to main modules/tiers to be described additionally denoting the system infrastructure language or platform stack chosen for development and deployment ( - Python, - PostgreSQL, - Java).

The presented schema is an evolution and expansion of the similar initial one provided as Fig. 7 in [1], and as noted in that referenced article, some of the platform functionalities and architecture build and extend upon the related efforts and outputs from preceding and parallel related projects, specifically from PULSE6 (partially described in [3, 5, 7]), NEVERMIND7 (in [6]), and PERISCOPE8 (in [4]).

**Fig. 1.** Updated BRAINTEASER ecosystem architecture with main data sources, consumers and flows, and detailed service tier breakdown.

RESTful APIs are the main interfacing and architectural approach on the back-end tier, with secure communication between the web services enhanced by industry standard JWT9 (JSON Web Token) lightweight encrypted encapsulation of request and response payloads. This extends also to the implementation and deployment of the AI models and routines themselves, with common unified human-readable and understandable interfaces specification based on JSONs, agnostic of the language the wrapped underlying

<sup>6</sup> https://www.project-pulse.eu.

<sup>7</sup> https://www.nevermindproject.org.

<sup>8</sup> https://periscopeproject.eu.

<sup>9</sup> https://jwt.io.

logic is written in, being strongly preferred across the complete services and tools ecosystem. Some of the additional key benefits of this for facilitated maintenance, scalability and sustainability of the platform beyond the Project developments are:

• OpenAPI (3.x) Specification (OAS) compliance, with ample open and available support and standardized toolsets for easier and semi-automated generation and maintenance of API specification and documentation live online (currently on Swagger<sup>10</sup> at https://brainteaser.belit.co.rs/gateway/swagger-ui/index.html, with relevant example provided on Fig. 2 below), as well as for API testing, mocking, and overall lifecycle governance and scaling.

*Collections* feature11 of the Postman REST API platform client has also been extensively used for building and maintaining request sets for practical integration testing of intra-service communication and data flows, and scheduled periodic batch server jobs, needed for execution management of most AI model routines, are also easier to consolidate and manage with uniform service calls (though still specific to application server/container or hosting server OS).

• the encapsulation of AI model and preprocessing routines (commonly written by data scientists as Python or R functions) within standard web service endpoints, similar to the interfacing (Java-based) invoking and managing ones of the core platform, also unifies and simplifies the deployment and CI/CD (Continuous Integration/Continuous Delivery) pipelines across the platform, including deployment containerization and scaling (with tools like Docker).

Concrete specifics of this web application server/container "wrapping" implementation for Python model routines are provided in the following section.

Development practice at this stage is to have each specific thematic domain set of AI models (for disease monitoring, progression prediction, patient stratification…) encapsulated in a dedicated wrapping microservice as it gets completed and delivered for platform integration (Fig. 3). Later towards the release of the overall integrated ecosystem, refactoring for optimal modularity can be performed, possibly merging some of the microservices (or most of them, into a practical modular monolith architecture), according to the results of the continuous models screening, in-silico simulation, evaluation, and improvement pipeline in the scope of the Project WP4, and according to the finally identified performance requirements and constraints.

The overall back-end service tier is similarly structured, mainly based on loosely coupled REST microservices [9], but with some practical trade-offs towards modular monolith, and the separation between domain-specific and infrastructural/utility orthogonal logic. The service packages and sets specific to the BRAINTEASER main thematic domains logic (supporting features and functionalities related to patients, caregivers, diseases & comorbidities, IoT devices and data in the system, etc.), presented as horizontal rectangles in the main top right frame on Fig. 1, are grouped into a couple of subprojects in development, and are currently using just three separate schemas/tablespaces in the Data Store underneath, divided at this stage mainly according to non-functional

<sup>10</sup> https://swagger.io.

<sup>11</sup> https://www.postman.com/collection.


**Fig. 2.** Example of testable specification/documentation of a platform API endpoint for ondemand invocation of a data pre-processing method generating inputs for the AI models for continuum disease monitoring (developed within the Work Package 5 (WP5) of the Project).

requirements – one containing highly sensitive potentially personally-identifying data (descriptive reported symptoms, specific socio-demographic and profile data…) kept protected fully encrypted in the database and throughout all handling in the system, other with the "regular" non-protected or already de-identified data, and the third for metadata. As the ingested data volume increases, the first two will likely separate to at least another additional dedicated to most bulky IoT sensed measurements data.

Microservices with separate underlying schemas supporting the infrastructural functionalities orthogonal to the domain logic (access control, authentication, security, utilities…) are represented as rectangles with vertical labels on Fig. 1, with the API Gateway package implementing the design pattern for basic service orchestration. Subsystems are hidden behind the Gateway façade service, acting not only as a proxy to those domain services but also validating requests (in terms of tokens, basic structure, sequencing…)

**Fig. 3.** Generic pipeline flows and microservice encapsulations of the integration of AI model and pre-processing methods with the core BRAINTEASER Platform services and Data Store

and documenting the specifications of all services through Swagger. Gateway also implements composition of calls for operations requiring calls to multiple services (or multiple calls to a single service), which is preferred to complex network of direct calls between sub-services. Code for request and response model, and endpoint definitions (URI, method, and docs) of sub-services are shared with Gateway by Git submodules in CI/CD.

Some additional most prominent employed service design patterns, mainly for the data collection, fusion, and provision to the applicative tools for disease monitoring and management, are described in [8].

## **3 Implementation**

Service tier core is currently implemented on the robust heavy-duty industry standard and proven Enterprise Java (Amazon Coretto 17 LTS12) web technology stack, leveraging Spring Boot13 framework 2.6.4 with all the functional programming and REST APIs support. Upgrade to the next Java LTS (Long-Term Support) version, expected to be Java 21 released around September 2023, is planned and being prepared for jOOQ14 (Java Object Oriented Querying) framework is used for object-relational mapping (ORM), and all is deployed and running on the Apache Tomcat 9.x web application server.

The industry-standard and common jUnit 5 framework15 is used for unit test generation, with REST Assured16 for baseline Java REST services semi-automated testing

<sup>12</sup> https://docs.aws.amazon.com/corretto/latest/corretto-17-ug/what-is-corretto-17.html.

<sup>13</sup> https://spring.io/projects/spring-boot.

<sup>14</sup> https://www.jooq.org.

<sup>15</sup> https://junit.org/junit5.

<sup>16</sup> https://rest-assured.io.

purposes, along with other extensive REST API development and lifecycle management tools specified above.

Other common and mostly newer open-source alternatives, like for example a complete Python-based stack covering also the AI models implementation, or JavaScript server-side implementations, still do not offer as comprehensive and robust support and performance in REST API services implementation, both object-oriented and functional programming, and cross-platform (heavy-duty web and mobile applications being the key BRAINTEASER tools for the end users) development.

GitLab serves as the code repository and version control system, as well as for CI/CD pipelines and control (except for the data tier, where Red Gate Flyway17 tool is used for database versioning management and migration/replication control).

Parallel Python (3.x.x) runtime is hosting the integrated AI model methods for sensory data pre-processing and disease monitoring, encapsulated in web service pipelines as described above, using the RESTful API implementation via the lightweight Flask<sup>18</sup> web framework deployed along with the Java-based core services stack on the platform back-end server cloud, with SQL Alchemy19 being the optional ORM framework in the Python-based stack. Principally, similar language-agnostic implementation would be viable for all AI models envisioned to be developed in the Project (more details in the next section with conclusions), supporting loosely coupled APIs for seamless scaling or significant changes (in cases like e.g., specific model routines getting completely rewritten and implemented in R or Julia, REST-based interfacing and service invocation and deployment control should remain unchanged, if a corresponding frameworks like Plumber or Genie are used instead of Flask).

Unified JSON-based communication across the platform also provides for some advantages on the data tier – implemented as PostgreSQL hybrid-relational Data Store, it features extensive support for JSON and binary JSON data types, querying, indexing and optimization. Consequently a lot of data that are natively structured in JSON documents as collected or generated in the ecosystem (evolving generic questionnaires, service configurations, intervention and gamified content…) are stored and queried in JSON format in the database, and fully deserialized into relational model entities only when necessary for main data consumption and performance criteria, or the structure evidently standardized and fixed in the long-term or permanently (like for the standardized questionnaire instruments exemplified in Sect. 1 above, or data model structures compliant with the relevant architectural standards in healthcare IT, mainly ISO/CEN 13606, openEHR, and HL7 FHIR…). This implementation has in development and deployment practice experiences by now shown equal or comparative performance in handling the mentioned document-structured data as using dedicated document-oriented databases like MongoDB, while at the same time retaining advantages of native relational data support (most of the data handled by the overall platform are relational by nature) or more comprehensive and robust transaction control (nesting, cascading), all in a single unified data store managing the complete heterogeneity of data. PostgreSQL has also shown satisfactory overall performance with the terabyte-level volumes of sensed

<sup>17</sup> https://flywaydb.org.

<sup>18</sup> https://palletsprojects.com/p/flask.

<sup>19</sup> https://www.sqlalchemy.org.

IoT data (billion-record tables) expected to be collected by the end of the Project, with proper indexing and query optimization, and using multiple available and well supported dedicated extensions (for scalability, time-series data management, etc.).

Leveraging the JSON-LD<sup>20</sup> format for Linked Data is also convenient over the complete uniform JSON platform interfacing and communication, for seamless expected upcoming semantic integration, requiring minimal overhead efforts and data model changes, with the BRAINTEASER Semantic Cloud [2] that has been developed and evolved during the two initial Project years mostly independently from the described platform. A referent similar semantic integration example is shortly described in Sect. 4.3 in [4].

## **4 Conclusion and Further Development and Evolution**

The presented platform architecture and deployed implementation in real-life clinical and home care settings on four BRAINTEASER study sites, integrating the novel working tools for improved ALS and MS monitoring and management released last year with the initial releases of the AI models for disease monitoring (and the supporting data pre-processing pipeline).

This integration of two key types of targeted ICT outputs of the BRAINTEASER Project through the described robust industry-standard scalable platform is to be a referent example of the integration approach based on loose coupling APIs and industry open standard human-readable and language-independent interface specifications, and its successful baseline implementation for further upcoming releases of additional and more advanced AI models and supporting pipelines (such as for ALS and MS progression prediction, patient stratification, and ambiental exposure modelling) in development until the end of 2024.

**Acknowledgement.** The presented work and developments are being funded by the European Commission Horizon 2020 programme grant agreement GA101017598, in the scope of the BRAINTEASER project.

## **References**


<sup>20</sup> https://json-ld.org.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Deriving Physiological Information from PET Images Using Machine Learning**

Olfa Gassara1(B), Belkacem Chikhaoui1,2(B), Rostom Mabrouk3(B) , and Shengrui Wang1(B)

<sup>1</sup> Department of Computer Science, Universit´e de Sherbrooke, Sherbrooke, QC J1K 2R1, Canada

{olfa.gassara,shengrui.wang}@usherbrooke.ca <sup>2</sup> AAII Research Institute, TELUQ University, Montreal, QC H2S 3L4, Canada

belkacem.chikhaoui@teluq.ca <sup>3</sup> Universit´e de Bishop's, Sherbrooke, QC J1M 1Z7, Canada

rostom.mabrouk@ubishops.ca

**Abstract.** Machine learning (ML) algorithms have become popular in recent years and have found increasing utility in the field of medical imaging, specifically in positron emission tomography (PET) imaging. The interest in ML in PET imaging for the study of neurodegenerative diseases stems from the potential of these techniques to analyze and predict the physiological parameters of biomarkers such as the total volume of distribution (Vt) in the organ or a structure of the organ to be explored. In this paper, we investigated whether the V<sup>t</sup> of [<sup>18</sup>F]-FEPPA radiotracer, an indicator of neuroinflammation, could be estimated directly in a non-invasive way, given the activity of the radiotracer in brain tissue. The study used several regression models to predict the [<sup>18</sup>F]-FEPPA V<sup>t</sup> in different brain regions where 31 regions of interest were defined for each of 24 patients with Parkinson disease and 20 healthy subjects, and were used to train four tree-based regression models. The predicted and reference values were compared by Bland-Altman analysis and regression model's performance was evaluated by the mean absolute error (MAE). The best result was obtained by the XGBoost model with a MAE of 2.6. Bland-Altman analysis results indicate that predicted V<sup>t</sup> are in average very close to the reference with a bias of 0.23 ∓ 2.82. Significant main effect of genotype on [<sup>18</sup>F]-FEPPA in both caudate and putamen have been preserved by predicted Vt values (p *<* 0.05). The results of paired t-test indicate that the difference between predicted and reference V<sup>t</sup> is not statistically significant in 6 out of 8 groups. The proposed algorithms provide a non-invasive and efficient tool to predict [<sup>18</sup>F]-FEPPA V<sup>t</sup> values, a hallmark of neuroinflammation that is believed to be a potential trigger for Parkinson's disease development.

**Keywords:** Machine Learning · PET imaging · Kinetic modeling

#### **1 Introduction and Related Work**

Machine learning (ML) is on the verge of revolutionizing medical diagnosis, especially in imaging-based specialties such as PET. This new discipline is bringing a wealth of innovations to the analysis of large datasets in critical clinical studies like those focused on neurodegenerative diseases. In this article, we propose a set of ML-based regression models to reproduce [<sup>18</sup>F]-FEPPA V<sup>t</sup> values, which are a neuroinflammation hallmark, in a non-invasive way.

Neuroinflammation is a complex process involving the activation of immune cells within the central nervous system (CNS) in response to injury or infection [1]. While it is normal and essential for a neuroinflammation to occur in the brain as a response to previous triggers, it has been proven that an excessive or chronic inflammation may lead to the development and progression of neurodegenerative diseases such as Parkinson's disease (PD) [2] and lead consequently to a loss of dopamine levels in the striatum region of the brain, more specifically in caudate and putamen regions.

Many factors contribute to this loss, including genetic factors [3], environmental factors [4], age [5], etc. Furthermore, several studies have shown that the activation of inflammatory pathways in the brain can contribute to the death of more dopamine neurons and amplify the motor symptoms of PD [6]. The inflammation in PD is actually characterized by the activation of particular glial cells in CNS, called microglia [7]. Recent studies have shown that a chronic activation of these cells can become harmful to the CNS [8]. Based on that, researchers have targeted neuroinflammation as a hallmark feature for PD's disease, as well as for other neurodegenerative diseases such as Alzheimer's disease [9], Huntington's disease [10], etc. Tracking neuroinflammation can potentially provide valuable insights into the mechanisms underlying these diseases, in addition to an accurate and earlier detection and diagnosis, enabling earlier interventions and treatments [11]. PET uses a special scanner to detect radiation emitted by a small amount of radioactive substance called radiotracer or radiopharmaceutical, that has been injected into the body. The used radiotracer binds to markers of inflammation or other physiological processes that are associated with these diseases.

Researches have proved that measuring microglia activation can be performed through quantifying a protein called translocator 18 kDa protein (TSPO). The expression of TSPO is upregulated when microglia are activated in response to injury or other stimuli in the brain, hence the interest of researchers to develop TSPO radiotracers [12,13]. [<sup>18</sup>F]-FEPPA is one of the newest second generation TSPO PET radiotracer with greater affinity for its target. However, the quantification of its distribution in the brain cells requires the determination of a metabolite-corrected arterial input function (AIF), which is practically done through arterial cannulation. Although the risk related to arterial cannulation is low, it represents an invasive and logistically demanding procedure. Besides, the discomfort caused by this procedure often discourages subjects from participating in PET studies. Given these considerations, much studies have been carried out to obviate the need for arterial cannulation such as Population-Based Input Function (PBIF) [14–17] and Image Derived Input Function (IDIF) [18–21].

Recent studies have explored the use of machine learning (ML) approaches for estimating AIF [22–24]. While the use of these approaches has not been extensively investigated, recent studies have reported promising results. As the aforementioned methods, MLIF has different challenges to be addressed like the high dependency of its results on the quantity and quality of the available training data.

To the best of our knowledge, this is the first study to investigate the estimation of pharmacokinetic parameters directly from the given Time Activity Curves (TACs) of different brain regions and, hence, avoid any use of AIF. By using ML models, we were able to give an approximate value of Vt, in a non-invasive way. Our results provide a novel perspective on PET images quantification.

Our paper addresses leveraging appropriate ML techniques to predict, in a non-invasive manner, the total volume of distribution (Vt) given the regional TACs. The data used in this study was acquired with [<sup>18</sup>F]-FEPPA radiotracer. We organized our paper as follows: in the first section, we will give an overview of the dataset acquisition, the reference V<sup>t</sup> estimation and the methods we used for Vt prediction. Next, we will discuss the results obtained from our ML models. In the conclusion, we will suggest some directions for future work.

## **2 Methods**

In this section, we present details on features that have been used as input to our models, including data acquisition, region of interest (ROI) delineation, TACs generation, input function used in the quantification of PET data previously published in [25] and the genotype of the subjects included in the analysis. The training data contain relevant information about the biochemical transformations of the tracer in each of the brain regions and the genotype of the studied subject, as well as the correct response to the [<sup>18</sup>F]-FEPPA Vt.

## **2.1 Data Acquisition**

Twenty four subjects with Parkinson's Disease and twenty healthy controls underwent an [<sup>18</sup>F]-FEPPA PET and magnetic resonance imaging scan. After radiotracer administration into the body of the patient and PET data acquisition, the collected PET raw data were reconstructed into images using ordered subset expectation maximization with point spread function (OSEM+PSF) reconstruction [26].

#### **2.2 ROI-Based Time Activity Curve Generation**

MRI images for all the subjects were acquired for co-registration with the corresponding PET images and the anatomical delineation of the 31 ROIs. The ROI template is transferred to the PET image space to extract the time activity curve for each ROI. The TACs are graphical representations of how radioactive tracers distribute and accumulate within tissues over time. In our study, dynamical series of images of [<sup>18</sup>F]-FEPPA PET have been visually checked for head-motion and corrected using frame-by-frame realignment [25].

#### **2.3 Input Function Measurement**

AIF is determined during the PET scanning by gathering blood samples at discrete time points from the subject's radial artery and measuring the concentration of the radioactive compound in every sample. Arterial blood was taken continuously at a rate 2.5 mL/min for the first 22.5 min after radioligand injection and the blood radioactivity levels were measured using an automatic blood sampling system (Model PBS-101 from Veenstra Instruments, Joure, The Netherlands). In addition, 4 to 8 ml manual arterial blood samples were obtained at 2.5, 7, 12, 15, 30, 45, 60, 90, and 120 min relative to time of injection. A biexponential function was used to fit the blood-to-plasma ratios. A Hill function was used to fit the percentage of unmetabolized radioligand. The dispersion effect was modeled as to the convolution with a monoexponential with dispersion coefficient of 16 s and corrected with iterative deconvolution [27].

#### **2.4 Polymorphism Genotyping**

The quantitative interpretations of [<sup>18</sup>F]-FEPPA are impacted by the large interindividual variability in binding affinity, which displays a trimodal distribution compatible with a co-dominant genetic trait [28]. Study of TSPO polymorphism explained the heterogeneity in binding potential by the difference in the affinity of the second-generation PET ligands for this protein. [<sup>18</sup>F]-FEPPA radiotracers bind TSPO in brain tissue from different subjects in one of three ways: highaffinity binders, mixed affinity binders, and low-affinity binders (HABs, MABs and LABs). The transport rate of radiotracer is 1.5 to 2-fold higher in HABs than MABs and 4-fold higher in HABs than LABs. Since LABs are very rare, we limited our data collection to the two groups HABs and MABs only. More insights on TSPO polymorphism can be found in [29].

#### **2.5 Kinetic Analysis**

Kinetic modeling is a mathematical approach used in PET imaging to quantify the pharmacokinetics of a radiotracer in various tissues. In this study, we used the 2-compartmental model (2-TCM) [30] to fit our data. This model assumes that the radiotracer in the tissue compartment can be either specifically bound to the target receptor (specifically bound compartment) or it can be free (nonspecifically bound compartment). The kinetics of tracer uptake have thus been modeled mathematically through differential equations describing the exchange rate of tracer concentrations among compartments in function of time as follows (1):

$$\frac{dC\_1}{dt} = K\_1 C\_p(t) - (k\_2 + k\_3)C\_1(t) + k\_4 C\_2(t) \tag{1}$$

$$\frac{dC\_2}{dt} = k\_3 C\_1(t) - k\_4 C\_2(t) \tag{2}$$

where:

*C*<sup>1</sup> is the tracer concentration in the non-displaceable compartment of the tissue (free and nonspecifically bound), *C<sup>p</sup>* is the tracer concentration in the plasma also known as Arterial Input Function (AIF), *C*<sup>2</sup> is the tracer concentration bound to the target receptors, *K*<sup>1</sup> is the rate constant for transfer of tracer from plasma to the tissue, *k*<sup>2</sup> is the rate constants for transfer of tracer from tissue to plasma, *k*<sup>3</sup> and *k*<sup>4</sup> are the rate constants for transfer of tracer from the non-displaceable compartment to the specific binding compartment of the tissue and vice versa, respectively. Knowing the tissue compartment comprises two different states of binding (non-displaceable + specific binding), the tissue concentration, *Ct*(*t*), is equal to the sum of the two states

$$C\_t(t) = C\_1(t) + C\_2(t) \tag{3}$$

Using the aforementioned differential equations, *Ct*(*t*) can be defined as follows:

$$C(t) = \frac{K\_1}{b\_2 - b\_1} \left[ \left( k\_3 + k\_4 - b\_1 \right) e^{-b\_1 t} + \left( b\_2 - k\_3 - k\_4 \right) e^{-b\_2 t} \right] \otimes C\_p(t) \tag{4}$$

where *b*<sup>1</sup> and *b*<sup>2</sup> expressions are

$$b\_{1,2} = \frac{1}{2} \left[ \left( k\_2 + k\_3 + k\_4 \right) \mp \sqrt{\left( k\_2 + k\_3 + k\_4 \right)^2 - 4k\_2 k\_4} \right] \tag{5}$$

and ⊗ denotes the mathematical convolution. In this study, we have a particular interest in deriving quantitative information about the total volume of distribution of the radiotracer as this kinetic parameter reflects the overall density and distribution of the TSPO receptors in the brain. Precisely, V<sup>t</sup> represents the ratio of the tracer amount in the target tissue at equilibrium to the amount of tracer in the plasma at the same time point. Mathematically, V<sup>t</sup> can be expressed in function of model rate constants as follows:

$$V\_t = K\_1 / k\_2 \left( 1 + k\_3 / k\_4 \right) \tag{6}$$

In TSPO studies, a higher V<sup>t</sup> indicates a greater amount of tracer binding to the target protein, suggesting a higher level of neuroinflammation in the tissue [31].

#### **2.6 Estimation of Reference Total Volume of Distribution**

An estimate of V<sup>t</sup> values was derived using the kinetic modeling tool of PMOD (https://www.pmod.com/web/). We utilized the blood TACs, plasma TACs, and regional TACs of the brain for each subject in our dataset to fit a reversible 2-TCM model, enabling us to estimate Vt values.

#### **2.7 Total Volume of Distribution Prediction**

After estimating the reference V<sup>t</sup> values corresponding to each brain region and for each subject, we built our dataset by concatenating the TACs of different subjects. Initially, the TAC file of each subject contains the different intervals of scans, defined by the start and end time of the scan, and the corresponding radiotracer concentration at this interval, with respect to each brain region. On average, we have 31 ROIs for each subject. Then, we added to our dataset 3 categorical variables specifying the brain ROI, the genotype (HAB or MAB) and the health status (Healthy Control or Parkinson). For V<sup>t</sup> prediction, a total of 44 subjects were included for the purpose of establishing predictive models using ML. Our predictive models are tree-based regression models, which use decision trees to predict continuous numerical values. The rationale behind choosing these approaches for our regression problem is their ability to handle the mixture of categorical and continuous variables as inputs, and to capture the non-linear relationships between variables. To find the optimal hyperparameters for our predictive models, a grid search approach was employed. Rather than dividing the available data into separate training and testing groups, 10-fold cross validation was utilized and the average performance was recorded.

#### **2.8 Total Volume of Distribution Evaluation**

To evaluate our models, the predicted V<sup>t</sup> values were compared with the ones estimated by kinetic modeling and denoted as reference Vt, for each region of interest, by the mean absolute error (*MAE*),

$$\text{MAE} = \frac{1}{n} \sum\_{i=1}^{n} |Vt\_i - \hat{V}t\_i| \tag{7}$$

We also used Bland-Altman as a graphical method, to compare the predicted values of our ML models against the reference values, and asses the level of agreement between them. By plotting the difference between *V*ˆ *<sup>t</sup>* and *V<sup>t</sup>* against their mean, it will help to identify any systematic bias between the two measurements, as well as the range of differences and outliers.

#### **3 Results and Discussion**

Results from comparisons between the reference and the predicted *V<sup>t</sup>* in terms of MAE are summarized in Table 1. As shown in this table, all the tree models have predicted V<sup>t</sup> with a mean absolute error ranging between 2.62 and 3, which is within an acceptable range of error for our particular problem, considering the median value of reference Vt. These results indicate that our models are able to predict the target variable with reasonable accuracy and provide a good fit to the data. After evaluating the performance of the four tree-based models, we found that XGBoost outperformed the other models in terms of its evaluation


**Table 1.** Summary of Mean Absolute Error values for the selected machine learning models

metrics. It specifically achieved a lower MAE, equal to 2.62 compared to other models, indicating that it has a better overall performance and it can predict the target variable with greater precision. For this model, we investigated the relative importance of each feature in predicting the target variable. Based on the importance scores of XGBoost model, we can conclude that the tracer uptake concentration for the first 7 timepoints are the most influential features in making V<sup>t</sup> predictions. In other words, these features are the most utilized by XGBoost in a split decision, while creating its decision trees.

In order to provide a more comprehensive evaluation of the performance of XGBoost, we used the bland-altman plots as showed in Fig. 1. Central and outer dashed lines indicate mean value and mean ± 1.96 SD. It is important to mention that bland-altman is displaying the difference and average between the reference and predicted V<sup>t</sup> for all the tissues of test data subjects. Figure 1 shows that the mean difference is 0 with a bias of 0.23 ± 2.82, among brain tissues, and 1.96 SD interval equal to −5.30 and +5.77, which is believed to be a tolerable result for a first attempt of predicting V<sup>t</sup> directly from tracer uptake concentration in brain tissues. Furthermore, we can observe that the majority of the data points fell within the limits of agreement, reflecting an overall good agreement between the reference and predicted values of Vt.

To further validate the results given by XGBoost, we investigated the ability of our model to highlight the genetic subgroup effects on TSPO binding. These effects have been previously explored in [25]. We are interested in reproducing these results, as we are analyzing the same dataset. In this part of our study, we will be focusing on putamen and caudate nucleus regions, as these regions are critically involved in the pathophysiology of Parkinson's disease. Given the two possible studied genetic subgroups (HAB or MAB) and health status (HC or PD), we can distinguish between 4 groups as shown in Fig. 2. A total of 12 subjects were substracted from the initial data, 3 subjects from each group (HAB-HC, HAB-PD, MAB-HC and MAB-PD) to constitute the test data and we trained XGBoost model on the remaining data. The results showing the effect of genotype (MAB or HAB) on estimated V<sup>t</sup> and predicted V<sup>t</sup> for the caudate nucleus and putamen are illustrated in Fig. 2. This figure suggests that we can preserve the same effect of genotype revealed by the kinetic modeling estimated Vt, with the exception of caudate V<sup>t</sup> values of one group, which is the PD group. According to ML predicted Vt, there is no significant difference in caudate Vt

**Fig. 1.** Bland-Altman plots of predicted and reference total volume of distribution *V<sup>t</sup>* using XGBoost model for all brain tissue regions.

**Fig. 2.** Comparison of kinetic modeling estimated *V<sup>t</sup>* and ML predicted *V<sup>t</sup>* in the caudate nucleus and in the putamen, for different genetic subgroups. The two Asterisks in the plot indicate statistical difference between two groups.

values within MAB-PD and HAB-PD, which is not the case for caudate V<sup>t</sup> values, estimated using kinetic modeling. Still we have consistent findings for HC group in both putamen and caudate Vt, and for PD group with regard to putamen V<sup>t</sup> values, which can be considered as satisfactory results.

In Fig. 3, we conducted a paired t-test to determine if there were any significant differences between the reference and predicted values of Vt. We found that predicted V<sup>t</sup> are significantly different from reference V<sup>t</sup> for MAB-PD and MAB-HC, respectively in the caudate nucleus and putamen regions. On the other hand, we have a good agreement between ML and kinetic modeling results for the remaining groups, in both regions, highlighting the consistency of our ML findings.

**Fig. 3.** Paired t-test showing differences of reference *V<sup>t</sup>* and predicted *V<sup>t</sup>* within subjects of the same subgroup, in both caudate nucleus and putamen tissues. The two Asterisks in the plot indicate statistical difference between two groups.

In summary, conventional methods used for quantitative evaluation of PET imaging are achieved by means of kinetic modeling, based on compartmental and non-compartmental approaches. This requires an accurate measurement of IF as well as the application of complex kinetic modeling approaches depending on the used tracer. One of the limitations of compartment modeling is that these models use iterative fitting including IF to calculate the least squares between the measured data and the model data, which can lead to problems of overfitting and lack of reproducibility. In particular, an inappropriate IF often leads to imprecision of the assessed rates. In contrast, our proposed method, based on machine learning approaches and the [<sup>18</sup>F]-FEPPA radiotracer dataset, provides a robust and reproducible solution. It is independent of the input function, representing a novelty in the quantitative analysis of PET.

## **4 Conclusion**

In this paper, we investigated the feasibility of non-invasively estimating the V<sup>t</sup> of [<sup>18</sup>F]-FEPPA radiotracer, an indicator of neuroinflammation, using its activity concentration in brain tissue. We used several non-linear regression models to predict the [<sup>18</sup>F]-FEPPA V<sup>t</sup> in 31 brain regions of interest over 24 patients with Parkinson Disease and 20 healthy subjects. The XGBoost model showed the best results with a MAE of 2.6. Bland-Altman analysis results indicate that predicted V<sup>t</sup> are in average very close to the reference with a bias of 0.23 ∓ 2.82. We also found that significant main effect of genotype on [<sup>18</sup>F]-FEPPA in both caudate and putamen have been preserved by predicted V<sup>t</sup> values (p *<* 0.05) for the majority of groups. The results of paired t-test indicate that the difference between predicted and reference V<sup>t</sup> is not statistically significant in 6 out of 8 groups. This study opens a new research direction in applying machine learning algorithms to provide a non-invasive and efficient tool to predict [<sup>18</sup>F]-FEPPA V<sup>t</sup> values, a hallmark of neuroinflammation that is believed to be a potential trigger for Parkinson's disease development. As part of future work, our goal will be to develop predictive models to estimate the four pharmacokinetic parameters, namely K1, k2, k3, and k4. By accurately deriving these parameters, we aim to improve the estimation of the physiological parameters of the total volume of distribution V<sup>t</sup> and the binding potential BP.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Situation-Aware Framework for Energy Saving in Unattended Activities of Daily Living in Smart Home Environments**

Richard O. Oyeleke(B) , Pranav Parekh, Sophia DiCuffa, and Andrew Quinlan

School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, USA {ooyeleke,pparekh5,sdicuffa,aquinla2}@stevens.edu

**Abstract.** Activities of daily living (ADLs) are basic self-care tasks that are necessary for independent living. It is also used to assess aging adults' functional ability. As people age, they experience impaired awareness, which affects their reasoning, cognitive ability, and consequently impedes their ability to manage or supervise ADLs successfully. Impaired awareness has a potential economic impact on aging adults. For example, recent studies have shown that aging adults with impaired awareness use more energy in homes due to deviation from normal execution of ADLs. This often results in incompletely executed ADLs which are left unattended or unsupervised. Unsupervised ADLs involving appliances with high wattage ratings (e.g., TV, air conditioner) can lead to an increase in the cost of home energy services used by aging adults who desire to age-in-place. Thus, in this paper, we propose a situation-aware framework that leverages a smart home resident's context information with respect to unattended ADLs in mitigating increased energy consumptions in smart home environments.

**Keywords:** Smart home · machine learning · ADLs · internet of things (IoT) · aging-in-place · situation awareness · aging adult

## **1 Introduction**

A smart home environment is an intelligent and inclusive residential space designed to enable aging adults to live comfortably and independently [1]. It is retrofitted with a network of sensors and IoT devices that can detect events, monitor, and recognize the activities of daily living (ADLs) of residents, respond, and adapt to their needs [2-4]. For example, smart homes can trigger a warning to notify residents of a potential fire hazard if they forget to turn off the stove. ADLs, such as using technology, preparing meals, and sleeping, are important tools used to assess an aging adult's functional ability [5]. For instance, if aging adults consistently leave ADLs unsupervised (e.g., resident frequently forgets to turn off the TV before going to bed or did not turn off the light after using the restroom), it may be an indication of a decline in their health and well-being. Several studies have shown a strong correlation between the amount of energy consumed in homes and how aging adults perform their ADLs. The projected population of aging adults (aged 65 and above) is expected to exceed 70 million by 2030 [6], and as they continue to age in place, the amount of home energy consumed will increase, resulting in higher costs for home energy services. Aging adults belong to an economically vulnerable group with limited options for consistent income [7, 8]. Therefore, it becomes crucial to develop a robust framework for energy management in older adults' homes to mitigate increasing energy consumption from unattended ADLs and to alleviates the associated financial burden on aging adults who choose to age in place. To address this, our study presents a situation-aware framework for mitigation of increase in energy consumption from detected unattended/unsupervised ADLs in smart home environments. Our proposed framework is designed to help residents sense and perceive unattended ADLs in the smart home environment, as well as the states of smart home devices/appliances, especially during episodes of impaired awareness. The remainder of this paper is organized as follows: Sect. 2 presents the related work, Sect. 3 highlights the proposed framework and methodology, Sect. 4 discusses the experiment design, the results and interpretation are presented in Sect. 5, and in Sect. 6, we present the conclusion and future work. Please note that we will use the terms 'unattended' and 'unsupervised' interchangeably throughout this paper.

## **2 Related Work**

This section presents the vision of this paper and concepts used in this study.

#### **2.1 Impaired Awareness in Activities of Daily Living**

Activities of daily living (ADLs) are self-care tasks performed to enhance an individual's quality of life (QoL) [9]. ADLs may include basic tasks such as grooming, dressing, and toileting, as well as more complex tasks that require stepwise procedures for their execution, for example, preparing meals, watching TV, and doing laundry [10]. As people age, they are more prone to age-related cognitive decline, which affects their reasoning and thinking skills, thereby impeding their ability to successfully manage or supervise complex ADLs [11]. When aging adults are unable to supervise ADLs effectively, it may suggest that they suffer from impaired awareness [12]. Impaired awareness causes aging adults to underestimate their functional decline and can have economic consequences and compromise their safety when they choose to age in place [13, 14].

For example, an aging adult with mild memory loss may forget to turn off the TV and/or the air-conditioner (AC)/fan in the living room before going to bed or may forget to turn off the light bulbs after using the restroom or may forget to turn off the stove and/or faucet after preparing a meal. Each of these scenarios has economic and safety consequences. For instance, if a resident fails to turn off the TV, AC, fan, and light bulbs for a long period of time in moments that suggest that the ADLs are incomplete or unattended, this could result in an increase in the home energy consumption and consequently raises the cost of home energy services. Moreover, when the stove or faucet are not turned off and left unsupervised for a long period of time, it may constitute a hazard or risk of accidents in homes (e.g., such as fire outbreak or slips due to a wet floor. However, the main objective of this study is to mitigate the potential economic consequences in terms of energy consumption from unsupervised ADLs.

#### **2.2 Situation Awareness for Energy Use Management in Smart Homes**

Situation awareness refers to the ability to perceive or sense devices in the environment at a given time instance and location, understand their operational context, and predict their future states or status [15]. Smart home environments consist of smart appliances and IoT devices that are connected and communicate via a wireless network. These smart appliances and IoT devices are tools used for the execution of ADLs. When aging adults develop impaired awareness, it impedes their ability to properly supervise and manage ADLs, causing them to leave one or more appliances in operating states and unattended, which leads to increasing home energy consumption. To address this problem and enable aging adults to continue live optimally and independently, a situationaware framework can be integrated into smart home environments. This framework enables aging adults to understand and perceive the contextual and operating states of the appliances used for the execution of their ADL in smart home environment and consequently help to mitigate unintended energy consumption. For example, when smart home appliances such as a TV, fan, or AC are in an operational state, each produces a distinct sound. These sounds provide useful context data that can be leveraged to gain an understanding of an ADL scene. Similarly, passive infrared (PIR) sensors can detect motion or human presence within a space at a given time to indicate if an ADL is supervised or unsupervised. By leveraging the predictive power of advanced machine learning algorithms and applying them to the robust context data from the smart home environments, aging adults can regain the awareness required to control the state of home appliances when left unattended. This can help prevent unintended energy consumption in unsupervised ADLs and alleviate the potential financial burden from increased energy use.

**Fig. 1.** Situation-Aware Framework for Energy Savings in smart home

## **3 Proposed Framework and Methodology**

The objective of our proposed situation-aware framework is to enable aging adults to understand and perceive the contextual and operating states of the appliances used for the execution of their ADL in smart home environment, and to mitigate unintended energy consumption when ADLs are left unsupervised. Specifically, we focused on three main smart home appliances that consume more energy in homes: TV, AC, and fan. These appliances were used as a case study for our experiment. Additionally, the proposed framework includes sensors for sensing lightening from lightbulbs spaces (e.g., restroom) and where human presence is not detected after an ADL has been initiated but not completed as would be expected for a normal execution of the ADL. In such scenarios, the user receives a notification to shut off the lightbulbs.

Figure 1 provides a description of the framework and its components. The smart devices are connected and controlled through the Alexa Echo Dot. The Raspberry Pi serves as the control hub of the smart home environment, detecting the presence of light and sound through its connected sensors. This helps to detect the presence of a resident within a space. If no human presence is detected in space, it sends a notification to the resident's phone instructing them to turn off any detected devices that are left unattended or not turned off. The Alexa Echo Dot can receive requests initiated by the user from our custom-built mobile app, allowing them to turn off the smart devices when they receive a notification on their phone from the Raspberry Pi.

#### **3.1 Audio Source Separation**

Blind source separation (BSS) refers to receiving a set of source signals from a mixed signal or a set of mixed signals with very little information of the corresponding source signals [16]. We require a separator for sound separation in our experiment and propose to use BSS to filter out a mixture of sound into its respective audio waveforms. We considered four main smart appliances within the smart home set-up.

**IVA Variations Used.** We employ four different variations of Independent Vector Analysis (IVA) in our study. Two of these variations are based on the gradient-based method. NaturalGradLaplaceIVA and GradLaplaceIVA. The other two variations use the auxiliary approach: Auxiliary Laplacian IVA (AuxLaplaceIVA) and Auxiliary Gaussian IVA (AuxGaussIVA). In the auxiliary approach, AuxLaplaceIVA utilizes the Laplacian function as its source model, while AuxGaussIVA uses the Gaussian function. On the other hand, both gradient-based methods consider the Laplacian function as their source model. However, they differ in how they optimize the KL Loss through gradient descent. Specifically, GradLaplaceIVA is like a gradient descent with momentum, as it incorporates an additional momentum hyper-parameter along with the learning rate.

**Why Dual Channel?** As part of our experiment, we assumed the presence of two functioning appliances at a given time. This assumption is based on the observation that the AC, fan, and heater are typically used independently without overlapping. When the room temperature is warm, residents are more likely to turn on the AC and at a higher temperature rather than putting on both the fan and the AC at the same time. Similarly, the resident would likely turn on the heater when the room temperature is cold. The fan is considered in scenarios with moderate temperatures. Additionally, we assumed the TV would always be turned on in all three scenarios. To avoid increased time and computational complexity, we tested the separator using three channels. However, since this significantly increases the processing time, we opted for a dual-channel method. When only one device is functioning in the dual-channel scenario, we modified the system to separate it as a blank output. For instance, an output prediction may include 'TV' and 'blank' or 'fan' and 'blank'.

**Why Noise?** An important assumption when using IVA is that the input sources should not be correlated with each other. However, in the case of a mixture such as the TV and fan, the IVA model would encounter an error due to their high correlation. Despite having different frequencies and amplitudes, the rate of change of the component is equivalent for both sources. To address this, we introduce noise to one of the source samples. There are several important parameters in the IVA function. These include the basic machine learning parameters such as the type of loss, number of epochs, and number of iterations. Additionally, two significant parameters are the algorithm spatial and apply projection back. The algorithm spatial parameter can take the values 'IP' or 'ISS', which determine how the algorithm iterates its demix filter. The second parameter, apply projection back, is a scaling function applied to the source and mixture to prevent larger values and reduce complexity. It can be set to either True or False, depending on whether we want to apply it or not.

**Hyper-parameters.** The essential hyper-parameter for the model is the noise coefficient (n) which refers to the degree of noise added to the input source. This is done to prevent the sources from being highly correlated to each other. The other hyper-parameters include the learning rate and the threshold value, which are common hyper-parameters while using machine learning models. The learning rate is used for optimization while the threshold value stops iterations once loss is at minimum.

## **4 Experiment Design**

## **4.1 Audio Classification**

Audio data for classification in most scenarios is pre-processed by conversion to mel spectrograms or by manipulating data through short time Fourier transform (STFT) [17]. The short time Fourier transform is a Fourier-based transform which determines the sinusoidal (sin) frequency and content of the local sections of a signal under study with respect to time [18]. Once we receive the spectrograms, the research problem is now reduced to that of a computer vision problem, hence, we rely on powerful convolutional neural network (CNN) architectures for the feature extraction.

**STFT.** The audio recordings were downsized to 16 kHz and then transformed from waveforms into the time frequency-domain signals by computing the STFT. The STFT splits the signal into windows of time and runs a Fourier transform on each window, preserving some time information, and returning a 2D tensor that you can run standard convolutions on.

**Spectrogram.** In the following step, the STFT (number of samples perseg = 255, overlap = 124, nfft = 256) was applied to the waveform signals to obtain the spectrogram images of size 129 x 124 (frequency x time) which was then fed into a simple convolutional neural network to train the model.

**CNN Model.** The CNN model has a convolution layer to down sample the input to enable the model to train faster and a normalization layer to normalize each pixel in the image based on its mean and standard deviation. The CNN model consists of four weight layers used sequentially as follows: conv2–32, conv2–64, maxpool2, FC-128, and FC-4.

**Important Parameters.** In our experiment, our parameters with the model include an epoch value, which is 10, along with a batch size of 64 which was trained on 712 audio datasets with a 4-class label: "TV", "AC" and "Fan". The loss function used is SparseCategoricalCrossEntropy since it is a classification problem. The optimization parameter used is Adam's Optimizer. We use the early stopping parameter to end iterations once minimum loss is reached per iteration. We also set the train-test split parameter as 0.6–0.2–0.2 with 60% for training data, 20% testing, and 20% for validation.

#### **4.2 DashHome Audio Dataset Collection**

The DashHome dataset consists of 60 s audio recordings of smart devices sound including TV, AC, fan and heater. There are 712 samples in.wav file format. The recordings were collected in a quiet room with a recording device at a sample rate of 44 kHz and then later re-sampled to 16 kHz. The AC and fan sounds were captured at different temperatures and fan speed, the TV sound was recorded with music or a show playing at volume 40, and the heater sound was captured when in high and low temperature mode. We consider the heater as the input to extract features from it, however it is not an output label, hence it is not considered in our output computations. The separator was trained with dual audio mixtures of fan-AC, fan-TV, AC-TV, and TV-heater. Each mixture was 10 min at the highest sensory level of the digital recorder. The TV sound was playing at a volume of 40, with the fan speed at four and AC at high. It should be noted that despite a couple of mixtures being trained on the heater, it was not considered for the calculation of ground truth device probability since the heater sound is very difficult to separate and there are few samples that gives us heater outputs.

#### **4.3 Smart Devices Setup**

The smart devices used for this study were Alexa-enabled devices connected to the Amazon Echo. They include Sony X90J 55 Inch TV, Vornado 660 AE Fan, Rollicool 14000 BTU Smart Air Conditioner and Philips Hue Smart A19 Lights. In the Alexa app, The IFTTT trigger skill (a virtual button that can be used to perform a request at the event of button pressed) was enabled and linked to the Alexa account. Then, an Alexa routine was created such that when the trigger button is pressed then the devices are instructed to be turned off. Finally, a webhook URL to trigger the button from a URL request was generated using an applet on ifttt.com using the logic "If receive a web request then Trigger the IFTTT Button", which will then issue the corresponding command to the Alexa device to turn off the smart devices automatically. The Raspberry Pi 4 with a configuration of 8GB ram and 64GB storage was used in this study with the Raspbian OS loaded into memory. The sensors connected to the Raspberry Pi include the microphone for collecting sounds from the environment, photoresistor for detecting the sensitivity of light bulbs, motion sensor for user location tracking. The function of the Raspberry Pi is to continuously collect the environmental sounds and detect if the sound is classified as one of the smart devices.

## **5 Results and Interpretation**

#### **5.1 Audio Classification**

We use a custom-made CNN model for our classification, and we achieved an accuracy of 84%. Further, we considered a threshold value of 0.30. If our probability is greater than the BSS threshold, we consider both outputs for the mixture.

#### **5.2 Audio Separation and Classification**

**Initial Studies**. We considered the following separator models which were trained on the respective source appliances. 'Ag' represents Auxiliary Gaussian, while 'Al' represents Auxiliary Laplacian.We considered only the auxiliary models as opposed to the gradientbased ones since they have a much better performance comparatively. 'ip' and 'iss' are the demix filter iteration differences, and the number at the end denotes how many iterations the model is trained for. We considered an effective mix of the different iterations, models, and methodologies to derive our output.

#### 1) Ag\_ip\_TVAC\_200.

2) Ag\_ip\_TVFAN\_200. 3) Ag\_ip\_TVHEATER\_200. 4) Ag\_iss\_FANAC\_300. 5) Ag\_iss\_FANAC\_200. 6) Ag\_iss\_FANAC\_100. 7) Ag\_iss\_TVAC\_300. 8) Al\_iss\_FANAC\_200. 9) Al\_ip\_TVAC\_200. 10) Ag\_ip\_TVFAN\_200.


**Table 1.** The probability of each ground truth smart device calculated from the BSSP

#### **About Table** 1.


**The BSSP Formula**.

$$p = \sum\_{i=1}^{n} \frac{\min\left(\frac{N}{i}, x\_i\right)}{N} \tag{1}$$

where,


**Model Comparisons**. Based on Table 1, we can conclude and compare the models as follows:


**Fig. 2.** Loss curves for the separator model.

**Understanding the Loss Curves**. The following points explain the loss curves represented in Fig. 2:



**Table 2.** Output of each machine learning model.

#### **5.3 Energy Savings Outcome**

Suppose an older adult resident turns on the TV, AC, and fan in the living room at 7:00 PM and then goes to bed at 8:00 PM and forgets to turn off the devices in the living room. When the resident wakes up the next day, the energy consumed during that period would have incurred a cost with zero service value because the resident was not present in the living room area where the ADLs were left unattended.

However, the Raspberry Pi component of our proposed framework can periodically detect human presence, listen to the sound emitted by the appliances, and infer its source as TV, fan, AC, or heater. This ADL context information is used to determine if notification needs to be sent to the resident's mobile phone instructing them about appliances that are left unattended and are to be turned off to save energy consumption and reduce cost. To provide a more specific estimate of energy savings, we can calculate the energy consumed every 30 min or hour based on the power consumption as shown in Table 2. The assumptions to consider during these calculations are as follows: (Table 3)



**Table 3.** Energy savings from smart devices in unattended ADLs.


$$E = P\_a \* T\_s - P\_m \* T \tag{2}$$

where, E is the energy saved for time *Ts, Pa* is the power rating of the appliance, *Pm* is the power rating of the energy management module (Raspberry Pi), *Ts* is the time of study.

## **6 Conclusion and Future Work**

The aging adult population belongs to the low-income category of our population and is susceptible to age-related impaired awareness. Impaired awareness limits aging adults' ability to successfully supervise ADLs and results in unanticipated economic consequences. For example, when home appliances are in operational states and left unattended, it leads to an increase in the amount of home energy consumed and consequently raises the cost expended on home energy service. It is therefore imperative to develop an effective solution to enable aging adults with impaired awareness to mitigate unintended energy use from unattended ADLs so that they can continue to afford to age-in-place. In this study, we proposed a situation-aware framework for mitigation of unintended energy consumption in unattended ADLs. The framework includes low-cost sensors and IoT devices such as Raspberry Pi, sound, light and motion sensors. In addition, the framework incorporates machine learning models that uses robust context data generated from the smart home appliances to enable the user to understand their ADLs environment, perceive smart home appliances and their operating states to mitigate unintended energy consumption when ADLs are unattended. The results obtained from the experiments are promising. For our future work, we plan to test the efficacy of our proposed framework in senior retirement homes in using longitudinal study.

## **References**


50 R. O. Oyeleke et al.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Adaptive Dashboard for IoT Environments: Application for Senior Residences**

Bessam Abdulrazak1,2(B) and Amin Rezaei<sup>1</sup>

<sup>1</sup> Ambient Intelligence Laboratory (AMI-Lab), Université de Sherbrooke, Département d'informatique, Faculté des Sciences, QC J1K 2R1 Sherbrooke, Canada {bessam.abdulrazak,amin.rezaei}@usherbrooke.ca

<sup>2</sup> Centre de recherche sur le vieillissement du CIUSSS de l'Estrie – CHUS, Sherbrooke, Canada

**Abstract.** Dashboards are powerful electronic tools that can provide actionable insights for healthcare professionals, especially in support of the increasing senior population. With advancements in technology and IoT infrastructure, remote patient monitoring has become a feasible option for healthcare professionals through dashboards. To best serve the diverse needs of healthcare professionals, dashboards should be tailored for each user, considering their roles, interests, and priorities. In this study we proposed AMI-Dash, a solution allows for dynamic design and information visualization to address the diversity in needs and priorities among different dashboard users while maintaining a high-level of performance, as evaluated through several technical aspects.

**Keywords:** Dashboards · Internet of Things (IoT) · Remote Health Monitoring (RHM) · Healthy Aging · Senior residences · Deployment

## **1 Introduction**

Dashboards can be a useful tool to deliver care to older adults while providing realtime health information to healthcare professionals. The advancement in the Internet of Things (IoT) and sensor technology allows for constant monitoring of seniors' health conditions, which is beneficial in reducing the work burden of healthcare professionals. This is especially important when we consider the shortage of healthcare professionals and the swelling population of older adults, leading to more pressure on the healthcare system and inevitable need for supportive solutions.

The population of older adults is increasing in numerous developed countries, including in Canada, where nearly a third of its population belongs to the baby boom generation [1]. This is putting more pressure on the already stretched healthcare system as the demand for healthcare and life aid services for baby boomers will continue to increase[2]. Remote Health Monitoring (RHM) is a new modern approach that has the potential to help healthcare professionals in Senior Residences by continuously monitoring individuals' health factors and transmitting them to healthcare professionals [3]. This can reduce the burden on healthcare professionals and the pressure on the healthcare system, as well as reduce costs and improve the quality of life for older adults. RHM systems generate heavy streams of data, but it is important to deliver this information to healthcare professionals in an understandable manner. Dashboards are powerful interfaces that can use RHM and act as a medium to deliver meaningful information to healthcare professionals and enable them to make informed and timely decisions. However, there are several problems associated with dashboard development and deployment in healthcare, including limited capabilities to respond to diverse needs, preferences, and roles in healthcare, the complexity of using multiple independent dashboards, and the high cost of dashboard development [4, 5]. The design of dashboards must constantly adapt to changing needs and be tailored to the specific preferences and roles of healthcare professionals. The large number of dashboards can lead to islands of knowledge and make it difficult for healthcare professionals to gain insights they need. Additionally, existing development approaches make it difficult to expand solutions to other healthcare centers, increasing costs. Therefore, the development of dashboards requires a deep understanding of real needs of health professionals and the healthcare workflow.

We aim in this study to achieve a tailored solution for a dashboard in the context of healthy aging and senior residences. We are interested in reducing the information load and mental workload of healthcare professionals and improving their effectiveness by providing them with tailored dashboards. Therefore, the overall aim of this research is to provide a cost-effective and efficient monitoring mechanism to reduce the workload of healthcare professionals. More specifically, the presented study aims to provide personalized dashboards integrated with the AMI-Lab platform [6], a complete IoT infrastructure for dynamic and quick deployment of IoT systems which is already being used in several research projects to follow up remotely older adult status.

The rest of the paper is organized as follows: Sect. 2 introduces the diverse approaches to handling tailored dashboards. Section 3 presents the proposed approach, the implementation as well as the results of the experiments and evaluations conducted. We finally outline the conclusion in Sect. 4.

## **2 Literature Review**

There are several studies on building tailored dashboards with diverse approaches to manage the requirements of each user and how to handle tailored dashboards accordingly. We can categorize the key strategies into three: a) Facilitate source code generation, b) Connecting to existing BI tools and c) Personalized solutions.

**a) Facilitate Source Code Generation:** Achieving tailored solutions in code level was one of the approaches taken in the literature. A number of studies took software production line in the literature [7]. This approach uses code templates to achieve dynamic creation of dashboards. User requirements and usage context is fed to the generation engine to generate the source code of the intended dashboard. Similarly, A model-driven approach developed by Palpanas *et al.* [8] follows the same approach using defined models. The difference in this study is that information provided as input for their system is fine-grained based on dashboard structure and it asks to provide more details including navigation, access controls and templates. Based on this provided information, a complete functional code for the dashboard is generated. In all mentioned works that use code approach, the developers have access to final source code after generation of the dashboard and they can modify it to meet their desired customization. Therefore, this solution provides a high level of flexibility. On the other hand, this approach depends on the existence of developers and technical people, and it cannot be used by final users of the dashboard. Moreover, existence of source code means that the final solution should pass all steps required for deployment of a software, ranging from compiling the code to deploying it on the final production system. Nevertheless, none of the studies using this approach mentioned provided specific plans for deployment which can be a very resource consuming task if it is not performed properly.

**b) Connecting to Existing BI Tools:** Interestingly, a number of existing solutions used external BI tools for the final rendering of the dashboard. Tundor *et al.* [9] proposed declaration description of the dashboard, and the final output of their system is a configuration layout which can be used by other BI solutions to represent the designed dashboard. Similarly, Santos *et al*. [10] proposed a knowledge graph which uses an ontology approach to generate dashboards automatically for smart cities. They used API methods to communicate with other BI solutions and reduce the efforts required to create the dashboard. Although leaving the responsibility of rendering the final dashboard can dramatically reduce development time and cost, it has a number of clear drawbacks. The main drawback of this approach is that the software cannot work standalone, and the user needs to link the output of proposed solutions to another external tool to be able to visualize data. Moreover, not all BI tools support common forms of visualization. An element on the designed dashboard might not be able to render on certain BI tools and therefore they cannot be shown on the final dashboard.

**c) Personalized Solutions:** A number of studies aimed to personalize the dashboard to users based on their need. We witnessed two different perspectives to achieve this goal. One perspective was to perform this procedure automatically based on the behavior and usage of end users while they are using the dashboard. For instance, Belo *et al.*[11] aimed to restructure the dashboard elements based on information collected during end-user usage. The other perspective requires explicit definition of user needs and then tailoring dashboard based on declared needs. This idea can be seen in studies of Ines *et al*. [12] and Tundo *et al*. [9]. More specifically, Ines *et al*. [12] designed a questionnaire to extract the needs of the end user and use collected information later to provide personalized dashboard. Similarly, Tundo *et al*. [9] also need an explicit declaration of intended dashboard. In both perspectives, authors assumed that the dashboard has the required visualization elements to reply to the needs of users and the main goal is to make existing elements more accessible for users. They did not mention how they want to handle the changes in users' needs over time or how to provide different users with different data.

**Overall**, the existing solutions in literature provided limited flexibility in terms of design and supporting new visual elements, most of the proposed solution focused on linking existing visual element to appropriate data sources and create a dashboard by combining these elements. Similarly, the majority of proposed solutions have not considered the need for adjusting dashboard after its creation and once the dashboard is created, there is no room for customization or trying different possible dashboard designs. In the existing solutions, we did not witness the option to provide a flexibility in designing the whole frame of a dashboard. Although the content of the dashboard is customized in these solutions, the structure and the frame of the dashboard is fixed for all users and use cases, and the need to change the structure is not supported.

After exploring the literature of tailored dashboards, in the next section we present our designed approach.

## **3 Design Approach**

To provide a solution to respond to the identified needs and enable healthcare professionals to easily create dashboards adapted to each deployment, AMI-Dash has been developed in collaboration with other teams in AMI-Lab with agile methodology. This methodology enabled us to assess the system throughout the development phase in several iterations and assure integrity of our solution with other elements of AMI-Platform to ensure that all parts of the system are working together. Dashboard Studio/Designer is the core of our proposed solution which enables healthcare professionals to design their intended dashboards and automatically adapt them to different environments based on information received from a Configuration Tool.

We followed a four-step agile approach to design AMI-Dash adaptive dashboards. In the **1**st **step**, we defined the requirements for the dashboards by analyzing existing work and grouping the requirements into end-user and technical needs. In the **2**nd **step**, we proposed a solution based on literature and technology to address the identified requirements, focusing on the concepts of "dynamicity" and "scalability." In the **3**rd **step**, we implemented a web-based prototype called AMI-Dash, designed to be userfriendly and adaptable to the real-world environment. In the **4**th **step**, we evaluated the prototype through a two-part evaluation process, measuring the technical performance against the defined requirements.

## **3.1 1st Step: Requirements**

We have identified five categories of technical requirements that involves dashboards in senior residences:

**Req. A) Privacy Prevention and Security:** Protected Health Information (PHI) is an important factor in healthcare dashboards. Legislation in many countries including Canada requires least privilege or "minimum necessary" access to personal health care information [13]. In another word, each health provider should be limited to access only portions of information that is essential for them to perform their tasks. However, the complexity of this requirement lies on the fact that health information is usually stored in a central storage for integrity and accessibility reasons [14].

**Req. B) Fast Easy Deployment of a Dashboard:** Deployment refers to delivering the final version of the designed dashboard to real users. Nowadays more complex software solutions are introduced in numerous domains including healthcare dashboards. These modern solutions usually involve adding third party systems to the solution including databases, firewalls and caching systems [15]. When it comes to deployment of new solutions, all these aspects should be considered to setup which leads to time-consuming procedure and needs of technical individuals and paper works regarding administrative permission.

**Req. C) Real-Time Data Visualization if Needed:** Real-time data refers to data which collected and processed in near real time and is available immediately after it is collected by sensing devices. This form of data can be specifically beneficial to reduce intervention time and improve patient safety, two important use-cases of dashboards in healthcare and senior residences. In both cases, intervention and safety, it is critical for healthcare professionals to notify any abnormal condition of the patient or the monitored person as quickly as possible.

**Req. D) Low Resource Consumption:** Dashboards usually include multiple charts and figures along with other visualization elements like maps and images which potentially can consume a high amount of resources. With the emergence of new RHM devices and IoT sensors, measured data for a large variety of health and environmental factors are available. Having such number of measured factors can be interpreted to have an even larger number of visualization elements on desired dashboards, which require handling a large amount of data and processing. Numerous dashboard solutions [16–18] are deployed on devices like tablets and smart TVs with limited computational resources. Therefore, it is critical for dashboard solutions to consume low resources of the system in order to be able to handle a large number of visualization elements and to be hosted on a variety of devices, especially when there is a need to load real-time data.

**Req. E) Scalability:** Dashboard solution should be able to handle unpredictable requirements in the future. Healthcare domain is very dynamic, and a wide variety of needs exists in this realm. With constant advancements in sensors and healthcare devices, it is highly likely to have new types of devices available in the market. These new devices might need new forms of visualization or new algorithms to enable healthcare professionals to analyze and interpret the data. Therefore, it is essential to be able to support emerging needs in healthcare domain.

#### **3.2 2nd Step: Proposed Approach**

Our approach is based on a number of subsystems to form the complete solution (Fig. 1). The main part of our solution is the Page-Designer which is responsible for enabling healthcare professionals to create and design dashboard pages based on their needs and preferences. This designer is equipped with functionality to communicate with IoT infrastructure of AMI-Lab, and visual elements of this page designer has attributes to connect to real devices in IoT infrastructure. As our proposed solution need to integrate the dashboard with IoT systems (fast dashboard creation), we need to have semantic information about the deployed sensors, which is received from Environmental Configuration Tool (AMI-ECT).

Once we can manipulate information on the deployed sensors and the deployment environment, there is a need for manipulating the representation (view) of this information. Chart-Designer provides healthcare professionals with the ability to manipulate dashboard charts with a great level of flexibility. It provides the ability to manipulate easily not only the layout of the pages and the arrangement of elements, but also, add new components and new visualization types that transform raw data to meaningful information. Lastly, Dashboard-Viewer is responsible for rendering the final layout of the designed dashboard for healthcare professionals with all required visualizations. This viewer along with previously mentioned parts of the solution are integrated with Authentication system of AMI-Platform to separate the design space of each healthcare professional and provide user tailored services.

**Fig. 1.** Overview of proposed solution

Following we discuss how our proposed solution replies to the identified requirements:

**Req. A) Privacy Prevention and Security:** We integrated an authentication system in our solution which handles all the functionality to define users (healthcare professionals) and assigning their roles. Based on healthcare professional information, dashboard system applies limitations according to their roles and gives them appropriate data access and management permission. As an example, the designer may wish to limit a page to be visible only for healthcare professionals with "Nurse" role. Dashboard Studio can handle that based on their identifier as the logic of showing a page is handled there.

**Req. B) Fast Easy Deployment of a Dashboard:** We proposed a subsystem called Dashboard-Viewer in our solution to automate the deployment procedure. Dashboard-Viewer is responsible for reading the configuration of requested dashboard from the data source. The input of Dashboard-Viewer is a JSON object that contains information about the elements inside the dashboard and their arrangement inside the final layout of rendered dashboard. This JSON object starts with a ROOT child which lists all its children and therefore enables the render engine to detect how to start rendering elements. Once configuration is loaded, a render engine in Dashboard-Viewer will parse the configuration to extract items and their arrangement in the dashboard and start rendering the designed dashboard and load its content. As our Dashboard-Viewer is always running on the server, there is no need to configure any additional system and a dashboard is instantly available once its design is finished.

**Req. C) Real-Time Data Visualization if Needed:** To support visualization of realtime data, dashboard components can receive live updates from their data sources and visualize data in a real-time manner. For each component there is a long-running background task which is responsible for regularly updating data of the component on regular intervals. We also enabled healthcare professionals to adjust update intervals based on the sensitivity of the element's data. For instance, they might set update intervals for a heartbeat chart to half a second while setting the update interval of sleep status to five seconds since it has less sensitive data.

**Req. D) Low Resource Consumption:** Dashboard-Viewer is the core part of the system responsible for rendering layout and contents of the dashboard, making it the main target for our efforts to make it efficient. One of our initiatives is to limit refreshing dashboard charts to the time that they are visible, this approach can greatly reduce resource consumption. If a chart is not visible on the page, it will stop sending any request until it becomes visible again. Using Async load methods is another optimization we added. All data elements inside the dashboard send their requests for data in asynchronous mode, which means that loading the data of each chart is not part of loading the dashboard itself. We also enabled the compression on our web server to reduce the size of information received from the server, which minimizes the size of data exchange.

**Req. E) Scalability:** All the elements inside the dashboard are components, and each component can be programed to perform a specific task or visualize information in a certain way. Being components, plays a key role in our implementation as it enables the system to accept new components so that in case we need a new visualization method or dashboard element, we can simply develop and add these components to our system.

#### **3.3 3rd Implementation of AMI-Dash**

AMI-Dash is composed of the following three important components (Environment Configuration Tool, Chart Designer and Dashboard Designer).

**a) Environment Configuration Tool:** We built the Environmental Configuration Tool (AMI-ECT) to generate the semantic information about the deployment place precisely so that it can be consumed by the dashboard studio to drive the generation engine adaptively. AMI-ECT provides a 3D designer in which healthcare professional is able to drag-and-drop physical elements that exist in the deployment place to create a floor plan of a building, as well as place sensors on it and specify their types (Fig. 2). The output of this component is an "Environment Model" which precisely describes the deployment area and contains all data needed to adapt the dashboard design.

**b) Chart Designer** execution can be seen in the Fig. 3. At the very first step the healthcare professional is supposed to create a new chart and specify the title (name) of this new chart and declare if it is a static chart or a template chart. If they choose template charts, the design can be automatically adapted to new environments provided by AMI-ECT. Once this initial information is provided by the healthcare professional, the next step is to specify how to fetch data from data sources and define its filters and timeframe that should be captured from the source. The easiest option is to ask the end user to write a query string. However, there are several clear drawbacks to that approach. First, the healthcare professional needs to have technical knowledge about the database and its structure and the knowledge to write appropriate queries. Apart from that, even if they had the knowledge working which queries, by writing a raw query we limit our logic to a specific database type used in our implementation and if there was any change in the database or its structure, it would directly affect the way end-user will need to define the query. Therefore, we equipped our system with a number of data panels that can help healthcare professionals to define query and filters.

An important aspect of the chart designer is the ability to design templates. Healthcare professionals can define charts as a static chart that will be shown as they are designed initially, or they can use templates which will be adapted to different spaces declared by AMI-ECT. For example, in one deployment we might have a hundred patients equipped with an identical sensor (e.g., blood pressure). In this situation, the design of the chart is the same in all these cases and the only change is refining the data source of the chart. We tackled this problem by invoking a template and linking the blood pressure visualization of all these patients to one template.

**Fig. 2.** Example of running configuration tool.

**Fig. 3.** Overview of all steps in designing charts with AMI-Dash

**C) Dashboard Designer:** The cornerstone of our solution lies in this part of the system as it enables healthcare professionals to design dashboard pages and adapt them to the real-world environment based on the description provided through AMI-ECT. Dashboard Studio consists of five main components which will be explained in the following sections. Figure 4 illustrates the high-level design of AMI-Dash Studio:

**Fig. 4.** Elements of Dashboard Studio

AMI-Dash studio (Fig. 4) consists of five main parts:


## **3.4 4th Step Evaluation**

We conducted an evaluation study to understand to which extent the proposed solution has replied to technical requirements (identified in 1st step). Following details about the study.

**Evaluation Setting:** AMI-Platform has been deployed and running for months now and in several projects located in Canada. We used our solution in two project settings:

#### 60 B. Abdulrazak and A. Rezaei


**Evaluation Metrics:** To validate the proposed solution, we targeted the full-wing metrics that are representative of the technical requirements:

**Fig. 5.** Snapshot of dashboard page designer

**Req. A) Privacy Prevention and Security:** Limiting data access to certain roles was an important identified problem. In all deployments we needed to deliver different dashboards to our partners. Our implemented version successfully handled loading information for all users and no issue regarding authentication is reported. To assure confidentiality in the communication channel, we used a tool called "Charles Proxy" to sniff data exchange between our proposed dashboard and its server. Manual check of headers and content of sending and received packets, confirms that all exchanged data is encrypted using standard TLS methods, which makes content confidential, even if it passed unsecured channels as it uses end-to-end encryption. By taking these two steps, our results confirm that our implementation is capable of replying to the identified requirement.

**Req. B) Fast Deployment:** The proposed solution is deployed in several real apartment with the aim of monitoring older adults at home. Dashboards deployed in these apartments have different floor plans, however, they mostly include five door sensors, seven multi-sensors which measure a number of parameters together (motion, temperature, humidity, and lux). A sleep mat is also part of the project to measure the sleep-related parameters. Similarly, in another project, we needed dashboards to visualize measured data of the deployed Apple Watches and Dexcoms. Once the installment of physical devices was finished, we started to design required charts for these projects in Chart Studio, followed by creating desired dashboard using AMI-Dash Studio, we were able to simply drag and drop the required visual elements and connect them to the appropriate data sources. The whole process of designing the required charts and the dashboard itself took roughly 20 min for apartment deployments. Finally, once the design step is finished, the dashboard is immediately accessible with no need of performing complex procedures. It worth mentioning that developing dashboards using ordinary methods can take several months to even years.

**Req. D) Low Resource Consumption:** CPU was in idle mode for our dashboard, suggesting enough efficiency to even run the dashboard in other media like smartphones and smart TVs.We run the dashboard with 30 figures and 5 images in the dashboard along with normal elements (Texts, Buttons and Containers). As it can be seen in Fig. 6, the CPU consumption is very little and 97% of the times the

**Fig. 6.** Overall resource consumption of our Dashboard Studio

The result of our comparison shows that there is almost no difference between the visualization of the designed dashboard and the normal web page. Additionally, our dashboard is used for a period of two months by AMI-Lab partners and no complaint regarding performance was received.

In another scenario, we measured memory consumed by the dashboard while our dashboard was running and contained a certain number of charts. Each time we added a new chart and checked how much memory is consumed by our dashboard. Overall, the results show a reasonable memory consumption (around 190 MB with 30 charts). The trend of memory consumption was increasing as the number of charts increased. However, after adding 12 charts this increase started to be less considerable. This has roots in our implementation methods in which when a chart is not visible, our system stops sending request and allocating resources for it.

**Req. C) Real-Time Data Visualization:** In order to measure the delay in visualization, we manually inserted a new set of data to our data storage and check how long it takes

**Fig. 7.** Memory consumption and number of charts on the dashboard

for the new data to be reflected on the dashboard. To this end, we used software called "Postman" which is designed to interact with web interfaces and enable us to directly communicate with our backend services and add new data to our data source. Our data storage model contains a timestamp field, which indicates the exact time a new document is inserted in our system. We used this field along with another function which returns exact current time (server time) and we added a temporary label on charts to show the difference (delay). We set the refresh time of the chart at 100 ms and observed the measured delay and in all cases the delay was less than 100ms, proving the capability of our system to show real-time data.

**Req. E) Scalability:** As we used our solution in real deployments of AMI-Lab we faced few use-cases which were not originally defined in our solution. However, as we have used a component-based approach in our design, we are able to easily develop new components to respond to any new need. As an example, a colleague needed to annotate chart data shown on the dashboard, which even needed interaction with data sources to store annotated information. The integration process took less than 30 min for integration, and the added component worked smoothly with no functional error.

## **4 Conclusion**

The study presented in this paper aims to propose a form of tailored dashboards that can allow healthcare professionals to create their own dashboards for senior residences. The motivation behind this study is to support healthy aging and help healthcare professionals deal with the increasing number of senior residents. The need for this study arose due to the limitations of existing technological solutions such as Remote Health Monitoring (RHM) and the Internet of Things (IoT) monitoring which do not provide a medium for healthcare professionals to understand the data generated by these devices.

The study followed an agile approach in four steps to reach its final solution, AMI-Dash. In the first step, the requirements for adaptive dashboards in senior residences were identified. In the second step, the solution was outlined based on the context of work, best practices in the literature of tailored dashboards and stacked technological solutions in AMI-Lab. In the third step, a prototype was implemented based on the outlined solutions. Finally, in the fourth step, an evaluation was performed to measure the usability of the solution for the end-user.

The real needs of healthcare professionals working in senior residences were explored through the literature of healthcare dashboards and the studies that were close to the context of work. The analysis of these studies led to the extraction of a number of requirements for dashboards in senior residences. The final dashboard designer was easy for healthcare professionals to use and provided the option to adapt the dashboard to different deployment places1.

There are some limitations to the study such as the current implementation being coupled with the AMI-Platform and the lack of integration with other systems and managerial aspects. However, future work can involve integrating the solution with other IoT infrastructures and existing management tools in senior residences.

#### **References**


<sup>1</sup> The results of the ergonomic evaluation of our AMI-Dash are the subject of another article.

(eds.) ISMIS 2014. LNCS (LNAI), vol. 8502, pp. 445–455. Springer, Cham (2014). https:// doi.org/10.1007/978-3-319-08326-1\_45


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Drug Intervention Follow up with Internet of Things: A Case Study**

Hassan M. Ahmed1(B) , Souhail Maraoui1, Bessam Abdulrazak1, Benoît Cossette2, and F. Guillaume Blanchet3,4,5

<sup>1</sup> Ambient Intelligence Laboratory (AMI-Lab), Université de Sherbrooke, Département d'informatique, Faculté des Sciences, Sherbrooke, QC J1K 2R1, Canada {hassan.mostafa.ahmed.fahmy,souhail.maraoui, bessam.abdulrazak}@usherbrooke.ca

<sup>2</sup> Centre de recherche sur le vieillissement du CIUSSS de l'Estrie – CHUS, Sherbrooke, Canada benoit.cossette@usherbrooke.ca

<sup>3</sup> Département de Biologie, Faculté des sciences, Université de Sherbrooke, Sherbrooke, Canada guillaume.blanchet@usherbrooke.ca

<sup>4</sup> Département de Mathématiques, Faculté des sciences, Université de Sherbrooke, Sherbrooke, Canada

<sup>5</sup> Département des sciences de la santé communautaire, Faculté de médecine et des sciences de la santé, Université de Sherbrooke, Sherbrooke, Canada

**Abstract.** Advancements on the Internet of Things (IoT) have enabled the development of advanced monitoring systems that can track human behavior and vital signs in real-time, which can have a real impact in the way healthcare is provided. This paper presents a system implementation to monitor and analyze a subject's behavior changes over time using IoT, with the objective of detecting the impact of an inhibitor drug on the subject's activity levels. In this research we present a case study by which we showed it is possible to follow the effect of an anticholinergic drug by means of an unobtrusive IoT system. We have monitored the physical activity of a subject in his residence for seven consecutive months to study the effect of the inhibiting drug doses introduced at three known specific timestamps. Following, we compared our detection results for the subject's physical activity change timestamps with the medical staff medication doses timestamps. Our results show that we can detect the physical activity change at close timestamps compared to those indicated by the medical staff.

**Keywords:** Physical Activity Change Detection · Outliers Detection · Medication Follow-up

## **1 Introduction**

The Internet of Things (IoT) is a rapidly growing network of interconnected devices that can share data and communicate with one another, and it has revolutionized a wide range of industries, including healthcare. IoT devices and systems can be used to gather and analyze data on patients, monitor their health conditions, and provide personalized and timely healthcare [1]. One area where IoT is particularly promising is in the development and monitoring of new drugs, such as inhibitor drugs used to slow down or stop specific biological processes or activities [2]. Monitoring the behavior and activity of subjects under the influence of such drugs is crucial to address two main issues: the drug effect itself and the pharmaceutical metrics [3, 4].

In this paper, we are presenting a proof of concept based on a case study to show we can follow the effect of drug intervention on a subject using IoT. To this end, we have monitored a subject in his residence where he was given three timestamped doses of anticholinergic drug. We used our IoT framework to monitor his daily physical activity inside the residence and analyzing the captured timeseries afterwards. To achieve such a task, the physical activity is monitored using motion sensors embedded in the ambient environment. Following, we used an outliers' detection technique applied to the physical activity of the subject that relied on a sliding window applied to the entire timeseries collected. The mean and standard deviation of the subject's physical activity was quantity within the sliding window. We considered the change in the subject's physical behavior to be defined as outliers from the calculated mean and standard deviation gathered across the sliding window.

This paper is organized as following, Sect. 2 comprises the related work in this particular field of research. The methodology is presented in Sect. 3. Results and discussion are presented in Sects. 4 and 5, respectively. Finally, the conclusion is presented in Sect. 6.

## **2 Related Work**

There has been significant interest in IoT research within the healthcare domain, resulting in the development of various applications for different diseases, settings, and populations. A comprehensive review conducted by Lin et al. [5] surveyed representative and active IoT application products and categorized their focuses into three care settings: (i) acute disease care, (ii) chronic disease care, and (iii) self-health management.

In acute disease care, it is crucial to collect patients' vital signs accurately and in a timely manner. For example, Gao et. al. [6] collaborated closely with medical professionals to identify the key needs in health emergencies. They designed and implemented a system that records patients' vital signs in real-time using wearable sensors, detects anomalies through a vital sign monitoring algorithm, and shares the data with authorized personnel via a web portal.

IoT applications in chronic disease management aim to enable remote health monitoring. For example, Al-Khafajiy *et al.* [7] presented Wearable Sensors for Smart Healthcare Monitoring System (SW-SHMS), a smart healthcare monitoring system that collects physiological data from wearable devices. In their system, the collected data is transmitted to a smartphone app via Bluetooth and then to the cloud over the internet. The processed data is displayed on a monitoring platform accessible by doctors. Another example is IMMED (Indexing MultiMEdia data from wearable sensors for diagnostics and treatment of Dementia) was proposed by Mégret et al. [8]. IMMED relies on wearable cameras to capture video and audio data of older adults' daily activities to assess cognitive decline.

As the importance of health monitoring becomes increasingly recognized, more IT applications are being developed to assist with self-health management. For example, Sadek et al. [9] proposed an unobtrusive Apnea detection using a bed sensor and Sadek et al. [10] proposed a sleep monitoring system using a smart bed. Additionally, smartwatches have been utilized in activity recognition [11] and as a stress detection using heart rate variability [12]. Another technique for monitoring the indoor daily activity was proposed by Ahmed et al. [13] who relied on a thermal sensor array-based (TSA-based) system that can monitor sleeping activity, daily activity, and the no-activity class of a person.

Drug intervention management is important for health management and started to attract attention of clinical research. For example, clinical researchers are interested in evaluating how deprescribing anticholinergic and sedative medications impact people behavior. Based on a literature review, Reeve et al. [14] defined deprescribing as "the process of withdrawal of an inappropriate medication, supervised by a health care professional with the goal of managing polypharmacy and improving outcomes." Although this definition has not yet been validated, it can be used as a basis for any deprescribing drug, even if further work is needed to get expert consensus to develop an internationally accepted definition [14].

Considering Reeves et al. [14] definition, in recent years there are accumulating evidence showing health improvement associated with deprescribing anticholinergic and sedative in older adults. For example, Ailabouni N. et al. [15] found that reducing anticholinergic and sedative medications in frail older people was feasible and beneficial and resulted in better health outcomes, fewer adverse effects, and improved mood and vigor. They also concluded that deprescribing with a patient-centered approach can be safe and effective. Dearing et al. [16] evaluated the effectiveness of a pharmacist-led medication optimization initiative using an electronic tool called the Drug Burden Index (BDI) Calculator in four hospital sites in Canada. This study aimed to achieve three objectives: First, to determine the impact of integrating an electronic clinical decision support tool, the DBI Calculator, into pharmacist medication optimization activities on medication changes and clinical outcomes; Second, to assess whether the effect of the intervention varies based on participant characteristics such as frailty status and sex, as well as the setting of the intervention, including different ward and hospital characteristics; and Third, to explore the implementation of the DBI Calculator into pharmacist-led medication optimization activities during inpatient admissions. The study found significant changes in DBI scores and clinical outcomes, indicating the potential benefit of such interventions. Similarly, Arvisais et al. [17] conducted a study to assess the efficacy of a pharmacist-physician intervention model in reducing the use of high-risk medications in older adult patients. The study reported that the intervention model was effective in reducing the use of high-risk medications. Also, Cossette et al. [18, 19] investigated the impact of a computerized alert system-based pharmacist-physician intervention model on Potentially Inappropriate Medications (PIM) use. The interventions resulted in a higher number of drug cessation and dosage reductions for targeted PIMs compared to a control group. In a pilot study conducted by Cossette et al. [20] in a primary care setting, an interdisciplinary pharmacist-physician intervention model based on alerts generated by a computerized alert system was implemented to reduce the use of PIM in community-dwelling older adults. The study found that a majority of the patients included in the study had medication change suggested by the pharmacist, indicating the potential effectiveness of this intervention model. These results can be generalized older adults living in different regions of the world. Studies conducted in New Zealand [21], Taiwan [22], and Texas [23] have highlighted the prevalence of PIM use and associated factors and suggested the need for screening tools such as the Beers criteria to identify PIMs and interventions to optimize or minimize their use.

Additionally, the use of PIMs can increase the likelihood of adverse drug reactions (ADRs), the harmful or unintended reactions to medications. The prevalence of ADRs is yet another significant concern for health care. Specifically, Beijer and de Blaey [24] found that among the older adult population one in six patients are for ADR as opposed to ~ 4% in the younger population [24]. Various studies have highlighted the burden of ADRs in hospital inpatients at large [25]. These studies have identified risk factors such as increasing age, admission to medical wards, and the number of regular medications. Strategies to minimize the burden of ADRs include computerized prescribing and monitoring systems, pharmacist involvement in ward rounds, improved monitoring, and education on prescribing practices. Harmful events resulting from PIMs also includes adverse drug events (ADEs) [26] that are more likely to occur in older adults with multiple chronic conditions as well as older adults taking specific medication classes. This highlights the importance of considering the overall health and medication profile of older adults to minimize the risk of ADEs. A study by Bressler et al. [27] emphasized the need for physicians to be continuously aware of potential adverse drug interactions in older patients and to consult pharmacists and/or patients to ensure appropriate medication prescribing.

As highlighted by numerous studies [18, 17, 27], ensuring appropriate medication prescription and monitoring of older patients is crucial to avoid ADEs and ADRs. This emphasizes the need for continuous monitoring of patients' behavior and vital signs, a task that can be facilitated using home-based non-intrusive IoT technology. In this context, our proposed system aims at extending the monitoring of patients' physical activity levels using IoT technology, with the objective of detecting the impact of an inhibitor drug on the subject's behavior changes over time.

#### **3 Methodology**

In this section, we first describe our methodology at the system implementation level where we are present our system architecture and our approach to deployment. Following, we present the data analysis algorithm we used in the data collection section.

#### **3.1 System Implementation**

*Architecture Overview:* This project relies on the AMI-Platform [28] solution we have developed. Data flows through the platform as follows: A cluster of heterogeneous sensors at the end-user layer collects environmental and biomedical data, such as temperature, heart rate, and location. This data is then sent to the physical gateway for pre-processing and transmitted to the cloud layer through a secure channel established by the network layer. At the cloud layer, the data is decoded, processed, and stored in databases. The business layer serves as the user interface, offering dynamic graphs and charts for improved data visualization.

*Deployment:* We setup two gateway nodes. The first node handles serial communication with the sleep mat sensor that collects Ballistocardiogram (BCG), from which we can infer on the participant's breathing rate and heartrate during sleep. The second node handles communication with all other deployed sensors that measures the participant's indoor activity, including door sensors but also motion sensors that report on luminance, humidity, and temperature.

#### **3.2 Data Collection and Analysis**

In this study, a subject has been monitored for 7 months, from the end of July 2021 to early January 2022. The objective of this study is to detect changes in the overall behavior of the monitored subject after being given doses of a drug that inhibits physical activity. We have imputed missing activity values using the approach proposed by Ahmed et al. [29]. The overall activity curve for the subject inside his residence during the entire monitoring period is presented by Fig. 1. The timestamps at which these doses are given to the subject are tabulated in Table 1.


**Table 1.** Timestamps at which actions were performed by the team.

The detection algorithm measures the mean and standard deviation within a 3-days sliding window of activity calculated along the complete time series for the subject's physical activity. Box-and-whisker plots were than calculated (Fig. 2) and used to define outliers for both the averages and standard deviations. Note that in our box-and-whisker plots outliers are calculated as all values above or below 1.5 times the interquartile range (the interquartile range where 50% of the data around the median). We defined outliers as change timestamps for the subject's behavior. These outlier values are then back projected to the time domain to obtain the corresponding change timestamps. The corresponding change timestamps are presented in Table 2. The mean and standard deviation timestamps by both average and standard deviation at which the behavior changes are plotted with red dots overlapped on the same overall activity curve and presented by Fig. 2.

The change detected by the mean sliding window conforms with the expected timestamps for the inhibition effect more than that detected by the standard deviation sliding window. This is because these timestamps are close enough in temporal domain to those timestamps at which the 3 doses are given to the subject. However, the standard deviation sliding window detects more change timestamps than the average sliding window does. Although the change is detected at the end of the window, the right change timestamp is corrected by the window size, i.e., three days are subtracted from the detected timestamp.

## **4 Data Analysis Results**

Daily activity time series for the subject under the effect of an activity inhibiting drug is presented in Fig. 1. There are two behavior states that can be found along the time series, and both have a similar meaning. One is characterized by a relatively low standard deviation, we referred to this state as the background activity state. The other is characterized by high standard deviations, we refer to this state as an (low or high) activity induced state. The background activity state is dominant across the monitoring period while the activity induced state occurs for short period in mid-August 2021, at the beginning of November 2021 and during the first week of December 2021. The background activity state is characterized by activity values ranging approximately between 5000 to 7000 movements (per day) with a standard deviation ranging between 0 and roughly 1800 (Fig. 3. Subject behavior changes timestamps overlapped on the activity timeseries). While the activity induced state is characterized by activity values higher than 7000 or lower than 4000 movements (per day).

**Fig. 1.** Subject physical activity time series.

**Fig. 2.** Boxplot and outliers for both average and standard deviation sliding window.

In our pilot study, 60% of the detected change (outliers) timestamps are associated with the standard deviation sliding windows' outliers while the remaining 40% of the detected change timestamps are associated to the mean sliding. Note also that there is a change detection overlap of for 25% of the detected timestamps. There are six groups of change timestamps distributed along the entire daily physical activity timeseries of the subject (Fig. 3. Subject behavior changes timestamps overlapped on the activity timeseries). The 1st group starts in late July and ends in early August 2021, the 2nd group span the second half of August 2021, the 3rd group is in late October 2021, the 4th group is at the beginning of November 2021, the 5th group is at the beginning of December 2021, and the 6th group is in late December 2021. The typical changes' timestamps are tabulated in Table 2.

**Fig. 3.** Subject behavior changes timestamps overlapped on the activity timeseries.


**Table 2.** Detected changes' timestamps and their corresponding values and source of detection.

## **5 Discussion**

For the inhibition drug experiment the average activity level is very high; about 6000 movements on average per day as the subject is a young person. Based on the timeseries, the subject's activity level declines from late July 2021 to mid-August 2021. This decline is conformed with the expected outcome of the inhibition drug given to the subject as the effect of the drug is to inhibit the physical activity of the subject. This observation is in-line with the timestamp marked by the medical staff as the drug's 1st dose intake, as reported in Table 1. There is another decline in the subject's physical activity from October 30th 2021, to November 9th November 2021, which matches the administration of the 2nd dose as reported by the medical staff. A 3rd decline in the subject's physical activity is observed and detected during the period extending from late December 2021 to early January 2022. In addition, a recovery period representing the subject's convalescence from the inhibition drug to his normal overall daily physical activity is observed from mid-August 2021 to early September 2021. A 2nd recovery period is also observed in mid-November 2021. From the subject's activity timeseries, we can observe that a convalescence period of roughly one week is required for the subject to recover from the effect of the inhibition drug.

#### **6 Conclusion**

In this paper we presented a promising application of IoT to monitor inhibitor drugs. We proposed to monitor the behavior and activity of individuals under the influence of drugs to assess their effectiveness, identify side effects, and make timely adjustments to drug levels based on patient status. As such, we have presented a novel approach for monitoring and analyzing the behavior of a subject in an indoor environment over a period of seven months. Our approach relies on the IoT AMI-Platform solution which collected data from a variety of sensors deployed in the participant's environment. Our method uses sliding windows to calculate the mean and standard deviation of the subject's activity over a 3-day period. Following, we rely on box-and-whisker plots to detect outliers in these values. These outliers represent changes in the subject's behavior, which can be further analyzed to understand the effects of specific interventions, such as the administration of an inhibitor drug.

Our results also show the effectiveness of our approach in detecting changes in behavior over a 7-month monitoring period. Specifically, we were able to observe the expected decline in the subject's physical activity following the administration of the inhibitor drug, as well as recovery periods following each dose. Such information is useful to medical professionals to evaluate the effectiveness of the drug and making decisions about its administration in the future. They also highlight our platform as a valuable tool for deploying and analyzing data collected from sensor networks in a variety of applications, including in healthcare as presented in our case study.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Biomedical and Health Informatics**

# **Towards a Medical Test Results Management System Based on Blockchain, Smart Contracts, and NFT: A Case Study in Vietnam**

N. T. Anh1(B) , V. H. Khanh<sup>1</sup>, N. T. Phuc<sup>1</sup>, T. D. Khoa<sup>1</sup>, H. G. Khiem<sup>1</sup>, N. D. P. Trong<sup>1</sup>, V. C. P. Loc<sup>1</sup>, N. T. Q. Duy<sup>1</sup>, T. Q. Bao<sup>1</sup>, D. M. Hieu1,2, H. T. Nghia<sup>1</sup>, N. T. K. Ngan<sup>2</sup>, and L. H. Huong1(B)

> <sup>1</sup> FPT University, Can Tho, Vietnam anhntce160237@fpt.edu.vn, huonglh3@fe.edu.vn <sup>2</sup> FPT Polytechnic, Can Tho, Vietnam

**Abstract.** The role of medical test results in the diagnosis and treatment of a patient's disease cannot be denied. Doctors and medical staff rely on these results to develop a treatment plan that meets the requirements of the patient's health status (i.e., physical condition) and disease type. In developing countries (i.e., Vietnam), we note that test results are recorded in paper versions and stored by patients. Thus, several solutions have been introduced for electronic medical records to compensate for medical test results management. However, in Vietnam, these approaches face many obstacles such as centralized processing (e.g., storage, analysis); non-transparency issues; scalability; availability; and so on. In this paper, we exploit the benefits of blockchain, smart contracts, and NFT technologies to solve the above disadvantages. Therefore, our work contributes to five aspects. (a) Collecting procedures for handling and storing test results of patients at hospitals in Ho Chi Minh City and Mekong Delta (i.e., Can Tho city) (b) Proposing a mechanism for sharing test results based on blockchain technology, smart contract, and NFT applied; (c) Presenting an NFT tool-based certification generation model; (d) implementing the proposed model based on smart contracts (i.e., proof-of-concept); and (e) deploying proof-of-concept on four EVMand NFT-supported platforms to find the most suitable one.

**Keywords:** Medical test result *·* Blockchain *·* Smart contract *·* NFT *·* Ethereum *·* Fantom *·* Polygon *·* Binance Smart Chain

## **1 Introduction**

Building a decentralized patient information management system to be able to share them among the different medical centers is an urgent requirement today, not only in developed countries but also in other developing countries [8]. While developed countries have a specific set of laws and policies regarding the sharing and use of personal/medical data of patients other than diagnostic and therapeutic purposes (eg, GDPR for European countries<sup>1</sup>, Privacy Act for Australia<sup>2</sup>), developing countries have not focused on synchronizing patient medical records [10]. In developing countries, personal data (i.e., including medical data) is not given due attention. Specifically, all information related to medical history and test results is stored locally at medical treatment facilities [6]. In addition, patients are not well aware of the process for handling and sharing their personal information - affecting patient privacy [5].

For medical data processing in developing countries (eg, Vietnam, India), most systems store information in the form of centralized storage. We also surveyed several hospitals in the Mekong Delta provinces (i.e., Ben Tre, Vinh Long) to observe how information about medical test results is stored. We found that the frontline hospitals in large cities store information (called electronic data) only focusing on the treatment route (i.e., patients undergoing internal medicine). inpatient) for outpatient (i.e., home) tests or treatment results are recorded in paper form (called paper data - ie, the procedure is described in Sect. 3).

Nevertheless, centralized storage carries with it many risks [18], for example easily being lost due to patient error or due to natural causes (eg, flood, fire). The loss of medical test results directly affects the diagnosis and treatment process. Hence, a solution is needed to store all this diagnostic information on electronic data. To solve this problem just focusing on infrastructure is not enough. Indeed, Sharma et al. [19] argue that centralized storage limits scalability and limits patient choice. We also note an example in Vietnam related to hospital transfer. Severe cases that cannot be treated at provincial hospitals (eg, Vinh Long, Ben Tre) are transferred to the larger hospitals in Ho Chi Minh City. Information related to testing results will be transferred based on paper documents (instead of directly retrieving from upper-level hospitals).

To solve the above problems, several approaches have combined Blockchain technology and smart contracts to store and process medical data, respectively. For example, Xiao et al. [30] has proposed a system called HGD (Health Care Data Gateway). Thanks to the transparency and decentralized storage of blockchain technology, patients can easily and securely store and share their data. Similarly, Kassab et al. [11] has provided analysis to demonstrate how the benefits of blockchain technology (i.e., synchronous processing, decentralized storage) can enhance data processing and storage capacity for healthcare systems. In addition to the above two preliminary studies, we also summarize some research directions on the same topic based on blockchain technology and smart contract in the related work section of the research paper.

One of the problems that the existing approaches have not solved (i.e., applied to developed countries) is the background of the users. In particular, it is not possible to require users (i.e., patients) to install a series of sharing policies for authorized users [22,28]. In addition, several studies have evaluated the feasibility of storing all medical data on-chain [31]. Most of the above studies provide an

<sup>1</sup> https://gdpr-info.eu/.

<sup>2</sup> https://www.ag.gov.au/rights-and-protections/privacy.

off-chain storage solution of information that is not directly related to the patient's treatment. This has two purposes: (a) restricts on-chain storage reduces the cost of query execution (i.e., data access) [4]; (b) restricts privacy violations because data stored on-chain is only accessible to users of the same system [16]. However, the above approach still faces problems when sharing their medical records with medical centers (i.e., medical staff: doctors, nurses, laboratory staff).

Our research problem in this paper is to fill the limitations on sharing medical data (i.e., medical test results) that apply to Vietnam. We have actually surveyed the testing process and its results storage of 2 large hospitals in Ho Chi Minh City (i.e., the University of Medicine and Pharmacy Hospital and Cho Ray Hospital) 2 hospitals in Can Tho (i.e., central general hospital, the Children's Hospital); 1 hospital in Ben Tre (i.e., Nguyen Dinh Chieu Hospital); and 1 hospital in Vinh Long Province (i.e., Vinh Long General Hospital) in November 2022. Our records range from the time the patient gets tested to the time the results are received and treatment at the hospital. Therefore, our work contributes on five aspects, namely (a) collecting procedures for handling and storing test results of patients at hospitals in Ho Chi Minh City and Mekong Delta (b) proposing a mechanism for sharing test results based on blockchain and smart contract technologies; (c) presenting an NFT tool-based generation model for storing medical test results; (d) providing the proof-of-concept of the proposed model; and (e) deploying our implement on 4 (ERC721 and ERC20)-supported platforms including BNB Smart Chain, Fantom, Polygon, and Celo to choose the most suitable platform for our proposed model.<sup>3</sup>

#### **2 Related Work**

To the best of our knowledge, most of the articles focus on proposing a model to protect personal information and medical data based on two approaches (i) authorization (i.e., access control) and blockchain technology. In this section we categorize state-of-the-arts based on the above two themes. Thereby, analyze the shortcomings in the current work and propose our model.

#### **2.1 The Approaches Based on Access Control Models**

Most state-of-the-arts exploit access control models (RBAC, ABAC, ABE) to ensure access to patient medical data. In particular, data retrieval requests must be approved by the data owner (i.e., patient). For example, Makubalo et al. [14] has synthesized access control model-based approaches to empowering patients to share medical information. Specifically, policies will help patients manage who (eg, nurses, doctors) is allowed to access their medical data and use it for a specific purpose (eg, treatment). Another example applying attribute-based

<sup>3</sup> We do not deploy smart contracts on ETH because the execution fee of smart contracts is too high.

encryption (ABE) was introduced by Barua et al. [1] and Yin et al. [32] to encrypt all medical data before storing it in the cloud. Only objects that satisfy the required attribute have the key to decrypt them. In addition, based on the Role-based Access Control model Chen et al. [2] described a requester role-based access management model. Besides, some methods apply dynamic policy model to increase flexibility in medical environment [9,21] or design two layers of policy, ie, i) security policy - protect data patient data from intrusion by users outside the system, and ii) privacy policy that protects patient information from agents in the system [28]. Combining dynamic policy-based approaches and the ABAC model, son et al. introduced a two-layer access control model based on medical environments [22].

#### **2.2 The Approaches Based on Blockchain Technology**

Chen et al. [3] proposed a model based on Blockchain technology applied to healthcare environment to store and control mining data from IoT devices. These devices store real-time medical data. All information stored on the chain is authenticated and controlled by the user. Madin et al. [13] has proposed a model of storing medical record information on blockchain and storing details of that information on IPFS. The purpose of this group of authors is to protect the privacy of patient data (i.e., limit access from users of the same system). MedAccess introduced the patient Health Record management system [15]. The proposed system focuses on protecting patient information from being altered (i.e., transparency) and privacy (offchain storage). Similar to the approach of [15], to minimize the amount of information stored on the chain, Zyskind et al. [33] presents an onchain and offchain patient data and information storage model.

For actual deployed solutions, HealthBank<sup>4</sup> combined blockchain and IPFS to propose a patient-centric model for sharing medical data. Several solutions are introduced to comply with privacy constraints (eg, GDPR). For example, Health-Nautica and Factom Announce Partnership<sup>5</sup> presented a decentralized data storage solution. and protect personal information based on Blockchain technology. Another approach based on data streaming is also proposed by IRYO<sup>6</sup> based on NuCypher KMS (ie [7]).

All of the above evaluation directions face two major challenges for both the access control model-based approach and the Blockchain and IPFS-based approach. Specifically, with the first approach of empowering users based on access policies, the approaches are limited when users have to create a series of policies to control access accordingly. These policies are prone to conflicts or redundancies if the number of access objects is large and there are many properties [28]. Moreover, getting used to a completely new system (i.e., defining security policies) also makes

<sup>4</sup> https://www.healthbank.coop/2018/10/30/healthbank-creates-the-first-patient-cen tric-healthcare-trust-ecosystem/.

<sup>5</sup> https://www.factom.com/company-updates/healthnautica-factom-announce-partn ership/.

<sup>6</sup> https://iryo.network/iryo whitepaper.pdf.

users encounter some obstacles due to lack of necessary background (especially for casual users) in developing countries. As for the second approach (blockchain and IPFS-based), it is agreed that this method brings a lot of efficiency in the process of storing and processing medical data, but on-chain data storage solutions and out-of-string also brings a lot of complications for the average user when they want to share. To limit the above risks, within the framework of the problem of sharing medical test results at medical centers in Vietnam, we combine a series of modern platforms including blockchain, smart contracts, and NFT. Specifically, instead of sharing policies to define groups of people who can retrieve test results or patient history, we create corresponding NFTs. This approach restricts the strict requirements of the technology platform such as security policy-based methods. The next section presents our proposed model based on NFT technology to share information with the respective objects.

## **3 The Blockchain-Based Medical Test Results Management System**

In this section, we summarize the traditional test results management process applied in Vietnam (i.e., in provinces and cities in the Mekong Delta) as well as propose a model based on blockchain technology. and smart contracts to solve the limitations of the traditional model as well as NFT technology to generate electrical test results for easy sharing in the medical environment.

#### **3.1 Traditional Medical Test Results Management Model**

**Fig. 1.** Traditional medical test results management model

Figure 1 shows the traditional process of testing and getting results based on four steps (i.e., depending on the hospital in the city or the country, the process is different). At provincial/major city hospitals, the waiting time to get tested and receive test results is very long. In particular, according to our perception at the University of Medicine and Pharmacy Hospital in Ho Chi Minh City. In Ho Chi Minh City, some patients have to be present from the hospital from 1:00 AM to queue to receive the order number even though the opening time there is 8:00 AM. Therefore, the procedure shown in Fig. 1 shows only the general steps and omits the detailed steps depending on the hospital. Specifically, the patient creates an archive of health information called medical record book at the medical center.<sup>7</sup> The patients undergo the necessary medical examinations before proceeding with the treatment. This information is extremely important - the doctor's diagnoses and judgments are based on the changes in the patient's health status through the test results. After receiving the results from the laboratory staff, the patient brings the results to the doctors and nurses to make judgments about their health status and corresponding treatments. All information about the diagnosis and treatment results are updated in the medical record book. Therefore, the loss of the patient's medical record book seriously affects the treatment process. Therefore, there is a need for a comprehensive solution to the problems related to the storage and sharing of patients' medical record books for both electronic and paper formats. The following section presents our solution based on current popular technologies: blockchain, smart contract and NFT applied to the Vietnamese environment.

## **3.2 Medical Test Results Management Model Based on Blockchain Technology, Smart Contract and NFT**

**Fig. 2.** Medical test results management model based on blockchain technology, smart contract and NFT

Figure 2 shows nine steps to build a medical test result management system based on blockchain technology, smart contract and NFT. Specifically, the patient must

<sup>7</sup> According to our records, most patients are given a paper version of the medical record book - one A number of medical conditions and treatment history or medical records are collected and stored on a local server.

create a global patience ID that is shared by the entire medical system in Vietnam (step 1). According to our observations, the role of patience ID global is not considered in the current system at the medical facility in Vietnam. Specifically, each patient can only receive one identification code (i.e., valid for the day) at each different hospital. The storage of patient information in medical facilities is not supported. Step 2 Synchronize patience ID global with the medical record book to store all information about the medical record and test results as well as the patient's medical history. This information is displayed on the user interface based on our provided services (UI services) in step 3. Other users can access patient information to get a list of test requests. of the patient (step 4); update test results (step 5); or get feedback on your doctor/nurse's medical condition, diagnosis, and treatment steps (step 6). All these steps are performed and managed based on pre-designed methods in smart contracts (step 7) and updated to the distributed ledger (step 8). The content of the NFTs is created in step 9 (i.e., the content of the NFTs - medical test results are presented in the Introduction) and shared with doctors and nurses for the purpose of patient care and treatment.

#### **4 Evaluation Scenarios**

#### **4.1 EVM-Supported Platforms**


**Fig. 3.** The transaction info (e.g., BNB Smart Chain)

Because the model for generating medical test results makes it easy for patients to manage test results (i.e., can be extended to medical records) as well as easily share them with other objects that users want. We want instead of defining security policies (eg, access control), we implement the proposed model on EVMenabled blockchain platforms instead of exploiting Hyperledger eco-system platforms because they are easy to open. wide (i.e., using existing platforms and systems). In addition, assessments based on system responsiveness (i.e., number of requests responded successfully/failed, system latency - min, max, average) have been evaluated by us in the previous research paper. Therefore, in this paper we determine the suitable platform for our proposed model. Specifically, we install a recommendation system on four popular blockchain platforms today, supporting Ethereum Virtual Machine (EVM), including Binance Smart Chain (BNB Smart Chain)<sup>8</sup>; Polygon<sup>9</sup>; Fantom<sup>10</sup>; and Celo<sup>11</sup>. Our implementations on these four platforms are also shared as a contribution to the article to collect transaction fees corresponding to the four platforms' supporting coins<sup>12</sup>, ie, BNB<sup>13</sup>; MATIC<sup>14</sup>; FTM<sup>15</sup>; and CELO<sup>16</sup>. For example, Fig. 3 details our three assessments of a successful installation on BNB Smart Chain (i.e., similar settings are shown for the other three platforms). Our implementations to evaluate the execution cost of smart contracts (i.e., designed based on Solidity language) run on testnet environments of four platforms in order to choose the most cost-effective platform to deploy. Our detailed assessments focus on the cost of performing contract creation, NFT generation, and NFT retrieval/transfer presented in the respective subsections related to i) Transaction Fee; ii) Gas limit; and iii) Gas used by transaction.

#### **4.2 Transaction Fee**


**Table 1.** Transaction fee

Table 1 shows the cost of creating contracts for the four platforms. It is easy to see that the highest transaction fee of the three requirements is contract creation for all four platforms. In which, the cost of BNB Smart Chain is the highest with the highest cost when creating a contract is 0.0273134 BNB (\$8.43); whereas, the lowest cost recorded by the Fantom platform with the highest cost for contract initiation is less than 0.00957754 FTM (\$0.001849). Meanwhile, the cost to enforce Celo's contract initiation requirement is lower than Polygon's with only \$0.004 compared to \$0.01. For the remaining two requirements (Create

<sup>8</sup> https://github.com/bnb-chain/whitepaper/blob/master/WHITEPAPER.md.

<sup>9</sup> https://polygon.technology/lightpaper-polygon.pdf.

<sup>10</sup> https://whitepaper.io/document/438/fantom-whitepaper.

<sup>11</sup> https://celo.org/papers/whitepaper.

<sup>12</sup> Implementation of theme models our release at 11/24/2022, 8:44:53 AM UTC.

<sup>13</sup> https://testnet.bscscan.com/address/0xafa3888d1dfbfe957b1cd68c36ede4991e104a 53.

<sup>14</sup> https://mumbai.polygonscan.com/address/0xd9ee80d850ef3c4978dd0b099a45a559f d7c5ef4.

<sup>15</sup> https://testnet.ftmscan.com/address/0x4a2573478c67a894e32d806c8dd23ee8e26f78 47.

<sup>16</sup> https://explorer.celo.org/alfajores/address/0x4a2573478C67a894E32D806c8Dd23E E8E26f7847/transactions.

NFT and Transfer NFT), we note that the cost of implementing them for all three platforms, Polygon, Celo, and Fantom is very low (i.e., negligible) given the cost. trades close to \$0.00. However, this cost is still very high when deployed on BNB Smart Chain with 0.00109162 BNB (\$0.34) and 0.00057003 BNB (\$0.18) for Create NFT and Transfer NFT, respectively.

#### **4.3 Gas Limit**


**Table 2.** Gas limit

Table 2 shows the gas limit for each transaction. Our observations show that the gas limits of the three platforms (i.e., BNB, Polygon, and Fantom) are roughly equivalent - where Polygon and Fantom are similar in all three transactions. The remaining platform (i.e., Celo) has the highest gas limit with 3,548,922; 142,040; and 85,673 for all three transaction types.

#### **4.4 Gas Used by Transaction**


**Table 3.** Gas Used by Transaction

Table 3 shows the amount of gas used when executing the transaction (i.e., what percentage of gas in total gas is shown in Table 2). Specifically, three platforms BNB, Polygon, and Fantom use 100% of Gas Limit for two transactions Contract Creation and Create NFT. Meanwhile, Celo uses 76.92% of the Gas limit for the above two transactions. For the last transaction of Transfer NFT, the highest Gas level was recorded by Fantom and Polygon with 93.41% of Gas limit; while BNB and Celo use 79.17% and 69.8% of Gas limit.

## **5 Discussion**

According to our observation, the transaction value depends on the market capitalization of the respective coin. The total market capitalization of the 4 platforms used in our review (i.e., BNB (Binance Smart Chain); MATIC (Polygon); FTM (Fantom); and CELO (Celo)) are \$50,959,673,206 respectively. \$7,652,386, 190; \$486,510,485; and \$244,775,762.<sup>17</sup> This directly affects the coin value of that platform – although the number of coins issued at the time of system implementation also plays a huge role. The total issuance of the four coins BNB, MATIC, FTM, and CELO is 163,276,974/163,276,974 coins; 8,868,740,690/10,000,000,000 coins; 2,541,152,731/3,175,000,000 coins and 473, 376,178/1,000,000,000 coins. The coin's value is conventionally based on the number of coins issued and the total market capitalization with a value of \$314.98; \$0.863099; \$0.1909; and \$0.528049 for BNB, MATIC, FTM, and CELO respectively.

Based on the measurements and analysis in 4 section, we have concluded that the proposed model deployed on Faltom brings many benefits related to system operating costs. In particular, generating and receiving NFTs has an almost zero (i.e., negligible) fee. Also, the cost of creating contracts with transaction execution value is very low (i.e., less than \$0.002). In future work, we plan to implement complex methods/algorithms (i.e., encryption and decryption) and more complex data structures to observe the costs for the respective transactions. Deploying the proposed model in a real environment is also a possible approach (i.e., implementing the recommendation system on the FTM mainnet). In our current analysis, we have not considered issues related to the privacy policy of users [23] (i.e., access control [21,22], dynamic policy [20,29]) - a possible approach would be implemented in upcoming work. Finally, infrastructure-based approaches (i.e., gRPC [12,25]; Microservices [17,26]; Dynamic transmission messages [27] and Brokerless [24]) can be integrated into the model of us to increase user interaction (i.e., API-call-based approach).

## **6 Conclusion**

Our article aims at a solution to share test result data in the medical environment based on the benefits of current technologies such as blockchain, smart contracts, and NFT. Our solution aims to replace the paper documents used in hospitals in the South of Vietnam, including hospitals in Ho Chi Minh City. Ho Chi Minh and the Mekong Delta. Specifically, these results are stored off-chain and generate corresponding NFTs upon request from treatment facilities. To reduce patient effort, we replace policy-based approaches with data-sharing solutions for disease care and treatment. Our solution is implemented on the Ethereum platform and Solidity language (i.e., smart contract development). In order to optimize the execution cost of basic transactions (i.e., contract creation, NFT creation, NFT transfer), we deploy proof-of-concept on four EVM-enabled platforms, including

<sup>17</sup> Our observation time is 12:00PM - 11/26/2022.

BNB, MATIC, FTM, and CELO. According to our observations (i.e., executed at the time of system evaluation), smart contracts deployed on Fantom bring more economic value (i.e., cost savings in execution) than the other three platforms. The future directions of the current work are presented in the Discussion section.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Agitated Behaviors Detection in Children with ASD Using Wearable Data**

Imen Montassar1(B) , Belkacem Chikhaoui<sup>2</sup>, and Shengrui Wang<sup>1</sup>

<sup>1</sup> Department of Computer Science, Universit´e de Sherbrooke, Sherbrooke, Qc, Canada

*{*Imen.Montassar,Shengrui.Wang*}*@usherbrooke.ca <sup>2</sup> Department of Science and Technology, Universit´e TELUQ, ´ Quebec City, Qc, Canada Belkacem.Chikhaoui@teluq.ca

**Abstract.** Children diagnosed with Autism Spectrum Disorder (ASD) often exhibit agitated behaviors that can isolate them from their peers. This study aims to examine if wearable data, collected during everyday activities, could effectively detect such behaviors. First, we used the Empatica E4 device to collect real data including Blood Volume Pulse (BVP), Electrodermal Activity (EDA), and Acceleration (ACC), from a 9 years-old male child with autism over 6 months. Second, we analyzed and extracted numerous features from each signal, and employed different classifiers including Support Vector Machine (SVM), Random Forest (FR), eXtreme Gradient Boosting (XGBoost), and TabNet. Our preliminary findings showed good performance in comparison with the state of the art. Notably, XGBoost demonstrated the highest performance in terms of accuracy, precision, recall, and F1-score. The accuracy achieved in this paper using XGBoost is equal to 80% which exceeds previous research.

**Keywords:** ASD *·* Agitated behaviors *·* Wearable data *·* Features extraction *·* Classification

## **1 Introduction**

Autism Spectrum Disorder (ASD) is a neurodevelopmental disability characterized by persistent difficulties in social communication and interaction [1]. Patients with ASD could display repetitive patterns of challenging behaviors such as self-injury, meltdown and tantrum, and aggression [2] which arise from a set of triggers.

The agitation has negative outcomes for both ASD patients and their caregivers. It can lead to increased stress levels and impaired quality of life [3]. Consequently, the need for assistive technologies to improve the well-being of children with ASD and their caregivers is growing. With the advance of technology, we could develop solutions to prevent these maladaptive behaviors. Before, behaviors were reported by caregivers. In such cases, the physical presence of the caregiver with the child during his daily activity is required. In this regard, the development of automatic detection methods based on wearable signals is needed to replace this constant physical assistance. Recently, the detection of agitated behaviors based on wearable data has grabbed the attention of scientists. Although, there is a lack of literature related to this topic.

Given the above-mentioned, this paper explores the feasibility of detecting agitated behaviors of ASD children based on their physiological and kinetic signals. We will compare different classification algorithms (SVM, Random Forest, XGBoost, and TabNet). The main contributions of our study are:


The remainder of this paper is as follows: Sect. 2, provides a literature review. We will give more details about our proposed models 3. Section 4 discusses our results and compares them to previous research. Finally, Sect. 5 concludes this paper.

## **2 Literature Review**

In this section, We will first define EDA and BVP signals, then we will review related work in the subsequent section.

#### **2.1 Physiological Signals**

Biomedical signals are very rich with information about changes in the psychophysiological systems. Hence, we will investigate the performance of some physiological signals to detect agitated behaviors of ASD children, basically EDA and BVP signals.

**Electrodermal Activity:** also known as Galvanic Skin Response GSR, reflects the autonomic changes in the electrical and conductivity properties of the skin consequent to an increased sweat glands activity [4,5]. EDA has been widely used as a physiological index in psychophysiological research since it is linked to autonomic emotional and cognitive processes. EDA serves as a valuable biomedical indicator in many fields such as emotions and behaviors [6].

EDA is an aggregate of two components: the tonic activity (Skin Conductance Level SCL) and the phasic activity (Skin Conductance Response SCR) [4]. Several methods have been developed to extract SCR and SCL from EDA. In this paper, we will use the *cvxEDA* [7] which relays on convex optimization and maximum a posteriori approach.

The phasic component corresponds to a series of SCR peaks typically characterized by three components: *onsets, peaks amplitude*, and *half-recovery*. Some methods have been suggested to identify these peaks and their characteristics. In our case, we used the method described in [8] to detect these peaks from SCR.

**Blood Volume Pulse:** is measured using the PPG sensor, which relays on the pulse oximeter technique to measure the blood volume in the micro-vascular of the target tissue [9]. In fact, the sensor illuminates the surface of the skin with IR light and measures changes in the back-scattered light and the portion absorbed by the skin [10]. Even though this technique is inexpensive and suitable for daily life applications, the BVP signal is susceptible to several artifacts. Nevertheless, health-related features such as heart rate and heart rate variability can be extracted from BVP.

Basically, the signal has three characteristic points: *diastolic point*, *systolic point*, and *dicrotic notch*.

In our study, we will use the *NeuroKit* [26] and *HRV analysis* [25] Python packages to process EDA and BVP signals. These packages offer comprehensive tools for the analysis and interpretation of physiological data.

#### **2.2 Related Works**

The literature on the detection of agitated behaviors using biomedical data is limited. While there have been studies investigating similar areas such as emotion recognition [11,12], anxiety, and stress detection [13] of ASD patients.

Alban et al. [14] developed three machine learning models (Decision Trees, SVM, Multi-Layer Perceptron) to detect challenging behaviors of a 10-year-old male ASD child in interaction with social robots and toys based on physiological and kinetic signals. From each signal, they extracted four time-domain features (mean, min, max, and std) from a window of two seconds. To balance the dataset, they applied resampling. Combining all the signals and using the MLP, they achieved the best accuracy equal to 97%. Although the result reflects a good performance, it suffers from subjectivity to the experimental protocol of the study (interaction with social robots and toys).

Meltdowns and tantrums often lead to impulsivity and agitation. Khullar et al. [15] designed a wristband with three sensors to collect Heart Rate (HR), Skin Temperature (ST), and Galvanic Skin Response (GSR) in order to detect meltdowns and tantrums in ASD patients. Different preprocessing techniques were applied (null values removal, filtering, normalization). A CNN+LSTM model was developed to detect these behaviors. The performance in terms of precision, recall, and F-1 is equal to 0*.*98, 0*.*95, and 0*.*97 respectively. Unfortunately, the authors did not provide any detail about the ASD patients or the data collection protocol.

Other studies have developed models to predict atypical behaviors. For instance, Goodwin et al. [16] developed a predictive model to anticipate agitated behaviors 1-min before they occur using physiological and kinetic signals. Their study involved 20, ASD patients. The signals were filtered and processed to extract 10 time-domain statistical features, and 2 binary aggression labels were also extracted and used to train a ridge-regularized logistic regression model. Two types of models were investigated: a global model using data from all participants, and 20-person dependent models, developed for each patient. The Area Under the Curve AUC of the global model is 0*.*71, while the average AUC for person-independent models was 0*.*84.

In a second study, Goodwin et al. [17] improved their previous models by using Principal Component Analysis PCA for features reduction and kernelbased SVM to predict aggressive behaviors 3 min in advance. The global model achieved an increased AUC of 0*.*98.

The major drawback of the aforementioned studies is that the datasets were collected in controlled environments under a specific experimental setup. Consequently, the results are subjective to the experimental protocols. In contrast, our study focuses on developing detection models for agitated behaviors in a real-world context during the everyday activities of an ASD child.

## **3 Proposed Models**

#### **3.1 Data Collection**

It is important to note that there is currently no publicly available dataset for this specific topic. Consequently, the first step consists in collecting real data from an ASD child using the Empatica E4.

We were able to recruit a 9-year-old male autistic child to take part in our experiments, although many families we contacted refused to participate. The patient suffers from moderate to severe autism and does not take any medication. Data collection took place over a period of 6 months during his daily activities (playing outside, doing homework, etc.).

E4 records EDA (4 Hz), BVP (64 Hz), Heart Rate HR (1 Hz), Inter Beat Intervals IBI, Skin Temperature ST (4 Hz), and acceleration ACC along the *x*, *<sup>y</sup>*, and *<sup>z</sup>−*axis (32 Hz). The wristband has an internal 60 hours memory capacity. Data transfer between E4 and the computer is made via an E4 manager Software that should be installed on the computer. The recorded signals are exported to *csv* files for further analysis.

The child wore the device and performed his activities in a naturalistic manner. The parents accompanied their child to annotate the moments of agitation. It should be noted that the participant refused to wear the device during school, resulting in most of the activities being conducted at home. Therefore, the moments of agitation were relatively rare compared to normal behaviors. In fact, the child acts aggressively at school or when confronting other people.

#### **3.2 Signal Processing**

The quality of signals significantly impacts the performance of a model. Mainly, in this step, we will apply filters to reduce the noise from raw signals, followed by the extraction of relevant features. The collected signals are corrupted by different sources of artifacts. The following paragraphs detail the distinct processing steps employed to extract relevant features from raw signals.

**EDA Processing:** EDA is hampered by its sensitivity to motion artifacts [19]. Raw EDA is firstly pre-processed using the wavelet-based-adaptive denoising procedure as described in [19]. Secondly, the signal is filtered using 4*th* order Butterworth low-pass forward-backward filter with *Fcut* = 0*.*5 Hz as in [20]. Thirdly, we employed min-max normalization to remove the difference between the different recordings. Moving forward, the next steps include the decomposition of EDA into SCR and SCL using the cvxEDA algorithm [7]. The three components are segmented into overlapping windows for features extraction [20,21]. In contrast to papers reviewed previously that have investigated a limited set of statistical features from raw EDA, our work encompasses a wider range of features from raw EDA, SCR, and SCL. More specifically, we extract time-domain features, peak-related features, and frequency-domain features. In total, we calculate 80 distinct features, which are summarized in Table 1.


**BVP Processing:** The BVP signals of our recordings are very noisy. We first, used the winsorization technique to remove values above 98*th* and below 2*sd* percentile as in [20]. Next, we applied a 6*th* order Butterworth band-pass forwardbackward filter with a frequency range [0*.*6*Hz <sup>−</sup>* <sup>3</sup>*.*3*Hz*] as in [22]. Later, we normalized the cleaned BVP using min-max. To extract systolic points, we used the pipeline described in [23], which are later cleaned using the HRV-analysis [25] to discard outliers and ectopic peaks [24]. RR intervals are calculated from BVP based on systolic points. The different types of features extracted from the RR interval basically comprise time-domain, frequency-domain, geometrical, and Poincare-plot features. The following Table 2 defines the 30 BVP features.


**Table 2.** Extracted features from BVP signal.

**ACC Processing:** We filtered the ACC signals as suggested by [27]. We applied a low-pass 4*th* order Butterworth filter with *Fcut* = 10 HZ. Features extracted from each axis include time-domain and frequency-domain features as summarized in Table 3. The total number of features extracted from ACC data is 69.


**Table 3.** Extracted features from ACC signals.

#### **3.3 Methodology**

Our main goal is to detect agitated behaviors of ASD patients using wearable data. For this purpose, we will establish different binary classifiers. While, as stated above, the dataset collected suffers from severe imbalance. For skewed class distribution, we typically apply cost-sensitive learning models [28]. This is because misclassifying agitated behaviors carries a higher cost compared to misclassifying normal behaviors during the model training process.

The first step after collecting the data consists in processing the different signals and extracting salient features. Subsequently, we will explore several types of classifiers: Support Vector Machine [30], Random Forest [29], XGBoost, and TabNet [32]. SVM works by finding a hyperplane that separates the classes of the training data. Random Forest is an ensemble learning algorithm based on multiple decision trees each of which is trained on a different subset of the training data and a random subset of features. XGBoost follows the principle of boosting [31] which consists in combining a set of weak classifiers to create a stronger one. TabNet on the other hand, is a deep-learning model designed for tabular data. It uses a sequence of attention mechanisms to evaluate the importance of each feature at every training step.

## **4 Experiments**

## **4.1 Setup**

Our dataset comprises multiple sessions, with the child exhibiting aggressive behaviors in only 6 of the recordings, and these behaviors were observed for a few minutes. To extract features, we segmented the recordings into windows of 30 seconds with a 50% overlap. For each window, we calculated the features illustrated in the previous Sect. 3.2. The next step consists in training the different models over a dataset containing segments from all sessions. To ensure optimal model performance, we conducted an extensive search for the best hyperparameters for each model using the Grid Search method. The hyperparameters that yielded the best results are presented below:


## **4.2 Evaluation Metrics**

We used stratified 10*−*fold cross-validations to test the performance of our models since it keeps the same dataset imbalance proportion. We calculated precision, recall, F-1 score, and balanced accuracy. Furthermore, we calculated the AUC and Precision-Recall AUC (PR AUC). AUC measures the model's performance by calculating the True Positive rate against the False Positive rate. On the other hand, PR AUC measures the precision and recall of a model. This metric is useful when the model suffers from imbalance.

## **4.3 Results**

In this section, we will present the results achieved by each classifier and compare them to previous research.

Table 4 corresponds to the average evaluation metrics values using the different classifiers and combining the features from EDA, BVP, and ACC signals.

These results revealed that the XGBoost model outperformed RF, TabNet, and SVM, achieving an 80% balanced accuracy. Specifically, precision, recall, F1-score, AUC, and PR AUC of XGBoost are 0*.*77*,* 0*.*60*,* 0*.*67*,* 0*.*80, and 0*.*72,


**Table 4.** Average evaluation metrics of the stratified 10-fold of the different classifiers using the combined features (Mean% *±* Standard Deviation%)

respectively. SVM achieved similar precision as XGBoost (0*.*7) while the recall is notably lower (0*.*38) resulting in a lower F1-score of 0*.*50. In terms of PR AUC, SVM achieved 0*.*61. RF model achieved the lowest average F1-score (0*.*41) and PR AUC (0*.*43). In the literature, TabNet outperformed XGBoost in several competitions including Kaggle, in our study, XGBoost yielded the best performance. One possible reason could be the limited amount of data to train TabNet. Nevertheless, it still achieved the second highest F1-score, AUC, and balanced accuracy with values of 56%*,* 76%*,* and 77% respectively, indicating acceptable preliminary results. Collecting more data and exploring the parameters of Tab-Net could potentially improve its performance. In addition to its performance, XGBoost exhibited faster training and validation times compared to TabNet.

We compare our results to a previous study [14] in Table 5. In [14], the authors developed an MLP classifier to detect the challenging behaviors of an ASD male child using four statistical features (min, max, mean, and std) from EDA, BVP, HR, and ACC. Table 5 compares the results achieved using XGBoost and the performance of the MLP in [14].


**Table 5.** Performance comparison of our model and the results of [14]

This comparative Table 5 shows that our method achieved higher performance. In fact, the features extracted in our paper contain a large amount of information, which could reflect the behavioral state of the child.

**Fig. 1.** The importance of features determined by XGBoost classifier of each signal EDA (a), BVP (b), and ACC (c)

With a focus to identify the contribution of each signal to our detection model, we calculated the features importance of the XGBoost classifier. The feature vector combines EDA, BVP, and ACC information respectively resulting in a 179*−*dimensional vector. Figure 1 displays the contribution of each feature of each signal: EDA 1a, BVP 1b, and ACC 1c.

Based on our results, the extracted features of the EDA, BVP, and ACC signal contributed 80*.*15%, 5*.*6%, and 14*.*25% respectively to XGBoost. These results confirm the literature assumption about the performance of EDA in identifying the psychophysiological state of individuals. As mentioned in Sect. 2, EDA has been extensively used in studies related to arousal investigation such as stress and anxiety, and emotion detection. Conversely, BVP contributes less to the detection of agitated behaviors, potentially due to the fact that this signal is very susceptible to environmental artifacts, leading to a less reliable heart rate and heart rate variability readings.

Previous research focused on kinetic data to detect agitated behaviors. However, this kind of signal solely could not contribute to a good generalization. Our results demonstrated the feasibility of using physiological biomarkers combined with kinetic data to detect agitated behaviors of people on the spectrum.

## **5 Conclusion**

Agitated behaviors represent significant challenges for individuals with autism and their caregivers, impacting their well-being. Technological solutions could improve their quality of life. This paper attempted to study the viability of using signals collected from wearable devices during everyday activities to detect agitation. The proposed framework involved pre-processing to remove additive noise, followed by relevant features extraction for each type of signal. The features were later combined to train different classifiers to detect agitated behaviors.

Our approach showed promising results across multiple evaluation metrics. More specifically, XGBoost achieved the highest performance in terms of balanced accuracy, precision, recall, F1-score, AUC, and PR-AUC.

These findings suffer from some limitations including a limited number of recorded moments of agitation. Moreover, we were only able to recruit a single ASD child. We firmly believe that the proposed models could be improved through the collection of a larger dataset from a larger number of ASD patients, hence refining the performance of classifiers.

The implication of our findings is significant in terms of enhancing the quality of life of patients with ASD. By providing caregivers with timely notification of moments of agitation, they can prevent harmful consequences associated with these destructive behaviors. Our proposed framework holds the potential to alleviate the negative impact of these behaviors and promote a safer and more supportive environment for patients with autism. Additionally, our future work will focus as well on developing predictive models capable of anticipating agitated behaviors before their occurrence. This proactive approach in this case will help caregivers to intervene early and prevent unwanted consequences.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Application of Performance Test Method in Korea for LED Optical Medical Device Samples**

Jae Young Lee1, In-ho Hwang2, and Sang Geon Park3(B)

<sup>1</sup> Department of Culture Convergence Technology Interdisciplinary programs, Changwon National University, 20 Changwondaehak-ro, Uichang-gu, Changwon-si, Gyeongsangnam-do 51140, Republic of Korea

<sup>2</sup> Korea Orthopedics and Rehabilitation Engineering Center, 26 Gyeongin-ro 10beon-gil,

Bupyeong-gu, Incheon 21417, Republic of Korea

ihhwang@kcomwel.or.kr

<sup>3</sup> Department of Mechatronics Convergence, Changwon National University, 20

Changwondaehak-ro, Uichang-gu, Changwon-si, Gyeongsangnam-do 51140, Republic of Korea

sgpark@changwon.ac.kr

**Abstract.** The development of light-based technologies has facilitated the use of phototherapy devices in the medical field. Representative phototherapy devices are laser surgical instruments and medical laser irradiators. While the former are used for the removal, ablation, and destruction of areas in need of treatment using the light of a strong laser; the latter irradiate the skin, induce skin regeneration, and relieve pain [1]. These medical devices are used under the guidance of healthcare providers in medical institutions and can be hazardous because they use strong light. According to the Korean Ministry of Food and Drug Safety, medical devices are divided into four classes according to the hazard level [2]. Devices such as lasers are currently represented by Class 3, which represents a high risk. However, various personal phototherapy devices have recently emerged and are used without any instructions from healthcare providers. Among them, representative personal phototherapy devices include those that use LED. Such devices are known to have effects such as pain and inflammation relief, scar healing, and skin regeneration according to previous studies [3]. With the progress of research, LED medical devices for skin improvement and beauty treatment are emerging in Korea. Thus, LED medical devices are used for various purposes. Certain devices only use LEDs, while others are used in com-bination with lasers their boundary with medical laser irradiators is vague. Based on the report on personal phototherapy devices by the Korea Consumer Agency, there were 172 adverse events of LED masks, which were home phototherapy devices, from 2018 to Au-gust 2020; they comprise 134 cases of skin and subcutaneous tissue injuries, six of burns, one of heat sensation and dyspnoea, and one of bruising [4]. As various phototherapy de-vices continue to be developed, assuring adequate safety is paramount. This study con-tributes to the industrial development of LED phototherapy devices and new phototherapy devices. We derived a performance test method to ensure safety related to the performance of LED phototherapy devices in the Korean context.

**Keywords:** LED phototherapy device 1 · safety 2 · Medical device 3

#### **1 Introduction**

The development of light-based technologies has facilitated the use of various types of phototherapy devices in the medical field. Representative phototherapy devices are laser surgical instruments and medical laser irradiators. The former are used for the removal, ablation, and destruction of areas in need of treatment using the light of a strong laser; the latter irradiate the skin, induce skin regeneration, and relieve pain [1]. These medical devices are used under the guidance of healthcare providers in medical institutions and can be hazardous because they use strong light. According to the Korean Ministry of Food and Drug Safety, medical devices are divided into four classes according to the hazard level [2]. Devices such as lasers are currently represented by Class 3, which is the grade with high risk. However, various types of personal phototherapy devices have recently emerged and are used without any instructions from healthcare providers. Among them, representative personal phototherapy devices include phototherapy devices using LED. LED phototherapy devices use a low-level phototherapeutic method known as Low-Level Laser Therapy (LLLT). LLLT is used for relief of pain or inflammation, scar healing, and skin resurfacing using low output power [3]. LED phototherapy devices with LLLT are used for skin improvement and cosmetic purposes. They vary in shape depending on the site at which to be used, and are of various types. Certain devices use only LEDs, while others are used in combination with laser—their boundary with medical laser irradiators is vague. Based on the report on personal phototherapy devices by the Korea Consumer Agency, there were 172 adverse events of LED masks, which were home phototherapy devices, from 2018 to August 2020. They comprise 134 cases of skin and subcutaneous tissue injuries, 6 of burns, 1 of heat sensation and dyspnoea, and 1 of bruise [4]. As various phototherapy devices continue to be developed, assuring adequate safety is paramount.

This study contributed to the industrial development of LED phototherapy devices and new phototherapy devices. We derived a performance test method to secure safety related to the performance of phototherapy devices using LEDs, from a Korean perspective.

## **2 Materials and Method**

#### **2.1 Determination of Performance Test Items Related to LED Phototherapy Devices**

It's determine to derive the performance test method to preserve the stability of LED optical medical devices and to obtain suitable test materials for these devices by investigating performance issues and conducting related performance tests. As a research method, we proceed by reviewing previous research and standards related to optical medical devices such as International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC). For the literature review, a literature search medium was used, and the search was conducted with "LLLT," "PBM," and "LED" as search keywords. Based on the reviewed literature, performance test items and methods were derived by examining key performance issues related to optical medical device therapy and LED optical medical device standards.

#### **2.2 Performance Test Method for the Phototherapy Device Sample**

We examined the safety of the LED phototherapy devices by applying the performance test items derived from this study to the LED medical device developed by T company, shown in Fig. 1. The company's LED phototherapy device samples were developed for the treatment of hair loss and scalp improvement and was shaped like a helmet. The LED phototherapy device comprises 54 lasers and 126 tri-color LEDs. Its performance items are 10 mW for the output power of the laser, 3 mW for the intended output power, and 650 nm for wavelength. Each LED has an output power of 3 mW, with wavelengths of red 625 nm, blue 455 nm, and infrared (IR) 845 nm. The key performance test items and methods of optical medical devices are obtained and used in the model to protect the validity of the test.

**Fig. 1.** LED phototherapy device sample.

## **3 Results**

#### **3.1 LED Optical Medical Device Literature Investigation**

As a result of the literature review, three volumes were reviewed for LLLT, two volumes for LED, two volumes for PBM optical medical device-related literature, and five volumes for related standards. In addition, two books related to Korean medical devices and photo-therapy were mentioned. By reviewing the research literature, main performance items according to the treatment of LED light medical devices, standards related to LED light medical devices, and performance test items and methods were obtained.

#### **3.1.1 Key Performance Points for Treatment Using LED Optical Medical Devices**

Phototherapy is a medical technique that has been used for thousands of years to treat various diseases. An example is sun therapy by Hippocrates, the father of Western medicine [5]. In modern times, various treatments using light have evolved. A common treatment that uses LEDs is photobiomodulation (PBM). This was previously called Low-Level Laser Therapy (LLLT) and was later changed to Low-Level Light Therapy by Kendrick C. Smith due to the advent of the next generation LED with higher photon intensity. Photobiologist [6]. After the change, the name was again changed to PBM due to uncertainty about the exact meaning of low power [7]. PBM was proposed by Andre Meister (1903–1984), a medical expert who researched and published the biological effects of low-power lasers in 1967 [8]. According to the literature, PBM is a treatment that uses a low-power laser or LED in the 1m to 50mW range to induce tissue regeneration and relieve inflammation. Red or near-infrared light with a wavelength of 600 nm to 1000 nm is used, and, unlike other laser treatments that destroy tissue, PBM is known for its photochemical effect similar to plant photosynthesis [9]. The mechanism of PBM is not yet clearly elucidated; however, when tissue is irradiated with low-output IR light, it is absorbed by a chromophore that penetrates the skin surface and absorbs light in a specific wavelength band. In the body, at this time [10], the absorbed light energy is converted into chemical energy and increases the synthesis of Adenosine Triphosphate (ATP), which is the energy source of the cells in the cell tissue, resulting in cell regeneration and rehabilitation, pain and inflammation relief, which is known [11, 12]. According to the literature related to LED-LLLT, the main performance issues for LED light sources used in PPM include wavelength, power density, and luminance. In terms of wavelength, the first law of light biology, the Grothes-Draper law, states that when the light of the appropriate wavelength is applied to the chromophore, the chromophore has no absorption nor reaction. Power density refers to the output used per unit area, and the unit is W*/*Cm^2. When the power density is insufficient, light absorption does not occur in the tissue, and when the power density is high, it is converted to excessive heat for the tissue, which could be harmful. Fluence (energy density) or "dose" refers to the amount of energy applied per unit area. The energy density is multiplied by the radiation time, and the unit is J*/*Cm^2. If the power density is too low, the irradiation time must be extended to achieve the best power density, which is why the desired results cannot be obtained [6, 10].

#### **3.1.2 Items for Standards Related to LED Phototherapy Devices**

To derive LED phototherapy device test items, the relevant standards were investigated by referring to the international standards, ISO and IEC, as well as Korean standards. Currently, there are two main categories of common standards for medical devices in South Korea. The classification consists of standards applied comprehensively and standards applied separately by medical device items. Additionally, it comprises medical devices that use electricity as a whole and those that do not use electricity [13]. Comprehensive standards are common standards for electromechanical safety, electro-magnetic waves and safety, and biological safety. The common standards for electromechanical safety are those applicable to the basic safety and essential performance of medical electrical appliances and medical electrical systems [13]. In the common standards on electromechanical safety, standards related to phototherapy devices were IEC 60601-1, IEC 60601-2-22, IEC 60825-1, IEC 60825-8, and IEC 62471-2. IEC 60601-1 is a set of standards that ensure the safety of medical electrical equipment, cover basic safety and essential performance requirements of medical electrical equipment, and prevent electrical, mechanical, or functional errors that present unacceptable hazards to patients and operators [14]. IEC 60601-2-22 is an individual standard of IEC 60601–1 standard applied to basic safety and essential performance of surgical, therapeutic, diagnostic, cosmetic, and animal laser instruments for use in humans or animals. IEC 60601-2-22 is a standard applicable to phototherapy devices corresponding to classes 3B to 4 and applied to the safety of laser output power. For example, on the 201.12.1.101 laser output power display, the actual laser output power measured in the operating area should not exceed ±20% of the set value [15]. IEC 60825-1 is a standard applied to the safety of laser products that emit light with a wavelength range of 180 nm to 1 mm. Per this standard, classes are di-vided into 1, 1M, 1C, 2, 2M, 3R, 3B, and 4, as described in Table 1 [16].


**Table 1.** Laser classes according to IEC 60825–1 [16].

These ratings are classified as per the accessible emission limit (AEL) according to the Maximum Permissible Exposure (MPE) in Annex A of IEC 60825-1, as shown in Table 2. The MPE provided in IEC 60825-1 presents additional information that can assist the manufacturer in assessing safety items for the intended use of the user. The MPE contained in the standard has been adopted from exposure limits published by the International Commission on Non-Ionized Radiation Protection, and MPE values have been used as the basis for safe design of products and for providing information to users. [16] IEC 60825-8 provides guidance to employers, competent authorities, laser safety managers, and others, on the safe use of laser and laser equipment classified as 3B and 4. An investigation identified potential side effects according to wavelengths in Appendix A of IEC 60825-1, which are shown in Table 2 [17].

#### **3.1.3 LED Phototherapy Device Performance Test Items and Method Derivation**

The main performance requirements and specifications related to performance test-ing for optical medical devices were grasped through the literature. It has been con-firmed that wavelength, power density, and fluorescence are key performance parameters for LED optical medical devices. If a wavelength that does not match the chromophore of the target to be treated is used, suitable results cannot be obtained; in the case of fluorescence that does not match, the desired result cannot be obtained, and thermal damage can be expected. Therefore here, the key performance test items obtained were wavelength, power, and power density, as shown in Table 3. For assessing wavelength, we used the wave-length accuracy test method according to IEC 62471-1 Annex B; the accurate calculation formula was used to obtain the measured value and the actual value as a percent-age, and the quality of the test ± -% within the error range was determined. Power density is the output per unit area, and an accuracy test for output and a test for illumination was obtained. Output Accuracy Test IEC 62471-1 Annex C. As a test method, the measured and the actual values are calculated as a percentage using a power meter. The quality of the test should be ± -% with an error margin of ± -%. The test for illumination is IEC 62471–1. 5, wherein we follow the measurements of the lamp and the lamp device and measure it using an illuminance meter. The test standard is IEC 62471-1. 4. 4. The test method that follows the hazard exposure limits is IEC 62471- 1.5. The test method was obtained ac-cording to the measurement of lamps and lighting fixtures, and IEC 62471-1 4. Exposure limit standards were followed. Energy density, one of the key performance factors, is the product of energy density and irradiation time, and the test item of energy density was obtained. However, irradiation time was excluded because it was difficult to be obtained. The test item will vary depending on the LED optical medical device.

#### **3.2 LED Optical Medical Device Literature Investigation**

Among the test items obtained from Sect. 3.1.3, the wavelength accuracy test was applied to LEDs built into the Company T model, and the same test was performed on the laser built into the model. To measure the wavelength, as shown in Fig. 2 below, we want to measure the main performance factors of the two models of the T company, the laser wavelength, and the LED wavelength, using the spectrometer of the T company. The two models have the same performance and we wanted to compare them by measuring the wavelengths in each mode.

As for the measurement method, the light source to be measured was aligned with the spectrometer's wavelength meter sensor, as shown in Fig. 3. The wavelength meas-ured was then checked by the monitoring function in the software on the computer con-nected to the spectrometer.


**Table 2.** Laser classes according to IEC 60825-1 [16].



#### **3.3 Performance Test Results of LED Phototherapy Devices**

The two samples of T company were numbered 1 and 2, and the laser and LED were measured for each mode of the samples. The set values for each mode of samples were the same for the laser and IR LED, and the set values for the laser and the IR LED were 650 nm and 845 nm, respectively. The set values for the red LED in H (Hair) mode and the blue LED in S (Scalp) mode were 625 nm and 455 nm, respectively. The measured values for the two samples are shown in Fig. 4.

The laser wavelengths per the mode of T company's samples measured with the spectrometer are shown in Fig. 5. The graph shows a maximum peak value of 654 nm.

**Fig. 2.** Wavelength performance test devices (Left: a spectrometer; Right: the sensor of the spectrometer)

**Fig. 3.** Wavelength measurement for the light source.

**Fig. 4.** Results of laser wavelength measurements by mode of T company's samples.

When the measured wavelength band was visually checked, the graphs overlapped because the measured laser wavelength bands were similar to the set value of 650 nm. However, as it was difficult to accurately differentiate the superimposed graph with the naked eye, the graph was adjusted to distinguish the laser wavelength band measured, as shown in Fig. 6. Using the adjusted graph, the measured wavelength for each mode of the two samples and the accuracy of the laser wavelengths were checked.

Laser measurements of T company's samples revealed 654 nm in H mode, 654 nm in S mode (Sample-1), 652 nm in H mode, and 653 nm in S mode (Sample-2). They differed by 2 to 4 nm from the set value of 650 nm in the laser. As each of the measured lasers was the same for each of the samples, Sample 1 indicated the same value. However, Sample 2, though it was the same object, demonstrated a difference of 1 nm depending on the mode. When each measured value was applied to the wavelength accuracy equation, Sample

**Fig. 5.** Enhanced graph of laser wavelengths measured in modes of T company's samples.

**Fig. 6.** LED wavelength graph measured in sample modes of T company's device.

1 showed an error range of ±0.615% because the values per mode were the same. As for Sample 2, the error range was ±0.31% in H mode and ±0.461% in S mode. The laser wave-lengths of the two samples were within the error range of ±5%, which was the test standard set by the manufacturer. For both samples, LED measurements with the spectrometer revealed the maximum peak values of 627 nm, 454 nm, and 838 nm, which were visible to the eye. When the graph was checked, both samples were found to be similarly matched and overlapped with the laser graph measured earlier. However, a difference of 1 to 3 nm was observed compared to the LED set values of 625 nm for red, 455 nm for blue, and 845 nm for IR. The graphs were adjusted, as shown in Figs. 7, 8, and 9 for precise determination. The respective graphs confirmed LED wavelengths by each mode and the accuracy of wave-lengths for the two samples.

The IR LED measured value was 838 nm in both H and S modes for Sample 1. In Sample 2, the values for H and S modes were 840 nm and 839 nm, respectively. For each wavelength, there were differences of 7 nm in Sample 1 and 5 to 6 nm in Sample 2 from the set value of 845 nm. The wavelength accuracy of Sample 1 was within ±0.828% because all measured values were the same. The wavelength accuracy of Sample 2 was ±0.591% in H mode and ±0.71% in S mode. Both samples were within the test standard of ±5%. In the case of IR LED and laser, the same object was applied for each mode. The measured values according to the mode were distinctly different from the set value.

**Fig. 7.** Graph of laser wavelengths.

**Fig. 8.** Red LED wavelength graph measured in the samples of T company's device.

The red LED of T company's samples was used only in H mode. The measured values in Samples 1 and 2 were 627 nm with a difference of 2 nm from the set value of 625 nm. The accuracy of the red wavelength was ±0.32% for both samples, thus indicating that both samples had an error range within ±5%. Unlike the laser and IR LED measured earlier, the red LEDs had the same values for Samples 1 and 2, and the difference from the set value was small.

**Fig. 9.** Blue LED wavelength graph measured in the samples of T company's device.

The blue LED of T company's samples is used only in S mode. The measured values were 455 nm in Sample 1 and 454 nm in Sample 2. In Sample 1, the measured value was the same as the set value of 455 nm, whereas Sample 2 had a difference of 1 nm. Regarding the accuracy of wavelength, Sample 1 had an error range of ±0% because the measured value was equal to the set value, whereas Sample 2 had ±0.219% in the error range. Thus, the values were within the test standard of ±5% in the error range. In this study, we tested the wavelength performance of both the laser and the LED used in the samples of the T company, from which we found that all measured values were within the reference values. The measured value of the blue LED was the closest to the set value than the data measured earlier, which was found stable, as shown in Fig. 9. The values of the wave-lengths measured in this study and their accuracy are shown in Table 4. In conclusion, all measured values were within ±5%, the test standard. However, the laser and IR LED, which are used regardless of mode, had larger differences between the measured values and the set values among the measurements.

## **4 Conclusions**

This study investigated items related to the performance of LED phototherapy devices to ensure their safety. Although LLLT using LED phototherapy devices, which have been recently examined, have no reported principle, it was possible to identify wavelength as the main performance item, which is the same for other phototherapy devices. The standard of phototherapy devices equipped with both LED and lasers such as T company's phototherapy device needs to be explored in combination with LED and laser standards. The investigation of standards was able to identify the classes of medical laser de-vices in IEC 60825-1 and their hazard depending on wavelength in IEC 60825-8, the appendix of IEC 60825-1. However, certain laser standards were found to be applied to 3B or higher classes of a laser. As the samples of T company's device used in this study were used in combination with LED and low-level laser, it was difficult to investigate the standards because of the differentiation between LED and laser. For LED phototherapy devices, IEC 62471-1, which is the same standard as lamp-based medical devices, was applied. Referring to IEC 62471-1 and the relevant literature and guidelines, this study de-rived the performance test method and test items for wavelength, the main performance item, and applied them to samples of the T company's device. The derived performance evaluation item was the accuracy test of the wavelength according to IEC 62471-1. The test was to determine whether the LED and laser wavelengths were stable. The test standard was within ±5% in the error range corresponding to the test standard of T company. The test method checked whether the measured wavelength shown through the spectrometer was within the error range through the percentage. According to the measurement results, the light sources of the samples complied with the test standard set by the manufacturer. The measured wavelength for samples differed from the set value by 1 to 2 nm in red and blue LEDs, and the accuracy error range of the wavelength was close to 0%. However, laser and IR LEDs, unlike red and blue LEDs, had a difference as large as 2 to 7 nm between the set value and the measured value, though they are used regardless of the mode. More-over, the error range was also shown to be close to 1%, unlike those of the red and blue LEDs. To make it closer to the set value, it is necessary to study the wave-length of photo-therapy devices whose accuracy is important for safety. We believe that a study on the wavelength of the phototherapy device similar to the set value would con-tribute to the safety of LED phototherapy devices and the industrial advancement of phototherapy de-vices similar to LED phototherapy devices.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Detecting the Pre-impact of Falls in the Elderly, Along with the Use of an Airbag Belt for Protection Against Femoral Neck Fractures**

Mohand O. Seddar1,3(B) , Guillaume Rao2(B), Anthony Fleury1(B) , and Maurice Kahn3(B)

> <sup>1</sup> IMT Nord Europe, CERI SN, Lille, France {anthony.fleury,mohand.seddar}@imt-nord-europe.fr <sup>2</sup> Aix Marseille Univ, CNRS, ISM, Marseille, France guillaume.rao@univ-amu.fr <sup>3</sup> INDIENOV, Marseille, France {mseddar,mkahn}@indienov.com

**Abstract.** Falls are a significant health risk for older adults, and fallrelated injuries are a leading cause of morbidity and mortality in this population. Elderlies are particularly vulnerable to falls due to age-related declines in mobility, balance, and muscle strength, as well as chronic medical conditions with use of certain medications. These injuries can range from minor bruises and scrapes to more severe like fractures, head trauma, or internal bleeding. To prevent falls in older adults, some solutions propose to ensure a safe living environment, others to maintain physical activity, and others to manage chronic medical conditions. This article presents the implementation and test of a system preventing hip fractures resulting from falls using a fall detection and prediction system designed to protect and alert individuals during falls.

**Keywords:** Pre-impact fall detection · fall detection · machine learning · tinyML · medical devices · IMU · e-health · airbag system

## **1 Introduction**

Falls among the elderly are currently a real human and financial scourge. 25 to 30% of people over 65 and 50% of those over 80 fall at least once a year. These are the first cause of emergency visits for people over 75 and the second cause of death by unintentional trauma. A significant proportion of falls result in femoral neck fractures, a real public health issue (Katsoulis, 2017 [1] Oberlin, 2016 [2]). Furthermore, research has demonstrated that the apprehension of falling is linked to unfavorable outcomes, including reduced engagement in everyday activities, decreased level of physical activity, increased likelihood of falling, symptoms of depression, and a lower overall quality of life [3]. According to INSERM (2017) [4], between 50,000 and 80,000 individuals in France, and approximately 1.5 million around the world (Silver Eco 2017 portal) experience fractures of the

**Fig. 1.** Belt Airbag illustration

neck of the femur. This type of fracture has severe implications as one-quarter of people pass away within one year, and 50% of those who survive experience a considerable loss of autonomy. This work is part of a project led by Indienov, a company that created a smart airbag belt capable of analyzing the movements of individuals and detecting anomalies (in this particular case falls and prefalls), and instantly activate a perfectly positioned airbag prevent fractures of the femoral neck or greater trochanter (Fig. 1). The belt will also be able to detect critical situations such as lying for a prolonged time.

## **2 Literature Review**

A large variety of fall detection systems are available and rely on diverse sensors (cameras, infrared motion sensors, LIDAR, and inertial motion units). These fall detection systems are generally classified into two categories: fixed systems (in the environment) and mobile/wearable systems (necklaces, watches, belts, phones, etc.) [5]. We focused on mobile systems using Inertial Motion Units.

#### **2.1 State of the Art**

Initial research concerning fall detection systems employed threshold-based algorithms, and demonstrated that by applying thresholds to vertical and horizontal velocities, it is possible to differentiate between daily activities and falls [6,7] with remarkable accuracy. In a study based on a sensor located on the chest [20], a fall detection algorithm is optimized using the Upper Fall Threshold (UFT) and Lower Fall Threshold (LFT) of the accelerometer combined with the UFT of the gyroscope for fall detection. The experimental results showed that the algorithm was able to detect falls compared to other daily movements with a sensitivity and specificity of 96.3% and 96.2%, respectively. The addition of gyroscope data significantly improves specificity over published results based solely on accelerometer data, as changes in angular velocity provide an additional indication of a fall event compared to other activities that may also experience high acceleration spikes. The use of LFTs and UFTs is equivalent to determining the window of fall and detecting it by means of its start and end.

Alternatively, Li et al. [8] used static postures and dynamic transitions between them to detect falls, defining a fall as a static "lying" posture with an unintentional transition to it. Numerous other studies have examined fall detection, including reviews, learning algorithms with automatic feature extraction, etc. [9–18], some of which have been found to be highly reliable [19].

Detecting the onset of a fall, or pre-fall detection, presents a challenging problem. Unlike fall detection, which relies on identifying the impact of a fall visible on acceleration signals, pre-fall detection requires detecting the early signs of an impending fall. As a result, accelerometer and gyroscope signals may not yet exhibit features indicative of a fall. The primary hurdle in solving this issue is to distinguish the crucial warning signs that trigger a fall while minimizing the number of inaccurate detections (false positive cases).

In the context of pre-fall detection, one of the earliest fall protection systems was proposed by G. Shi et al. [21] and later enhanced [22]. It relies on an inertial motion unit and airbag deployment, along with a Support Vector Machine (SVM) prediction model and an embedded Digital Signal Processing (DSP) unit.

In recent years, machine learning models have gained popularity due to their good performances in fall and pre-fall detection. In a comparative study conducted by Yu et al. [24], a hybrid ConvLSTM model was evaluated and compared to other existing models. The results showed that the ConvLSTM model outperformed other models for all three classes (no drop, drop before impact, and drop), with average sensitivities of 93.15%, 93.78%, and 96.00% for no drop, drop before impact, and drop, respectively. Furthermore, the model demonstrated higher specificities for all three classes (96.59%, 94.49%, and 98.69%) compared to the LSTM and CNN models. These results suggest that the proposed hybrid model outperforms the LSTM and CNN models, providing high detection accuracy (particularly for the pre-impact drop). The combination of convolution and recurrent models allows feature extraction and temporal linkage between the data, leading to satisfactory results compared to using these models separately.

Recent fall prediction methods have shown very promising results (as shown in Table 1). However, one important issue highlighted in the literature is the need to compare the performance of fall detection algorithms using real-world data,


**Table 1.** Fall prediction algorithms comparison

**Table 2.** Comparison of the Kfall model performances on simulated data and real data. Source : [27]


as significant discrepancies have been observed between the results obtained from simulated falls data of healthy individuals and real falls data from at-risk populations [11]. The Bourke and Chen algorithms perform the best in terms of sensitivity and specificity, but all algorithms exhibit lower performance on real falls compared to simulated falls. The latter being also true for the work from Yu et al. (as shown in Table 2) despite the very high performance obtained on simulation data (99.32%, 99.03% of sensitivity, and specificity, respectively). This difference in performance was mainly attributed to the fact that the movements in real falls do not exhibit the same signatures as simulated falls, especially in terms of the amplitude before impact [28].

## **3 Methods and Material**

#### **3.1 Data Collection**

As mentioned above, the problem of motion and fall data for at-risk individuals currently remains one of the biggest challenges. The first step consists of obtaining real, reliable motion data that comes from our target audience. Then, in addition to retrieving widely used movement data from the internet (especially in research, e.g. SisFall [30], MobiFall [29], kFall [27] etc.), a specific protocol based on [29,30] was developed to gather experimental data (Table 3 and 4).

A data collection campaign has been initiated in EHPAD (institutions for dependent elderly people) (Dataset 03, Dataset 04), in which multiple volunteers were equipped with dataloggers (embedded sensors with IMU + SD card attached to a belt worn at the waist) to monitor their activities throughout the day (and in some cases, the entire week from morning to evening). The purpose of the database obtained from this data collection is to develop solutions and algorithms adapted to the target population, the elderly, where detecting pre-fall events is of utmost importance. As the participants' profiles are highly suitable


**Table 3.** Public Data Overview


**Table 4.** Acquired Data Overview

for this purpose, the collected data is expected to provide valuable insights and aid in validating the developed solutions and algorithms.

#### **3.2 Configuration and Setup**

After examining the SisFall and MobiFall datasets, we discovered that they were not optimal for predicting falls because they lacked temporal information on the pre-fall phase. Even though there has been a lot of research on fall detection, there hasn't been much done on prefall detection. The key contrast between prefall detection and fall detection is found here. Prefall detection focuses on detecting the beginning of a fall, which happens over a brief period of time and then passes. In other words, throughout the fall, we can only see the beginning of the fall during a specified time period. On the other side, fall detection focuses on identifying a fall when it really occurs, generally after the impact has taken place. At any time after the fall, it is relatively simple to identify because the impact of the fall clearly leaves an imprint in the signals. Therefore, we had to identify and label the pre-fall phase, which may have introduced bias into the labeling process. In contrast, the kfall dataset contained more detailed information on the onset and impact of falls, making it more useful for fall prediction. Additionally, the age and profiles of the participants in the various datasets were not very representative, so we generated new data for activities of daily living and falls that included a wide range of realistic scenarios.

For the labeled data, participants followed a predefined protocol that included activities from the SisFall dataset as well as additional daily activities such as walking, using stairs with a cane or walker, doing household chores, and dancing. We also included wheelchair activities. Unlabeled data were collected from participants engaged in their normal activities throughout the day, resulting in data that better reflects real-world scenarios. For data recording, we used dataloggers, which are developed by the same company, and we recorded the videos of these simulations. The IMU (MPU6050) that we used was equipped with a 3-axis accelerometer and gyroscope, with a resolution of ±8 [g] for the accelerometer and ±1000◦/s for the gyroscope. The sampling frequency (200 [Hz]) is defined based on the data analysis and signal processing.

## **3.3 Data Processing and Feature Extraction**

**Data Cleaning and Labelling:** To reduce accelerometer sensitivity noise, we used a 3rd order Butterworth lowpass filter with a cutoff frequency of 20 [Hz] as first step. We then divided the fall data into four stages (Fig. 2): non-risk, pre-fall, the impact, and the post-fall. This categorization was based on notes taken during fall simulations (such as the type of fall, subject, and duration) and recorded video data (including the start time of the fall, the end time of the pre-fall phase, the time of ground contact, and the time of impact).

**Data Analysis and Feature Selection:** An initial visual inspection was performed to analyze the collected data. This enabled the identification of movement-specific signatures and variable behavior, and the tracking of their fluctuations over time in relation to distinct activities and subject profiles (as shown in Fig. 3). A statistical examination of the information obtained from various activities was performed with the aim of identifying indicators that can

**Fig. 2.** Fall Data Segmentation.

**Fig. 3.** Activities signal examples - Acc: accelerometer (acc x, acc y, acc z), Gyro: gyroscope (gyro x, gyro y, gyro z) - 3 Normal Activities (1, 2, 3) vs one Fall Activity (4)

**Fig. 4.** Analysis of the data distribution examples - compare the data distribution over different activities (ADLs vs FALLs)


**Table 5.** Selected Variables for Feature calculation

distinguish between ADLs and falls (Fig. 4). Once this stage was completed, we selected the variables that would be used in computing the features (listed in Table 5). Subsequently, statistical computations were carried out on each variable, resulting in a collection of potential features for pre-fall detection (Table 6). The features were calculated in sliding windows of 0.5[s].

**Algorithm and Features:** For feature selection, we started an optimization process over all the features (cited on Table 6), based on recursive elimination of features by KNN (K Nearest Neighbors) and RF (Random Forest) algorithms, so we tested several combinations depending on the type of falls we were proposing and the performances we were getting. Ultimately, we chose an algorithm based on the RF (random forest) and the most important features, which we selected


**Table 6.** Selected Features for Calculation

**Fig. 5.** Fall Detection/Prediction System.

for their relevance to decision making. Our solution detects pre-falls and falls (Fig. 5), then, sends an alert to relatives and caregivers (previously selected).

#### **3.4 Model Deployment in Embedded Architecture**

Due to their inherent resource constraints, embedded systems are unable to support sophisticated inference models. As a result, implementing AI models in these systems becomes difficult, especially in real-time scenarios when characteristics like response speed, space complexity, and computing complexity are critical. Focusing on low complexity choices is necessary since these limits impose extra factors to take into account when choosing the right model. Deep learning techniques, for instance, may not be practical due to their inference, time, and space complexity, necessitating careful attention throughout implementation. As a result, it may be necessary to develop customized methods to optimize model training, conversion, and implementation for better optimization and utilization of hardware resources such as DSP and FPU. Various methods (weight pruning, quantization, encoding, etc.) and solutions (TensorFlow Lite Micro, EdgeML, STM32Cube.AI, etc.) are available to address this challenge [31], but they require careful consideration and planning to achieve the desired results in the context of TinyML.

## **4 Experiments and Evaluation**

## **4.1 Experiments**

Initially, we tested our solution on daily life and fall acquired data to ensure its functionality (this was the Experiment 00). Our ultimate goal was to develop a functional and customized solution for the airbag belt. Therefore, we implemented and deployed the solution in the embedded system (belt) and conducted three types of experiments.


The first and second tests were conducted by young and healthy individuals.

## **4.2 Evaluation**

We established a set of evaluation criteria for our solutions, which serve to measure their quality and degree of reliability. These criteria include:


Finally, we use performance as an overall measure of how well our solution performs based on the aforementioned criteria. This value is calculated using a weighted sum of sensitivity, specificity, and time lead. Based on the desired outcomes, it is possible to set the weights of the parameters, such as emphasizing sensitivity, specificity, or lead time.

Performances = (Psn ∗ Sensivity) + (Psp ∗ Specifity) + (Ptm ∗ time lead). (9) Performances <sup>∈</sup> [0*,* 1], P: weights, P<sup>i</sup> <sup>∈</sup> [0*,* 1]*,* -P<sup>i</sup> = 1, time lead normalized


**Table 7.** Experimental results

## **5 Experimental Result Analysis**

The Dataset 02 was used for training, and subsequent optimization techniques were implemented to select a subset of the data that would yield lightweight and tailored models optimized for embedded systems. To assess the effectiveness of our solution, we employed different methods depending on the experiment. For Exp. 0 and Exp. 1, we evaluated the performance based on the number of simulations (each activity labeled as ADL or Fall), enabling us to determine false positive and false negative rates. For other experiments (exp. 2 and exp. 3), we based our evaluations on the number of activity hours and participant-reported information. If the participant doesn't fall but a fall was predicted or detected, it is considered a false positive. If the participant falls, but we didn't detect anything, it is considered a false negative. However, if the person falls and we predict it, it was considered as a true positive.

Generally, falls in real life are infrequent, so our solution specificity is highlighted in exp. 2 and exp. 3. When we encounter false positives or false negatives, we analyze the data and incorporate it into subsequent training sessions. The benefit of these experiments is that we can evaluate and validate our solution under real-world conditions with participants who are natural and free, unlike other evaluation tests. Therefore, our solution is exposed to situations (activities) not seen during training, representing a challenge for us. The evaluation results that we currently have are presented in (Table 7). It should be noted that the evaluation of our solution is still ongoing, with particular focus on (Exp. 03).

## **6 Discussion**

Based on the initial results from experiments (exp. 00 and exp. 01), it appears that fall detection is facilitated by the fact that signals already contain the necessary information to accurately detect falls. This is supported by the high level of sensitivity and specificity observed. The fall prediction models showed a decrease in performance when the environment and subjects were changed (between exp. 00 and exp. 01), which is likely attributed to the prior compression of the models before deployment on embedded devices. This compression was intended to reduce the models space-time complexity and make them more lightweight. The two additional experiments exposed our solution to real-world conditions, which allowed participants greater freedom of movement, resulting in signals that differed, sometimes markedly, from what we had observed in our learning database. Moreover, as the belt-wearing was subtle, participants behaved more naturally, revealing movements we had not encountered during training.

## **7 Conclusion**

This study involved the development of a fall detection system and a fall prediction system, which were adapted and deployed on an airbag belt to test their effectiveness in real-life conditions. While the theoretical results were comparable to other works, the goal was to create a functional system that performed well in both theory and practice. This is in contrast to some studies that relied on theoretical results or private motion data to which the researchers did not have access. However, the lack of movement data from elderly people and real fall data posed significant challenges. Therefore, the plan for the future is to launch a collection campaign in EHPAD to gather natural data over a long period of time, including real falls that can be added to the learning base.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Missing Health Data Pattern Matching Technique for Continuous Remote Patient Monitoring**

Teena Arora, Venki Balasubramanian, and Andrew Stranieri(B)

Federation University Australia, Ballarat, Australia teenaarora@students.federation.edu.au, {v.balasubramanian, a.stranieri}@federation.edu.au

**Abstract.** Remote patient monitoring (RPM) has been gaining popularity recently. However, health data acquisition is a significant challenge associated with patient monitoring. In continuous RPM, health data acquisition may miss health data during transmission. Missing data compromises the quality and reliability of patient risk assessment. Several studies suggested techniques for analyzing missing data; however, many are unsuitable for RPM. These techniques neglect the variability of missing data and provide biased results with imputation. Therefore, a holistic approach must consider the correlation and variability of the various vitals and avoid biased imputation. This paper proposes a coherent computation pattern-matching technique to identify and predict missing data patterns. The performance of the proposed approach is evaluated using data collected from a field trial. Results show that the technique can effectively identify and predict missing patterns.

**Keywords:** Missing health data · Pattern-matching technique · Remote patient monitoring · IoMT · Smart healthcare

## **1 Introduction**

In recent years, wireless vital sign sensors, commonly known as the Internet of Medical Things (IoMT), have broadly extended the boundaries of remote patient monitoring (RPM) [1, 2]. These sensors remotely and continuously measure a patient's five vital signs, such as oxygen saturation (SpO2), blood pressure (BP), body temperature (Temp), respiratory rate (RR), and heart rate (HR), which allows for early identification of any abnormalities or deterioration in the patient's conditions [3]. Continuously monitored patient data and its extracted features can be used for decision-making, early clinical event prediction models [4], and automated risk analysis [5, 6].

The modified early warning score (MEWS) is widely used in hospital wards for nurses to assess a patient's condition and raise alerts [7]. First introduced as an early warning score (EWS) [8], it was replaced with the modified early warning score (MEWS) [9, 10]. Weighted scores from 0 to 3 for each five vital signs parameter datasets are assigned from thresholds on raw values; single vital parameter scores are summed to generate a MEWS score [10, 11], as shown in Table 1. Several variations to patient deterioration using MEWS and other approaches, including single parameter, multiple parameters, the aggregate score, and combination system methodologies, have been reviewed [11]; however, approaches commonly assume the minimal value of the parameter if there is no value for a particular parameter. Therefore, missing data refers to any parameter value with no recorded data in the five vital signs parameter dataset., Much research has focused on dealing with missing data [12, 13]; however, missing data in continuous vital signs in RPM is an unexplored area. The number of parameter data observed continuously in RPM makes the database highly susceptible to missing data occurrence, and its analysis is a significant challenge. Missing data is generally categorized into three categories: Missing completely at random (MCAR), Missing at random (MAR), Missing not at random (MNAR) [13].


**Table 1.** Modified Early Warning Score [9, 10]

The amount of data missing in any clinical setting depends upon various causes, such as delayed data transmission, environmental issues, connectivity issues due to a network, sensor malfunction, power loss, and sensor detachment from the skin. Consequently, missing or poor-quality data will impact the evaluation of vital sign abnormalities, clinical event detection algorithms, or the risk analysis of the patient [14].

Extensive work exists on analyzing missing data in the literature [13, 15]. A general regression neural network (GRNN) and successive geometric transformation model (SGTM) were applied in a study to handle the missing clinical data [16]. Missing data may frequently occur in RPM monitoring, and the amount of recorded data variations makes it challenging to analyze as these algorithms need significant data for training [17]. In addition, these techniques often ignore unique patterns in data and correlations between the observed vitals.

This study proposes a similar pattern match to continuously provide missing values imputation based on a temporal association of vitals in a sliding window approach. Patterned early modified warning score (PMEWS) is introduced in the research as a pre-processing feature for finding pattern matches for the missing vital signs values. This pattern-matching technique helps identify the trends and patterns in the data even when the variability of the missing data is significant. This approach uses the required vital signs data from a clinical trial from our RPM real-time implementation [18]. The main contribution of this study is summarized below:


This paper is organized as follows: Sect. 2 outlines the related work of this research. Section 3 describes our proposed solution. Section 4 describes our results, and Sect. 5 concludes the paper.

## **2 Related Work**

Imputation is the recovery-based approach to overcome the limitation of complete case analysis if the missing data falls under the MCAR category [14]. Imputation is used to replace the missing data values using single mean imputation, conditional mean imputation, and last observation carries forward (LOCF), multiple imputations (MI), and full information maximum likelihood (FIML) [19]. Single-mean imputation, where the missing data is imputed by the mean of the recorded value of that attribute, has a limitation that affects the dataset's variation. In comparison, conditional mean imputation intensifies the multivariate relationship in the data. Due to these limitations of mean and conditional imputation, these are recommended for baseline measurements in randomized trials [20]. LOCF is similar to single imputation, which replaces the missing values with the last observed value. MI has three phases: the imputation phase, the analysis phase, and the pooling phase [21]. MI generates n replacements for missing values. Then these completed datasets generated with n imputed values are analyzed and integrated into the final output. MI applies to both MAR and MCAR. FIML best suits the MAR context as it considers only the observed data and ignores non-response data [22]. These imputation methods are inappropriate if the missing data falls under the MNAR category.

Random Forest is inherently a MI approach, and it finds the mean of the data from numerous unpruned classification or regression trees for missForest [23, 24]. Bayesian ridge regression used the probability distributor when designing the linear regression, which enables the automated process for the missing data [25]. A cluster's variable mean, or mode is used to impute the missing values in the hot/cold deck [26]. Using the Euclidean distance, the K-nearest neighbour imputes the missing values with a similar one by defining the similarity between two values [26].

Normalization, formatting, and synchronization of the data are also necessary [27]. Normalizing the recorded data variables is essential as various scoring systems, such as the sequential organ failure assessment (SOFA) and MEWS, were registered at different frequencies [28]. Removing abnormal values from the data is also described in the literature as a part of the pre-processing [29]. One typical statistical analysis, the complete case, removes all the rows with the missing values and studies all accounts with the exclusive data [30]. This method has a disadvantage: it will reduce the amount of valuable data, producing biased results. Von Russum [12] analyzes techniques such as linear interpolation, spline interpolation, last observation, mean-forward, and cluster-based imputation for missing values. Imputation using these resulted in early warning score (EWS) miscalculations ranging from 1% to 8%. These methods produced more biased results than oversized observation windows [12]. A deep learning-based protocol for accurately predicting the missing data is also proposed in the literature [31]. Expectation Maximization (EM) [32] is a popular algorithm for imputing the missing data. One disadvantage of EM is that the recorded data should be highly correlative to get more reliable information [33].

Existing approaches perform well in imputation accuracy, though their efficiency is greatly affected by various factors such as computational time, complexity, and correlation of parameters. Our proposed pattern-matching imputation approach considers factors such as the correlation of the parameters, which helps to reduce the biased imputation and computational time can be set using the sliding window to show efficient performance in handling the missing values.

#### **3 Pattern-Matching for Missing Values**

The structure of our method is divided into two stages: Initially, the data is pre-processed using the patterns created using the MEWS [34] discussed in Sect. 3.1. Pre-processing the data helps ensure an accurate assessment of the patient's condition. Pre-processing includes removing erroneous values, no recorded vital values at a time, abnormal values, and high-frequency noise [35]. In the second stage, a pattern match is found for the missing values in an observation window. These two components work together to provide a high similarity match for the missing data.

Vital signs data for parameters systolic blood pressure, blood oxygen, body temperature, respiratory rate, and heart rate streams at a discrete time for each parameter using the wearable sensors recorded from patients in a general ward [18] were used in the current study. Publicly available datasets were not used to ensure data is simulated real data and were not curated in any other ways. The imputation method considers the data's temporal context and more accurately predicts the missing values using a sliding window. A sliding window has been used to find patterns for a particular data segment [21, 22]. A six-hour prediction window for clinical deterioration is considered ideal [36]. The sliding window is predefined in our algorithm.

A sliding window analyses a subset of data within the process window of fixed length, often with a minimum overlap of one data point and shifting with the window increment time in the process window. The notations used for the pattern-match algorithm are the number of vitals to be recorded (TMV), Sliding window time (t), String patterned array (SP[i]), Variables (Str1, Str2), Counter for the matched character (CSP[k]). Figure 1 shows the sliding window increment and overlaps in the process window. T10, T20, T30, T40, T50, and T60 denote the time in Fig. 1. The sliding window (*sw<sup>i</sup>* <sup>δ</sup>) consists of different time slots *t i* (δ+*n*) shown in Eq. (1).

**Fig. 1.** Sliding window with the overlap and window increment

The process window length (L), where (L - N) N is the set of all the natural numbers, and the slide window increment (δ), where (1 < = δ < L), can be decided by the physician, as shown in Fig. 1. If urgent attention is needed, a shorter window increment time can deliver more frequent information on the patient's condition. On the other hand, a longer window increment can be employed to lower the computation burden and boost the analysis accuracy if the patient's condition is stable and not urgent. A significant observation window with personalized decision-making and imputation produced positive results for missing values on maternal health data [14]. The window number (i) where (i - N). Notation (*t i* (δ+*n*) ) will be written as TS for calculations in the rest of the paper, as shown in Eq. (2).

$$sw^{l}\_{\delta} = (t^{l}\_{(\delta+0)}, t^{l}\_{(\delta+1)}, t^{l}\_{(\delta+2)}, \dots, \dots, \dots, \dots, \dots, t^{l}\_{(\delta+n)}) \tag{1}$$

$$\text{TS} = (\mathbf{t}\_{\text{(\\$+n)}}^{\text{i}}) \tag{2}$$

The physician will decide the start time (T) and the time slot (TS) for the observation. The slide window increment (δ) value will be valid for the time slot (TS) and can be changed for the next process window (L). "Patterned modified early warning score" (PMEWS) is described in the next section. The PMEWS forms the basis for similar pattern matches for missing vitals values. The method is described in the next section after PMEWS.

#### **3.1 Pattern Modified Early Warning Score (PMEWS)**

Our method for handling missing values in our data set is to make a pattern of the values from the recorded set, replacing the missing values with letters or symbols. As shown in Table 2, recorded raw, vital values at any data points are converted into their respective MEWS score to form the pattern, where 'N' is used for the null values. Another significant benefit of using the pattern with the letters is that it avoids using any previously registered threshold value that could affect the prediction of the patient's risk assessment by providing a biased estimation. In our method, patterns can be recorded in any format.

An ordered sequence of MEWS values that make a pattern is recorded from the patient's observed data and is labelled as the PMEWS. The pattern array best suits sensor signal discrepancies because of null values and discrete-time readings from different sensors. This step is the second most vital step in raw data transformation.


**Table 2.** Pattern Formation

For sliding windows (*sw<sup>i</sup>* <sup>δ</sup>) consisting of time slots (TS), threshold data values of patients' vitals are recorded after processing them into their relevant MEWS score (M), and the resulting pattern is stored in an integer array *P*[*TS*]. Equation (3) shows an example of the pattern at the time slot (TS). MHR, MBP, MT, MSPO2, and MRR are the MEWS scores derived from the raw, vital data for heart rate, blood pressure, temperature, oxygen saturation and respiratory rate, respectively.

$$P[TS] = \{\text{MHR, MBP, MT, MSPO2, MRR}\} \tag{3}$$

The following section describes the pattern-match algorithm to identify similar patterns for missing values of vital signs.

#### **3.2 Pattern-Matching Algorithm**

Our main aim for this algorithm is to match the pattern with the missing value with the closest matching pattern that appeared in the sliding window. When the match is found, imputing the missing value parameter with the found match values is performed. For example, in Table 3, the raw data is recorded for the heart rate (HR), temperature (T), blood pressure (BP), oxygen saturation (SpO2), and respiratory rate (RR) and relevant patterns are created for the time one minute to four minutes. HR is missing at minute four with the pattern 'NNN20' and will get pattern-match at minute one '0NN20' as the highest number of symbols matched and imputed with the HR value of 99.


**Table 3.** Pattern Match

After pre-processing, the first step is to initialize the variables T, L, t, TMV, δ, *sw<sup>i</sup>* δ*,* and TS. The pattern *P*[*TS*] is generated from MEWS for the recorded raw values. Loop to traverse the string pattern SP[], from the start of a sliding window, i = T to TS is set. If there is a pattern SP[i + 1] with no null values, the counter i is incremented for the following index. Otherwise, another inner loop is set for the string pattern array with the index j = T to i to ensure all patterns are considered to match null data values. To ensure a match, two variables, str1 and str2, are set with SP[j] and SP[i + 1], respectively. A call to the function count passes these two string variables to check if str1 contains str2. For a matching pattern character case, the count is maintained for the pattern and stored in counter array CSP[k]. The counter array CSP[k] from j = T to TS is checked for the condition if the number of character matches is highest for the number of vitals observed, as shown in Table 4. Relative patterns and count values are displayed as suggestions for outcomes. Timeslot TS is updated with the sliding window increment time δ for traversing the sliding window loop. This algorithm is an iterative process in the sliding window to ensure all patterns are considered.

The proposed algorithm that accomplishes our pattern matching is shown as follows:

## **4 Result and Discussion**

To assess the plausibility of our technique, we used the dataset collected from the clinical trial [18]. The trial dataset has the raw values for the vital parameters Temp, BP, SpO2, RR, HR, and the time for each record, as shown in Table 3. For this study, only two parameters, SpO2 and HR, were imputed. The dataset was split into additional sets of various record lengths. MI and EM are commonly used to impute missing values; our proposed method is compared against these. Normalization is an essential step in preprocessing our data, as it helps scale the data and make it more suitable for analysis by reducing the biased outcome. Normalization is performed using the MEWS. Starting with the entire collection of values, we removed various percentages of the SpO2 and HR to test the precision of our technique. The values of the missing parameters in the dataset were then imputed using our pattern-match imputation, EM and MI in the same sliding window. The highest match for parameters is selected for the imputation by our algorithm. For comparison, we calculated the root mean squared error (RMSE) for the imputed value with the starting values from the whole dataset. Table 4 presents results obtained from four different datasets using different imputation methods.

For dataset D1:


For dataset D2:


For dataset D3:


For dataset D4:


Figures 2 and 3 show the variation of RMSE of two of our used parameters, SpO2 and heart rate (HR). In the context of missing data, the sliding window approach and the PMEWS have proven very useful as the algorithm demonstrates comparable results. The algorithm considers every pattern in the sliding window before the missing pattern and uses these patterns to find a similar match. Then it slides the window across the process window to cover all the patterns that appeared in that timeframe.


**Table 4.** Results of the Datasets.

The proposed algorithm's accuracy was satisfactory, which suggested that this approach is practical for finding a similar effective match for the missing vital parameter values in medical datasets generated in RPM settings. It was observed that most of the similar patterns in data appeared after the normalization phase of pre-processing of the data. The algorithm can leverage these patterns to find identical matches for the missing vital values, which can improve the accuracy of the prediction. These findings have implications for the researchers, physicians, and practitioners who work with the vital sign data set with the missing values in RPM for patient risk assessment analysis.

**Fig. 2.** Root Mean Square Error for SpO2

**Fig. 3.** Root Mean Square Error for HR

## **5 Conclusion**

This study presents a method for pattern-matching the missing values in the RPM vital sign dataset using the PMEWS and the sliding window. This method offers the identification of similar patterns in the data, and using the PMEWS, this technique is further improved by leveraging these patterns. Compared to other approaches, this technique considers the temporal context of the missing values, which can be particularly important in the medical domain where the timings of the measurements can have critical implications for patient care. This method also has the potential to revolutionize how continuous RPM data is processed and analyzed, leading to more efficient healthcare delivery. Future research could explore other pre-processing techniques to improve the pattern match further. Overall, the proposed method can improve the accuracy of the medical data analysis for decision-making, specifically for the missing vitals values.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **How to Assess the Effectiveness of Arm Support: A Systematic Review**

Yun-hwan Lee1(B) and Jongbae Kim2(B)

<sup>1</sup> Department of Occupational Therapy, The Graduate School, Yonsei University, Seoul, South Korea

enn1210@gmail.com

<sup>2</sup> Department of Occupational Therapy, College of S/W Digital Healthcare Convergence,

Yonsei University, Seoul, South Korea

jongbae@yonsei.ac.kr

**Abstract.** Arm support is a typical assistive technology device to assist shoulder and elbow movements in those with reduced upper extremity muscle strength due to neurological lesions. Recently, the assisting method, range, and manipulation method of arm support are also changing with the development of technology. Accordingly, an assessment system is required for appropriate matching and measuring the effectiveness in the clinical field. This study examines the direction of the assessment process by analyzing studies that measured the effectiveness of arm support through a systematic literature review. The databases were collected using Embase, CINAHL Plus with Full text,Web of Science, and Scopus. 19 studies were selected according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) flow chart. Most studies have been conducted on people with neuromuscular disorders. Assessment area included performance, function, usability, satisfaction, and psychosocial impact. And more than half of the studies measured performance and function. There were various assessment methods to measure the effectiveness including assessment tools, kinematics, physical examination, questionnaire, observation, and EMG, with assessment tools accounting for more than half of the studies. Most studies have set up assessment environments based on tasks related to ADLs and IADLs to measure the effectiveness. Currently, various methods such as assessment tools and kinematics were applied to measure the effectiveness of arm support. The assessment tool was used the most among them. However, assistive technology-based assessment tools are extremely limited. Therefore, it is required to develop an assessment tool centered on assistive technology based on performance related to ADL and IADL in the future.

**Keywords:** Arm support · Assessment · Systematic review

## **1 Introduction**

Arm support is an assistive technology device (ATDs) that has been used since the 1960s to supplement the performance of activities of daily living (ADLs) of those who (e.g., Spinal cord injury, Muscular dystrophy, Multiple sclerosis) limited upper extremity muscle strength [1]. It provides stability to the proximal part of the shoulder by removing gravity acting on the upper limb through an armrest and assists shoulder and elbow movement through a power source [2]. In recent, the power source of arm support is developing with the development of technology. Because of this, people with disabilities who use arm support can receive support for a wider range of movements. According to a systematic review by Lee et al. (2021), the proportion of arm supports was the highest among the upper extremity ATDs to assist people with disabilities in their ADLs [3]. Also, they found that the power source for assisting the user's movement is changing from a spring to an actuator, electricity to assist the user's active movement according to the development of technology. Changes in power sources will affect how users manipulate arm support and the extent to which it can assist a user's functional movements [4]. Therefore, in the clinical field, it is required to set up an assessment system that can prescribe appropriate arm support according to the function of the people with disabilities and measure the effectiveness after the intervention. Standardized assessment about ATDs relies on the subjective judgment of the user, such as satisfaction and psychosocial impact [5, 6]. In addition, objective assessment systems and methods for performance are limited. This is a critical barrier to assessing the multifaceted effectiveness of ATDs. Therefore, the purpose of this study is to collect and systematically analyze studies that have been assessed about arm support, and to suggest directions for establishing an assessment system.

## **2 Method**

#### **2.1 Database and Search Term**

This study searched for studies that applied arm support as an intervention in 4 literature search databases (Embase, CINAHL Plus with Full text, Web of science, and Scopus). Search terms were "Arm support" AND (Assessment OR Evaluation OR Measurement) and search period was set to the past 10 years.

#### **2.2 Selection Strategy**

The studies were selected according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement [7].

Studies were included if they met the following criteria: (1) studies on people with disabilities, (2) studies that assessed after providing arm support as an intervention, (3) can see the full text, (4) studies written in English or Korean. And, studies were excluded if they met the following criteria: (1) studies that not provided arm support as an intervention, (2) studies about reliability and validity of assessment tool, (3) studies on people without disabilities, (4) Systematic review & meta-analysis, (5) Grey literature, (6) Studies that could not open the full text.

Literature selection was carried out by two researchers. In case of disagreement between the researchers, it was discussed and agreed upon. Consequently, 19 studies were selected for this study (Fig. 1).

**Fig. 1.** PRISMA flow chart

#### **2.3 Analysis**

The selected articles were analyzed frequency for the type of disability, assessment area, assessment method, frequency of assessment tools by area, and measurement method of assessment tools.

## **3 Result**

## **3.1 Type of Disability**

In terms of the type of disability, most of the studies have been conducted on people who have decreased performance in daily activities mainly due to decreased muscle strength in the shoulders and elbows. Sixteen studies were conducted on people with muscular dystrophy which is a loss of strength from the proximal to the distal part of the shoulder. It is occupied the largest proportion. Eight studies were conducted on people with spinal muscular atrophy. And three studies were conducted on people with spinal cord injury, stroke, and arthrogryposis, respectively (Table 1).


**Table 1.** Type of disabilities on subjects

#### **3.2 Assessment Area**

In the included studies, Upper extremity performance and function were primarily assessed about arm support interventions. This is consistent with the results of previous research [3]. It can be confirmed that it is an important factor to measure the function and ADL performance in the assessment of arm support intervention. In addition, it was found that assessment was performed through the subjective judgment on usability, satisfaction, psychosocial influence, fatigue, and workload according to the use of the arm support (Table 2).


**Table 2.** Assessment area included studies

#### **3.3 Assessment Method**

The included studies used either quantitative or qualitative method to assess the effect of arm support on the area. Most studies used standardized assessment tool to measure effectiveness of arm support. By using a standardized assessment tool, it is possible to secure the validity and reliability of the results. Kinematics through 3D motion analysis was used to measure performance and function based on tasks (e.g., Workspace, Range of motion). Physical examination which is the traditional method of measuring function mainly assessed range of motion (ROM) with a goniometer and manual muscle strength (MMT). Through qualitative methods such as questionnaires and observations, factors that are difficult to measure quantitatively were assessed (Table 3).


**Table 3.** Assessment Method included studies

#### **3.4 Assessment Tool by Area**

Looking at the assessment tools by area, tools for measuring performance and function occupied about 80% of the total. Since there is no assessment tool centered on assistive technology, the occupational therapy assessment tool was used or modified for assess on performance and function. There were two assistive technology-focused assessment tools: QUEST which measures satisfaction with the device and service process, and PIADS which measures the psychosocial impact of using assistive technology (Table 4).


**Table 4.** Assessment tool by area included studies

#### **3.5 Measurement Method of Assessment Tool**

Most assessments measure effectiveness through the performance of tasks related to ADL, which are mostly utilized to measure performance and function. A questionnairebased assessment tool is used to investigate users' subjective feelings about the effectiveness of arm support. There were also studies that used semi-structured interview and observation to measure the effectiveness of the arm support (Table 5).

**Table 5.** Measurement method of assessment tool included studies


## **4 Conclusion**

Arm support is an assistive device that compensates for shoulder and elbow movements to help those with reduced upper extremity muscle strength perform activities ADLs. Due to the development of technology, the assisting method and range, and operation method of arm support are also changing. Accordingly, a system for properly matching and assessing effects in consideration of the user's function, environment, and activity is required in the clinical field. In this study, we examined assessment trends and suggest an appropriate direction for assessment by systematic review on studies that measured the effectiveness of arm support. First, the assessment area for arm support is mainly focused on performance and function. The use of assistive devices interacts not only with the user but also with the activity and environment. Therefore, a comprehensive assessment considering not only users but also activities and environments should be conducted. Second, assessment tools are the most popular way to measure the effectiveness of arm support. However, when looking at the trend of tools by assessment area, the assessment tools centered on assistive technology are extremely limited. The majority of studies use modifications of existing occupational therapy assessments to measure performance and function. It is necessary to develop an assistive technology-centered performance assessment tool or method. Third, the effectiveness of arm support is measured through task-based performance. Assigned tasks are mainly composed of ADLs and instrumental activities of daily living (IADLs). Therefore, it is required to establish an environment for measuring effects based on ADLs and IADLs related to using arm support. We expect this to be useful in the process of measuring the effect after providing arm support as an intervention for those who have decreased performance in daily activities mainly due to decreased muscle strength in the shoulders and elbows.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Wellbeing Technologies**

# **Ambient Assisted Living Service Conception in Nursing Homes: From Reinforced Aging in Place Services Towards Smart Digital Coordination Platform**

Juliette Reerink-Boulanger1,2(B) , Rémi Locquet1, Caroline Ric1, and Alain Somat2

<sup>1</sup> Hospitalité Saint Thomas de Villeneuve, Direction Innovation Recherche et Formation, 29 rue Charles Cartel, 22400 Lamballe, France {Juliette.Boulanger,Remi.Locquet,Caroline.Ric}@HSTV.fr <sup>2</sup> LP3C - EA1285 - Laboratoire de Psychologie : Cognition, Comportement, Communication, Université Rennes 2, Rennes, France {Juliette.Reerink-Boulanger,Alain.Somat}@univ-rennes2.fr

**Abstract.** This article presents the ethical and responsive governance process designed by the Hospitalité Saint Thomas de Villeneuve to address aging in place services, care and needed support. We will first describe the genesis of a supportive community living services organization and its technical translation into Smart Digital Health Coordination Platform. Secondly, we will highlight the Living Lab methodological approach to support change. Finally, the results section will describe change evaluation program engaged and future work perspectives.

**Keywords:** Aging in place · nursing homes · living lab · change

## **1 Introduction**

By 2030, the increase of healthcare expenditure related to the baby-boomer generation will worsen as the European population grows older. It is therefore urgent to prevent frailty risks and deliver health and social supportive services to address elderly complex care situations at home.

On the one hand, the proportion of people over 65 years increases rapidly, and European society is challenged to prevent decline of functional capacity by addressing user's needs at home with appropriate and optimal health service packages. On the other hand, loneliness, geographical isolation and vulnerability, three social determinants, are also known to impact mortality, morbidity and health care system use [1, 2]. Health and social determinants thus need to be addressed together for better aging in place situations.

Innovative social and medical coordinated care solutions are therefore required to prevent complex situations that lead to institutionalization and nursing homes. This is the primary goal of the "Reinforced Aging In Place Services" project, EAIPS, namely "*Dispositifs Renforcés d'Accompagnement à Domicile* (DRAD) in France, a French experimentation financed project since 2018 [3]. Due to its organizational major ambition, EAIPS-DRAD project is already generalized and integrated into the French code of social action and families. It is currently named "Territorial Resources Centre for Nursing Homes" (*Centre de Ressources Territoriaux Gérontologiques*) in the French administrative system. Patients benefit, efficiency and medico-economic evaluation are three major criteria currently being evaluated in the EAIPS-DRAD project by the DGCS (Social Cohesion and Solidarity Public Policies General Direction).

The aim of this paper focuses on how living lab methodological approaches support care activities change in nursing homes. Consequently, this study will try to analyze the various acceptance determinants of a smart digital coordination platform designed to reinforce aging in place services.

#### **1.1 From Nursing Homes to Reinforced Aging in Place Services – Dispositifs Renforcés d'ACCompagnement à Domicile**

Aging in place is a major public health issue with medical, economic and social impacts. In France, social and medico-social housing facilities are facing an unprecedented crisis that legitimates a careful consideration of this model. Indeed, industrial processes applied to human care, degradation of working conditions [4], numerous adjustments that have been made to cope with covid pandemic situations, their effects and organizational impacts are so many crisis indicators and urge for a deep and sustainable model change [5].

A variety of aging in place solutions currently exist, some are called intermediate forms of residential provision [6], with varied types of residential, community and supportive services [7, 8]. This term defines a new type of housing which is closer to the home than to the social and medical/social sector. EAIPS-DRAD system can be considered as a bridge between home and institution, especially regarding of two organizational characteristics: its organization "outside the walls" [9] and its provision of assisted technological services [10].

The choice made by HSTV is to provide an "outside the walls solution", a form of nomadic services brought from inside the nursing home to outside elderly accommodations [11]. It represents a multidimensional and multi-professional expertise based on different actors (gerontologist, nurses, caregivers of the nursing home) who take into account the patient's desire to remain at home. Additionally, Reinforced Aging In Place Services/DRAD are completed by sensors and applications that are likely to slow down loneliness and/or possible depression [12]. The aim of this system is to anticipate the changing needs of elderly people aging in place, within a structured and coordinated local community living service (medico-social and health services).

#### **1.2 Reinforced Aging in Place Services (EAIPS-DRAD)**

As we get older, health and care needs are not likely to stay progressive and continuous. Therefore, RAIPS/DRAD experiments an organizational system whose mission is to offer an alternative to admission in nursing homes for the elderly with who might have functional or cognitive incapacities but would rather stay home. As a matter of a fact, one of the HSTV non-profit organization's nursing homes observed in 2016 that it was no longer possible to fulfill the growing needs of the nearby population: more than 300 elderly people were on the waiting list. A survey conducted the same year among its residents showed that none of them would have chosen the nursing home if they could have stayed at home. The idea of a "nursing home outside the walls" service was developed and co-designed with staff and families. It has been experimented since 2018 within 5 institutional home care of HSTV group, and based on three principles:


These efforts to promote new types of living spaces for older people are not new but to date, there was no coordinated and evaluated deployment in France. Today, nearly 150 people benefit from RAIPS/DRAD services within 5 nursing facilities.

From now on, because this experimental phase has been positively evaluated, this plan can officially provide aging in place preventive support inside the walls of the nursing home. These are the new missions called "Territorial Resources Centre for Nursing Homes" (*Centre de Ressources Territoriaux Gérontologiques*). April 27, 2022 bylaw1 gives precise specifications for these missions.

#### **1.3 Smart Digital Health and Coordination Platform - PALOMA**

Thanks to the non-profit organization executive board and local council of HSTV, Brittany IT Services and IT Consulting (*GCS E-santé*) and the Regional Heath Authority (*Agence Régionale de Santé de Bretagne*), the PALOMA smart coordination platform was born. Its goal is to enable people with a loss of independence to stay at home with the help of the nearest Territorial Resources Center (CRT). Placed at the center of the system, the nursing home, is the resource center, in the sense of CRT.

The challenge of developing such a platform is associated with the tremendous changes that health and medico-social nursing homes sector are facing. Their missions are progressively evolving from residential settings to Territorial Resources Centers (CRT) for the elderly populations living nearby. Their aim is to provide responses to the changing needs of frail seniors, primarily by acting preventively and curatively to allow them to stay at home by providing support. Two major issues must thus be addressed in the smart digital platform conception:


<sup>1</sup> Arrêté du 27 avril 2022 relatif à la mission de centre de ressources territorial pour les personnes âgées - Légifrance (legifrance.gouv.fr).

The scope of the PALOMA platform project therefore includes the support and the deployment of CRT by offering a global, interoperable and predictive digital tool (see Fig. 1).

The digital solution has been designed to help elderly people maintain their independence and stay at home by improving the coordination of social, medico-social and health sector players in order to:


**Fig. 1.** Smart digital health and coordination platform –PALOMA

Thus, this platform aims at acting on 4 dimensions:


PALOMA smart digital health coordination platform aims at providing continuous support from primary signs of frailty in everyday life, complex care management while aging in place (RAIPS/DRAD) and to institutionalized care in nursing homes.

## **2 Literature Review**

PALOMA's platform development is a strategic priority for the Hospitalité Saint-Thomas de Villeneuve non-profit organization. The HSTV's living lab is in charge of coordinating the development stages, allocating the necessary resources, undergoing the qualityassessment process as well as monitoring the pilot sites deployment. In order to ensure strategic alignment between technical development and supportive community services, HSTV's ethics committee has been consulted throughout the project. The innovative and methodological approach is developed hereafter.

#### **2.1 The Living Lab Approach Implemented at the Hospitalité Saint Thomas de Villeneuve**

In 2021, a five-year strategic project on user experience, attractiveness and innovation has been launched with HSTV local council approval. To implement, develop, and structure an innovation culture, a living lab project has been launched the same year. It aims at ensuring open innovation policy, attached to the value of the HSTV non-profit organization.

Several definitions of Living-labs co-exist [13]. Forum LLSA2 in France, and ENoLL3 in Europe both describe living labs as open innovation ecosystems. In HSTV's living lab, the methodological approach follows the core principle of: "open innovation" carried by the user, "co-creation" and experimentation of the uses of products and services, "multiple stakeholders" and the importance of "real-life environment" with the aim of defining and developing new products, services, public and community systems or new business models [14]. Livings labs thus constitute a new research paradigm based on a participatory dimension through collaboration with end users and a mode of innovation described as open.

Considering living labs as potential sources of innovation stimulation [15], a strategic priority project of Living lab has been structured within HSTV in 2021. Firstly, support from the Forum LLSA has been requested in order to design the value creation and the raison of being HSTV's living lab, as well as elaborate its governance, development process and economic model [16]. Secondly, in 2022, a complementary approach has been implemented with the training program provided by the Ensembll Living Lab in Lille Metropolitan area. In a dynamic and learning process, co-designed methodology, animation postures, evaluation, ethical and governance process of the living lab project has been structured. Thirdly, a working and learning expedition during autumn 2022 with the directory and scientific board of the University of Lille has contributed to rely on a definition of living lab activities as " a process of special attention paid to enabling environment in inclusive space" [17].

#### **2.2 The Purpose of HSTV'S Living Lab: Supporting Change Activities in Health and Medico-Social Institutions**

The structuration of the HSTV Living Lab activities previously described led to define its raison of being, which is to meet the needs of change management for HSTV's care and medico-social facilities for change support. A value proposition creation has been established to take up to the challenges faced by HSTV's health and medico-social institutions:

<sup>2</sup> https://www.forumllsa.org.

<sup>3</sup> https://www.enoll.org/about-us/


Change in the health and medico-social sector are numerous, and the ambition of the living lab is to go beyond a time-limited project management mindset to contribute to a "performance of use" [18]. This concept relies on centrality of work which leads to the opportunity of value proposition and contributes to achieve performance.

## **2.3 How Technological Artifact PALOMA Processes Organizational Change?**

Firstly, to move from a project management logic to the emergence of performance of use concept, we propose to challenge our activity model change with simple questions [19]:


## **2.4 Ethical and Responsive Evaluation Program in the Design Process of Smart Health Digital Services for Users**

Secondly, to support change requires appraising the effects of a complex situation. Based on Stake's "Responsive Evaluation" model [20], the evaluation approach associates the experiences of patients and beneficiary and family members for shared decision making, and collective participation. Additionally, the evaluation approach focuses on the educational, learning and empowerment processes rather than the solely measures of effects and outcomes of the change process.

In the HSTV's living lab, evaluation process was reduced to four steps, including:


#### **2.5 Multilevel Acceptance Framework Assessment for Smart Health Digital Services for Users**

Thirdly, to support Ambient Assisted Living (AAL) product and service conception, user needs requirements and activity limitations assessment are two crucial components to target independent living. AAL system acceptance requires a micro level of system acceptability combined with a macro acceptability level for innovative practices acceptance [21].

To support change within a complex activity system, an "expansive learning cycle" involving several phases is initiated (cf. Fig. 2) and aims, through co-construction, at elaborating the bases of new practice decision and collective participation.

**Fig. 2.** Sequence of learning steps in a development cycle by expansive learning [22]

## **3 Living Lab Approach: A Responsive and Ethical Evaluation Program to Co-Create the Paloma Platform**

We built up a responsive ethical program. We proposed to analyze the co-design work of the platform through the prism of these different ethical evaluation stages. Indeed, the Living Lab participants (users and medical and health professionals) questioned the meaning of work activities and service relationship during the design of the smart digital platform. In this paper, our aim is to describe change evaluation program engaged in order to better apprehend organization change. This reflexive approach, led to the identification of a four-stage research-action framework which we propose to describe here:


## **3.1 Step 1: Ethics Committee Group Referral**

The development of the PALOMA platform raises a certain amount of ethical questions. In 2022, two members of the Living lab, a gerontologist and a Nursing Home Director of HSTV group, wrote down a document to raise these issues. A synthesis of the referral document identified the risks of opposition between HSTV care values and the digital platform (Table 1 below).

**Table 1.** HSTV care values and risks of opposition summary raised by the digital health and coordination platform –PALOMA


These 6 questions addressed to the HSTV Ethic Committee group aim at providing ethical guidance for smart platform specification development. They particularly points out strategic dimensions such as:


## **3.2 Step 2: Controversies Arising from the Use of the Platform**

In order to materialize these ethical questions associated with the use of the PALOMA platform, a collective workshop was organized. Several Living Lab community users contributed to co-design the digital platform controversies: HSTV' representative (health professionals, doctors, UX designer, innovation manager, nursing home director, information system director), Aract Bretagne Association (work condition non-profit organization), France Asso Santé Association (patient non-profit organization), and Telegrafik (smart digital platform software company associated in the Paloma development). Workshop objectives were to collectively debate on digital and Artificial Intelligence health service design principles conception for healthy aging. The controversies were designed based on the ethical principles of medical, care and digital Artificial Intelligence (AI) project conception design4:

The workshop was held during the Britany regional conference on Ethics and Artificial Intelligence in Health organized by the Ministerial Delegation for eHealth (DNS) and Regional Health Authority (*Agence Régionale de Santé de Bretagne*) on December 8, 2022. An invitation workshop based on "Digital data use in health to support healthy aging" was launched.

50 people took part in the workshop (12 men and 38 women): 26 caregiver students, 24 health professionals or user representatives. All of them had a previous hour conference on AI concepts in Health provided by Ministerial Delegation for eHealth.

The controversies used during the workshop were:


#### **3.3 Step 3: Ethical Shared Principles Synthesis During Collective Participation on the Debate of Smart Digital Platform Conception**

During the workshop, a facilitator welcomed the participants and invited them over a period of one hour, to discuss the controversial issues. As an introduction, a short brief on the technical aspects of the platform (predictive algorithms of frailty risks signals) was provided. The objective of a collective debate on ethical issues, work impact, quality of care with IA use was explained. According to the concept of "moving debate", participants had to occupy the workshop space according to their opinion: the "agree" on one side of the room, the "disagree" on the other. They were then invited to express out loudly, as many arguments as possible and to switch camps if they agreed with the argument expressed.

Each argument, from one side or the other, was stated successively, trying to rally the opposite side. The arguments exchanged were successively noted on post-it notes by a co-facilitator on a paper-board: the arguments that led to agreement were noted in blue whereas the arguments that led to disagreement in green.

During the workshop, the ethical design principles, as defined by the DNS in the "ethic by design guide", were explained. A categorization was then realized: the post-it notes were grouped in respect of the ethical principles guidelines. Eight ethical principles for the digital platform conception were identified:


<sup>4</sup> Ethic\_By\_Design\_Guide\_VF (esante.gouv.fr).


### **3.4 Step 4: Use Case Design to Engage Discussion with the Ethical Committee**

In order to get a concrete representation of the potential risks of the smart digital platform, 4 use cases have been designed. They consist of 4 different potential types of data use by 4 types of elderly and professional personas (i.e. fictional character). Each of them has been designed with the help of healthcare professionals (1 gerontologist, 2 nurse coordinators in the RAIPS Project, 1 nurse coordinator in a nursing home). To perform the data use case design, several criteria have been described:


These smart digital health use-cases were presented at HSTV Ethic Committee group on May 24, 2023. They intended to provide ethical discussion, guidelines and specifications of Paloma's platform data use with a user-centric perspective. They also address future possible work conditions and quality of care impacts. Finally, these platform data use cases highlight the responsibility and governance processes that HSTV non-profit organization needs to develop.

## **4 Conclusion**

A smart digital platform conception is not limited to technical infrastructure construction. As HSTV chose to develop its own organizational settings for aging in place, the nonprofit organization is being challenged with the implementation of a living lab approach to address ethical issues that digital community service conception raises.

This papers tried to examine the impact of a technological artefact on the professional activity changes associated with the conception of a smart digital and coordination platform named PALOMA. Involving professionals stakeholders to decision making process and digital co-conception is still a hard work to do because it implies to put in the spotlight new work identities, different nature of professional relations, ways of managing information, new values and norms elaboration, responsibility and missions.

Further work will integrate methodological material (description of use-cases and personas) and results data (ethics committee specifications; work activity scenario simulations, governance mechanism identification). Measurement of sustainable work conditions, quality of life, professional and environmental empowerment will be the our next step to evaluate if those determinants can be achieved within the smart digital health and coordination platform.

**Acknowledgments.** The research was supported by the Brittany IT Services and IT Consulting (*GCS E-santé*) and the Regional Health Authority (Agence Régionale de Santé de Bretagne). The authors would like to thank Lise Delcourt, ARACT work organization engineer who helped designing the workshop. The authors would also like to thank the various HSTV's living lab community members, Gérard le Goff, Lionel Jan, Martin Favre, Jean-Bernard Gauvin, Carole Zisa-Garat for helpful discussion on the smart digital health coordination platform conception evaluation.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Current State of Access to Assistive Technology and Its Related Services in Dominican Republic: A Pilot Study**

Rosanny Araysy Muñoz Collado1(B) and JongBae Kim<sup>2</sup>

<sup>1</sup> Department of Occupational Therapy, The Graduate School, Yonsei University, Seoul, South Korea rosannymunozc@yonsei.ac.kr <sup>2</sup> Department of Occupational Therapy, College of S/W Digital Healthcare Convergence, Yonsei University, Wonju, South Korea jongbae@yonsei.ac.kr

**Abstract.** The Assistive Technology supply is highly fragmented and almost nonexistent in many middle and low-resource settings with limited purchasing and no service delivery systems. In the Dominican Republic, despite the existence of a legal framework on disability, there is no comprehensive source of data about the availability of Assistive technology devices. Therefore, the purpose of this study was to investigate about the access, use, barriers to access and users' satisfaction of Assistive Technology (AT) in the province of Espaillat, in Dominican Republic. The WHO's rapid Assistive Technology Assessment tool (rATA) survey was implemented as a stand-alone technique to 15 outpatient of rehabilitation services. Despite the small sample size, the information gathered provides valuable insight into the use of assistive products in the Dominican Republic. The findings of the study highlighted cost-related factors influencing AT access and low level of satisfaction.

**Keyword:** Assistive technology · Dominican Republic

## **1 Introduction**

In actuality, 1 billion people need assistive technology (AT) to lead productive, inclusive and dignified lives, but only 1 in 10 people globally have access to the AT they need. Access to AT is essential for many people to maintain and improve function, health and well-being, and to participate in education, work and social activities. Among the people who commonly need AT are older people, people with disabilities and people living with chronic conditions. As the world population ages and the prevalence of noncommunicable diseases increases, the need for AT will continue to rise [1, 2].

Few statistics have been systematically gathered in countries to show the demand for AT among national populations and the need that is currently unmet, despite the urgency and the worldwide imperative to enhance access to AT. [3] The AT supply is highly fragmented and almost non-existent in many middle and low-resource settings with limited purchasing power and no service delivery systems. Historically, most of the governments in low/middle-income countries have not prioritized this sector [4].

In low and middle income countries, as most of the Latin American countries, assistive technologies are not universally accessible for a number of reasons, including high costs, limited availability, a lack of governance, inadequate financing, widespread lack of knowledge, and a shortage of staff with the requisite training [5].

In the Dominican Republic, 12.3% of the overall population lives with a disability, according to the 2010 Census on Population and Residence, representing 1 of every 8 people and constituting a total of 1,160,847. Physical and motor disabilities, visual impairments, attention deficit disorder, and auditory impairment are the most prevalent disabilities in the nation [6].

Despite the existence of a legal framework on disability, as in most of the developing countries, in the Dominican Republic there is no comprehensive source of data about the availability of Assistive technology devices. This can be explained with the fact that some of the main providers or leading suppliers of AT products and services are non-state actors, and databases are frequently maintained on an organizational basis with rarely external sharing (as is the case with NGOs) [6–8].

Therefore, for AT purchasers, such as governments, UN agencies, and civil society organizations, to decide whether to invest in the infrastructure of the AT supply chain, as well as for AT designers and distributors to identify the populations in need of ATs, data on Dominican AT supply and demand is required.

The purpose of this study is to investigate about the access and use of Assistive products(AP) in the province of Espaillat, in Dominican Republic, as well as the barriers to access and users' satisfaction with products provided. Furthermore, the study aimed to gain more insight into the methods people use to meet their needs for AT and produce evidence on the strengths and weaknesses of the current system by choosing a target population of community-dwelling individuals attending rehabilitation services provided by the local public healthcare system.

## **2 Method**

With a target demographic of 15 outpatient of a public rehabilitation center, the WHO's rapid Assistive Technology Assessment tool (rATA) survey [9] was implemented as a stand-alone technique.

The selected rehabilitation center is a public rehabilitation center of Espaillat province, and it is in charge of providing rehabilitation services for people with all kinds of disabilities, including acquired brain damage, spinal cord injury, neurological, orthopedic, rheumatologic, and post-intensive care unit rehabilitation, among many other diseases. According to this viewpoint, its rehabilitation services are responsible for managing all functional domains addressed by the rATA (mobility, vision, hearing, cognition, communication, and self-care).

The study received ethical approval from the Yonsei University Institutional Review Board. All surveys were conducted through face to face interviews in January-February 2023.

#### **3 Result**

In total, 15 individual responses (73,3% females; 26,7% males) were collected in the period January-February 2023. Overall, most respondents lived in rural areas (73.3%) and were adults aged 18–64 years (66,7%), followed by adults 65 + years (33.3%). (Table 1) A closer examination of the data revealed that the distribution of AT users across the Espaillat province geographic area was well balanced, with no discernible differences across districts.


**Table 1.** Demographic data

The higher use of AP was reported for individuals with a functional limitation in the mobility domain. The most commonly reported AT product in use was the cane (7), followed up by manual wheelchair (3), rollator (2), orthoses for lower limb (2), glasses of low vision (2) and axillary elbow crutches (1) (Table 2).

In terms of the sources to access the AP, they were most commonly obtained from a family member or friends (46.7%) or the private sector (26.6%) followed by the nongovernment organization (NGO) sector sources (13.3%), while only 6,7% were obtained through the public sector sources or were self-made. The majority of the users (33.3%) paid out-of-pocket for their AP(s) or relied on family/friends (46,7%) and only 13.3 and 6.7% were paid by government funding or a NGO respectively. Most AT users traveled less than 5 km (40%) or 6–25 km (33.3%) to obtain their AP(s) (Table 3).


When asking the users about their satisfaction, the majority (53%) of the users reported being dissatisfied with their AP. And when it comes to the unmet need, the most commonly reported reason was 'cannot afford' (40%), followed by AP unavailable (33.3%), lack of time (13.3%), lack of support (6.7%), and AP not suitable (6.7%). (Table 3).


**Table 3.** Assistive Product Information

(*continued*)


#### **Table 3.** (*continued*)

Regarding unmet needs, some users mentioned simply needing a replacement for the same product they were using, but other products such as powered wheelchairs, therapeutic footwear, fall detectors, hearing aids, smart phones/tablets/PDA, GPS locators, chairs for shower/bath/toilet and grab-bars/Handrails were mentioned. (Table 4).



(*continued*)


**Table 4.** (*continued*)

*Note: The responders selected up to 3 products they need but don't have or have but need a replacement*

## **4 Discussion**

Despite the small sample size, the information gathered provides valuable insight into the use of assistive products in the Dominican Republic. During interviews, it became clear that we were raising awareness of the wide range of products and services that fall under the umbrella term of 'assistive technology' as defined by the WHO, and that a large-scale national questionnaire would be of significant benefit in raising population awareness in the Dominican Republic.

The findings of the study highlighted cost-related factors influencing AP access. For example, among AP users, aside from those acquired from a family member or friend, APs were most commonly obtained from private providers and paid out of pocket, and the most frequently reported barrier to AP use was high cost. This suggests a gap in public provision of AP in this setting, which is consistent with studies conducted in other developing countries [10, 11] and indicates that low/no cost AP provision is still limited, despite the presence of some Public institutions and NGOs providing rehabilitation services in Espaillat province. This could be due to these organizations' limited resources and capacity to deliver at a bigger scale.

When discussing unmet needs, the majority of users only had mobility assistive products and a minority vision products, however products to assist in hearing, communication and personal care were reported as needed, but not currently used, highlighting the small range of AP that the respondents in this population had access to.

Similar to previous research in low and middle income countries, mobility and vision related APs are used more frequently and their unmet is more commonly reported than APs for other functional domains [12]. These results might be explained by a number of variables, such as the accessibility of these services in Espaillat and the population's stronger knowledge and comprehension of mobility and vision issues in comparison to other areas, given that these functional challenges are generally more evident and well-known.

The reported unmet demands might indicate the need for additional or alternative APs in addition to replacements. This may be due to the high cost, unavailability and the lengthy waits for medical examinations, which push users to utilize products that are defective or simply worn off. Information on how to obtain new prescriptions is also lacking. In addition, because the majority of the items on the funding-approved product lists are mobility aids, many other domains are neglected.

Regarding the access to AT products, the recommendation section revealed a low level of satisfaction. The most common complaint was about the government provision, being considered "insufficient" and "providing low levels of user needs and satisfaction". Respondents also stated that, after charity, the only option was to obtain the AP from private companies at a very high cost. A number of respondents stated that they had no recommendations to make because they were unfamiliar with the system, emphasizing the lack of awareness and education about APs and their services.

This study had some limitations, such as the small sample size. Another limitation is that the findings of this study cannot be generalized to other settings in the Dominican Republic because the government institutions, private institutions and non-governmental organizations (NGOs) that provide rehabilitation services vary by region. As a result, it is recommended that future surveys be conducted in other parts of the country.

## **5 Conclusion**

Access to AP for disabled people and the elderly should be considered a national priority in order to prevent social inequalities and improve the Dominican population's quality of life. Costs, lack of information, and product availability all work together to create significant unmet needs, which is a barrier to fair development and the enhancement of general well-being.

The current country's AP services are not providing the enough coverage. While acknowledging and enhancing existing supply lines, new measures should address the lack of understanding regarding AP.

In order to get the right solutions for the provision of assistive technology in humanitarian contexts, it is needed to develop products that can be easily sourced, environmentally friendly, and easily maintained.

Based on the current findings, actions should be taken to promote the collection of data on AP access in order to determine the current state of AP systems across the country and lay the groundwork for system improvement. Additional research is needed to determine how to fulfill demand in the Dominican Republic in order to solve the huge gap in access to appropriate assistive technology.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **A Self-quantified Based Dashboard for Supporting Aged-Workforce in Industry 4.0**

Patricia Abril-Jimenez, Sergio Gonzalez-Martinez, and María Fernanda Cabrera-Umpierrez(B)

Universidad Politécnica de Madrid, ETSIT, 28040 Madrid, Spain {pabril,sgonzalez,chiqui}@lst.tfo.upm.es

**Abstract.** With the new Industry 5.0 future factories can effectively face the aged workforce challenge, making workflows more enriched and flexible and capable to increase work well-being. This paper described how self-quantified worker could be a successful tool to achieve with this with a careful collaborative design. Our vision aims at empowering the aged workers and engage them with the work community based on adapting the factory shop floor routines to their changing needs while they age and support the aged worker to understand and develop his/her own competence.

**Keywords:** Self-regulatory strategies · work wellbeing · aged workforce

## **1 Introduction**

The European workforce is rapidly ageing. In 2030, workers aged 55–64 will make up 30% or more of the workforce in many countries, while the phenomenon of workforce shrinkage phenomenon is also evident with a decline of 0.4% every year between now and 2040 [1]. As a consequence, many governments are planning to raise the official retirement age, resulting in longer working lives with direct effects on the management of the workforce of companies. There is evidence that sustainable long-term employment can be associated with positive health outcomes if working conditions are appropriate [2, 3]. This is especially relevant in the case of industrial workplaces, as they appear to be increasingly dependent on the skills, knowledge, and experience of mature workers, and they must maintain the workability and employability of workers with adaptable working conditions. Industry 5.0 is booming to put workers at the core of the technological revolution initiated a decade ago by Industry 4.0, facilitating worker positioning to more complex activities such as abstraction and problem solving. Although Industry 4.0 could make an intelligent decision through real time communication and cooperation between manufacturing things thanks to Artificial Intelligence (AI) and the Internet of Things (IoT), Industry 5.0 introduces the principles of social fairness and sustainability in the digitalization and AI-driven technologies process initiated a decade ago, to highlights the importance of human and machines cooperation [4].

This new paradigm enables workers, and specially aged workers explore new opportunities for expanding their capabilities by the qualitative enrichment of their work, changing their role to supervision and solving unexpected problems, a position where the aged workers can find major benefits from the experiences and skills. However, fast-evolving labor conditions are also a source of daunting challenges for mature workers. They need from a friendly, flexible work arrangement in the work environment to help mature and experimented workers cope with their intrinsic and extrinsic body and life changes as years ago by and while working places introduce new workflows and technologies.

The combination of the digitalization process initiated during Industry 4.0, the vision of the human center of Industry 5.0, and the adaptation of industrial environments to occupational safety and health (OSH) practices enable the promotion of adaptable and sustainable working lives. Workability enhancing models suggest different actions and strategies to promote it at different levels [5, 6]. In the case of older workers, most of the proposed solutions recommend a stage model that addresses different dimensions of daily living of individuals that cover aspects such as health, competence and skills, and motivation, in the promotion of health at the core. In all of them, health promotion plays an essential role, but understanding the work adjustments needed to compensate for ageing-related functional changes. Redesigning work tasks according to the strengths, needs, and abilities of each individual is crucial to ensure the well-being, productivity, and workability of ageing workers, and an important success factor in successful ageing. Therefore, innovative models to promote better health and life expectancy in older workers are necessary to improve their opportunities to enhance their later life.

This paper analyses the collaborative design of a self-quantified dashboard to introduce improvements in worker well-being and self-track for task performance improvement and increase efficiency and productivity. This dashboard aims to lead to positive behavioural changes to maintain well-being at work and in private life, towards a successful ageing process, allowing workers to play an assisting role in workability and the successful transition to an active and healthy retirement.

## **2 The Design of the Self-quantified Dashboard**

When designing human-centered solutions in an industrial environment, the individual worker point of view is central in the design and evaluation activities. New solutions should be designed to meet the real needs of workers while being meaningful and engaging with their daily work activities. Consequently, the medium- and long-term results of the proposed solution should have a positive impact on worker well-being, productivity, and workability.

Our aim is to design a dashboard that supports self-management of mental and physical wellness outcomes while aged workers are embedded in an adaptable intelligent working environment that supports them with different interventions that support the beneficial, productive, and safe ageing process. This dashboard aims to lead to positive behavioral changes to maintain well-being at work and in private life, towards a successful ageing process. The dashboard design aims to adapt the principles of the Quantified Self movement to the real needs of the aged industry workers to favour the engagement with the adaptation intervention proposed by their managers. Quantified self-movement refers to the practices of tracking and self-measurement of personal health data as a practise orientated toward self-improvement and healthy lifestyle routines [7].

The initial design involved aged workers from the beginning of the process. Focus groups were held with the participation of 16 employees from four different industrial settings in Spain and Germany, all over 50 years old. During the focus group the researchers participating in the session used a funnel approach (REF) to discover the workers daily routines, their needs regarding the ageing process and the possibility to include ICT to support them or to mitigate negative aspects of the work due to the ageing process. During this first process the unmet needs and requirements about ageing, workability and private life-work balance emerge.

**Fig. 1.** Main Barriers, Needs and requirements identified during the Focus Group Session

Based on this initial identification of requirements, a hierarchical representation of goals, questions, and measurements discovered during the session with the workers were described using a GQM model [8], and based on this model, designers proposed a first visualization mockup of the dashboard. This first design included three aspects considered for a successful ageing process: physical, cognitive, and social/mental domains that were in alignment with the needs and requirements discovered during focus group sessions and organized in the GQM model. In this sense, the three domains were presented separately for the work environment and for private life. Following the approach proposed by Hall, the dashboards build on common and well-established self-tracking approaches. Based on the data automatically collected by worker personal devices (smartwatch and smartphone), feedback and environmental context awareness perform automatic detection of the worker's behavior pattern and stimulates it to self- challenges and encourages him to achieve the goals.

Once the first mockup was designed, a set of collaborative workshops were performed. In these workshops, a variable number of age workers, designers, and technicians participated to iterative co-create the final version of the dashboard. In the workshops, the participants used the available mockup to highlight the important data, prioritize and link them, using a pre-attentive processing technique. Thanks to this, the participant was able to iteratively identify the graphical properties of each piece of information; hence, the mockup was refined and improved between workshops. In addition to the graphical representation, the participants discussed and selected self-quantified measures that better improve workers' performance towards goal attainment.

During the workshop sessions, field notes were collected, while suggested changes were made to the prototype to continue discussion. After each of the workshop sessions, the notes were reviewed looking for common preferences among the participants, which were incorporated into the dashboard design, and it was used as a starting point for the next workshop session.

The emotional status of the worker plays a key role in the dashboard due to its influence in workability and private life. To avoid lack of participation or negative attitudes toward quantification of their roles due to the influence of emotional status, selfregulation techniques based on goal attributes were proposed to support self-qualified approaches. In this way, the individual worker proposed his/her own goals, allowing workers who set specific challenges to appreciate the qualified self-characteristics. This simple technical aspect has the potential to make workers perceive their change more positively and take greater benefits from it [9].

To implement this approach, a dedicated algorithm was designed and employed. This algorithm automatically monitors the level of achievement of each goal using the data gathered by smartphones and smartwatches mainly, but other sensors network available on the factory shopfloor could be also added, and according to the gathered data and the moment of the day, different motivational messages are sent to the users to encourage them to work harder or maintain this behaviour. Ideally, workers should achieve their challenges not only on the physical level, but also on the socialisation level, so motivational messages are aimed at balancing these life aspects. The rules on the delivery of motivational messages were discussed in the different co-creation sessions, with the aim of guaranteeing a fair and stimulating motivational system. In these, particularly, motivational messages always try to show positive aptitude.

The final version of the worker dashboard consists of 4 main sections presented below.

#### **2.1 My Messages**

This section aims to present notifications and messages to the user. This directly addresses the need for recall and awareness identified during the focus groups. Even this section is not really a graphical dashboard functionality, workers demanding the need of having a global view of those aspects that should be considered during the day, as well as reminders that contribute to the long-term positive habits acquisition. The section has the basic functionality of a mailing box, so they can manage the messages with the following features read, filter, and delete. Different types of messages are shown using different types of icons that have been chosen after research on the different types of information to be shown.

#### **2.2 My Daily Habits and My Job**

My daily habits and My job are the core sections of the dashboard, and they are interconnected, hence, workers can perceive the influence of their daily habits on their workability and on their private life. The emotional state of the worker plays a central role in this evaluation and follow-up. The graphical representation of the data shows a relative scale that aims to show the level of achievement of the personal goals. The graphic representation is supported by a colour scale that represents a low, medium, or high level of achievement, which also depends on the time of day. In this way, the level of achievement and motivational messages are adapted to the daily routine of the user. Additional textual information is provided to support the interpretation of the data and contribute to employee motivation and long-term positive habit acquisition.

In support of the self-quantified strategies adopted in the dashboard, a self-assessment questionnaire has been included in the My Job section. This questionnaire attempts to collect additional information from the user, in terms of stress, concentration, tiring after work, and pain during the workday, and allows dashboard algorithms and early detection of worker problems that can affect your workability or private life. The information of these variables is also presented in the form of a trend graph, supported by a three-colour scale (from red to green).

Finally, the dashboard presents a set of graphical information that supports workers with ergonomic assessment of the job position, hence preventing physical problems and work breaks, and prevent the detection of situations that cause stress, fatigue, and lack of concentration.

#### **2.3 My Goals**

The My Goals section is the core of the self-regulation theory implemented in the dashboard. Thanks to this section, workers can personalise their goals and better manage their daily habits. Additionally, goals are used to personalise the visual analytics and feedback messages that workers receive during the day according to their achievement of their goals (Fig. 2).

**Fig. 2.** Different versions of the My daily habits sections developed during the collaborative workshops

#### **3 Evaluation Results**

User acceptance was measured using a self-developed acceptance questionnaire with the participation of real potential users. The questionnaire consisted of 11 items on the following dimensions: Perception of ease of use, perceived usefulness, intention of behavioral use, and self-efficacy. Items are rated using a 5-Likert scale (0, Extremely disagree-extremely agree). In addition, a common-based UTAUT questionnaire was also used. A total of 125 workers over 50 years participated in the evaluation.

180 P. Abril-Jimenez et al.

Almost all participants perceived the system as useful for their daily tasks at work. The functionalities proposed in the dashboard were considered useful and covered the needs and barriers identified by the workers. Participants indicated their desire to use the system and they considered that they would not have significant learning problems using it properly.

The results (Table 1) also clarified some subjective statements. Feedback on the following items was particularly positive: (1) The tool was clear and easy to understand, (2) receiving recommendations through the tool would provide you useful measurements of your health (for example, your cardiac rhythm), and (3) you can use the tool if someone shows you how to do it first.


**Table 1.** Results of the acceptance questionnaire

The UTAUT results (Fig. 1) show that the more positive domains were performance expectancy (the degree to which an individual believes that using a system will help achieve gains in job performance) and attitude (the person's acceptance process) while the less positive domains were social influence (the degree to which others believes will affect someone to use the new system) and willing to pay (Confidence, commitment, and motivation to accomplish to acquire the solution) (Fig. 3).

**Fig. 3.** UTAUT questionnaire results

## **4 Discussion**

More than 200 participants, designers, developers, experts in labour health, and final users have participated in the co-creation process presented in this paper. As a result of this work, a dedicated dashboard has been successfully implemented to support and accompany aged workers in their ageing process, so that they can maintain their abilities and develop their potential. The results of the needs and requirement elicitation study show that older workers have positive intentions to use technological support that mitigates the impact of hard work conditions on their physical and emotional health and are willing to stay active as much as possible if they have good health conditions. The workers appreciated the increased sense of early detection of potential work issues that negatively affect both their workability and their health.

Close cooperation with aged workers gave us valuable insight into the discovery of the critical requirements for both the development and implementation of the dashboard, but also facilitates the development of current demographic change policies.

Workers indicated their willingness to use the system since the results improved the well-being even though there is little evidence of the technological development of older workers; this agrees with the finding from [10–12]. Furthermore, in this study, most of the 125 workers reported very good acceptance and high satisfaction in terms of usability. The influential role of improving the balance of private work on motivation, perception, and engagement with the proposed solutions emerged in worker interviews. In this sense, critical requirements are related to the adaptation of the working environment to normal functional decline. Early detection of physical and mental problems has the potential to alleviate factors that negatively impact the workability of these workers, contributing to increased job participation and successful retirement. In this sense, this work contributes directly to the research agenda proposed by [13], providing older workers with new assistive technology and new age-friendly workspaces.

One limitation of this study is that, even though the large number of real users involved during the whole process, especially at the last stage, during the usability assessment, it is a need to corroborate the good acceptance results with acceptability studies using the real setting pilot. The lack of this fully piloted implementation during this study allows only a purely descriptive evaluation of the tendencies and potential results. Therefore, the high number of real users during the whole process is sufficient to refine the dashboard design as an optimal process. Design procedures, made as usercentred design as possible, reduce failure risk due to acceptability or usability issues during a real-setting pilot. Aspects such as the impact on workability and well-being were not addressed in this first study. For this reason, a full pilot of the solution is planned, with the participation of more than 100 workers, to show the efficacy of the proposed dashboard in supporting the workability and well-being of the affected workers.

This study is the first milestone on the way to achieve successful intervention and technology support systems that have a positive influence on the successful ageing process of older workers before their retirement.

## **5 Conclusions**

Due to demographic change, there is a need for greater participation of older adults in society. Policy developments in industrial countries incorporate work as a means to achieve this, but the practical implementation of these policies is required to be with the aim of improving the understanding of the aged works, daily routines, and accompany them in the ageing process, so they can maintain their abilities, developing their potential; this paper proposes the co-creation of a dashboard.

Drawing upon self-tracking devices, dashboards, and corporative dashboards used to monitor workability and productivity, this article uses conceptual and empirical techniques to guide the research and design work of technological tools supporting the workability and well-being of aged workers. The main contribution of this paper is the application of co-creation techniques to a more general understanding of the specific need of aged workers in workability and well-being supported tools.

**Acknowledgments.** The heading should be treated as a 3rd level heading and should not be assigned a number.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Effectiveness of a Korean Smart Home Modification Program: Focused on People with Physical Disabilities**

KwangTae Moon, Yeong-hun Kwon, and Jongbae Kim(B)

Department of Occupational Therapy, College of Software Digital Healthcare Convergence, Yonsei University, Seoul, South Korea {otbuilder,yhkwon97,Jongbae}@yonsei.ac.kr

**Abstract.** The purpose of this study was to investigate the effects of the Korean Smart Home Modification Program (KSHMP) on the activities of daily living and health-related quality of life of people with physical disabilities. The study used a single-group pre-post design with 10 people with physical disabilities living at home. During the intervention period, the KSHMP was implemented, which included occupational profile, smart home installation, setup, training, task-based feedback, and monitoring. Post-intervention changes in activities of daily living and health-related quality of life were assessed with the Canadian Occupational Performance Measure (COPM) and EuroQual-5 Dimensions (EQ-5D). After the KSHMP, all 10 subjects improved their activities of daily living and quality of life. In addition, the occupational performance of all subjects was maintained. These results show that a customized smart home has a positive impact on improving the activities of daily living and quality of life of people with retardation and is an efficient alternative.

**Keywords:** Smart Home · Home Modification · Quality of Life · Disabled Persons · Korea

## **1 Introduction**

People with physical disabilities (PwPD) frequently have difficulty managing and accessing their homes, depend on a caregiver for assistance in completing their daily-life activities [1]. Based on the Institute of Medicine model published by Pope and Brandt, the degree of disability was reported to vary depending on the individual's ability and support for the physical-social environment [2]. The per-son-environment-occupationperformance model is a measurement of the degree of occupational performance, and it changes with the appropriate interaction of the person, environment, and occupation [3]. It is important to note that an increase in occupational performance can create meaningful occupations, promote occupation participation, and improve quality of life [4, 5]. Their study suggested that PwPD, with increased occupation-al performance, were associated with creating and satisfying independent activities of daily livings (ADL) that could further improve their quality of life [6]. Home modification programs include improved housing structure, relocation of furniture, installation of amenities, efficient moving space, and education for changing environmental adaptation, and Advanced technologies such as smart homes, enable PwPD to improve their ADL and participate more in life [7, 8]. The medical rehabilitation system in Korea consists of multidisciplinary teams, including doctors, occupational therapists, and social workers, in rehabilitation medical institutions, and is implementing a home modification program as part of the discharge plan [9]. However, it is difficult to reflect sensor lighting, access outlets, control brightness, and use auto doors, or people with stroke and spinal cord injury; ADL independency limitations remain in homes [10, 11]. In addition, the requirement for a home environment that meets the physical functions and needs of PwPD is increasing [12, 13]. A smart home is defined as a technology and service environment that combines in-formation and communication technology with human space and devices to provide value, such as safety, convenience, economy, and pleasure [14]. Based on research on people using smart homes, controllers, such as voice recognition, tablets, and remote controls, are key technologies for improving the quality of life of PwPD and can remotely control home appliances, phone use, alarm settings, lighting on/off, windows, doors, and applications for daily activities. However, this has mainly been conducted abroad [15, 16]. In Korea, owing to different barriers that limit the use and effectiveness of smart homes, such as complex use, high cost, lack of institutional maintenance, and opinions deeming smart homes unnecessary, the only available studies that apply smart homes to a testbed; clinical studies that apply smart homes to actual homes are limited [9, 17]. The objective of this study is to gather information on the applicability, effectiveness, and limitations of providing a Korean Smart Home Modification Program (KSHMP) to community practice areas. This is the first step in developing programs that promote quality of life, occupational performance, and home management.

## **2 Method**

## **2.1 Participants**

From December 2022 to May 2023, we visited ten participants at home and provided KSHMP. The characteristics of the participants were 5 men and 5 women, with an average age of 46.7 years, five spinal cord injuries, two strokes, and three cases of cerebral palsy.

#### **2.2 Korean Smart Home Modification Program (KSHMP)**

The KSHMP consisted of three phases: occupational profile, intervention and monitoring, and follow-up. In the first phase, researchers visited participants' homes to collect baseline information, identify key needs and smart devices in use, assess the occupation space, observe, and analyze occupational performance, and explain and discuss home modifications. In the second phase, researchers installed and set up the smart home in each participant's home. Researchers trained participants and caregivers on how to use the smart home and provided task-oriented feedback and monitoring. In the third phase, we conducted a post-test and interviewed participants about their experiences with the smart home. The KSHMP was determined by space through a literature review of the impact of smart homes on daily life [18]. The applications and contents of the smart home in this study are shown in Table 1.


**Table 1.** Smart home by space

## **2.3 Occupational Profile**

The occupational profile is summary of a participant's occupational history and experiences, patterns of daily living, interests, values, needs, and smart home relevant contexts. Using a client-centered approach, the occupational therapist collected information to understand what participants want and need through SHT and to identify past experiences and interests that can help them understand current problems and problems. The researcher evaluated the participants' occupational performance, satisfaction, and health-related quality of life before and after KSHMP. Canadian Occupational Performance Measure (COPM) was used for occupational performance and satisfaction, and EuroQol-5 dimension (EQ-5D) was used for health-related quality of life [19, 20].

## **2.4 Home Modification**

Home modification was performed to improve the participants' indoor ADL. After checking the participants' bedroom environment, the researcher installed controllers and smart devices based on the participants' physical functions to lower environmental barriers and increase convenience [21]. The researcher monitored and supplemented this to adapt to the changed smart home environment and improve ADL during the study period.

#### **2.5 Smart Home Education**

The researchers taught participants and guardians to understand and use smart homes. It supplemented the manual provided by smart device manufacturers and provided it to individuals in an easy-to-understand manner.

#### **2.6 Task-Oriented and Simplify Occupational Performance Skills**

In this study, the task-oriented approach is a training method focused on special functional tasks that combine musculoskeletal for nervous systems, providing occupationbased routines for participants rather than repeatedly practicing normal patterns [22]. The researcher prioritized meaningful activities for each topic to build a task based on the smart home, which was changed for each participant based on the occupational profile. However, if the participants first performed a task or failed to perform it independently, they could learn how to perform the task accurately by referring to the error-free fourstep learning in Thivierge et al. [23]. We modified the participants' fewer effective skills to make them efficient and allow them to perform their occupations with less physical effort [24].

#### **2.7 Statistical Analysis**

Statistical analyses in this study were performed using JAMOVI version 2.3.26.0 software. Participants' general characteristics and pre-post assessments were presented as means and standard deviations. Due to the small sample size (n = 10), a non-parametric test, the Wilcoxon signed-rank test, was used to test whether there was a central difference between the pre- and post-tests between the two groups. The level of significance was set at P *<* 0.05.

## **3 Results**

#### **3.1 Participants and Intervention**

Study participants were recruited based on selection criteria through the Gangwon-do Regional Health & Medical Center for People with Disabilities and the Seoul Northern Regional Health & Medical Center for People with Disabilities. A total of 10 people participated in the study, and their pre- and post-assessment results are shown in Table 2. All participants were treated at the Department of Family Medicine and Rehabilitation, and the intervention began after they agreed to participate in the study. This study was conducted with the approval of Yonsei University Wonju Institutional Review Board (1041849-202211-SB-218-02). After receiving training on the application of the smart home intervention, a licensed occupational therapist visited the participants' homes from December 2022 to May 2023 to implement the smart home modification program.


**Table 2.** COPM, EQ-5D change Scores for Participants Who Had a KSHMP.

## **4 Discussion**

A smart home is a new home environment for people with physical disabilities in the New Normal era and is important for health-related quality of life [12]. However, rehabilitation experts feel that the smart home is cumbersome and has limitations in its application compared to the development of fourth industrial technology. The fast-developing smart home is a promising new model for the satisfactory daily lives of physically handicapped people. PwPD can experience daily comfort when using a smart home, which coincides with a study indicating that an environment suitable for physical ability tends to lower the dependence on daily life and increase the health-related quality of life [25]. Participants in this study applied the smart home technology to remain in their home environment for as long as possible. The occupational therapist evaluated the occupational performance in the bedroom based on the occupational profile. Through the occupational profile and occupational performance evaluation, uncomfortable activities due to the physical limitations of the study participants were observed, and the strengths and barriers of the environment were identified and modified. These points were similar to the existing home modification procedures [26]. Based on previous studies, home modification has been reported to reduce parental burden and housing support costs through the improvement of housing structure, relocation of households, installation of convenience facilities, secured movement, application of auxiliary devices, changed environmental adaptation, and housing applications [27]. In this study, education and tasks related to the smart home were provided to allow the research participants to adapt to the smart home; occupational performance skills were simplified. All study participants experienced the difficulty of learning how to use a smart house; however, they noted that it was essential as they became accustomed to it. As can be observed from previous studies, it was initially evaluated as cumbersome because of the limitations of smart home applications for physically disabled people. However, they became accustomed to the smart home and could participate in limited activities despite uncomfortable hands and walking [28, 29]. Owing to these changes, it was confirmed that they participated in the occupation by controlling the lights, stands, fans, heaters, air conditioners, and curtains in the bedroom, even with their physical restrictions.

## **5 Conclusion**

The study identified interventions and home modifications that can assist PwPD in managing their condition within their homes. Implementing the KSHMP has the potential to encourage the development of cost-effective service delivery models. Utilizing the KSHMP to improve occupational performance and health-related quality of life for PwPD may broaden the scope of research and interventions used in practice, eventually extending to the community for occupational therapists.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Real-Time Multiple Object Tracking for Safe Cooking Activities**

Hubert Ngankam(B) , Philippe Dion, H´el`ene Pigot , and Sylvain Giroux

Laboratoire DOMUS, D´epartement d'informatique, Universit´e de Sherbrooke, Sherbrooke, Canada

*{*hubert.ngankam,philippe.dion,helene.pigot,sylvain.giroux*}*@usherbrooke.ca https://domus.recherche.usherbrooke.ca/

**Abstract.** This work presents a real-time system for tracking multiple object in the context of meal preparation when using the Cognitive Orthosis for CoOKing (COOK). This system is called SafeCOOK. It aims to provide more capabilities to detect some dangerous situations that the current system does not consider. For example, it can locate a utensil or other kitchen object that has been left on the cooking surface of the stove while a meal is being prepared. This system uses a hybrid method based on YOLO and KCF to detect, track and drop cooking utensils as they enter and leave the cooking area, and is capable of monitoring an entire cooktop in real-time with a single camera. The software has been implemented on an embedded platform in the smart stove and has been added to it. The system produces good segmentation and tracking results at a frame rate of 1 to 4 frames per second, as demonstrated in extensive experiments using video sequences under different conditions.

**Keywords:** Ambient Intelligence *·* Object tracking *·* Multi-object tracking *·* Context-Aware *·* YOLO *·* KFC

## **1 Introduction**

The availability and accessibility of home monitoring technologies have enabled friends, family members, and caregivers to track the activities of older adults [10]. Tracking the activities of daily living (ADLs) of older adults is also important for measuring the safety conditions of older adults [14]. Smart homes are equipped with Internet of Things (IoT) devices that monitor the elderly and communicate real-time data/information to stakeholders [2,13]. Ambient intelligence applications, such as motion and position detection, fall detection, telemonitoring, cognitive technology assistants, sensor-based wearable systems, and automated home monitoring, are among the tools to provide safety precautions for older adults during their daily activities [9,10].

Among these ADLs, meal preparation is one of the most important activities of daily living [6,22]. Especially when used as part of the home maintenance assessment for people with or without loss of independence. Activities such as meal preparation have recently been explored to exploit the assistive capabilities of ambient assistance solutions to prevent dangerous situations [4,18,22]. In the context of meal preparation, we have identified 3 categories of so-called dangerous situations. The category of omissions concerns objects appearing on the cooking surface, the oven door open, and the burners left on without a pot on it [7,14]. In the category of an absence of supervision, it is a question of being present in front of the stove during the cooking process. This category varies greatly depending on the temperature of the burners, the contents of the pans, the temperature of the oven, and the cooking time. The last category is the presence in the home during the use of the stove. We will focus on the dangerous situations of the forgetfulness category in the following.

Contextual awareness is particularly important in a high-risk activity such as meal preparation, where assistive technologies must be able to identify and assess problems as they arise and manage dangerous situations in real-time. Except that some elements of context are not observable by motion or contact sensors [3]. The integration of computer vision and deep regression network-based systems into smart stoves would provide more context to recognize certain dangerous situations during meal preparation. The combination of using a different camera, IoT sensors, and real-time processing of this data allows for the introduction of multiple objects tracking techniques [5,12,19].

Multiple object tracking (MOT) is a research topic related to computer vision [11,16,17,21]. When we talk about MOT, it is largely about locating multiple objects, maintaining their identity, and generating their trajectories from an input video. As a mid-level activity in computer vision, multiple object tracking underlies high-level tasks such as pose estimation, action recognition, and behavior analysis [11]. It has many practical applications, such as visual surveillance, human-computer interaction, and virtual reality [11].

In this paper, MOT is used for the detection of objects appearing on the top of the stove to improve safety during meal preparation when a round is left hot with a cloth on it; (2) the ability to maintain or abandon the tracking of detected objects to take into account the evolution of the context. Since a dangerous situation can be caused by different types of objects appearing on the stove, we developed a real-time algorithm to extract multiple perspective views from a single capture device to track objects in real-time and stop tracking once the object leaves the camera's field of view over the stove.

The remainder of this paper is organized as follows. Section 2 briefly introduces the COOK system. Section 3 presents our object-tracking pipeline. Section 4 presents how this algorithm is applied to the COOK system. Section 5 presents the results and a brief discussion. Finally, Sect. 6 concludes the work.

## **2 COOK: Cognitive Orthosis for CoOKing**

The Cognitive Orthosis for coOKing (COOK) is an innovative, context-aware, stove-connected smart tablet application designed to optimize the independence of individuals with cognitive deficits during meal preparation [7,18]. This application was originally designed to specifically target the cognitive deficits of people with moderate or severe traumatic brain injury [6,14]. But now the system is designed for anyone who wants a safe environment for meal preparation. The COOK system addresses non-emergency situations. The system assists the person with tasks related to meal preparation (ingredients to prepare, planning, safety rules, etc.). It can detect pre-defined hazardous situations through various sensors, including motion and door sensors, ultrasonic range finders, door openings, flame and high-temperature sensors, and smart switches as shown in Fig. 1.

**Fig. 1.** Example of some sensors installed on the smart stove.

The main components are (1) a Self-monitoring Security System, (2) a Cognitive Assistance Appliance, and (3) a Configuration System. Only the selfmonitoring Security System will be discussed in the rest of this paper.

#### **2.1 Self-monitoring Security System**

In autonomous operation, i.e. not depending on the other components of the system, the Self-monitoring Security System (SSS) has the role of monitoring all the so-called dangerous or hazardous situations [1]. While the user is cooking, if necessary, the SSS will progressively indicate the presence of a potentially hazardous situation that must be resolved within a predefined time frame. The SSS is designed to monitor three hazardous situations: supervision, omissions, and presence. The safety system collects information from the sensors, detects critical errors and hazardous situations, and, if necessary, turns off the stove and calls for help. Through the sensor infrastructure, the safety system can detect the temperature of the top of the stove, the presence of a person near the stove, the opening of the oven door, the temperature of each burner, the amount of electric current flowing on each burner, etc.

Several standard and predefined security rules are provided by the safety system. However, to monitor dangerous situations, the SSS uses sensors such as motion and door sensors, ultrasonic distance sensors, flame, and hightemperature sensors. These sensors are not able to distinguish between different objects that arrive at the burner surface. Some of these objects are kitchen towels, hands, cooking utensils, etc. Therefore, the SSS needs a preventive assistance model that captures each user's idiosyncrasies, personalizes preventive assistance, and provides context-aware assistance regarding food preparation hazards. The objective of this version of the SSS is to extend its potential to detect new dangerous situations. By adding a camera to the existing sensors and a new real-time object-tracking model to the SSS. The following situations could be easily detected in conjunction with user context information.


To sum up, to have a module that makes it possible to follow in real-time several objects that appear on the cooktop during the preparation of a meal.

## **3 Multi-object Tracking**

Predicting the object's location (bounding box) with class is called object detection [21]. By locating the potential object locations in each frame, object detection can provide observations for detection-based object tracking. Therefore, either in every frame or when the object first appears in the video, each tracking method requires an object detection mechanism [11].

Object tracking is one such application of computer vision where an object is detected in a video. Otherwise, it is the process of identifying the same object and keeping track of its location with the unique label as it moves around in a video. For example, you have a video of someone cooking a meal, and you want to track the location of the frying pan constantly throughout the video in real-time by estimating its trajectory. Object tracking can be done using one of the two approaches that exist in the field. One is called Single Object Tracking (SOT) and the other is called Multiple Object Tracking (MOT) [3,16].

Multiple Object Tracking is when different objects are tracked at the same time within the same video or the same set of frames [17]. For example, you have a video of a meal preparation activity, and you want to track the location of the pans, spoons, and utensils continuously throughout the video in real-time by estimating their trajectory. The proposed real-time multiple object tracking (MOT) systems typically follow the tracking-by-detection paradigm [5,12]. It includes (1) a detection model for the target location and (2) an appearance embedding model for data association. In other words, the proposed MOT approach has to accurately detect the objects in each frame and provide a consistent label for them. There are certain cases where the visual appearance of the object is not clear as long as there is a moving object in the food preparation activity. For example, a spoon may be hidden behind a frying pan. In all such cases, detection would fail while tracking would succeed. That's why we combine these two modes to provide a safe environment for food preparation.

Detecting and localizing the object in a video in a fraction of a second and with high accuracy is expected and required from the proposed MOT algorithm. Due to the variety of stove burners and backgrounds with similar colors or textures as the object, this detection speed can be significantly affected. Another significant difficulty in the context of meal preparation is occlusion. This is when the object in question is partially or completely occluded by another object. A further problem is that an object in a video can have different sizes and directions. In food preparation, there are often two objects crossing each other. How do you know which is which? This problem is known as identity switching. Another problem with MOT, which is also important in detecting and recognizing objects, is called Motion Blur. Motion blur is when the object is blurred due to the motion of the object or the motion of the camera. As a result, the object no longer looks the same. A quick move of the hand in front of the camera to grab an object on the cooking surface or to turn the soup in the pot.

Two object tracking techniques are used to solve these problems: Detection-Based Tracking and Detection-Free Tracking [20]. In detection-based tracking, successive video frames are presented to a pre-trained object detector. The detector generates detection hypotheses that are used to build tracking trajectories. In detection-free tracking, a fixed number of objects must be manually initialized in the first frame. In subsequent frames, it then tracks these objects. It is not able to deal with the case where new objects appear in the intermediate frames.

Typically, objects move back and forth during meal preparation. For our context, this means that non-tracking detection is not appropriate. In addition, new objects appearing and disappearing must be automatically supported, i.e., recognized when they appear and forgotten when they disappear. Therefore, tracking-based detection could meet our needs. Finally, the tracker is used to make the remaining predictions, and the object detector is run for every n frames. For tracking over a long period of time, such as food preparation, this approach is very suitable.

#### **3.1 General Algorithm**

To track objects appearing on the cooking surface of the stove during a meal preparation activity in real-time, this study proposes a hybrid method. For this purpose, the Kernelized Correlation Filter (KCF) [8], which is particularly fast in tracking, is used in combination with the YOLO [15], which is an accurate and relatively fast detection model. The workflow of the proposed method is shown in Fig. 2. First, the objects in the video image were detected by the YOLO algorithm, and the surface target was continuously tracked by the KCF algorithm. After tracking a certain number of frames, the detection mechanism is re-introduced, and then the output of the target frame is obtained through the feature fusion algorithm based on machine learning, and when new targets appear in the field of view, the new targets are initialized and tracked.

**Fig. 2.** General Algorithm for Safe COOK using YOLO and KCF.

To summarize, an input in the form of a video file is processed in the pipeline, and the output is the bounding box of each object related to food preparation detected in the video frame, with its identity obtained from the tracking process, with the possibility of automatically adding an object or stopping tracking it. Two main concepts are proposed: (1) estimation model; and (2) data mapping. The estimation model is used to estimate the next bounding box of a detected bounding box, while the data mapping is used to map the predicted next bounding boxes to the actual or detected bounding boxes to link them into a unique identification number. The KCF filter must be used to update and predict the data with the equations described for each step to complete the tracking process. The value is a measurement from each frame. In this case, it is represented as the center of gravity of the bounding box found by YOLO. This process must be performed along the entire video stream, with each frame considered a different state and the predicted values updated recursively with the measurements.

## **3.2 YOLO Detector Model**

This algorithm is called You Only Look Once (YOLO) because it identifies objects and their locations using bounding boxes by looking at the image only once [15]. YOLO is proposed for real-time object detection in high-speed video using a convolutional neural network architecture. It has several advantages over traditional methods, including high speed, low computational costs, and fewer background errors. Instead of processing each class separately, the YOLO network turns the detection problem into a regression problem where the image is used only once as the input network. This makes it much faster than traditional object detection algorithms to determine the location of detected objects in the video frames. The single deep convolutional neural network used by YOLO is capable of predicting all the bounding boxes and the class probabilities for those boxes at the same time, using features from the entire image. It is suitable for end-to-end training in real-time while maintaining good average accuracy. The YOLO object detector is fast because it works in one step. The image is first divided into an SxS grid; each grid cell predicts B bounding boxes, including the coordinates, the width, the height, the confidence values of the box, and the conditional class probabilities.

#### **3.3 Kernelized Correlation Filter Tracker Model**

Correlation measures how similar two patterns are, with more similar patterns being more highly correlated. It is obtained by the pairwise product of corresponding pixels. Correlation filtering, which uses trained sample images that capture the appearance of objects of interest, is a widely used method in visual tracking applications. Unfortunately, the large sample size requirement of correlation filters becomes computationally demanding, conflicting with real-time requirements. However, the limitation of the number of samples can be at the expense of performance. A novel idea that significantly increases the tracker's computational speed is to exploit the circulant structure of the data matrix for larger sample sizes and to extend it using a kernel trick. This is known as the Kernelized Correlation Filter (KCF) tracker [8]. To perform fast tracking, the KCF reveals the relationship between the circulant matrix and the discrete Fourier transform. The circulant matrix rows consist of the target model and the model's cyclic shifts. The ridge regression used to learn the image windows is defined in the frequency domain using the Discrete Fourier Transform. There are two main phases, training, and detection, in the KCF tracking algorithm. In the training phase, there is a training of the classifier based on a set of samples. Threedimensional color or oriented gradient histograms are used as feature vectors in this phase. In the detection phase, a new sample is detected by the operation of the Fourier transform. Typically, this involves multiplying the weights obtained after training by the frequency domain test images to find the possible position of the target. Before any computation can be performed, each tracker requires the extraction of frames. Once the frames have been extracted, the KCF needs a training model with the image patch at the initial position of the target.

## **4 Multi-object Tracking in COOK**

In this study, we propose a fast object-tracking pipeline that combines accurate YOLO detection and fast KCF tracking to track specific objects for the preparation of meals. Typically, several objects appear and disappear quickly on the cooking surface during a meal preparation activity. For this reason, we have added an extra step to perform a fast drop tracking. This solution was chosen because it can be useful to work in an environment where resources are limited, such as in an embedded system. The overall architecture of the implemented system is shown in Fig. 3. The system uses deep learning, OpenCV, and Python to detect objects in images and video streams using the YOLO object detector. We created an additional dataset of images with different types of kitchen utensils to train the YOLO models and test the proposed methods.

**Fig. 3.** Architecture of the Safe COOK system, showing how the MOT can provide more information in the context of hazardous situation detection.

In this architecture, the camera collects data from the stove and uses the ZeroMQ (ZMQ) protocol to transfer the data. ZeroMQ is a high-performance asynchronous messaging library intended for use in distributed or concurrent applications. This information is then sent to OpenCV, which runs the proposed solution for multiple object detection. The proposed real-time method, through an object detection, localization, and correction process, automatically detects and tracks the utensil appearing in a sequence of images. The central coordinates (x, y) of YOLO are updated every thirty frames. This is a trade-off between accuracy and speed.

A home kitchen with a stainless steel countertop and a spiral kitchen layout was used to record the meal preparation videos. The videos were recorded

**Fig. 4.** Sample frames from our training dataset. Showing detection of hands, stoves, pans, and spoons.

with a high-resolution camera. The camera was connected directly to a 4 GB Raspberry PI 4 model B+. Image transport was done using imageZMQ, a set of Python classes that transport OpenCV images between computers using PyZMQ messaging. Experiments were done for the YOLOv5 network. Existing data like OpenImage provides a large set of trained and annotated images (bounding box) with 600 classes. However, this number of classes is limited, and some classes do not have enough images to do the training properly.

Therefore, data from OpenImage and Google Colab were used for training on an IntelR Core TM i7-3770 CPU 3.40GHz 4 cores with 8 threads and Raspberry Pi 4 with 4 GB. Mainly we labeled the following classes: Pan, Hand, Stove, and Spoon as shown in Fig. 4. There are many datasets for testing object tracking methods in videos. However, these datasets do not consist of meal preparation videos. Therefore, to create a dataset for our specific safe cook app that tracks meal preparation, we tagged 3500 frames. From this dataset, 1000 frames are used for the training of the YOLOv5 models, while the remaining 2500 frames are used for the testing.

## **5 Results and Discussion**

This solution adds further context to the detection of hazardous situations. It provides data that is combined with sensor information to offer greater safety during meal preparation. However, some problems need to be solved to make it more optimal.

Foremost, YOLO does not recognize the uniqueness of certain objects from one image to the next. For example, there is no way to be sure that the detected pan is the same pan that was detected in the previous image. Our solution is to compare the position of objects detected by YOLO with the position of objects detected by tracking. So if we track an object of the same type at the position of the YOLO detected object (with some error depending on the implementation), we consider them the same objects. With this approach, we are now able to recognize the type and the instance of the objects. For example, we will be able to distinguish between pan1, pan2, and so on.

The second problem is that it is possible for the tracking to start tracking a similar object instead of the previously detected object. This happens even when using KCF, which was specifically chosen for its ability to easily stop tracking when the object is lost. This is most likely to happen with objects that are in proximity to identical objects. For example, how can you track an object that has the same color as the image background? How can you be sure that you still have the right object on the track? The solution chosen for this problem is to use object detection (YOLO) with a very low confidence level.

**Fig. 5.** Performance for tracking and detecting objects in interaction

The objects detected with this confidence level are compared to the tracked objects. In other words, does the object on the track have a minimal chance of being the object on the track? If this is not the case, then we will stop tracking this object. Since the tracking is usually stopped because of this condition and not because of the loss of tracking, the overall result is a detection algorithm that makes it easier to detect objects (minimum confidence level), but at the same time correctly identifies objects (higher initial confidence level). On the other hand, the effectiveness of monitoring is greatly reduced with this solution. Therefore, it is ideal to find another solution that would easily drop tracking.

The object detection and tracking techniques of YOLO and KCF can be easily substituted or replaced for different models due to the modularity of our algorithm. We replaced KCF with other tracking algorithms to evaluate the effectiveness of our tracking loss algorithm. Starting from the simple use of YOLO without tracking, then YOLO with single tracking without detecting multiple instances of objects, then YOLO with tracking of multiple instances of objects, and finally YOLO with tracking of multiple instances of objects with drop tracking. The sequences obtained are shown in Fig. 5.


**Table 1.** Performance evaluation of the different methods with and without tracking.

Note that the multiple tracking algorithms without drop tracking have the best performance metric. However, it often tracks the wrong objects. So on the whole it is not optimal. In addition, although YOLO alone has a better performance metric than the full algorithm, the full algorithm (MOT+drop) has better detection and is still able to do detection at sufficient speed (realtime) to be useful in the context of the Safe Cook project. The performance quantified on a video of the experiment is shown in Table 1.

## **6 Conclusion**

SafeCOOK is a real-time system for tracking multiple objects associated with preparing a meal while using the Cognitive Orthosis for CoOKing (COOK). It provides more contextual information to the safety self-monitoring system for the detection of potentially dangerous situations that are not currently managed.

The proposed system is a low-cost monitoring system without a graphics card based on YOLOv5 and KCF. The main idea is to reduce the limitations of the resulting tracking system by combining a tracker and a detector to take advantage of both methods. In order to meet the requirements of the fast and frequent appearance and disappearance of cooking utensils on the cooking surface, we propose in this paper an algorithm that fuses the detection, the object tracking, and the fast object dropout tracking. The fusion algorithm is designed in such a way that the detection and tracking information is complementary in order to fulfill the tasks of detecting hazardous situations during food preparation in different scenarios. The findings show that the detection and tracking fusion algorithm can effectively detect and track kitchen tools in different scenarios, and the success rate exceeds that of a single detection and tracking algorithm.

## **References**

1. Amaro, M.C.O., et al.: An ontology model for a context-aware preventive assistance system: reducing exposition of individuals with traumatic brain injury to dangerous situations during meal preparation, pp. 1–8 (2016)


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Towards a Medical Waste Classification System Based on Blockchain, Smart Contracts, and NFT Technologies**

Hien Q. Nguyen1(B) , Nguyen D. P. Trong<sup>1</sup>, Luong H. Huong<sup>1</sup>, Khoa T. Dang<sup>1</sup>, Khiem H. Gia<sup>1</sup>, Phuc N. Trong<sup>1</sup>, Hieu L. Van<sup>1</sup>, Loc V. C. Phu<sup>1</sup>, Duy N. T. Quoc<sup>1</sup>, Nguyen H. Tran<sup>1</sup>, Anh N. The<sup>1</sup>, Huynh T. Nghia<sup>1</sup>, Bang L. Khanh<sup>1</sup>, Kiet L. Tuan<sup>1</sup>, Ngan N. T. Kim<sup>2</sup>, Bao Q. Tran<sup>2</sup>, Hieu M. Doan<sup>2</sup>, and Hong K. Vo1(B)

> <sup>1</sup> FPT University, Can Tho, Vietnam hiennqce172010@fpt.edu.vn, khanhvh@fe.edu.vn <sup>2</sup> FPT Polytechnic, Can Tho, Vietnam

**Abstract.** The end of the covid-19 epidemic has revealed many weaknesses in the health system and the medical waste treatment process. In particular, the ineffective treatment of medical waste has also contributed to the explosion in the number of infections in some countries (i.e., India, Brazil, and Vietnam). Several studies have found that even developed countries (i.e., with better infrastructure and health services than the world average) have to face emergencies during this time. epidemic. Therefore, the amount of medical waste dumped into the environment is extremely terrible. The waste generated suddenly during this period includes protective gear, masks, and vaccines that burden the waste treatment process. There are several approaches to exploiting Blockchain technologies to solve the problem of direct contact between the stages: medical staff - transportation staff - waste disposal staff to minimize unintended spread. However, to thoroughly solve the current waste classification and treatment processes, a more reward/punishment solution is needed. Specifically, we propose a model to assess the compliance/violation level of waste sorting and treatment in medical centers and isolation areas based on current popular technologies: blockchain, smart contracts, and NFTs.

**Keywords:** Medical waste treatment *·* Blockchain *·* Smart contracts *·* NFT *·* Ethereum *·* Fantom *·* Polygon *·* Binance Smart Chain

## **1 Introduction**

The passing of the Covid epidemic has revealed inadequacies in the treatment of medical waste. Studies conducted by [10] have shown that medical waste directly contributes to the spread of Covid disease. Other studies have also shown that upgrading medical facilities (i.e., waste disposal processes) is not enough to ensure a safe, closed process in a healthcare environment. Therefore, a series of approaches have been born [1,17] to speed up the processing process to support medical doctors and e-medical staff, where direct contact is minimized (ie., faceto-face). These solutions also support the doctor/nurse-patient visit process. Specifically, patients will report information online about their symptoms. The doctors will give the corresponding advice to help the patient have an effective treatment plan.

However, for the medical waste treatment process, where there is a need for interaction between many stakeholders (i.e., nurses, doctors, patients, staff in medical center departments, as well as employees of waste transportation/disposal companies). Indeed, 99% of items (including medical equipment and supplies) become junk after use within the first six months of first use [13]. Therefore, many approaches [3,5] have presented processes based on central processing in the waste treatment process before the time of the Covid epidemic. However, these solutions do not take into account the risk reduction of direct contact between stakeholders (eg, health workers - transportation companies or transport companies - waste treatment companies). In addition, the approach based on traditional centralized data storage faces many risks, such as a lack of transparency between information transfer stages [9]. Centralized storage also prevents information authentication between the parties involved in the traditional waste treatment process [6,7]. In addition, information is easily hacked or stolen by malicious users [20] because all information is stored centrally and is difficult to recover when attacked by hackers or natural disasters (eg, fire or flood).

Because of the above risks, many approaches exploit Blockchain technology as an alternative to centralized storage, and lack of transparency [14]. In the matter of waste treatment, the majority of systems deployed on two main platforms, Ethereum [8] and Hyperledger [12] offer solutions medical waste treatment (see Related work for details). However, these approaches have not yet provided a specific definition for reward behaviors and violations for individuals and organizations for their waste sorting behavior.

Therefore, in this article, we propose a medical waste classification model based on the combination of a series of current advanced technologies including: Blockchain, smart contract, and NFT (see Approach section for more details). In addition, to demonstrate the effectiveness of the proposed model we implemented a proof-of-concept based on the Ethereum platform (i.e., see 2 section for reasons to use this platform instead of other platforms) combined with ERC 721 (i.e., Ethereum's NFT issuance certificate). We also install smart contracts on three EVM (Ethereum Virtual Machine) enabled platforms including BNB smart chain, Fantom, Polygon and Celo (i.e., see Evaluation for more details).

Therefore, our work contributes on four aspects. (a) Propose a mechanism for classification and treatment of medical waste based on blockchain technology and smart contract; (b) propose a model to create reward/punishment decisions for individuals/organizations based on NFT technology; (c) implement the proposed model based on smart contracts and the proposed model (i.e., proof-of-concept); and (d) deploying proof-of-concept on 4 supporting platforms (ERC721 - NFT of ETH) and EVM (deploying smart contract implemented in solidity language) including BNB Smart Chain, Fantom, Polygon, and Celo.<sup>1</sup>

## **2 Related Work**

#### **2.1 Ethereum-Based Waste Treatment System**

Two popular examples in applying the Ethereum platform in building a supply chain management model for waste treatment in the healthcare environment are [8]. For electronic medical devices (eg, CT scanner, X-Ray machine, pacemaker, etc) that are no longer in use (e.g., damaged, expired) at the centers healthcare is called e-waste, Gupta et al. [8] proposes a system to treat the above types of waste based on Blockchain technology (i.e., Ethereum). To develop processing policies that comply with strict requirements at medical centers, the authors have built a system of smart contracts. In order to raise awareness among users about waste disposal, each user will receive a reward if they complete the sorting before sending them for disposal. However, these solutions only apply smart contracts to distribute reward points instead of storing evidence of reward receipts or violations in the waste disposal process - as we have applied (i.e., based on NFT).

Another model was introduced by Laura et al. [11], which is implemented on the Ethereum platform. This system helps observers in assessing violations in solid waste segregation (computers and smartphones). The difference between this paper and other approaches is that it proposes a remote management model based on QR code (i.e., generated from classification to processing). For crossborder processes (i.e., between countries rather than within the same country) Schmelz et al. [16] introduces an Ethereum-based system, privacy protection model, for garbage disposal. Information that is not related to the waste treatment process will be stored offchain instead of putting them on-chain. This approach is similar to our NFT-based certificate and metadata storage model. However, our focus is on encouraging standard behavior in waste sorting as well as violations of the above procedure.

#### **2.2 Waste Treatment System Based on Hyperledger Fabric**

In this section, we show three case studies in applying the Hyperledger platform to propose a medical waste treatment model. Most waste products during the Covid 19 pandemic are listed as hazardous waste to health (i.e., 99% of items after use are medical waste [13]). So they combine blockchain and smart contracts to classify and dispose of garbage instead of just centralized local storage. For example, Trieu et al. [12] proposes a waste treatment model called MedicalWast-Chain. This proposed model is aimed at the treatment of medical waste from

<sup>1</sup> We do not deploy smart contracts on ETH because the execution fee of smart contracts is too high.

medical centers, reuse of tools, the process of transferring medical supplies waste (i.e., protective gear, gloves, masks, etc.) and waste treatment processes in factories. They aim for a solution to help parties trace the source of waste and toxic levels during the pandemic. Similar to the above approach, Ahmad et al. [2] aims at the traceability model of personal protective equipment for healthcare workers (i.e., doctors, nurses, testers) during the pandemic. To assist with validation of waste treatment processes (i.e., stakeholder interactions), Dasaklis et al. [4] proposed a blockchain-based system set up on smartphones.

However, the above approaches do not pay too much attention to the regeneration/refurbishment process. Neither of the above approaches offers a reasonable solution for handling (i.e., reward and handling of violations). Specifically, the above solutions (for both Hyperledger Eco-system and Ethereum platforms) only focus on the management model of the waste treatment chain from the place of origin (i.e., medical centers) to factories. In this paper, we not only offer a model to manage the waste sorting process, but also provide a solution for rewarding and handling user violations based on NFT technology.

## **3 Approach**

#### **3.1 The Traditional Model of Medical Waste Treatment**

**Fig. 1.** The traditional model of treatment and classification of medical waste

To build a model of treatment and classification of medical waste that is commonly applied today, we refer to the process of classifying and treating medical waste during the Covid-19 pandemic. In the documents that can be downloaded and referenced from CDCs in different parts of the world, we choose Vietnam for two reasons: i) Vietnam has successfully controlled through the first two waves of Covid-19 in 2019 and early 2020; ii) is a developing country - undeveloped waste treatment infrastructure can be applied in other countries around the world. Specifically, in contrast to complicated waste treatment processes - difficult to apply to other countries due to budget constraints and modern equipment - the application model in Vietnam can be applied. in other countries around the world. The process of classification and treatment of medical waste during the Covid-19 epidemic was signed by the Ministry of Health in 2019 [15]. The steps are shown in Fig. 2. Specifically, Fig. 2 shows five sources of medical waste classification and five treatment steps. Sources of medical waste include treatment place (i.e., hospital, military barracks), testing, vaccination, and the personal place under quarantine (eg,household, apartment). For medical waste classification, the first three steps are carried out at healthcare centers (i.e., classification, separation, and collection) where all hazardous waste is sent to factories for treatment disposal (i.e., destruction) including transportation and treatment.

**Fig. 2.** The Covide-19 pandemic-related medical waste treatment sample in Vietnam

Figure 1 shows the steps of traditional garbage classification and treatment. Step 1 presents the waste collection process in departments in medical facilities (e.g., hospitals). These types of waste include medical equipment and supply. Step 2 presents the process of self-segmentation and reuse in the medical environment. At the end of this process, products belonging to the recycling medical product group are reused (step 3) while medical waste is disposed of (step 4). All medical waste is sent to the waste treatment area (step 5). Here, depending on the treatment requirements, the medical waste is classified into the corresponding treatment processes (i.e., recycling plant and sorting factory - step 6).

The risks in the traditional medical waste treatment process have been summarized and presented in the Introduction and Related work sections. However, in order to improve people's sense of self-segregation and treatment of medical waste, we propose a model that combines blockchain, smart contract, and NFT technologies to create certificates in classification at medical centers thereby identifying compliance/violation.

**Fig. 3.** Model of treatment and classification of medical waste based on blockchain technology, smart contract and NFT

Figure 3 presents nine steps of classification and treatment of medical waste using blockchain technology, smart contract and NFT. In particular, doctors, nurses, and other medical center staff (eg, clinics, hospitals) must confirm constraints on waste segregation at the workplace (step 1). These requirements are presented in the form of guidelines and commitments (i.e., specify compliance/violations when sorting medical waste). Here, we only classify into two main groups including: recyclable waste at medical centers and hazardous waste. Step 2 presents healthcare workers with self-segregation of waste as required in step 1. Hazardous items must be isolated from the treatment and medical care area (i.e., waiting for the transporter to arrive at the centers). waste treatment center). In step 4, hospitals provide guidance documents to cleaning staff regarding the assessment of compliance/violation behaviors by doctors and nurses. Step 5, present the cleaning staff's monitoring process for all types of waste after being sorted (i.e., check the process for hazardous waste). Step 6 shows the cleaning staff to check the types of recyclable waste. These assertions are sent to the respective functions (i.e., name the functions in the smart contract) - step 7. Step 8 generates NFTs corresponding to compliance/violations based on cleaning's evaluation process. staff for individuals/organizations at medical centers. The entire process is stored in the distributed ledger (step 9).

## **4 Implementation**

Our reality model focuses on two main purposes: i) data manipulation (i.e., medical waste) - initialization, query and update - on blockchain platform and ii) creation of NFTs for each user's (i.e., individual/organization) reward and violation behavior based on their behavior in waste sorting/disposal.

#### **4.1 Initialize Data/NFT**

**Fig. 4.** Initialize data/NFT

Figure 4 shows the steps to initialize medical waste data. These types of waste include medical equipment (i.e., expired/damaged) or medical supplies (eg, masks, PPE, injections). These types of waste are required to be classified into different categories (i.e., discard, reuse) depending on their level of toxicity. Then descriptions of the type of waste are added to each specific garbage bag. Each trash bag has a unique address to separate them with the type of waste. In addition, information about the sorter as well as in which department, time, and location is also added to the metadata of the trash bag. As for the storage process, services support concurrent storage (i.e., distributed processing as a peer-to-peer network) on a distributed ledger - supporting more than one user for concurrent storage to reduce system latency. The medical waste data is organized as follows:

```
medicalWasteDataObject = {
"medicalWasteID": medicalWasteID,
"staffID": staffID,
"type": type of waste,
"apartmentID": apartmentID,
"quantity": quantiy,
"unit": unit,
"packageID": packageID,
"time": time,
"location": location,
"state": null,
"reUse": Null};
```
Specifically, in addition to the information to extract the content (i.e., place of origin, weight, type of waste, etc.), we also store information related to the status of the garbage bags at the hospital (ie. , "state" and "reUse" - defaults to Null). Specifically, "state" changes to 1 if the corresponding garbage bag has been shipped out of the medical center (i.e., for the type of waste to be treated); value 0 - pending. Meanwhile, "reUse" presents the value 1 when the type of waste (i.e., medical device) is reused (i.e., value 0 - pending). Non-hazardous wastes (i.e., non-toxic to the environment and user's health). After the waste sorting phase, the cleaning staff will check if they are in accordance with the process and wait for validation before synchronizing on the chain (i.e., temporarily stored on the data warehouse). Then the pre-designed constraints in the Smart Contract are called through the API (i.e., name of function) to sync them up the chain. This role of accreditation is extremely important because they directly affect the waste treatment process, as well as the premise for rewarding or sanctioning individuals and organizations.

#### **4.2 Data Query**

Similar to the data initialization steps, the data query process also supports many simultaneous participants in the system for access (i.e., distributed model). Support services receive requests from Cleaning staff or nurses/doctors to access data. Depending on the query object we have different access purposes. Specifically, Cleaning staff queries with the purpose of checking the classification process or transferring hazardous medical waste to the shipping company. In contrast, healthcare professionals can query data to find reusable medical tools (i.e., out of stock). Figure 5 shows the steps to query medical waste data. These requests are sent as requests (i.e., pre-designed services as API calls) from the user to the smart contracts available in the system (i.e., name of function) before retrieving the data. from the distributed ledger. All retrieval requests are also saved as query history for each individual or organization. In case the corresponding information is not found (eg, wrong ID), the system will send a message not found results. For the NFT query process, all support services are provided as APIs.

## **4.3 Data Updated**

The data update routine is invoked only after verifying that the data exists on the thread (i.e., after executing the corresponding data query procedure). In this section, we assume that the search data exists on the string. Where none exists, the system sends the same message to the user (see Sect. 4.2 for details). Similar to the two processes of query and data initialization, we support update services in the form of APIs to receive requests from users before passing them to smart contracts (i.e., name of function) for processing. The purpose of this process is to update the time and location of the garbage bags during transportation and handling of medical waste. Thereby, the administrator can trace the status of medical waste treatment/transportation from medical centers to waste treatment companies. Figure 6 shows the procedure for updating medical waste data. For NFTs (i.e., available) the update process includes only moving from the owner's address to the new (i.e., new owner). If any information is updated on an existing NFT, it will be stored as a new NFT (see Sect. 4.1 for details).

## **5 Evaluation**

Since the proposed model rewards/penalizes for compliance/violation of the medical waste classification process, we implement the recommendation model on EVM-enabled blockchain platforms instead of mining platforms under the Hyperledger eco-system. In addition, assessments based on system responsiveness (i.e., number of requests responded successfully/failed, system latency - min, max, average) have been evaluated by us in the previous research paper. Therefore, in this paper we determine the suitable platform for our proposed model. Specifically, we install a recommendation system on four popular blockchain platforms today, supporting Ethereum Virtual Machine (EVM), including Binance Smart Chain (BNB Smart Chain)<sup>2</sup>; Polygon<sup>3</sup>; Fantom<sup>4</sup>; and Celo<sup>5</sup>. Our implementations on these four platforms are also shared as a contribution to the article to collect transaction fees corresponding to the four platforms' supporting coins<sup>6</sup>, ie, BNB<sup>7</sup>; MATIC<sup>8</sup>; FTM<sup>9</sup>; and CELO<sup>10</sup>. Our implementations to evaluate the execution cost of smart contracts (i.e., designed based on Solidity language) run on testnet environments of four platforms in order to choose the most cost-effective platform to deploy. Our detailed assessments focus on the cost of performing contract creation, NFT generation and NFT retrieval/transfer (i.e., updating NFT ownership address) presented in the respective subsections related to i) Transaction Fee; ii) Gas limit.

<sup>2</sup> https://github.com/bnb-chain/whitepaper/blob/master/WHITEPAPER.md.

<sup>3</sup> https://polygon.technology/lightpaper-polygon.pdf.

<sup>4</sup> https://whitepaper.io/document/438/fantom-whitepaper.

<sup>5</sup> https://celo.org/papers/whitepaper.

<sup>6</sup> Implementation of theme models our release at Nov-24-2022 04:19:05 AM +UTC.

<sup>7</sup> https://testnet.bscscan.com/address/0x94d93a5606bd3ac9ae8b80e334dfec74d0075e ce.

<sup>8</sup> https://mumbai.polygonscan.com/address/0x48493a3bb4e7cb42269062957bd541d5 2afc0d7a.

<sup>9</sup> https://testnet.ftmscan.com/address/0x48493a3bb4e7cb42269062957bd541d52afc0 d7a.

<sup>10</sup> https://explorer.celo.org/alfajores/address/0x48493A3bB4E7cB42269062957Bd541 D52aFc0d7A/transactions.


**Table 1.** Transaction fee

#### **5.1 Transaction Fee**

Table 1 shows the cost of creating contracts for the four platforms. It is easy to see that the highest transaction fee of the three requirements is contract creation for all four platforms. In which, the cost of BNB Smart Chain is the highest with the highest cost when creating a contract is 0.02731376 BNB (\$8.41); whereas, the lowest cost recorded by the Fantom platform with the highest cost for contract initiation is less than 0.009577666 FTM (\$0.001840). Meanwhile, the cost to enforce Celo's contract initiation requirement is lower than Polygon's with only \$0.004 compared to \$0.01. For the remaining two requirements (Create NFT and Transfer NFT), we note that the cost of implementing them for all three platforms, Polygon, Celo, and Fantom is very low (i.e., negligible) given the cost. trades close to \$0.00. However, this cost is still very high when deployed on BNB Smart Chain with 0.00109162 BNB (\$0.34) and 0.00057003 BNB (\$0.18) for Create NFT and Transfer NFT, respectively.

#### **5.2 Gas Limit**


**Table 2.** Gas limit

Table 2 shows the gas limit for each transaction. Our observations show that the gas limits of the three platforms (i.e., BNB, Polygon, and Fantom) are roughly equivalent - where Polygon and Fantom are similar in all three transactions. The remaining platform (i.e., Celo) has the highest gas limit with 3,548,968; 142,040; and 85,673 for all three transaction types.

## **5.3 Discussion**

According to our observation, the transaction value depends on the market capitalization of the respective coin. The total market capitalization of the 4 platforms used in our review (i.e., BNB (Binance Smart Chain); MATIC (Polygon); FTM (Fantom); and CELO (Celo)) are \$50,959,673,206; \$7,652,386,190; \$486,510,485; and \$244,775,762.<sup>11</sup> This directly affects the coin value of that platform - although the number of coins issued at the time of system implementation also plays a huge role. The coin's value is conventionally based on the number of coins issued and the total market capitalization with a value of \$314.98; \$0.863099; \$0.1909; and \$0.528049 for BNB, MATIC, FTM, and CELO respectively. Based on the measurements and analysis, we have concluded that the proposed model deployed on Faltom brings many benefits related to system operating costs. In particular, generating and receiving NFTs has an almost zero (i.e., negligible) fee. Also, the cost of creating contracts with transaction execution value is also meager (i.e., less than \$0.002).

In future work, we proceed to implement more complex methods/algorithms (i.e., encryption and decryption) as well as more complex data structures to observe the costs for the respective transactions. Deploying the proposed model in a real environment is also a possible approach (i.e., implementing the recommendation system on the FTM mainnet). Currently, we have not considered issues related to the privacy policy of users (i.e., access control [18], dynamic policy [19]) - a possible approach would be implemented in upcoming research activities.

## **6 Conclusion**

Our work proposes a reward/punishment model for individual/organizational medical waste sorting behavior. Our proposal is based on blockchain technology, smart contracts (i.e., transparency, distributed storage, and processing), and NFT (i.e., immutable) to build waste separation and treatment processes. medical. We have implemented proof-of-concept on the Ethereum platform and smart contracts (i.e., contract creation, NFT create, NFT transfer) based on the Solidity language. We also tested smart contracts on four EVM-enabled platforms (i.e., BNB, MATIC, FTM, and CELO) to find the right platform (i.e., the lowest smart contract execution cost). Our analysis in the evaluation section demonstrated that implementations on Fantom offer more benefits than the other ones.

## **References**


<sup>11</sup> Our observation time is 12:00PM - 11/26/2022.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Short Contributions: Medical Systems and E-Health Solutions**

# **Quantitative Pressure Measurement in Areas at High Risk of Pressure Ulcers in Different Positions: Pilot Study**

Junhee Lee1,2 and Joon Shik Yoon2,3(B)

<sup>1</sup> Department of Physical Medicine and Rehabilitation, Hallym University Dongtan Sacred Heart Hospital, 7, Keunjaebong-gil, Hwaseong-si, Gyeonggi-do 18450, Republic of Korea <sup>2</sup> Department of Physical Medicine and Rehabilitation, Korea University College of Medicine, 73, Goryeodae-ro, Seongbuk-gu, Seoul 02841, Republic of Korea

rehab46@korea.ac.kr

<sup>3</sup> Department of Physical Medicine and Rehabilitation, Korea University Guro Hospital, 148 Gurodong-ro, Guro-gu, Seoul 08308, Republic of Korea

**Abstract.** This study aimed to quantitatively analyze pressure parameters in different high-risk areas depending on the position. We reviewed the clinical records of trials of 20 healthy adults on a multi-actuated bed accompanied with pressure sensor mat. We collected average, maximal, minimal pressure, and area in the supine and bilateral side-tilt positions. Also, we analyzed the difference between each at-risk area, depending on positions. In the supine position, pressure parameters of the head, shoulders, sacrum, coccyx, and heels showed significant differences, except between the right and left heels. In the right side-tilt position, all pressure measurements of the ear, shoulder, elbow, hip, knee, and lateral ankle were significantly different. In the left side-tilt position, most of the pressure parameters of the ear, shoulder, elbow, hip, knee, and lateral ankle were significantly different, except between the elbow and ankle. We found that frequent position changing is more important than achieving optimum positioning.

**Keywords:** Patient care · Patient positioning · Pressure ulcer

## **1 Introduction**

Pressure ulcers are skin and/or underlying tissue lesions caused by the limitation of blood circulation owing to the combination of increased pressure, friction, and sheer force on specific parts of the body [1, 2]. These pressure injuries are closely related to loss of quality of life, impairment in mobility, and increased mortality, both directly and indirectly [3].

Repositioning patients every 2 h (q2hr) to prevent continuous high pressure on a specific body part is the mainstream of pressure sore care, and it has been considered a guideline for general patient care [4, 5]. However, previous studies have shown that standard turning does not sufficiently relieve the high skin-bed pressure on peri-sacral areas, such as the sacrum, coccyx, and ischial tuberosity, even in non-disabled adult subjects. This may explain why pressure ulcers still develop despite preventive methods, including regular patient repositioning [6]. In addition, recording repositioning has not been standardized or automated, and remains manual. This causes a lack of optimal repositioning and incomplete documentation of a patient's position.

Several studies have focused on a preventive solution for pressure ulcers with realtime position monitoring systems. In particular, devices that use sensors to detect pressure are considered to be one of the most effective methods because they minimize care-giver's effort [7]. It is also important because there has been issues about the pressure monitoring while sleeping. Furthermore, these devices used to modify care plans while preventing the development of pressure ulcers [8]. Comprehensive understanding of the monitoring information gathered from devices is necessary to increase optimal patient repositioning by clinical staff. An integrated analysis is needed between pressure monitoring information and known positions to prevent pressure ulcers.

There is a lack of research evaluating pressure measurements in various positions considering the risk areas of each position. This study aimed to clarify the key clinical implications for preventing pressure ulcers by quantitatively analyzing acquired pressure parameters in high-risk areas depending on position.

## **2 Materials and Methods**

#### **2.1 Study Design**

In this study, we retrospectively analyzed the clinical records of trials conducted on a multi-actuated bed with an applied pressure sensor. From August to September 2022, the clinical records of 20 healthy individuals were obtained. The inclusion criteria were: 1) over 18 years of age, 2) no underlying disease, and 3) no limitations on activity. The reviewed data were recorded in a structured form. Additionally, we collected demographic data such as age, gender, height, weight, and body mass index (BMI).

This study was approved by the Institutional Review Board of Korea University Guro Hospital (2022GR0463).

#### **2.2 Pressure Sensing with Multi-actuated Bed**

A calibrated XSENSOR IX500:256.256.22 (XSENSOR Technology Corporation, Calgary, Canada) pressure mapping system with X3 software (v8) was used. XSENSOR is considered the gold standard for pressure mapping and has been used in several studies [5, 9, 10]. The XSENSOR mat consists of 65,536 sensing points, has a total area of 63.5 cm × 53.3 cm and a sensing area of 29.5 cm × 29.5 cm. The manufacturer's specifications state that the pressure mat showed high durability in subsurface testing, an accuracy rate of ±10% of the calibrated values, a sampling frame rate of 6.2 frames per second, and a spatial resolution of 1.15 mm. The device was calibrated to measure pressure from 0 to 200 mmHg. IP readings were transferred from the XSENSOR mat to a handheld monitor.

A modern automatic bed setting (Jeong In ENS Corporation, Republic of Korea) was used in conjunction with the XSENSOR mat. Repositioning to the lateral tilt (i.e., turning) position and supine position are the most routine methods for patients' position change [11]. The multi-actuated bed supported three standard positions for the prevention of pressure ulcers: right-side tilt, 30°; left-side tilt, 30°; and supine [12].

**Fig. 1.** Illustration of three different position with measurement. (a) is side-tilt 30° position, (b) is supine position and (c) is example of pressure sensing measurements.

In particular, the side-tilt and supine positions have different high-risk areas for the development of pressure ulcers. In the supine position, pressure ulcers frequently occur at the head (occiput), bilateral shoulders (scapula), sacrum, coccyx, and bilateral heels. In the lateral tilt position, pressure ulcers easily develop in the ear, shoulder (upper humerus), elbow outer side (lateral epicondyle), hip (greater trochanter), knee outer side (fibular head), and lateral ankle outer side (lateral malleolus). [13, 14] We considered these differences regarding positioning on the evaluations and selected the most highly recorded areas in each risk area.

The sensor mat was placed above the bed, and the pressure was continuously monitored for 10 min in three positions. In total, 30 min was needed for each evaluation [5]. Recorded data included average pressure (N/cm2), peak pressure (N/cm2), minimal pressure (N/cm2), and area (cm2). The settings are illustrated in Fig. 1.

#### **2.3 Statistical Analysis**

Using theWilcoxon signed-rank test, the quantitative difference between pressure parameters depending on the risk areas of each position was analyzed. For the descriptive analysis, the median and inter-quarter range of measurements were suggested for metric evaluations. SPSS version 26.0 software (SPSS Inc., Chicago, IL, USA) was used for all analyses, with the statistical significance level set at p *<* 0.05.

#### **3 Results**

Ten patients were male and 10 were female. The median age of participants was 28 years (25.25–31.5); median height was 167.0 cm (162.0–172.75); median weight was 63.5 kg (50.25–70.0); and median BMI was 21.12 kg/m2 (18.75–23.87).

#### **3.1 Pressure Measurements at Risk Area of Supine Position**

The pressure parameters in the supine position in high-risk areas are listed in Table 1. All four pressure measurements of the head and bilateral shoulders, right heel; coccyx and left heel, right heel were different. Pressure measurements of the head and left heel were different, except for average pressure. The pressure measurements of the sacrum and right heel were different, except for maximal pressure. Pressure measurements of the left shoulder and left heel; right shoulder and left heel; sacrum and left heel were different, except for minimal pressure. Pressure measurements of the left shoulder and sacrum, coccyx; right shoulder and sacrum, coccyx; sacrum and coccyx were different, except in the area. The average and minimal pressures of the left and right shoulders were different. The maximal pressure and areas of the head and coccyx; left shoulder and right heel were different. The areas of the right shoulder and right heels were different. There was no statistically significant differences between the right and left heels.


**Table 1.** At-risk area of pressure ulcer in supine position.

Abbreviations: P, pressure; Rt., right; Lt., left

#### **3.2 Pressure Measurements at Risk Area of Right Side-Tilt Position**

The pressure measurements in the side-tilt position to the right in the high-risk areas are presented in Table 2. All four pressure measurements of the ear and shoulder, elbow, knee, ankle; shoulder and knee, ankle; elbow and hip; hip and knee, ankle were different. The pressure measurements of the ear and hip were different, except for maximal pressure. The pressure measurements of the shoulder and elbow were different, except for minimal pressure. The pressure measurements of the shoulder and hip were different, except for the area. The minimal pressures and areas of the knee and ankle were different. The minimal pressures at the elbow and ankle were different. The elbow and knee areas were different.

#### **3.3 Pressure Measurements at Risk Area of Left Side-Tilt Position**

The pressure measurements in the side-tilt position to the left in high-risk areas are also presented in Table 2. All four pressure measurements of the shoulder and knee; elbow and hip; hip and knee, ankle were different. The pressure measurements of the ear and hip were different, except for maximal pressure. The pressure measurements of the ear and shoulder, elbow, knee, ankle; shoulder and elbow; knee and ankle were different, except for minimal pressure. Pressure measurements of the shoulder and hip were different, except for the area. The maximal pressure and area of the shoulder and ankle were different. The elbow and knee areas were different. There was no statistically difference between the elbow and the ankle.


**Table 2.** At-risk area of pressure ulcer in side-tilt position.

(*continued*)


**Table 2.** (*continued*)

Abbreviations: P, Pressure; Rt., Right; Lt., Left; Lat., Lateral

#### **3.4 Pressure Change Depending on Position**

To detect the pressure change depending on positions, we compared the areas with anatomical rotation. We matched the followings: head (occiput) on supine position and bilateral ears on side-tilt position, bilateral shoulder on supine position and bilateral shoulders on side-tilt position, sacrum on supine position and bilateral hip on side-tilt position, bilateral heel on supine position and bilateral lateral ankle on side-tilt position.

Compared to the pressure on head (occiput) in supine position, average pressure was stationary on right ear and decreased on left ear. Also, maximal and minimal pressure were decreased on right ear and increased on left ear, respectively. In comparison of the pressure on bilateral shoulder in supine position and side-tilt position, average and maximal pressure were increased on right side-tilt shoulder; and minimal pressure was decreased on right side-tilt shoulder. In addition, maximal pressure was increased on left side-tilt shoulder; average pressure and minimal pressure were decreased on left side-tilt shoulder. Compared to the pressure on sacrum in supine position, all pressure parameters were increased on right hip. Also, average and minimal pressure were increased and maximal pressure was decreased on left hip. In comparison between the pressure on bilateral heal in supine position and bilateral lateral ankle in side-tilt position, all pressure parameters were decreased on bilateral side-tilt positions.

#### **4 Discussion**

Position change for reducing pressure and preventing ulcers have been the standard methods of patient care [1, 12, 15, 16]. However, studies showed failure in reducing the incidence of pressure ulcers [15, 18–20]. Although maintaining the skin contact pressure below 32 mmHg is expected to decrease the possibility of developing pressure ulcers, the effectiveness of repositioning patients for pressure sore prevention remains unclear [6].

Previous studies conducted with non-disabled population and pressure mapping system [5, 6]. They emphasized the high-risk of pressure area as peri-sacral area and greater trochanters. Our research was the first study which quantitively tracked the pressure parameters with position change and describe the clear pressure difference between each high-risk pressure ulcer areas. Furthermore, we clarified the importance of frequent position change in patient care for preventing pressure ulcers.

Depending on the position, various findings were observed with quantitative measurements. In the supine position, the average, maximal, and minimal pressure measurements were higher on the head, sacrum, and coccyx than on the shoulder and heel. This might be due to the bilateral distribution of body weight in the shoulder and heel. In the right and left side-tilt positions, we compared the right side and the left side of the body, respectively. The average, maximal, and minimal pressure measurements were higher on the bilateral ear than on the bilateral shoulder and elbow on the upper side of the body. In addition, these three pressure parameters appeared to be higher on the bilateral hip than on the bilateral knee and lateral ankle on the lower side of the body.

Although there were minor differences in the composition of pressure measurements with statistical significance, we clarified the clear differences between pressure values in each risk area. In the supine position, most pressure parameters of the head, shoulder, sacrum, coccyx, and both heels showed significant differences, except between the right and left heels. In addition, in the right side-tilt position, all pressure measurements of the ear, shoulder, elbow, hip, knee, and lateral ankle were significantly different. In the left side-tilt position, most of the pressure parameters of the ear, shoulder, elbow, hip, knee, and lateral ankle were significantly different, except between the elbow and ankle. The difference between right-side tilt and left-side tilt might be explained by the small amount of recruited data, and it would be expected to be covered by larger scale analysis.

In comparing descriptive measurements with positional rotation, the changes in pressure parameters at each high-risk area did not show a consistent pattern. This result suggest that it is more important to change the posture frequently than to make an ideal posture in patient care for preventing pressure ulcers.

This study has some limitations. First, the pressure measurement by the sensor did not directly express the exact pressure loading on the tissue [20]. However, pressure sensor tracking remains one of the most popular methods for pressure monitoring. The clear presentation of pressure change in real time reflects a good representation of the pressure on the surface of the skin. Therefore, it allows us to intuitively recognize the increasing and decreasing pressures. Second, we only used the measured records of participants with a single brand of modern ward bed and sensor monitoring interface. However, we expected that the results would be similar, regardless of the type of hardware system. The pressure measurements at various positions are expected to be different for different beds/sensor mats; however, there is no doubt that the general trends should be reflected in the results. Finally, we studied the data of healthy adults, but not of patients. In patients with a high risk of developing pressure ulcers, the condition of the soft tissue in the affected area is clearly different from that of the general population. Therefore, there are many variables to consider regarding the risk factors for pressure ulcers, such as nutritional status, capillary blood flow, friction force and shear force. In this point of view, we considered that healthy participants were more likely to show a clear tendency of pressure measurements in each high-risk area at different positions and positional change. Further study of patients with classified impairments is necessary, because these results with normal population may appear differently depending on the patient's variables.

## **5 Conclusion**

This study quantitatively examined pressure distribution depending on position and found that the risk areas for pressure ulcers in different positions have different pressure distributions in a healthy population. These findings suggest clinical implications for positioning depending on patient impairment. Further studies with a larger patient population are needed to evaluate the actual risk of developing pressure ulcers in patients and to establish positioning methods to achieve optimum status to decrease pressure ulcer risk.

**Acknowledgments.** This research was funded by the Translational Research Program for Care Robots funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HK20C0018).

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **PNRG: Knowledge Graph-Driven Methodology for Personalized Nutritional Recommendation Generation**

Aminah Bilal Lodhi1, Muhammad Abdullah Bilal2, Hafiz Syed Muhammad Bilal1(B) , Kifayat Ullah Khan2, Fahad Ahmed Satti1, Shah Khalid1, and Sungyoung Lee3(B)

<sup>1</sup> National University of Sciences and Technology (NUST), H-12 Campus, Islamabad, Pakistan {alodhi.mscs19seecs,bilal.ali,fahad.satti, Shah.khalid}@seecs.edu.pk <sup>2</sup> FAST National University of Computer and Emerging Sciences, Islamabad Campus, Islamabad, Pakistan {i190616,kifayat.alizai}@nu.edu.pk <sup>3</sup> Kyung Hee University Global Campus, Yongin, South Korea sylee@oslab.khu.ac.kr

**Abstract.** Chronic Diseases are a prevalent problem that affects millions of people worldwide. It is a prevalent health condition that requires careful diet and medication management and preventing chronic diseases. Traditional approaches to nutritional recommendation generation often rely on generic guidelines and populationbased data, which may not account for individual dietary needs and preferences variations. In this paper, we propose a knowledge graph driven methodology for generating highly personalized nutritional recommendations that leverage the power of knowledge graphs to integrate and analyze complex data about an individual's health, lifestyle, and dietary habits. Our methodology employs a multi-step process that includes data collection and curation, knowledge graph construction, and personalized recommendation generation. We illustrate the effectiveness of our approach through a case study in which we generate personalized nutritional recommendations for a sample individual based on their specific health and dietary goals.

**Keywords:** Chronic Kidney Disease · Knowledge Driven Approach · Personalized Nutritional Recommendation System

## **1 Introduction**

Chronic kidney disease (CKD) is a long-lasting condition that can lead to a range of complications, including cardiovascular disease, decreased immune response, anemia, fluid retention and bone disease. Chronic kidney disease accounts for a very high risk for cardiovascular diseases [1] among other diseases. The high probability of developing conditions like hypertension can be countered to some extent using lifestyle changes such as increased physical activity, better dietary habits, and regular monitoring of medical reports related to these conditions. On the other hand, medical conditions such as hypertension and diabetes can be successfully managed by changes in lifestyle such as dietary restrictions. These lifestyle changes include recommendations such as reduced salt intake as well as an increase in the consumption of fruits and vegetables while decreasing the consumption of processed food items [3].

## **1.1 Effect of Diet on CKD Patients**

Chronic kidney disease (CKD) is a condition in which the kidneys are not working as effectively as they should to filter waste products from the blood. Diet plays a critical role in managing CKD. Some of the effects of diet on CKD patients which we have taken Center of Disease Control and prevention [4] include sodium, protein, potassium, phosphorus, vitamins (such as vitamin D [5], iron [6], and calcium [7]), minerals, and fluids.

## **1.2 Knowledge Graph-Based Application for Nutritional Recommendations**

In this paper, we propose a knowledge graph-driven methodology for generating highly personalized nutritional recommendations that leverage the power of knowledge graphs to integrate and analyze complex data about an individual's health, lifestyle, and dietary habits.

The development of applications that provide personalized health management and nutritional guidance [8] has the potential to improve the quality of life for CKD patients by facilitating self-management and providing access to specialized knowledge. In this paper, we present the development of an application that incorporates a range of features to support personalized health management for CKD patients.

The app also includes medication reminders, nutritional guidance, drug interactions, CKD stage Calculation, and a log of medical tests and Food. This research paper aims to evaluate the effectiveness of our healthcare app in managing CKD.

The application incorporates the following features:


## **2 Related Work**

Previously work has been done regarding the implementation of tech-based solutions for chronic kidney disease most notably of which is the work done by Chatterjee et. Al [9] in his work they have taken a similar approach where lifestyle data can be logged into an app and because of it, a qualitative analysis of a patient's lifestyle can be performed. Although the methodology proposed by this paper includes lifestyle data, it builds upon that by adding dietary data as well as medical test data performed by hospitals as well. Due to the addition of these parameters, the purpose of this paper is not only to help patients better understand the effect of their lifestyle choices but also as an analysis tool for medical professionals.

Another study that targeted the same domain created a web-based platform for helping women living in rural areas with obesity [10]. They used a web-based system to provide guidance tailored specifically to people living in hard-to-reach places. One of the limitations of that paper was that only participants that were interested in physical activity and nutrition were only interested in using the system. We have tried to solve the problem by taking proper input from the doctors as well and trying to incorporate the system into the treatment regimen of the patients.

Another similar study [11] done related to this field provided a similar system to ours regarding the fact that the platform proposed by them had the ability to keep track of medical reports, physical activity, and dietary data and the data could be shared with medical professionals, but the platform was only targeted to meet the needs of pregnant women having diabetes.

## **3 Methodology**

## **3.1 Overview**

Using an algorithmic approach, we developed a tool that generates personalized lifestyle recommendations for CKD patients based on their individual test results. The web application was developed using a combination of open-source tools and programming languages, the architecture includes NodeJS, Express, MongoDB, React, and Tailwind CSS, D3 library and Knowledge graphs as shown in Fig. 1. The user data has been stored in MongoDB while the knowledge base is used to represent the interactions between the drugs, food etc. successfully divide the application specific data and the domain specific data. Along with this, patients will be able to log their non-invasive health data such as blood pressure, heart rate, and blood sugar levels. This would allow patients to monitor their health condition using a graphical user interface. The patients would also be able to log in to their medications to get drug interaction information about the medicine that they are taking.

## **3.2 Server Side**

This system's aim would be to ultimately provide recommendations regarding their health data and allow them a safe mechanism for self-monitoring that can also help the doctor regarding their medical decisions. We designed the prototype of this system using the agile methodology and as per the recommendations provided by it, we divided the development of the prototype into five parts: requirement gathering, requirement design, development iterations, testing, and finally feedback.

## **3.3 Data Storage**

First, we collect data from various sources, such as medical records, wearable devices, and food intake logs, to create a profile of the individual's health, lifestyle, and dietary habits. Finally, we generate personalized nutritional recommendations for the individual based on their specific health and dietary goals, using a combination of rule-based and data-driven approaches.

**Fig. 1.** Application Architecture

**Fig. 2.** Ontology Design Methodology

We have constructed a knowledge graph that represents the relationships between different entities in the data, such as foods, nutrients, health conditions, and lifestyle factors. To design the ontology, we used OD101 as the knowledge base would be domain specific as shown in Fig. 2. We have used an RDF-based knowledge base that uses RDF, RDFS, and OWL to represent concepts in an interconnected form. Using an RDF triple store as shown in Fig. 3 allows us to publish the data in 4-star format according to Tim Berner Lee's linked data criteria. This would also allow us to use the reasoning engines like the HermiT or Pellet reasoner which allows us to make decisions based on knowledge instead of data which helps us explain the reasoning behind the recommendations that are made. These reasoning engines also provide the advantage of finding associations in data that are not perceivable by the layman.

Knowledge bases also provide a better and more natural format for drug interactions. In our application, this is best seen by the fact that drug interactions naturally have a symmetric property as shown in Fig. 4. That knowledge bases can very easily present while other data storage systems fail to do so. Domain and range restrictions also provide a check for drug interaction data. The symmetric and reflexive axioms also provide a more natural representation of the knowledge represented in the knowledge base as all entities in a drug or food interaction inherently have a symmetric and reflexive relationship.

The methodology followed by the recommendation system follows the following steps:


**Fig. 3.** RDF Architecture

	- interacts\_with: this object property handles the interactions that take place between drugs and food items as well as drugs with other drugs. This object property is symmetric and hence has the advantage that the inference engine would complete any missing links in the knowledge base.

**Fig. 4.** Drugs/Food Interaction

#### **3.4 Domain Specific Features**

The most used method for calculating the stage of CKD is based on the estimated glomerular filtration rate (eGFR), which is calculated using the serum creatinine level, age, gender, and race of the patient. The eGFR provides an estimate of the rate at which the kidneys are filtering waste from the blood. The stages of CKD are classified based on the eGFR and the presence of kidney damage.

**Fig. 5.** Relationship between drugs and foods

## **4 Evaluation Methodology**

We had 10 volunteers of different genders, heights and weights and age groups, and through CKD stage calculation we have grouped them into 5 groups as shown in Table 1.


**Table 1.** Details of Volunteers

We recommended nutrients based on their medical tests and recorded their noninvasive health data e.g., blood pressure and heartbeat for 6 weeks. Over this specific period, with proper recommendations and logging of their health data and medical records, they were able to control their disease in a more efficient way, as the feedback taken from the survey, we conducted from the volunteers as shown in Table 2.


**Table 2.** Survey conducted of volunteers for reusability of application.

## **5 Result**

The development of a prototype done through the agile methodology allowed us to regularly obtain feedback from patients as well as health professionals. The product contained proper visualizations of both types of data: numerical and categorical. This allows the patients as well as health professionals to quantitively assess the lifestyle of a patient during a normal day. The prototype can also provide the possibility for patients to set goals for their day which includes exercise time etc.

## **6 Conclusion**

By leveraging the power of knowledge graphs to integrate and analyze complex data, our approach can generate highly personalized recommendations that are both effective and sustainable. Our research findings suggest that personalized recommendations can significantly improve patient outcomes and reduce the progression of CKD. By using the app, patients can track their progress, receive guidance, and support, and connect with their healthcare provider. We believe that our healthcare app has the potential to revolutionize CKD management by providing patients with a comprehensive and personalized approach to their care. We encourage CKD patients to download and use the app to take control of their health and manage their condition effectively. With the right tools and support, CKD patients can lead fulfilling and healthy lives. Personalized lifestyle recommendations based on individual test results can improve the outcomes of CKD patients.

Further research is needed to evaluate the effectiveness of this approach in a clinical setting.

## **7 Future Work**

RDF being a non-proprietary format, also has the advantage that by providing access through the SPARQL endpoint, the knowledge base can potentially act as an open-source repository. This would allow a centralized repository containing all drug data as well as data related to their interactions with other drugs, food items and medical conditions.

Although our current ontology has been designed to only accommodate use cases for drug-drug and drug-food interactions, the ontology can be expanded to accommodate interactions between drugs and medical conditions as well as drug and genetic variations. This presents the opportunity to increase the scope of the application in the future from assisting doctors with providing them with insights about the patient's lifestyle and providing recommendations related to the food and drug recommendations, to the system providing personalized recommendations based on the patient's invasive as well as non-invasive medical data such as genetic data. An example of a drug interaction with genetic properties individuals is where the optimal dosage of Warfarin, an anticoagulant that acts by reducing the activity of vitamin K, is dependent on the patients CYP2C9 and VKORC1 genotype [12]. The same principle applies with medical conditions affected by drugs. Acetaminophen, commonly known as Tylenol if taken in high doses can have adverse reactions in kidney disease patients [2].

**Acknowledgments.** This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the Grand Information Technology Research Center support program(IITP-2022– 2020-0–01489) supervised by the IITP(Institute for Information & communications Technology Planning & Evaluation) and by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (IITP-2022–0-00078, Explainable Logical Reasoning for Medical Knowledge Generation), (IITP-2017–0-00655, Lean UX core technology and platform for any digital artifacts UX evaluation).

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **For People with Disabilities Who Need Medical Care "Smart Elephant Whole Body Exercise Machine" Development and Clinical Translational Research**

Seungwan Yang1 and Jongbae Kim2(B)

<sup>1</sup> Department of Occupational Thearpy, The Graduate School, Yonsei University, Seoul, South Korea <sup>2</sup> Department of Occupational Thearpy, College of S/W Digital Healthcare Convergence, Yonsei University, Wonju, South Korea

jongbae@yonsei.ac.kr

**Abstract.** It is difficult for inpatient rehabilitation patients to continue to perform rehabilitation exercises in the community after leaving the hospital. This is because various exercise programs, which are not medically proven, do not reflect the specificity of the individual and are performed collectively due to administrative and financial convenience. The purpose of this study is to evaluate and compare the effects of exercise programs using the Smart Elephant total body exercise device and walking on mental and physical outcomes with real-time monitoring to develop a customized rehabilitation exercise program optimized for people with disabilities. To conduct this study, five non-disabled people living in the community were selected to participate in the exercise programs of Intervention A (walking), Intervention B (walking and cycling), and Intervention C (cycling) for 9 weeks to determine the effects on physical function measures, psychosocial, mental and quality of life health outcomes, participants' feedback and satisfaction surveys, and changes in Electromyography (EMG), Electrodermal Activity (EDA), Temperature (TEMP), and oxygen saturation (SpO2) during the intervention. It is believed that it can be used as a basis for customized rehabilitation exercise that provides a validated rehabilitation exercise service model for people with disabilities in the community.

**Keywords:** Whole Body Exercise Machine · person with Disability health · exercise program

## **1 Introduction**

As living standards improve, access to healthcare expands, and people become more concerned about their health, morbidity, mortality, and life expectancy are increasing (Jang & Ham, 2014). However, unlike the development of medical technology, it is difficult for rehabilitation patients who were hospitalized to continue rehabilitation exercises in the community after leaving the hospital.

Rehabilitation has been shown to improve mobility, physical functioning, and quality of life for people with disabilities suffering from illness. Mental health improvements (Blondell et al., 2014). However, while there are many companies developing equipment for older adults and people with disabilities, many of the products developed are not clinically validated, so their use is problematic and few are actually used by older adults and people with disabilities. The Elephant Total Body Exerciser is a total body exercise device that is consistently sold to neurological and musculoskeletal patients. It is used as an upper and lower limb rehabilitation device for people with disabilities in most rehabilitation hospitals and nursing homes. In addition, a study (Shin, 2016) analyzing the design satisfaction of rehabilitation medical devices using an ergonomics perspective (Shin, 2016).

The purpose of this study is to provide a validated rehabilitation exercise service model for people with disabilities in the community through a customized rehabilitation exercise study optimized for individuals by evaluating and comparing the effects of smart elephant exercise devices and walking exercise programs with real-time monitoring on mental and physical outcomes. To this end, we will confirm the effects of exercise programs on assessment tools and vital signs through a study on non-disabled people before conducting a study on disabled people.

## **2 Method**

## **2.1 Participants**

To conduct this study, we used five community-dwelling, non-disabled individuals (Table 1). According to Virzi (1992) and Nielson (2000), five users are the most efficient way to identify usability and problems of a particular product, and they reported that 80% of the problems can be found arithmetically. The inclusion criteria for the study were adults aged 18 years or older, at least 3 months since the diagnosis of any type of disease, free of cognitive and psychological disorders, able to communicate, and able to understand the purpose of the study and participate in the study. Exclusion Criteria is Subjects were excluded from the study if they had neurological or musculoskeletal disorders that made it difficult for them to use a whole body exercise device, if they were participating in other organized research projects, or if they had medical problems that made it difficult for them to participate in the study, as determined by the healthcare provider.

## **2.2 Smart Elephant Whole Body Exercise Machine**

The upper and lower extremity rehabilitation exercise machine is a combined upper and lower extremity rehabilitation device that focuses on basic and essential rehabilitation treatments such as strengthening, preventing contractures, and increasing joint range of motion. It consists of an aluminum wheel that turns a weighted wheel that is driven evenly by a chain connected to a rotating body as the crank rotates, a brake device that stops the weighted wheel, and a load adjustment device using magnets, so it is a device. By applying healthcare monitoring technology to this device, it provides feedback to each user on the number of times, time, resistance, and intensity to enhance the effectiveness of exercise.


**Table 1.** General characteristics of participants (n = 5)

#### **2.3 Study Protocol**

This study used a Single Experimental Research Design, the ABC Design, to clarify the cause and effect of the intervention in five non-disabled individuals. The exercise program consisted of Intervention A (walking), Intervention B (walking and exercise machine), and Intervention C (exercise machine) for 9 weeks, with each intervention lasting 3 weeks. Each intervention was conducted twice a week for 30 min, for a total of 6 sessions, and to determine the effectiveness of the exercise program, biosignal data was measured in real time and monitored by the user through a biosignal meter called Biosignalplux Professional. An intervention evaluation tool consisting of a physical test, fall risk assessment, psychosocial test, mental health test, and satisfaction test was conducted as a pre-assessment before intervention A, a post-assessment after intervention A, a post-assessment after intervention B, and a post-assessment after intervention C. The intervention evaluation tool was administered to the participants.

#### **2.4 Evaluation**

In this study, several papers were reviewed to evaluate the effect of exercise programs on the physical and psychosocial, and among them, representative evaluation tools used for the disabled in the physical and mental domains, which are necessary factors to improve the quality of life, were used as follows.

#### **2.4.1 Physical Function Measures**

To measure physical function measures, we used the Borg Rating of Perceived Effort (RPE). This tool is roughly equivalent to heart rate per minute as an indicator of exercise intensity (Kwon et al., 2018).

#### **2.4.2 Psychosocial, Mental and Quality of Life Health Outcomes**

To measure psychosocial, mental and quality of life health care outcomes, we used the Korean version of the WHOQOL-BREF, which was developed by the World Health Organization to measure quality of life for use in all cultures and standardized by Min (2000).

## **2.4.3 Participants' Feedback and Satisfaction Surveys**

To evaluate participants' feedback and satisfaction surveys, we used the Self Efficacy for exercise (SEE) instrument, which assesses self-efficacy for exercise and is self-reported. Self-efficacy has been cited as an important factor influencing physical activity (Lee et al., 2007).

## **2.4.4 Biosignal Acquisition Applications**

Measure biosignals in real time while performing a task. The tool used is the Biosignalplux Professional. It can provide up to 10 h of continuous signal streaming wirelessly at 16-bit resolution per channel at a sampling rate of up to 3.000 Hz with eight specialized sensors of your choice, and it can record eight data simultaneously. The vital signs to be measured by this tool.

**2.4.4.1 Electromyography (EMG).** Measuring muscle activation by bioelectricity. In this study, surface electromyography was used to quantitatively measure maximal voluntary contraction values. The sampling rate was 1000 Hz and the notch filter was 60 Hz. The measurement site was selected for the externsor muscles, which are the most used muscles when using the whole body exercise device, and the sensor Ch.1 was measured for the left biceps brachi, Ch.2 for the right biceps brachi, Ch.3 for the left quadriceps, and Ch.4 for the right quadriceps.

**2.4.4.2 Electrodermal Activity (EDA).** By measuring the change in sweat secretion, the sweat glands are active and the sympathetic nervous system is active. In this study, sensors were attached to the 2nd and 3rd fingers, and changes in the average value of skin conductance were measured in the signal parameters of each intervention.

**2.4.4.3 Temperature (TEMP).** It is used to measure physical or ambient temperature and provides an analog output of temperature change, with a short response time and high accuracy. In this study, the temperature was measured immediately after the intervention to check the body temperature and to measure the change between each intervention.

**2.4.4.4 Oxygen Saturation (SpO2).** The SpO2 sensor is designed to estimate the oxygen saturation level in the finger using two LEDs, and the reflected light from each of these LEDs is absorbed by a photodiode that converts the current into a digital value, from which the blood oxygen saturation can be derived. In this study, the average oxygen saturation value was measured to determine changes in the intervention.

## **3 Result**

## **3.1 Physical Function Measures**

The results of the paried t-test for within-group comparison of PRE at pre test, post test 1, post test 2, post test 3, and post test 4 are shown in Table 2. In RPE, the score was 7.2 ± 0.44 at pre test and 9.6 ± 1.14 at post test 1, which was a statistically significant improvement (P *<* .05). In post test 2, the score was 11.6 ± 1.14, a statistically significant improvement compared to post test 1 (P *<* .05). In post test 3, the score was 14.4 ± 2.07, a statistically significant improvement compared to post test 2 (P *<* .05). There was also a statistically significant improvement when comparing pre test to post test 3 (P *<* .05).


**Table 2.** Characteristic of the RPE Test of Total Score

\*: P *<* .05

#### **3.2 Psychosocial, Mental and Quality of Life Health Outcomes**

The results of theWilcoxon signed rank test for within-group comparisons ofWHOQOL-BREF at pretest, post test 1, post test 2, and post test 3 are shown in Table 3. The WHOQOL-BREF showed a statistically significant improvement from 79.8 ± 8.31 at pretest to 81.8 ± 8.58 at posttest 1 (P *<* .05). In post test 2, the score was 81.8 ± 10.47, which was not statistically significant compared to post test 1 (P *>* .05). In post test 3, the score was 85.0 ± 9.82, which was not a statistically significant difference compared to post test 2 (P *>* .05). There was a statistically significant improvement when comparing pre test to post test 3 (P *<* .05).


\*: P *<* .05

#### **3.3 Participants' Feedback and Satisfaction Surveys**

The results of Wilcoxon signed rank test for within-group comparison of SEE in pre test, post test 1, post test 2, and post test 3 are shown in Table 4. SEE scored 782.0 ± 213.23 in pre test and 878.0 ± 182.67 in post test 1, which is not a statistically significant difference (P*>*.05). In post test 2, the score was 878.0±182.67, a statistically significant improvement compared to post test 1 (P *<* .05). There was no significant difference in post test 3, with a score of 880.0 ± 275.77, compared to post test 2 (P *>* .05). There was also no statistically significant difference when comparing pre test to post test 3 (P *>* .05).

#### **3.4 Electromyography (EMG)**

The results of the Wilcoxon signed rank test for within-group comparison of EMG in interventions A, B, and C are shown in Table 5. In EMG, it was 0.74 ± 0.43 in A and



0.72 ± 0.35 in B, a statistically significant improvement (P *<* .05). In C, there was no statistically significant difference compared to B at 0.80 ± 0.31 (P *>* .05). There was also no statistically significant difference when comparing A and C (P *>* .05).

**Table 5.** Characteristic of the EMG Test of Total Score


#### **3.5 Electrodermal Activity (EDA)**

The results of the Wilcoxon signed rank test for within-group comparison of EDA in interventions A, B, and C are shown in Table 6. EDA was 4.22 ± 1.82 in A and 4.27 ± 2.23 in B, which was not statistically significant (P *>* .05). In C, the score was 5.76 ± 1.33, a statistically significant improvement compared to B (P *<* .05). When comparing A and C, there was a statistically significant improvement (P *<* .05).

**Table 6.** Characteristic of the Electrodermal Activity of Total Score


\*: P *<* .05

#### **3.6 Temperature (TEMP)**

The results of Wilcoxon signed rank test for within-group comparison of Temporature in A, B, and C interventions are shown in Table 7. Temporature was 36.53 ± 0.22 in A and 36.55 ± 0.11 in B, which was not statistically significant (P *>* .05). In C, it was 36.57 ± 0.09, which is not a statistically significant difference compared to B (P *>* .05). There was also no statistically significant difference when comparing A and C (P *>* .05).


**Table 7.** Characteristic of the Temperature of Total Score

\*: P *<* .05

#### **3.7 Oxygen Saturation (SpO2)**

The results of Wilcoxon signed rank test for intra-group comparison of oxygen saturation in A, B, and C interventions are shown in Table 8. The oxygen saturation was 98.50 ± 0.68 in A and 98.56 ± 0.50 in B, which was not statistically significant (P *>* .05). In C, it was 98.50 ± 0.50, which was not statistically significant compared to B (P *>* .05). There was also no statistically significant difference when comparing A and C (P *>* .05).

**Table 8.** Characteristic of the oxygen saturation of Total Score


\*: P *<* .05

#### **4 Discussion**

The purpose of this study was to evaluate and compare the effects of real-time monitoring smart elephant whole body exercise device and walking exercise program on mental and physical outcomes, and to study customized rehabilitation exercise optimized for people with disabilities. To this end, we evaluated and compared the effects of using the smart elephant whole body exercise device and walking exercise programs on mental and physical outcomes in five non-disabled people in the community. The results showed that the physical function measures, PRE assessment, showed a significant improvement between each intervention from pre test to post test 3, and the WHOQOL-BREF of psychosocial, mental and quality of life health outcomes showed that walking exercise affected the psychosocial, mental and quality of life health outcomes more than before exercise, and the elephant whole body exercise device affected the psychosocial, mental and quality of life health outcomes more than before exercise. In other words, the effects of exercise on psychosocial, mental and quality of life health outcomes could be seen, but there were no differences between each exercise program. In the SEE, which examined participants' feedback and satisfaction surveys, there was no difference between pre- and post-exercise, but the combined walking and elephant exercise program had a impact on participants than the walking program. In other words, The study results indicate promising improvements in physical function measures, psychosocial and mental health outcomes, as well as participants' feedback and satisfaction. The observed statistically significant improvements in the Borg Rating of Perceived Effort (RPE) and the WHOQOL-BREF scores support the effectiveness of the exercise interventions. The findings suggest that the Smart Elephant Whole Body Exercise Machine and walking exercises have a positive impact on the physical and mental well-being of individuals with disabilities.

The biosignal data showed that the walking plus elephant exercise program had a significant improvement in EMG compared to walking, and the elephant exercise program had a significant effect on EDA compared to walking and walking plus elephant exercise. However, there were no statistically significant differences between the interventions in temperature and oxygen saturation.

## **5 Conclusion**

It was necessary to standardize the effects of the exercise program by conducting a study on non-disabled people before conducting a study on people with disabilities. Also, a study on non-disabled people was conducted before the mid- to long-term project for people with disabilities to identify deficiencies in safety and operation. A limitation of the study is that it is difficult to generalize the results because the experiment was conducted only with women in their early twenties. Therefore, in future studies, an appropriate number of sampling and balanced sex subjects are needed to increase statistical significance. In addition, since it was conducted for non-disabled people, not the disabled, it was not possible to use the initially set evaluation tools. Evaluations in the areas of daily living and falls were excluded from the results because it is difficult to compare and evaluate the functions of people with disabilities because their functions are perfect and do not change. In addition, we tried to check the electrocardiography (ECG) of vital signs, but the sensor generated a lot of noise during exercise, so we could not extract accurate data, so it was excluded from the results.

Due to the improvement of living standards, expanding access to medical services, and increasing interest in people's health, disease-related mortality and life expectancy are gradually increasing, but unlike the development of medical technology, it is difficult for rehabilitation patients who were hospitalized to continue to perform rehabilitation exercises in the community after leaving the hospital. This is because various exercise programs that are not medically proven do not reflect individual specificity and are performed collectively due to administrative and financial convenience (Bae et al., 2010). This study will be used as a basis for customized rehabilitation exercises that provide a proven rehabilitation exercise service model for people with disabilities in the community, and it is hoped that the lives of people with disabilities will become health.

**Acknowledgments.** This study was supported by (Intelligent Rehabilitation Sports Mediation Research Project (R&D) (20230208DEA - 00), National Rehabilitation Center, Ministry of Health and Welfare, Korea.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Analysis of Virtual Reality Based on the Internet of Things on Human Psychology 'Internet of Thoughts' (IoThs) for Rich Content Extraction Applied Natural Language Processing and Deep Learning**

Pascal Muam Mah1(B) , Iwona Skalna1,2, Tomasz Pełech-Pilichowski1,3, Tomasz Derlecki1,4, Mahmoud Nasr1,5,6, Eric Munyeshuri1,7, Gilly Njoh Amuzang1,8, Micheal Blake Somaah Itoe1,8, and Ning Frida Tah1,8

<sup>1</sup> Department of Technical Informatics and Telecommunications, AGH University of Science and Technology, Krakow, Poland {mah,skalna,tomasz,derlecki,nasr,Eric.Munyeshuri,

Gilly.Njoh.Amuzang,Micheal.Blake.Somaah.Itoe,

Ning.Ning.Tah}@agh.edu.pl

<sup>2</sup> Department of Business Informatics and Management Engineering, AGH University of Science and Technology, Krakow, Poland

<sup>3</sup> Institute of Computer Science, AGH University of Science and Technology, Krakow, Poland <sup>4</sup> Department of Economics and Management Industry, AGH University of Science and Technology, Krakow, Poland

<sup>5</sup> Department of Biomedical Engineering, AGH University of Science and Technology, Krakow, Poland

<sup>6</sup> Department of Engineering Mathematics and Physics, Faculty of Engineering and Technology, Future University, Cairo, Egypt

<sup>7</sup> Department of Production Engineering and Logistics Enterprise, Silesian University of Technology, Gliwice, Poland

<sup>8</sup> Department of Logistics Management, Silesian University of Technology, Gliwice, Poland

**Abstract.** In the past decade, a lot of challenges to access, assess, and to acquire the needed technological opportunities to teach computers what naturally comes from the human brain and to understand how we naturally react when we rely on technology. The ability to document human thoughts, reactions and behavior to computers has led to the coming of NLP, AI, Dl, & ML. Aim to understand the influence of IoT on humans with the use of DL to achieve content correctness and accuracy with virtual technology. Studies show that the way we think, react, and do the things we think "Internet of thoughts" reflect our personality. The way we think determines the way we react and the way we do things are based on how we think. Technology advancement has reinforced a lot of changes in humans which makes humans vulnerable to personal content exposure misappropriation due to the continuously changing nature of humanity and language. The study uses NLP, DL and behavior-oriented drive and influential function and results show that IoT based on VR influences human psychology "Internet of Thoughts".

**Keywords:** Human Psychology · Virtual reality · Deep learning · Natural language processing · Internet of things · Internet of Thoughts · Content extraction

## **1 Introduction**

Deep learning (DL) is a computer program called machine learning that provides access to techniques that teach computers to do what comes naturally to humans [1]. Natural language processing (NLP) is a subfield of computer linguistics, computer science, and artificial intelligence (AI) that connects and enables the interactions between computers and human natural language. Technology advancement has reinforced a lot of changes in humans [2]. The precipitated changes in the internet world have modified the way we see the outside environment, manage our affairs and make changes to our bodies and environment. Due to precipitated changes in internet things (IoT), the attraction effect on human psychology "internet of thoughts" (IoThs) now occurs when the preference for a target increases relative to our choice option that is dominated by the targetoriented services of internet services. The eye is an important part of human life [4]. The way we see things determine the way we react to them in real life. Virtual reality (VR) helps to redesign prior reactions and feelings through virtual performance [5]. The eyes see content different from virtual technology [9]. Virtual reality build in humans a strong emotional engagement with data and can help tremendously in data extraction [7]. It shapes the way we see the world around us. Virtual reality (VR) is a computersimulated experience that employs pose tracking and 3D near-eye displays to give users an immersive feeling of a virtual world via hardware (headsets) and software designers to create VR experiences such as virtual museums [8]. Due to these advances in technology, human react responses and behaviors to real-world natural artifacts for museums are now perceived differently because of virtual reality [6]. There exist a new struggle to balance the much important technology such as virtual reality and the physical facilities represented in 3D systems and object identifiers by virtual reality [10]. Natural language processing (NLP) is a sub-field of artificial intelligence (AI) that uses computers to understand, analyze, and extract meaning from human language in a smart, sustainable, and useful manner Developers uses natural language processing to organize structure knowledge to perform tasks such as automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, and topic segmentation [3]. Natural language processing uses common word processor operations to handle text and treat them as a sequence of symbols taking appropriate consideration of the hierarchical structure of language.

## **2 Literature Review**

The internet of thoughts (IoThs) "Internet of Thought (IoThs)" is part of the human mental process that built psychological associations with models of computer systems and uses the products and services of the internet of things to influence the way we think, react and do things based on how we think. The development of models and modeling systems software that represents human natural thoughts has allowed other technologies like the internet of things to influence the way we think, react and do things the way we think. One of the advantages that IoTs have over humans is their ability to connect living things and non-living. The process of the internet of things is far beyond human capacity and it's influencing the way humans think, act and behave. The psychoanalytic theory explains that cognitive psychology engages humans with things based on the way they see, access, and feel about them. The higher the cognitive processes the more deeply the influence on human ways of think, acting, and doing things. Identifying things and resolving differences with items of similar nature have a higher achievement with the internet of things smart grip. The deep desire humans experience when in an encounter to identify an item, exerts a tense atmospheric situation in humans. The inherent feelings that arose during a moment of identifying amongst alternatives relate to Freud's thermodynamics theory acknowledgments power of internet service. Due to advanced developments, identifying someone based on his or her original and choice of words is becoming increasingly possible. This has made it easier for internet service to influence human choice, The advances in IoTs have a strong grip on human intuitive attention, decision making, learning, judgment, reasoning, thinking, and cognitive processes.

#### **2.1 Basic Seven (7) Principles of Unprecedented Changes on Human Psychology "Internet of Thoughts"**

The study provides an explanation to the unprecedented changes in the way we think, react and do things we do think in terms of seven (7) principles below. The following definition explains the role of these elements in changes in our activities.

**Cultural Heritage.** This relates to the culture we inherited from birth from where we grow up. The language we speak depicts part of our culture. Certain languages are associated with certain cultures. In the world, the English language depicts British culture, American English depicts American culture, French culture depicts French culture and the Polish language depicts Polish culture. Cultural heritage offers a key opportunity for the development and deployment of new IoTs systems, with potential benefits both for the culture [11].

**Educational Standards.** The educational standard of a person changes the way he or she thinks, reacts, and does things. It is often very easier to judge people just by way of observing them act, speak and behave. The educational status of someone instills a certain culture in the person's altitude. It is easier to be connected with real-world affairs by being acknowledged the all-around happenings [12, 22].

**Environmental Situation.** The environment in which people grow pre-determine some of their altitudes. Persons in urban areas have a way of behaving that is different from those in rural areas. It is understood that the urban environment exposes people more than the rural environment. The growth of skill enhancement expands one's sense of self and adaptation to the environmental and personal growth facilitation model of curiosity informs our research [13].

**Information Availability.** The availability of information changes the way we think, react and do things. The availability of information has a way to instill a degree of culture in us. Having a long list of alternatives normalizes internal desire. It is known that information is the power to mental growth. Having information helps to advance moderate choices as compared to the situation with no or limited access to information. Information availability allows one to suggest that information availability has an effect on the habit of thinking [14, 18].

**Information Accessibility.** The way we think, react, and do things depend on information accessibility. This has a great way to influence us. There are some countries with little or no free access to certain information which course citizens of such countries to have a different perception than others who have access. The availability of access is the availability of knowledge. The accessibility of cue information in memory affects which decision strategy individuals rely on [15].

**Information Efficiency.** The ease with which we obtain information helps a lot to shape our way of thinking, reacting, and doing things. Some peoples turn out to be very selfish with knowledge and information. A wide range of factors influencing the individual. Subjective norm is activated when individuals become aware that certain behaviour that they perform have adverse effects on issues they belief, and that behavioural action will have a positive and significant impact on the aspects the individual values [18].

**Cognitive Resource Efficiency.** There is a quality achieved such as an experience, intelligence, competence, and task-relevant knowledge [17]. The flexible allocation of cognition during translation is assumed to codetermine overall efficiency in translation [16].

#### **2.2 The Theory of Psychoanalysis Applied Basic Seven (7) Principles of Unprecedented Changes on Human Thoughts**

The figure below explains the relationship between the identified elements known as principles of unprecedented changes on human thoughts with Freud's Theory of thermodynamics of psychoanalysis.

Figure 1 represents ID which is the Unconscious drive by IoTs. Pleasure-oriented and self-drive by inherent instincts by the internet which can be attributed to cultural heritage, educational standards, and environmental situation. EGO is the Consciousness rational driven by situational reality. The balance demand of the Id and superego can be attributed to information availability, information accessibility, and information efficiency. SUPEREGO is the state of Morality Concerned. The personality developed through socialization is attributed to cognitive resource efficiency.

**Fig. 1.** Psychoanalysis applied basic Seven (7) principles of unprecedented changes on human thoughts. Source: author's own copy

#### **2.3 Deep Learning Interrelation with Natural Language Processing and Machine Learning**

The section represents information, applications, and system software that enable and enhances data extraction. Most data from simple text, to classifications, patterns and clustering pass through one of the following programming languages below depending on the context of information.

**Fig. 2.** Deep learning interrelation. Source: author's own copy

Figure 2 represents components of natural language processing, deep learning, and machine learning. Over the past years, the field of natural language processing has been propelled forward by an explosion in the use of deep learning models [19]. As natural language processing and Machine Learning (ML) tools rise in popularity, it becomes increasingly vital to recognize the role they play in shaping societal biases [20]. Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. After the advances made in Computer Vision using deep learning tools, NLP has metamorphosed recently. Deep learning is represents learning algorithms in deep learning that natural language processing uses to run its applications. Applying deep learning to solve fundamental problems in NLP has tremendous strides in deep learning applications to NLP. Recently, the NLP community has witnessed many breakthroughs due to the use of deep learning. Deep learning (DL), is a subfield of machine learning (ML) [21].

## **3 Results**

In this section, we provide information on how text or speech content can be extracted and classified into different parts of the speech based on data obtained via NLP means that help in decision-making for digital services. The influence is symbolized as BIF = F(D) which is said "f of d" equal to Eq = ∫*(*D*)* -MRS MR × BS Metrics range is up of (nouns, adjectives, verbs, adverbs, interjections, prepositions, conjunctions, pronouns, determiners, and numerals). Metrics range substitute is made up of (nouns, adjectives, verbs, adverbs, interjections, numerals, and prepositions).

**Fig. 3.** Metric range and metric range substitute. Source: author's own copy

Above Fig. 3, provide data classified into different groups Class A and Class B. The group mark with red are made up of metric range substitute and metrics range for a single individual called Class Band that with green is called Class Awhich constitute of metrics range substitute and metrics range. To obtain influence rate, metric range substitute is divided by sum of metric range then multiple by behavior score. *Class A MR =11 and* =9 while *Class B :::)) MR =12 and* =11.

**SOLUTION CLASS A** *Class A MR =11 and* =9

*MR=Metrics Range* = *Metrics Range Substitute*

*BS=Behavior Score (Also human five sense organs) =*

====)))) ====)))

**SOLUTION CLASS B** *Class B MR =12 and* =11

*MR=Metrics Range* = *Metrics Range Substitute*

*BS=Behavior Score (Also human five sense organs) =*

$$\begin{array}{rcl} \stackrel{\textstyle \longrightarrow \longrightarrow}{\longrightarrow} (\stackrel{\textstyle \longrightarrow}{\longrightarrow})))) \\ \stackrel{\textstyle \perp \perp}{\longrightarrow} 5 \\ \stackrel{\textstyle \perp \perp}{\longrightarrow} 0)) \ \stackrel{\textstyle \longrightarrow}{\longrightarrow} 4.58 \end{array}$$

From the statistics we can say that the influence score is grade " Very Good" as per classification is achieve by both class but Class B is better than Class A with behavior score of 4.58 and 4.0909 respectively.

## **4 Applied Method**

The section explains the stages used to develop the classification and how data were selected and classified into a deep learning model. The study uses two sample texts and is classified into groups called Class A and Class B as per Fig. 3. This section represents the various parts of speech selected by the author. We consider these parts of speech as those that convey information in a simplified manner to the lowest level that even a language beginner can learn from.

Figure 4 represents a deep learning model of a recurrent neural network. The hidden layer is represented by ten (10) parts of speech. The ten (10) parts of speech are grouped into two fractions called metrics range substitute (nouns, adjectives, verbs, adverbs, interjections, numerals, and prepositions) and metrics range (nouns, adjectives, verbs, adverbs, interjections, prepositions, conjunctions, pronouns, determiners, and numerals). The metrics range substitute detailed summaries of a speech and provide meaning in the simplest form while the metrics range contains all of the parts of speech. Datasets are automatically differentiated in the hidden layer sector and sent to the output with two layers.

Figure 5 represents the behavior layers of content extraction charge of generating coordinated verbal gestural and facial behaviors for realizing an important of each utterance behavior to the require context. This help the extractor or practitioner determine influence of each utterances. This includes the synthesis of intelligible verbal and gestural acts per see and each combination allows a continuous flow of human-like multimodal utterance as digital services streamline. The main research purpose of this algorithm is to analyze the degree of richness user content.

**Fig. 4.** Applied Deep Learning on Content Extraction using Natural Language Processing. Source: author's own copy

**Fig. 5.** (Scoring model) Virtual reality and internet of things on human psychology. Source: author's own copy

## **5 Virtual Reality and Internet of Things Influence Human Psychology**

This section presents the parts of virtual reality that engages with the internet of things to influence human's way of thinking, reacting, and responding to situations. Humans are face with challenges of complex system, applications, and multiple technologies (Fig. 6).

**Fig. 6.** Virtual reality and internet of things influence human psychology. Source: author's own copy

There is no technology that was aim to confuse uses except a redirection of the intended use for something locally. This has been the work of haters of healthy and secure world. This study examine four (4) aspects of virtual technology that internet of things have fully integrated to give humanity good life. These four parts of virtual reality technology engage the text and images of a user. Virtual reality technology is a cutting-edge intelligent system that is currently processing the connecting technology of artificial psychology and is also gradually coming to maturity with the help of the Internet of Things.

**Immersion and the Internet of Things on Human Psychology.** Immersion is simply a sense of reality that promote interest and encourage participation. Immersion in the internet of things helps humans' to boost their way of reasoning with a comprehensive technology that combined with high technology to achieve a multi-field study under the virtual background using the internet of things to connect with the outside world and to realize the interactive feelings. Without the internet of things, there will be no virtual reality technology. The importance of immersion has been seen in today's world real life with tremendous changes and support services known as remote work and studies.

**Engagement and the Internet of Things on Human Psychology.** The internet of things offers a unique opportunity to help empower humans and improve societies' engagement. Empowerment improves the human way of reasoning, responsibility, and interaction. Engagement creates a digitalized altitude of dominant coalition motivation and the ability to deploy the resources. The ability of humans to feel relevant plays an important role in their way of reasoning and manners of doing things. In the healthcare sector, human psychology is very important in determining health issues. A positive mindset helps the health practitioner to facilitate treatment which can be achieved by deploying virtual technology.

**Interaction and the Internet of Things on Human Psychology.** The internet of Things helps create more significant learning spaces for human interaction in virtual systems. The ability to allow humans from a free physical background to run some modernized experience exercises is very important to human life and extremely shapes human psychology. Virtual objects are very important in advancing interaction for internet-based systems.

**Imagination and the Internet of Things on Human Psychology.** The internet of things provides means for systems to surpass human imagination with good and positive systems. The context of the internet of things is becoming the number one system that is reinforcing organizations to rethink their value creation and methodology. With the emergence of the Internet **of** Things, it is easier and cheaper to make information available about virtual physical objects as this information can be automatically created, and distributed by simple stream of imagination.

## **6 Conclusion**

The problem of content extraction is very essential in managing the amount of content, but other important issues associated with this are the viewpoint of the creator, quality of content, usability of content, and multiple views without changing the original content. This has not been the case in the past which is why this study applied deep learning to natural language processing using parts of speech to evaluate the changing nature of humans and the content of their information. Based on the behavior-oriented drive and influential function of the internet of things on human psychology "internet of thoughts" on content extraction, results show a score grade "Very Good" as per classification for both classes A and B but Class B is better than Class A with behavior score of 4.58 and 4.09 respectively. The study concluded that the internet of things based on virtual reality technology influences human psychology "internet of thoughts (IoThs)" on content extraction using natural language processing and deep learning models. This will help E-health services be able to accurately determine patient's situations remotely and prescribe treatment. Digital objects like pointers, editor's boards and working space should be part of Internet of Things with high visibility to health improve cognitive importance and promote healthcare and well-being of uses.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Innovative Smart Care Space Solutions: Integrating Technology and Universal Design for People with Significant Severe Disabilities**

Kuem Ju Lee1,2 and Won-Kyung Song1,2(B)

<sup>1</sup> Department of Rehabilitative and Assistive Technology, National Rehabilitation Center, Seoul 01022, Republic of Korea {kjlee74,wksong}@korea.kr <sup>2</sup> Translational Research Program for Care Robots, National Rehabilitation Center, Seoul 01022, Republic of Korea

**Abstract.** This study aimed to develop and implement smart care spaces (SCS) to improve the quality of life and convenience for individuals with significant severe disabilities. By analyzing Korean and international case studies, the study integrated technology, universal design principles, and stakeholder feedback, along with care robots for daily living assistance. Various areas within the SCS address the unique needs of people with severe disabilities. The first SCS was constructed in the Nuri Hall of the National Rehabilitation Center in Korea as a pilot project. Future plans include continuous improvement and utilization of the space based on user experiences, feedback, and results of usability evaluation to reduce the caregiving burden and promote inclusive environments.

**Keywords:** people with significant severe disabilities · care robots · service model · smart care space

## **1 Introduction**

The rapid aging of the Korean population has led to an increasing demand for care services, while the younger generation's population structure is decreasing at an unprecedented rate. As the number of older individuals with disabilities continues to increase, there is an urgent need to address the care burden and determine effective solutions. This is particularly important since many severely disabled older adults and those with reduced mobility spend most of their time at home with caregivers, greatly impacting their quality of life.

Previous studies investigating the living environments of people with disabilities [1–3] revealed that their living conditions play a significant role in determining their quality of life. In response, the Korean local governments have been implementing housing improvement projects for the disabled, such as installing residential convenience facilities for low-income registered persons with disabilities [4]. However, the creation of barrier-free spaces that allow for active and personalized care, accommodating both people with disabilities and their caregivers, is urgently needed.

As part of the 4th industrial revolution [5], the development of care robots [1, 6– 8] has the potential to assist caregivers and reduce their care burden. However, the environmental factors need to be considered for the successful integration and utilization of care robots. Moreover, as Korea is projected to become a super-aged society by 2026, the number of households requiring elderly care is expected to increase, further increasing the demand for home care solutions.

To explore how care robots and services can be effectively implemented in various settings, we conducted a comprehensive analysis of the literature, field cases, and stakeholder interviews through a care robot translational research and service model development project (2019–2022). Based on our findings, we constructed a smart care space (SCS) on the second floor of Nuri Hall in the National Rehabilitation Center, Korea. This paper aims to discuss the design, implementation, and implications of this innovative SCS, which integrates technology and universal design for people with significant severe disabilities, to address the care challenges in the care of the aging population with and into disabilities.

## **2 Materials and Methods**

From January to December 2020, in order to develop the appropriate SCS for severely disabled older adults, results of the literature review of similar cases in Japan and Korea were analyzed, and various cases were collected by visiting the target sites that had a barrier-free design at home and abroad. A total of 11 institutions were visited to collect case studies. Next, 7 domestic cases such as that in Shinnae Medical Safety House and 3 foreign cases such as that in Denmark were visited and benchmarked based on the collected data. Then the results were used to determine the SCS design.

In this study, the British Design Council in 2005 adopted the double diamond process, which involves problem solving carried out through a four-step process (discovery, definition, development, delivery), as a service design developed and analyzed the effective user experiences.


In order to establish the SCS for the severely disabled, the areas that need improvements were determined through stakeholder meetings with the severely disabled and caregivers. Interviews were conducted with experts, persons with severe disabilities, and caregivers.

## **3 Results**

#### **3.1 Survey of Current Status and Stakeholder Interviews for Smart Care Space Design**

Between 2019 and 2020, the appropriated SCS model was designed by determining the current situation and gathering expert opinions, related to the residential environments of domestic wheelchair users. We collected case studies carried out in various facilities based on a comprehensive literature review.

Through stakeholder interviews surrounding Smart Care Spaces (SCS), the prospective opportunities and users' needs of care receivers and caregivers were analyzed. These needs are critical to be accommodated within their communal living space. Given that care receivers frequently spend the bulk of their day either in a supine position or require assistance, it's imperative for them to be able to independently operate key devices or equipment. Simultaneously, caregivers showcased significant apprehension towards unfamiliar technology or devices, especially when utilized for the first time, underscoring the necessity for straightforward and intuitive user experiences.

As such, the SCS should be envisioned as an interactive environment in which care receivers and caregivers cohabit and partake in daily life activities together. The focus should be on providing services that cater to individual needs and alleviate common inconveniences.

Furthermore, it was found that the majority of Korean caregivers are older women with a generally low level of technological proficiency and a pronounced fear of new technologies. This demographic tends to react sensitively when interacting with technological devices, thereby necessitating an approach that is user-friendly and non-intimidating.

In expert interviews, it was proposed that the incorporation of Internet of Things (IoT) technologies is crucial to enhancing user satisfaction and the overall quality of life in smart care spaces. These technologies could encompass voice recognition for simple on/off functionalities, sensor-driven alarm systems, behavior detection technology, and services that respond to user-intent. The main implication from this recommendation is the need for these technologies to be effortlessly usable and conveniently manageable by the end users.

The different notable domestic cases that showcased our SCS designs were as follows: Shinnae Medical Safety House, which features safety devices and space designs tailored to individuals with disabilities; Hanssem Smart Home Model House, which utilizes a Wi-Fi-based smart home system; Seongnam Senior-friendly Comprehensive Experience Center, Seongnam Senior Smart Home, and the Home Adaptation Training Center of the National Transportation Rehabilitation Hospital, which emphasize the senior living spaces that incorporate advanced technology from the 4th industrial revolution era, with a focus on the Internet of Things (IoT) technology; and the Korea Institute of Science and Technology (KIST), Gachon University Ambient Assisted Living (AAL), and Bundang Seoul National University Hospital, which offer separated spaces for experiments, exhibitions, and monitoring, as well as for usability evaluations.

Additionally, we benchmarked several international cases that showcased innovative solutions not yet implemented in Korea. These included the installation of folding furniture attached to the wall and wall mounted-type lifts that provided assistance during spatial movement at Musholm (Denmark, a residential facility for people with muscular disorders); the application of universal design principles to enhance usability at the Life and Living Center (Helsinki, Finland); and SilviaBo (Sweden, a residential facility for older adults with dementia), which had simplified spatial structures and improved indoor circulation for wheelchair users.

#### **3.2 Smart Care Space Design**

**Main Entrance and Room Door.** Wheelchair users find it easier to operate sliding doors rather than pull-style doors. Although the main entrance door features a set of double pull-style doors for ease of installation, all room doors have been designed with sliding door systems. When designing entrances and corridors, the required effective width and front effective distance should be considered. For bedroom access, the effective passage width, activity space, bed structure, and flooring should be taken into account. Installing semi-automatic or automatic sliding doors at the entrance is recommended, along with securing activity space in front of and behind the entrance to avoid collisions with hallway pedestrians. To accommodate wheelchair users, a toilet door should have an effective width of at least 0.8 m, a step of 2 cm or less, and a sliding door rather than a hinged one, as this provides more activity space in front of the door and added convenience.

**Living Room and Hallway.** Architectural considerations should be made to ensure a clear and obstruction-free pathway for wheelchair users. Since it can be challenging for wheelchair users to move in a straight line when their left and right manipulation power is not equal, a wider walking width is necessary. Power wheelchairs require a width of 800 to 900 mm, and the rotation widths should be 2,000 mm for power wheelchairs and 1,800 mm for manual wheelchairs. By securing an aisle width of more than 1,200 mm for emergency beds, placing safety handles in corridors, and installing unit households with emergency alarm lights at entrance tops, emergency contact devices inside households, and motion detectors, safety can be ensured.

**Restroom.** To prevent head injuries, a sufficient distance should be maintained between the toilet and the wall, and wheelchair users should have access to either side of the toilet depending on their dominant hand. The effective toilet size should measure at least 1.4 m in width and 1.8 m in depth, with a 0.75 m or wider activity space secured on one side of the toilet for side wheelchair access. The toilet seat height should be between 0.4 m and 0.45 m from the floor, and an adjustable wall-mounted toilet should be installed to accommodate wheelchair users, older adults, and children. A user-friendly flushing mechanism, such as a light-sensing push-button or lever type, should be installed.

**Others.** The height of wall hangings, electrical outlets, switches, storage shelves, and hangers should be between 0.8 m and 1.2 m from the floor; if a bed is not used, the installation location of electrical outlets and switches should be within 0.35 m. The doorbells for hearing-impaired disabled individuals must be installed in rooms, bedrooms, and bathrooms. The floor surface of all spaces should be level, be finished with non-slip material even when wet, and have an effective floor area of more than 1.4 m in width and 1.8 m in depth.

#### **3.3 Design and Construction of SCS**

**Design of SCS.** Based on the literature review and current status survey, an initial SCS design draft was developed and refined using the input from architectural experts (Fig. 1A). Design drawings were then used to gather and apply the following field case characteristics (Fig. 1B): 1) incorporating double doors (final width of 1,600 mm) of at least 1,100 mm at the main entrance for easy movement of emergency beds and wheelchairs; 2) designing steps within 20 mm at the entrance to facilitate wheelchair movement; 3) removing steps at the bathroom entrance and installing a water drainage trench to prevent water from flowing back into the main room; 4) using wheeled furniture or beds for easy mobility; 5) maximizing space utilization with wall-mounted, foldable furniture; 6) positioning ceiling-mounted lifts, height-adjustable washbasins, and foldable shower beds for improved bathroom usability; 7) creating a multipurpose room near the entrance for lift storage; 8) arranging a straight-line wall-mounted lift for convenient bathroom access by connecting the toilet to the main room; 9) designing a barrier-free entrance and toilet considering the turning radius of electric wheelchairs (more than 1,500 mm); and 10) installing sliding bathroom doors instead of hinged doors for optimal space usage.

**Fig. 1.** Smart care space construction design (draft). (A) Space size and design draft, (B) Smart care space final design draft

**Establishment of Smart Care Space (Nuri Hall, National Rehabilitation Center).** The National Rehabilitation Center's Smart Care Space was established as part of the Ministry of Health and Welfare's care robot translational research and service model development project, which started in 2019. The space, built for demonstrations like usability evaluations of care robots or care devices, is the first in Korea to address the needs of both severely disabled individuals and their caregivers. The initial SCS was constructed in the National Rehabilitation Center's Nuri Hall (December 2019) and opened in 2020 (Fig. 2). It integrates care robots and various sensor-based IoT facilities and devices into six spaces to create a convenient and safe care environment that accommodates various care scenarios and physical conditions, emphasizing IoT technology and universal design features.

The SCS was designed to harmonize with the needs of severely disabled individuals and their caregivers, thus enabling care robots to assist with daily tasks seamlessly. It also serves as a demonstration space for caregiver burden analysis and care robot usability evaluations. The service environment for severely disabled individuals includes a Google speaker and application, functioning as the central controller in the bedroom, which connects to other devices in the house and provides personalized services. Voice commands such as "Come out" and "I'll go out" can sequentially control various devices, while natural language voice commands enable users to manage air-conditioning, lighting, and weather information on-demand. The UI and UX were designed to facilitate device control.

**Fig. 2.** SCS built in the National Rehabilitation Center, Korea (Photos of 6 spaces of SCS). SCS: smart care space.

## **4 Discussion**

The SCS offers a revolutionary approach to create tailored, efficient, and accessible care environments for people with significant severe disabilities. First, it integrates cutting-edge technology, universal design principles, and valuable stakeholder feedback to address the unique needs of and challenges faced by both disabled individuals and their caregivers. The incorporation of the barrier-free concept in the SCS, enables the efficient use of care robots, thus enhancing the effectiveness of this innovative space. Second, as a model for future development, the SCS aims to improve the quality of life for those with disabilities and reduce caregiver burden through the incorporation of care robots and other assistive devices. By continuously evolving based on user experiences and feedback, the SCS can potential transform the care landscape, fostering more inclusive and supportive environments for disabled individuals and their caregivers.

The Nuri Hall of National Rehabilitation Center's SCS allows the utilization of care robots that provide technical assistance in caregiving and enhancing care quality, while also serves as a demonstration space for care robot prototypes. Considering the various stakeholders' input, the SCS integrated technologies related to daily living convenience, such as barrier-free design, smart controllable IoT devices, voice control applications, and wall-mounted transfer devices tailored for wheelchair users. Opened in 2020, the space caters to the needs of severely disabled individuals, primarily wheelchair users, by designing and organizing spaces based on daily life scenarios illustrated through journey maps. It also features pressure ulcer prevention and posture-changing beds, excretion care robots, and meal assistance devices.

The SCS have been designed and built to enhance the provision of beneficial services in a care environment by integrating networked care robots that can interact with individuals. In light of this, there is a pressing need to explore methodologies for the joint expansion and provision of care robots and care environments, particularly for those most vulnerable, including people with severe disabilities, older adults with notably limited mobility, and caregivers in Korea.

## **5 Conclusions**

The SCS's most distinguishing feature is its capacity to serve as a genuine residential environment, unlike many living labs or experience centers in Korea. The active participation and choice of users in the design process play crucial roles in the discovery and innovation of new ideas [9, 10]. In the future, patients at the National Rehabilitation Hospital, caregivers, and their families can directly use the space, enabling them to implement similar scenarios and features within their communities. This study does have some limitations. First, the SCS was designed in alignment with Korea's barrierfree housing blueprint, with specific reference to the living spaces of severely disabled individuals. Moving forward, there is a clear need for research that gathers data from the actual residences of a diverse range of users. Second, it is essential to assess the usability of the SCS across users with various disabilities to ensure inclusivity and accessibility in its truest sense. The SCS model will provide knowledge and guidance while continuously improving the space through the user's feedback on areas for enhancement after actual residency or usability evaluations, ultimately demonstrating a reduction in caregiver burden.

**Acknowledgments.** This study was funded by the Translational Research Program for Care Robots (grant number: HK19C0002, HK23C0002) from the Ministry of Health and Welfare of South Korea. We would like to express our sincere gratitude to the various researchers, experts, and stakeholders who contributed to the development and construction of the Smart Care Space. Their invaluable insights, advice, and collaboration were essential in realizing this innovative project, and we deeply appreciate their dedication and support.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **An Approach for Modeling Annotation in the e-Health Domain**

Zayneb Mannai1(B) , Anis Kalboussi2,3, and Ahmed Hadj Kacem1

<sup>1</sup> ReDCAD Research Laboratory and Faculty of Economics and Management, University of Sfax, Sfax, Tunisia

mannaizayneb889@gmail.com, ahmed.hadjkacem@fsegs.rnu.tn

<sup>2</sup> Higher Institute of Computer Science and Management, University of Kairouan, Kairouan,

Tunisia

anis.kalboussi@isigk.rnu.tn

<sup>3</sup> ReDCAD Research Laboratory, University of Sfax, Sfax, Tunisia

**Abstract.** In our research, we established a medical annotation model in the form of an ontology in an effort to ensure data interchange amongst medical annotation systems. We employ the "patient partner" approach to involve the patient in the medical annotative activity. In fact, the patient will be able to register, annotate, and comprehend the comments made in his medical file utilizing this new paradigm.

**Keyword:** Annotation · Medical record · Interoperability · Patient · Ontology

## **1 Introduction**

Over the last two decades, health information systems have experienced a significant technological revolution, which has affected the quality of services provided by these systems, their functionalities, the extent of their applications, and the sorts of users that use them [1].

As a result of this transformation, the Electronic Medical Record (EMR) is one of the components of medical information systems that are being transformed. It is true that with the use of ICT (information and communication technologies) in the medical area, the medical record, which was previously on a physical medium, has now been converted to a digital version. The latter is defined as an electronic statement that contains a summary of treatments, diagnoses, and patient follow-ups, as well as any written communications between healthcare specialists [2].

Currently, scientific research is seeing a slew of initiatives to include the patient in the decision-making process along his treatment course. This is beneficial to the healthcare worker, the patient, and the whole healthcare system because it encourages collaboration, cooperation, and knowledge exchange, and it contributes to an improvement in the safety and quality of treatment. Patient participation in the healthcare system is focused on a variety of levels, including acquiring medical knowledge, educating, establishing the treatment protocol, providing direct care, organizing care, and so on [3].

In this study we will investigate the applicability of the patient partner approach to the area of computerized medical record annotation. Indeed, traditionally held and examined solely by a medical team in a single health establishment, local electronic medical files are now shared and consulted by new interveners, the hosts, and the patients. An important part of the host's job is to create an application that includes both patient care and data preservation. Electronic medical records cannot be created or accessed without a patient's agreement or authorization. Furthermore, he is entitled to see his medical records as well as papers pertaining to the diagnosis and treatments that are relevant to him. In this sense, the patient should also have the right to comprehend the annotations that have been made to his medical record.

A medical annotation is, in reality, a comment that is written in the patient's medical file for a particular purpose. In this case, it is done by the healthcare practitioner and is meant to be read by another healthcare professional or by the patient themselves. As a result, we can say that medical annotation is a fundamental part of the healthcare system. She is the one who encourages open dialogue and information exchange among the many parties involved. However, a review of the literature on numerous annotation systems revealed that all of these techniques remain basic and are not widely used in the health system [4]. Patients are unable to access the content of the annotation systems examined in the literature, they are unable to communicate their ideas in a meaningful manner, and as a consequence, the annotation is unable to promote collaboration among various stakeholders. It will cease to work properly, resulting in a breakdown of communication between healthcare practitioners and patients.

Each annotation system investigated in the literature has its own set of standards and conventions. The inability to integrate diverse programs prevents data interchange across applications. Due to trade blockage, a healthcare professional cannot communicate or receive data from peers or transfer information across systems. This restricts the movement of both patients and healthcare providers. Also, the patient cannot send personal data to his doctor or a health practitioner, resulting in data loss. Indeed, this highlights the need to address the challenge of interoperability among different annotation systems. The objective of this work is to develop an ontology that ensures the semantic layer of interoperability.

So, what follows is an attempt to answer a fundamental question:

What generalized model of medical annotation should we use to promote health professionals and patients to exchange annotations in electronic medical records?

The second section of this paper provides a comprehensive understanding of the different types of medical records and highlights the crucial importance of facilitating their sharing.

The third section delves deep into an extensive exploration of the diverse definitions of annotation found in the literature. This thorough analysis captures a wide range of perspectives offered by researchers in the field.

The fourth section introduces a meticulously developed ontology specifically designed for layered annotations within the medical record. This ontology serves as a robust framework that ensures semantic interoperability across various medical annotation systems. By standardizing the annotation process, this ontology significantly enhances data exchange and promotes seamless collaboration among healthcare professionals.

The fifth section is a conclusion that presents a comprehensive summary of our findings while also providing valuable insights for future research in the domain of medical record annotation.

## **2 Computerization of Medical Record**

## • **Medical record sorts**

A patient's medical record is defined as a statement that contains a summary of treatments, diagnoses, and patient follow-ups, as well as any written interactions between health care providers.

Examining various forms of medical data gives critical information for medical annotation modeling. It makes it easier to interpret medical data, identify specific annotation requirements, build suitable annotation schemas, and validate annotations.

In this part, we will look at the various forms of medical records and highlight their key advantages.

• **The EMR Electronic Medical Record and the CCR Computerized Clinical Record:**

The EMR and CCR are electronic equivalents of the traditional medical record, which is recorded on paper with a pencil. They are memories that allow the acquisition of personal data from patients [Article 1 of the Canadian Uniform Act on Electronic Commerce]. These files are deemed complete since they contain all of the essential information and specialized words relevant to a given speciality and also create the proper instruments for trade practice.

## • **The EMR Electronic Medical Record and the EHR Electronic Health Record:**

The EMR is distinguished from the EHR by the Canadian Medical Protective Association (CMPA)1. The EHR delivers complete information from many providers across numerous provinces and regions. Nevertheless, this sort of record provides less detail than the EMR (it does not contain all the information presented in the EMR). The concept of electronic record sharing arises from the capacity of a health care practitioner to adhere to data generated and kept by someone else. As a result, he may study them and use them within the context of his specialty's practices. He can also contribute information to it. We can look at Quebec, where the QHR (Quebec Health Record) was implemented, and France, where the SMR (Shared Medical Record) was constructed. The medical records implemented in France and Quebec are concrete examples that illustrate initiatives for sharing medical records, aimed at improving care coordination and patient management2.

<sup>1</sup> Les dossiers de santé électroniques: perspectives de la responsabilité médicale. Aout 2008. p. 5. http://www.cmpa-acpm.ca/cmpapd04/docs/submissions-papers/pdf/comelectronic-health-records-f.pdf. ousur www.cmpa-acpm.ca. http://www.cdpdj.qc.ca/publicati ons/memoire-pl-59-renseignements-sante.pdf.

<sup>2</sup> ASIP Santé, Rapport d'activités 2010. PP 26 et 28. http://esante.gouv.fr/sites/default/files/ASIP-RA2010.pdf.

#### 272 Z. Mannai et al.

## • **The local EMR and the shared EMR:**

Local EMRs are used and preserved only by its authors and are entirely their responsibility. Yet, more healthcare personnel have access to shared EMRs, which explains the rise in the number of stakeholders and managers.

The various kinds of medical records are contrasted in the table below (Table 1).


**Table 1.** Kinds of medical records

## • **Shared medical record actors**

Local electronic medical records, which were previously held and accessed solely by a team at a single health facility, are now shared and consulted by new stakeholders, the hosts and the patients.

**Host:** The Host's functionality entails creating an application that combines processing and preservation of patient data3,4.

**Patient:** the establishment, accommodation, and access to the SMR or QHR cannot take place without the patient's authority and agreement. The latter has the right to see his file, as well as the records pertaining to his diagnoses and treatments, and even to obtain information pertaining to his file.

**Healthcare professionals involved:** The public health code mandates that medical practitioners get the patient's permission before consulting their shared EMR. Within the confines of the professional practice of their trade, they are accountable for

<sup>3</sup> Article 19, de la loi sur les services de santé et les services sociaux.

<sup>4</sup> Article L 1111-21 du code de la santé publique.

any confidential information pertaining to the patient. So, they must keep all sensitive information hidden.

## • **Contributions of medical record sharing**

The digital support relating to the medical file is an effective tool which contributes to facilitate the research of the information concerning the patients, to help to process the data and to ensure the storage.

The table opposite (Table 2) shows the contribution of sharing medical record on an individual and collective level [5–7].


**Table 2.** Benefits of medical record sharing

In this sense, we can state that the share of patient file ensures the efficient operation of the care protocol, helps to raise the standard of medical care, ensures data security, and helps stakeholders coordinate while enabling data access. Also, by allowing the patient access to his medical record, it is feasible to give him detailed explanations of his condition and course of treatment. As a result, both the professional and the patient have a vested stake in the outcome.

## **3 Annotation of Medical Record**

Having a precise understanding of what constitutes an annotation is crucial for accurately describing an annotation ontology. In fact, the definition of annotation has sparked diverse perspectives among researchers regarding this concept. These varied viewpoints have played a significant role in enhancing our comprehension of annotation and have laid the groundwork for constructing a pertinent ontology that encompasses the concepts and relationships associated with annotation. As a result, the annotation ontology embodies a comprehensive understanding of the annotation concept by taking into account multiple researchers' perspectives.


## **4 Medical Annotation Model**

Our objective is to propose a formal model of annotation to aid in the resolution of issues that arise while researching annotation systems [16–18].

In 2001, the W3C released the initial draft of the Annotea standard, which aimed to define an annotation model based on the RDF representation. Over time, this model underwent significant enhancements by the dedicated research team, eventually leading to its official adoption as a standard by the W3C in 2014. Despite the evident success of RDF technology in the semantic web domain, upon which Annotea is built, the adoption of Annotea for annotating systems development has been limited among researchers. To address this issue, researchers [19] developed an ontology that drew inspiration from the Annotea model. We utilized this ontology as a fundamental basis and further refined it to tailor it specifically to the medical field.

Our ontology development followed the established methodology employed at Stanford University, which encompasses seven fundamental steps: Domain definition of the ontology, Reuse of existing ontologies from the literature (if available), Definition of the set of important terms, Definition of classes and their hierarchy, Definition of properties of the classes, Definition of attribute facets, and Creation of instances. By following this methodology, we ensured a systematic and comprehensive approach to ontology development for our research [19].

To effectively illustrate the foundational concepts of our ontology, we can take the example of a radiologist who wants to annotate the medical field of his patient. The radiologist, as an annotator belonging to the medical team, has the patient's consent to access their medical record and make annotations. On a specific date, in a particular healthcare facility, using their PC, the radiologist wishes to write an annotation on a specific paragraph of the patient's electronic medical record. The radiologist uses their PC and connects to the healthcare facility's medical information system, which contains the patient's electronic medical record.

While navigating through the record, the radiologist locates the paragraph on which they want to write an annotation. To attach the annotation to the target paragraph, the radiologist uses an anchor, which can be a feature provided by the medical information system or a dedicated annotation tool. They select the relevant paragraph and access the section reserved for scientific annotations within the document. In this section for annotations, the radiologist writes their annotations to capture the important points. They may provide additional information, observations, interpretations, preliminary diagnoses, or other relevant information related to the content of the paragraph. The annotations are saved in the patient's electronic medical record, typically in a structured format and associated with the corresponding anchor, thereby linking the annotation to the specific paragraph. The objective of this annotation is to retain the important points and provide additional information for future reference. This can help other healthcare professionals understand the radiologist's reasoning, clinical decisions, and evaluations when reviewing the patient's medical record.

The proposed ontology includes the following concepts:

## • **Physical attribute**

**Anchor:** An annotation's anchor is a way for an annotator to define where the annotation should be placed within the document.

**Content:** it is a trace of the mental representation that the annotator has developed concerning the object of interest.

**Target:** the target serves as the basis for the annotation. The healthcare professional has the possibility of placing his annotation either in the section reserved for scientific annotations, or in the section reserved for adopted annotations, as the case may be. The patient can keep his notes in a part that has been reserved for him.

The following figure highlights the subclasses of the physical attributes class

## • **Circumstantial attribute**

Circumstantial attribute encompasses all aspects of contact with the environment (date, place, tool, validity).

## • **Partnership Attribute**

These characteristics arise when many types of stakeholders collaborate together on a project.

**Intervenor:** a person who has the authorization of the patient to consult and annotate his electronic medical record is called an intervenor.

**Medical team:** in this case, the speaker is a member of the medical profession.

**Non-medical team**: the intervener, in this case, is not a health professional.

Patient: the patient is a practitioner who has the power to take notes in his medical file.

**Patient:** The patient is entitled to view and make notes in his medical record.

**Host:** it is the role of the host to create an application that includes both the processing and archiving of patient data.

**Patient's family:** Electronic medical records (EMRs) can be viewed and annotated by a patient's family members with the patient's permission.

**Level of analysis:** The term "level of analysis" refers to the process of analyzing the extensive data available on the patient.

**Partnership level:** the different stakeholders must collaborate in order to reach a consensus on a decision. For this reason, they are required to go through the four degrees of partnership.

**Transmission level:** this level signifies the beginning of the discourse phase between the different parties involved in the process.

**Planning level:** at the planning stage, all the different stakeholders work together to plan each step of the care pathway that the patient will go through.

**Action level:** it is about going to the act of care and implementing all the activities previously prepared.

**Informative unit:** an informative unit is a collection of information on an illness or medical event that pertains to a patient.

**Quantifiable piece of information**: for example, the number of hours slept or meals consumed during the course of a day.

**Qualitative piece of information:** patients' emotional levels or their general health information are examples of qualitative data.

**Personalization parameters:** they refer to the characteristics that distinguish one patient from another in terms of reading style and annotation generation process. Personalization parameters, as stated in earlier sections, are as follows: (learning level, media preference, language preference, specific requirements, beliefs, social norms, psychological data).

**Access authorization:**the authority required for a user to have access to protected data or resources in a medical record concerning a certain patient.

## • **Semantic Attribute**

Semantic attributes are the characteristics which make it possible to adapt the annotation to the function to which it applies. They help make sense of annotations by providing context.

**Perlocutionary attribute:** an annotation is produced for a specific purpose by the person who makes it.

**Production objective:** the objectives of the annotator with reference to the creation of his note.

**Reading objective:** these are the annotator's expectations for reading his note.

**Communication object:** the communication object identifies the subject of the annotation, ie the subject of reading the annotation.

**Reuse context:** the reuse circumstance is a representation of how the annotation was intended to be reused.

The (Fig. 1) illustrates the proposed ontology's class hierarchy.

The class hierarchy view displays asserted and inferred class hierarchies. By default, the stated class hierarchy is shown. The asserted class hierarchy view is one of the

**Fig. 1.** Class hierarchy

primary navigational features in PROTEGE. It is displayed as a tree, with the nodes of the tree standing for the classes. Sub/super class relationship in the class hierarchy is represented by a child-parent relationship in the tree.

In our ontology, we have the class "**Medical annotation**" is the parent class of the classes "**Physical attribute**", "**Semantic attribute**", "**Circumstantial attribute**" and "**Partnership attribute**" which are sister classes that appear under their parent class "**Medical annotation** ".

"**Anchor**", "**Target**" and "**Content**" are sister classes derived from the "**Physical attribute**" class. "**Anchor**" is the parent class of the "**Scientific annotation**", "**Adaptive annotation**" and "**Patient annotation**" classes. "**Communication object**", "**Perlcutionary attribute**" and "**Reuse context**" are equivalent classes. They are the daughter classes of the "**Semantic attributes**" class.

The "**Partnership attribute**" class is parent class of the classes: "**Partnership level**", "**Access permission**" and "**Stakeholder**". "**Stakeholder**" derives into two classes which are: "**Medical team**" and "**Non Medical Team**". "**Host**", "**Patient's family**", "**Patient**" are the daughter classes of the "**Non medical"** class. "**Transmission level**", "**Analysis level**", "**Planing level**" and "**Action level**" are the subclasses of the "**Partnership level**" class.

(Figure 2) showcases the object properties within the proposed ontology, providing a visual representation of the relationships established between entities.

#### 278 Z. Mannai et al.

#### **Fig. 2.** Object properties

**Fig. 3.** Data properties

(Figure 3) provides a visual representation of the data properties integrated into the proposed ontology, illustrating the specific attributes and characteristics associated with the entities.

(Figure 4) presents an example axiom for the class 'MedicalAnnotation. This axiom specifies that instances of the class "MedicalAnnotation" must have at least one "PartnershipAttribute", at least one "SemanticAttribute", at least one "CircumstantialAttribute", and at least one "PhysicalAttribute". The quantifier (some) indicates that there exists at least one instance that satisfies the restriction.

**Fig. 4.** Example of axiom

## **5 Conclusion**

In conclusion, the development of an ontology for annotation in the medical domain serves the purpose of creating a standardized and structured representation of medical annotations. By capturing the knowledge and semantics associated with these annotations, the ontology enables easier utilization, sharing, and interoperability across diverse applications and systems. The proposed ontology undergoes discussions and validation by domain experts, and it is essential to emphasize that the validation of ontology performance in medical applications is an ongoing process. User feedback and regular evaluations play a crucial role in identifying areas that require improvements and allowing for adjustments to be made to the ontology accordingly. As part of future work, we plan to implement an annotation system based on this ontology, further enhancing the efficiency and effectiveness of medical annotation processes.

**Acknowledgement.** This work was partially supported by the LABEX-TA project MeFoGL: "Méthodes formelles pour le Génie logiciel".

## **References**


19. Noy, N.F., Mcguinness, D.L.: Développement d'une ontologie 101: Guide pour la création de votre première ontologie. Université de Stanford, Stanford, Traduit de l'anglais par Anila Angjeli (2000). http://www.bnf.fr/pages/infopro/normes/pdf/no-DevOnto.pdf

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Enhancing the Measurement of the Range of Motion Using Multi-camera Learning Approaches**

Giseop Noh1(B) , Juyoung Ahn2, and Bogja Jeoung3

<sup>1</sup> Department of Software Convergence, Cheongju University, 298, Daeseong-ro, Cheongwon-Gu, Chungcheongbuk-do, Cheongju-si, Republic of Korea kafa46@cju.ac.kr

<sup>2</sup> Marketian C&E, 155, Misagangbyeonhangang-ro, Hanam-si, Gyeonggi-do, Republic of Korea <sup>3</sup> Department of Exercise Rehabilitation, Gachon University, 191 Hambakmoero, Yeonsu-gu,

Incheon, Republic of Korea

bogja05@gachon.ac.kr

**Abstract.** Measuring the range of motion (ROM) is one of the important tasks in medical or healthcare sectors. However, person-to-person measurement is timeconsuming and requires resources. In this paper, we propose an approach to estimate ROM using machine learning algorithm equipped with computer vision technology based on data-driven experiments. We describe the setup to gather experimental dataset to learn the angle of human joints in 2D space. From the extensive experiments and multi-linear regression learning approach, our method can reduce estimation error by 11.1% in average. We draw conclusions that machine learningbased, data-driven approaches could predict ROM better than using vision camera only.

**Keywords:** Range of Motion · Machine Learning · Multi-camera · Multi-linear Regression · Rehabilitation

## **1 Introduction**

The measurement of the range of motion (ROM) is critical in health care as well as clinic treatment professionals. Also, the importance of ROM is increased in the field of rehabilitation exercises (RE) from surgery, disability, accidents and so on. The measurement devices of ROM (MDROM) can be applied to yield ROM information only if ROM values are measured by well trained and experienced persons with relatively high precision. However, P2P measurement with MDROM has several drawbacks. First, P2P with MDROM needs resources such as education, training for professional knowledge which follows cost and certain period of time. Second, it has constraints since at least two persons must be met in same time and physical space. Third, even though a person who measures is well trained, the results are not always be consistent since it depends on the persons.

With the rapid progress of artificial intelligence (AI) in image processing recently, there is a possibility that these shortcomings can be compensated. AI bears various areas. One of the AI-based image processing areas is the detect pose landmarks in an image or video streaming. MediaPipe provides open application programming interfaces (API) [3], called 'ML kit' to software developers for the purpose of ease and fast building applications. Right figure of Fig. 1 is an example figure provided by 'ML kit pose detection API' (we will use 'ML-kit' for brevity hereinafter). ML-kit provides 33 coordinates of body landmarks including entire body (face, arms, legs, etc.) For each coordinate is consisted of three values representing x-, y-, and z-axis of a landmark.

Image processing powered by machine learning algorithms show the powerful ability to replace P2P MDROM. Also, researchers show practical implementations in various areas such as underwater running [4], marker less system [5], pose matching [6], measuring single-leg squat kinematics [7] and so on.

**Fig. 1.** Two examples for measuring human body. P2P based ROM measurement (left) [1] and Machine learning based pose (landmarks) detection [2].

Besides medical and rehabilitation, RE machines are already on commercial market. For examples, multi-purpose device including rehabilitation, therapy and fitness is introduced in [8], mobility-abled rehabilitation exercise equipment [9], rehabilitation machine for people with disabilities [10], home healthcare device with IoT [11], and so on. However, almost RE doesn't have functionalities for ROM using computer vision systems. We argue the main reason is that output data from computer vision is not reliable and hard to implement onto RE in terms of accuracy compared to P2P approaches.

We focus on the machine to be used in RE using machine learning to facilitate ROM with vision systems in this paper. We developed new RE machines during our research project mainly focused on upper body rehabilitation exercise/training machine. To exploit the computer vision systems, we installed camera devices and developed analysis software for various motions in rehabilitation exercise. One of the challenging problems is that since the images from rehabilitation training is 2D-coordinate, it is not trivial to estimate body angles in actual 3D-coordinates in real world. The easiest approach is to apply multiple cameras to measure ROM on RE. However, mounting several cameras is disadvantageous in terms of RE operation, maintenance, data processing of computing devices inside RE, and movability to elsewhere. We addressed the problem-solving approach utilizing machine learning to overcome insufficient 2D space information in 3D world.

## **2 Problem Definition**

## **2.1 Problems of 2D Information in 3D World**

To give better experience and systemic rehabilitation exercise, we developed a RE, which can monitor, and analysis 13 types of motions based on computer vision system. We report the possible rehabilitation actions of RE we developed in Table 1. The RE is consisted of exercise devices, information screen, computing devices, and a camera. The detailed figure is shown in Fig. 2.


**Table 1.** Supported rehabilitation motions in our RE.

**Fig. 2.** The configuration of the RE we developed. A camera is installed on head area of RE

We utilized ML-kit to detect landmarks of human body, which supports to extract 3D coordinates of 33 landmark points in real time. For more detailed information, refer to [12]. Even though the ML-kit produce 3D-coordinates with the tuple values, (x, y, z). The x and y are the landmark coordinates normalized to [0, 1], and z is the depth at the midpoint of hips.

Problems arises here is that if we consider only 2D image, the estimation of ROM has errors by nature. For example, if we measure angle of in shoulder-elbow-wrist of ROM in left elbow anterior up and down (Motion # 5 in Table 1) using only one camera as shown in Fig. 1, the angle is equal to 185°. However, if we measure the angle after moving the location of the camera to the left-side of the person the value of angle is equal to 118° which shows not acceptable errors. For the intuitive understanding, we portrayed the problematic situation in Fig. 3.

**Fig. 3.** The landmarks from front camera (right) and those from the left side of person

185 and 118° from the left- and right-hand figure of Fig. 3, respectively. Note that we applied well known mathematical equation to compute angle between three points in 2D space stated in Eq. (1).

$$angle = \left| \frac{radians}{180} \times \pi \right|,$$

$$radians = \arctan\left(\frac{c\_\text{y} - b\_\text{y}}{c\_\text{x} - b\_\text{x}}\right) - \arctan\left(\frac{a\_\text{y} - b\_\text{y}}{a\_\text{x} - b\_\text{x}}\right) \tag{1}$$

where *a*, *b*, *c* are the 2D coordinates such that *a* = *ax*, *ay* ,

*b* = *bx*, *by* , *c* = (*cx*, *cy*) and |·| is absolute function, respectively.

This example is just for one situation among 13 rehabilitation motions. To mitigate errors under information loss between the 2D and 3D space, we introduce machine learning approach in this paper.

#### **2.2 Problem Formulation**

We restrict the problem such that it is just for one situation among 13 rehabilitation motions. To mitigate Given a dataset *D*, we want to learn how to predict more precise coordinates information to estimate ROMs. As our notation, we define observed dataset *D* = {(*oi*, *yi*)} *N <sup>i</sup>*=1, where *oi* is observed information and expressed by vector space, *yi* is label (true value), (·) is tuple containing a pair of data, *N* is the total number of data tuples, respectively. The maximum dimension of our data is R1×24, consisting of 24 = 8 *points*(*landmarks*) × 3(*x*, *y*,*z*). Also, we select the first step of this problem applying simple machine algorithm, multiple linear regression (MLR), which is simple but fast learning algorithm. We will use root mean squared error (RMSE) to evaluate our approach, which can be stated as Eq. (2)

$$RMSE = \frac{1}{N} \sqrt{\left(\mathbf{y}\_l - \hat{\mathbf{y}}\_l\right)^2},\tag{2}$$

*where y <sup>i</sup> is the predicted value by MLR.*

## **3 Experimental Setup**

To gather *D*, we conducted experiments for each rehabilitation motions as stated in Table 1. The experiments are conducted via 13 test subjects (refer Table 2). Each test subject performs 5 times for each rehabilitation motion.

**Fig. 4.** Camera positions in experiments, where the numbers on solid black lines are distances between a camera and test subject, the red numbers beside cameras are the heights of cameras from ground, the figures inside of yellow circles are the camera indexes.

To gather various possible combinations of camera position, we installed four additional cameras with different distances, heights, angle at which looking at the test subject. Finally, we have five camera configuration including rehabilitation measuring equipment (marked as ME in Fig. 4). We simulate five formations which are 9, 10, 12, 2, 3 o'clock direction from the line of sight of the test subject as portrayed in Fig. 4. Each camera record and save videos in remote storage (server) at the same time to yield the dataset *D* of machine learning. After finishing all experiments, we facilitated ML-kit algorithms to extract all possible coordinates among 33 landmarks. Since the camera yields 60 FPS (frame per second), we pullout 60 images per second. Finally, we secured 1,816,704 images from the experiments. Using the images, we extract all landmarks.


**Table 2.** Anonymized test subjects

ML-kit extracts 1,773,234 coordinates of landmarks after dropping NONE values which failed to get coordinates by ML-kit, which is*N* = 1, 773, 234 in *D* = {(*oi*, *yi*)} *N <sup>i</sup>*=1. The *xi* is a vector with size R1×<sup>24</sup> consist of 3D coordinates (*x*, *y*,*z*) from landmark #11, #12, #13, #14, #15, #16, #23, and #24 in Fig. 4. To acquire *yi* (label values, or equivalently ground truth values), we build mapping table for each action in Table 3.


**Table 3.** Camera mapping to generate *yi*

Note: 1landmark: index is referenced from Fig. 4, 2coordinate: subscript of *x*, *y* in coordinates is the index of landmark, and 3motion: index is referenced from Table, all value in table is the camera index from Fig. 4

We set *yi* to be R1×<sup>12</sup> since the final task of prediction is elbow angle of right and left in this paper. To compute right and left angle, we apply Eq. (1) landmark #12, #14, #16 for right elbow and #11, #13, #15 for left elbow angle, respectively. We divided *D* into train and test set applying ration 80% and 20%, respectively. Finally, we trained train set by applying MLR, and predicted *yi* after data learning (i.e., after all train steps).

## **4 Result Analysis**

## **4.1 Prediction Performance in RMSE**

We predict *yi* of test set (landmark #11, #12, #13, #14, #15, #16, #23 values) using *xi* of test set via MLR trained model. Let the predicted value be *y <sup>i</sup>*. Then we compute RMSE errors between *yi* and *y <sup>i</sup>* using Eq. (2). The overall error is 0.07064 and the least error is 0.02726 shown in right wrist *x*-coordinate and the largest error is 0.11332 found in left elbow *y*-coordinate, respectively. The RMSE comparison is shown in Table 4 and visual representation is portrayed in Fig. 5.

**Fig. 5.** Prediction RMSE errors between *yi* and *y i*.

## **4.2 Prediction Errors Between Left- and Right-Hand**

To compare the prediction behavior between right and left, we report scatter plots of shoulder and elbow in Fig. 6. We omit other plots due to the page limit.


**Table 4.** Prediction RMSE errors between *yi* and *y i*.

**Fig. 6.** Comparison prediction errors between left- and right-hand.

#### **4.3 Performance of ROM Measurement**

Since elbow angles is only considered among ROM in this study, we averaged all RMSEs using observed coordinates and predicted coordinates via MLR, respectively. Note that the ground truth is *yangle*\_*from*\_*groundTruth* using *y* in Sect. 3 and we have two types of *y* such that *y angle*\_*from*\_*observe* and *y angle*\_*form*\_*MLR* for angle comparison. The comparison between two angle estimation is reported in Table 5.


**Table 5.** Prediction RMSE errors between elbow angle estimation approach.

\* Performance enhancement is estimated such that *<sup>a</sup>*−*<sup>b</sup> <sup>a</sup>* × 100

## **5 Conclusions**

It is an important task to estimate ROMs as accurate as possible in rehabilitation exercise. Since the computer vision technology is plausible to apply into detecting human landmarks, it is promising in motion analysis in that cost effective and time saving. However, detecting and measuring the landmarks is not trivial yet. We addressed the way of measuring ROMs facilitating machine learning approach. In this paper, simple MLR is used to learn human elbow angle based on data. We developed how the machine learning approaches can be used and generate training and test dataset. Even though the test cases and test subjects are limited, we think the possibility of our approach could be addressed sufficiently. After implementation and experiments, we showed that machine learning based ROM estimation is possible approach in terms of enhancing the accuracy. If more enhanced algorithm is tested and evaluated in our domain, the better enhancement will be achieved. We leave this as our future research.

**Acknowledgements.** This study was supported by the Translational R&D Program on Smart Rehabilitation Exercises (NCR-TRSRE-Eq. 01A), National Rehabilitation Center, Ministry of Health and Welfare, Korea.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Short Contributions: Wellbeing Technologies**

# Towards a Personalized Business Services Recommendation System Dedicated to Preventing Frailty in Elderly People

Ghassen Frikha1(B) , Xavier Lorca<sup>1</sup>, Hervé Pingaud<sup>2</sup>, Christophe Bortolaso<sup>3</sup>, Katarzyna Borgiel<sup>3</sup>, and Elyes Lamine1,4

<sup>1</sup> University of Toulouse, IMT Mines Albi, Industrial Engineering Center, Route de Teillet, 81013 Albi Cedex 9, France

{ghassen.frikha,xavier.lorca,elyes.lamine}@mines-albi.fr

<sup>2</sup> CNRS-LGC, Champollion National University Institute,

University of Toulouse, Albi, France

herve.pingaud@univ-jfc.fr

<sup>3</sup> Research and Innovation Division, Berger-Levrault, Labège, France

{christophe.bortolaso,katarzyna.borgiel}@berger-levrault.fr

<sup>4</sup> University of Toulouse, ISIS, Champollion National University Institute, Rue Firmin-Oulès, 81104 Castres, France

Abstract. Frailty is a clinical syndrome associated with ageing that characterizes an intermediate state between robust health and loss of autonomy. To preserve the abilities of older adults and prevent dependency, it is important to identify and evaluate their frailty. This approach is part of a dependency prevention strategy, based on a thorough understanding of their medical, social, and living environment. This understanding is usually acquired through significant data collection using standardized evaluation surveys. The obtained data is then analyzed to provide personalized recommendations for the beneficiaries' lifestyles. Our article presents the concept of frailty and a personalized recommendation system aimed at helping citizens prevent frailty. This system uses an innovative self-assessment approach designed for older adults, without necessarily involving healthcare professionals.

Keywords: Recommendation systems *·* Elderly *·* Frailty *·* Recommendation algorithm *·* Personalized recommendations

## 1 Introduction

As the global population ages, preserving physical, cognitive, and psycho-social abilities is essential for healthy aging. Age-related declines in these abilities can lead to frailty, a syndrome characterized by a decline in physiological systems, a decrease in functional reserves, and vulnerability to stress [4]. Unlike disability, frailty can often be reversed or minimized with personalized interventions [16] aimed at preserving abilities.

Managing frailty involves a three-step process, including frailty screening and the definition of a personalized intervention plan, as well as follow-up to prevent or delay disability and dependence and improve the quality of life of elderly individuals [8]. Several European countries, including France, are developing innovative solutions for elderly individuals and their caregivers to improve their quality of life in the context of the Silver Economy [5]. Our work is aligned with this goal of promoting ∠ Successful Ageing ∠ and behaviours that promote good health by creating a personalized service recommendation engine for the prevention of frailty.

In this article, we provide an overview of the medical context and issues related to population ageing that motivated our research. We also discuss the link between frailty and intrinsic capacities. In the third section, we present our innovative recommendation platform, ∠ Senselife ∠, which is based on a thorough analysis of user profiles and provides personalized recommendations for services to meet their specific needs. Finally, we conclude the article by summarizing our results and discussing future perspectives for practical applications to improve the quality of life of elderly individuals.

## 2 Context

In order to comprehend the driving forces behind our research and the obstacles to be surmounted in advancing a recommendation engine, it is necessary to acquire a foundational understanding of the medical knowledge surrounding the factors that contribute to loss of autonomy in older adults, as well as how this can be addressed in an ongoing process of care.

## 2.1 Frailty and Intrinsic Capacities

The syndrome of frailty is characterized by a decline in physiological reserve capacities that impact adaptation mechanisms to stress. Psychological, social, economic, and behavioural factors, as well as comorbidities, influence its clinical manifestation. It is linked to increased mortality risk, disability, falls, hospitalization, and institutionalization [12]. Consequently, early interventions to prevent the progression of frailty and disability are crucial. Frailty evaluation allows older adults to be classified into three categories: robust, pre-frail, and frail.

Prevalence of frailty and pre-frailty in EU countries is around 15% and 50%, respectively, among those aged 65 and older based on the study [13].

The concept of intrinsic capacities (IC) was introduced by the World Health Organization (WHO) to enable a more comprehensive and suitable evaluation of the ageing population. IC refers to the sum of all physical and mental abilities an individual can use throughout their life and includes five domains that summarize their capacities [1].

Understanding the relationship between IC and frailty is crucial. Indeed, frailty is a state of vulnerability that can occur in elderly people due to a decrease in intrinsic capacities. Thus, intrinsic capacities are the basic capacities that an elderly person needs to function independently.

Detecting and evaluating frailty is challenging due to the variety of available measurement tools. A review by [6] identified 29 different measures, ranging from simple screening instruments to complex ones. Meanwhile, the assessment of intrinsic capacities (IC) in older adults is a growing field. A proposed scoring system by [9] assesses IC in older adults and highlights the complementarity between measuring frailty and IC. Conducting a comprehensive evaluation of both frailty and IC is crucial for enhancing the quality of life and well-being of older adults.

#### 2.2 Frailty Personalized Intervention Plans Implementation Challenges

To manage frailty in older adults, a personalized intervention plan involving collaboration between multiple stakeholders is necessary (up to 13 actors can be included) [2,15]. However, its implementation can be complex and resourceintensive, limiting accessibility to a significant number of older adults.

Creating a personalized intervention plan for frailty in older adults requires a comprehensive evaluation of the individual's situation, including measures of intrinsic capacity and frailty. Identifying available services and personal information, such as preferences and history, also play a crucial role in developing an effective plan.

To develop an accurate and dynamic personalized intervention plan, a decision support system capable of recommending services taking into account their medical, social, and environmental profile would be useful. To this end, we will present our recommendation platform, called Senselife, in detail in the following section. It will enable an interactive, precise, and personalized response to the specific needs of each older adult, based on their frailty situation and personal preferences.

## 3 Seneselife Platform

The goal of the Senselife platform is to address the challenge of managing frailty in older adults by utilizing self-evaluations to provide personalized recommendations tailored to their needs and preferences. The platform offers a variety of features such as the ability for users to express their living conditions and match supply and demand for services. Additionally, it provides a selection of services aimed at strengthening the functional abilities of older adults based on their intrinsic capacity. However, the platform needs to define and use knowledge to create tailored recommendations. This requires identifying key knowledge to characterize users and services, representing it, and integrating it with the software structure to improve recommendations. The scheme of the platform is illustrated in Fig. 1, which is composed of three main components: data collection,

Fig. 1. Senselife Platform

recommendation generation, and services consumption. Citizen satisfaction and service utilization are used to adapt future recommendations.

The platform's recommendations will heavily rely on the data it collects. The primary sources of this data will be the citizen's self-evaluation survey responses and connected devices. The survey's questions are chosen from different frailty surveys. They were selected based on whether the citizen can answer them alone (for example, ADL<sup>1</sup> questions) and whether they provide an understanding of the user's condition. Additionally, service providers are responsible for defining their services on the platform, which is the second input for the recommendation engine.

The collected data will feed the recommendation engine, which will present the citizen with a list of services that meet their needs, factoring in their personal characteristics and preferences.

#### 3.1 Citizen Profile

In the context of recommendation, user profiling is the process of collecting and analyzing information about users to understand their preferences, habits, and behaviors. It is a key element in improving the relevance and quality of recommendations [7]. To this end, we have developed a conceptual model of the user profile for our platform, which is the elderly citizen in our case. This

<sup>1</sup> ADL: Activities of Daily Living survey. https://www.alz.org/careplanning/downloads/katz-adl.pdf.

model takes into account all relevant elements to characterize the citizen while remaining extensible. As shown in Fig. 2a, a citizen profile consists of 5 parts:

Fig. 2. Excerpt from the Senselife meta-model


Although our platform can generate an estimated frailty score based on the data collected, it's essential to acknowledge that this score may not be as accurate as those obtained by healthcare professionals. For example, the 6 questions of the ADL survey can help us to understand a specific aspect of the person and have a frailty score on a scale of 6. Other scores are also calculated which will give us more comprehension about the person. In summary, our primary goal is to gather information about the citizen to better understand their situation and develop personalized recommendations, rather than providing an exact frailty score.

## 3.2 Services

Service providers are responsible for creating services on the platform. We offer a service model that will assist providers in creating their services while enabling the recommendation engine to efficiently comprehend the input data. Figure 2b displays this model.


## 3.3 Recommender Engine

The Senselife platform employs a recommendation engine that provides services for frailty prevention. With the help of algorithms, this recommendation engine suggests specific services to elderly individuals based on their personal information and circumstances. The engine evaluates the data provided by the elderly person and suggests the most relevant services using the knowledge bases and the rule base available. The knowledge base contains the different personal characteristics, needs, the service types and the relation between them. The rule base contains the rules that will be used to manipulate the data in the knowledge base. The validation of this rules are based on the expertise of the health-care professionals.

There are various recommendation techniques, such as collaborative filtering algorithms, content-based algorithms, and knowledge-based algorithms. The choice of algorithm depends on the specific requirements and objectives of the recommendation system [14]. Our recommendation engine uses a knowledgebased recommendation technique that enables increased personalization. This technique analyze information about users, items, or their relationships to suggest relevant items using a knowledge base. This approach infers the relationship between a user's needs and a possible recommendation, as explained in [3].

Fig. 3. Recommendation Engine overview

When we zoom-in on Fig. 1 in the recommendation engine section, we can see Fig. 3, which provides a schematic representation of the knowledge-based recommendation technique. Ultimately, the elderly person receives a filtered recommendation based on their preferences.

## 4 Conclusion and Perspectives

The article focuses on the vulnerability of elderly people and suggests personalized service recommendations for frailty prevention. The proposal is a recommendation platform that assesses the older person in a multidimensional way, taking into account both intrinsic abilities and measures of frailty.

This approach is crucial in understanding an individual's needs to determine the most suitable services. The platform uses a self-evaluation approach, which is user-friendly and accessible to everyone. The recommendation technique of the engine is presented (knowledge-based one), with an explanation of its different steps and the knowledge bases used.

The platform provides a holistic solution to ensure that elderly people receive the necessary support without relying solely on healthcare professionals. To achieve this goal, the approach requires attention to various aspects. We identify several research avenues for future work, including validating citizen and service profile models, verifying the effectiveness of the incremental approach, and considering the optimization and complexity aspects of the algorithms used by the recommendation engine.

## References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **How Dual-Tasking Using VR Affects the Elderly: A Systematic Review**

Jun-Su Choi1, Min-Ye Jung2(B) , and Junghun Aj Kim<sup>3</sup>

<sup>1</sup> Department of Occupational Therapy, Graduated School, Yonsei University, Seoul, South

Korea

12junsu@yonsei.ac.kr

<sup>2</sup> Department of Occupational Therapy, College of S/W Digital Healthcare Convergence, Yonsei

University, Seoul, South Korea

Minye@yonsei.ac.kr

<sup>3</sup> Department of Occupational Therapy, Far East University, Eumseong-Gun, Gamgok-myeon 27601, South Korea

**Abstract.** This study systematically reviewed research that applied dual-task interventions using VR technology for balance and cognitive tasks among older adults. Ten databases were searched following the PRISMA guideline, and 18 studies were selected based on their evidence levels and risk of bias. The selected studies consisted of 10 RCTs, 6 non-randomized controlled trials, and one each of case and qualitative studies. The types of balance tasks included standing and sitting postures, and all studies utilized cognitive tasks that required concentration. A total of 30 physical assessment tools and 42 cognitive assessment tools were identified. The results showed that virtual reality interventions improved balance and cognitive abilities among older adults and had a positive effect on fall prevention. These findings suggest that VR technology can be an effective tool for improving the physical and cognitive health of older adults.

**Keywords:** Virtual reality · Dual-task · Cognition · Balance · Elderly

## **1 Introduction**

VR is being used by integrating various technologies to experience realistic representations in a virtual space. Research on dual-task training using VR has been conducted on a diverse range of subjects, including individuals with Parkinson's disease, as well as children, adults, and the elderly of all genders and ages. While numerous interventions involving VR programs and devices for dual-task training have been conducted in foreign countries, research utilizing these technologies for such training is still lacking in Korea.

Dual-task training involves performing cognitive tasks while engaging in motor activities, creating a situation of cognitive-motor interference where efforts are divided between two tasks [1]. This induces a state of conflict, requiring high levels of attention and concentration during the execution of motor tasks.

Dual-task training is easily implemented through VR technology, allowing for personalized exercise, rehabilitation, and interventions for individuals. This approach can be particularly beneficial for older adults and patients requiring rehabilitation, as it can stimulate interest and promote engagement, leading to enhanced daily functioning and recovery [2–4].

It should be noted, however, that the application of VR technology also has its disadvantages, with cybersickness being identified as a common issue among some users [5]. This is a particularly critical issue for individuals with disabilities and the elderly [6]. Head-mounted displays (HMDs) have been identified as a major cause of cybersickness [7, 8]. Therefore, for the elderly, VR programs utilizing technologies other than HMDs are more suitable.

This study aims to systematically review domestic and international research on dual tasks that combine balance and cognitive tasks using VR for older adults, examining the level of evidence, types of tasks used, validation tools, intervention duration, and effects. It intends to provide directions for VR-based dual task research in Korea and offer evidence-based approaches for researchers working with older adults.

## **2 Methods**

#### **2.1 Literature Selection and Study Design**

This study examined the effects of dual tasks combining balance and cognitive tasks using VR devices on older adults. The literature selection process was conducted using the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines. The level of evidence of the selected studies was evaluated using the evidence level assessment developed by Abresman, Scheer, and Lieberman (2008) and divided into five levels [9].

#### **2.2 Database and Search Terms**

To find relevant studies for the topic, a three-step search strategy was employed between October and December 2022. Firstly, previous research trends were identified using medical and health literature analysis databases (PubMed, CINAHL) and Google Scholar. Secondly, search terms were selected. Lastly, systematic reviews that had been previously conducted on topics related to VR were consulted to select the paper search databases [10, 11].

For domestic databases, the search was conducted using the following search terms: "Elderly" AND "Virtual reality" AND "Balance" AND "Dual task" AND "Cognition" on the Research Information Sharing Service (RISS). For foreign databases, the search was conducted using the same search terms on ACM Digital Library, SCOPUS, PubMed, ScienceDirect, CINAHL, EMBASE, Web of Science, PsycINFO, and IEEE Xplore. Studies conducted between 1960 and December 2022 were included. After collecting all papers retrieved from each database, the final papers were selected according to the following selection and exclusion criteria. In order to ensure that the selected studies were relevant to the research topic, a set of selection criteria and exclusion criteria were used. The inclusion criteria were: (1) Studies that targeted older adults, (2) Studies that used virtual reality to mediate dual tasks related to balance and cognitive tasks, (3) Papers written in Korean or English, and (4) Studies for which the full text is available. The exclusion criteria were: (1) Studies that did not use VR technology as an intervention, (2) Studies that used head-mounted display (HMD) devices for mediation, (3) Studies that targeted older adult patients with central nervous system disorders or chronic diseases, (4) Systematic reviews and meta-analyses, and (5) Thesis studies and protocol studies.

## **2.3 Method of Results Analysis**

For the selected studies, we analyzed the types of dual tasks combining balance and cognitive tasks that elderly individuals perform using VR devices. First, we divided the types of balance and cognitive tasks performed during dual-task interventions using VR and compiled them into a table. Second, we analyzed the assessment tools used to evaluate the effectiveness of dual-task interventions using VR and compiled the results into a table.

## **3 Results**

## **3.1 Selection of Studies**

A total of 3983 studies were retrieved from the initial search of 10 databases. After removing duplicates (n = 421) using EndNote and manual review, we assessed the titles and abstracts of the remaining papers for relevance to Virtual Reality, dual-task, and balance, and excluded irrelevant papers, such as review and meta-analysis papers, papers without full texts, papers with inappropriate study designs, and papers with participants not relevant to the research topic. After the first screening, we collected 137 studies and further excluded irrelevant papers by checking the full texts, resulting in the final selection of 18 papers (Fig. 1).

## **3.2 Level of Evidence**

The evaluation of the level of evidence for the final selection of 18 studies revealed that 10 studies were classified as level 1, accounting for 55.6% of the total. Three studies each were classified as level 2 and level 3, accounting for 16.7% each. Additionally, there were two studies (11.0% in total) classified as level 5 in the evidence level, which included one case report and one qualitative study (Table 1).

## **3.3 Types of Balance and Cognitive Tasks in Dual-Task Interventions**

The analysis of the dual-task intervention using VR, divided into balance task types and cognitive task types, showed that the balance task type had the highest number of tasks in the standing position, with 15 tasks (83.3%). Two tasks (11.1%) were performed in the seated position, and one task (5.6%) included both standing and seated balance tasks. For the cognitive tasks, attention had the highest number of tasks with 18 (32.1%), followed by memory with 16 (28.6%), and executive function with 14 (25.0%). Planning tasks accounted for 4 (7.1%), and calculation and inhibition/interference control each accounted for 2 (3.6%) (Table 2).

**Fig. 1.** Flow Diagram of the Literature Selection Process



**Table 2.** Classification of Balance Types and Cognitive Types of Double Tasks

#### **3.4 Analysis of Evaluation Tools for Assessing the Effects of VR-Based Dual-Task Interventions**

Through the selected studies, a total of 30 evaluation tools were identified for measuring the effects of VR-based dual-task interventions on physical functions of older adults, including 26 for balance assessment and 4 for motor function evaluation. Among the balance assessment tools, Mini-BESTest and TUG were the most frequently used, with 3 studies (10.0%) each. Motor function evaluation was conducted with 4 tools, including fNIRS. A total of 42 evaluation tools were used for cognitive function assessment, which was further categorized into executive function, memory, attention, MCI screening test, and overall cognitive function assessment tools. TMT was the most commonly used tool (8 studies, 19.0%) for executive function assessment, while WMS-R was the most frequently used tool (3 studies, 7.1%) for memory assessment. All 5 studies (11.9%) assessing attention utilized Stroop. MoCA was the most frequently used tool (6 studies, 14.3%) for MCI screening. For overall cognitive function assessment, fNIRS and VFT were the most commonly used tools, each in 2 studies (4.8%) (Table 3).


**Table 3.** Measured Dependent Variables and Outcomes

(*continued*)


**Table 3.** (*continued*)

ADAS: Alzheimer's Disease Assessment Scale, BBS: Berg Balance Scale, CCVLT: The Chinese version of the California Verbal Learning Test, CNT40: Computerized nerocognitive function test, CWT: The Chinese version of the Stroop Colour and Word Test, DSST: Digit Symbol Substitution Task, DST: Digit Span Test, EEG: Electroencephalogram, EXIT-25: The Executive Interview 25, FES-I: The Falls efficacy scale, FGA: Fuctional Gait Assessment, fNIRS: Functional NIRS, FSST: Four Square Step Test, GUG: Get-up-and-go Test, HbO2: Oxygenated Hemoglobin, LoS: Limits of Stability Scores, MBT: Mind-Body Trainer, K-MMSE: Korea-Mini Mental State Exam, Mini BESTest: Mini Balance Evaluation Systems Test, MMSE: Mini Mental State Exam, MoCA: Montreal Cognitive Assessment, RBANS: the Repeatable Battery for the Neuropsychological Status, SFT: Senior Fitness Test, SPPB: Short Physical Performance Battery, TMT-A: Trail making test-A, TMT-B: Trail making test-B, TOVA: Test of Variables of Attention, TUG: Timed Up and Go, VFT: Verbal Fluency Test, VST: Victoria Stroop Test, WAIS-R: Wechsler Adult Intelligence Scale-Revised, WAIS-4: Wechsler Adult Intelligence Scale-4, WMS-R: Wechsler Memory Scale-Revised, YBT: Y-Balance Test, 10MWT: 10m Walking Test.

## **4 Conclusion**

In this study, we examined research that provided dual-task interventions combining balance and cognitive tasks using VR. We analyzed the types of balance and cognitive tasks used to provide dual-task performance through VR, evaluation tools for analyzing the effects of dual-task performance using VR on the elderly, and the physical, cognitive, and emotional effects, fall prevention rates, and usability of dual-task performance with VR technology. VR dual-task interventions for elderly individuals may have positive effects on their physical balance and cognition. However, it is important to select appropriate challenging tasks based on their cognitive abilities and to create diverse content that can stimulate their interest. Furthermore, it has been observed that while there are numerous existing assessment tools for evaluating physical and cognitive performance separately in the literature review of previous studies, there is a lack of assessment tools specifically designed to evaluate dual-task performance itself. VR technology facilitates the integration of physical and cognitive tasks in dual-task paradigms. Therefore, there is a need for research on the reliability and validity of assessment tools that can evaluate dual-task performance using VR technology.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Comparative Usability Testing Between Lightweight Power Wheelchairs: Focused on People with Physical Disabilities in the Community**

Yeong-hun Kwon1, Dong-wan Kim1, and Jongbae Kim2(B)

<sup>1</sup> Department of Occupational Therapy, The Graduate School, Yonsei University, Seoul, South Korea Yhkwon97@yonsei.ac.kr

<sup>2</sup> Department of Occupational Therapy, College of S/W Digital Healthcare Convergence, Yonsei University, Wonju, South Korea jongbae@yonsei.ac.kr

**Abstract.** There have been studies on Lightweight Power Wheelchairs (LPW) that compensate for the disadvantages of large and heavy powered wheelchairs for the activities of the people with physical disabilities in the community, but there is a lack of research to understand usability. Accordingly, we compared the usability of LPWs developed in Korea for 5 wheelchair participants and caregivers to find out their effectiveness, efficiency, and satisfaction. As a result of the usability testing, there was a difference between the two LPWs in effectiveness and satisfaction. LPW1 was analyzed as grade B/'Good' usability LPW2 was analyzed as grade A/'Best Imaginable' usability. The usability of the heavier LPW2 was highly appreciated. This can be interpreted that users feel a sense of stability in the LPW with a certain amount of weight. In future studies, it is considered necessary to study the appropriate weight of LPW that users can feel stable.

**Keywords:** Lightweight Power Wheelchair · Usability Testing · People with Disabilities

## **1 Introduction**

According to the Ministry of Health and Welfare of Korea in 2020, the rate of physical disability among community-dwelling people with disabilities were the highest [1]. People with severe physical disabilities use mobility assistive devices when moving to the community. Among them, powered wheelchairs are generally used when users cannot propel independent wheelchairs, and according to Lee's study, powered wheelchair users reported low satisfaction with the weight and size of powered wheelchairs and postservice [2]. Accordingly, Lightweight Power Wheelchairs (LPW) with the function of powered wheelchairs at home and abroad are being studied [3–6].

Usability is specified according to the ISO-9241-11 guidelines, and usability testing is to investigate the interaction between users and products in the process of constructing and executing usability scenarios [7]. According to Kim, in order to increase the usability of assistive devices, it is important to reflect the needs of the people of disabilities from the early stage of development, and to this end, usability testing is required [8]. However, research on usability testing of LPW is insufficient at domestic and abroad.

Kim conducted a comparative usability testing on a hybrid manual-powered wheelchair but focused on wheelchair driving [9]. In the case of people with severe physical disabilities, it is difficult to manage independent powered wheelchairs, so help from caregivers is essential. Accordingly, it is important to find out the usability of wheelchairs from the perspective of caregivers in product development and improvement [10].

Therefore, in this study, we would like to compare the usability of two LPWs recently developed in Korea in accordance with the ISO-9241-11 guidelines from the perspective of users and caregivers. Through the analysis of the usability testing results, we intend to derive factors to be considered in order to increase the usability of users and caregivers and to provide basic data for future LPW research.

## **2 Method**

## **2.1 Participants**

Participants in this study were recruited from people with physical disabilities who use powered wheelchairs. According to Nelson's research, 5 people with physical disabilities who agreed to participate in the study were finally selected [11]. The general characteristics of the study participants are shown in Table 1.


**Table 1.** Characteristics of participants

#### **2.2 Inclusion and Exclusion Criteria**

If the participants met (1) A power wheelchair can be used independently and (2) There is no cognitive difficulty in understanding the research process and agrees to the research voluntarily, they were included in the study, and if they met (3) Do not currently use a powered wheelchair and (4) Participant currently has difficulty using an power wheelchair, they were excluded from the study.

#### **2.3 Tools**

#### **2.3.1 Lightweight Power Wheelchair**

In this study, two types of LPW manufactured in Korea were used (Fig. 1). Unlike existing power wheelchairs, it is light in weight and can be folded and stored, so it can be carried in a vehicle. Another advantage is that the battery can be easily installed and removed. It is equipped with the same driving device as the existing power wheelchair and can be steered and speed controlled using a controller such as a joystick.

Both LPWs have the same 8-inch front wheel and 14-in. rear wheel. The battery capacity of the two LPWs is the same at 24 V and 16 Ah. The motor specifications of the two LPWs are the same at 250 W and 4600 rpm. The controller of both LPWs is the same. The weight of the two LPWs is different. The weight of LPW1 is 29.7 kg, and the weight of LPW2 is 31 kg. The frame sizes of the two LPWs are different. LPW1 is 105–58–84 cm (length, width, height), LPW2 is 103–58–96 cm (length, width, height).

**Fig. 1.** Lightweight power wheelchair

#### **2.4 Usability Testing**

In the International Organization for Standardization (ISO), usability is defined according to the ISO-9241-11 standard. Usability refers to effectiveness, efficiency, and satisfaction when a user uses a system, device, or service to achieve a specific purpose [12]. Effectiveness is to measure whether the user has performed the desired task accurately and perfectly. Efficiency includes time, costs invested to achieve goals, the degree of effort, and fatigue. Satisfaction is the evaluation of an individual's cognitive, emotional, and psychological responses, such as comfort and positive attitude, when a user performs a task using a product [12, 13]. Effectiveness was measured through task success. Efficiency measured (1) task performance time (TPT), (2) effort and fatigue required to perform tasks using the product through The Borg Rating of Perceived Examination Scale (RPE), and (3) subjective difficulty when performing tasks through the Differential Rating Scale (DRS). Satisfaction measured acceptance, satisfaction, and overall usability of the product through (1) Acceptability Rating Scale (ARS) and (2) System Usability Scale (SUS) [14–17]. RPE is measured in the range of 6 to 20 points, 6 points are no exertion, 20 points are interpreted as maximum exertion, DRS is interpreted as −3 points to 3 points, −3 points to very difficult and 3 points to very easy. ARS ranges from −3 to 3 points, and −3 points are interpreted as very unacceptable and 3 points as very acceptable. The SUS converts the participants' responses into 100 points and substitutes them for the indicator to confirm satisfaction and overall usability (Fig. 2).

For usability testing, 3 scenarios were created, and 18 detailed tasks were included under them. Each scenario is divided into a pre-driving process (S1), a driving process (S2), and a post-driving process (S3) (Table 2). S1 and S3 were performed by the caregivers, and S2 was performed by the participants. Usability testing was conducted by establishing a testbed similar to the environment that can be experienced when moving in a wheelchair in the community. The testbed was aimed at testing the stages of the driving process (Fig. 3).


**Table 2.** Scenario for usability testing

**Fig. 2.** Indicators of usability testing (RPE, DRS, ARS, SUS)

**Fig. 3.** Wheelchair driving testbed

## **2.5 Data Analysis**

Analysis was conducted to evaluate the effectiveness, efficiency, and satisfaction of each detailed task performed using LPW1 and LPW2. Effectiveness was expressed as a percentage by calculating the successful proportion of tasks performed by each LPWs. To measure efficiency and satisfaction, average values and standard deviations of TPT, RPE, DRS, ARS, and SUS of each LPWs were calculated using Excel and then substituted into each evaluation index.

## **3 Result**

Section 3.1 was written based on the results of scenario 2 to find out about the usability of users using LPWs. Section 3.2 was written based on the results of S1 and S3 to find out the usability in setting the LPWs by the caregivers.

## **3.1 S2 Tasks Performed by the Participants**

#### **3.1.1 Effectiveness**

The effectiveness of the two LPWs was confirmed by analyzing the success or failure of the task. All participants succeeded in performing LPW1 except for one participant who did not succeed in performing T10. LPW2 was successfully performed by all participants at once in all scenarios. Thus, LPW1 showed 80% effectiveness and LPW2 showed 100% effectiveness.

#### **3.1.2 Efficiency**

TPT were averaged 11.5 s (LPW1) and averaged 6.2 s (LPW2). RPE scores averaged 9.7 points for LPW1 and 9.4 points for LPW2. Both LPWs were usable with very light effort. DRS scores averaged 1.3 points for LPW1 and 1.4 points for LPW2. Both LPWs were analyzed close to a barely easy level. (Table 3).


**Table 3.** Result of S2 TPT, RPE, DRS

### **3.1.3 Satisfaction**

ARS scores averaged 1.6 for both LPWs. Both LPWs were analyzed close to a moderately acceptable level. (Table 4). As a result of SUS analysis, LPW1 scored averaged 77.6 points (Black line), which was measured as grade B/'Good' usability. LPW2 scored averaged 86.4 points (Red line), which was measured as grade A/'Best imaginable' usability (Fig. 4).

## **3.2 S1, S3 Tasks Performed by the Caregivers**

## **3.2.1 Effectiveness**

All caregivers performed successfully in all scenarios at once. Thus, LPW1 and LPW2 showed 100% effectiveness.

**Fig. 4.** Result of S2 SUS (Color figure online)

**Table 4.** Result of S2 ARS


#### **3.2.2 Efficiency**

TPT were averaged 6.7 s (LPW1) and averaged 7.9 s (LPW2). RPE scores averaged 7.3 points for LPW1 and 6.7 points for LPW2. Both LPWs were usable with extremely light effort. DRS scores averaged 2.9 for both LPWs. Both LPWs were analyzed close to a very easy level.

#### **3.2.3 Satisfaction**

ARS scores averaged 3 points for LPW1 and 2.9 points for LPW2. Both LPWs were analyzed close to a very acceptable level.

## **4 Conclusion**

Wheelchair users face a number of obstacles when traveling in a community environment. Through this comparative usability testing, the usability of two LPWs developed in Korea was examined from the perspective of users and caregivers. From the user's point of view, the usability was investigated by driving in a driving environment that can be experienced when moving to the community using a LPW, and by setting the battery and cushion of the LPW from the caregiver's point of view.

There was a difference between the two LPWs in terms of effectiveness and satisfaction in the participant's driving, and there was no significant difference in efficiency. There was no significant difference between the two LPWs in the performance of the caregivers.

As a result of the usability testing, the usability of the heavier LPW2 was highly evaluated. This can be interpreted that users feel a sense of stability in the LPW with a certain amount of weight. In future studies, it is considered necessary to study the appropriate weight of LPW that users can feel stable.

This study has a limitation in that it did not include powered wheelchair users with various types of disabilities by limiting the subjects to people with physical disabilities. It is believed that future research needs to evaluate usability considering various types of disabilities. It is expected that this study will be used as basic data in future research on LPWs.

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **A Study on Standard for Safety and Performance Requirements for Care Robots**

Soonjae Ahn1(B) , Sunwoo Yuk2, Woojin Yu3, and Inhyuk Moon1

<sup>1</sup> Dong-Eui University, 176 Eomgwang-no, Busanjin-gu, Busan 47340, Korea {asj,ihmoon}@deu.ac.kr <sup>2</sup> Korea Orthopedics and Rehabilitation Engineering Center, 26 Gyeongin-ro 10Beon-gil, Bupyeong-gu, Incheon 21417, Korea

sunwoo@kcomwel.or.kr <sup>3</sup> Korea Testing Laboratory, 87 Digital-ro 26-gil, Guro-gu, Seoul 08389, Korea ktl\_ywj@ktl.re.kr

**Abstract.** Recently, care robots are being developed that incorporate robotics into assistive products that focus on daily care for the physically disabled or elderly with reduced physical function. However, although care robots can reduce the physical burden of human intervention, they can also be dangerous depending on their situational awareness. This study describes a standardization that defines safety requirements for care robots and includes an evaluation method to test their safety based on these requirements. As an example of the application of this standard, a standard for the safety and performance method of a meal assistant robot is shown. This standardization study is expected to contribute to the spread of care robots in the future.

**Keywords:** Care robot · Standard · Safety · Performance · Requirement

## **1 Introduction**

The classification and terminology of assistive products are addressed explicitly in the ISO 9999 standard [1]. The standard defines an assistive product as "optimizing a person's functioning and reducing disability." In other words, an assistive product can optimize a person's functioning or reduce disability. With the recent development of robotics, many assistive devices are incorporating robotics to improve their functionality. The terms robot and robotics are well-defined in ISO 8373:2021 [2].

A robot is defined as a "programmed actuated mechanism with a degree of autonomy to perform locomotion, manipulation or positioning," and robotic technology is defined as "practical application knowledge commonly used in the design of robots or their control systems, especially to raise their degree of autonomy" [2]. Since IEC/TR 60601- 4-1 defines a medical robot as a "robot intended to be used as medical electrical equipment or system," an assistive robot can be defined as a "robot intended to be used as an assistive product" [3]. Similarly, a care robot is a "robot intended to be used for caring persons with disability or elderly persons."

These care robots are intended to support the care, nursing, and physical activities of the care recipient, easing the burden of caregiving and improving the quality of life of the elderly or disabled and their families. Care robot products are developed around basic activities such as Feeding, Personal toileting, Hoist, and Posture-changeable. However, there are still no clear definitions, item classifications, and safety standards for care robots, so their dissemination has many difficulties.

In this study, a care robot is defined as a "robot that assists the daily life of the disabled or the elderly who have difficulties in daily life." However, although care robots can be physically burdensome by reducing human intervention, they can also be dangerous depending on their ability to recognize situations. Therefore, we propose a standard that defines safety requirements specific to care robots and includes an evaluation method to test safety based on these requirements. As an example of applying this standard, we show a standard for Feeding robots' safety and performance methods.

As for standards related to care robots, ISO 21856, the common standard for assistive devices, was recently published [4]. It consists of 29 clauses, with general requirements in clause four and individual requirements in clauses 5 through 26. The international standard for medical rehabilitation robots is IEC 80601-2-78 [5]. This standard adds mechanical hazards defined by robots to the mechanical hazards in IEC 60601-1, a common standard for medical devices. It is necessary to refer to it in the care robot standard because it considers the characteristics of robots. IEC 60601-1, a common safety standard for all medical devices, states that the primary user may be a patient rather than a clinical professional. Therefore, for medical robots with a degree of autonomy (DOA), the user's situational awareness can be an essential safety factor [6]. Since the primary users of care robots are non-expert care receivers, and caregivers, the risk of losing situational awareness should also be considered. Methods for evaluating this risk include usability engineering in the field of medical devices [7] and usability evaluation in the field of assistive devices.

Medical devices, assistive devices, and children's products that have been used for a long time have well-documented safety requirements. However, the field of care robots, which has a short history, still lacks safety standards. This study proposes standards for safety and verification methods suitable for care robots by referring to existing standards.

#### **2 Safety Standard for Care Robots**

#### **2.1 Definition of Care Robot**

Care or care service is defined as "the act of providing physical Care or care service as "the act of providing physical or mental assistance to the elderly or disabled who have difficulty maintaining daily activities on their own."

Thus, a care robot can be defined as "a robot that assists the daily life of the disabled or the elderly who have difficulties in daily life."

#### **2.2 Types of Care Robots**

Care robots include Feeding robots, Hoist robots, Personal toileting robots, posturechangeable robots, and Rollator robots to reduce the burden on caregivers and improve the quality of life of care recipients (Fig. 1) [9].

**Fig. 1.** Types of care robots [9]

## **2.3 Structure of Care Robot Safety Standard**

The safety standards for care robots are divided into common safety standards, individual safety standards, care data standards, and care robot degrees of autonomy (Fig. 2).

**Fig. 2.** Hierarchy of care robot standards

The common safety standard for care robots covers general safety requirements that should be applied to care robots. In contrast, the individual care robot standard covers requirements that are difficult to apply in general in consideration of the characteristics of individual care robots and must be considered for each item. Therefore, the Care Data Standard and the Care Robot Degree of Autonomy Standard are different standards that can be referenced for care robots.

#### **2.4 General Requirements**

For risk analysis and management of care robots, the procedures of ISO 14971 or ISO 12100 are applied [10, 11]. In addition, for care robots classified as medical devices, IEC 62366-1 is used to identify hazardous situations [7], and for care robots with embedded programs, IEC 62304 is used [12]. In particular, for care robots with autonomy, the loss of the user's situational awareness is considered a hazard [13].

Multifunctional care robots with two or more functions must meet the safety and performance requirements in individual standards according to the primary function proposed by the manufacturer. Functions other than the primary function shall be classified as minor and meet common safety requirements.

If the care robot supports wired and wireless communication (Wi-Fi, Bluetooth, USB, RS-232, LAN, etc.) capabilities, it must be designed for availability, confidentiality, and integrity within the risk management process.

#### **2.5 Safety Requirements for Care Robots**

Safety requirements are categorized into electrical safety, mechanical safety, cleaning and disinfection, environmental factors, and hazardous materials.

Electrical safety is required to comply with national safety certification requirements. If there are no national safety certification requirements, at least meet the requirements for leakage current, withstand voltage, and over temperature. In addition, built-in batteries, emergency stop, protective stop, and electronics compatibility are required (Table 1). Finally, mechanical safety requirements are provided for the structure, pinching, squeezing, and restraint to ensure that the user of the care robot is not at risk of mechanical harm.

Requirements for hazardous substances are provided for parts of the robot that come into contact with the human body. The allowable values of hazardous substances are based on the Korean certification (KC) standard for children's products [8].

To prevent harm from noise, the standard requires a maximum noise level of no more than 50 dB in sleeping environments, 65 dB indoors, and 85 dB outdoors. In addition, the standard requires classification for water resistance according to IEC 60529 [13].

#### **2.6 Example of Feeding Robot**

This section introduces the individual requirements of a feeding robot. The shape of a feeding robot is shown in Fig. 3. The safety of the feeding robot is further tested for impact energy, static load, and repetitive durability (Table 2). The performance of the feeding robot is further evaluated through pose accuracy (ISO 9283) [14], continuous use time, and feeding assistance success rate (Table 3).

**Fig. 3.** Feeding Robot


#### **Table 1.** Electrical Safety Requirements for care robots


**Table 2.** Additional Safety Requirements for Feeding robot


**Table 3.** Performance Requirements for Feeding robot

## **3 Conclusion**

Recently, various research projects on care robots have been started and are in progress. However, due to the lack of relevant standards, there may be problems with the steps for licensing or certification for future productization. Therefore, we developed a standard for safety requirements for care robots in this study.

The requirements for electrical safety and mechanical safety, which are safety requirements for medical devices, were investigated and applied [6]. The mechanical risk factors defined by robots were additionally reflected by referring to ISO 13482 [16]. In addition, since care robots use robotics and the primary users are non-experts, the risk of situational awareness loss was considered by referring to IEC 80601-2-78, an international standard for rehabilitation robots [5], and cybersecurity was also added. Performance issues will be addressed in individual standards for care robots to be developed in the future.

In a situation where standardization of safety evaluation of care robots is actively needed, it is expected that securing product safety and consumer trust through normal development will be very important for the successful settlement and expansion of the care robot industry. Furthermore, the developed standards can be used for licensing and establishing policies for care robots, and manufacturers will be able to ensure safety in the development stage of care robots.

The addition of robot and robot technology definitions, automatic toileting systems, power assist units for assistive products for walking, and feeding robots to ISO 9999 has paved the way for developing and institutionalizing international standards for care robots [1]. Therefore, it can be promoted as an international standard based on the current standards. In addition, we will also develop standards for individual care robot items so that care robots currently under development can be released to the market with guaranteed safety and performance.

**Acknowledgment.** This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HJ20C0040, HI21C0484).

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Economic Analysis Methods Applied to Assistive Technology Devices and Services: A Literature Review**

Dong-wan Kim1 and Jongbae Kim2(B)

<sup>1</sup> Department of Occupational Therapy, The Graduate School, Yonsei University, Seoul, South Korea <sup>2</sup> Department of Occupational Therapy, College of S/W Digital Healthcare Convergence, Yonsei University, Wonju, South Korea jongbae@yonsei.ac.kr

**Abstract.** Assistive technology devices (ATD) and services (ATS) are generally applied to assist the people with disabilities and the elderly who have limited independent performance in daily life. The Korean government is also providing support through various projects, but the evidence for its effectiveness is lacking, and its application is also limited. Therefore, it is necessary to objectively measure the value of ATD and present evidence. This study analyzed studies that measured the economic and non-economic values of ATD and ATS through a systematic literature review, and looked at the overall economic analysis process. The database was compiled using KCI, NDSL, Web of Science and Scopus. Nineteen studies were selected according to a set procedure for literature screening, and the quality level of studies was analyzed through the CHEERS checklist 2022. Most studies have analyzed hearing aids, and most studies have used cost-effectiveness techniques. Measurement instruments were most commonly used to measure health and quality of life, accounting for approximately 60% of all. The quality level of the studies was calculated as 'Good' and 'Average' grades. In the future, it is expected that a high-quality economic analysis study will be conducted to measure the economic and non-economic values of various ATD and ATS in Korea.

**Keywords:** Assistive Technology · Economic Analysis · Literature Review

## **1 Introduction**

The World Health Organization emphasized that efforts to remove environmental and social barriers are needed to overcome the difficulties experienced by people with disabilities, and it has been confirmed that the people with disabilities rely on other people or assistive technology devices when necessary to carry out their daily activities [1].

The Korean government supports assistive technology device for the people with disabilities and the elderly through 9 projects in 4 departments (Ministry of Health and Welfare, Ministry of Employment and Labor, Ministry of Science and ICT, Ministry of Education) and 1 department (Ministry of Patriots and Veterans Affairs). Researchers reported that the use of assistive technology improves independence and quality of life, affects family and social burdens, and reduces the help of others [2–5]. This means that it is important to provide appropriate and effective assistive technology devices according to the needs and abilities of the subjects, and in-depth evaluation by occupational therapists, regular inspection of assistive technology devices, and cooperation with other experts are required [6, 7].

In Korea, the health insurance budget was gradually estimated from 2017 to 2022 to alleviate the burden of the vulnerable, and in particular, health care plans to alleviate economic difficulties and cost burdens were emphasized to ensure people-centered universal health [8–10]. Accordingly, plans and measures to efficiently use the limited budget of the state are required [11]. Kong and Jung [12], cost-effectiveness, cost-utility, and cost-benefit analysis are common techniques for confirming economic feasibility. However, related research in Korea is very scarce. According to WHO [2], through the support of appropriate assistive technology devices, the state not only reduces medical costs and additional welfare budgets due to repeated hospitalization of users such as the disabled and the elderly, but also provides socio-economic value by enabling users' economic activities reported to be able to create.

Therefore, this study analyzes the economic feasibility of providing assistive technology devices or services, systematically reviews studies that derive economic and noneconomic values, and analyzes economic analysis techniques and measurement tools used in the economic analysis. Through this, plans to efficiently use the government's limited budget in the field of rehabilitation in Korea will be presented.

## **2 Method**

#### **2.1 Database and Search Term**

The databases for literature search for this study are the National Digital Science Library (NDSL), Korea Citation Index (KCI), Scopus, and Web of science. Articles included domestic and foreign academic journals published from February 2018 to February 2023 based on the publication date.

Search terms are ("Economic analysis" OR "economic evaluation" OR "Costbenefit" OR "Cost-effectiveness" OR "Cost-efficiency") AND ("Assistive device" OR "Assistance tool" OR "Assistive Technology device" OR "Assistive Technology service" OR "Assistive Technology") was searched, and it was translated into Korean and used in the KCI and NDSL databases. Papers retrieved from each database were integrated through Endnote 20, duplicated articles were removed, and then selected based on the following inclusion and exclusion criteria.

#### **2.2 Selection Criteria**

Each article was selected if it satisfies the following criteria.


And, if each article satisfies the following criteria, it was excluded.


Since this study only considered studies analyzing the economic feasibility of assistive technology devices and services and studies reviewed through peer review, systematic reviews, meta-analyses, and degree studies were excluded. In addition, articles that could not be read or whose economic feasibility was not analyzed were excluded. The literature was independently reviewed by two researchers and finally selected through discussion. Finally, 19 articles were selected (Fig. 1).

#### **2.3 Quality Level of Research**

The quality level of the final selected articles was evaluated using the CHEERS (Consolidated Health Economic Evaluation Reporting Standards) checklist. The checklist was announced in 2013, the qualitative level of the economic evaluation was evaluated by Drummond et al. [13], after responding with 'Yes', 'No' or 'Not Applicable', if the item meets the guidelines, 1 point (yes), and half of it, 0.5 points (partial), if not satisfied at all, it can be calculated as 0 points (no). As the checklist was revised in 2022 through an expert Delphi survey from 2020, so the 2022 version of the revised CHEERS checklist was used in this study. The CHEERS checklist 2022 has added items related to patient or service recipient, general public, community or stakeholder engagement and involvement, and terminology has been expanded for broader applicability. And modifications have been made to make the analysis more transparent and to help readers decision-making, including items such as whether the economic analysis was planned and whether the model used could be used by anyone [14].

#### **2.4 Analysis**

In this study, the frequency of Type of Assistive Technology Device (ATD) or Service (ATS), Type of economic analysis methods, and Measurement tools used in the article was analyzed, and the Results of the Quality level of research were analyzed.

**Fig. 1.** Flow diagram of article selection.

## **3 Result**

#### **3.1 Analyzed Assistive Technology Device (ATD) or Service (ATS)**

Among the types of assistive technology devices and services analyzed in the final selected literature, hearing aids showed the highest frequency, and the study was conducted targeting the people with hearing disabilities due to disease and the elderly population. Next, there were many analyzes on assistive technology, hearing service, and manual wheelchair. In addition, the economics of vehicle modification, prosthesis, rollator, and upper limb robot was analyzed in one article each (Table 1).


**Table 1.** Type of Assistive Technology Device or Service.

#### **3.2 Used Economic Analysis Methods**

The techniques used in the analysis were cost-effectiveness, cost-benefit, and cost-utility analysis, and cost-effectiveness techniques were used most frequently. Among the economic analysis methods mentioned above, the cost-effectiveness method was used the most, and it was also used in various assistive technology device and service studies, but the high proportion of the papers on economic analysis on hearing aids had an impact. Additionally, there was one article each on the social value of assistive technology and telecare (ATT), and vehicle modification (Table 2).

**Table 2.** Type of Economic Analysis Method


#### **3.3 Measurement Tools by Cost and Outcome**

In the analyzed studies, costs were generally determined through unit costs, based on national surveys or published data. Direct costs typically included initial consultation, intervention costs such as assistive technology devices or services, evaluation and training, and, if necessary, surgery costs. Indirect costs included transportation and opportunity costs for individuals and caregivers, and non-working hours.

Outcome tools measured as a result of economic analysis were analyzed. Of the 11 tools, the tools to evaluate health and quality of life accounted for a high proportion of 5, and among them, quality adjusted life year (QALY) was used the most. QALY is a wellknown measure of the degree to which a treatment or system improves quality of life while prolonging lifespan, and it has been pointed out that it is a measure tailored to the individual rather than society. Next, EQ-5D and HUI3 were used as questionnaire tools to measure health utility. The EQ-5D responds to mobility, self-care, usual activities, pain/discomfort, and anxiety/depression. Similarly, HUI3 responds to the state of eight attributes, such as vision, hearing, and ambulation, etc. And according to the purpose of the study, tools to evaluate elderly people with dementia and tools to evaluate people with disabilities who are working were used.

In addition, five articles analyzed complications prevented through assistive technology devices and services, cost reduction, and durability of devices themselves as outcomes, and two articles that studied social values measured non-economic values through questionnaires and CVM (Table 3).


**Table 3.** Outcome Measurement tool of included studies

#### **3.4 Results of the Quality Level of Research**

The total score range of CHEERS 2013 checklist ranges from 0 to 24, and higher scores mean excellent quality levels, and can be divided into excellent (24 points), good (17.5 to 23 points), average (11.5 to 17 points), and poor (11 points or less) [13]. However, since the revised CHEERS 2022 checklist has not yet been presented and consists of 28 items, this study classified it into excellent (28 points), good (20.5–27.5 points), average (13.5–20 points) and poor (13 points or less) based on previous studies. As a result of the analysis, the total score ranged from 17 to 27 points, and one of the four studies conducted according to CHEERS checklist scored the highest with 27 points. Two articles that studied the cost-benefit of lightweight manual wheelchairs were excluded from the analysis because only limited information was provided. Therefore, 17 articles were evaluated in this study (Table 4).


**Table 4.** Level of Quality of included studies

## **4 Conclusion**

The application of assistive technology devices and services has a great impact on the lives of people with disabilities, improves their lives, and enables new attempts, so they must be attempted. The Korean government is also making great efforts for this, but so far, limited application is being made. Therefore, this study attempted to provide basic data for improving the existing system by systematically reviewing studies that analyzed the economic and non-economic values of assistive technology devices and services. As a result of analyzing previous studies, both economic and non-economic values of assistive technology devices and services for the people with disabilities were measured, and the economic value was mainly calculated through a unit cost based on national survey and published data, and the non-economic value was calculated through QALY, questionnaires and conditional value evaluation (willing to pay). Also, Outcome measurement tools were mainly related to health and quality of life. Through this, the government will provide efficient support without wasting budget, and people with disabilities can expect to improve their quality of life by reducing the burden in purchasing and using necessary assistive devices and services. Lastly, In Korea, there are few studies that have confirmed the effectiveness, benefits, and utilities of assistive technology devices and services on the health and life of people with disabilities who use and receive them. Accordingly, it is expected that researches that measure the economic and non-economic values of various ATDs and ATSs in Korea according to an objective checklist will be actively conducted in the future.

## **References**

1. Lakshmi, V.V., Paul, M.M.: Importance of assistive technologies and programmes for economic empowerment of disabled farmers-a review. Int. J. Farm Sci. **8**(2), 37–42 (2018)


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Ontology Alignment for Accurate Ontology Matching: A Survey**

Hasham Khan<sup>1</sup>, Muhammad Saqib<sup>1</sup>, Hasan Ali Khattak<sup>1</sup>, Syed Imran Ali1(B), and Sungyoung Lee<sup>2</sup>

<sup>1</sup> School of Electrical Engineering and Computer Sciences, National University of Science & Technology (NUST), Islamabad, Pakistan *{*hkhan.mscs21seecs,msaqib.mscs21seec, hasan.alikhattak,imran.ali*}*@seecs.edu.pk <sup>2</sup> Department of Computer Science and Engineering, Kyung Hee University, Giheung-gu, Yongin 17104, South Korea

sylee@oslab.khu.ac.kr

**Abstract.** Edge computing, a distributed computing architecture within the knowledge-defined network (KDN), faces challenges due to the significant disparities and data heterogeneity among its nodes, hindering their interaction. Ontology, a solution within the Semantic Web, is well-suited for addressing data heterogeneity and matching ontologies effectively. However, ontology matching presents difficulties due to non-linear mathematical issues. To overcome these challenges, the generative adversarial network (GAN), an unsupervised learning method, has emerged as a promising tool. GAN consists of two models with distinct objectives trained against eachother to achieve optimal outcomes. This paper introduces SA-GAN, an algorithm that combines GAN with simulation-based annealing to enhance its effectiveness. SA-GAN utilizes a stagnation counter to expedite the convergence speed of GAN. Through experiments conducted on a renowned ontology benchmark, the paper demonstrates that SA-GAN, along with other ontology matching algorithms, can identify the best alignments. Consequently, SA-GAN facilitates the construction of bridges in edge computing, improving its overall effectiveness.

**Keywords:** Edge Computing · Ontology Matching · Simulation-Based Annealing Generative Adversarial Network (SA-GAN)

## **1 Introduction**

The new paradigm of knowledge-defined networking (KDN) combines artificial intelligence (AI), software-defined network analytics (NA), and software-defined networking (SDN). Within KDN, edge computing empowers edge nodes with storage and processing capabilities to perform data forwarding, recognition, and other services. However, these edge nodes often have different data representations, leading to data heterogeneity. Ontology, at the very exact moment, acts as a model of reference for information transmission, allowing for the exact correct semantic standardized meaning to date [10]. To address this, edge node ontology is used to represent concepts and relationships in specific domains. Ontology, a fundamental technique in the Semantic Web, is widely employed to tackle data heterogeneity by effectively modeling knowledge and formalizing relationships and concepts. The System Upgrade Process in Fig. 1 is an example of how ontology is applied and mapped to interconnected concepts. Ontology helps identify similar domains, definitions, terms, and their connections, creating a graphical map of entities and their associations, including links to other corporate domains.

**Fig. 1.** An Ontology about System Upgrade

Ontology construction is subjective, resulting in heterogeneity in ontologies due to different techniques used for representing ideas and relationships [8]. In an information-rich era, multiple sources produce similar content, making it impossible for systems to have identical ontologies. Ontology matching provides a direct approach to leverage ontologies by identifying heterogeneous entities across multiple ontologies. When given two distinct ontologies with discrete items like attributes and classes, ontology matching involves determining relationships such as equality and formation among these entities [11].

Ontology meta-matching is a complex problem that involves determining suitable weights for combining multiple similarity measures, ensuring highquality alignment. Unsupervised learning is well-suited for large-scale ontology meta-matching, as it doesn't rely heavily on manual labeling [9]. Generative adversarial networks (GANs) are popular in unsupervised learning. They consist of a generator and a discriminator. The generator produces simulated samples, while the discriminator provides feedback on their authenticity. Through iterations, the generator improves until the discriminator can't distinguish between real and generated samples. GANs have found applications in various fields, including computer vision, and have inspired innovative approaches in ontology matching.

Rest of the paper is organized in the following order: Sect. 2 gives an outline of the state of the art literature review where we outline the work done. Section 3 presents the model of adversarial learning based ontology alignment. While Sect. 4 gives and outline for the experimental analysis we have done. Finally the paper gives a comparison to Instance Matching using Knowledge Graph in Sect. 5 and concludes the work in Sect. 6.

## **2 Literature Survey**

Ontology matching bridges the semantic gap between different domain representations but can be time-consuming. The use of diverse modeling approaches is necessary due to the multiple perspectives in perceiving things. Integrating domains often involves representing combined knowledge using ontologies and employing ontology matching approaches to align them. Our literature review focused on recent publications in the past three years and revealed a widespread interest in ontology matching. Our objective was to classify and identify research trends in ontology matching while developing a reference framework for integrating and categorizing these materials.

Ontology matching bridges the gap between domain representations. Integrating diverse perspectives requires different modeling approaches using ontologies. Our literature review identified trends and developed a framework. We examined a meta-strategy-based KG matching approach that aligns schema and instance-level entities. Experimental results favor algorithm-based metaheuristic approaches over CNEA-based KG matching [11].

We introduce a heuristic evaluation measure and an optimization model for ontology matching. Our approach utilizes an Adaptive Evolutionary Algorithm (AEA) and outperforms reference alignment-based measures. It avoids local optima and excels in determining high-level alignments compared to existing systems. Our AEA-based ontology meta-matching system generates superior alignments independently [4].

Early work on graph alignment employs a graph embedding algorithm and an absolute orientation rotation method. Studies have shown that this method is effective for aligning structurally similar ontologies and is more robust against alignment noise when dealing with graphs of different sizes and architectures. Future research aims to explore various embedding methodologies and integrate the method with additional features, such as text-based information, in a hybrid matching system [1].

Normalization of Word and representation learning are employed to enhance the optimization of ontology alignment approaches using feature-based methodologies. Word normalization helps capture various domain terms, while representation learning discovers semantic and conceptual information, expanding alignment possibilities. Entity embedding learning within the representation learning approach is investigated [2].

Ontology heterogeneity in Knowledge Delivery Networks (KDNs) hampers collaboration among edge nodes. Ontology matching tackles this challenge by assessing similarity. We propose an adversarial learning model based on simulated annealing for ontology meta-matching. It optimizes weights and thresholds using a Generator-Discriminator relationship, outperforming previous systems and achieving effective ontology meta-matching [3].

We propose a mapping procedure to generate a semantic-compatible ontology for Digital Dentistry (DD) using a base ontology and three reference ontologies. Additionally, we suggest a deep learning-based method to identify key factors contributing to depression in DD. Due to limited data availability caused by the COVID-19 pandemic, further experimentation is needed to validate the proposed methods [5].

DeepOM is a deep learning-based ontology matching system that handles large ontologies without partitioning. It generates concept embeddings using a reference ontology and an auto-encoder, resulting in accurate and compact representations. DeepOM outperforms other systems in matching large-scale ontologies. The use of an auto-encoder for concept embeddings proves effective, with all DeepOM parameters contributing to improved matching [6].

This study proposes an approach for aligning IIoT ontologies using NLP. It learns vector embeddings for items and relations using ontology metadata and structure. Experimental results consistently outperform the baseline model BERT INT in HR and MRR scores by 1.2–2.7%. Limitations include synthesizing the ontology dataset due to unavailability of real-world data. However, the model benefits from learning language embeddings, facilitating identification of nodes with comparable semantic meaning. The structural encoder accurately aligns nodes, eliminating biases and imposing context [7].

## **3 Model of Adversarial Learning Based**

In this study, the meta-matching problem of ontology is treated as a constrained optimization problem. To solve it, popular heuristic algorithms like simulated annealing, genetic algorithm (GA), and technique are commonly used. These algorithms excel in handling complex optimization problems, especially with large datasets (e.g., SA).

The choice of a specific heuristic algorithm depends on the ontology's characteristics and the optimization problem's nature. Selecting the right algorithm is vital for achieving effective and efficient meta-matching of ontologies (Fig. 2).

The use of genetic algorithms (GA) to solve the meta-matching problem in ontology has two main limitations: slow convergence and premature convergence.

**Fig. 2.** GANs Framework

These problems can reduce the algorithm's effectiveness and efficiency in finding optimal solutions.

To address these challenges, we propose a new approach called SA-GAN. It combines simulated annealing with the adversarial training framework inspired by generative adversarial networks (GANs). We use the Metropolis criterion to generate optimal characteristics for solving the ontology meta-matching problem. In the context of GANs, our methodology has two components: the Generator and Discriminator models.

#### **3.1 Objective Function**

The rate of recall within the domain of ontology matching is computed as the ratio of accurate alignments chosen from the overall count of alignments generated by the ontology matching system. It assesses the system's capacity to accurately recognize and incorporate pertinent matches in its outcomes. The recall rate provides an important evaluation metric for assessing the completeness and comprehensiveness of an ontology matching system's results. It indicates how well the system captures the true matches in the given ontologies and helps in evaluating its performance. The following is a definition of the formula:

$$recall = \frac{len(A \cup R')}{len(A)}\tag{1}$$

The proportion of accurate alignments we select the overall number of total alignments chosen is called precision. It assesses the ontology matching system's precision. The following is the formula:

$$precision = \frac{len(A \cup R')}{len(R')} \tag{2}$$

We expect recall and accuracy to be equal (both ranges are between 0 and 1), but this isn't always the case, therefore we included f - measure which is a harmonic average to assess the match of ontology outcomes. It even emphasizes our system's performance in a more intuitive manner. In addition, the formula can be changed. the outcome is nearer to recall when it is nearer to the 0, while its result is nearer to the precision the nearer it is to 1. (Xue & Wang, 2015a) The function f-measure is given below:

$$f-measure = \frac{recall \times precision}{\alpha \times recall + (1 - \alpha) \times precision} \tag{3}$$

In the SA-GAN framework for ontology meta-matching, the goal of the Generator is to generate optimal similarity thresholds and weights that lead to the optimization of alignment quality and maximize the value of the f-measure. The f-measure is a metric that combines precision and recall to assess the overall effectiveness of the ontology matching system.

The ultimate objective of the SA-GAN framework is to achieve a suitable combination of weights and similarity criteria that yield good performance in terms of the f-measure. The Generator plays a crucial role in this process, guided by the Discriminator's feedback on the f-measure. The framework involves interactive training of both the Generator and Discriminator parameters to optimize the meta-matching process.

Figure 3 illustrates the overall framework, highlighting the interactions and iterative training between the Generator and Discriminator to achieve effective ontology meta-matching.

**Fig. 3.** SA-GAN Framework

In ontology meta-matching, when the generator's weights and threshold make the discriminator's precision and recall both 1, regardless of how it trains, the f-measure is 1, the discriminator will end training and now the last threshold and weights will be displayed out (Fig. 5).

#### **3.2 Simulated Annealing Optimizer**

This paper explores using simulated annealing, an optimization algorithm, for parameter optimization in ontology meta-matching. Simulated annealing is effective for numerical optimization problems with many local minima. The authors propose utilizing simulated annealing as a generator optimizer to speed up training and improve results. They suggest adjusting the update probability of the algorithm to enhance exploration of the parameter space and achieve better optimization outcomes in ontology meta-matching. Here E is either equal to 0 or less than 0, now updated solution is superior to the previous solution, and the efficient solution is thus revised. Otherwise, based on the existing temperature condition, it will accept the new answer with a fixed likelihood. In the optimization procedure for the generator, the authors introduce Algorithm 1, which describes the calculation process using simulated annealing with a modification to address efficient solutions. The goal is to find the best optimal solution within the SA-GAN epoch.

$$P = \begin{cases} 1, & \Delta E \le 0 \\ e^{-\Delta E/t}, \Delta E > 0 \end{cases}$$

In this algorithm, the temperature adjustment is controlled using a geometric-based reduction. The temperature (T) is updated using the formula T = current temperature/m β, where m is a rate parameter of cooling. This geometric reduction allows the annealing process to speed up, gradually reducing the temperature.

Simulated annealing accepts worse options at high temperatures to explore the search space and potentially reach the global maximum. The authors introduce a stagnation counter in Algorithm 1 to measure solution efficiency. If the same result occurs multiple times, it indicates a lack of progress and termination of the algorithm. This modified approach aims to enhance the generator's effectiveness in finding optimal solutions for ontology meta-matching within each SA-GAN epoch.

#### **3.3 Using Gradient Descent and Discriminator**

Because discriminator must be educated in the f-measure, it is thought of as a univariate function optimization issue. We use gradient descent, iterative optimization which is considered to be first order process for calculating differentiable functions' local minima, to swiftly identify the ideal case.

Here algorithm gradient descent follow the rule i.e., in case the univariate function f-measure(α) is identified and it also is distinguishable in the neighborhood of the point a, f-measure(α) decreases with speed at point a in the different direction of its gradient, rf-measure(α). It follows that, if b = a − γΔf − measure(α), now considering γRp less then f −measure(α) ≥ rf −measure(α). keeping this in notice, it should begin with an initial approximation of its local minimum x<sup>0</sup> with f − measure(α), so we know here is a sequence of x in a way that x0, x1, x2, ..., xn, xn+1 so that Xn+1 = X<sup>n</sup> − nΔf − measure(Xn), n ≥ 0. Therefore, f −measure(x0) ≥ f −measure(x1) ≥ f −measure(x2), if it proceed good, its sequence is converged obtaining desired local minimum finally.

We varied parameters' range in its discriminator in the experiment, creating a wave at 0.46 to 0.56 that we chose from a sensitivity test approximately. Because we aim to enhance both the precision rate and the recall rate during the training of parameter, here, we even need the end outcome to be optimized in the last f-measure calculation, which is 0.51.

#### **3.4 Training Process of Model**

SA-GAN, a revolutionary adversarial learning model that offers a fresh approach to the problem of ontology meta-matching. The flowchart related to our process of matching is shown in Fig. 5 during the stage of pre-processing, it finishes extracting alignments of ontology and calculating our matrix of similarity. Then we convert matrix of similarity in our data set which allows our model to be trained. The generator and discriminator models are then trained using an adversarial technique in the training step. Finally, to achieve the final ontology alignments, the weights and threshold produced during adversarial training were combined for similarity computation.

## **4 Experimental Analysis**

In this work evaluated their ontology matching strategy using a benchmark track provided by the Ontology Alignment Evaluation Initiative (OAEI). They compared their generated alignments with the correct correspondences in the reference alignment to measure performance. This allowed for a comprehensive evaluation and comparison with other matching systems in the OAEI benchmark.

Table 1 gives a detailed overview regarding OAEI benchmark. Here Table 1's first column shows the ID in our test cases, which corresponds to several sorts of ontologies in the experiment. We can discover that our system is sensitive to which features and our system is inconsiderate to which features to using various sorts of test sets, and we can make more precise adjustments to the system based on this information (Fig. 4).

Here SA-GAN model is undergone training for maximum 50 epochs in the adversarial training step. Set Max Temperature to 200, Min Temperature to 100, Maximum Stay Times to 30 and iterations to 99 for fast-SA optimizer. The weights and threshold are Set for the generator at random initial is set to 0.51 and range is set to 0.46 to 0.55 for the discriminator, and epochs and learning rate for the algorithm gradient descent to 11, 0.01 correspondingly (Fig. 6).


**Table 1.** OAEI's Benchmark Description

**Fig. 4.** SA-GAN Flowchart


**Fig. 5.** Comparisons of OAEI's and f-measure


**Fig. 6.** OAEI's Benchmark for experimental Analysis

## **4.1 Experimental Results**

In the experiment, the evaluation function's parameter varies, requiring identification of a suitable range for the discriminator's training parameter. The objective is to control the parameter within a manageable range while improving both recall and precision in ontology matching results. Table 2 presents results achieved in various benchmarks with six groups in the range. Adjusting the range between 0.45 and 0.55 enhances precision and recall.

Researchers compare their method's benchmark results with participants from OAEI, including CODI, MapPSO edna, GeRMeSMB, TaxoMap, Falcon, and AROMA. Table 3 shows their method performs well in five types of OAEI benchmarks. However, GeRMeSMB's results in 201-208 differ slightly from Falcon's in 301-304 due to the lack of entity context and structure consideration, along with imprecise semantic-based similarity metrics.

## **5 Comparison to Instance Matching in Knowledge Graph**

**Problem Focus:** While the proposed framework focuses on ontology metamatching, which involves aligning and optimizing different ontologies, instance matching in the knowledge graph aims to identify and match specific entities or instances across different knowledge graphs.

**Optimization Approach:** The researchers in the proposed framework utilize heuristic algorithms such as simulated annealing and genetic algorithm to solve the complex optimization problem of ontology meta-matching. These algorithms iteratively adjust weights and similarity criteria to improve the alignment results. On the other hand, instance matching in the knowledge graph typically relies on graph-based algorithms and similarity measures specific to entity attributes and relationships.

**Evaluation Metric:** The framework's evaluation metric revolves around the f-measure, which considers both precision and recall. The objective is to achieve an f-measure of 1 by iteratively adjusting the weights and thresholds using the Generator and Discriminator. In contrast, instance matching in the knowledge graph often employs metrics like precision, recall, and F1-score at the instance or entity level.

**Training Process:** The proposed framework's training process involves interactive training of the Generator and Discriminator parameters. The Generator aims to find optimal weights and similarity criteria, while the Discriminator provides feedback by assessing the alignment results. Training continues until the Discriminator's precision and recall reach 1, indicating a successful match. In instance matching, the training process focuses on learning and optimizing similarity measures and matching rules specific to the attributes and relationships of instances.

**Scope and Data Considerations:** The proposed framework deals with ontology meta-matching and addresses the issue of semantic heterogeneity induced by knowledge representation differences. It requires access to ontological metadata and structures to guide the alignment process. Instance matching in the knowledge graph, on the other hand, focuses on identifying correspondences between specific instances, relying on available attributes and relationships within the knowledge graphs.

**Applicability and Limitations:** The proposed framework enhances ontology meta-matching, achieving a high f-measure through optimized alignment results. It has been evaluated on benchmark datasets, but its effectiveness may vary depending on ontology characteristics and available metadata. On the other hand, instance matching techniques in the knowledge graph target specific entities within the graph, each with its own strengths and limitations based on the chosen approach. By including this detailed comparison section, readers gain insights into the distinct contributions and limitations of the proposed framework for ontology meta-matching and instance matching in the knowledge graph.

## **6 Conclusion and Future Work**

Semantic heterogeneity in knowledge-defined networks arises from differences in knowledge representation, impacting collaboration among edge nodes. Ontology matching addresses this issue by determining weights and confidence levels for multiple similarity assessment methodologies. This study proposes a simulated annealing-based adversarial learning framework for ontology meta-matching. It optimizes a single-objective model by iterative tuning of parameters and achieving improved matching outcomes compared to previous systems.

Future work aims to enhance ontology alignments by considering entity structure and improving confidence value accuracy. Addressing the slower convergence of the framework during training is also a focus, involving parameter adjustments and optimization for faster convergence.

**Acknowledgements.** This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the Grand Information Technology Research Center support program(IITP-2022-2020-0-01489) supervised by the IITP (Institute for Information & communications Technology Planning & Evaluation) and by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (IITP-2022-0-00078, Explainable Logical Reasoning for Medical Knowledge Generation), (IITP-2017-0-00655, Lean UX core technology and platform for any digital artifacts UX evaluation).

## **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Author Index**

#### **A**

Abdulrazak, Bessam 51, 65 Abril-Jimenez, Patricia 175 Ahmed, Hassan M. 65 Ahn, Juyoung 282 Ahn, Soonjae 322 Ali, Syed Imran 338 Amuzang, Gilly Njoh 249 Anh, N. T. 79 Arora, Teena 130

#### **B**

Balasubramanian, Venki 130 Bao, T. Q. 79 Bilal, Hafiz Syed Muhammad 230 Bilal, Muhammad Abdullah 230 Blanchet, F. Guillaume 65 Borgiel, Katarzyna 295 Bortolaso, Christophe 295

#### **C**

Cabrera-Umpiérrez, María Fernanda 16, 175 Cappon, Giacomo 16 Chikhaoui, Belkacem 26, 92 Choi, Jiwon 3 Choi, Jun-Su 304 Cossette, Benoît 65 Cossu, Luca 16

#### **D**

Dang, Khoa T. 205 Derlecki, Tomasz 249 DiCuffa, Sophia 38 Dion, Philippe 192 Doan, Hieu M. 205 Duy, N. T. Q. 79

#### **F**

Facchinetti, Andrea 16 Fleury, Anthony 117 Frikha, Ghassen 295

#### **G**

Gassara, Olfa 26 Gia, Khiem H. 205 Giroux, Sylvain 192 Gonzalez-Martinez, Sergio 16, 175

#### **H**

Hieu, D. M. 79 Huong, Luong H. 79, 205 Hwang, In-ho 104

#### **I**

Itoe, Micheal Blake Somaah 249

#### **J**

Jeoung, Bogja 282 Jovanovi´c, Aleksandar 16 Jung, Min-Ye 304

#### **K**

Kacem, Ahmed Hadj 269 Kahn, Maurice 117 Kalboussi, Anis 269 Khalid, Shah 230 Khan, Hasham 338 Khan, Kifayat Ullah 230 Khanh, Bang L. 205 Khanh, V. H. 79 Khattak, Hasan Ali 338 Khiem, H. G. 79 Khoa, T. D. 79 Kim, Dong-wan 313, 330 Kim, Hayoung 3 Kim, Jaesuk 3 Kim, JongBae 167 Kim, Jongbae 144, 184, 239, 313, 330 Kim, Junghun Aj 304 Kim, Ngan N. T. 205 Kosti´c, Borko 16 Kwon, Junhwan 3 Kwon, Oyun 3 Kwon, Yeong-hun 184, 313

© The Editor(s) (if applicable) and The Author(s) 2023 K. Jongbae et al. (Eds.): ICOST 2023, LNCS 14237, pp. 351–352, 2023. https://doi.org/10.1007/978-3-031-43950-6

#### **L**

Lamine, Elyes 295 Lee, Jae Young 104 Lee, Junhee 221 Lee, Kuem Ju 261 Lee, Sungyoung 230, 338 Lee, Yun-hwan 144 Loc, V. C. P. 79 Locquet, Rémi 155 Lodhi, Aminah Bilal 230 Lorca, Xavier 295

#### **M**

Mabrouk, Rostom 26 Mah, Pascal Muam 249 Mannai, Zayneb 269 Maraoui, Souhail 65 Montassar, Imen 92 Moon, Inhyuk 322 Moon, KwangTae 184 Muñoz Collado, Rosanny Araysy 167 Munyeshuri, Eric 249

#### **N**

Nasr, Mahmoud 249 Ngan, N. T. K. 79 Ngankam, Hubert 192 Nghia, Huynh T. 79, 205 Nguyen, Hien Q. 205 Noh, Giseop 282

#### **O**

Oh, Kyeong Teak 3 Ottaviano, Manuel 16 Oyeleke, Richard O. 38

#### **P**

Parekh, Pranav 38 Park, Sang Geon 104 Pełech-Pilichowski, Tomasz 249 Phu, Loc V. C. 205 Phuc, N. T. 79 Pigot, Hélène 192 Pingaud, Hervé 295

#### **Q**

Quinlan, Andrew 38 Quoc, Duy N. T. 205

#### **R**

Rao, Guillaume 117 Reerink-Boulanger, Juliette 155 Rezaei, Amin 51 Ric, Caroline 155

#### **S**

Saqib, Muhammad 338 Satti, Fahad Ahmed 230 Seddar, Mohand O. 117 Skalna, Iwona 249 Somat, Alain 155 Song, Won-Kyung 261 Stranieri, Andrew 130

#### **T**

Tah, Ning Frida 249 The, Anh N. 205 Tran, Bao Q. 205 Tran, Nguyen H. 205 Trong, Nguyen D. P. 79, 205 Trong, Phuc N. 205 Tuan, Kiet L. 205

#### **U**

Uroševi´c, Vladimir 16

#### **V**

Van, Hieu L. 205 Vo, Hong K. 205 Vojiˇci´c, Nikola 16

#### **W**

Wang, Shengrui 26, 92

#### **Y**

Yang, Seungwan 239 Yoo, Sun K. 3 Yoon, Joon Shik 221 Yu, Woojin 322 Yuk, Sunwoo 322