A Review and Comparative Study of Works that Care is Monitoring

Detection and Therapy of Children with Autism Spectrum Disorder

MOHANNED. A. ALJBORI, AMEL MEDDEB-MAKHLOUF, AHMED FAKHFAKH

ENET’com of Sfax University,

TUNISIA

Abstract: - Recognizing human activity from video sequences and sensor data is one of the major challenges in

human-computer interaction and computer vision. Health care is a rapidly developing field of technology and

monitoring that only requires the patient to be within a few meters of the sensor. Nowadays, the Internet of

Things, wearable devices, mobile technologies, and improved communication and computing capabilities have

given rise to innovative mobile health solutions, and several research efforts have recently been made in the

field of autism spectrum disorders (ASD). This technology may be particularly useful for some rapidly

changing emotional states, especially people with ASD. Children with ASD have some disturbing activities,

and usually cannot speak fluently. Instead, they use signs and words to establish rapport, so understanding their

needs is one of the most challenging tasks for healthcare providers, but monitoring the disease can make it

much easier. We study in this work more than 50 collected articles that have made a significant contribution to

the field were selected. Indeed, the current paper reviews the literature to identify current trends, expectations,

and potential gaps related to the latest portable, smart, and wearable technologies in the field of ASD. This

1 Introduction

Since the past years, we have witnessed great

progress and great effort to build innovative

computer vision applications that cover different

fields, despite the very wide spectrum of these

applications, few solutions have been designed to

help people with autism. Computer vision

technologies for supervising people with ASD are

still in their infancy. There has been limited research

especially those related to the security of autistic

children during an autistic crisis. Highly functional

autistic children often go through severe autistic

crises and breakdown can occur due to sensory

overload. It can happen when children are alone and

lose control of their behavior by unintentionally

hurting themselves or others. An autistic child may

display the variable symptoms of a breakdown in his

abnormal behavior according to different scenarios,

[1]. COVID-19 has played a huge role and has been

opportunity to use these technologies to

simultaneously monitor the health and well-being of

a patient there is a lot of data physiological signals

and parameters used in physiological and emotional

evaluation provided by these devices, such as (heart

rate, respiratory rate, electrical activity of the skin,

skin, and body temperature, blood pressure, blood

oxygen saturation, electromyography,

electroencephalogram), all these sensors mentioned

above, it can contribute significantly to the

characterized by abnormal communication and

social interaction, with restricted and repetitive

behaviors. There is no doubt that people with autism

are a growing part of our society. In addition, the

most recent studies conducted by the Centers for

Disease Control and Prevention (CDC) in 2018

estimated that the prevalence of autism could be

between 90 and 120 individuals out of 10,000,

which is about 1% of the population, [4].

behavior and intentions, they have difficulty

thinking outside of their routine The current world is

moving towards an intelligent society based on the

Internet of Things, where AI-enabled devices will

be everywhere. People will get help through it,

which will reduce human intervention across a

variety of sectors. It would be beneficial to design

these devices for autistic individuals as well because

they can significantly reduce the requirements for

human assistance. These devices can help them

special way of interacting and understanding, [6].

doctors, beds, and nurses is imminent around the

world and the cost of treating chronic diseases

continues to increase. In such emergencies, the

smart monitoring app is particularly useful, as it

automatically triggers an alert in the event of a crisis

based on an analysis of abnormal facial expressions

of children with autism or by analyzing the data of

other sensors associated with the patient, [7].

popular research topics due to the availability of

sensors and accelerometers, lower cost, lower power

consumption, real-time data streaming, and

advances in computer vision, machine learning,

Artificial Intelligence (AI), and the Internet of

Things. The procedure for identifying human

are data collection, feature extraction, classification,

and recognition activities, [8], [9], [10], [11], [12].

Most researchers used different ML and DL

algorithms such as SVM, LR, NB, ANN, KNN, etc.,

and DL algorithms such as CNN, Recurrent Neural

Network (RNN), etc. to monitor autism. Apart from

these algorithms, they used different image-

processing methods for feature extraction and

adopted different rule-based methods such as fuzzy

logic to classify ASD. Systems based on Internet of

devices to analyze activity, movement, and the state

of the environment. Various sensors and devices

have been widely used in research work on autism,

where the data collected from different sensors are

sent to the smart grid to communicate with the

system. A smart grid is a network through which

connectivity is provided to the various entities

involved in the healthcare system, [15], [16].

Furthermore, the design of technologies and their

integration into the concept of smart cities is

monitored in a regular way to help them with any

kind of problem, [17]. Indeed, image processing is a

growing technology and can be used in various

fields such as medical imaging, computer vision,

computer graphics, etc. Image processing is a

method that takes an image as an input, performs

some operations on it, and gives the image as an

output. Image processing operations are divided into

image optimization, restoration, segmentation,

extraction, and recognition. Image processing

for children to become self-reliant. Nowadays,

smartwatches and other wearable devices have

enough processing power and memory to help

children with autism. Can help collect health data

from integrated sensors that can help monitor. In

general, wearable devices and mobile technology

especially in the field of health care, [19].

become a popular and active research area for

researchers over the past two decades. However, it

remains a complicated task due to some unsolvable

issues such as sensor motion, sensor placement,

background clustering, and the inherent variance in

how autistic people perform different activities.

one of the challenging issues in computer vision in

activities from a dataset. There are challenges for

the potential clinical adoption of those wearable

technologies. First, most existing devices on the

market are designed for the general population with

little attention to people with ASD. Second, the use

of these devices requires training for users and their

caregivers. Third, even when devices are designed

for people with ASD, the cost of purchasing and

gaining continuous service for some devices can be

expensive for parents and caregivers. There are

standard among different organizations and regions,

and there is a need for improvement to ensure data

integrity. As extensive data must be gathered, it will

likely make the system rather complicated, which

will lead to difficulties in communication and data

management.

The rest of the paper is structured as Section 2

contains the related work of recent research papers

in the field. Section 3 briefly describes the various

methods and techniques used in ASD monitoring

2 Related Works

Interacting with autistic children is one of the most

challenging issues that their families and caregivers

deal with. In recent years, systems based on the

been committed to treating ASD or diagnosing

diseases, but few relevant papers have been

submitted to investigate ASD monitoring regimens

in the same way. Several reviews of relevant

literature have been published, examining

technologies that can enable physiological and

emotional monitoring remotely, using different

sensor modalities.

people with ASD are still in their infancy. There has

in the Figure 1.

Fig. 1: Related studies classification for autistic

from video templates to predict position with

subject position. The LSTM model calculates the

temporal relationship in feature maps to analyze the

measure of irregularity. Moreover, to deal with such

irregularities, a time-sensitive alert-based decision-

generate early warnings for the clinician and

caretaker.

evaluating the accuracy of the Eye-Tracker

algorithm, a phone-based eye-tracking technology,

in distinguishing between gaze towards the eyes and

mouth of a face displayed on a smartphone screen.

This technology can be used to monitor gaze

aversion behaviors in individuals with ASD over an

extended period through a mobile-monitoring tool.

insights into the progression of their condition over

time. It is also under the same context as above.

same context, that suggested a system autism

screening system that replaces the conventional

scoring functions in classic screening methods with

deep learning algorithms. The system is composed

of a mobile application that provides the user

interface for capturing questionnaire data. an

intelligent ASD detection web service that interfaces

CNN, a database to store the subject’s responses and

test results, and the CNN screening algorithm that

detects autistic traits. The Autism AI app is required

to communicate with the web service that interfaces

and implements CNN. The app captures and verifies

relevant user data (behavioral traits and

demographic features) and feeds them to CNN via

the web service. The Autism AI app also generates a

report that the user can provide to health

professionals.

video by the tracking-learning-detection method.

Secondly, they divide these tracking trajectories into

two components: (I) the length; and (II) the angle.

Afterward, they calculate an accumulative

histogram for each component. Finally, they adopt a

three-layer Long Short-Term Memory (LSTM)

propose a framework to help classify ASD children

in raw video data This framework has consisted of

four stages: (i) manual labeling; (ii) eye tracking;

(iii) histogram computation for the length and the

angle, respectively; and (iv) construction of long

short-term memory (LSTM) network for

classification. Firstly, they manually choose the eye

area and apply Tracking-Learning-Detection (TLD)

to obtain trajectory features, which can be divided

into the length feature and the angle feature. In this

stage, which is the stage of showing the results.

study [24]. This work is based on the idea of the

ability of adolescents with autism to self-tracking

using customizable tools to suit their needs in

adulthood, and whether this tool will be successful

in this challenge or not. In this work, they

investigated how adolescents with ASD kept track

of their everyday lives using a custom self-tracking

platform, called Omni-Track. The Omni-Track

experiments and analyze the collected data. They

installed Omni-Track on the participants’

smartphones to observe how they used it over a

period, to elicit feedback on how the use of Omni-

Track may or may not have addressed their needs

and concerns, and to critique the technology by

describing their experiences.

aims to introduce children to various emotions,

improve their emotional state, recognize emotional

and providing guidance on learning through AI. The

system also uses two databases - Emotions and Data

from students with autism. The fifth process,

guiding learning through AI, is further broken down

into four subprocesses, which include obtaining the

result of recognizing the emotions of a student with

reactions, preparing a personal teaching

methodology based on a recognized emotional state,

and making recommendations to improve the

response of students with autism. It is also under the

same previous context.

autism spectrum disorder (ASD) using various

machine-learning methods. They experimented with

feature selection methods, analyzed the impact of

different features, and tested different machine-

Random Forest, MLP, and CNN with their default

hyperparameters, except for MLP where they used 1

hidden layer with 65 hidden units and SGD solver.

For CNN, they used a basic 1D convolutional layer,

followed by a pooling layer, dense layer, and output

layer. It is also under the same context as the one

that precedes it.

environment all day long without caregivers. The

SWB consists of an accelerometer, gyroscope,

magnetometer, GPS tracker, heart rate sensor,

pedometer, and temperature sensor. The output

device be a sound box and a computer screen, which

will contain a camera to take images of the patient

for emotion analysis every 5 min. AI models will

analyze the health condition of the autistic

individual and operate through some visuals and

sounds.

comprised of a wristwatch-like protective case

integrated into an elastic band, maintaining the

contact of the sensors with the skin. The electronic

module contains an inductive (contactless) battery

charging system, with the receiver coil located on

the outer face of the module.

The same method was used in the study with some

other additions that suggested a wearable emotional-

based e-Healthcare controller using electrodermal

activity and speech recognizer sensors to obtain

physiological data from autistic children. The

system architecture is divided into two modules, the

first effective controller and e-Healthcare modules.

The second effective module comprises an

emotional controller, an emotional mobile

application, a dedicated MATLAB server, and a

electricity in the sympathetic autonomic nervous

system and the Speech recognition sensor is used to

capture some sets of commands from the patients.

The recommender sub-module gathers inputs from

autistic patients, which are further analyzed and

classified using a fuzzy inference system (FIS) to

map out emotion into seven emotional outputs.

quite like the previous studies, this study aimed to

evaluate HR during different interactive activities as

has been merged with three electrocardiographs and

a piezoelectric sensor to monitor cardiorespiratory

activity in children with ASD and language disorder

(LD). The monitoring system includes a

microcontroller with a wireless interface able to

transmit the acquired parameters to a personal

computer for post-processing analysis. The wearable

sensor device integrated a three-lead

electrocardiograph (ECG) sensor for cardiac activity

monitoring. The system is characterized by an ad-

movements recorded by wearable sensors and try to

detect abnormal movements in patients with ASD in

a novelty detection framework. In this proposed

deep normative model, a movement disorder

behavior is treated as an extreme of the normal

range or, equivalently, as a deviation from the

movement detection, they are using wearable

sensors, modern detection is defined as detecting

atypical movements in the test phase while only

normal movements are available in the training

phase. In this study, they consider a probabilistic

modern detection approach consisting of three steps:

the first step is learning the distribution of normal

movements using a probabilistic to denoising

autoencoder. The second step is quantifying the

deviation of each test sample from the distribution

under the same context, they proposed a system for

assisting in intervention strategies for Autism

Spectrum autism spectrum disorder fuzzy logic-

based expert system. The system collects data from

four sensors (GPS, heartbeat, accelerometer, and

sound) via smartwatches, and consists of three units:

sensing, data processing, and application. The data

processing unit uses the expert system and external

database to process the collected data, determine a

has been proposed that works on smart devices and

performs the function of monitoring and

management, as well as receiving notifications

about any activity related to the sick child, in

addition to his actual location, daily measurements,

and other data.

proposed in this study contains nodes. These nodes

represent multiple sensors, some of which specialize

in sensing the skin and others for monitoring

displayed on smartphones and personal computers.

study proposes a system based on EEG sensor data

for people with autism. Then this data is classified

and its features are extracted using a binary pattern

to create spectral images of the brain. The extracted

ShuffleNet, and SqueezeNet). These models are

trained. Previously, we extracted image features in a

deep, uncomplicated way. A two-layered ReliefF

algorithm is used for feature ranking and feature

selection.

like the previous work, which proposed a health

monitoring system consisting of four parts: a

wearable module for collecting physiological data,

an intelligent medicine box module for managing

management, data processing, medication

management, and alerting abnormal data. The

system uses several algorithms for monitoring,

including the effective independence method (EFI),

QR Decomposition Method, modal kinetic energy

method (MKE), modal strain energy method (MSE),

and Guyan Model Reduction Method. These

algorithms are used for configuring sensors on the

degrees of freedom of the model, ensuring effective

monitoring of the patient's physiological condition.

caregiver, or anybody else concerned about the

incident in the house. It consists of three basic

modules—the mobile module, the web service

module, and the sensor module. The mobile module

communicates with the web service module through

Wi-Fi or cellular network. An Android application

configures the user’s preferences such as the way

he/she would like to be informed about an incident.

These alert options can involve either playing audio,

starting a vibration or displaying an animated image.

processing module, where these data are stored,

analyzed, and inferred. The sensor module controls

and manages the sensors.

[37], proposed a framework that includes several

devices, including Tri-axial accelerometers to

discrete and transferable devices with flexible

placement. Accelerometers were applied in front of

the child at the wrists, waist, and/or ankles, or

within pant pockets. A maximum of six

accelerometers were placed on each child. Video

recordings were also obtained for each child, using

three cameras. Two side cameras and an overhead

camera. These recordings provide labeling for the

accelerometer data. These devices are used for Self-

injurious behavior detection. The purpose of using it

various sensors including MQ-135, MQ-3, and

DHT11 that are connected to an Arduino Uno

board, which acts as the processor. If any of these

parameters exceed a threshold level, an alert is

triggered, and appropriate actions can be taken. The

data generated by the sensors is stored in the

Thingspeak cloud and if any abnormal condition is

detected, the user is notified through the Pushbullet

application. The system also includes a buzzer as an

actuator to alert the user of danger. It also falls

deviations in the behavior of children. The system

uses deep learning of artificial neural networks to

recognize the movements and facial expressions of

the child. It also includes virtual and augmented

reality to create educational modules for social

adaptation, telemedicine technology for remote

monitoring by doctors, and IoT technology for

managing mobile health devices. The system

includes a set of mobile medical devices and a

program complex for device control, which

software for remote recognition of emotions. The

system also includes a hardware-software system for

psychological relaxation using the impact of virtual

reality. The system uses methods of intelligent data

processing to support medical decision-making

through the analysis of data from mobile medical

questionnaires.

previous work, which proposed a business process

model and notation (BPMN) extension to enable the

Internet of Things IoT-aware business process

(BP)modeling. Second, they present IoT-fog-cloud

based architecture, which (i) supports the distributed

inter and intralayer communication as well as the

real-time stream processing, (ii) enables the

multiapplication execution within a multitenancy

child, and coronavirus disease 2019 monitoring

systems.

sensor to monitor the interaction between two

people using a Bidirectional Long Short-Term

Memory Neural Network (BLSTM-NN). The 3D

skeleton of each user is detected and tracked using

the Kinect, and the data is modeled using BLSTM-

BLSTM-NN which is pre-trained to recognize

skeletal movement and extract its kinematic

features.

based on extracting features of temporal and spatial

activities as well as fine facial expressions of

children with autism during a collapse crisis and in

the natural state of the same person. The previously

recorded data is compared using the Kinect camera

in both cases (natural, collapse crisis). Temporal and

while eliminating repeated and insignificant

features. It is also under the same context as above.

This method was used in the study [43], to develop

a robot-assisted therapy (RAT) system that enables

psychological disposition and mapping it to

appropriate actions under the supervision of a

therapist. The system generates task-based social

behaviors to achieve therapeutic goals and allows

robot control to be shared with human therapists to

ensure safe and ethical behaviors. The system

applies to various therapeutic scenarios and

platforms and analyzes recorded data to provide

information to different parties. An advanced

sensory system translates multisensory data, such as

suggested in the study [44], is a robot-mediated

therapy and assessment system for children with

autism spectrum disorder (ASD) of mild to

moderate severity and minimal verbal capabilities.

The system uses an NAO humanoid robot with an

additional mobile display to present emotional cues

and solicit appropriate emotional responses. The

mobile phone displays emotions using a custom-

designed mobile application called "Emotions

Selector," which accepts control messages to switch

accumulate an attention score. The Haar classifiers

are used to detect the faces of patients from the

image frames received from the mobile camera. The

assessment system increases the attention score if

the patient is facing the front of the robot body,

where the mobile phone is attached, and decreases

the attention score if the patient's face is not

detected. The operator can change the preset

increment and decrement values for individual

patients, and this score value is updated in each

2 (Appendix).

We classified the discussions of previous studies

put each discussion group under a specific title as

follows:

sensor, privacy, and security issues, and perhaps the

cost, which depends on the type of device used. The

methodology proposed in the study [34], based on

of reading data. The system uses algorithms that are

not computationally complex and accurate in

determining the features of the extracted data.

However, we still believe that the system has some

drawbacks, such as the size of the samples allocated

to train the system, and thus a lack of diversity of

data, clinical applicability, and generalizability of

the system. The methodology used in the study [35],

includes smart monitoring units, wearable sensors,

an intelligent management system for used

of a simple statistical system with little complexity,

effective detection of abnormal behavior, and the

generality of the system. However, the system has

methodology.

of which are accuracy in the process of tracking

patient behavior, efficiency in the process of

extracting time series features from video frames,