A modified neural network model for Real-time Driver
Drowsiness detection system
APASH ROY
Department of CSE, NSHM knowledge Campus Durgapur, INDIA
DEBAYANI GHOSH
Department of ECE, Thapar University, Patiala, Punjab, INDIA
Abstract: - World is running fast. With the speed of communication technology, there is a boom in the
transportation industry also. The transportation vehicles are operating day and night to provide proper
support of the need. This is really tiring for the transportation workers, especially the drivers who are
driving the vehicle. A slight negligence of a driver may cause huge loss. The increasing number of road
accidents are therefore a big concern. There is huge research going on to comfort the drivers and
increase the security features of vehicles to avoid accidents. Here is this work, a model is proposed,
which can efficiently detect driver drowsiness. The work mainly focused on building the learning
model. A modified convolutional neural network is built to solve the purpose. The model trained with
a dataset of 7000 images of open and closed eyes. For testing purposes, some real-time experiments are
done by some volunteer drivers in different conditions, like gender, day, night, etc. the model is really
good for daytime and if the driver is not wearing any glass. But with a glass in the eye and in night
condition the system needs improvements.
Key-Words: Computer Vision, CNN, Drowsiness Detection, Machine Learning, face detection
Received: March 9, 2022. Revised: October 17, 2022. Accepted: November 22, 2022. Published: December 31, 2022.
1. Introduction
With the advancement of various fields, the
demand for transportation became high. To
meet the demand, the vehicles are running day
and night. This is really tiring for the
transportation workers, especially the drivers
who are driving the vehicle. A slight
negligence of a driver may cause huge loss.
The increasing number of road accidents are
therefore a big concern. According to the road
accident report published in the website of
ministry of road transport and highways,
Government of India, there were 366138 road
accidents in India in the year 2020, as updates
last till date [1]. One of the major causes is
alertness of the driver, especially during night
time. With the advancement of automated
vehicle technology, the research is going on
different aspects of safety too. The approach to
detect driver drowsiness is one of them. Here
we are in search of an automated system that
can detect drowsiness and alert the driver.
Machine learning is used in diversified areas
like handwritten character recognition [3, 5,
7], medical image processing [2, 4, 6].
Here in this work, a neural network based
model is proposed, which can efficiently
detect driver drowsiness and alert the driver.
The work mainly focused on building the
learning model. A modified convolutional
neural network is built to solve the purpose.
The model trained with a dataset of 7000
images of open and closed eyes. For testing
purposes, some real-time experiments are
done by some volunteer drivers in different
conditions, like gender, day, night, etc. the
model is really good for daytime and if the
driver is not wearing any glass. But with a
glass in the eye and in night condition the
system needs improvements.
2. Study of the Literature
Over the recent years, detection of driver
fatigue has generated quite a lot of research
interest and can be broadly classified into four
categories [8]. The first category deals with
methods based on physiological signals
obtained from the driver, for example,
electroencephalograph (EEG),
electrocardiograph (ECG), and
electrooculogram (EOG) signals [9, 10]. It has
been shown that these methods have good
predictive ability. However, obtaining clean
datasets in this case offers considerable
challenge in designing such methods [11, 12].
The second category relies on the behaviour of
the driver such as decrease in the grip strength
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
77
Volume 4, 2022
on the steering wheel or the lack of ability to
control the steering wheel, both provide a
measure for the driver's fatigue [13]. The
departure of the vehicle from the intended
trajectory, that is, deviation of the vehicle state
can also be a good measure for the driver's
fatigue and forms the foundation for the third
category [14]. Finally, the driver's drowsiness
can also be detection through physiological
reactions from the driver, such as closed eye
over a duration, which is the focus of the
current work. The frame wise facial
expression and their ratio is used for detecting
the drowsiness in [15]. Another work based on
facial expressions [16] claims 95.58%
sensitivity and 100% accuracy for o-line
detection with SVM classifier. Eye closure
and yawning ratios is also used as facial
expression, and classified through machine
learning algorithm to detect drowsiness [17].
Considering eye as the only facial expression
to detect sleepiness is sufficient and is
established in many work [18, 19].
3. Proposed Methodology
3.1. The Overall System
The methodology adapted in the current work
is shown in Fig. 1. The first stage consists of a
webcam that captures the real-time video of
the driver. It then locates the face as the first
region of interest (ROI) from then the eye,
which is the second region of interest. The
second stage consists of a previously trained
Convolutional Learning Model that is used to
classify; the job of the classifier is to classify
the state of the eyes as `open' or `close'. The
final stage of the proposed system is to ring an
alarm if the eyes are found close for some
threshold value (3 seconds in this case).
Figure 1: The Overall System
3.2. Pre-processing
Before producing the input, the image is
converted to grayscale image and basic
transformations- translation, rotation and
scaling are applied. To translate the original
coordinates (X, Y) to a new translated
coordinate (Xt, Yt) by the shift coordinate (Tx,
Ty), the basic rule of translation is performed
as follows-
[Xt] = [X] + [Tx]
[Yt] = [Y] + [Ty]
(1)
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
78
Volume 4, 2022
To rotate the images from coordinate (X, Y) to
the desired angle Ɵ with new coordinate (Xr,
Yr), the basic scaling technique is used as
follows-
[Xr, Yr] = [X, Y]
for positive rotation
= [X, Y]
for negative rotation
(2)
Scaling is achieved by multiplying the original
coordinates (X, Y) with a scaling factor (Sx,
Sy), which is calculated by comparing the
desired input size and the given input image
size. The desired coordinate points are (Xs, and
Ys) are calculated as follows
(3)
Finally, the Input layer, the image matrix of
size 24x24 is produced as input to the input
layer of the model.
3.3. The Model
We now briefly explain the Convolutional
Neural Network model, which we use here for
classifying the images of the eye as open or
closed. A CNN model is a feed-forward
Neural Network which consists of the
following layers: (i) Input layer, (ii) Hidden
layers and an (iii) Output layer. The hidden
layers consist of Convolution layers, activated
with ReLU function and pooling layers. In
particular, the work- flow of our CNN model
is as shown in Fig. 2.
The hidden layers consist of 3 Convolution
layers, each activated with ReLU function to
extract significant features from the input
images to rectify the feature maps. Note that
the ReLU activation function is as follows:
f(x) = max(0, x).
(4)
Each convolution layer is succeeded by a
pooling layer that reduces the dimensionality
of the feature map. Here, we have
implemented a max pool operation in the
pooling layer.
In the first convolution layer, the input data is
convoluted with 32 filters of size 3x3 with a
stride of 1. So it produces a matrix of 22x22
elements in output. Then a max pool layer with
2x2 filter and stride 2 is used to reduce the
dimensionality of the matrix. Some dropouts
are also used. In the same manner, another two
convolutional layers with 32 and 64 filters are
used. Then finally a flatten layer is used to
prepare vector input for a fully connected
network, which is activated using softmax
activation.
The fully connected layer classifies the eye as
open or close" based on the input feature map.
The CNN model is first pre-trained with a set
of 70,000 eye images, for classification. Then,
in real-time, the same CNN model is used to
classify the state of the eyes from the frames
of the video, captured through the webcam.
Figure 2: The neural network model
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
79
Volume 4, 2022
4. Results and Discussion
4.1. Implementation
The implementation is done in a real time
manner. It captures real time video frames
through a camera and spots the eye as a region
of interest. Using the help of a pre-trained
CNN model it classifies if the eye is opened or
closed and gives an alarm sound if the eye is
found closed for a predefined threshold time
(here 10 second in our case). A webcam is
used to capture the image, then several python
packages are used like OpenCV to detect the
face and eye, Keras to build the model,
TensorFlow as Keras used it as backend, and
finally pygame to play alarm sound.
Table 1: observed output during trials
Driver
Sex
No of
trial
Success in
Day with
Glass
Success in
Day
without
Glass
Success
in Night
with
Glass
Success
in Night
without
Glass
Total
trial
Total
Success
D1
M
25
22
25
19
20
100
86
D2
M
25
24
25
19
21
100
89
D3
M
25
24
25
18
21
100
88
D4
M
25
21
25
19
21
100
86
D5
M
25
23
25
18
22
100
88
D6
M
25
22
25
16
22
100
85
D7
M
25
25
25
19
20
100
89
D8
F
25
25
25
18
21
100
89
D9
F
25
22
25
19
21
100
87
D10
F
25
24
25
16
21
100
86
Total
250
232
250
181
210
1000
873
%
--
92.8
100
72.4
84
87.3
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
80
Volume 4, 2022
4.2. Dataset
The CNN model is trained with a dataset with
70000 eye images, found in the website
https://www.kaggle.com/datasets/serenaraju/y
awn-eye-dataset-new. It consists of open and
close eye images taken from different persons
of both male and female gender. Various light
conditions are also covered.
Figure 3: Summary of trials
4.3. Trials
A total of 1000 trials by 10 volunteer drivers
(7 males and 3 females), is taken into
consideration. The test is taken in day and
night times and with or without glass. Fig. 3
shows the summary of the results. With a good
light condition during day times, and if the
driver is without glasses, the system shows an
outstanding result of 100% accuracy. But, if
the driver is wearing glasses, or the light
condition is not appropriate during night
times, the system seeks refinement. However,
considering all scenarios for day and night,
drivers with or without glasses, male and
female, the overall accuracy is 87.9%, which
is quite encouraging.
5. Conclusions
There are a huge number of road accidents
recorded due to drowsiness of drivers. To
overcome this, people are looking for an
automated system which can detect and alert
the driver if there is drowsiness. Here in this
work one such approach is proposed. A simple
architecture with a camera, a CNN trained
with huge data set is used to detect the
drowsiness. The camera captures the real time
video; the system spots the face and finally the
eyes as region of interest. Then the eye region
is fed into the CNN, which is pre-trained to
classify the eyes are closed or open. If the eyes
are found close for 3 seconds at-a-stretch, the
system rings an alarm, and the driver gets alert.
The trials are showing outstanding result
without a glass and day time. But when the
light conditions are tough at night or the drive
is with a glass, the system needs refinement,
which is the focus of our future work.
0
10
20
30
40
50
60
70
80
90
100
Success
in Day
with
Glass
Success
in Day
without
Glass
Success
in Night
with
Glass
Success
in Night
without
Glass
Summary of results
in trial
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
81
Volume 4, 2022
References:
[1]. https://morth.nic.in/road-accident-in-india
[2]. Swapnil Deshmukh, Apash Roy, Early
Detection of Diabetic Retinopathy using
Vessel Segmentation based on Deep Neural
Network, proceedings of International
conference on "Recent Advances in
Materials, Manufacturing and Machine
Learning (RAMMML-2022), Nagpur, 26th -
27th April 2022.
[3]. Jagdish Kumar, Apash Roy, DograNet- a
comprehensive offline Dogra handwriting
character dataset, Journal of Physics, 2251
(2022) 012008 IOP Publishing
doi:10.1088/1742-6596/2251/1/012008.
[4]. Samia Mushtaq, Apash Roy, Tawseef
Ahmed Teli, A Comparative Study on
Various Machine Learning Techniques for
Brain Tumor Detection Using MRI, Global
Emerging Innovation Summit (GEIS-2021),
2021, 125-137, Bentham Science Publishers.
DOI:10.2174/97816810890101210101
[5]. A. Roy and D. Ghosh, "Pattern
Recognition based Tasks and Achievements
on Handwritten Bengali Character
Recognition," 2021 6th International
Conference on Inventive Computation
Technologies (ICICT), Coimbatore, India,
2021, pp. 1260- 1265, IEEE. doi:
10.1109/ICICT50816.2021.9358783.
[6]. S V Deshmukh and A Roy, An empirical
exploration of artificial intelligence in
medical domain for prediction and analysis
of diabetic retinopathy: Review, J. Phys.:
Conf. Ser. 1831 012012. doi:10.1088/1742-
6596/1831/1/012012.
[7]. Apash Roy, Handwritten Bengali
Character Recognition- a Study of works
during current decade, Advances and
Applications in Mathematical Sciences
(0974-6803). Volume 18, Issue 9, July 2019,
Pages 867-875.
[8]. L. Wang, X. Wu, and M. Yu, Review of
driver fatigue/drowsiness detection methods,
Journal of Biomedical Engineering, vol. 24,
no. 1, pp. 245248, 2007.
[9]. G. Borghini, L. Astol, G. Vecchiato, D.
Mattia, and F. Babiloni, Measuring
neurophysiological signals in aircraft pilots
and car drivers for the assessment of mental
workload, fatigue and drowsiness,
Neuroscience and Biobehavioral Reviews,
vol. 44, pp. 5875, 2014.
[10]. T. Myllylä, V. Korhonen, E. Vihriälä et
al., Human heart pulse wave responses
measured simultaneously at several sensor
placements by two MR-compatible bre optic
methods, Journal of Sensors, vol. 2012,
Article ID 769613, 8 pages, 2012.
[11]. M. Simon, E. A. Schmidt, W. E. Kincses
et al., EEG alpha spindle measures as
indicators of driver fatigue under real traffic
conditions, Clinical Neurophysiology, vol.
122, no. 6, pp. 11681178,2011.
[12]. S. K. L. Lal and A. Craig,
Reproducibility of the spectral components
of the electroencephalogram during driver
fatigue, International Journal of
Psychophysiology, vol. 55, no. 2, pp.
137143, 2005.
[13]. B. T. Jap, S. Lal, P. Fischer, and E.
Bekiaris, Using EEG spectral components to
assess algorithms for detecting
fatigue,Expert Systems with Applications,
vol. 36, no. 2, pp. 23522359, 2009.
[14]. C. X. Huang, W. C. Zhang, C. G. Huang,
and Y. J. Zhong, Identication of driver state
based on ARX in the automobile
simulator,Technology & Economy in Areas
of Communications, vol. 10, no. 2, pp. 6063,
2008.
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
82
Volume 4, 2022
[15]. Rohith Chinthalachervu, Immaneni Teja,
M. Ajay Kumar, N. Sai Harshith, T. Santosh
Kumar, Driver Drowsiness Detection and
Monitoring System using Machine Learning,
Journal of Physics: Conference Series 2325
(2022) 012057 IOP Publishing
doi:10.1088/1742-6596/2325/1/012057
[16]. Mr. J. Haribabu, T. Navya, P. V.
Praveena, K. Pavithra, K. Sravani, DRIVER
DROWSINESS DETECTION USING
MACHINE LEARNING, Journal of
Engineering Sciences, Vol 13, Issue 06,
June/2022.
[17]. N. Prasath, J. Sreemathy and P.
Vigneshwaran, "Driver Drowsiness
Detection Using Machine Learning
Algorithm," 2022 8th International
Conference on Advanced Computing and
Communication Systems (ICACCS), 2022,
pp. 01-05, doi:
10.1109/ICACCS54159.2022.9785167.
[18]. S. Cheerla, D. P. Reddy and K. S.
Raghavesh, "Driver Drowsiness Detection
using Machine Learning Algorithms," 2022
2nd International Conference on Artificial
Intelligence and Signal Processing (AISP),
2022, pp. 1-6, doi:
10.1109/AISP53593.2022.9760618.
[19]. Al Redhaei, Y. Albadawi, S. Mohamed
and A. Alnoman, "Realtime Driver
Drowsiness Detection Using Machine
Learning," 2022 Advances in Science and
Engineering Technology International
Conferences (ASET), 2022, pp. 1-6, doi:
10.1109/ASET53988.2022.9734801.
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
83
Volume 4, 2022
Contribution of Individual Authors to the
Creation of a Scientific Article (Ghostwriting
Policy)
The authors equally contributed in the present
research, at all stages from the formulation of the
problem to the final findings and solution.
Sources of Funding for Research Presented in a
Scientific Article or Scientific Article Itself
No funding was received for conducting this study.
Conflict of Interest
The authors have no conflicts of interest to declare
that are relevant to the content of this article.
Creative Commons Attribution License 4.0
(Attribution 4.0 International, CC BY 4.0)
This article is published under the terms of the
Creative Commons Attribution License 4.0
https://creativecommons.org/licenses/by/4.0/deed.en
_US
Engineering World
DOI:10.37394/232025.2022.4.10
Apash Roy, Debayani Ghosh
E-ISSN: 2692-5079
84
Volume 4, 2022
