Deep Learning Based Multi-Modal Biometric Security System Using

Visible Light Communication

ARTHI.R, MANOJKUMAR.D, AKSA ABRAHAM, ALLADA RAHUL KISHAN, ALEKHYA

SATTENAPALLI

Department of Electronics and Communication Engineering

SRM Institute of Science and Technology,

Ramapuram Campus, Chennai.

INDIA

Abstract: Multi-biometric system is an advanced technology which has a wide application space in the field of information security.

This work proposes the design of a personal identification system based on a combination of biometric inputs such as face, finger vein,

iris and fingerprint. Viola jones algorithm is used for face detection. Convolutional neural network (CNN) with different optimisers are

used to steeply raise the image texture and extract high definition distinct features of the input images. The deep dream image algorithm

accompanies CNN by visualizing these images and by highlighting the image features learnt by the network. These images are used for

understanding and diagnosing network behaviour. This network obtains a high recognition rate, which proves to be better performing than

traditional algorithms. In addition to these, a high-speed advanced wireless communication technology (Li-Fi) is used in combination with

GSM which would act as an alert system that effectively helps in notifying the supervisory authority, if the system is being trespassed

without proper authentication.

Key-Words: - Multi-biometric, Convolutional Neural Network (CNN), Viola Jones Algorithm, Deep Dream Image Algorithm, SGDM,

RMS Prop, Adam, Light Fidelity (Li-Fi)

Received: March 10, 2021. Revised: October 16, 2021. Accepted: December 12, 2021. Published: January 7, 2022.

1 Introduction

Biometrics contribute significantly in the field of

information security and plays a cardinal role in the

area of authentication and access control. Biometric

is metrics concomitant with human attributes.

Biometric authentication is a mark of identification

[1] and has many applications for personal and

industrial needs. Biometric authentication is an

encapsulation of biometric identifiers and high-end

processing algorithms. Biometric identifiers are

classified into various characteristics. A multi-

biometric system is an advanced technology that

fuses various biometrics [2] or different sets of the

same biometrics in order to tighten the security and

prevent the system from spoof attacks. Multi-

biometric systems have a higher rate of security [3]

when compared to unimodal biometric systems. The

fusion of the different biometrics or different sets

of same biometric takes place in different stages

of the processing stage. These algorithms

strengthen the core of entire system.

Various algorithms are used for processing the

biometric input. CNN is one of the most enhanced

and highly accurate algorithms that has a higher rate

of recognition and accuracy. It works on the principle

of biological visual cortex. The neurons in brain is

the study pattern and origin sets the base of this

algorithm. These neural networks are used in various

arenas of a processing stage such as object detection,

image classification, image enhancement, etc. CNN

has a specific workflow that includes loading the

data, analyzing the data, data preprocessing, creating

the network, modelling the data, training the model,

model evaluation of the test set, predicting labels and

finally creating the classification report. Once the

image input has been processed and matched, the

processed and matched input [ 4] were mapped to the

user specific application or the industry specific

application.

Biometrics has a wide space of applications that

includes specific and combined applications [5]. Data

transmission plays a vital role [6] in various varieties

of application based needs. GSM is one of the most

common means of transmission. The security breach

has been notified via sending a message to the

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022

authorized person using GSM.

The distinct characteristics of this paper can be

concluded in three-fold: Firstly, we propose a multi-

biometric system using four physiological biometric

identifiers namely face, finger vein, iris and

fingerprint. Viola jones algorithm has been used for

face detection. Secondly, the input images are

processed using convolutional neural network

algorithms. The mechanism of occlusion has been

used to adjust the parameters and also the number of

layers in the network. The deep dream algorithm

accompanies CNN by visualizing these images and

by highlighting the image features learnt by the

network. These images are used for understanding

and diagnosing network behavior. Thirdly, high

speed advanced wireless communication technology

(Li-Fi) is used in combination with GSM and an alert

system which effectively helps in notifying the

supervisory authority, if the system is being

trespassed without proper authentication.

Section 1 contains the Introduction. Section 2

gives a list of all the related works that is used in this

paper to make it more informative. Section 3 contains

the System Architecture and the usage of components

that has been used in the following system. Section 4

discusses about the Results where an overall

experience on the possibilities of the various

optimizers used and the alert system is given. Section

5 concludes and discusses about the future work.

2 Related Work

A new multi-biometric system with many traits such

as face, voice and iris are integrated [7] with a multi-

model biometric system to overcome the limitations

of the unimodal biometric system. The ROC curves

of three-single biometric were plotted to compare the

accuracy rate of each system. The result obtained

from the ROC curves states that the multi-model

system provides better accuracy than the individual

system.

From [8], the authors had proposed a new concept

in the year 2018 that is CNN for extracting the

features and identifying the user. The authors

observed that in the field of authentication of

biometrics, CNN produces a result with roughly

100% accuracy rate. The layers which have the

capacity to produce high quality results are fully

connected and Convolutional. From [9] various

biometric systems are compared. CNN does the

extraction of features. According to the author, the

multi-model gives a result of high accuracy along

with excellent performance as compared to the

unimodal. It has also determined the score level

fusion gives 100% accuracy than feature level fusion

that gives only a rate of 99.22%.

From [10] the training on fractionally occluded

images is reduced. A training process is proposed to

reduce the spatial region of the filters. Three types of

regularizations are considered. From [11], [12] and

[13], the optical wireless transmission of data has

been proposed and LED’s were used. It was proved

that the data rate is effective, and also provides an

alternate solution for the radio frequency spectrum

crisis.

3 System Architecture

The complete System Architecture has been

divided into three parts. In the first part, the input

images of face, finger vein, iris and fingerprint are

obtained using the biometric devices and the images

are preprocessed, cropped, resized, and integrated

with CNN. In the second part, three different

optimizers are used and they are varied with each

simulation to obtain the most efficient optimizer.

Changes are made to the hyper parameters that

calculate the loss and gain accordingly.

In the third part, high speed advanced wireless

communication technology (Li-Fi) has been used in

combination with GSM and an alert system which

effectively helps in notifying the supervisory

authority, if the system is being trespassed without

proper authentication. The process of the system

considers the input image of the face, finger vein, iris

and fingerprint biometric has been taken with the

help of biometric devices respectively. The Proposed

System Architecture of multi-biometric Transmitter

is as shown in Fig.1. and Receiver with an alert

system is as shown in Fig.2.

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022

Fig. 1. Architecture of Multi-Biometric

Transmitter

Fig. 2. Architecture of Multi-Biometric Receiver

with an alert system.

The input image after getting preprocessed

[14] gets converted from RGB to gray to make

extraction easier as it doesn't have any color

information. The input image is preprocessed,

converted and operated by CNN and later is

processed by the deep dream image algorithm which

highlights the important features of the image. This

highlighted image has been passed to the layers in

CNN. The training of CNN was done with three

different optimizers, namely: SGDM, RMSPROP,

ADAM for comparison of the best system in terms of

speed, accuracy, etc. Occlusion has been applied to

obtain the exact feature that the network is looking

for. These processed trained input images are

matched with biometric images of the authorized

person from the database and thereby the results are

obtained. While matching takes place, the system has

been subdivided into two parts based upon the results

of matching. If matching takes place successfully for

all the mentioned biometrics, the person was given

access to the system. Else, the system passes through

the security control that triggers the Li-Fi and GSM

present in the system. Li-Fi helps in data transmission

and through GSM the message is sent to the

concerned authority, if the security breach takes

place.

2.1 Deep Dream Image Algorithm

Deep Dream Image Algorithm is a visualization

technique that deals with the features of a specimen

by processing an image in the network layer. The

algorithm uses gradient ascent [15] [16] [17] [18] that

is set equal to the activations from that layer done on

the image. Therefore, this maximizes the activations

of that layer. The way deep dream algorithm can be

used is:

(1) The given input image is passed to the deep

dream algorithm and the network is trained along

with back propagation to modify the input image

in turn to diminish categorization errors.

(2) When the image input has been given into

the network, this algorithm utilizes to emphasize

the essential features in the specified input

image. These layers can extract out those features

exactly rather than focusing on the other features.

The important advantage with the latter one is that

the weights are not mandatory to be changed when

the classification error occurs. Instead of varying the

weights of the network, it can also be rectified by

highlighting the other features which are not

highlighted before. Therefore, the proposed work

prefers to go with the latter use.

2.2 CNN Architecture

The proposed system architecture consists of an

input layer, 3 convolution layers, namely (conv1,

conv2, conv3), 1 fully connected layer. The

parameters of the network are as shown in Table 1.

Table 2. shows the various optimizers which were

used and also the hyper parameters which were tuned

precisely to avoid the problem of overfitting and

under fitting.

Table 1. Layers of ConvNet and

Parameters

Model

Conv1

Conv2

Conv3

FCL

Convolution

layers

1 Fully

connected

layer

32 x 3 x 20

ReLu

Batch

Normalization

Dropout (0.5)

64 x 3 x 20

ReLu

Batch

Normalizat

ion

64 x 3 x

ReLu

Batch

Normaliz

ation

Dropout

(0.75)

SoftMa

Table 2. Optimizers used and its hyper

parameters

Optimizers

Hyper parameters

● SGDM

● RMS Prop

● Adam

● Momentum: 0.9000

● Initial Learning Rate: Varies

● L2 Regularization: 1.0000e-04

● Max Epochs: 20

● Mini Batch Size: 64

● Verbose: 1

● Verbose Frequency: 50

● Validation Frequency: 50

● Validation Patience: Inf

3 Results and Discussions

The experimental outcome discussed in the

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022

proposed work gives clarity about the feature

extraction at different layers, Mini batch accuracy

over the Learning rate and activation strength

variations for iteration level.

3.1 Activations of CNN layers

The inner processing of all visible layers for all

the four biometrics is obtained. The input image is

taken and it is converted to the binary image to avoid

inter class variations as shown in Fig.3., Fig.4.,

Fig.5., and Fig.6. The input image is passed to

convolution layer to extract the important features

that are necessary. The ReLu activation is then

applied to the image to increase the non-linearity

thereby removing negative values from the image.

As seen, Max pooling layers are dark in nature to

grab maximum value in the image and also to avoid

the area that are not the features of the images. By

this, the parameters are reduced and therefore the

model won't over fit on that information. The desired

classes are quantized in this layer. After the

processing of the image by this layer, the image is

classified on the basis of SoftMax classifier and the

matching output has been displayed.

Fig. 3. Activations of Face input at different

layers of the network

Fig. 4. Activations of Finger vein input at

various layers of the network

Fig. 5. Activations of Iris input at various layers of

the network

Fig. 6. Activations of Finger print input at different

layers of the network

Fig. 7. Occlusion Result of the image after brute

force approach

The image obtained from the occlusion is obtained

and is shown above. The probability of all the three

classes are obtained as shown in the Fig.7. After

adjusting the parameters and number of layers in the

network with help of the previous image obtained

from occlusion mechanism, we could see that the

network now obtains the important features from the

region of eyes.

3.2 CNN Architecture

Fig.8, Fig.9, Fig.10 and Fig.11. shows the efficiency

of three different optimizers, namely SGDM, Adam

and RMS prop plotted between learning rate and mini

batch accuracy. The comparison between these

optimizers [19] obtained represents the most efficient

and accurate optimizer for different input images.

Fig. 8. Performance of Face Biometric

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022

Fig. 9. Performance of Vein Biometric

Fig. 10. Performance of Iris Biometric

The input images are trained and iterated more

than 20 times. The mini batch size for the training of

layers is set as 64. The layers are trained with the

above-said optimizers with different learning rates.

Fig. 11. Performance of Finger Print Biometric

SGDM optimizer gives the peak value of the

accuracy rate for finger vein recognition. Adam

optimizer gives the peak value of accuracy rate for

other biometrics as the mini batch accuracy rate is

one hundred-percent for the first three learning rates

(0.0001, 0.001, and 0.01) and gradually decreases for

the rest of the learning rates. The other optimizers

show a hundred percent accuracy only for the first

two learning rates (0.0001, 0.001) and shows a steep

decrease for the rest of the learning rates. The loss is

minimized for Adam optimizer in case of iris, face

and finger print biometrics and SGDM optimizer in

case of finger vein biometric. The time elapse for

finger vein biometric is comparatively lesser than

other biometrics.

3.3 Activation Strength Variations

The Activation strength was varied with respect

to iterations is shown in Fig.12.

Fig. 12. Graphical representation of activation

strength variations

The result was observed with convolution layer 1.

The proposed work clearly infers from the graph that

Activation strength with deep dream has been

increasing when compared to activation strength

without deep dream. It was also observed that the

frequency of categorical errors has been very much

decreased when using deep dream algorithm.

3.4 Hardware outputs for alert system

The hardware part is divided into two parts i.e.

Transmitter and Receiver. Fig.13. shows the

transmitter which is interfaced with the software part

of the system. The serial port connection sends the

data to the transmitter when the matching

criteria/conditions are not met. Hence, this data is

used to trigger the hardware part of the system. From

the software part of the system, COM3 port is used to

send data through the TTL converter to the Arduino.

The Arduino (in transmitter) processes the data

received by the Software. This processed data is the

sent to the light transmitter which transmits to the

receiver part of the Li-Fi.

Fig. 13. Li-Fi Transmitter

Fig.14. shows the receiver part which is interfaced

with the transmitter part. The receiver part receives

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022

the data and is then sent to the Arduino

(Microcontroller) and then processes this data to

trigger LED, buzzer, GSM, and LCD display. LED

will glow red light (blinking) indicating this as an

emergency.

The buzzer produces a beep sound to warn. GSM

triggers the message immediately to the supervised

authority on the system being trespassed. LCD

displays on the screen. The combination of all these

hardware components together can potentially warn

the supervisory authority.

The Fig.15 and Fig.16 delineates the working of

the system. The biometric system transmits the data

(in the form of a char) during every authentication.

Different char helps us in distinguishing if the access

has to be granted or be denied.

Fig.15.Working of Li-Fi transmitter

Fig. 16. Working of Li-Fi receiver

The data which is transmitted from biometric

system is received at transmitter side. If the data

belongs to the category to which access can be

granted, the transmitter ends up in giving the access

to the system. If the data belongs to the

unauthenticated category, the lifi transmitter

transmits the data in the form of Light (Li-fi as in

Fig.15) to the receiver end as shown in Fig.16.

The receiver then glows up the LED, displays

message on LCD, rings a buzzer and also transmits a

message with the help of GSM triggered by the

Receiver to the possessor of the system.

Fig. 17. Message transmitted through GSM

Fig.17. shows the message transmitted by the

GSM from the receiver side warning the possessor

that some authenticated person is trying to trespass

the system.

4 Conclusion and Future Work

Combination of various biometrics quiets the

concern of security in secured places. The network

that the proposed work experimented on contains 3

convolutional layers and 1 fully connected layers.

CNN algorithm has been used for processing the

acquired images and for classifying the images. The

accurate parameters and layers are used due to the

help of the mechanism of occlusion. The categorical

errors are reduced with the help of Deep dream image

algorithm by improving the activation strength of

layers. The accurate tuning of hyper-parameters are

done with the help of occlusion mechanism which

results in the reduction of overfitting and under fitting

problems. CNN is observed to concentrate only on

the necessary features. Three optimizers namely

Adam, RMS prop, SGDM has been used for

Fig. 14. Li-Fi Receiver

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022

simulation and its performance were compared based

on learning rate and their accuracy. From the above

results, it can be concluded that Adam provides

higher stability in the network with respect to

increase in learning rates. An alert system is adjoined

with the biometric system for warning the security

breach and a message has been sent to the authorized

person via GSM. In future, Learning rates can be

increased therefore the time elapsed for the training

could be increased. Many more features can be

appended with the existing biometrics. The net can

be simulated further by using the other optimizers

such as Adagrad, Adadelta, etc. The Grad-CAM can

be used in place of occlusion to provide the better

tuning of hyper parameters and also to get the better

network predictions. Hence, this can reduce the

activity bias. Li-Fi can be replaced by any other faster

means of communication, which can potentially

reduce the line-of-sights problems also.

References:

[1] R.F Tazim, M. M. M Miah, S. S. Surma, M.

T. Islam, C. Shahnaz, & S. A Fattah,

“Biometric Authentication Using CNN

Features of Dorsal Vein Pattern Extracted

from NIR Image”, TENCON 2018-2018

IEEE Region 10 Conference, 2018, pp.1923-

1927.

[2] S Chaudhary, R Nath, “A New Multimodal

Biometric Recognition System Integrating

Iris, Face and Voice”, in International

Journal of Advanced Research in Computer

Science and Software Engineering,2015, pp

145-50.

[3] R Devi and P Sujatha, “A Study On

Biometric And Multi-Modal Biometric

System Modules, Applications, Techniques

And Challenges”, Conference on Emerging

Devices and Smart Systems, 2017, pp. 267-

271 [ICEDSS].

[4] S. Albawi, T. A Mohammed, and S. Al-

Zawi, “Understanding of a Convolutional

Neural Network” in International

Conference on Engineering and Technology,

2017, pp. 1-6 [ICET].

[5] K Lai, S. Samoil, S. N. Yanushkevich, & G.

Collaud, “Application Of Biometric

Technologies In Biomedical Systems” in

The 10th International Conference on Digital

Technologies 2014, pp. 207-216.

[6] M. A Ozkan, & S. B. Ors, “Data

Transmission via GSM Voice Channel for

End To End Security”, IEEE 5th

International Conference on Consumer

Electronics-Berlin, 2015, pp. 378-382,

[ICCE-Berlin].

[7] M. Elhoseny, A Elkhateb, A Sahlol, & A. E

Hassanien, “Multimodal Biometric Personal

Identification And Verification”, Advances

in Soft Computing and Machine Learning in

Image Processing in Springer, Cham 2018,

pp. 249-276.

[8] A Rattani, N. Reddy & R. Derakhshani,

“Multi-Biometric Convolutional Neural

Networks for Mobile User Authentication”

in IEEE International Symposium on

Technologies for Homeland Security

HST, pp. 1-6, 2018.

[9] A. S. Al-Waisy, R. Qahwaji, S. Ipson, & Al-

Fahdawi, S, “A Multimodal Biometric

System For Personal Identification Based On

Deep Learning Approaches” in Seventh

International Conference on Emerging

Security Technologies EST,2017, pp. 163-

168.

[10] E. Osherov, & M. Lindenbaum, “Increasing

CNN Robustness to Occlusions by Reducing

Filter Support” in Proceedings of the IEEE

International Conference on Computer

Vision, 2017, pp. 550-561.

[11] D. Javale, C. A. Sashittal, S. Wakchaure, A.

M. Phadnis, S. S. Patil, & , R. S. Shahane, “A

New Approach to Wireless Data

Transmission Using Visible Light”, in

Fourth International Conference on

Computing Communication Control and

Automation,2018, pp. 1-4, [ICCUBEA].

[12] J. K. Xavier, N. Raveen, A. R. Aneesh & M.

V. Sarada, “Data Transfer Using Light

Fidelity Technology”, in International

Journal of Pure and Applied

Mathematics, vol. 119(15),2018, pp.903-

908.

[13] A.Sarkar, S. Agarwal & A. Nath, “Li-Fi

technology: Data Transmission through

Visible Light”, in International Journal of

Advance Research in Computer Science and

Management Studies, vol. 3(6), 2015.

[14] S. Fairuz, M. H. Habaebi, E. M. A. Elsheikh,

& A. J. Chebil, “Convolutional Neural

Network-based Finger Vein Recognition

using Near Infrared Images” in 7th

International Conference on Computer and

Communication Engineering, pp. 453-

458[ICCCE 2018].

[15] E. L. Spratt, “Dream Formulations and Deep

Neural Networks: Humanistic Themes in the

Iconology of the Machine-Learned Image”,

2018 [arXiv preprint arXiv: 1802.01274].

[16] Arthi. R, Jai Ahuja, Sachin Kumar,

Pushpendra Thakur, Tanay Sharma, “Small

Object Detection from Video and

Classification using Deep Learning”, In:

Springer Lecture Notes in Electrical

Engineering Advances in Systems, Control

and Automations, Proceedings of

ETAEERE-2020, Published online on March

2021, Vol 708, pp 100-107.

[17] C. Aravindan, R. Arthi, R. Kishankumar, V.

Gokul, S. Giridaran, “A Smart Assistive

System for Visually Impaired to inform

acquaintance using Image Processing (ML)

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022

supported by IoT” for book titled “Hybrid

Artificial Intelligence and IoT in Healthcare”

with the Springer Premier Book Series

"Intelligent Systems Reference Library”

Vol: 209, PP 149-164

[18] Arthi. R, Allada Rahul Kishan, Aksa

Abraham, Alekhya Sattenapalli “Centralized

Intelligent Authentication System Using

Deep Learning with Deep Dream Image

Algorithm”, In: Springer Lecture Notes in

Electrical Engineering Advances in Power

Systems and Energy Management,

Proceedings of ETAEERE-2020, Published

online on Jan 2021, Vol 690, pp.169-178.

[19] S. V. Reddy, K. T. Reddy, & V.

ValliKumari, “Optimization of Deep

Learning Using Various Optimizers, Loss

Functions and Dropout” in International

Journal of Recent Technology

Engineering vol.7,2018, pp.448-455.

Creative Commons Attribution License 4.0

(Attribution 4.0 International, CC BY 4.0)

This article is published under the terms of the Creative

Commons Attribution License 4.0

https://creativecommons.org/licenses/by/4.0/deed.en_US

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2022.17.4

Arthi R, Manojkumar D, Aksa Abraham,

Allada Rahul Kishan, Alekhya Sattenapalli

E-ISSN: 2224-2856

Volume 17, 2022