Abstract — The purpose of the study is to analyse and compare the most common machine learning and deep learning techniques

used for computer vision 2D object classification tasks. Firstly, we will present the theoretical background of the Bag of Visual

words model and Deep Convolutional Neural Networks (DCNN). Secondly, we will implement a Bag of Visual Words model, the

VGG16 CNN Architecture. Thirdly, we will present our custom and novice DCNN in which we test the aforementioned

implementations on a modified version of the Belgium Traffic Sign dataset. Our results showcase the effects of hyperparameters

on traditional machine learning and the advantage in terms of accuracy of DCNNs compared to classical machine learning methods.

As our tests indicate, our proposed solution can achieve similar - and in some cases better - results than existing DCNNs

architectures. Finally, the technical merit of this article lies in the presented computationally simpler DCNN architecture, which

1. Introduction

N our modern day and age, the amount of data provided daily

by individuals and businesses is rapidly increasing due to

new technological advancements. The use of the Internet of

Things and especially Machine-to-Machine communication

channels have helped create a large interconnected network of

computing devices. Specifically, the increasing use of mobile

devices equipped with a large variety of sensors, various

everyday embedded devices, and everyday tasks such as web

time and implement techniques harnessing the power of

Artificial Intelligence (AI) to provide valuable data insights

categorised into descriptive, diagnostic, predictive, and/or

prescriptive results [1].

Additionally, AI is a field that shows great potential as, due to

Moore’s law that states the computing processing power yearly

increases while the cost is decreased, we are now capable of

handling rapidly and most importantly reliably data points

generated on the spot. Analytically, AI is not a new term as one

data classification tasks, this concept was later used as a

stepping stone to enhance data insights in the field of Computer

vision as it introduced detailed object classification and

recognition in image/video datasets. Lastly, recent advances in

the field of deep learning ANNS [5], [6], [7] focus on data

classification and image recognition tasks via deep learning

techniques [8], [9] in several fields. Most notably this includes

e-commerce [10], finance [11], humanitarian aid [12],

education [13], healthcare [14], and ecological informatics [15].

mentioned process, presenting a detailed comparison of recent

ML and deep learning techniques. Specifically, the outline of

our paper is as follows: firstly, we will present the aims and

objectives of our study. Secondly, we will briefly discuss some

of the most widely used ML data and image classification

techniques in the industry. Thirdly, we will present and focus

on ML and Deep Learning solutions specialising in

classification problems. Fourthly, we will present a detailed

comparison analysis between these traditional ML and Deep

Learning techniques. Fifthly, we will discuss our comparative

widely used techniques regarding data and image

classification. Specifically, our tests are two folded:

1. ML techniques: using the bag of visual words model

(BOVW), K-nearest neighbours and support vector

machine algorithms

2. Deep Learning techniques: using deep convolutional

neural networks, i.e., a pre-trained model and the

proposed ANN

Additionally, using the above-mentioned methods, we aim to

address the following:

4. Suggest a novice CNN architecture for image

classification

Lastly, the technical novelty of this article is not only

presenting a comparative study between traditional ML and

Deep Learning techniques, but suggesting a new CNN that

achieves accuracy levels of slightly over 90% - and in some

cases higher - similarly with the most recent scientific

advances in the field.

3. Background and Methods

the rise of cloud computing infrastructures, and advances in

hardware acceleration techniques (either using GPUs or remote

data centres), 2D object recognition research has rapidly

increased. Specifically, recent research suggests that these

challenges do not pose a computational problem [16], [17],

efforts to provide innovative solutions that take under

power and low-cost methods. This means that emphasis must

be given to study, understand, and ameliorate existing

computational mathematics approaches and methods.

classification techniques focusing on the K-nearest neighbours’

algorithm (KNN) and support vector machine (SVM) classifier.

Moreover, regarding Deep Learning techniques we will

showcase the implementation of two custom convolutional

neural networks (CNN), i.e., one with and one without the use

of a pretrained model. Lastly, for each of the method used we

highlight the most important mathematic components and

address common neural network issues such as overfitting.

In order to conduct this comparative analysis, we have

descriptor [26].

the operating principle behind the BOVW model supports that

to encode all the local features of an image, a universal

representation of an image must be created. This model

compares the examined image with the generated

representation of each class and generates an output based on

following:

5) SVM classifier

The SVM classifier uses an algorithm to select the surface

that lies equidistant from the 2 nearest vector spaces [35]. This

is achieved via classifying dataset into different classes and

calculating the margin between each class, thus creating vectors

that support this margin region (support vectors). Additionally,

in cases where several classes occur such as in our image

recognition dataset, research also uses the “one versus all”

approach where a classifier is trained for each class [36].

Moreover, the mathematical equation for a generic SVM

definition is the following:

Where, instead of a space (x), a space of a kernel K is used.

Lastly, based on the kernel space, the equations derived are

the following:

Where the norm (w) of the equation is:

hierarchical feature learning [38]. These networks take

advantage of the spatial information available on the ever-

increasing areas of images during the system’s processing.

More specifically, due to their structure and their properties

(e.g., Local Receptive Fields, Shared weights, Pooling) they

significantly reduce the parameters and therefore the

computational power over traditional feed-forward fully

connected networks.

overfitting. In the following sections we will explain in detail

each NN architecture as well as the activation functions and

design principles.

custom classifier. Specifically, we use fully connected layers

with batch normalisation and dropout for regularisation, and

mish activation function [41]. Lastly, the final layer of the

classifier consists of 34 neurons using SoftMax activation

functions, one for each class of the dataset.

high-level and more abstract features which are a combination

of low-level features and are problem-specific to each dataset.

architecture is presented in Figure 1 where the input dataset

dimensions are: (128,128,3).

consists of a convolutional layer, where we perform BN and a

max pooling. The second stage uses the same pattern twice, i.e.,

a convolutional layer followed by batch norm, another

convolutional layer and BN and finally max pooling. Lastly, the

Fig.1. Architecture of the proposed NN where convolution, max

pooling, and fully connected layers are highlighted in yellow, orange,

and purple respectively

network. It is noted that in deep learning it is common practice

to double the number of channels (or in our case filters) after

each pooling layer as the knowledge of each layer becomes

more spatial. As we get deeper into the network, each pooling

layer divides the spatial dimensions by 2 thus doubling the

number of filters and available kernels without the risk of rapid

ReLU [45], [46], [47], and Swish [48] and after our tests, we

conclude that Mish [49] is the optimal one. This function

performed better than its rivals in terms of accuracy as it

generated the smaller loss rates and a smoother loss landscape

representation as its behaviour suggests that it avoids saturation

due to capping. The Mish function is defined as follows:

Moreover, a fact of great importance is that smooth activation

functions allow optimal information propagation deeper into

the neural network, thus achieving higher accuracy and

steady learning rate of 0.0002. Analytically, 80% of our

examined dataset was used for training and 20% for validation.

Moreover, to eliminate overfitting, we have used a BN

technique similar to [51] and dropout technique.

measurements of the distributions for all the examined

activations of each batch of our training examples at all levels.

During the feed propagation phase, a normalisation of the

output is applied for each batch, thus each batch average

approximately reaches zero values and standard deviation

is selected to be disactivated (i.e. output 0). This acts as a means

to force the NN to span out its usage and not deeply rely on

specific neurons, thus generalising our solution and reducing

the possibility of developing “single point of error” systems.

This random value is selected based on a Bernoulli distribution,

disactivating connections, NN behaviour is determined as:

Lastly, we used data augmentation methods, where training

data are provided based on a probabilistic approach, i.e.,

calculating a possibility variable in real time during the training

phase and transforming images on the fly. This technique was

used during the training and not the testing or validation phase.

Except for comparing the dictionary size and focusing on

Table 4. Training Accuracy and Loss between Proposed and

pretrained model (VGG16)

Fig.7. Confusion matrix of the optimal SVM (Kernel=

inter, Vocab_size = 3000).

Fig.8. Confusion matrix of the optimal KNN ( K= 10,

0.1321 alike. Moreover, the system presented had a high

validation and testing accuracy of 97.16% and 97.44%

respectively, achieving optimal results in various test cases. In

addition, the pretrained model used as a point of reference and

comparison with our suggested CNN, initially scored lower loss

finally reached validation and testing loss of 0.1011 and 0.1191

and validation and testing accuracy of 97.5% and 97.16% alike.

The comparison between the proposed and pretrained model

shows that our system is capable of achieving similar results

with existing CNNs but authors suggest that for medium to

to be used either as a standalone solution or as part of existing

vehicles’ automatic control systems used to validate the

prediction (i.e. ground truth) of other sensory data points (e.g.

from camera or lidar) predictions.

Finally, future system expansions should shift their attention

into developing a dynamic model that automatically searches

and combines existing activation functions to achieve better

results. Moreover, we need to expand our architecture similarly

to the studies of vision transformers designs, studying related

datasets such as [58], [59], [60] as well as self-supervised