Detection of Steel Structures Degradation through a UAVs and
Artificial Intelligence Automated System
ANTONINO FOTIA1, RAFFAELE PUCINOTTI1, VINCENZO BARRILE2
1PAU Dept., Mediterranea University, Reggio Calabria,
Via Graziella, Feo di Vito, 89124 Reggio Calabria,
ITALY
2DICEAM Dept., Mediterranea University, Reggio Calabria,
Via Graziella, Feo di Vito, 89124 Reggio Calabria,
ITALY
Abstract: - In recent times, the need for the management and monitoring of steel structures (bridges, but also
buildings) has become more and more important; consequently, a new phase has opened up aimed at the
surveillance and monitoring of these structural types with the objective of their protection and preservation,
also through preventive maintenance activities. Leaving aside the world of large structures (industrial buildings,
bridges, etc.), the reality of metal-framed buildings in Italy is not yet strongly established. For this reason,
particular attention must be paid to these types of structures. The application of experimental monitoring
techniques, however, involves the succession and chaining of various established procedures. Visual inspection
is generally the first step to assess any deterioration, but it becomes quite difficult for elements at significant
heights. The operational difficulties can be reduced by the UAV drone. Image processing using soft computing
techniques also offers the possibility of speeding up the inspection by human operators, who can limit
themselves to assessing any damaged parts already selected by artificial intelligence. It is, therefore, necessary
to establish appropriate automatic or semi-automatic inspection procedures mainly aimed at providing useful
indications to operators on intervention priorities. An automatic monitoring and management procedure is
therefore presented, which provides for the detection and evolution of degradation on structural elements and
joints of existing steel structures. The implemented methodology follows five main phases: (a) images
acquisition by UAVs; (b) 3D creation with geometry and degradation; (c) data processing and defect detection;
(d) creation of an "evolutionary" database, able to update the degradation on the basis of the acquisitions made
in subsequent inspections by UAVs; (v) implementation of the structure (with its defects) within a structural
analysis software FEM (Finite Element Method).
Key-Words: -
Computational Intelligence Systems, Soft Computing, Dynamical Systems, Management and surveillance.
Received: March 19, 2022. Revised: October 18, 2022. Accepted: November 14, 2022. Published: December 31, 2022.
1 Introduction
The global transport infrastructure system and in
particular the Italian one is obsolete. The Italian
Ministry of Infrastructure has given a great impetus
towards the monitoring of the road system in
general with reference to both seismic aspects,
landslide movements and flood aspects. In recent
years, there has been a series of instability and
collapses that affect transport networks in general,
causing damage to the community and endangering
the safety of users. For this reason, the Italian
Ministry of Infrastructure started a series of studies
and research activities related to road safety; in
particular, in 2020 guidelines were issued
concerning the monitoring of transport
infrastructures from a
structural/geotechnical/hydraulic point of view. For
this reason, there is a growing demand for better
monitoring of the condition of transport
infrastructure worldwide. Bridges are key
components of transport infrastructure and require
such monitoring. In Europe, most road
infrastructures were built from 1945 to 1965, in the
post-war period. The load conditions of the bridges
have recently changed due to increasing transport
volumes and vehicle sizes. In addition, most of these
bridges are subject to gradual deterioration over
time and many are now structurally deficient, [1],
[2]. Rehabilitation and life extension of these
structures raise important maintenance and safety
issues. However, an increase in bridge inspections
to address existing structurally deficient bridges has
considerable costs and practical implications for
road owners and managers.
Automated monitoring of structures plays a key
role in the digitization of our infrastructure, [3], [4].
WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS
DOI: 10.37394/23201.2022.21.25
Antonino Fotia, Raffaele Pucinotti,
Vincenzo Barrile
E-ISSN: 2224-266X
231
Volume 21, 2022
Bridges, in particular, are increasingly equipped
with measurement technology so that their condition
can be permanently monitored. With reference to
newly built bridges, static and dynamic monitoring
techniques are already proceeding with [5]. The
inclusion of monitoring systems that provide useful
indications on the evolution of infrastructures [6], in
predictive maintenance, such measurement data are
used to initiate maintenance procedures at an early
stage to avoid bridge closures. The difficulty is to
evaluate the data in such a way that bridge
conditions are calculated automatically. Machine
Learning (ML) methods can be useful in this
evaluation.
The difficulty lies in applying the developed ML
methods to actual measured data in practice and
generating added value for the owners and operators
of bridge structures, [7], [8], [9]. Bridge monitoring
systems generate large amounts of data. A manual
evaluation is often unrealistic or only to a limited
extent.
Steel bridges usually have high-strength bolt
connections for the assembly of the load-bearing
elements of the structure. However, these bridges
are often used in adverse environments and are
subject to corrosion, vibration and fatigue, and
thermal cycling, which can contribute to bolt
damage, [10]. Due to the enormous energy released
by brittle fracture, fractured bolts will fail. Damage
to the bolts will threaten the safety of bridges and
may even lead to serious accidents. Therefore, it is
necessary to monitor the condition of the bolt during
daily operation and maintenance.
In the last years, computer vision technology has
gained considerable attention as an interdisciplinary
subject and has been used in monitoring and
inspection activities of civil infrastructure to
enhance the efficiency and accuracy of manual
visual inspection.
A quasi-automatic bolt looseness detection
method was proposed by Huynh et al, [11], in which
plausible bolts were detected using a CNN-based
object detector and the rotation angle of each bolt
was measured by the Hough line transform. A
method for measuring the angle of rotation of bolt
slack using a CNN-based object detector was
proposed by Zhao et al., [12]. A computer vision-
based method integrating perspective transformation
to detect bolt looseness for flange connections was
designed by Wang et al., [13]. However, there is no
automated procedure for all these works.
In this study, we focus mainly on the data
acquisition methodology and the type of monitoring
tested, focusing on the detection of deterioration in
bolted plates using neural networks.
2 Materials and Methods
2.1 Methodology
Bolted joints are very common and important in
engineering structures, [14], [15], [16]. Due to the
extreme service environment and load factors, bolts
often become loose or even slackened. Detecting
loose or disengaged bolts in real time or in a timely
manner is an urgent necessity in practical
engineering, that is crucial for maintaining structural
safety and durability, [10]. Recently, a lot of
machine learning and deep learning techniques and
methods to detect bolt loosening have been
proposed. However, in most of these studies, images
of bolts captured in the laboratory are used for
model training. The images are acquired under well-
controlled light, distance and viewing angle
conditions. It should be noted that in practical
engineering, the above-mentioned well-controlled
laboratory conditions are not easy to realize, and
images of real bolts often have blurred edges,
oblique perspective, partial occlusion and
indistinguishable colors, etc., which cause trained
models obtained under laboratory conditions to lose
their accuracy or falsify.
The proposed system is developed in 4 main phases,
according to the flowchart in the figure (see Fig. 1).
Fig. 1: Flowchart of the proposed methodology.
- Preliminary phase:
o survey and identification of areas to be
examined: necessary step to assess the
presence of any obstacles that the drone
may encounter and definition of points of
interest;
- Flight:
o Design of the flight plan and acquisition:
necessary step to define the flight trajectory
(way point), the acquisition spatial intervals
(choice of image acquisition positions), the
WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS
DOI: 10.37394/23201.2022.21.25
Antonino Fotia, Raffaele Pucinotti,
Vincenzo Barrile
E-ISSN: 2224-266X
232
inclination and rotation angle of the camera,
the travel speed, the duration of the mission;
o Flight and image acquisition: the drone
automatically flies along the planned
trajectory and acquires the images needed
for 3D modelling, orthophoto generation,
degradation detection;
- Processing:
o Reconstruction of 3D model and high-
resolution orthophotos (orthomosaic)
o Detection of deterioration via CNN
o Deterioration image processing assignment;
o Visualisation of deterioration on ortho-
mosaic model;
o Structural analysis of infrastructure for
different deterioration scenarios;
- Visualisation of results
o Creation of a historical database of points of
interest;
o Querying the database of overlapping
layers.
The objective of using drones and planning flight
plans is to allow the end user to always be able to
acquire images from the same position and at the
same angle conditions (using cameras with the same
characteristics), which is necessary to avoid pre-
processing rectification.
On the other hand, the goal of Machine Learning
is to extract knowledge or information and
correlations from these data.
A dataset for the detection of 'bolt' objects in
natural scene images was developed by
implementing it with datasets freely available on the
web, [17]. In this application, the reference
categories applied to object detection are two "head"
and "nut".
Advanced object detection models (such as
YOLO v5) were used for testing the dataset. The
evaluation results show that the bolt target detection
model trained using this dataset can detect and
classify the bolt head and bolt nut well in the natural
environment. In the YOLO v5-l model, the average
accuracy of the two main categories reaches 97.38%
and 91.88% respectively. The proposed dataset
bridges the gap in the current field of bolt object
detection.
3 Case Study
The operations were tested on a low-traffic road in
the territory of the town of Cardeto (RC), Southern
Italy, in an area with low density and traffic. It is an
interurban road infrastructure that connects the city
center with a village that runs along the bank of a
river (Fig. 2).
Fig. 2: Bridge case study.
In order to determine the trajectory of the UAV, it is
first necessary to provide an initial inspection to
avoid interference with any obstacles during data
acquisition, to prepare the take-off and/or landing
point and data transmission, to verify the possible
presence of overflying areas subject to vehicular or
pedestrian traffic, and then to provide for the
installation of all appropriate 'precautionary'
measures to mitigate the danger (parachutes, control
cables, etc.).
Having determined the space of the trajectory
and the points of interest for the inspection (areas to
be acquired), the information was transposed using a
flight planner and then the trajectory was drawn,
and the take-off-landing/data transmission and
image acquisition points were determined (Fig. 3).
Fig. 3: Automated flight plan.
WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS
DOI: 10.37394/23201.2022.21.25
Antonino Fotia, Raffaele Pucinotti,
Vincenzo Barrile
E-ISSN: 2224-266X
233
This provides repeatability of the flight and images
(with the same camera used) that can be easily
compared, especially in terms of changes in the
structure (presence of degradation, improvement
interventions, etc.).
Once the data and information necessary for
analyzing the bridge had been acquired, the various
phases of data processing were conducted. The
images were processed using the Metashape
software and the digital 3D model was reconstructed
from which the geometric characteristics were
extracted. In addition, orthophotos were produced,
in which the various processing steps performed on
the individual frames can be reported.
Subsequently, individual frames were processed
through the proposed model in order to identify
individual bolts. Image processing follows these
steps image acquisition; (2) pre-processing; (3)
processing; and (4) detection of deterioration, [15].
Recently, deterioration detection methods based on
deep learning have been proposed, [16]. CNN has
disadvantages such as a high computation cost and a
long operation time. However, the reduction of the
areas to be inspected within a single frame allows
significant regions of the infrastructure to be
scanned more quickly, [17], [18]. Once the different
frames have been acquired, masks have been
designed, which allows the processing time of
acquisitions made at later times to be significantly
reduced.
In Fig.4 a Flow Chart explaining the image
processing system for detecting the deterioration
[19] [20], [21].
Fig. 4: Explanatory Flow Chart of the deterioration
image processing system
Fig. 5 shows an example of a bolt loosening
evaluation. The frame acquired at instant t (Fig. 5a)
is processed through an edge detection algorithm to
enhance its contours (Fig. 5b) and facilitate
comparison with the frame acquired and processed
at instant t+1 (Fig. 5c). The acquisition of frames at
different instants through the use of UAVs
facilitates this comparison operation, having
designed through a flight planner the acquisition
points while maintaining constant distance and
camera angle.
(a) (b)
(c)
Fig. 5: An example of a bolt loosening evaluation.
Fig.6a shows the results of processing a single
frame where all present bolts are recognized through
YOLO v5. Fig. 6b shows the processing of a frame
in which only the bolts present in certain areas are
recognized. The processing of such frames is
sufficient for the identification of the deteriorations
(slippages and breaks) through the comparison with
the frames acquired in subsequent shots. The same
frames are also used through the use of homologous
points for the identification of the same bolts on
overlapping portions of frames where processing
has failed (Fig.7).
WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS
DOI: 10.37394/23201.2022.21.25
Antonino Fotia, Raffaele Pucinotti,
Vincenzo Barrile
E-ISSN: 2224-266X
234
(a)
(b)
Fig. 6: a) locating bolt heads on deck b) locating
bolt heads on the pier.
Fig.7 shows an example where bolts belonging to
the same area are not recognised by the algorithm
and therefore require further processing for
comparison with frames showing the same areas
where, however, all bolts have been recognised.
Fig. 7: Zoom on an area with processing problems -
red arrows indicate bolts not recognised by the
processing system.
The proposed system also consists of a system for
visualizing the results, which makes it possible to
compare the evolution over time of the variations
occurring on the bridge. Given the repeatability of
the flight plan, the images acquired from a given
position will frame the same areas and thus be easily
comparable, creating a temporal database. This
information available on the visualization platform
helps and enables the operator to 'visually process'
the changes.
In particular, a filter (0-1) will then cause the
image itself to be added to the database whenever a
change from the previous temporal acquisition is
detected. Changes in degradation, depending on the
point of acquisition, will be associated with a
particular element (waypoint determined in the
preliminary phase) and recorded on a CSV file. The
acquisition data and analysis results can then be
made available within a user-questionable platform,
making the results accessible to end-users for
prioritized action considerations.
4 Conclusion
As is well known, Italy's building and infrastructure
heritage suffer from significant problems related to
the age of construction., the state of maintenance
and the particular vulnerabilities due to the adoption
of construction techniques that are not always
adequate with respect to the possible load
conditions.
The antiquity of Italy's infrastructure network
and the absence of a reference database for planning
the maintenance work the infrastructure network
needs is one of the most widespread problems in our
country. As bridges become obsolete, inspection and
maintenance requirements increase [22-23]. In this
work, in accordance with the new guidelines on
Risk Classification and Risk Management and
Safety Assessment and Monitoring of Existing
Bridges, a new methodology is proposed to establish
an appropriate inspection procedure and
intervention priorities. Traditionally, bridge
maintenance has been based on visual inspection
methods that are highly variable and lack resolution
and can only detect damage when it is visible.
Therefore, structurally deficient bridges can be left
undiscovered. A number of bridge collapses have
occurred due to a lack of information on structural
capacity and it is therefore clear that visual
inspection alone may not be adequate for
monitoring the condition of bridges. In countries
such as Japan, which is prone to natural disasters, it
is recommended that monitoring of engineering
WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS
DOI: 10.37394/23201.2022.21.25
Antonino Fotia, Raffaele Pucinotti,
Vincenzo Barrile
E-ISSN: 2224-266X
235
infrastructure such as bridges be conducted
continuously.
Monitoring and structural diagnostics are
particularly topical issues as techniques that can be
usefully adopted for safety assessment and more
generally for the proper management of existing
constructions. In fact, monitoring overlaps with
structural diagnostics activities with the aim of
checking the behavior of constructions over time,
with longer observation periods, even of several
years, aimed at controlling the evolution of certain
aspects of interest, such as, for example, the
presence of some structural damage or form of
degradation, or the existence and entity of cracking
frameworks. By verifying their possible evolution
over time, monitoring provides overall feedback
with which to ascertain the change in structural
characteristics, which may be indicative of a
significant increase in structural damage.
With reference to a case study, an experimental
and automated methodology capable of acquiring
geometric information and the state of degradation
of a bridge in Cardeto was presented.
This information, available on the visualization
platform, helps and enables the Authority to
determine the priority of maintenance interventions.
Acknowledgements:
We would like to thank MTC for their willingness
to experiment.
References:
[1] P. Chupanit and C. Phromsorn, The importance
of bridge health monitoring, International
Science Index, vol. 6, pp. 135–138, 2012.View
at: Google Scholar
[2] Y. Fujino and D. M. Siringoringo, “Bridge
monitoring in Japan: the needs and strategies,”
Structure and Infrastructure Engineering, vol.
7, no. 7-8, pp. 597–611, 2011.
[3] S. L. Davis and D. Goldberg, The Fix We're In
For: The State of Our Nation's Bridges 2013,
Transportation for America, Washington, DC,
USA, 2013.
[4] A. Žnidarič, V. Pakrashi, E. O'Brien, and A.
O'Connor, A review of road structure data in
six European countries, Proceedings of the
ICE: Urban Design and Planning, vol. 164, no.
4, pp. 225–232, 2011.
[5] R., Pucinotti, G., Fiordaliso, Multi-span steel–
concrete bridges with anti-seismic devices: A
case study, Frontiers in Built Environment,
2019.
[6] A. Bonelli, O.S. Bursi, R. Ceravolo, S. Santini,
N. Tondini, A. Zasso, Dynamic Identification
and Structural Health Monitoring of a Twin
Deck Curved Cable-Stayed Footbridge: The
“Ponte del Mare” of Pescara in Italy, Fifth
European Workshop on Structural Health
Monitoring, 2010.
[7] X. Q. Zhu and S. S. Law, Wavelet-based crack
identification of bridge beam from operational
deflection time history, International Journal
of Solids and Structures, vol. 43, no. 7-8, pp.
2299–2317, 2006.
[8] A. K. Pandey, M. Biswas, and M. M. Samman,
Damage detection from changes in curvature
mode shapes, Journal of Sound and Vibration,
vol. 145, no. 2, pp. 321–332, 1991.
[9] G. Lederman, Z. Wang, J. Bielak et al.,
Damage quantification and localization
algorithms for indirect SHM of bridges, in
Bridge Maintenance, Safety, Management and
Life Extension, chapter 83, pp. 640–647, CRC
Press, New York, NY, USA, 2014.
[10] A., Fotia, M.R., Alvaro, F., Oliveto, R.,
Pucinotti, Safety Management of Existing
Bridges: A Case Study . Lecture Notes in
Networks and Systems, 2022, 482 LNNS, pp.
2268–2277.
[11] T.-C. Huynh, J.-H. Park, H.-J. Jung, and J.-T.
Kim, Quasi-autonomous bolt-loosening
detection method using vision-based deep
learning and image processing, Automation in
Construction, vol. 105, article 102844, 2019.
[12] X. F. Zhao, Y. Zhang, and N. N. Wang, Bolt
loosening angle detection technology using
deep learning, Structural Control & Health
Monitoring, vol. 26, no. 1, article e2292, 2019.
[13] C. Y. Wang, N. Wang, S. C. Ho, X. M. Chen,
and G. B. Song, Design of a new vision-based
method for the bolts looseness detection in
flange connections, IEEE Transactions on
Industrial Electronics, vol. 67, no. 2, pp. 1366–
1375, 2020.
[14] S. H. Chen, F. Cerda, P. Rizzo, J. Bielak, J. H.
Garrett, and J. Kovacevic, Semi-supervised
multiresolution classification using adaptive
graph filtering with application to indirect
bridge structural health monitoring, IEEE
Transactions on Signal Processing, vol. 62, pp.
2879–2893, 2014.
[15] Barrile, V., Fotia, A., Leonardi, G., Pucinotti,
R. Geomatics and soft computing techniques
for infrastructural monitoring. Sustainability
(Switzerland), 2020, 12(4), 1606
[16] Barrile, V., Bilotta, G., Fotia, A., Bernardo, E.
Road Extraction for Emergencies from Satellite
WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS
DOI: 10.37394/23201.2022.21.25
Antonino Fotia, Raffaele Pucinotti,
Vincenzo Barrile
E-ISSN: 2224-266X
236
Imagery. Lecture Notes in Computer Science
(including subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in
Bioinformatics), 2020, 12252 LNCS, pp. 767–
781
[17] Fotia, A., Barrile, V. Geomatics and Soft
Computing Techniques for Road Infrastructure
Monitoring. Communications in Computer and
Information Science, 2022, 1507 CCIS, pp.
313–324.
[18] Barrile, V., Fotia, A., Leonardi, G., Pucinotti,
R. Geomatics and Soft Computing Techniques
for Infrastructural Monitoring. Sustainability
2020, 12, 1606.
https://doi.org/10.3390/su12041606
[19] Barrile, V., Fotia, A., Bernardo, E., Bilotta, G.,
Modafferi, A. Road Infrastructure Monitoring:
An Experimental Geomatic Integrated System.
Lecture Notes in Computer Science (including
subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in
Bioinformatics), 2020, 12252 LNCS, pp. 634–
648
[20] Barrile, V., Bernardo, E., Candela, G., Bilotta,
G., Modafferi, A., Fotia, A. Road infrastructure
heritage: From scan to infrabim. WSEAS
Transactions on Environment and
Development, 2020, 16, pp. 633–642
[21] Barrile, V., Bernardo, E., Fotia, A., Candela,
G., Bilotta, G. Road safety: Road degradation
survey through images by UAV. WSEAS
Transactions on Environment and
Development, 2020, 16, pp. 649–659, 67
[22] Angiulli G.;Barrile V.;Cacciola M. SAR
imagery classification using Multi-class
Support Vector Machines; Journal of
Electromagnetic Waves and Applications.
Volume 19, Issue 14, 2005, Pages 1865-1872
[23] Barrile, V; Meduri G; Bilotta G. Laser scanner
surveying techniques aiming to the study and
the spreading of recent architectural structures.
Proceedings of the 2nd WSEAS International
Conference on Engineering Mechanics,
Structures and Engineering Geology, EMESEG
'09 2009, Pages 25-28
WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS
DOI: 10.37394/23201.2022.21.25
Antonino Fotia, Raffaele Pucinotti,
Vincenzo Barrile
E-ISSN: 2224-266X
237
Volume 21, 2022
Contribution of Individual Authors to the
Creation of a Scientific Article (Ghostwriting
Policy)
The authors equally contributed in the present
research, at all stages from the formulation of the
problem to the final findings and solution.
Sources of Funding for Research Presented in a
Scientific Article or Scientific Article Itself
No funding was received for conducting this study.
Conflict of Interest
The authors have no conflicts of interest to declare
that are relevant to the content of this article.
Creative Commons Attribution License 4.0
(Attribution 4.0 International, CC BY 4.0)
This article is published under the terms of the
Creative Commons Attribution License 4.0
https://creativecommons.org/licenses/by/4.0/deed.en
_US