Real-time Inspection System Based on Moire Pattern and YOLOv7 for
Coated High-reflective Injection Molding Product
OUNGSUB KIM1,2, YOHAN HAN1,2, JONGPIL JEONG1
1Department of Smart Factory Convergence, Sungkyunkwan University,
2066, Seobu-ro, Jangan-gu, Suwon-si, Gyeonggi-do, 16419,
REPUBLIC OF KOREA
2AI Machine Vision Smart Factory Lab, Dev,
296, Sandan-ro, Danwon-gu, Ansan-si, Gyeonggi-do, 15433,
REPUBLIC OF KOREA
Abstract: Recently, with the development of smart factories, innovation through automation is being carried out
in various fields of industry. In particular, because quality control requires a lot of man-hours, many studies are
being conducted to replace workers with machine vision. We proposed a real-time inspection system based on
YOLOv7 using moire patterns to automate quality inspection in the industry. In particular, the inspection system
was successfully applied in the actual industrial manufacturing environment by overcoming the limitations of
the applying inspection system to high-reflective products. Not only did we confirm the possibility of applying
YOLOv7 to industrial sites, but our proposed optical system can also be used for the inspection of other high-
reflective products.
Key-Words: Inspection System, Deep Learning, Object Detection, Machine Vision, Smart Factory, Moire Pattern
Received: May 9, 2022. Revised: October 23, 2022. Accepted: November 25, 2022. Published: December 31, 2022.
1 Introduction
Industry 4.0 is driving many innovations in
manufacturing. A smart factory is a very important
issue in Industry 4.0. Smart factory refers to a
method that enables more efficient business
operations by collecting and analyzing various types
of big data generated in the manufacturing
environment to improve productivity and quality.
Implementing a smart factory has now become an
essential element to strengthen a company’s
competitiveness and sustainability.
In particular, the quality control (QC) process
for inspecting defects in the manufacturing process
is one of the processes that requires a lot of
manhours. Companies are trying to implement a
real-time inspection system for automation in the
quality control process. However, many problems
need to be solved to inspect coated high-reflective
injection molded products. The three most difficult
problems in coated high-reflective injection molded
products are:
•
Various types of defects occur irregularly.
•
It is difficult to distinguish between real defects and
dust on the image
•
It is difficult to see various defect types in one
optical system.
In this paper, YOLOv7, an object detection
algorithm, was used to detect irregular defects. In
addition, we propose a system that solves the two
problems described above by utilizing the Moire
pattern and detects atypical defects in cylindrical
high-reflective products in real-time. Our main
contributions are summarized as follows:
•
First, we propose a method of detecting very small
defects using the Object Detection algorithm. A
real-time inspection system that satisfies the tact-
time and detection accuracy in the actual process
environment was implemented using YOLOv7.
•
Second, we propose an optical system design
optimized for cylindrical high-reflective products.
Real defects and dust are distinguished by using
Moire pattern. In addition, both Bright Field and
Dark Field are implemented to increase detection
accuracy for various types of defects.
The structure of this paper is as follows. Section
2 introduces research or background knowledge
related to the system proposed in this paper.
Section 3 explains the structure and method of a
real-time inspection system based on YOLOv7.
Section 4 describes the experimental environment
and the experimental results. Finally, Section 5
describes the results of the study and future
research.
WSEAS TRANSACTIONS on COMPUTER RESEARCH
DOI: 10.37394/232018.2022.10.16
Oungsub Kim, Yohan Han, Jongpil Jeong
E-ISSN: 2415-1521
120
Volume 10, 2022
2.1 Machine Vision
The introduction of automation in quality control
(QC) in the manufacturing industry has
revolutionized manufacturing as repetitive tasks can
be replaced by machines. Mechanisms that had to
be performed by humans in the past caused various
human errors and inefficiency. These tasks have
traditionally been performed by human operators,
but these challenges make machine vision systems
even more attractive. The main components of a
typical vision system are described in references,
[1], [2], [3], [4]. Fig. 1 shows a simple block
diagram for a machine vision system, [5].
Fig. 1: A Simple Block Diagram for a Vision
System Operation
2.2 Object Detection
For a computer to analyze images perfectly, it must
not only focus on classifying different images but
also try to accurately estimate the concept and
location of objects contained in each image, [6].
These tasks are called object detection and have
been studied in various fields such as image
classification, [7], [8], human behavior analysis,
[9], face recognition, [10], and autonomous
driving, [11], [12]. However, it is difficult to
perform perfect viewpoint object detection due to
large variations in the lighting conditions, poses,
occlusions, and lighting conditions. Therefore, in
recent years, there has been a lot of interest in this
field. Studies related to object detection can be
largely divided into three categories: information
area selection, feature extraction, and classification,
and related studies are described in references, [13],
[14], [15], [16]. The development roadmap of the
framework of general object detection methods can
be seen in Fig. 2
2.3 YOLOv7
The YOLOv7 model was proposed by Wang et al. In
2022, we realized faster rates and higher in the
COCO dataset. YOLOv7 includes several trainable
bags of freebies so that real-time detectors can
significantly improve detection accuracy without
increasing inference costs. We also study how the
module reparameterization strategy can effectively
replace the original module and how the dynamic
label allocation strategy can allocate different output
layers. The speed and accuracy exceed other
detectors in the range of 5 160 FPS. It also supports
both mobile GPUs and GPU devices from edge to
cloud. In the future, this model can be deployed in
practical working applications and utilized in real-
time industrial inspection systems, [17].
Fig. 2: Two Types of Frameworks: Region
Proposal Based and Regression/Classification
Based
2 Related Work
WSEAS TRANSACTIONS on COMPUTER RESEARCH
DOI: 10.37394/232018.2022.10.16
Oungsub Kim, Yohan Han, Jongpil Jeong
E-ISSN: 2415-1521
121
Volume 10, 2022
YOLOv7 is evaluated to have the best performance
among the recently announced object detection
algorithms. In Fig. 3, you can see the performance
evaluation table of Object Detections, [18].
Fig. 3: Comparison with Other Real-time Object
Detectors
2.4 Moire Pattern
Oster (1964) and Oster (1968) proposed the Moire
pattern as a tool for the optical examination of
surface structures in materials science (e.g.,
mechanical distortion and experimental strain
analysis during thermal expansion), [19]. Moire
pattern refers to a pattern in which a certain pattern
repeats regularly, as shown in Fig. 4. The Moire
pattern can make the small curves of a three-
dimensional shape stand out more.
Fig. 4: Example of Moire Pattern
3 Real-time Inspection System based
on Moire Pattern and YOLOv7
3.1 Target Product
In this paper, a study was conducted to detect
defects on the surface of a cosmetic case, which is a
coated high-reflection injection molding product,
using deep learning. Fig 5. shows the description
of the target product and defect types
Fig. 5: Target Product / Defect Image Example
The size of the product is a height of 32.52mm, a
diameter of 22.25mm, and has a cylindrical shape.
The defect occurs on the top and side, and the types
that occur are as follows: Scratch, Bad point, Weld,
Oil, Finger Printing, and Pollution.
3.2 Optical Setting
The target product is a product coated on the
surface of an injection molded product, and it is
difficult to accurately acquire a defect image due to
diffuse reflection due to the high-reflective surface.
We constructed an optimal optical system that can
detect defects in all aspects of the product by taking
images of the product using 10 cameras, Dark Field
and Bright Field, and Moire Pattern. Working
distance (WD) is 180mm, and Field of view (FOV)
is 50mmx50mm. The concept is described in Fig 6.
Among the types of defects that occur in the
product, Bad Point is difficult to distinguish from
dust on the image. We used the Moire pattern to
solve the problem. In the left image, dust and bad
points are treated as NG. On the other hand, in the
image on the right, Bad Point is distorted on the
stripe and is clearly distinguished from Dust. Fig. 7
shows the difference between when Moire pattern is
applied and when it is not.
Dark Field and Bright Field (with Moire pattern)
were cross-applied to solve the problem that some
defects were not expressed on the image when
Moire
WSEAS TRANSACTIONS on COMPUTER RESEARCH
DOI: 10.37394/232018.2022.10.16
Oungsub Kim, Yohan Han, Jongpil Jeong
E-ISSN: 2415-1521
122
Volume 10, 2022
Fig. 6: Equipment / Optical Setting
Fig. 7: Distinction between Applying the Moire
Pattern and Not
Pattern was applied. In Fig. 8, it can be identified
that the Finger Printing defect is covered in Dark
Field.
Fig. 8: Inspecting Pollution Defect on Dark Field
3.3 System Architecture
Existing anomaly detection techniques have various
classes of defect types and were difficult to apply to
manufacturing inspection systems that need to
detect irregular defects. We propose a system that
uses an object detection algorithm to detect defects
by dividing defect types into each class and learning,
and if no objects are detected, it is judged normal.
This content is schematized in Fig. 9. We trained
on 23,852 bad images
Fig. 9: Concept Diagram of Real-time Inspection
System Based on YOLOv7
In order to take and analyze 10 images within
1.2sec, which is the Tact-time in the actual
production line, YOLOv7, which is evaluated as the
best in terms of current speed and prediction
accuracy, was adopted. Fig. 10 shows the network
structure diagram of YOLOv7 where captured
images are processed, [20].
Fig. 10: Network Architecture Diagram of the
YOLOv7
3.4 Application
The experiment was conducted in an actual
production line of a cosmetic container
manufacturer in Gangwon-do, Korea. Equipment
was manufactured to satisfy the existing
production capacity of
1.2sec/ea. The input part sorts the product and
regularly puts it into the inspection system. The
optical setting described in section 3.2 is
implemented in the optical setting part. The
OK/NG output part pushes NG through the cylinder.
The monitoring part shows images and statistics
being inspected in real-time. You can see an actual
case in Fig. 11. A brief system architecture can be
found in Fig. 12.
WSEAS TRANSACTIONS on COMPUTER RESEARCH
DOI: 10.37394/232018.2022.10.16
Oungsub Kim, Yohan Han, Jongpil Jeong
E-ISSN: 2415-1521
123
Volume 10, 2022
Fig. 11: Application in an Actual Production Line
and Image for Each Part
Fig. 12: Simple Hardware Architecture of Real-
time Inspection System Based on YOLOv7
4 Experiment Results
4.1 Experience Environment
Table. 1 describes the specifications of the hardware
used in this system.
In this paper, we aim to secure inspection speed
and detection capability applicable to the actual
manufacturing environment. Therefore, the
evaluation index of the experiment evaluated the
values of miss-detection and over-detection when
the actual product was inspected by the inspection
system.
4.2Result
In this paper, we proposed an inspection system
using the Moire pattern and YOLOv7. Therefore, we
evaluated how effective Moire pattern and YOLOv7
were, and each test targeted 1000 products (OK:
700, NG: 300). Table. 2 shows the comparison
results when Moire pattern is applied and when it is
not applied, and when YOLOv4 and YOLOv7 are
applied.
When Moire Pattern was applied, miss-detection
improved performance by about 5 times and over-
detection by more than 2 times. Compared to
YOLOv4, it was confirmed that YOLOv7 improved
by about 30%.
Table 1. Specification of the hardware
Industrial
Computer
Analysis
Processor
RAM
GPU
Control
Processor
RAM
GPU
Camera
Sensor Format
Shutter
Sensor Type
Mono/Color
Resolution
Lens
Focal length
Lens mount
Table 2. Experiment result
Apply Moire
Pattern
O O
O O
O O
Applied
YOLOv4
Applied
YOLOv7
Total
Amount
1,000
1,000
1,000
1,000
Defect
Inspection
408
356
343
322
Miss
Detection
102
87
21
17
Over
Detection
210
143
107
73
Miss Detection
Rate
10.2%
8.7%
2.1%
1.7%
Over Detection
Rate
21%
14.3%
10.7%
7.3%
WSEAS TRANSACTIONS on COMPUTER RESEARCH
DOI: 10.37394/232018.2022.10.16
Oungsub Kim, Yohan Han, Jongpil Jeong
E-ISSN: 2415-1521
124
Volume 10, 2022
5 Conclusion
We proposed a real-time inspection system using
YOLOv7 and Moire pattern for coated high-
reflective injection molding products. As shown in
Table 2, the Moire pattern was able to obtain very
effective results for high-reflective products, and
the applicability of the inspection system using
YOLOv7 to industrial applications was also
confirmed.
However, it is true that inspection performance
is still insufficient for application in industrial
applications. In addition, it is difficult to properly
secure defective samples in an actual industrial
environment. In order to solve these problems,
future research will propose a method to solve the
class imbalance problem.
$FNQRZOHGJPHQW
This work was supported by the Technology
Development Program (Project Number:
1415178709, 20015644) funded by the Ministry of
Trade, Industry and Energy(MOTIE, Korea)
Corresponding author: Professor Jongpil Jeong
References:
[1]
Awcock GJ, Thomas R,”Applied image
processing”, London: Mac Millan New Press
Ltd., 1995.
[2]
Pugh A, Robot sensors, UK: IFS Publication
Ltd, 1986.
[3]
Davies ER,”Machine vision theory. Algorithms,
practicalities”, UK: IFS Publication Ltd, 1986.
[4]
Bastuchech CM, ”Techniques for real time
generation of range images. In: Proceedings
on computer vision and pattern recognition”,
San Diego, p. 262-8, 1989.
[5]
H. Golnabi, A. Asadpour, ”Design and
application of industrial machine vision
systems”, Robotics and Computer-Integrated
Manufacturing, 23, pp. 630-637, 2007.
[6]
P. F. Felzenszwalb et al., “Object detection
with discriminatively trained part-based
models”, IEEE Trans. Pattern Anal. Mach.
Intell., vol. 32, no. 9, pp. 1627–1645, Sep.
2010.
[7]
Y. Jia et al., “Caffe: Convolutional architecture
for fast feature embedding”, in Proc. ACM MM,
pp. 675–678, 2014.
[8]
A. Krizhevsky et al., “ImageNet classification
with deep convolutional neural networks”, in
Proc. NIPS, pp. 1097–1105, 2012.Z. Cao et
al., “Realtime multi-person 2D pose estimation
using part affinity fields”, in Proc. CVPR, pp.
1302-1310, 2017.
[9]
Z. Yang and R. Nevatia, “A multi-scale cascade
fully convolutional network face detector”, in
Proc. ICPR, pp. 633–638, 2016.
[10]
C. Chen et al., “DeepDriving: Learning
affordance for direct perception in autonomous
driving”, in Proc. ICCV, pp. 2722–2730, 2015.
[11]
X. Chen et al., “Multi-view 3D object
detection network for autonomous driving”, in
Proc. CVPR, pp. 6526–6534, 2017.
[12]
R. Girshick et al., “Rich feature hierarchies for
accurate object detection and semantic
segmentation”, in Proc. CVPR, pp. 580–587,
2014.
[13]
R. Girshick, “Fast R-CNN,” in Proc. ICCV,
2015, pp. 1440–1448.
[14]
S. Ren et al., “Faster R-CNN: Towards real-time
object detection with region proposal networks”,
in Proc. NIPS, pp. 91–99, 2015.
[15]
J. Redmon et al., “You only look once: Unified,
real-time object detection”, in Proc. CVPR, pp.
779–788, 2016.
[16]
Jianfeng Zheng, Hang Wu, Han Zhang, Zhaoqi
Wang and Weiyue Xu, ”Insulator-Defect
Detection AlgorithmBased on Improved
YOLOv7”, Sensors 2022, 22, 8801.
[17]
Wang, C.Y., Bochkovskiy A. and Liao, H.Y.M.,
”YOLOv7: Trainable bag-of-freebies sets new
state-of-the-art for real-time object detectors”
arXiv, arXiv:2207.02696, 2022.
[18]
Nishijima Y, Oster G, ”Moire patterns: Their
application to refractive index and refractive
index gradient measurements”, Journal of the
Optical Society of America 54, pp. 1-5, 1964
[19]
Kailin Jiang, Tianyu Xie, Rui Yan, Xi Wen,
Danyang Li, Hongbo Jiang Ning Jiang, Ling
Feng, Xuliang Duan and JianjunWang, ”An
Attention Mechanism-Improved YOLOv7
Object Detection Algorithm for Hemp Duck
Count Estimation”, Agriculture, 12, 1659,
2022.
Creative Commons Attribution License 4.0
(Attribution 4.0 International, CC BY 4.0)
This article is published under the terms of the Creative
Commons Attribution License 4.0
https://creativecommons.org/licenses/by/4.0/deed.en_US
WSEAS TRANSACTIONS on COMPUTER RESEARCH
DOI: 10.37394/232018.2022.10.16
Oungsub Kim, Yohan Han, Jongpil Jeong
E-ISSN: 2415-1521
125
Volume 10, 2022