An Anti-Noise Gearbox Fault Diagnosis Method based on Multi-Scale

Transformer Convolution and Transfer Learning

JINLIANG WU, XIAOYANG ZHENG, XINGLONG PEI

School of Artificial Intelligence,

Chongqing University of Technology,

Pufu Avenue, Longxing Town, Yubei District, Chongqing,

CHINA

proposed in this paper. Concretely, a multi-scale coarse-grained procedure is incorporated to capture different

and complementary features from multiple scales and filter random noises to some extent. Meanwhile, the

Transformer composed of an attention mechanism is utilized to extract high-level and effective features and the

transfer learning strategy is applied to overcome the limitation of insufficient fault samples for model training.

Finally, the experiments are conducted to verify the effectiveness of the proposed method. The results show

that the proposed method has higher accuracy and robustness under noisy environments compared with

previous methods.

Key-Words: - Gearbox, Fault diagnosis, Transformer, Convolution Network, Transfer Learning.

safe operation of mechanical systems in modern

society. As an important and common component of

mechanical equipment, the gearbox has been

extensively applied in various fields such as

transportation, agriculture production, space flight,

aviation, and so on, [1]. Due to the influence of

various working environmental conditions and load

intensity, the gearbox is vulnerable to various faults

which could lead to accidents and loss, [2].

Consequently, it is of critical significance to

traditional fault diagnosis methods, features are

usually extracted manually from original vibration

signals and then fed into machine learning models to

obtain fault diagnosis results. For example,

empirical mode decomposition and support vector

machine are combined to diagnose the faults of gear

reducer in [3]. To realize gearbox fault diagnosis,

deep Boltzmann machines are developed for deep

representations of the statistical parameters of the

wavelet packet transform in [4]. To solve the

model parameters estimation in [6]. However,

owing to the limitations of shallow structure and

manual feature extraction, the traditional fault

diagnosis method is hard to learn complex nonlinear

relations and intrinsic fault features.

Deep learning methods can solve the problem of

low nonlinear performance of the shallow neural

network and have superior generalization ability.

Due to its unique advantages and potential in

automatic feature extraction and pattern recognition,

different degradation states, achieving higher

accuracy with lower computational time cost

compared with traditional diagnosis methods.

Zhang et al., [8], introduced a novel method based

on recurrent neural network (RNN) to exploit the

temporal information of time-series data and learn

inserts confidence and classification rules into the

deep network structure to enable the model to have

good pattern recognition performance and can

adaptively determine the network structure and

obtain a good understanding of the features learned

by the deep network.

The method based on deep learning mentioned

above has achieved certain success in some

applications, but there are still certain limitations.

DBN to exhibit a satisfactory performance due to its

structural characteristics, [10]. Different from the

structure of traditional CNN and RNN, Transformer

has a new special network architecture solely based

on attention mechanisms and abandons recurrence

and convolutions structure entirely, [12]. To be

specific, all the related operations in the

Transformer network are order-independent and

parallelizable. With the rapid development of the

Transformer network, it has been widely applied

of the gearbox, the measured vibration signals

collected from sensors installed on the house usually

contain multiple intrinsic oscillatory modes, [18].

Consequently, the vibrations signals contain

complex patterns at multiple time scales and

inherent multi-scale characteristics, [19], [20]. Due

to the limitation of the inherent structure, it’s hard

for traditional CNN or RNN to capture multi-scale

features from vibration signals.

Furthermore, there is usually an assumption that the

distribution when training a deep learning network

model for fault diagnosis, [21]. However, due to the

complex working environment and variable load

conditions in real industrial applications, it’s

probably hard to obtain adequate fault data for

training network models. Although Transformer has

strategy, the network model can be pre-trained in

the source domain dataset which provides a massive

amount of prior knowledge for training, and then the

model can be fine-tuned in target domain dataset

whose data amount is small and effective fault data

is insufficient. Therefore, this paper proposed a

multi-scale Transformer convolution network

(MTCN) based on transfer learning for gearbox

fault diagnosis. The main contributions of this paper

are summarized as follows:

3) The use of transfer learning strategy enables

the proposed model to be trained on a target dataset

with limited effective fault samples.

The rest of this paper is organized as follows. The

related method theories are introduced in Section 2.

The details of the proposed MTCN architecture is

elaborated in Section 3. The results of the

comparative experiment to evaluate the proposed

model against some other common methods is

analyzed in Section 4. The conclusion of this paper

2 Related Theories

2.1 Scaled Dot-Product Attention

As a core module of the Transformer encoder,

scaled dot-product attention is used to map a query

vector and a set of key-value vector pairs to an

output vector, [12]. To be specific, the input of

scaled dot-product attention is composed of the

query and keys of dimension

and values of the

pooling layer. For the output from each scale, the

final outputs are concatenated together to be fed into

the classification layer to finish fault diagnosis.

Lastly, it is worth mentioning that Fig. 3 illustrates

the coarse-grained procedure for s = 2, 3

of effective fault samples for training are not

available in certain cases, the proposed method

MTCN is trained by transfer learning. Concretely,

the MTCN is pre-trained with randomly initialized

parameters on the source domain dataset which

for gearbox fault diagnosis and the general

procedure can be concluded as the following six

steps:

target domain datasets are collected from

different experimental facilities.

dataset are obtained by dividing the source domain

dataset and target domain datasets and each

sample is processed through FFT.

gearbox fault diagnosis

fault types and one normal type.

the CWRU-bearing dataset is set as the source

domain dataset. The amount of each fault type in the

CWRU bearing dataset is 2400 and 1600 samples of

it are randomly selected to use as a training set and

the rest samples are adopted as a testing set. The

SEU gearbox dataset is set as the target domain

In the pre-training stage, the Adam optimizer

with a learning rate of 1e-4 is adopted to train the

proposed model in the source domain dataset.

Moreover, the batch size is set as 600 and the

training epoch is set as 200. In the fine-tuning stage,

the classifier of the proposed model is modified to

adapt to the target task according to the number of

fault type in the target domain dataset. Similarly, the

Adam optimizer is used to fine-tune the parameters

of the proposed model and the learning rate is set as

RTX 3080 Ti GPU 12G, Pytorch 1.10.0 and Python

3.6.

target domain dataset of the transfer experiment. To

be specific, the training diagnosis accuracy of the

ten trials all reach 100% and the smallest diagnosis

accuracy of the testing is still up to 99.80%. It can

be clearly seen that the proposed model obtains

outstandingly high accuracy both in training and

Moreover, the confusion matrix with the

classification accuracy and misclassification error of

each fault type from the trial with the smallest

testing accuracy is presented in Fig.8. Its horizontal

axis refers to the predicted label and the ordinate

axis represents the actual label of the testing sample.

The result shows that the classification accuracy of

each fault type is up to 100% and there is no

misclassification error, which demonstrates that the

the t-distributed stochastic neighbor embedding (t-

SNE) which can effectively project the dimension of

high-level features into lower dimensions and

realize feature visualization. Fig. 9 presents the

feature visualization results of the transformer

encoder layer and convolution layer. There are nine

and the convolution layer enhances the effect, which

confirms that the proposed model has an excellent

ability to extract useful features and can distinguish

different fault types effectively.

On the other hand, in order to illustrate the

influence of different parameter settings on the

diagnosis accuracy of the proposed method, nine

groups of different experiments are carried out on

the proposed method. As shown in Table 2, the

training accuracy is relatively high under different

parameter settings, but when the training batch size,

epoch, and learning rate are relatively small, the test

accuracy will be reduced due to its influence.

Appropriately increasing the batch size and epoch of

but the final test accuracy has not been improved.

of the proposed model under a noise environment,

eight groups of control experiments are carried out.

(10)

where

and

experiment from the first group to the seventh

group, and the last group without adding noise to

represent the fault diagnosis under normal

conditions.

In the comparative experiments, four different

excellent fault diagnosis methods are adopted to

compare with the proposed method. The details of

these four methods can be summarized as: (1)

WDCNN, [24],: Raw vibration signals are used as

input and the wide kernels are applied in the first

different scales in a parallel way by combining

multi-scale learning with CNN. (4) TCN, [10]:

Transformer encoder with two layers is combined

with three same CNN blocks to realize fault

diagnosis.

For each method, it is executed 10 times, and

the average value of the 10 trials is taken as the final

result. As shown in Table 3 and Fig. 10, the

proposed model and the comparative model all

perform outstanding results reaching to 99% when

dB, but it still performs poorly when SNR is - 6 dB.

The proposed method and MSCNN all use multi-

scale learning, but the accuracy of the proposed

model doesn’t change significantly with the

decrease of SNR. Compared with the state-of-art

fault diagnosis model TCN, the proposed method

has fewer layers and blocks and it obtains superior

performance of 92.98% accuracy in extremely noisy

conditions of -6 dB which is higher than TCN. As a

result, the comparison experiment demonstrates that

5 Conclusion

To realize the high accuracy of gearbox fault

diagnosis and to deal with the limitation of the rarity

of fault samples in certain cases, a novel multi-scale

Transformer convolution network based on a

transfer learning strategy named MTCN is proposed

creatively in this paper. The coarse-grained

procedure incorporated in the proposed model can

not only enable the MTCN to learn complementary

multi-scale features but also can filter high-

frequency perturbations and random noises to some

extent. The proposed MTCN can extract rich fault

features and perform accurate pattern recognition

through the superior learning capability of the

Transformer and enhance the efficiency of training

this research is to expand the capability of the

proposed MTCN on unbalanced datasets.

References:

[1] Y. G. Lei, J. Lin, M. J. Zuo, Z. J. He,

Condition monitoring and fault diagnosis of

planetary gearboxes: A review, Measurement,

Vol. 48, 2014, pp. 292-305.

[2] X. H. Liang, M. J. Zuo, Z. P. Feng, Dynamic

Measurement, Vol. 45, 2012, pp. 30-40.

[4] C. Li, R.V. Sanchez, G. Zurita, M. Cerrada,

D. Cabrera, R.E. Vásquez, Gearbox fault

diagnosis based on deep random forest fusion

of acoustic and vibratory signals, Mechanical

Systems & Signal Processing. Vol. 76, 2016,

283-293.

[5] F. F. Chen, B. P. Tang, R. X. Chen, A novel

fault diagnosis model for gearbox based on

wavelet support vector machine with immune

[7] G. Q. Qiu, Y. K. Gu, Q. Cai, A deep

convolutional neural networks model for

intelligent fault diagnosis of a gearbox under

different operational conditions,

Measurement, Vol. 145, 2019, pp. 94-107.

[8] Y. Zhang, T. Zhou, X. Huang, L. Cao, and Q.

Zhou, Fault diagnosis of rotating machinery

based on recurrent neural networks,

Measurement, Vol. 171, 2021, pp. 108774.

[9] J. B. Yu, G. L. Liu, Knowledge extraction

[10] X. L. Pei, X. Y. Zheng, J. L. Wu, Rotating

[12] A. Vaswani, N. Shazeer, N. Parmar, J.

[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D.

[15] A. Prakash, K. Chitta, A. Geiger, Multi-

[16] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V.

[17] J. Devlin, M. W. Chang, K. Lee, K.

[19] H. Liu and M. Han, A fault diagnosis method