Dynamic Vision-Based Machinery Fault Diagnosis With Cross-Modality Feature Alignment

Xiang Li; Shupeng Yu; Yaguo Lei; Naipeng Li; Bin Yang

doi:10.1109/JAS.2024.124470

Volume 11 Issue 10

Oct. 2024

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2024 > 11(10): 2068-2081

X. Li, S. Yu, Y. Lei, N. Li, and B. Yang, “Dynamic vision-based machinery fault diagnosis with cross-modality feature alignment,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 10, pp. 2068–2081, Oct. 2024. doi: 10.1109/JAS.2024.124470

Citation:

X. Li, S. Yu, Y. Lei, N. Li, and B. Yang, “Dynamic vision-based machinery fault diagnosis with cross-modality feature alignment,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 10, pp. 2068–2081, Oct. 2024. doi: 10.1109/JAS.2024.124470

Citation:

PDF( 1497 KB)

Dynamic Vision-Based Machinery Fault Diagnosis With Cross-Modality Feature Alignment

doi: 10.1109/JAS.2024.124470

Funds: This work was supported by the National Science Fund for Distinguished Young Scholars of China (52025056), the China Postdoctoral Science Foundation (2023M732789), the China Postdoctoral Innovative Talents Support Program (BX20230290), and the Fundamental Research Funds for the Central Universities (xzy012022062)

More Information

Author Bio:
Xiang Li (Senior Member, IEEE) received the B.S. degree in engineering mechanics and the Ph.D. degree in mechanics from Tianjin University, in 2012 and 2017, respectively. He is currently an Associate Professor with the Key Laboratory of Education Ministry for Modern Design and Rotor-Bearing System, Xi’an Jiaotong University. Prior to joining Xi’an Jiaotong University, he was a Postdoctoral Fellow in the Center for Intelligent Maintenance Systems (IMS) at the University of Cincinnati, US. He was a Visiting Scholar with University of California at Merced, US in 2015. His research interests include industrial artificial intelligence (AI), machine vision, neuromorphic computing, and industrial big data. He serves as Associate Editor for IEEE Transactions on Neural Networks and Learning Systems, and Expert Systems With Applications. He is on the editorial boards of IEEE/CAA Journal of Automatica Sinica, Measurement, Measurement Science and Technology, etc. He is selected as the Highly Cited Researcher by Clarivate

Shupeng Yu received the B.E degree at the School of Mechanical Engineering from Xi’an Jiaotong University, in 2022. He is currently pursuing the Ph.D. degree in mechanical engineering at the Key Laboratory of Education Ministry for Modern Design and Rotor-Bearing System, Xi’an Jiaotong University. His research interests include machine vision, intelligent fault diagnosis, contactless vibration measurement, and event-based camera

Yaguo Lei (Senior Member, IEEE) received the B.S. and Ph.D. degrees in mechanical engineering from Xi’an Jiaotong University in 2002 and 2007, respectively. He is currently a Full Professor of mechanical engineering at Xi’an Jiaotong University. Prior to joining Xi’an Jiaotong University in 2010, he was a Postdoctoral Research Fellow with the University of Alberta, Canada. He was also an Alexander von Humboldt Fellow with the University of DuisburgEssen, Germany. His research interests include machinery condition monitoring and intelligent maintenance, dynamic modeling, signal processing, intelligent fault diagnosis, and remaining useful life prediction. He has pioneered many fault diagnosis and prognosis methods, and applied them to industrial areas, such as wind turbines, trains and industrial robots, etc. Dr. Lei is a Fellow of ASME, IET and ISEAM, a Senior Member of IEEE, and a Senior Member of Chinese Mechanical Engineering Society, Operations Research Society of China and Chinese Association of Automation, respectively. He is currently an Associate Editor or Editorial Board Member of more than ten journals, including IEEE Transactions on Industrial Electronics and Mechanical Systems and Signal Processing, etc

Naipeng Li (Member, IEEE) received the B.S. degree in mechanical engineering from Shandong Agricultural University in 2012, and the Ph.D. degree in mechanical engineering from Xi’an Jiaotong University in 2019. He is currently an Associate Professor of mechanical engineering with Xi’an Jiaotong University. He was also a Visiting Scholar with Georgia Institute of Technology, USA. His research interests include machinery condition monitoring, intelligent fault diagnostics, and remaining useful life prediction of machinery

Bin Yang (Member, IEEE) received the B.S. and M.S. degrees in mechatronics engineering from Chang’an University in 2014 and 2017, respectively, and the Ph.D. degree in mechanical engineering from Xi’an Jiaotong University in 2022. He is currently an Assistant Professor with the Key Laboratory of Education Ministry for Modern Design and Rotor-Bearing System, Xi’an Jiaotong University. He was also a visiting Ph.D. student of the Centre of Maintenance Optimization and Reliability Engineering (CMORE) at the University of Toronto, Canada. His research interests include intelligent fault diagnosis and condition monitoring of machines
Corresponding author: Bin Yang, e-mail: binyang@xjtu.edu.cn
Received Date: 2024-02-21
Revised Date: 2024-03-18
Accepted Date: 2024-04-05

Available Online: 2024-07-30

Abstract

Abstract

Intelligent machinery fault diagnosis methods have been popularly and successfully developed in the past decades, and the vibration acceleration data collected by contact accelerometers have been widely investigated. In many industrial scenarios, contactless sensors are more preferred. The event camera is an emerging bio-inspired technology for vision sensing, which asynchronously records per-pixel brightness change polarity with high temporal resolution and low latency. It offers a promising tool for contactless machine vibration sensing and fault diagnosis. However, the dynamic vision-based methods suffer from variations of practical factors such as camera position, machine operating condition, etc. Furthermore, as a new sensing technology, the labeled dynamic vision data are limited, which generally cannot cover a wide range of machine fault modes. Aiming at these challenges, a novel dynamic vision-based machinery fault diagnosis method is proposed in this paper. It is motivated to explore the abundant vibration acceleration data for enhancing the dynamic vision-based model performance. A cross-modality feature alignment method is thus proposed with deep adversarial neural networks to achieve fault diagnosis knowledge transfer. An event erasing method is further proposed for improving model robustness against variations. The proposed method can effectively identify unseen fault mode with dynamic vision data. Experiments on two rotating machine monitoring datasets are carried out for validations, and the results suggest the proposed method is promising for generalized contactless machinery fault diagnosis.
- Condition monitoring,
- domain generalization,
- event-based camera,
- fault diagnosis,
- machine vision

FullText(HTML)

References(40)

References

[1]	Y. Lei, N. Li, and X. Li, Big Data-Driven Intelligent Fault Diagnosis and Prognosis for Mechanical Systems. Springer, 2022.
[2]	J. Huang, Z. Li, and Z. Zhou, “A simple framework to generalized zero-shot learning for fault diagnosis of industrial processes,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 6, pp. 1504–1506, 2023. doi: 10.1109/JAS.2023.123426
[3]	X. Gu, Y. Shang, Y. Kang, J. Li, Z. Mao, and C. Zhang, “An early minor-fault diagnosis method for lithiumion battery packs based on unsupervised learning,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 3, pp. 810–812, 2023. doi: 10.1109/JAS.2023.123099
[4]	W. Zhang and X. Li, “Data privacy preserving federated transfer learning in machinery fault diagnostics using prior distributions,” Structural Health Monitoring, vol. 21, no. 4, pp. 1329–1344, 2021.
[5]	Z. Zhang, C. Guan, H. Chen, X. Yang, W. Gong, and A. Yang, “Adaptive privacy-preserving federated learning for fault diagnosis in internet of ships,” IEEE Internet of Things J., vol. 9, no. 9, pp. 6844–6854, 2022. doi: 10.1109/JIOT.2021.3115817
[6]	B. Yang, Y. Lei, X. Li, and N. Li, “Targeted transfer learning through distribution barycenter medium for intelligent fault diagnosis of machines with data decentralization,” Expert Systems With Applic ations, vol. 244, p. 122997, 2024. doi: 10.1016/j.eswa.2023.122997
[7]	M. Xia, H. Shao, D. Williams, S. Lu, L. Shu, and C. W. de Silva, “Intelligent fault diagnosis of machinery using digital twin-assisted deep transfer learning,” Reliability Engineering and System Safety, vol. 215, p. 107938, 2021.
[8]	A. Glover, A. Dinale, L. D. S. Rosa, S. Bamford, and C. Bartolozzi, “luvHarris: A practical corner detector for event-cameras,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 10087–10098, 2022. doi: 10.1109/TPAMI.2021.3135635
[9]	L. Wang, T. K. Kim, and K. J. Yoon, “Joint framework for single image reconstruction and super-resolution with an event camera,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7657–7673, 2022. doi: 10.1109/TPAMI.2021.3113352
[10]	L. Yu, X. Zhang, W. Liao, W. Yang, and G. S. Xia, “Learning to see through with events,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8660–8678, 2023.
[11]	Y. Li, Y. Chen, K. Zhu, C. Bai, and J. Zhang, “An effective federated learning verification strategy and its applications for fault diagnosis in industrial IoT systems,” IEEE Internet of Things J., vol. 9, no. 18, pp. 16835–16849, 2022. doi: 10.1109/JIOT.2022.3153343
[12]	X. Li, W. Zhang, X. Li, and H. Hao, “Partial domain adaptation in remaining useful life prediction with incomplete target data,” IEEE/ASME Trans. Mechatronics, vol. 29, no. 3, pp. 1903–1913, 2024.
[13]	B. Yang, Y. Lei, X. Li, N. Li, and A. K. Nandi, “Label recovery and trajectory designable network for transfer fault diagnosis of machines with incorrect annotation,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 4, pp. 932–945, 2024. doi: 10.1109/JAS.2024.124386
[14]	X. Li, C. Zhang, X. Li, and W. Zhang, “Federated transfer learning in fault diagnosis under data privacy with target self-adaptation,” J. Manufacturing Systems, vol. 68, pp. 523–535, 2023. doi: 10.1016/j.jmsy.2023.05.006
[15]	W. Zhang, M. Xu, H. Yang, X. Wang, S. Zheng, and X. Li, “Data-driven deep learning approach for thrust prediction of solid rocket motors,” Measurement, vol. 225, p. 114051, 2024. doi: 10.1016/j.measurement.2023.114051
[16]	T. Zhou, T. Han, and E. L. Droguett, “Towards trustworthy machine fault diagnosis: A probabilistic bayesian deep learning framework,” Reliability Engineering and System Safety, vol. 224, p. 108525, 2022.
[17]	J. Liu, C. Zhang, and X. Jiang, “Imbalanced fault diagnosis of rolling bearing using improved MsR-GAN and feature enhancement-driven CapsNet,” Mechanical Systems and Signal Processing, vol. 168, p. 108664, 2022. doi: 10.1016/j.ymssp.2021.108664
[18]	B. Yang, Y. Lei, X. Li, and C. Roberts, “Deep targeted transfer learning along designable adaptation trajectory for fault diagnosis across different machines,” IEEE Trans. Industrial Electronics, vol. 70, no. 9, pp. 9463–9473, 2023. doi: 10.1109/TIE.2022.3212415
[19]	T. Han, W. Xie, and Z. Pei, “Semi-supervised adversarial discriminative learning approach for intelligent fault diagnosis of wind turbine,” Information Sciences, vol. 648, p. 119496, 2023. doi: 10.1016/j.ins.2023.119496
[20]	T. Han and Y. Li, “Out-of-distribution detection-assisted trustworthy machinery fault diagnosis approach with uncertainty-aware deep ensembles,” Reliability Engineering System Safety, vol. 226, p. 108648, 2022. doi: 10.1016/j.ress.2022.108648
[21]	Y. Chen, M. Rao, K. Feng, and M. J. Zuo, “Physics-informed LSTM hyperparameters selection for gearbox fault detection,” Mechanical Systems and Signal Processing, vol. 171, p. 108907, 2022. doi: 10.1016/j.ymssp.2022.108907
[22]	Y. Chen, M. Rao, K. Feng, and G. Niu, “Modified varying index coefficient autoregression model for representation of the nonstationary vibration from a planetary gearbox,” IEEE Trans. Instrumentation and Measurement, vol. 72, pp. 1–12, 2023.
[23]	X. Li, S. Yu, Y. Lei, N. Li, and B. Yang, “Intelligent machinery fault diagnosis with event-based camera,” IEEE Trans. Industrial Informatics, vol. 20, no. 1, pp. 380–389, 2024. doi: 10.1109/TII.2023.3262854
[24]	S. Guo and T. Delbruck, “Low cost and latency event camera background activity denoising,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 785–795, 2023. doi: 10.1109/TPAMI.2022.3152999
[25]	X. Chen, X. Li, S. Yu, Y. Lei, N. Li, and B. Yang, “Dynamic vision enabled contactless cross-domain machine fault diagnosis with neuromorphic computing,” IEEE/CAA J. Autom. Sinica, vol. 11, no. 3, pp. 788–790, 2024. doi: 10.1109/JAS.2023.124107
[26]	Y. Zhou, G. Gallego, X. Lu, S. Liu, and S. Shen, “Event-based motion segmentation with spatio-temporal graph cuts,” IEEE Trans. Neural Networks and Learning Systems, vol. 34, no. 8, pp. 4868–4880, 2023.
[27]	P. R. G. Cadena, Y. Qian, C. Wang, and M. Yang, “SPADE-E2VID: Spatially-adaptive denormalization for event-based video reconstruction,” IEEE Trans. Image Processing, vol. 30, pp. 2488–2500, 2021. doi: 10.1109/TIP.2021.3052070
[28]	C. Ryan, B. O’Sullivan, A. Elrasad, A. Cahill, J. Lemley, P. Kielty, C. Posch, and E. Perot, “Real-time face and eye tracking and blink detection using event cameras,” Neural Networks, vol. 141, pp. 87–97, 2021. doi: 10.1016/j.neunet.2021.03.019
[29]	Z. Chen, J. Wu, J. Hou, L. Li, W. Dong, and G. Shi, “ECSNet: Spatio-temporal feature learning for event camera,” IEEE Trans. Circuits and Systems for Video Technology, vol. 33, no. 2, pp. 701–712, 2023.
[30]	A. Rahate, R. Walambe, S. Ramanna, and K. Kotecha, “Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions,” Information Fusion, vol. 81, pp. 203–239, 2022. doi: 10.1016/j.inffus.2021.12.003
[31]	B. Goyal, A. Dogra, D. C. Lepcha, D. Koundal, A. Alhudhaif, F. Alenezi, and S. A. Althubiti, “Multi-modality image fusion for medical assistive technology management based on hybrid domain filtering,” Expert Systems With Applications, vol. 209, p. 118283, 2022. doi: 10.1016/j.eswa.2022.118283
[32]	L. A. Passos, J. P. Papa, J. D. Ser, A. Hussain, and A. Adeel, “Multimodal audio-visual information fusion using canonical-correlated graph neural network for energy-efficient speech enhancement,” Information Fusion, vol. 90, pp. 1–11, 2023. doi: 10.1016/j.inffus.2022.09.006
[33]	S. Qiu, H. Zhao, N. Jiang, et al., “Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges,” Information Fusion, vol. 80, pp. 241–265, 2022. doi: 10.1016/j.inffus.2021.11.006
[34]	J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multimodality benchmark to fuse infrared and visible for object detection,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, pp. 5802–5811, 2022.
[35]	W. Li, R. Huang, J. Li, Y. Liao, Z. Chen, G. He, R. Yan, and K. Gryllias, “A perspective survey on deep transfer learning for fault diagnosis in industrial scenarios: Theories, applications and challenges,” Mechanical Systems and Signal Processing, vol. 167, p. 108487, 2022. doi: 10.1016/j.ymssp.2021.108487
[36]	G. Chakrapani and V. Sugumaran, “Transfer learning based fault diagnosis of automobile dry clutch system,” Engineering Applications of Artificial Intelligence, vol. 117, p. 105522, 2023. doi: 10.1016/j.engappai.2022.105522
[37]	W. Zhang, X. Li, H. Ma, Z. Luo, and X. Li, “Universal domain adaptation in fault diagnostics with hybrid weighted deep adversarial learning,” IEEE Trans. Industrial Informatics, vol. 17, no. 12, pp. 7957–7967, 2021.
[38]	S. Asutkar and S. Tallur, “Deep transfer learning strategy for efficient domain generalisation in machine fault diagnosis,” Scientific Reports, vol. 13, no. 1, p. 6607, 2023. doi: 10.1038/s41598-023-33887-5
[39]	S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “Convnext v2: Co-designing and scaling convnets with masked autoencoders,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, pp. 16133–16142, 2023.
[40]	W. Lu, B. Liang, Y. Cheng, D. Meng, J. Yang, and T. Zhang, “Deep model based domain adaptation for fault diagnosis,” IEEE Trans. Industrial Electronics, vol. 64, no. 3, pp. 2296–2305, 2017. doi: 10.1109/TIE.2016.2627020

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(13) / Tables(8)

Get Citation

PDF

XML

Article Metrics

Article views (149) PDF downloads(47)

Highlights

A contactless machine fault diagnosis method is proposed with dynamic vision
A cross-modality alignment method is proposed for vision and accelerometer data
An event erasing method is proposed to enhance model robustness
The proposed method can diagnose unseen machine fault with dynamic vision

Dynamic Vision-Based Machinery Fault Diagnosis With Cross-Modality Feature Alignment

doi: 10.1109/JAS.2024.124470

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content