Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis

Qiyue Wang; Wenhua Jiao; Peng Wang; YuMing Zhang

doi:10.1109/JAS.2020.1003518

Volume 8 Issue 2

Feb. 2021

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2021 > 8(2): 334-343

Qiyue Wang, Wenhua Jiao, Peng Wang and YuMing Zhang, "Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 334-343, Feb. 2021. doi: 10.1109/JAS.2020.1003518

Citation:

Qiyue Wang, Wenhua Jiao, Peng Wang and YuMing Zhang, "Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 334-343, Feb. 2021. doi: 10.1109/JAS.2020.1003518

Citation:

PDF( 3131 KB)

Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis

doi: 10.1109/JAS.2020.1003518

More Information

Author Bio:
Qiyue Wang received the B.Eng. degree in welding engineering from Harbin Institute of Technology in 2016. Currently, he is pursuing the Ph.D. degree in electrical engineering at the University of Kentucky, USA. His current research interests include human-robot interaction (HRI), machine learning, and their applications in welding robots. He has been a manufacturing intern in Tesla, Inc., Fremont, CA, during June–December 2018. Mr. Wang has published one paper in IEEE Transactions on Automation Science and Engineering and one paper in IEEE Robotics and Automation Letters

Wenhua Jiao received the B.S. and Ph.D. degrees in electrical engineering from University of Kentucky, USA in 2015 and 2020, respectively. His current research interests include image processing and intelligent welding. He has been an intern in Tesla, Inc., Fremont, CA, from January to May in 2018

Peng Wang (M’17) is an Assistant Professor in the Department of Electrical and Computer Engineering and Department of Mechanical Engineering at the University of Kentucky, USA. He received the B.S. and M.S. degrees in information science from Beijing University of Chemical Engineering in 2010 and 2013, and the Ph.D. degree in mechanical and aerospace engineering from Case Western Reserve University in 2017, respectively. His research interests are in the areas of stochastic modeling, machine learning, and data fusion for machine performance prediction under uncertainty, manufacturing process modeling, and product quality assessment

YuMing Zhang (SM’97) is a Professor and the James R. Boyd Professor in electrical engineering at the University of Kentucky, USA. He received the B.S. (1982) and M.S. (1984) degrees in control major and Ph.D. (1990) degree in welding major all from Harbin Institute of Technology. His research interests include welding processes sensing and control, robotic/intelligent welding, and human-robot collaborative welding. He served American Welding Society (AWS) as Lead Principal Reviewer for the Welding Journal and Chair for the Technical Papers Committee. He is a co-author for over 200 peer-reviewed journal publications and has been granted 10 US patents. His recognitions include Fellow of the AWS, ASME, and SME, William Irrgang Memorial Award (2017) from AWS, Dean’s Award for Excellence in Research (2016) from College of Engineering at the University of Kentucky, and Poster Paper Prize and Application Paper Finalist (2002) from the 15th IFAC World Congress
Corresponding author: P. Wang is with the Institute for Sustainable Manufacturing and the Department of Electrical and Computer Engineering, and also with the Department of Mechanical Engineering, University of Kentucky, Lexington, KY 40506 USA (e-mail: edward.wang@uky.edu)
Received Date: 2020-07-03
Revised Date: 2020-08-31
Accepted Date: 2020-09-19

Available Online: 2020-09-30

Abstract

Abstract

This paper presents an innovative investigation on prototyping a digital twin (DT) as the platform for human-robot interactive welding and welder behavior analysis. This human-robot interaction (HRI) working style helps to enhance human users’ operational productivity and comfort; while data-driven welder behavior analysis benefits to further novice welder training. This HRI system includes three modules: 1) a human user who demonstrates the welding operations offsite with her/his operations recorded by the motion-tracked handles; 2) a robot that executes the demonstrated welding operations to complete the physical welding tasks onsite; 3) a DT system that is developed based on virtual reality (VR) as a digital replica of the physical human-robot interactive welding environment. The DT system bridges a human user and robot through a bi-directional information flow: a) transmitting demonstrated welding operations in VR to the robot in the physical environment; b) displaying the physical welding scenes to human users in VR. Compared to existing DT systems reported in the literatures, the developed one provides better capability in engaging human users in interacting with welding scenes, through an augmented VR. To verify the effectiveness, six welders, skilled with certain manual welding training and unskilled without any training, tested the system by completing the same welding job; three skilled welders produce satisfied welded workpieces, while the other three unskilled do not. A data-driven approach as a combination of fast Fourier transform (FFT), principal component analysis (PCA), and support vector machine (SVM) is developed to analyze their behaviors. Given an operation sequence, i.e., motion speed sequence of the welding torch, frequency features are firstly extracted by FFT and then reduced in dimension through PCA, which are finally routed into SVM for classification. The trained model demonstrates a 94.44% classification accuracy in the testing dataset. The successful pattern recognition in skilled welder operations should benefit to accelerate novice welder training.
- Digital twin (DT),
- human-robot interaction (HRI),
- machine learning,
- virtual reality (VR),
- welder behavior analysis

FullText(HTML)

References(42)

References

[1]	H. S. Kang, J. Y. Lee, S. Choi, H. Kim, J. H. Park, J. Y. Son, B. H. Kim, and S. D. Noh, “Smart manufacturing: Past research, present findings, and future directions,” Int. J. Precis. Eng. Manuf. - Green Technol., vol. 3, no. 1, pp. 111–128, Jan. 2016. doi: 10.1007/s40684-016-0015-5
[2]	N. Q. Wu, Z. W. Li, K. Barkaoui, X. O. Li, T. Murata, and M. C. Zhou, “IoT-based smart and complex systems: A guest editorial report,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 69–73, Jan. 2018. doi: 10.1109/JAS.2017.7510748
[3]	A. Kusiak, “Smart manufacturing must embrace big data,” Nature, vol. 544, no. 7648, pp. 23–25, Apr. 2017. doi: 10.1038/544023a
[4]	M. Ghahramani, Y. Qiao, M. C. Zhou, A. O. Hagan, and J. Sweeney, “AI-based modeling and data-driven evaluation for smart manufacturing processes,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 4, pp. 1026–1037, Jul. 2020. doi: 10.1109/JAS.2020.1003114
[5]	Q. L. Qi and F. Tao, “Digital twin and big data towards smart manufacturing and industry 4.0: 360 degree comparison,” IEEE Access, vol. 6, pp. 3585–3593, Jan. 2018. doi: 10.1109/ACCESS.2018.2793265
[6]	F. Tao, H. Zhang, A. Liu, and A. Y. C. Nee, “Digital twin in industry: State-of-the-art,” IEEE Trans. Ind. Inform., vol. 15, no. 4, pp. 2405–2415, Apr. 2019. doi: 10.1109/TII.2018.2873186
[7]	S. Haag and R. Anderl, “Digital twin – Proof of concept,” Manuf. Lett., vol. 15, pp. 64–66, Jan. 2018. doi: 10.1016/j.mfglet.2018.02.006
[8]	T. DebRoy, W. Zhang, J. Turner, and S. S. Babu, “Building digital twins of 3D printing machines,” Scr. Mater., vol. 135, pp. 119–124, Jul. 2017. doi: 10.1016/j.scriptamat.2016.12.005
[9]	T. Mukherjee and T. DebRoy, “A digital twin for rapid qualification of 3D printed metallic components,” Appl. Mater. Today, vol. 14, pp. 59–65, Mar. 2019. doi: 10.1016/j.apmt.2018.11.003
[10]	R. Soderberg, K. Warmefjord, J. S. Carlson, and L. Lindkvist, “Toward a digital twin for real-time geometry assurance in individualized production,” CIRP Ann., vol. 66, no. 1, pp. 137–140, 2017. doi: 10.1016/j.cirp.2017.04.038
[11]	R. Cupek, M. Drewniak, A. Ziebinski, and M. Fojcik, ““Digital twins” for highly customized electronic devices – Case study on a rework operation,” IEEE Access, vol. 7, pp. 164127–164143, Nov. 2019. doi: 10.1109/ACCESS.2019.2950955
[12]	H. Zhang, Q. Liu, X. Chen, D. Zhang, and J. W. Leng, “A digital twin-based approach for designing and multi-objective optimization of hollow glass production line,” IEEE Access, vol. 5, pp. 26901–26911, Oct. 2017. doi: 10.1109/ACCESS.2017.2766453
[13]	R. Dong, C. Y. She, W. Hardjawana, Y. H. Li, and B. Vucetic, “Deep learning for hybrid 5G services in mobile edge computing systems: Learn from a digital twin,” IEEE Trans. Wirel. Commun., vol. 18, no. 10, pp. 4692–4707, Oct. 2019. doi: 10.1109/TWC.2019.2927312
[14]	R. H. Guerra, R. Quiza, A. Villalonga, J. Arenas, and F. Castano, “Digital twin-based optimization for ultraprecision motion systems with backlash and friction,” IEEE Access, vol. 7, pp. 93462–93472, Jul. 2019. doi: 10.1109/ACCESS.2019.2928141
[15]	R. L. Zhao, D. X. Yan, Q. Liu, J. W. Leng, J. F. Wan, X. Chen, and X. F. Zhang, “Digital twin-driven cyber-physical system for autonomously controlling of micro punching system,” IEEE Access, vol. 7, pp. 9459–9469, Jan. 2019. doi: 10.1109/ACCESS.2019.2891060
[16]	Y. L. Fang, C. Peng, P. Lou, Z. D. Zhou, J. M. Hu, and J. W. Yan, “Digital-twin-based job shop scheduling toward smart manufacturing,” IEEE Trans. Ind. Inform., vol. 15, no. 12, pp. 6425–6435, Dec. 2019. doi: 10.1109/TII.2019.2938572
[17]	B. Akteke-Ozturk, G. W. Weber, and G. Koksal, “Optimization of generalized desirability functions under model uncertainty,” Optimization, vol. 66, no. 12, pp. 2157–2169, Sept. 2017. doi: 10.1080/02331934.2017.1371167
[18]	A. Ozmen, E. Kropat, and G. W. Weber, “Robust optimization in spline regression models for multi-model regulatory networks under polyhedral uncertainty,” Optimization, vol. 66, no. 12, pp. 2135–2155, Dec. 2017. doi: 10.1080/02331934.2016.1209672
[19]	B. Schleich, N. Anwer, L. Mathieu, and S. Wartzack, “Shaping the digital twin for design and production engineering,” CIRP Ann., vol. 66, no. 1, pp. 141–144, 2017. doi: 10.1016/j.cirp.2017.04.040
[20]	J. L. Wang, L. K. Ye, R. X. Gao, C. Li, and L. B. Zhang, “Digital twin for rotating machinery fault diagnosis in smart manufacturing,” Int. J. Prod. Res., vol. 57, no. 12, pp. 3920–3934, 2019. doi: 10.1080/00207543.2018.1552032
[21]	P. Jain, J. Poon, J. P. Singh, C. Spanos, S. R. Sanders, and S. K. Panda, “A digital twin approach for fault diagnosis in distributed photovoltaic systems,” IEEE Trans. Power Electron., vol. 35, no. 1, pp. 940–956, Jan. 2020. doi: 10.1109/TPEL.2019.2911594
[22]	Y. Liu, L. Zhang, Y. Yang, L. F. Zhou, L. Ren, F. Wang, R. Liu, Z. B. Pang, and M. J. Deen, “A novel cloud-based framework for the elderly healthcare services using digital twin,” IEEE Access, vol. 7, pp. 49088–49101, Apr. 2019. doi: 10.1109/ACCESS.2019.2909828
[23]	F. Tao, M. Zhang, Y. S. Liu, and A. Y. C. Nee, “Digital twin driven prognostics and health management for complex equipment,” CIRP Ann., vol. 67, no. 1, pp. 169–172, 2018. doi: 10.1016/j.cirp.2018.04.055
[24]	M. K. Zhou, J. F. Yan, and D. H. Feng, “Digital twin framework and its application to power grid online analysis,” CSEE J. Power Energy Syst., vol. 5, no. 3, pp. 391–398, Sept. 2019.
[25]	S. M. M. Rahman, “Cyber-physical-social system between a humanoid robot and a virtual human through a shared platform for adaptive agent ecology,” IEEE/CAA J. Autom. Sinina, vol. 5, no. 1, pp. 190–203, Jan. 2018. doi: 10.1109/JAS.2017.7510760
[26]	H. H. Zhang and S. K. Agrawal, “An active neck brace controlled by a joystick to assist head motion,” IEEE Robot. Autom. Lett., vol. 3, no. 1, pp. 37–43, Jan. 2018. doi: 10.1109/LRA.2017.2728858
[27]	Z. Ma and P. Ben-Tzvi, “RML glove – An exoskeleton glove mechanism with haptics feedback,” IEEE/ASME Trans. Mechatronics, vol. 20, no. 2, pp. 641–652, Apr. 2015. doi: 10.1109/TMECH.2014.2305842
[28]	G. L. Du, P. Zhang, and X. Liu, “Markerless human-manipulator interface using leap motion with interval Kalman filter and improved particle filter,” IEEE Trans. Ind. Inform., vol. 12, no. 2, pp. 694–704, Apr. 2016. doi: 10.1109/TII.2016.2526674
[29]	K. Zinchenko, C. Y. Wu, and K. T. Song, “A study on speech recognition control for a surgical robot,” IEEE Trans. Ind. Inform., vol. 13, no. 2, pp. 607–615, Apr. 2017. doi: 10.1109/TII.2016.2625818
[30]	J. I. Lipton, A. J. Fay, and D. Rus, “Baxter’s homunculus: Virtual reality spaces for teleoperation in manufacturing,” IEEE Robot. Autom. Lett., vol. 3, no. 1, pp. 179–186, Jan. 2018. doi: 10.1109/LRA.2017.2737046
[31]	Q. Y. Wang, W. H. Jiao, R. Yu, M. T. Johnson, and Y. M. Zhang, “Virtual reality robot-assisted welding based on human intention recognition,” IEEE Trans. Autom. Sci. Eng., vol. 17, no. 2, pp. 799–808, Apr. 2020. doi: 10.1109/TASE.2019.2945607
[32]	M. Borges, A. Symington, B. Coltin, T. Smith, and R. Ventura, “HTC vive: Analysis and accuracy improvement, ” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Madrid, Spain, 2018, pp. 2610–2615.
[33]	D. Whitney, E. Rosen, D. Ullman, E. Phillips, and S. Tellex, “ROS reality: A virtual reality framework using consumer-grade hardware for ROS-enabled robots, ” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Madrid, Spain, 2018, pp. 1–9.
[34]	L. K. Shi, C. D. Tong, T. Lan, and X. H. Shi, “Statistical process monitoring based on ensemble structure analysis,” IEEE/CAA J. Autom. Sinica, 2018. DOI: 10.1109/JAS.2017.7510877
[35]	P. Y. Zhang, S. Shu, and M. C. Zhou, “An online fault detection model and strategies based on SVM-grid in clouds,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 2, pp. 445–456, Mar. 2018. doi: 10.1109/JAS.2017.7510817
[36]	B. Eygi Erdogan, S. Özöğür-Akyüz, and P. Karadayı Ataş, “A novel approach for panel data: An ensemble of weighted functional margin SVM models,” Inf. Sci, 2019. DOI: 10.1016/j.ins.2019.02.045
[37]	M. S. Erden and T. Tomiyama, “Identifying welding skills for training and assistance with robot,” Sci. Technol. Weld. Join., vol. 14, no. 6, pp. 523–532, 2009. doi: 10.1179/136217109X437150
[38]	Y. K. Liu, Y. M. Zhang, and L. Kvidahl, “Skilled human welder intelligence modeling and control: Part I – Modeling,” Weld. J., vol. 93, no. 2, pp. 46-s–52-s, 2014.
[39]	Y. K. Liu, Y. M. Zhang, and L. Kvidahl, “Skilled human welder intelligence modeling and control: Part Ⅱ – Analysis and control applications,” Weld. J., vol. 93, no. 5, pp. 162-s–170-s, 2014.
[40]	A. J. Smola and B. Scholkopf, “A tutorial on support vector regression,” Stat. Comput., vol. 14, no. 3, pp. 199–222, Aug. 2004. doi: 10.1023/B:STCO.0000035301.49549.88
[41]	T. P. Minka, “Automatic choice of dimensionality for PCA,” in Proc. 13th Int. Conf. Neural Information Processing Systems, Cambridge USA, 2000, pp. 577–583.
[42]	F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, no. 85, pp. 2825–2830, 2011.

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(11) / Tables(3)

Get Citation

PDF

XML

Article Metrics

Article views (8890) PDF downloads(162)

Highlights

A digital twin framework is developed for human-robot interactive welding.
Virtual reality enhances the interactive ability of the digital twins with users.
A data-driven method is developed for welder behaving pattern recognition.

Digital Twin for Human-Robot Interactive Welding and Welder Behavior Analysis

doi: 10.1109/JAS.2020.1003518

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content