Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments

Yuzhen Liu; Ziyang Meng; Yao Zou; Ming Cao

doi:10.1109/JAS.2020.1003530

Volume 8 Issue 2

Feb. 2021

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2021 > 8(2): 344-360

Yuzhen Liu, Ziyang Meng, Yao Zou and Ming Cao, "Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 344-360, Feb. 2021. doi: 10.1109/JAS.2020.1003530

Citation:

Yuzhen Liu, Ziyang Meng, Yao Zou and Ming Cao, "Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments," IEEE/CAA J. Autom. Sinica, vol. 8, no. 2, pp. 344-360, Feb. 2021. doi: 10.1109/JAS.2020.1003530

Citation:

PDF( 2754 KB)

Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments

doi: 10.1109/JAS.2020.1003530

Funds: This work was supported in part by the Institute for Guo Qiang of Tsinghua University (2019GQG1023), in part by Graduate Education and Teaching Reform Project of Tsinghua University (202007J007), in part by National Natural Science Foundation of China (U19B2029, 62073028, 61803222), and in part by the Independent Research Program of Tsinghua University (2018Z05JDX002)

More Information

Author Bio:
Yuzhen Liu received the B.S. degree in measurement and control technology and instrumentation from the School of Artificial Intelligence and Automation, Huazhong University of Science & Technology in 2017. He is currently working toward the Ph.D. degree with the Department of Precision Instrument, Tsinghua University. His research interests include state estimation, visual-inertial localization and mapping, sensor fusion, and visual object tracking

Ziyang Meng (SM’20) is currently an Associate Professor with the Department of Precision Instrument, Tsinghua University. He received the B.S. degree with honors from Huazhong University of Science & Technology in 2006, and Ph.D. degree from Tsinghua University in 2010. He was an exchange Ph.D. student at Utah State University, USA, from 2008 to 2009. Prior to joining Tsinghua University, he held postdoc, researcher, and Humboldt research fellow positions at, respectively, Shanghai Jiao Tong University, KTH Royal Institute of Technology, Sweden, and Technical University of Munich, Germany from 2010 to 2015. He was selected to the national “1000-Youth Talent Program” of China in 2015. His research interests include distributed control and optimization, space science, and intelligent navigation technique. He serves as an Associate Editor for Systems & Control Letters and IET Control Theory & Applications

Yao Zou received the B.S. degree in automation from Dalian University of Technology (DUT) in 2010, and Ph.D. degree in control science and engineering from Beihang University (BUAA, formerly named Beijing University of Aeronautics and Astronautics) in 2016. He was a Post-Doctoral Research Fellow with the Department of Precision Instrument, Tsinghua University from 2017 to 2018. He is currently an Associate Professor with the School of Automation and Electrical Engineering, University of Science and Technology Beijing. His current research interests include nonlinear control, unmanned aerial vehicle control, and multiagent control

Ming Cao (SM’16) is a Professor of systems and control with the Engineering and Technology Institute (ENTEG) at the University of Groningen, The Netherlands, where he started in 2008. He received the bachelor degree in 1999 and the master degree in 2002 from Tsinghua University, and the Ph.D. degree in 2007 from Yale University, USA, all in electrical engineering. From September 2007 to August 2008, he was a Postdoctoral Research Associate with the Department of Mechanical and Aerospace Engineering at Princeton University, USA. He worked as a research intern during the summer of 2006 with the Mathematical Sciences Department at the IBM T. J. Watson Research Center, USA. He is the 2017 and inaugural Recipient of the Manfred Thoma medal from the International Federation of Automatic Control (IFAC) and the 2016 Recipient of the European Control Award sponsored by the European Control Association (EUCA). He is a Senior Editor for Systems and Control Letters, and an Associate Editor for IEEE Transactions on Automatic Control. He is a Vice Chair of the IFAC Technical Committee on Large-Scale Complex Systems. His research interests include autonomous agents and multi-agent systems, complex networks, and decision-making processes
Corresponding author: Y. Z. Liu and Z. Y. Meng are with the Department of Precision Instrument, Tsinghua University, Beijing 100084, China (e-mail: lyz17@mails.tsinghua.edu.cn; ziyangmeng@tsinghua.edu.cn)
¹https://www.dji.com/cn
²https://wiki.bitcraze.io/
³In Algorithm 1, we test the sub profiles.
⁴http://www.optitrack.com/
Received Date: 2020-09-08
Accepted Date: 2020-10-01

Abstract

Abstract

There are two main trends in the development of unmanned aerial vehicle (UAV) technologies: miniaturization and intellectualization, in which realizing object tracking capabilities for a nano-scale UAV is one of the most challenging problems. In this paper, we present a visual object tracking and servoing control system utilizing a tailor-made 38 g nano-scale quadrotor. A lightweight visual module is integrated to enable object tracking capabilities, and a micro positioning deck is mounted to provide accurate pose estimation. In order to be robust against object appearance variations, a novel object tracking algorithm, denoted by RMCTer, is proposed, which integrates a powerful short-term tracking module and an efficient long-term processing module. In particular, the long-term processing module can provide additional object information and modify the short-term tracking model in a timely manner. Furthermore, a position-based visual servoing control method is proposed for the quadrotor, where an adaptive tracking controller is designed by leveraging backstepping and adaptive techniques. Stable and accurate object tracking is achieved even under disturbances. Experimental results are presented to demonstrate the high accuracy and stability of the whole tracking system.
- Nano-scale quadrotor,
- nonlinear control,
- position-based visual servoing,
- visual object tracking

FullText(HTML)

¹https://www.dji.com/cn
²https://wiki.bitcraze.io/
³In Algorithm 1, we test the sub profiles.
⁴http://www.optitrack.com/

References(47)

References

[1]	K. Fregene, “Unmanned aerial vehicles and control: Lockheed martin advanced technology laboratories,” IEEE Control Syst. Mag., vol. 32, no. 5, pp. 32–34, Oct. 2012. doi: 10.1109/MCS.2012.2205474
[2]	X. Zhang, B. Xian, B. Zhao, and Y. Zhang, “Autonomous flight control of a nano quadrotor helicopter in a GPS-denied environment using on-board vision,” IEEE Trans. Ind. Electron., vol. 62, no. 10, pp. 6392–6403, Oct. 2015. doi: 10.1109/TIE.2015.2420036
[3]	G. W. Cai, J. Dias, and L. Seneviratne, “A survey of small-scale unmanned aerial vehicles: Recent advances and future development trend,” Unmanned Syst., vol. 2, no. 2, pp. 175–199, Apr. 2014. doi: 10.1142/S2301385014300017
[4]	V. Srisamosorn, N. Kuwahara, A. Yamashita, T. Ogata, and J. Ota, “Human-tracking system using quadrotors and multiple environmental cameras for face-tracking application,” Int. J. Adv. Robot. Syst., vol. 14, no. 5, pp. 1–18, Oct. 2017.
[5]	A. Briod, J. C. Zufferey, and D. Floreano, “Optic-flow based control of a 46 g quadrotor,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Tokyo, Japan, 2013, pp. 149–158.
[6]	D. Palossi, J. Singh, M. Magno, and L. Benini, “Target following on nano-scale unmanned aerial vehicles,” in Proc. 7th IEEE Int. Workshop on Advances in Sensors and Interfaces, Vieste, Italy, 2017, pp. 170–175.
[7]	D. Floreano and R. J. Wood, “Science, technology and the future of small autonomous drones,” Nature, vol. 521, no. 7553, pp. 460–466, May 2015. doi: 10.1038/nature14542
[8]	P. Serra, R. Cunha, T. Hamel, D. Cabecinhas, and C. Silvestre, “Landing of a quadrotor on a moving target using dynamic image-based visual servo control,” IEEE Trans. Robot., vol. 32, no. 6, pp. 1524–1535, Dec. 2016. doi: 10.1109/TRO.2016.2604495
[9]	H. Cheng, L. S. Lin, Z. Q. Zheng, Y. W. Guan, and Z. C. Liu, “An autonomous vision-based target tracking system for rotorcraft unmanned aerial vehicles,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Vancouver, Canada, 2017, pp. 1732–1738.
[10]	A. Rodriguez-Ramos, A. Alvarez-Fernandez, H. Bavle, P. Campoy, and J. P. How, “Vision-based multirotor following using synthetic learning techniques,” Sensors, vol. 19, no. 21, pp. 4794, Nov. 2019. doi: 10.3390/s19214794
[11]	Y. J. Yin, X. G. Wang, D. Xu, F. F. Liu, Y. L. Wang, and W. Q. Wu, “Robust visual detection-learning-tracking framework for autonomous aerial refueling of UAVs,” IEEE Trans. Instrum. Meas., vol. 65, no. 3, pp. 510–521, Mar. 2016. doi: 10.1109/TIM.2015.2509318
[12]	P. Campoy, J. F. Correa, I. Mondragón, I. Mondragón, C. Martínez, M. Olivares, L. Mejías, and J. Artieda, “Computer vision onboard UAVs for civilian tasks,” J. Intell. Robot. Syst., vol. 54, no. 1–3, pp. 105–134, 2009. doi: 10.1007/s10846-008-9256-z
[13]	J. Pestana, J. L. Sanchez-Lopez, S. Saripalli, and P. Campoy, “Computer vision based general object following for GPS-denied multirotor unmanned vehicles,” in Proc. American Control Conf., Portland, USA, 2014, pp. 1886–1891.
[14]	R. Li, M. Pang, C. Zhao, G. Y. Zhou, and L. Fang, “Monocular long-term target following on UAVs,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Las Vegas, USA, 2016, pp. 29–37.
[15]	Z. Kalal, K. Mikolajczyk, and J. Matas, “Tracking-learning-detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1409–1422, Jul. 2012. doi: 10.1109/TPAMI.2011.239
[16]	C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in Proc. 13th European Conf. Computer Vision, Zurich, Switzerland, 2014, pp. 391–405.
[17]	B. Babenko, M. H. Yang, and S. Belongie, “Robust object tracking with online multiple instance learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1619–1632, Aug. 2011. doi: 10.1109/TPAMI.2010.226
[18]	S. Hare, S. Golodetz, A. Saffari, V. Vineet, M. M. Cheng, S. L. Hicks, and P. H. S. Torr, “Struck: Structured output tracking with kernels,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 10, pp. 2096–2109, Oct. 2016. doi: 10.1109/TPAMI.2015.2509974
[19]	D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, San Francisco, USA, 2010, pp. 2544–2550.
[20]	J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 3, pp. 583–596, Mar. 2015. doi: 10.1109/TPAMI.2014.2345390
[21]	J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “Exploiting the circulant structure of tracking-by-detection with kernels,” in Proc. 12th European Conf. Computer Vision, Florence, Italy, 2012, pp. 702–715.
[22]	M. Danelljan, G. Häger, F. S. Khan, and M. Felsberg, “Discriminative scale space tracking,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 8, pp. 1561–1575, Aug. 2017. doi: 10.1109/TPAMI.2016.2609928
[23]	J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv preprint arXiv: 1804.02767, 2018.
[24]	K. M. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 2, pp. 386–397, Feb. 2020. doi: 10.1109/TPAMI.2018.2844175
[25]	D. L. Zheng, H. S. Wang, W. D. Chen, and Y. Wang, “Planning and tracking in image space for image-based visual servoing of a quadrotor,” IEEE Trans. Ind. Electron., vol. 65, no. 4, pp. 3376–3385, Apr. 2018. doi: 10.1109/TIE.2017.2752124
[26]	F. Chaumette and S. Hutchinson, “Visual servo control. I. Basic approaches,” IEEE Robot. &Automat. Mag., vol. 13, no. 4, pp. 82–90, Dec. 2006.
[27]	N. Guenard, T. Hamel, and R. Mahony, “A practical visual servo control for an unmanned aerial vehicle,” IEEE Trans. Robot., vol. 24, no. 2, pp. 331–340, Apr. 2008. doi: 10.1109/TRO.2008.916666
[28]	M. G. Popova and H. H. Liu, “Position-based visual servoing for target tracking by a quadrotor UAV,” in Proc. AIAA Guidance, Navigation, and Control Conf., San Diego, USA, 2016, pp. 2092–2103.
[29]	W. B. Zhao, H. Liu, F. L. Lewis, K. P. Valavanis, and X. L. Wang, “Robust visual servoing control for ground target tracking of quadrotors,” IEEE Trans. Control Syst. Technol., vol. 28, no. 5, pp. 1980–1987, Sept. 2020. doi: 10.1109/TCST.2019.2922159
[30]	F. Rinaldi, S. Chiesa, and F. Quagliotti, “Linear quadratic control for quadrotors UAVs dynamics and formation flight,” J. Intell. &Robot. Syst., vol. 70, no. 1–4, pp. 203–220, Apr. 2013.
[31]	D. Mellinger, M. Shomin, and V. Kumar, “Control of quadrotors for robust perching and landing,” in Proc. Int. Powered Lift Conf., Philadelphia, USA, 2010, pp. 205–225.
[32]	Y. Z. Liu and Z. Y. Meng, “Visual object tracking for a nano-scale quadrotor,” in Proc. 15th Int. Conf. Control, Automation, Robotics and Vision, Singapore, 2018, pp. 843–847.
[33]	T. Yang, N. Sun, H. Chen, and Y. C. Fang, “Neural network-based adaptive antiswing control of an underactuated ship-mounted crane with roll motions and input dead zones,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 3, pp. 901–914, Mar. 2020. doi: 10.1109/TNNLS.2019.2910580
[34]	N. Sun, D. K. Liang, Y. M. Wu, Y. H. Chen, Y. D. Qin, and Y. C. Fang, “Adaptive control for pneumatic artificial muscle systems with parametric uncertainties and unidirectional input constraints,” IEEE Trans. Ind. Informatics, vol. 16, no. 2, pp. 969–979, Feb. 2020. doi: 10.1109/TII.2019.2923715
[35]	B. Zhao, B. Xian, Y. Zhang, and X. Zhang, “Nonlinear robust adaptive tracking control of a quadrotor UAV via immersion and invariance methodology,” IEEE Trans. Ind. Electron., vol. 62, no. 5, pp. 2891–2902, May 2015. doi: 10.1109/TIE.2014.2364982
[36]	Y. Zou and W. Huo, “Adaptive tracking control for a model helicopter with disturbances,” in Proc. American Control Conf, Chicago, USA, 2015, pp. 3824–3829.
[37]	P. Marantos, C. P. Bechlioulis, and K. J. Kyriakopoulos, “Robust trajectory tracking control for small-scale unmanned helicopters with model uncertainties,” IEEE Trans. Control Syst. Technol., vol. 25, no. 6, pp. 2010–2021, Nov. 2017. doi: 10.1109/TCST.2016.2642160
[38]	M. W. Mueller, M. Hamer, and R. D’Andrea, “Fusing ultra-wideband range measurements with accelerometers and rate gyroscopes for quadrocopter state estimation,” in Proc. IEEE Int. Conf. Robotics & Automation, Seattle, USA, 2015, pp. 1730–1736.
[39]	M. Greiff, “Modelling and control of the crazyflie quadrotor for aggressive and autonomous flight by optical flow driven state estimation,” M.S. thesis, Lund University, Sweden, 2017.
[40]	Y. J. Li, “The design and implementation of four rotor UAV fixed landing system based on machine vision,” M.S. thesis, South China University of Technology, Guangzhou, China, 2015.
[41]	R. Mahony, V. Kumar, and P. Corke, “Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor,” IEEE Robot. &Autom. Mag., vol. 19, no. 3, pp. 20–32, Sept. 2012.
[42]	Y. Zou, “Trajectory tracking controller for quadrotors without velocity and angular velocity measurements,” IET Control Theory &Appl., vol. 11, no. 1, pp. 101–109, Jan. 2017.
[43]	M. Huang, B. Xian, C. Diao, K. Y. Yang, and Y. Feng, “Adaptive tracking control of underactuated quadrotor unmanned aerial vehicles via backstepping,” in Proc. American Control Conf., Baltimore, USA, 2010, pp. 2076–2081.
[44]	B. Zhu and W. Huo, “Robust nonlinear control for a model-scaled helicopter with parameter uncertainties,” Nonlinear Dyn., vol. 73, no. 1–2, pp. 1139–1154, Jul. 2013. doi: 10.1007/s11071-013-0858-z
[45]	M. Krstić, I. Kanellakopoulos, and P. Kokotović, Nonlinear and Adaptive Control Design. New York, UK: Wiley, 1995.
[46]	A. Levant, “Higher-order sliding modes, differentiation and output-feedback control,” Int. J. Control, vol. 76, no. 9–10, pp. 924–941, 2003. doi: 10.1080/0020717031000099029
[47]	Y. Wu, J. Lim, and M. H. Yang, “Object tracking benchmark,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1834–1848, Sept. 2015. doi: 10.1109/TPAMI.2014.2388226

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(14) / Tables(1)

Get Citation

PDF

XML

Article Metrics

Article views (2377) PDF downloads(125)

Highlights

This paper proposes a complete visual object tracking and servoing control system using a tailor-made 38 g nano-scale quadrotor platform. This tracking system is composed of a versatile and robust visual object tracking module, and an efficient PBVS control module. Due to the limited payload, a lightweight monocular visual module is integrated to equip the quadrotor with the capability of object tracking. Additionally, we present a micro positioning deck to provide more stable and accurate pose estimation for the quadrotor.
This paper proposes a novel object tracking algorithm, i.e., RMCTer, where a two-stage short-term tracking module and an efficient long-term processing module are tightly integrated to collaboratively process the input frames. Compared with the tracking algorithms such as STUCK, DSST and KCF, the proposed tracker is more applicable in the presence of the variations of object appearance and can effectively compensate the visual tracking errors thanks to the adequate model modification provided by the long-term processing module.
This paper proposes an adaptive PBVS control algorithm by leveraging the backstepping and adaption techniques. The proposed controller is robust against the uncertain model parameters and the existence of external disturbances, and their exact model information is not needed in the design of the controller.

Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments

doi: 10.1109/JAS.2020.1003530

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content