Proximity Based Automatic Data Annotation for Autonomous Driving

Chen Sun; Jean M. Uwabeza Vianney; Ying Li; Long Chen; Li Li; Fei-Yue Wang; Amir Khajepour; Dongpu Cao

doi:10.1109/JAS.2020.1003033

Volume 7 Issue 2

Mar. 2020

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2020 > 7(2): 395-404

Chen Sun, Jean M. Uwabeza Vianney, Ying Li, Long Chen, Li Li, Fei-Yue Wang, Amir Khajepour and Dongpu Cao, "Proximity Based Automatic Data Annotation for Autonomous Driving," IEEE/CAA J. Autom. Sinica, vol. 7, no. 2, pp. 395-404, Mar. 2020. doi: 10.1109/JAS.2020.1003033

Citation:

Chen Sun, Jean M. Uwabeza Vianney, Ying Li, Long Chen, Li Li, Fei-Yue Wang, Amir Khajepour and Dongpu Cao, "Proximity Based Automatic Data Annotation for Autonomous Driving," IEEE/CAA J. Autom. Sinica, vol. 7, no. 2, pp. 395-404, Mar. 2020. doi: 10.1109/JAS.2020.1003033

Citation:

PDF( 1956 KB)

Proximity Based Automatic Data Annotation for Autonomous Driving

doi: 10.1109/JAS.2020.1003033

More Information

Author Bio:
Chen Sun (SM’17) received the M.A.Sc degree in electrical & computer engineering from University of Toronto, Canada in 2017 and B.Eng. degree in automation from the University of Electronic Science and Technology of China in 2014. He is currently working toward the Ph.D. degree in mechanical & mechatronics engineering at University of Waterloo. He is a Member of Cognitive Autonomous Driving Lab (CogDrive) and supervised by Prof. Dongpu Cao and Prof. Amir Khajepour. His research interests include end-to-end autonomous driving, safety validation for cyber-physical systems, planning, and control for robots

Jean M. Uwabeza Vianney received the BSc. degree from University of Calgary, Canada in 2015. He is currently a second year MASc Student in mechanical & mechatronics engineering at University of Waterloo. He is a Member of Cognitive Autonomous Driving Lab (CogDrive) and supervised by Prof. Dongpu Cao. Prior to starting a master’s program, Jean was a Product Support Analyst at Applanix for more than 2 years. His research interests include end-to-end autonomous driving, vision-based systems, sensor fusion, machine learning, geospatial data engineering, prediction, and planning for autonomous driving

Ying Li received the M.Sc. degree in remote sensing from Wuhan University in 2017. She is currently working toward the Ph.D. degree with the Mobile Sensing and Geodata Science Laboratory, Department of Geography and Environmental Management, University of Waterloo, Canada. Her research interests include autonomous driving, mobile laser scanning, intelligent processing of point clouds, geometric and semantic modeling, and augmented reality

Long Chen (M’12) received the B.Sc. degree in communication engineering and the Ph.D. degree in signal and information processing from Wuhan University. He is currently an Associate Professor with the School of Data and Computer Science, Sun Yat-sen University. His research interests include autonomous driving, robotics, and artificial intelligence where he has contributed more than 70 publications. He serves as an Associate Editor for IEEE Transactions on Intelligent Transportation Systems

Li Li (S’05–M’06–SM’10–F’17) is currently an Associate Professor with the Department of Automation, Tsinghua University, where he was involved in artificial intelligence, intelligent control and sensing, intelligent transportation systems, and intelligent vehicles. He has authored over 90 SCI-indexed international journal papers and over 50 international conference papers. Dr. Li was a Member of the Editorial Advisory Board for Transportation Research Part C: Emerging Technologies, a Member of the Editorial Board of Transport Reviews and ACTA Automatica Sinica. He serves as an Associate Editor for the IEEE Transactions on Intelligent Transportation Systems

Fei-Yue Wang (S’87–M’89–SM’94–F’03) received the Ph.D. degree in computer and systems engineering from the Rensselaer Polytechnic Institute, USA, in 1990. In 1990, he joined the University of Arizona, USA, where he became a Professor and the Director of the Robotics and Automation Laboratory and the Program in Advanced Research for Complex Systems. In 1999, he founded the Intelligent Control and Systems Engineering Center, Institute of Automation, Chinese Academy of Sciences (CAS), under the support of the Outstanding Overseas Chinese Talents Program from the State Planning Council and the 100 Talent Program from CAS. In 2002, he joined the Laboratory of Complex Systems and Intelligence Science, CAS, as the Director, where he was the Vice President for Research, Education, and Academic Exchanges with the Institute of Automation from 2006 to 2010. In 2011, he was named the State Specially Appointed Expert and Director of the State Key Laboratory for Management and Control of Complex Systems. His current research interests include methods and applications for parallel systems, social computing, parallel intelligence, and knowledge automation. Dr. Wang was elected as a Fellow of INCOSE, IFAC, ASME, and AAAS. In 2007, he was a Recipient of the National Prize in Natural Sciences of China and the Outstanding Scientist by ACM for his research contributions in intelligent control and social computing. He was a Recipient of the IEEE Intelligent Transportation Systems (ITS) Outstanding Application and Research Awards in 2009, 2011, and 2015 and the IEEE SMC Norbert Wiener Award in 2014. He was the Chair of IFAC TC on Economic and Social Systems from 2008 to 2011. He has been the General or Program Chair of more than 30 IEEE, INFORMS, ACM, and ASME conferences. He was the Founding Editor-in-Chief of the International Journal of Intelligent Control and Systems from 1995 to 2000 and the IEEE ITS Magazine from 2006 to 2007. He was the Editor-in-Chief of the IEEE Intelligent Systems from 2009 to 2012, and the IEEE Transactions on ITS from 2009 to 2016. He is currently the Editor-in-Chief of the IEEE Transactions on Computational Social Systems and the Founding Editor-in-Chief of the IEEE/CAA Journal of Automatica Sinica, and the Chinese Journal of Command and Control. He was the President of the IEEE ITS Society from 2005 to 2007, the Chinese Association for Science and Technology, USA, in 2005, and the American Zhu Kezhen Education Foundation from 2007 to 2008. He was the Vice President of the ACM China Council from 2010 to 2011. He is currently the President of the IEEE Council on Radio Frequency Identification. Since 2008, he has been the Vice President and the Secretary General of the Chinese Association of Automation

Amir Khajepour received the Ph.D. degree in mechanical engineering from the University of Waterloo, Canada, in 1996. He is a Professor of the Department of Mechanical and Mechatronics Engineering at the University of Waterloo. He holds the Canada Research Chair in Mechatronic Vehicle Systems, and NSERC/General Motors Industrial Research program that applies his expertise in several key multidisciplinary areas including system modeling and control of dynamic systems. His research has resulted in many patents and technology transfers. He is the author of more than 400 journal and conference publications as well as several books. Dr. Khajepour is a Fellow of the Engineering Institute of Canada, the American Society of Mechanical Engineers, and the Canadian Society of Mechanical Engineering. His research interests include the development of hybrid powertrains (electric and air hybrid engines), component sizing and power management design through concurrent optimization, vehicle modeling through real-time simulation and hardware-in-the-loop, active and adaptive suspension systems, vehicle stability, ultra-high-speed robotics, automated laser fabrication, and micro-electrical-mechanical systems

Dongpu Cao (M’13) received the Ph.D. degree from Concordia University, Canada, in 2008. He is a Canada Research Chair in Driver Cognition and Automated Driving, and currently an Associate Professor and Director of Waterloo Cognitive Autonomous Driving (CogDrive) Lab at University of Waterloo, Canada. His current research interests include driver cognition, automated driving, and cognitive autonomous driving. He has contributed more than 180 publications, 2 books and 1 patent. He received the SAE Arch T. Colwell Merit Award in 2012, and three Best Paper Awards from the ASME and IEEE conferences. Dr. Cao serves as an Associate Editor for IEEE Transactions on Vehicular Technology, IEEE Transactions on Intelligent Transportation Systems, IEEE/ASME Transactions on Mechatronics, IEEE Transactions on Industrial Electronics and ASME Journal of Dynamic Systems, Measurement and Control. He was a Guest Editor for Vehicle System Dynamics and IEEE Transactions on SMC: Systems. He serves on the SAE Vehicle Dynamics Standards Committee and acts as the Co-Chair of IEEE ITSS Technical Committee on Cooperative Driving
Corresponding author: F.-Y. Wang is with the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China (e-mail: feiyue.wang@ia.ac.cn)
Received Date: 2020-01-21
Revised Date: 2020-02-04
Accepted Date: 2020-02-12

Available Online: 2020-02-17

Abstract

Abstract

The recent development in autonomous driving involves high-level computer vision and detailed road scene understanding. Today, most autonomous vehicles employ expensive high quality sensor-set such as light detection and ranging (LIDAR) and HD maps with high level annotations. In this paper, we propose a scalable and affordable data collection and annotation framework, image-to-map annotation proximity (I2MAP), for affordance learning in autonomous driving applications. We provide a new driving dataset using our proposed framework for driving scene affordance learning by calibrating the data samples with available tags from online database such as open street map (OSM). Our benchmark consists of 40 000 images with more than 40 affordance labels under various day time and weather even with very challenging heavy snow. We implemented sample advanced driver-assistance systems (ADAS) functions by training our data with neural networks (NN) and cross-validate the results on benchmarks like KITTI and BDD100K, which indicate the effectiveness of our framework and training models.
- Affordance learning,
- autonomous vehicles,
- data synchronization,
- scene understanding

FullText(HTML)

References(38)

References

[1]	F.-Y. Wang, N.-N. Zheng, D. Cao, C. M. Martinez, L. Li, and T. Liu, “Parallel driving in cpss: a unified approach for transport automation and vehicle intelligence,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 4, pp. 577–587, 2017. doi: 10.1109/JAS.2017.7510598
[2]	C. Lv, D. Cao, Y. Zhao, D. J. Auger, M. Sullman, H. Wang, L. M. Dutka, L. Skrypchuk, and A. Mouzakitis, “Analysis of autopilot disengagements occurring during autonomous vehicle testing,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 58–68, 2017.
[3]	Y. Xing, C. Lv, L. Chen, H. Wang, H. Wang, D. Cao, E. Velenis, and F.-Y. Wang, “Advances in vision-based lane detection: algorithms, integration, assessment, and perspectives on acp-based parallel vision,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 3, pp. 645–661, 2018. doi: 10.1109/JAS.2018.7511063
[4]	B. Jähne and H. Haußecker, Computer Vision and Applications: A Guide for Students and Practitioners, Elsevier, 2000.
[5]	S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Malaysia; Pearson Education Limited, 2016.
[6]	A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: the kitti dataset,” The Int. J. Robotics Research, vol. 32, no. 11, pp. 1231–1237, 2013. doi: 10.1177/0278364913491297
[7]	Z. Chen, J. Zhang, and D. Tao, “Progressive Lidar adaptation for road detection,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 3, pp. 693–702, 2019. doi: 10.1109/JAS.2019.1911459
[8]	C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: learning affordance for direct perception in autonomous driving,” in Proc. IEEE Int. Conf. Computer Vision, 2015, pp. 2722–2730.
[9]	H. Guo, D. Cao, H. Chen, C. Lv, H. Wang, and S. Yang, “Vehicle dynamic state estimation: state of the art schemes and perspectives,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 2, pp. 418–431, 2018. doi: 10.1109/JAS.2017.7510811
[10]	V. Rausch, A. Hansen, E. Solowjow, C. Liu, E. Kreuzer, and J. K. Hedrick, “Learning a deep neural net policy for end-to-end control of autonomous vehicles,” in Proc. IEEE American Control Conf., 2017, pp. 4914–4919.
[11]	A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: an open urban driving simulator,” in Proc. 1st Annual Conf. Robot Learning, 2017, pp. 1–16.
[12]	S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: high-fidelity visual and physical simulation for autonomous vehicles,” in Field and Service Robotics. Springer, 2018, pp. 621–635.
[13]	S. Kato, S. Tokunaga, Y. Maruyama, S. Maeda, M. Hirabayashi, Y. Kitsukawa, A. Monrroy, T. Ando, Y. Fujii, and T. Azumi, “Autoware on board: enabling autonomous vehicles with embedded systems,” in Proc. ACM/IEEE 9th Int. Conf. Cyber-Physical Systems, 2018, pp. 287–296.
[14]	H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving models from large-scale video datasets,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2017, pp. 3530–3538.
[15]	A. Seff and J. Xiao, “Learning from maps: visual common sense for autonomous driving,” arXiv Preprint arXiv: 1611.08583, 2016.
[16]	M. Haklay and P. Weber, “Open street map: user-generated street maps,” IEEE Pervas Comput., vol. 7, no. 4, pp. 12–18, 2008. doi: 10.1109/MPRV.2008.80
[17]	F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell, “BDD100K: a diverse driving video database with scalable annotation tooling,” arXiv Preprint arXiv: 1805.04687, 2018.
[18]	M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016, pp. 3213–3223.
[19]	S. Lee, J. Kim, J. Shin Yoon, S. Shin, O. Bailo, N. Kim, T.-H. Lee, H. Seok Hong, S.-H. Han, and I. So Kweon, “Vpgnet: vanishing point guided network for lane and road marking detection and recognition,” in Proc. IEEE Int. Conf. Computer Vision, 2017, pp. 1947–1955.
[20]	G. Li, Y. Yang, and X. Qu, “Deep learning approaches on pedestrian detection in hazy weather,” IEEE Trans. Industrial Electronics, 2019. [Online]. Avaliable: https://ieeexplore.ieee.org/document/8880634/
[21]	X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, and R. Yang, “The apolloscape dataset for autonomous driving,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, 2018, pp. 954–960.
[22]	A. Sauer, N. Savinov, and A. Geiger, “Conditional affordance learning for driving in urban environments,” arXiv Preprint arXiv: 1806.06498, 2018.
[23]	C. Sun, J. M. U. Vianney, and D. Cao, “Affordance learning in direct perception for autonomous driving,” arXiv Preprint arXiv: 1903.08746, 2019.
[24]	S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, “VQA: visual question answering,” in Proc. IEEE Int. Conf. Computer Vision, 2015, pp. 2425–2433.
[25]	S. Ruder, “An overview of multi-task learning in deep neural networks,” arXiv Preprint arXiv: 1706.05098, 2017.
[26]	S. Rezaei and R. Sengupta, “Kalman filter-based integration of dgps and vehicle sensors for localization,” IEEE Trans. Control Systems Technology, vol. 15, no. 6, pp. 1080–1088, 2007. doi: 10.1109/TCST.2006.886439
[27]	S. E. Li, G. Li, J. Yu, C. Liu, B. Cheng, J. Wang, and K. Li, “Kalman filter-based tracking of moving objects using linear ultrasonic sensor array for road vehicles,” Mechanical Systems and Signal Processing, vol. 98, pp. 173–189, 2018. doi: 10.1016/j.ymssp.2017.04.041
[28]	S. Gao, Y. Hou, H. Dong, S. Stichel, and B. Ning, “High-speed trains automatic operation with protection constraints: a resilient nonlinear gainbased feedback control approach,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 4, pp. 992–999, 2019. doi: 10.1109/JAS.2019.1911582
[29]	J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli, “Kinematic and dynamic vehicle models for autonomous driving control design,” in Proc. IEEE Intelligent Vehicles Symp. 2015, pp. 1094–1099.
[30]	K. T. Leung, J. F. Whidborne, D. Purdy, and P. Barber, “Road vehicle state estimation using low-cost GPS/INS,” Mechanical Systems and Signal Processing, vol. 25, no. 6, pp. 1988–2004, 2011. doi: 10.1016/j.ymssp.2010.08.003
[31]	H. Schafer, E. Santana, A. Haden, and R. Biasini, “A commute in data: the comma2k19 dataset,” arXiv Preprint arXiv: 1812.057522018, 2018.
[32]	S. Miura, L.-T. Hsu, F. Chen, and S. Kamijo, “GPS error correction with pseudorange evaluation using three-dimensional maps,” IEEE Trans. Intelligent Transportation Systems, vol. 16, no. 6, pp. 3104–3115, 2015. doi: 10.1109/TITS.2015.2432122
[33]	H. Sairo, D. Akopian, and J. Takala, “Weighted dilution of precision as quality measure in satellite positioning,” IEE Proceedings-Radar,Sonar and Navigation, vol. 150, no. 6, pp. 430–436, 2003. doi: 10.1049/ip-rsn:20031008
[34]	K. Czarnecki and R. Salay, “Towards a framework to manage perceptual uncertainty for safe automated driving,” in Proc. Int. Conf. Computer Safety, Reliability, and Security. Springer, 2018, pp. 439–445.
[35]	E. Herrera-Viedma, F. Herrera, and F. Chiclana, “A consensus model for multiperson decision making with different preference structures,” IEEE Trans. Systems,Man,and Cybernetics–Part A:Systems and Humans, vol. 32, no. 3, pp. 394–402, 2002. doi: 10.1109/TSMCA.2002.802821
[36]	P. M. Kebria, A. Khosravi, S. M. Salaken, and S. Nahavandi, “Deep imitation learning for autonomous vehicles based on convolutional neural networks,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 1, pp. 82–95, 2019.
[37]	Z. Wu, C. Shen, and A. Van Den Hengel, “Wider or deeper: revisiting the resnet model for visual recognition,” Pattern Recognition, vol. 90, pp. 119–133, 2019. doi: 10.1016/j.patcog.2019.01.006
[38]	J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F. F. Li, “Imagenet: a large-scale hierarchical image database,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009, pp. 248–255.

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(8) / Tables(2)

Get Citation

PDF

XML

Article Metrics

Article views (2799) PDF downloads(99)

Highlights

Affordable and scalable uncertainty-aware Image2Map annotation scheme for ADS.
Proximity-based label extraction with GNSS calibration and EKF based on kinematic vehicle model.
Present a scene understanding benchmark for multi-label learning collected in Canada winter.

Proximity Based Automatic Data Annotation for Autonomous Driving

doi: 10.1109/JAS.2020.1003033

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content