IEEE/CAA Journal of Automatica Sinica
Citation: | L. L. Fan, S. Li, Y. Li, B. Li, D. P. Cao, and F.-Y. Wang, “Pavement cracks coupled with shadows: A new shadow-crack dataset and a shadow-removal-oriented crack detection approach,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 7, pp. 1593–1607, Jul. 2023. doi: 10.1109/JAS.2023.123447 |
Automatic pavement crack detection is a critical task for maintaining the pavement stability and driving safety. The task is challenging because the shadows on the pavement may have similar intensity with the crack, which interfere with the crack detection performance. Till to the present, there still lacks efficient algorithm models and training datasets to deal with the interference brought by the shadows. To fill in the gap, we made several contributions as follows. First, we proposed a new pavement shadow and crack dataset, which contains a variety of shadow and pavement pixel size combinations. It also covers all common cracks (linear cracks and network cracks), placing higher demands on crack detection methods. Second, we designed a two-step shadow-removal-oriented crack detection approach: SROCD, which improves the performance of the algorithm by first removing the shadow and then detecting it. In addition to shadows, the method can cope with other noise disturbances. Third, we explored the mechanism of how shadows affect crack detection. Based on this mechanism, we propose a data augmentation method based on the difference in brightness values, which can adapt to brightness changes caused by seasonal and weather changes. Finally, we introduced a residual feature augmentation algorithm to detect small cracks that can predict sudden disasters, and the algorithm improves the performance of the model overall. We compare our method with the state-of-the-art methods on existing pavement crack datasets and the shadow-crack dataset, and the experimental results demonstrate the superiority of our method.
[1] |
L. Zhang, F. Yang, Y. D. Zhang, and Y. J. Zhu, “Road crack detection using deep convolutional neural network,” in Proc. IEEE Int. Conf. Image Processing, Phoenix, USA, 2016, pp. 3708–3712.
|
[2] |
Y. Shi, L. M. Cui, Z. Q. Qi, F. Meng, and Z. S. Chen, “Automatic road crack detection using random structured forests,” IEEE Trans. Intell. Trans. Syst., vol. 17, no. 12, pp. 3434–3445, Dec. 2016. doi: 10.1109/TITS.2016.2552248
|
[3] |
S. J. Schmugge, L. Rice, J. Lindberg, R. Grizziy, C. Joffey, and M. C. Shin, “Crack segmentation by leveraging multiple frames of varying illumination,” in Proc. IEEE Winter Conf. Applications of Computer Vision, Santa Rosa, USA, 2017, pp. 1045–1053.
|
[4] |
A. Benedetto, F. Tosti, L. Pajewski, F. D’Amico, and W. Kusayanagi, “FDTD simulation of the GPR signal for effective inspection of pavement damages,” in Proc. 15th Int. Conf. Ground Penetrating Radar, Brussels, Belgium, 2014, pp. 513–518.
|
[5] |
Q. Zou, Y. Cao, Q. Q. Li, Q. Z. Mao, and S. Wang, “CrackTree: Automatic crack detection from pavement images,” Pattern Recognit. Lett., vol. 33, no. 3, pp. 227–238, Feb. 2012. doi: 10.1016/j.patrec.2011.11.004
|
[6] |
N. Tanaka and K. Uematsu, “A crack detection method in road surface images using morphology,” in Proc. IAPR Workshop on Machine Vision Applications, Chiba, Japan, 1998, pp. 154–157.
|
[7] |
Q. Q. Li and X. L. Liu, “Novel approach to pavement image segmentation based on neighboring difference histogram method,” in Proc. Congr. Image and Signal Processing, Sanya, China, 2008, pp. 792–796.
|
[8] |
T. Ahonen, A. Hadid, and M. Pietikainen, “Face description with local binary patterns: Application to face recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 12, pp. 2037–2041, Dec. 2006. doi: 10.1109/TPAMI.2006.244
|
[9] |
M. Salman, S. Mathavan, K. Kamal, and M. Rahman, “Pavement crack detection using the Gabor filter,” in Proc. 16th Int. IEEE Conf. Intelligent Transportation Systems, The Hague, Netherlands, 2013, pp. 2039–2044.
|
[10] |
A. Zhang, K. C. P. Wang, B. X. Li, E. H. Yang, X. X. Dai, Y. Peng, Y. Fei, Y. Liu, J. Q. Li, and C. Chen, “Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network,” vol. 32, no. 10, pp. 805–819, Oct. 2017.
|
[11] |
P. P. Lu, C. Cui, S. B. Xu, H. E. Peng, and F. Wang, “SUPER: A novel lane detection system,” IEEE Trans. Intell. Veh., vol. 6, no. 3, pp. 583–593, Sept. 2021. doi: 10.1109/TIV.2021.3071593
|
[12] |
F. Yang, L. Zhang, S. J. Yu, D. Prokhorov, X. Mei, and H. B. Ling, “Feature pyramid and hierarchical boosting network for pavement crack detection,” IEEE Trans. Intell. Trans. Syst., vol. 21, no. 4, pp. 1525–1535, Apr. 2020. doi: 10.1109/TITS.2019.2910595
|
[13] |
K. Lu, “Advances in deep learning methods for pavement surface crack detection and identification with visible light visual images,” in Proc. Conf. Computer Vision and Pattern Recognition, Virtual, 2020.
|
[14] |
G. J. Wang, J. Wu, R. He, and B. Tian, “Speed and accuracy tradeoff for LiDAR data based road boundary detection,” IEEE/CAA J. Autom. Sinica, vol. 8, no. 6, pp. 1210–1220, Jun. 2021. doi: 10.1109/JAS.2020.1003414
|
[15] |
L. Jiang, Y. Xie, and T. Ren, “A deep neural networks approach for pixel-level runway pavement crack segmentation using drone-captured images,” in Proc. Conf. Computer Vision and Pattern Recognition, Virtual, 2020.
|
[16] |
H. Wang, Y. J. Yu, Y. F. Cai, X. B. Chen, L. Chen, and Y. C. Li, “Soft-weighted-average ensemble vehicle detection method based on single-stage and two-stage deep learning models,” IEEE Trans. Intell. Veh., vol. 6, no. 1, pp. 100–109, Mar. 2021. doi: 10.1109/TIV.2020.3010832
|
[17] |
B. Bešić and A. Valada, “Dynamic object removal and spatio-temporal RGB-D inpainting via geometry-aware adversarial learning,” IEEE Trans. Intell. Veh., vol. 7, no. 2, pp. 170–185, Jun. 2022. doi: 10.1109/TIV.2022.3140654
|
[18] |
W. Y. Liu, G. F. Ren, R. S. Yu, S. Guo, J. K. Zhu, and L. Zhang, “Image-adaptive YOLO for object detection in adverse weather conditions,” in Proc. 36th AAAI Conf. Artificial Intelligence, 2022, pp. 1792–1800.
|
[19] |
H. Tian, T. Deng, and H. M. Yan, “Driving as well as on a sunny Day? Predicting driver’s fixation in rainy weather conditions via a dual-branch visual model” IEEE/CAA J. Autom. Sinica, vol. 9, no. 7, pp. 1335–1338, Jul. 2022. doi: 10.1109/JAS.2022.105716
|
[20] |
G. Volk, S. Müller, A. von Bernuth, D. Hospach, and O. Bringmann, “Towards robust CNN-based object detection through augmentation with synthetic rain variations,” in Proc. IEEE Intelligent Transportation Systems Conf., Auckland, New Zealand, 2019, pp. 285–292.
|
[21] |
K. H. Liu, Z. H. Ye, H. Y. Guo, D. P. Cao, L. Chen, and F.-Y. Wang, “FISS GAN: A generative adversarial network for foggy image semantic segmentation,” IEEE/CAA J. Autom. Sinica, vol. 8, no. 8, pp. 1428–1439, Aug. 2021. doi: 10.1109/JAS.2021.1004057
|
[22] |
J. Huyan, W. Li, S. Tighe, R. R. Deng, and S. Yan, “Illumination compensation model with k-means algorithm for detection of pavement surface cracks with shadow,” J. Comput. Civ. Eng., vol. 34, no. 1, p. 04019049, Jan. 2020. doi: 10.1061/(ASCE)CP.1943-5487.0000869
|
[23] |
C. X. Guo, B. Fan, Q. Zhang, S. M. Xiang, and C. H. Pan, “AugFPN: Improving multi-scale feature learning for object detection,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, USA, 2019, pp. 12592–12601.
|
[24] |
E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 640–651, Apr. 2017. doi: 10.1109/TPAMI.2016.2572683
|
[25] |
X. W. Hu, C.-W. Fu, L. Zhu, J. Qin, and P.-A. Heng, “Direction-aware spatial context features for shadow detection and removal,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 11, pp. 2795–2808, Nov. 2020. doi: 10.1109/TPAMI.2019.2919616
|
[26] |
S. N. Xie and Z. W. Tu, “Holistically-nested edge detection,” in Proc. IEEE Int. Conf. Computer Vision, Santiago, Chile, 2015, pp. 1395–1403.
|
[27] |
M. A. Hedeya, E. Samir, E. El-Sayed, A. A. El-sharkawy, M. F. Abdel-Kader, A. Moussa, and R. F. Abdel-Kader, “A low-cost multi-sensor deep learning system for pavement distress detection and severity classification,” in Proc. 8th Int. Conf. Advanced Machine Learning and Technologies and Applications, Cairo, Egypt, 2022, pp. 21–33.
|
[28] |
Y. Liu, M.-M. Cheng, X. W. Hu, J.-W. Bian, L. Zhang, X. Bai, and J. H. Tang, “Richer convolutional features for edge detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 8, pp. 1939–1946, Aug. 2019. doi: 10.1109/TPAMI.2018.2878849
|
[29] |
J. S. Tang and Y. L. Gu, “Automatic crack detection and segmentation using a hybrid algorithm for road distress analysis,” in Proc. IEEE Int. Conf. Systems, Man, and Cybernetics, Manchester, UK, 2013, pp. 3026–3030.
|
[30] |
H. Oliveira and P. L. Correia, “CrackIT-An image processing toolbox for crack detection and characterization,” in Proc. IEEE Int. Conf. Image Processing, Paris, France, 2014, pp. 798–802.
|
[31] |
M. D. Yan, S. B. Bo, K. Xu, and Y. Y. He, “Pavement crack detection and analysis for high-grade highway,” in Proc. 8th Int. Conf. Electronic Measurement and Instruments, Xi’an, China, 2007, pp. 4-548–4-552.
|
[32] |
J. Zhou, P. S. Huang, and F.-P. Chiang, “Wavelet-based pavement distress detection and evaluation,” in Proc. SPIE of 5207, Wavelets: Applications in Signal and Image Processing X, San Diego, USA, 2003, pp. 728–739.
|
[33] |
P. Subirats, J. Dumoulin, V. Legeay, and D. Barba, “Automation of pavement surface crack detection using the continuous wavelet transform,” in Proc. Int. Conf. Image Processing, Atlanta, USA, 2006, pp. 3037–3040.
|
[34] |
M. Quintana, J. Torres, and J. M. Menéndez, “A simplified computer vision system for road surface inspection and maintenance,” IEEE Trans. Intell. Trans. Syst., vol. 17, no. 3, pp. 608–619, Mar. 2016. doi: 10.1109/TITS.2015.2482222
|
[35] |
V. Baltazart, P. Nicolle, and L. Yang, “Ongoing tests and improvements of the MPS algorithm for the automatic crack detection within grey level pavement images,” in Proc. 25th European Signal Processing Conf., Kos, Greece, 2017, pp. 2016–2020.
|
[36] |
T. S. Nguyen, S. Begot, F. Duculty, and M. Avila, “Free-form anisotropy: A new method for crack detection on pavement surface images,” in Proc. 18th IEEE Int. Conf. Image Processing, Brussels, Belgium, 2011, pp. 1069–1072.
|
[37] |
C. Ieracitano, A. Paviglianiti, M. Campolo, A. Hussain, E. Pasero, and F. C. Morabito, “A novel automatic classification system based on hybrid unsupervised and supervised machine learning for electrospun nanofibers,” IEEE/CAA J. Autom. Sinica, vol. 8, no. 1, pp. 64–76, Jan. 2021. doi: 10.1109/JAS.2020.1003387
|
[38] |
P. Prasanna, K. J. Dana, N. Gucunski, B. B. Basily, H. M. La, R. S. Lim, and H. Parvardeh, “Automated crack detection on concrete bridges,” IEEE Trans. Autom. Sci. Eng., vol. 13, no. 2, pp. 591–599, Apr. 2016. doi: 10.1109/TASE.2014.2354314
|
[39] |
X. Li, H. B. Duan, Y. L. Tian, and F.-Y. Wang, “Exploring image generation for UAV change detection,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 6, pp. 1061–1072, Jun. 2022. doi: 10.1109/JAS.2022.105629
|
[40] |
K. Gopalakrishnan, S. K. Khaitan, A. Choudhary, and A. Agrawal, “Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection,” Constr. Build. Mater., vol. 157, pp. 322–330, Dec. 2017. doi: 10.1016/j.conbuildmat.2017.09.110
|
[41] |
J. Zhang, L. Pan, Q. L. Han, C. Chen, S. Wen, and Y. Xiang, “Deep learning based attack detection for cyber-physical system cybersecurity: A survey,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 3, pp. 377–391, Mar. 2022. doi: 10.1109/JAS.2021.1004261
|
[42] |
K. Hacıefendioğlu and H. B. Başağa, “Concrete road crack detection using deep learning-based faster R-CNN method,” Iran. J. Sci. Technol.,Trans. Civ. Eng., vol. 46, no. 2, pp. 1621–1633, Apr. 2022. doi: 10.1007/s40996-021-00671-2
|
[43] |
G. X. Hu, B. L. Hu, Z. Yang, L. Huang, and P. Li, “Pavement crack detection method based on deep learning models,” Wirel. Commun. Mob. Comput., vol. 2021, p. 5573590, May 2021.
|
[44] |
H. Lv, C. Liu, X. W. Zhao, C. Y. Xu, Z. Cui, and J. Yang, “Lane marking regression from confidence area detection to field inference,” IEEE Trans. Intell. Veh., vol. 6, no. 1, pp. 47–56, Mar. 2021. doi: 10.1109/TIV.2020.3009366
|
[45] |
L. L. Fan, H. W. Zhao, Y. Li, S. Li, R. Zhou, and W. B. Chu, “RAO-UNet: A residual attention and octave UNet for road crack detection via balance loss,” IET Intell. Trans. Syst., vol. 16, no. 3, pp. 332–343, Mar. 2022. doi: 10.1049/itr2.12146
|
[46] |
L. Pauly, H. Peel, S. Luo, D. Hogg, and R. Fuentes, “Deeper networks for pavement crack detection,” in Proc. 34th Int. Symp. Autom. and Robotics in Construction, Taipei, China, 2017, pp. 479–485.
|
[47] |
J. E. Hoffmann, H. G. Tosso, M. M. D. Santos, J. F. Justo, A. W. Malik, and A. U. Rahman, “Real-time adaptive object detection and tracking for autonomous vehicles,” IEEE Trans. Intell. Veh., vol. 6, no. 3, pp. 450–459, Sept. 2021. doi: 10.1109/TIV.2020.3037928
|
[48] |
Y. X. Wang, S. Qiu, D. Li, C. D. Du, B.-L. Lu, and H. G. He, “Multi-modal domain adaptation variational autoencoder for EEG-based emotion recognition,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 9, pp. 1612–1626, Sept. 2022. doi: 10.1109/JAS.2022.105515
|
[49] |
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F. F. Li, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Miami, USA, 2009, pp. 248–255.
|
[50] |
K. G. Zhang, Y. T. Zhang, and H.-D. Cheng, “CrackGAN: Pavement crack detection using partially accurate ground truths based on generative adversarial learning,” IEEE Trans. Intell. Trans. Syst., vol. 22, no. 2, pp. 1306–1319, Feb. 2021. doi: 10.1109/TITS.2020.2990703
|