Geographic, Geometrical and Semantic Reconstruction of Urban Scene from High Resolution Oblique Aerial Images

Xiaofeng Sun; Shuhan Shen; Hainan Cui; Lihua Hu; Zhanyi Hu

doi:10.1109/JAS.2017.7510673

Volume 6 Issue 1

Jan. 2019

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2019 > 6(1): 118-130

Xiaofeng Sun, Shuhan Shen, Hainan Cui, Lihua Hu and Zhanyi Hu, "Geographic, Geometrical and Semantic Reconstruction of Urban Scene from High Resolution Oblique Aerial Images," IEEE/CAA J. Autom. Sinica, vol. 6, no. 1, pp. 118-130, Jan. 2019. doi: 10.1109/JAS.2017.7510673

Citation:

Xiaofeng Sun, Shuhan Shen, Hainan Cui, Lihua Hu and Zhanyi Hu, "Geographic, Geometrical and Semantic Reconstruction of Urban Scene from High Resolution Oblique Aerial Images," IEEE/CAA J. Autom. Sinica, vol. 6, no. 1, pp. 118-130, Jan. 2019. doi: 10.1109/JAS.2017.7510673

Citation:

Xiaofeng Sun, Shuhan Shen, Hainan Cui, Lihua Hu and Zhanyi Hu, "Geographic, Geometrical and Semantic Reconstruction of Urban Scene from High Resolution Oblique Aerial Images," IEEE/CAA J. Autom. Sinica, vol. 6, no. 1, pp. 118-130, Jan. 2019. doi: 10.1109/JAS.2017.7510673

PDF( 73171 KB)

Geographic, Geometrical and Semantic Reconstruction of Urban Scene from High Resolution Oblique Aerial Images

doi: 10.1109/JAS.2017.7510673

Xiaofeng Sun^{1, 2
,},
Shuhan Shen^{1, 2
,
,},
Hainan Cui^{1, 2
,},
Lihua Hu^{1, 3
,},
Zhanyi Hu^{1, 2
,}

1.
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China
3.
School of Computer Science & Technology, Taiyuan University of Science and Technology, Taiyuan 030024, China

Funds:

the National Natural Science Foundation of China 61421004

the National Natural Science Foundation of China 61402316

the National Natural Science Foundation of China 61333015

the National Natural Science Foundation of China 61632003

Doctoral Research Fund of Taiyuan University of Science and Technology under grant 20162009

National Key Technologies R & D Program 2016YFB0502002

More Information

Author Bio:
Xiaofeng Sun received the B.S. degree from the Chongqing University in 2008, and the M.S. degree from Chinese Academy of Surveying and mapping in 2011. He is currently pursuing the Ph.D. degree with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences. His research interests include computer vision and remote sensing, with a particular interest in 3D urban scene reconstruction and object-level scene understanding. (e-mail: xiaofeng.sun@nlpr.ia.ac.cn)

Shuhan Shen received the B.S. and M.S. degrees both from Southwest Jiao Tong University in 2003 and 2006 respectively, and the Ph.D. degree from Shanghai Jiao Tong University in 2010. Since 2010, he has been with the National Laboratory of Pattern Recognition at Institute of Automation, Chinese Academy of Sciences, where he is currently an Associate Professor. His research interests include 3D computer vision, and in particular in image based 3D modeling of large scale scenes, 3D perception for intelligent robot, and 3D semantic reconstruction. (e-mail: shshen@nlpr.ia.ac.cn)

Hainan Cui received the B.S. degree from the South China University of Technology in 2011, and the Ph.D. degree from University of Chinese Academy of Sciences in 2016. Since 2016, he has been with the National Laboratory of Pattern Recognition at Institute of Automation, Chinese Academy of Sciences, where he is now an Assistant Professor. His research interests include computer vision, with a particular interest in using vast amounts of imagery to reconstruct and visualize the world in 3D. (e-mail: hncui@nlpr.ia.ac.cn)

Lihua Hu received the B.S., M.S., and Ph.D. degrees all from Taiyuan University of Science and Technology. Now she is an Associate Professor in the same university. Her research interests include 3D computer vision and image data mining. (e-mail: sxtyhlh@126.com)

Zhanyi Hu received the B.S. degree in automation from the North China University of Technology in 1985, the Ph.D. degree in computer science from the University of Liege, Belgium, in Jan. 1993. Since 1993, he has been with the Institute of Automation, Chinese Academy of Sciences. He was the organization committee co-chair of ICCV 2005, and a program co-chair of ACCV 2012. His current research interests include biology-inspired vision and large scale 3D scene reconstruction from images. (e-mail: huzy@nlpr.ia.ac.cn)
Corresponding author: Shuhan Shen, e-mail: shshen@nlpr.ia.ac.cn
Received Date: 2016-10-20
Revised Date: 2017-02-21
Accepted Date: 2017-03-15

Abstract

Abstract

An effective approach is proposed for 3D urban scene reconstruction in the form of point cloud with semantic labeling. Starting from high resolution oblique aerial images, our approach proceeds through three main stages:geographic reconstruction, geometrical reconstruction and semantic reconstruction. The absolute position and orientation of all the cameras relative to the real world are recovered in the geographic reconstruction stage. Then, in the geometrical reconstruction stage, an improved multi-view stereo matching method is employed to produce 3D dense points with color and normal information by taking into account the prior knowledge of aerial imagery. Finally the point cloud is classified into three classes (building, vegetation, and ground) by a rule-based hierarchical approach in the semantic reconstruction step. Experiments on complex urban scene show that our proposed 3-stage approach could generate reasonable reconstruction result robustly and efficiently. By comparing our final semantic reconstruction result with the manually labeled ground truth, classification accuracies from 86.75% to 93.02% are obtained.
- Oblique aerial image,
- point cloud,
- semantic labeling,
- urban reconstruction

FullText(HTML)

References(43)

References

[1]	G. Gröger, T. H. Kolbe, C. Nagel, and K. Häfele, "OGC city geography markup language (CityGML) en-coding standard, version 2. 0, " OGC 12-019, 2012.
[2]	G. Gröger and L. Plümer, "CityGML—interoperable semantic 3D city models, " ISPRS J. Photogramm. Remote Sens. , vol. 71, pp. 12-33, Jul. 2012. https://www.sciencedirect.com/science/article/abs/pii/S0924271612000779
[3]	P. Musialski, P. Wonka, D. G. Aliaga, M. Wimmer, L. van Gool, and W. Purgathofer, "A survey of urban reconstruction, " Comput. Graph. Forum, vol. 32, no. 6, pp. 146-177, May. 2013. http://www.researchgate.net/publication/259464590_A_Survey_of_Urban_Reconstruction
[4]	F. Lafarge, "Some new research directions to explore in urban reconstruction, " in Proc. Joint Urban Remote Sensing Event, Lausanne, Switzerland, 2015. https://www.researchgate.net/publication/281329593_Some_new_research_directions_to_explore_in_urban_reconstruction
[5]	J. Heinly, J. L. Schönberger, E. Dunn, and J. M. Frahm, "Reconstructing the world in six days, " in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, MA, USA, 2015. https://www.researchgate.net/publication/308867152_Reconstructing_the_world_in_six_days
[6]	J. M. Frahm, P. Fite-Georgel, D. Gallup, T. Johnson, R. Raguram, C. C. Wu, Y. H. Jen, E. Dunn, B. Clipp, S. Lazebnik, and M. Pollefeys, "Building Rome on a cloudless day, " in Proc. 11th European Conf. Computer Vision: Part IV, Heraklion, Crete, Greece, 2010, pp. 368-381. doi: 10.1007%2F978-3-642-15561-1_27
[7]	S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. M. Szeliski, "Building Rome in a day, " Commun. ACM, vol. 54, no. 10, pp. 105-112, Oct. 2011. http://www.researchgate.net/publication/221111210_Building_Rome_in_a_day?ev=prf_cit
[8]	Y. Furukawa and J. Ponce, "Accurate, dense, and robust multi-view stereopsis, " in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 2007, pp. 1-8. https://www.ncbi.nlm.nih.gov/pubmed/20558871?dopt=Abstract
[9]	S. H. Shen, "Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes, " IEEE Trans. Image Process. , vol. 22, no. 5, pp. 1901-1914, Jan. 2013. https://www.ncbi.nlm.nih.gov/pubmed/23322763
[10]	Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski, "Towards internet-scale multi-view stereo, " in Proc. IEEE Conf. Computer Vision and Pattern Recognition, San Francisco, CA, USA, 2010, pp. 1434-1441. https://www.researchgate.net/publication/221363626_Towards_Internet-scale_Multi-view_Stereo
[11]	Y. Furukawa and J. Ponce, "Accurate, dense, and robust multiview stereopsis, " IEEE Trans. Pattern Anal. Mach. Intell. , vol. 32, no. 8, pp. 1362-1376, Aug. 2010. https://www.ncbi.nlm.nih.gov/pubmed/20558871?dopt=Abstract
[12]	E. Tola, V. Lepetit, and P. Fua, "DAISY: An efficient dense descriptor applied to wide-baseline stereo, " IEEE Trans. Pattern Anal. Mach. Intell. , vol. 32, no. 5, pp. 815-830, Apr. 2010. http://med.wanfangdata.com.cn/Paper/Detail/PeriodicalPaper_PM20299707
[13]	R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision(2nd edition), Cambridge: Cambridge University Press, 2004.
[14]	Y. Egels and M. Kasser, Digital Photogrammetry. London: CRC Press, 2003.
[15]	C. C. Wu, "Towards linear-time incremental structure from motion, " in Proc. 2013 Int. Conf. 3DTV-Conf. , Seattle, WA, USA, pp. 127-134. https://www.researchgate.net/publication/261447449_Towards_Linear-Time_Incremental_Structure_from_Motion
[16]	N. Snavely, S. M. Seitz, and R. Szeliski, "Modeling the world from internet photo collections, " Int. J. Comput. Vis. , vol. 80, no. 2, pp. 189-210, Nov. 2008. doi: 10.1007/s11263-007-0107-3
[17]	R. Szeliski, Computer Vision: Algorithms and Applications. London: Springer, 2010.
[18]	A. Petzold and E. Pitz, "The historical origin of the pulfrich effect: A serendipitous astronomic observation at the border of the milky way, " Neuro-Ophthalmology, vol. 33, no. 1-2, pp. 39-46, Jul. 2009. doi: 10.1080/01658100802590829
[19]	M. J. Hannah, "Computer Matching of Areas in Stereo Images, " Ph. D. dissertation, Stanford Univ., Stanford, CA, USA, 1974. https://www.researchgate.net/publication/33896376_Computer_matching_of_areas_in_stereo_images
[20]	D. Marr and T. Poggio, "Cooperative computation of stereo disparity, " Science, vol. 194, no. 4262, pp. 283-287, Oct. 1976. doi: 10.1007%2F978-1-4684-6775-8_9
[21]	D. Scharstein and R. Szeliski, "A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, " Int. J. Comput. Vis. , vol. 47, no. 1-3, pp. 7-42, Apr. 2002. doi: 10.1023/A%3A1014573219977
[22]	S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, "A comparison and evaluation of multi-view stereo reconstruction algorithms, " in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition, New York, NY, USA, 2006. https://www.researchgate.net/publication/221364550_A_Comparison_and_Evaluation_of_Multi-View_Stereo_Reconstruction_Algorithms
[23]	C. Strecha, W. von Hansen, L. Van Gool, P. Fua, and U. Thoennessen, "On benchmarking camera calibration and multi-view stereo for high resolution imagery, " in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Anchorage, AK, USA, 2008. https://www.researchgate.net/publication/37463836_On_Benchmarking_Camera_Calibration_and_Multi-View_Stereo_for_High_Resolution_Imagery
[24]	R. Jensen, A. Dahl, G. Vogiatzis, E. Tola, and H. Aanaes, "Large scale multi-view stereopsis evaluation, " in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014, pp. 406-413. https://www.researchgate.net/publication/286108269_Large_Scale_Multi-view_Stereopsis_Evaluation
[25]	Available: http://vision.middlebury.edu/mview/eval/. Accessed on: 2016.
[26]	B. Wu, X. Sun, Q. C. Wu, M. L. Yan, H. Q. Wang, and K. Fu, "Building reconstruction from high-resolution multiview aerial imagery, " IEEE Geosci. Remote Sens. Lett. , vol. 12, no. 4, pp. 855-859, Apr. 2015. https://www.researchgate.net/publication/271388569_Building_Reconstruction_From_High-Resolution_Multiview_Aerial_Imagery
[27]	A. Ferro, D. Brunner, and L. Bruzzone, "Automatic detection and reconstruction of building radar footprints from single VHR SAR images, " IEEE Trans. Geosci. Remote Sens. , vol. 51, no. 2, pp. 935-952, Aug. 2013. https://www.researchgate.net/publication/224233846_Automatic_Detection_and_Reconstruction_of_Building_Radar_Footprints_From_Single_VHR_SAR_Images
[28]	G. Sithole, "Filtering of laser altimetry data using a slope adaptive filter, " Int. Arch. Photogramm. Remote Sens. , vol. XXXIV-3/W4, pp. 203-210, Oct. 2001.
[29]	K. Q. Zhang, S. C. Chen, D. Whitman, M. L. Shyu, J. H. Yan, and C. C. Zhang, "A progressive morphological filter for removing nonground measurements from airborne LIDAR data, " IEEE Trans. Geosci. Remote Sens. , vol. 41, no. 4, pp. 872-882, Jun. 2003. https://www.researchgate.net/publication/3203162_A_progressive_morphological_filter_for_removing_nonground_measurements_from_airborne_LIDAR_data
[30]	X. G. Lin and J. X. Zhang, "Segmentation-based filtering of airborne LiDAR point clouds by progressive densification of terrain segments, " Remote Sens. , vol. 6, no. 2, pp. 1294-1326, Feb. 2014. https://www.researchgate.net/publication/266912443_Segmentation-Based_Filtering_of_Airborne_LiDAR_Point_Clouds_by_Progressive_Densification_of_Terrain_Segments
[31]	G. Sithole and G. Vosselman, "Experimental comparison of filter algorithms for bare-Earth extraction from airborne laser scanning point clouds, " ISPRS J. Photogram. Remote Sens. , vol. 59, no. 1-2, pp. 85-101, Aug. 2004. https://www.sciencedirect.com/science/article/abs/pii/S0924271604000140
[32]	F. Rottensteiner, "Advanced methods for automated object extraction from LIDAR in urban areas, " in Proc. IEEE Int. Geoscience and Remote Sensing Symp. , Munich, Germany, 2012, pp. 5402-5405. https://www.researchgate.net/publication/261274892_Advanced_methods_for_automated_object_extraction_from_LiDAR_in_urban_areas
[33]	J. X. Zhang, X. G. Lin, and X. G. Ning, "SVM-based classification of segmented airborne LiDAR point clouds in urban areas, " Remote Sens. , vol. 5, no. 8, pp. 3749-3775, Jul. 2013.
[34]	M. Carlberg, P. R. Gao, G. Chen, and A. Zakhor, "Classifying urban landscape in aerial lidar using 3D shape analysis, " in Proc. 16th IEEE Int. Conf. Image Processing, Cairo, Egypt, 2009. https://www.researchgate.net/publication/224114520_Classifying_urban_landscape_in_aerial_LiDAR_using_3D_shape_analysis
[35]	J. Niemeyer, J. D. Wegner, C. Mallet, F. Rottensteiner, and U. Soergel, "Conditional random fields for urban scene classification with full waveform LiDAR Data, " in Proc. ISPRS Conf. Photogrammetric Image Analysis, Munich, Germany, 2011, pp. 233-244. doi: 10.1007%2F978-3-642-24393-6_20
[36]	F. Lafarge and C. Mallet, "Creating large-scale city models from 3D-point clouds: a robust approach with hybrid representation, " Int. J. Comput. Vis. , vol. 99, no. 1, pp. 69-85, Aug. 2012. doi: 10.1007/s11263-012-0517-8
[37]	Y. Verdie, F. Lafarge, and P. Alliez, "LOD generation for urban scenes, " ACM Trans. Graph. , vol. 34, no. 3, Article ID 30, Apr. 2015. https://www.researchgate.net/publication/276458701_LOD_Generation_for_Urban_Scenes
[38]	J. Y. Rau, J. P. Jhan, and Y. C. Hsu, "Analysis of oblique aerial images for land cover and point cloud classification in an urban environment, " IEEE Trans. Geosci. Remote Sens. , vol. 53, no. 3, pp. 1304-1319, Aug. 2015. https://www.researchgate.net/publication/265855971_Analysis_of_Oblique_Aerial_Images_for_Land_Cover_and_Point_Cloud_Classification_in_an_Urban_Environment
[39]	Available: http://www.photometrix.com.au/australis/, Accessed on: 2016.
[40]	H. N. Cui, S. H. Shen, W. Gao, and Z. Y. Hu, "Efficient large-scale structure from motion by fusing auxiliary imaging information, " IEEE Trans. Image Process. , vol. 24, no. 11, pp. 3561-3573, Jun. 2015. http://med.wanfangdata.com.cn/Paper/Detail/PeriodicalPaper_PM26111397
[41]	S. N. Sinha, D. Steedly, and R. Szeliski, "A multi-stage linear approach to structure from motion, " in Proc. European Conf. Computer Vision, Heraklion, Crete, Greece, 2010.
[42]	S. Arya, D. M. Mount, N. S. Netanyahu, R. Silverman, and A. Y. Wu, "An optimal algorithm for approximate nearest neighbor searching fixed dimensions, " J. ACM, vol. 45, no. 6, pp. 891-923, Nov. 1998. https://www.researchgate.net/publication/220430768_An_Optimal_Algorithm_for_Approximate_Nearest_Neighbor_Searching_Fixed_Dimensions
[43]	R. Castro, T. Lewiner, H. Lopes, G. Tavares, and A. Bordignon, "Statistical optimization of octree searches, " Comput. Graph. Forum, vol. 27, no. 6, pp. 1557-1566, Jan. 2008. https://www.researchgate.net/publication/220507053_Statistical_optimization_of_octree_searches?ev=prf_cit

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(22) / Tables(1)

Get Citation

PDF

XML

Article Metrics

Article views (1756) PDF downloads(44)

Geographic, Geometrical and Semantic Reconstruction of Urban Scene from High Resolution Oblique Aerial Images

doi: 10.1109/JAS.2017.7510673

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Export File

Citation

Format

Content