Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network

Long Sun; Zhenbing Liu; Xiyan Sun; Licheng Liu; Rushi Lan; Xiaonan Luo

doi:10.1109/JAS.2021.1004009

Volume 8 Issue 7

Jul. 2021

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2021 > 8(7): 1271-1280

Long Sun, Zhenbing Liu, Xiyan Sun, Licheng Liu, Rushi Lan and Xiaonan Luo, "Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network," IEEE/CAA J. Autom. Sinica, vol. 8, no. 7, pp. 1271-1280, July 2021. doi: 10.1109/JAS.2021.1004009

Citation:

Long Sun, Zhenbing Liu, Xiyan Sun, Licheng Liu, Rushi Lan and Xiaonan Luo, "Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network," IEEE/CAA J. Autom. Sinica, vol. 8, no. 7, pp. 1271-1280, July 2021. doi: 10.1109/JAS.2021.1004009

Citation:

PDF( 1238 KB)

Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network

doi: 10.1109/JAS.2021.1004009

Funds: This work was supported in part by the National Natural Science Foundation of China (61772149, 61866009, 61762028, U1701267, 61702169), Guangxi Science and Technology Project (2019GXNSFFA245014, ZY20198016, AD18281079, AD18216004), the Natural Science Foundation of Hunan Province (2020JJ3014), and Guangxi Colleges and Universities Key Laboratory of Intelligent Processing of Computer Images and Graphics (GIIP202001)

More Information

Author Bio:
Long Sun received the B.S. degree from Yunnan University of Finance and Economics, Kunming, China, in 2018. He is currently a Master student at the School of Computer Science and Information Security, Guilin University of Electronic Technology, China. His current research interests include image/video restoration, computational photography

Zhenbing Liu received the B.S. degree from Qufu Normal University, Shandong, China in 2003, and the M.S. and Ph.D. degrees from Huazhong University of Science and Technology, Wuhan, China in 2006 and 2010, respectively. He was a Visiting Scholar in Department of Radiology, University of Pennsylvania, Philadelphia, USA, in 2015. He is currently a Professor and Doctoral Supervisor at School of Computer and Information Security, Guilin University of Electronic Technology, Guilin, China. His main research interests include image processing, machine learning, and pattern recognition

Xiyan Sun received the M.S. degree from Guilin University of Electronic Technology, China in 2002, and the Ph.D. degree in science and engineering from National Astronomical Observatories, Chinese Academy of Sciences, China in 2006. During 2006–2014, she stayed in Guangxi Key Laboratory of Precision Navigation Technology and Application to research satellite navigation and signal processing. She is now working at the Guangxi Experiment Center of Information Science. Her research interests include remote-sensing information processing, satellite navigation and location service, and machine learning. She is the author of two books, more than 100 articles, and more than 30 inventions

Licheng Liu received the B.S. degree in information and computing science from the China University of Geosciences, Wuhan, China, in 2010, the M.S. degree in applied mathematics from Hunan University, Changsha, China, in 2012, and the Ph.D. degree in software engineering from the University of Macau, China, in 2016. He is currently an Associate Professor with the College of Electrical and Information Engineering, Hunan University, China. His research interests include image processing, sparse representation, pattern recognition, and machine learning

Rushi Lan received the B.S. and M.S. degrees from Nanjing University of Information Science and Technology, Nanjing, China in 2008 and 2011, respectively, and the Ph.D. degree from University of Macau, China in 2016. He is currently an Associate Professor in the School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China. His research interests include image classification, image denoising, and metric learning

Xiaonan Luo received the B.S. degree in computational mathematics from the Jiangxi University, Nanchang, China, in 1983, the M.S. degree in applied mathematics from Xidian University, Changsha, China, in 1985, and the Ph.D. degree in computational mathematics from the Dalian University of Technology, Dalian, China, in 1991. He was the Director of the National Engineering Research Center of Digital Life (2011–2016), Sun Yat-sen University, Guangzhou, China. He is currently a Professor with the School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China. His research interests include computer graphics, CAD, image processing, and mobile computing. He received the National Science Fund for Distinguished Young Scholars granted by the National Nature Science Foundation of China
Corresponding author: L. C. Liu is with the College of Electrical and Information Engineering, Hunan University, Changsha 410082, China (e-mail: rslan2016@163.com)
Received Date: 2020-09-04
Revised Date: 2020-11-04
Accepted Date: 2020-11-20

Available Online: 2021-01-12

Abstract

Abstract

The tradeoff between efficiency and model size of the convolutional neural network (CNN) is an essential issue for applications of CNN-based algorithms to diverse real-world tasks. Although deep learning-based methods have achieved significant improvements in image super-resolution (SR), current CNN-based techniques mainly contain massive parameters and a high computational complexity, limiting their practical applications. In this paper, we present a fast and lightweight framework, named weighted multi-scale residual network (WMRN), for a better tradeoff between SR performance and computational efficiency. With the modified residual structure, depthwise separable convolutions (DS Convs) are employed to improve convolutional operations’ efficiency. Furthermore, several weighted multi-scale residual blocks (WMRBs) are stacked to enhance the multi-scale representation capability. In the reconstruction subnetwork, a group of Conv layers are introduced to filter feature maps to reconstruct the final high-quality image. Extensive experiments were conducted to evaluate the proposed model, and the comparative results with several state-of-the-art algorithms demonstrate the effectiveness of WMRN.
- Convolutional neural network (CNN),
- lightweight framework,
- multi-scale,
- super-resolution

FullText(HTML)

References(58)

References

[1]	B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.-K. Liang, M. Levoy, and P. Milanfar, “Handheld multi-frame superresolution,” ACM Trans. Graph., vol. 38, no. 4, pp. 28:1–28:18, July. 2019. doi: 10.1145/3306346.3323024
[2]	Z. Ren, K. Qian, Z. Zhang, V. Pandit, A. Baird, and B. Schuller, “Deep scalogram representations for acoustic scene classification,” IEEE/CAA Journal of Automatica Sinica, vol. 5, no. 3, pp. 662–669, 2018. doi: 10.1109/JAS.2018.7511066
[3]	A. A. M. Muzahid, W. Wan, F. Sohel, L. Wu, and L. Hou, “Curvenet: Curvature-based multitask learning deep networks for 3D object recognition,” IEEE/CAA Journal of Automatica Sinica, vol. 8, no. 6, pp. 1194–1204, 2020.
[4]	R. Lan, Y. Zhou, Z. Liu, and X. Luo, “Prior knowledge-based probabilistic collaborative representation for visual recognition,” IEEE Trans. Cybernetics, vol. 50, no. 4, pp. 1498–1508, 2020. doi: 10.1109/TCYB.2018.2880290
[5]	X. Wang, K. C. Chan, K. Yu, C. Dong, and C. C. Loy, “Edvr: Video restoration with enhanced deformable convolutional networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA, 2019, pp. 1954–1963.
[6]	R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, “Video super-resolution via deep draft-ensemble learning,” in Proc. IEEE Int. Conf. Computer Vision, Santiago, Chile, 2015, pp. 531–539.
[7]	R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in Proc. Asian Conf. Computer Vision, Singapore, 2014, pp. 111–126.
[8]	B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 2017, pp. 1132–1140.
[9]	C. Wang, Z. Li, and J. Shi, “Lightweight image super-resolution with adaptive weighted learning network,” arXiv preprint arXiv:1904.02358, 2019.
[10]	R. Keys, “Cubic convolution interpolation for digital image processing,” IEEE Transactions on Acoustics,Speech,and Signal Processing, vol. 29, no. 6, pp. 1153–1160, 1981. doi: 10.1109/TASSP.1981.1163711
[11]	R. Timofte, R. Rothe, and L. Van Gool, “Seven ways to improve example-based single image super resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 1865–1873.
[12]	J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010. doi: 10.1109/TIP.2010.2050625
[13]	J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 5197–5206.
[14]	S. Guo, Z. Yan, K. Zhang, W. Zuo, and L. Zhang, “Toward convolutional blind denoising of real photographs,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 1712–1722.
[15]	P. Liu, H. Zhang, K. Zhang, L. Lin, and W. Zuo, “Multi-level waveletcnn for image restoration,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 2018, pp. 886–88609.
[16]	X. Xu, D. Sun, J. Pan, Y. Zhang, H. Pfister, and M.-H. Yang, “Learning to super-resolve blurry face and text images,” in Proc. IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 251–260.
[17]	D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng, “Progressive image deraining networks: A better and simpler baseline,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 3932–3941.
[18]	B. Li, F. Zhao, Z. Su, X. Liang, Y.-K. Lai, and R. L. Rosin, “Example based image colorization using locality consistent sparse representation,” IEEE Trans. Image Processing, vol. 26, no. 11, pp. 5188–5202, 2017. doi: 10.1109/TIP.2017.2732239
[19]	C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in Proc. European Conf. Computer Vision, Zurich, Switzerland, 2014, pp. 184–199.
[20]	J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2016.
[21]	J. Kim, J. Kwon Lee, and K. Mu Lee, “Deeply-recursive convolutional network for image superresolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp.1637–1645.
[22]	Y. Tai, J. Yang, and X. Liu, “Image super-resolution via deep recursive residual network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 2790–2798.
[23]	W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video superresolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 1874–1883.
[24]	R. Lan, L. Sun, Z. Liu, H. Lu, Z. Su, C. Pang, and X. Luo, “Cascading and enhanced residual networks for accurate single-image superresolution,” IEEE Trans. Cybernetics, vol. 51, no. 1, pp. 115–125, Jan. 2021. doi: 10.1109/TCYB.2019.2952710
[25]	R. Lan, L. Sun, Z. Liu, H. Lu, C. Pang, and X. Luo, “Madnet: A fast and lightweight network for single-image super resolution,” IEEE Trans. Cybernetics, vol. 51, no. 3, pp. 1443–1453, Mar. 2021. doi: 10.1109/tcyb.2020.2970104
[26]	N. Ahn, B. Kang, and K.-A. Sohn, “Fast, accurate, and lightweight super-resolution with cascading residual network,” in Proc. European Conf. Computer Vision, Munich, Germany, 2018, pp. 252–268.
[27]	T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 11057–11066.
[28]	G. Gao, D. Zhu, M. Yang, H. Lu, W. Yang, and H. Gao, “Face image super-resolution with pose via nuclear norm regularized structural orthogonal procrustes regression,” Neural Computing and Applications, vol. 32, no. 9, pp. 4361–4371, 2020. doi: 10.1007/s00521-018-3826-1
[29]	K. Zhang, S. Gu, R. Timofte, Z. Hui, X. Wang, X. Gao, D. Xiong, S. Liu, R. Gang, and N. Nan, “Aim 2019 challenge on constrained super-resolution: Methods and results,” in Proc. IEEE Int. Conf. Computer Vision Workshops, Seoul, Korea (South), 2019, pp. 3565–3574.
[30]	C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. doi: 10.1109/TPAMI.2015.2439281
[31]	W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 5835–5843.
[32]	Y. Tai, J. Yang, X. Liu, and C. Xu, “Memnet: A persistent memory network for image restoration,” in Proc. IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 4549–4557.
[33]	C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in Proc. European Conf. Computer Vision, Amsterdam, Netherlands, 2016, pp. 391–407.
[34]	M. D. Zeiler, G. W. Taylor, and R. Fergus, “Adaptive deconvolutional networks for mid and high level feature learning,” in Proc. IEEE Int. Conf. Computer Vision, Barcelona, Spain, 2011, pp. 2018–2025.
[35]	W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super-resolution with deep laplacian pyramid networks,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 41, no. 11, pp. 2599–2613, 2018.
[36]	Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image superresolution using very deep residual channel attention networks,” in Proc. European Conf. Computer Vision, Munich, Germany, 2018, pp. 286–301.
[37]	Z. Hui, X. Wang, and X. Gao, “Fast and accurate single image superresolution via information distillation network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 723–731.
[38]	Z. Wang, J. Chen, and S. C. H. Hoi, “Deep learning for image super-resolution: A survey,” CoRR, vol. abs/1902.06068, 2019. [Online]. Available: http://arxiv.org/abs/1902.06068
[39]	C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 105–114.
[40]	J. Johnson, A. Alahi, and F. Li, “Perceptual losses for real-time style transfer and super-resolution,” CoRR, vol. abs/1603.08155, 2016. [Online]. Available: http://arxiv.org/abs/1603.08155
[41]	X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in Proc. European Conf. Computer Vision Workshops, Munich, Germany, 2018, pp. 63–79.
[42]	A. Marquina and S. J. Osher, “Image super-resolution by tvregularization and bregman iteration,” Journal of Scientific Computing, vol. 37, no. 3, pp. 367–382, 2008. doi: 10.1007/s10915-008-9214-8
[43]	C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 2818–2826.
[44]	C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in Proc. AAAI Conf. Artificial Intelligence, San Francisco, California, USA, 2017.
[45]	M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition), Salt Lake City, UT, USA, 2018, pp. 4510–4520.
[46]	K. Hung, Z. Zhang, and J. Jiang, “Real-time image super-resolution using recursive depthwise separable convolution network,” IEEE Access, vol. 7, pp. 99 804–99 816, 2019. doi: 10.1109/ACCESS.2019.2929223
[47]	S. Liu, D. Huang, and Y. Wang, “Receptive field block net for accurate and fast object detection,” in Proc. European Conf. Computer Vision, Munich, Germany, 2018, pp. 404–419.
[48]	S. Gao, M. Cheng, K. Zhao, X. Zhang, M. Yang, and P. H. S. Torr, “Res2net: A new multi-scale backbone architecture,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 2, pp. 652–662, Feb. 2021. doi: 10.1109/TPAMI.2019.2938758
[49]	F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in Proc. Int. Conf. Learn. Representations, San Juan, Puerto Rico, 2016.
[50]	W. Ren, S. Liu, L. Ma, Q. Xu, X. Xu, X. Cao, J. Du, and M. Yang, “Lowlight image enhancement via a deep hybrid network,” IEEE Trans. Image Processing, vol. 28, no. 9, pp. 4364–4375, Sep. 2019. doi: 10.1109/TIP.2019.2910412
[51]	D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in Proc. Int. Conf. Learn. Representations, San Diego, CA, USA, 2015.
[52]	R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA, 2017, pp. 1110–1121.
[53]	M. Bevilacqua, A. Roumy, C. Guillemot, and M. line Alberi Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” in Proc. British Machine Vision Conf. BMVA Press, Surrey, Australia, 2012, pp. 135.1–135.10.
[54]	R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in Curves and Surfaces, J.-D. Boissonnat, P. Chenin, A. Cohen, C. Gout, T. Lyche, M.-L. Mazure, and L. Schumaker, Eds. Berlin Heidelberg, Germany: Springer, 2012, pp. 711–730.
[55]	P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898–916, May 2011. doi: 10.1109/TPAMI.2010.161
[56]	Y. Matsui, K. Ito, Y. Aramaki, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using manga109 dataset,” CoRR, vol. abs/1510.04389, 2015. [Online]. Available: http://arxiv.org/abs/1510.04389
[57]	Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Processing, vol. 13, no. 4, pp. 600–612, April. 2004. doi: 10.1109/TIP.2003.819861
[58]	K. Zhang, W. Zuo, and L. Zhang, “Learning a single convolutional superresolution network for multiple degradations,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 3262–3271.

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(6) / Tables(3)

Get Citation

PDF

XML

Article Metrics

Article views (1934) PDF downloads(128)

Highlights

A novel weighted multi-scale residual block (WMRB) is proposed, which can not only effectively exploit multi-scale features but also dramatically reduce the computational burden.
A global residual shortcut is deployed, which adds high frequency features to generate more clear details and promote gradient information propagation.
Extensive experiments show that the WMRN model utilizes only a modest number of parameters and operations to achieve competitive SR performance on different benchmarks with different upscaling factors.

Lightweight Image Super-Resolution via Weighted Multi-Scale Residual Network

doi: 10.1109/JAS.2021.1004009

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content