Residential Energy Scheduling With Solar Energy Based on Dyna Adaptive Dynamic Programming

Kang Xiong; Qinglai Wei; Hongyang Li

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2025 > In Press, Accepted Manuscript

K. Xiong, Q. Wei, and H. Li, “Residential energy scheduling with solar energy based on dyna adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 2, pp. 1–11, Feb. 2025.

Citation:

K. Xiong, Q. Wei, and H. Li, “Residential energy scheduling with solar energy based on dyna adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 2, pp. 1–11, Feb. 2025.

K. Xiong, Q. Wei, and H. Li, “Residential energy scheduling with solar energy based on dyna adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 2, pp. 1–11, Feb. 2025.

Citation:

K. Xiong, Q. Wei, and H. Li, “Residential energy scheduling with solar energy based on dyna adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 2, pp. 1–11, Feb. 2025.

PDF( 4671 KB)

Residential Energy Scheduling With Solar Energy Based on Dyna Adaptive Dynamic Programming

Funds: This work was supported in part by the National Key Research and Development Program of China (2021YFE0206100, 2018YFB1702300), the National Natural Science Foundation of China (62073321), the National Defense Basic Scientific Research Program (JCKY2019203C029), and the Science and Technology Development Fund, Macau SAR, China (0015/2020/AMJ)

More Information

Author Bio:
Kang Xiong received the master degree in computer science and technology from Dalian University in 2021. He is currently pursuing the Ph.D. degree in intelligent science and systems at the Institute of Systems Engineering, Macau University of Science and Technology, Macau, China. His research interests include adaptive dynamic programming, reinforcement learning, and their applications to microgrid energy management

Qinglai Wei (Senior Member, IEEE) received the B.S. degree in automation, and the Ph.D. degree in control theory and control engineering, from Northeastern University, in 2002 and 2009, respectively. From 2009–2011, he was a Postdoctoral Fellow with the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences. He is currently a Professor of this institute. He has authored three books, and published over 70 international journal papers. His research interests include adaptive dynamic programming, neural-networks-based control, optimal control, and nonlinear systems and their industrial applications.Dr. Wei is an Associate Editor of IEEE Transactions on Automation Science and Engineering since 2017, IEEE Transactions on Consumer Electronics since 2017, IEEE Transactions on Cognitive and Developmental Systems, IEEE Transaction on Systems Man, and Cybernetics: Systems since 2016, Information Sciences since 2016, Neurocomputing since 2016, Optimal Control Applications and Methods since 2016, Acta Automatica Sinica since 2015, and has been holding the same position for IEEE Transactions on Neural Networks and Learning Systems during 2014–2015 and since 2019. He is the Secretary of IEEE Computational Intelligence Society (CIS) Beijing Chapter since 2015. He was Registration Chair of the 12th World Congress on Intelligent Control and Automation (WCICA2016), 2014 IEEE World Congress on Computational Intelligence (WCCI2014), the 2013 International Conference on Brain Inspired Cognitive Systems (BICS 2013), and the Eighth International Symposium on Neural Networks (ISNN 2011). He was the Publication Chair of 5th International Conference on Information Science and Technology (ICIST2015) and the Ninth International Symposium on Neural Networks (ISNN 2012). He was the Finance Chair of the 4th International Conference on Intelligent Control and Information Processing (ICICIP 2013) and the Publicity Chair of the 2012 International Conference on Brain Inspired Cognitive Systems (BICS 2012). He was guest editors for several international journals. He was a recipient of Shuang-Chuang Talents in Jiangsu Province, China, in 2014. He was a recipient of the Outstanding Paper Award of Acta Automatica Sinica in 2011 and Zhang Siying Outstanding Paper Award of Chinese Control and Decision Conference (CCDC) in 2015. He was a recipient of Young Researcher Award of Asia Pacific Neural Network Society (APNNS) in 2016

Hongyang Li received the Ph.D. degree in computer application technology from the University of Chinese Academy of Sciences in 2022. He is currently working as an Assistant Professor with the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences. His research interests include adaptive dynamic programming, reinforcement learning, and optimal control and their industrial applications
Corresponding author: Kang Xiong, e-mail: 2109853wmi30001@student.must.edu.mo
Received Date: 2024-06-28
Accepted Date: 2024-07-18

Available Online: 2024-11-15

Abstract

Abstract

Learning-based methods have become mainstream for solving residential energy scheduling problems. In order to improve the learning efficiency of existing methods and increase the utilization of renewable energy, we propose the Dyna action-dependent heuristic dynamic programming (Dyna-ADHDP) method, which incorporates the ideas of learning and planning from the Dyna framework in action-dependent heuristic dynamic programming. This method defines a continuous action space for precise control of an energy storage system and allows online optimization of algorithm performance during the real-time operation of the residential energy model. Meanwhile, the target network is introduced during the training process to make the training smoother and more efficient. We conducted experimental comparisons with the benchmark method using simulated and real data to verify its applicability and performance. The results confirm the method’s excellent performance and generalization capabilities, as well as its excellence in increasing renewable energy utilization and extending equipment life.
- Adaptive dynamic programming (ADP),
- dynamic residential scenarios,
- optimal residential energy management,
- smart grid

FullText(HTML)

References(31)

References

[1]	H. H. Aly, “A proposed intelligent short-term load forecasting hybrid models of ANN, wnn and KF based on clustering techniques for smart grid,” Electric Power Systems Research, vol. 182, p. 106191, 2020. doi: 10.1016/j.jpgr.2019.106191
[2]	T. Liu and T. Shu, “On the security of ANN-based AC state estimation in smart grid,” Computers and Security, vol. 105, p. 102265, 2021.
[3]	Z. Cabrane, J. Kim, K. Yoo, and S. H. Lee, “Fuzzy logic supervisor-based novel energy management strategy reflecting different virtual power plants,” Electric Power Systems Research, vol. 205, p. 107731, 2022. doi: 10.1016/j.jpgr.2021.107731
[4]	A. Alnasser and H. Sun, “A fuzzy logic trust model for secure routing in smart grid networks,” IEEE Access, vol. 5, pp. 17896–17903, 2017. doi: 10.1109/ACCESS.2017.2740219
[5]	G. Hafeez, K. S. Alimgeer, and I. Khan, “Electric load forecasting based on deep learning and optimized by heuristic algorithm in smart grid,” Applied Energy, vol. 269, p. 114915, 2020. doi: 10.1016/j.apenergy.2020.114915
[6]	U. Asgher, M. Babar Rasheed, A. S. Al-Sumaiti, A. Ur-Rahman, I. Ali, A. Alzaidi, and A. Alamri, “Smart energy optimization using heuristic algorithm in smart grid with integration of solar energy sources,” Energies, vol. 11, no. 12, p. 3494, 2018. doi: 10.3390/en11123494
[7]	L. Yin, S. Luo, and C. Ma, “Expandable depth and width adaptive dynamic programming for economic smart generation control of smart grids,” Energy, vol. 232, p. 120964, 2021. doi: 10.1016/j.energy.2021.120964
[8]	R. Yu, W. Zhong, S. Xie, Y. Zhang, and Y. Zhang, “QoS differential scheduling in cognitive-radio-based smart grid networks: An adaptive dynamic programming approach,” IEEE Trans. Neural Networks and Learning Systems, vol. 27, p. 2, 2016. doi: 10.1109/TNNLS.2016.2522803
[9]	S. Singh, Q. Z. Sheng, E. Benkhelifa, and J. Lloret, “Guest editorial: Energy management, protocols, and security for the next-generation networks and internet of things,” IEEE Trans. Industrial Informatics, vol. 16, no. 5, pp. 3515–3520, 2020. doi: 10.1109/TII.2020.2964591
[10]	A. Keshtkar and S. Arzanpour, “An adaptive fuzzy logic system for residential energy management in smart grid environments,” Applied Energy, vol. 186, pp. 68–81, 2017. doi: 10.1016/j.apenergy.2016.11.028
[11]	M. A. Hossain, H. R. Pota, S. Squartini, and A. F. Abdou, “Modified PSO algorithm for real-time energy management in grid-connected microgrids,” Renewable Energy, vol. 136, pp. 746–757, 2019. doi: 10.1016/j.renene.2019.01.005
[12]	M. A. Mohamed, A. M. Eltamaly, and A. I. Alolah, “PSO-based smart grid application for sizing and optimization of hybrid renewable energy systems,” PloS One, vol. 11, p. 8, 2016.
[13]	M. Ahrarinouri, M. Rastegar, and A. R. Seifi, “Multiagent reinforcement learning for energy management in residential buildings,” IEEE Trans. Industrial Informatics, vol. 17, no. 1, pp. 659–666, 2021. doi: 10.1109/TII.2020.2977104
[14]	L. Yu, W. Xie, D. Xie, Y. Zou, D. Zhang, Z. Sun, L. Zhang, Y. Zhang, and T. Jiang, “Deep reinforcement learning for smart home energy management,” IEEE Internet of Things J., vol. 7, no. 4, pp. 2751–2762, 2019.
[15]	Q. Wei, G. Shi, R. Song, and Y. Liu, “Adaptive dynamic programming-based optimal control scheme for energy storage systems with solar renewable energy,” IEEE Trans. Industrial Electronics, vol. 64, no. 7, pp. 5468–5478, 2017. doi: 10.1109/TIE.2017.2674581
[16]	Q. Wei, D. Liu, Y. Liu, and R. Song, “Optimal constrained self-learning battery sequential management in microgrid via adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 2, pp. 168–176, 2016.
[17]	Q. Wei, Z. Liao, and G. Shi, “Generalized actor-critic learning optimal control in smart home energy management,” IEEE Trans. Industrial Informatics, vol. 17, no. 10, pp. 6614–6623, 2020.
[18]	M. Boaro, D. Fuselli, F. D. Angelis, D. Liu, Q. Wei, and F. Piazza, “Adaptive dynamic programming algorithm for renewable energy scheduling and battery management,” Cognitive Computation, vol. 5, no. 2, pp. 264–277, 2013. doi: 10.1007/s12559-012-9191-y
[19]	J. Yuan, S. Y. Samson, G. Zhang, C. P. Lim, H. Trinh, and Y. Zhang, “Design and hil realization of an online adaptive dynamic programming approach for real-time economic operations of household energy systems,” IEEE Trans. Smart Grid, vol. 13, no. 1, pp. 330–341, 2021.
[20]	Y. Zhu, D. Zhao, X. Li, and D. Wang, “Control-limited adaptive dynamic programming for multi-battery energy storage systems,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 4235–4244, 2018.
[21]	F. Wei, Z. Wan, and H. He, “Cyber-attack recovery strategy for smart grid based on deep reinforcement learning,” IEEE Trans. Smart Grid, vol. 11, no. 3, pp. 2476–2486, 2019.
[22]	R. Manojkumar, C. Kumar, and S. Ganguly, “Optimal demand response in a residential pv storage system using energy pricing limits,” IEEE Trans. Industrial Informatics, vol. 18, no. 4, pp. 2497–2507, 2021.
[23]	M. B. Gough, S. F. Santos, T. AlSkaif, M. S. Javadi, R. Castro, and J. P. Catalão, “Preserving privacy of smart meter data in a smart grid environment,” IEEE Trans. Industrial Informatics, vol. 18, no. 1, pp. 707–718, 2021.
[24]	D. Liu, Y. Xu, Q. Wei, and X. Liu, “Residential energy scheduling for variable weather solar energy based on adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 36–46, 2017.
[25]	H. Saber, M. Ehsan, M. Moeini-Aghtaie, H. Ranjbar, and M. Lehtonen, “A user-friendly transactive coordination model for residential prosumers considering voltage unbalance in distribution networks,” IEEE Trans. Industrial Informatics, 2022.
[26]	Z. Li, Y. Xu, X. Feng, and Q. Wu, “Optimal stochastic deployment of heterogeneous energy storage in a residential multienergy microgrid with demand-side management,” IEEE Trans. Industrial Informatics, vol. 17, no. 2, pp. 991–1004, 2020.
[27]	R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM Sigart Bulletin, vol. 2, no. 4, pp. 160–163, 1991. doi: 10.1145/122344.122377
[28]	T. Yau, L. N. Walker, H. L. Graham, A. Gupta, and R. Raithel, “Effects of battery storage devices on power system dispatch,” IEEE Trans. Power Apparatus and Systems, vol. 1, pp. 375–383, 1981.
[29]	X. Fang, J. Wang, G. Song, Y. Han, Q. Zhao, and Z. Cao, “Multi-agent reinforcement learning approach for residential microgrid energy scheduling,” Energies, vol. 13, no. 1, p. 123, 2019. doi: 10.3390/en13010123
[30]	Q. Wei, Z. Liao, R. Song, P. Zhang, Z. Wang, and J. Xiao, “Self-learning optimal control for ice-storage air conditioning systems via data-based adaptive dynamic programming,” IEEE Trans. Industrial Electronics, vol. 68, no. 4, pp. 3599–3608, 2020.
[31]	T. Huang and D. Liu, “A self-learning scheme for residential energy system control and management,” Neural Computing and Applications, vol. 22, no. 2, pp. 259–269, 2013. doi: 10.1007/s00521-011-0711-6

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(14) / Tables(2)

Get Citation

PDF

XML

Article Metrics

Article views (11) PDF downloads(1)

Residential Energy Scheduling With Solar Energy Based on Dyna Adaptive Dynamic Programming

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Export File

Citation

Format

Content