A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search
Volume 12 Issue 2
Feb.  2025

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 19.2, Top 1 (SCI Q1)
    CiteScore: 28.2, Top 1% (Q1)
    Google Scholar h5-index: 95， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2025  >  12(2): 403-413
K. Xiong, Q. Wei, and  H. Li,  “Residential energy scheduling with solar energy based on Dyna adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 2, pp. 403–413, Feb. 2025. doi: 10.1109/JAS.2024.124809
Citation: K. Xiong, Q. Wei, and  H. Li,  “Residential energy scheduling with solar energy based on Dyna adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 12, no. 2, pp. 403–413, Feb. 2025. doi: 10.1109/JAS.2024.124809
shu

Residential Energy Scheduling With Solar Energy Based on Dyna Adaptive Dynamic Programming

doi: 10.1109/JAS.2024.124809
Funds:  This work was supported in part by the National Key Research and Development Program of China (2024YFB4709100, 2021YFE0206100), the National Natural Science Foundation of China (62073321), the National Defense Basic Scientific Research Program (JCKY2019203C029), and the Science and Technology Development Fund, Macau SAR, China (0015/2020/AMJ)
More Information
    Abstract
  • Learning-based methods have become mainstream for solving residential energy scheduling problems. In order to improve the learning efficiency of existing methods and increase the utilization of renewable energy, we propose the Dyna action-dependent heuristic dynamic programming (Dyna-ADHDP) method, which incorporates the ideas of learning and planning from the Dyna framework in action-dependent heuristic dynamic programming. This method defines a continuous action space for precise control of an energy storage system and allows online optimization of algorithm performance during the real-time operation of the residential energy model. Meanwhile, the target network is introduced during the training process to make the training smoother and more efficient. We conducted experimental comparisons with the benchmark method using simulated and real data to verify its applicability and performance. The results confirm the method’s excellent performance and generalization capabilities, as well as its excellence in increasing renewable energy utilization and extending equipment life.

     

    • FullText(HTML)
  • loading
    • References(31)
  • [1]
    H. H. Aly, “A proposed intelligent short-term load forecasting hybrid models of ANN, wnn and KF based on clustering techniques for smart grid,” Electric Power Systems Research, vol. 182, p. 106191, 2020. doi: 10.1016/j.jpgr.2019.106191
    [2]
    T. Liu and T. Shu, “On the security of ANN-based AC state estimation in smart grid,” Computers and Security, vol. 105, p. 102265, 2021.
    [3]
    Z. Cabrane, J. Kim, K. Yoo, and S. H. Lee, “Fuzzy logic supervisor-based novel energy management strategy reflecting different virtual power plants,” Electric Power Systems Research, vol. 205, p. 107731, 2022. doi: 10.1016/j.jpgr.2021.107731
    [4]
    A. Alnasser and H. Sun, “A fuzzy logic trust model for secure routing in smart grid networks,” IEEE Access, vol. 5, pp. 17896–17903, 2017. doi: 10.1109/ACCESS.2017.2740219
    [5]
    G. Hafeez, K. S. Alimgeer, and I. Khan, “Electric load forecasting based on deep learning and optimized by heuristic algorithm in smart grid,” Applied Energy, vol. 269, p. 114915, 2020. doi: 10.1016/j.apenergy.2020.114915
    [6]
    U. Asgher, M. Babar Rasheed, A. S. Al-Sumaiti, A. Ur-Rahman, I. Ali, A. Alzaidi, and A. Alamri, “Smart energy optimization using heuristic algorithm in smart grid with integration of solar energy sources,” Energies, vol. 11, no. 12, p. 3494, 2018. doi: 10.3390/en11123494
    [7]
    L. Yin, S. Luo, and C. Ma, “Expandable depth and width adaptive dynamic programming for economic smart generation control of smart grids,” Energy, vol. 232, p. 120964, 2021. doi: 10.1016/j.energy.2021.120964
    [8]
    R. Yu, W. Zhong, S. Xie, Y. Zhang, and Y. Zhang, “QoS differential scheduling in cognitive-radio-based smart grid networks: An adaptive dynamic programming approach,” IEEE Trans. Neural Networks and Learning Systems, vol. 27, p. 2, 2016. doi: 10.1109/TNNLS.2016.2522803
    [9]
    S. Singh, Q. Z. Sheng, E. Benkhelifa, and J. Lloret, “Guest editorial: Energy management, protocols, and security for the next-generation networks and internet of things,” IEEE Trans. Industrial Informatics, vol. 16, no. 5, pp. 3515–3520, 2020. doi: 10.1109/TII.2020.2964591
    [10]
    A. Keshtkar and S. Arzanpour, “An adaptive fuzzy logic system for residential energy management in smart grid environments,” Applied Energy, vol. 186, pp. 68–81, 2017. doi: 10.1016/j.apenergy.2016.11.028
    [11]
    M. A. Hossain, H. R. Pota, S. Squartini, and A. F. Abdou, “Modified PSO algorithm for real-time energy management in grid-connected microgrids,” Renewable Energy, vol. 136, pp. 746–757, 2019. doi: 10.1016/j.renene.2019.01.005
    [12]
    M. A. Mohamed, A. M. Eltamaly, and A. I. Alolah, “PSO-based smart grid application for sizing and optimization of hybrid renewable energy systems,” PloS One, vol. 11, p. 8, 2016.
    [13]
    M. Ahrarinouri, M. Rastegar, and A. R. Seifi, “Multiagent reinforcement learning for energy management in residential buildings,” IEEE Trans. Industrial Informatics, vol. 17, no. 1, pp. 659–666, 2021. doi: 10.1109/TII.2020.2977104
    [14]
    L. Yu, W. Xie, D. Xie, Y. Zou, D. Zhang, Z. Sun, L. Zhang, Y. Zhang, and T. Jiang, “Deep reinforcement learning for smart home energy management,” IEEE Internet of Things J., vol. 7, no. 4, pp. 2751–2762, 2019.
    [15]
    Q. Wei, G. Shi, R. Song, and Y. Liu, “Adaptive dynamic programming-based optimal control scheme for energy storage systems with solar renewable energy,” IEEE Trans. Industrial Electronics, vol. 64, no. 7, pp. 5468–5478, 2017. doi: 10.1109/TIE.2017.2674581
    [16]
    Q. Wei, D. Liu, Y. Liu, and R. Song, “Optimal constrained self-learning battery sequential management in microgrid via adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 2, pp. 168–176, 2017.
    [17]
    Q. Wei, Z. Liao, and G. Shi, “Generalized actor-critic learning optimal control in smart home energy management,” IEEE Trans. Industrial Informatics, vol. 17, no. 10, pp. 6614–6623, 2020.
    [18]
    M. Boaro, D. Fuselli, F. D. Angelis, D. Liu, Q. Wei, and F. Piazza, “Adaptive dynamic programming algorithm for renewable energy scheduling and battery management,” Cognitive Computation, vol. 5, no. 2, pp. 264–277, 2013. doi: 10.1007/s12559-012-9191-y
    [19]
    J. Yuan, S. Y. Samson, G. Zhang, C. P. Lim, H. Trinh, and Y. Zhang, “Design and hil realization of an online adaptive dynamic programming approach for real-time economic operations of household energy systems,” IEEE Trans. Smart Grid, vol. 13, no. 1, pp. 330–341, 2021.
    [20]
    Y. Zhu, D. Zhao, X. Li, and D. Wang, “Control-limited adaptive dynamic programming for multi-battery energy storage systems,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 4235–4244, 2018.
    [21]
    F. Wei, Z. Wan, and H. He, “Cyber-attack recovery strategy for smart grid based on deep reinforcement learning,” IEEE Trans. Smart Grid, vol. 11, no. 3, pp. 2476–2486, 2019.
    [22]
    R. Manojkumar, C. Kumar, and S. Ganguly, “Optimal demand response in a residential pv storage system using energy pricing limits,” IEEE Trans. Industrial Informatics, vol. 18, no. 4, pp. 2497–2507, 2021.
    [23]
    M. B. Gough, S. F. Santos, T. AlSkaif, M. S. Javadi, R. Castro, and J. P. Catalão, “Preserving privacy of smart meter data in a smart grid environment,” IEEE Trans. Industrial Informatics, vol. 18, no. 1, pp. 707–718, 2021.
    [24]
    D. Liu, Y. Xu, Q. Wei, and X. Liu, “Residential energy scheduling for variable weather solar energy based on adaptive dynamic programming,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 36–46, 2018.
    [25]
    H. Saber, M. Ehsan, M. Moeini-Aghtaie, H. Ranjbar, and M. Lehtonen, “A user-friendly transactive coordination model for residential prosumers considering voltage unbalance in distribution networks,” IEEE Trans. Industrial Informatics, vol.18, no. 9, pp. 5748–5759, 2022.
    [26]
    Z. Li, Y. Xu, X. Feng, and Q. Wu, “Optimal stochastic deployment of heterogeneous energy storage in a residential multienergy microgrid with demand-side management,” IEEE Trans. Industrial Informatics, vol. 17, no. 2, pp. 991–1004, 2020.
    [27]
    R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM Sigart Bulletin, vol. 2, no. 4, pp. 160–163, 1991. doi: 10.1145/122344.122377
    [28]
    T. Yau, L. N. Walker, H. L. Graham, A. Gupta, and R. Raithel, “Effects of battery storage devices on power system dispatch,” IEEE Trans. Power Apparatus and Systems, vol. 1, pp. 375–383, 1981.
    [29]
    X. Fang, J. Wang, G. Song, Y. Han, Q. Zhao, and Z. Cao, “Multi-agent reinforcement learning approach for residential microgrid energy scheduling,” Energies, vol. 13, no. 1, p. 123, 2019. doi: 10.3390/en13010123
    [30]
    Q. Wei, Z. Liao, R. Song, P. Zhang, Z. Wang, and J. Xiao, “Self-learning optimal control for ice-storage air conditioning systems via data-based adaptive dynamic programming,” IEEE Trans. Industrial Electronics, vol. 68, no. 4, pp. 3599–3608, 2020.
    [31]
    T. Huang and D. Liu, “A self-learning scheme for residential energy system control and management,” Neural Computing and Applications, vol. 22, no. 2, pp. 259–269, 2013. doi: 10.1007/s00521-011-0711-6
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(14)  / Tables(2)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (385) PDF downloads(46) Cited by()

    Highlights

    • An improved ADHDP method that incorporates learning and planning ideas
    • Dyna architecture to enhance learning efficiency and generalization performance
    • Improved utility function enhances renewable energy utilization
    • Rule system guarantees the feasibility of executing actions

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return