Price-Based Residential Demand Response Management in Smart Grids: A Reinforcement Learning-Based Approach

Yanni Wan; Jiahu Qin; Xinghuo Yu; Tao Yang; Yu Kang

doi:10.1109/JAS.2021.1004287

Volume 9 Issue 1

Jan. 2022

IEEE/CAA Journal of Automatica Sinica

JCR Impact Factor: 15.3, Top 1 (SCI Q1)

CiteScore: 23.5, Top 2% (Q1)
Google Scholar h5-index: 77， TOP 5

Turn off MathJax

Article Contents

Article Navigation > IEEE/CAA Journal of Automatica Sinica > 2022 > 9(1): 123-134

Y. N. Wan, J. H. Qin, X. H. Yu, T. Yang, and Y. Kang, “Price-based residential demand response management in smart grids: A reinforcement learning-based approach,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 1, pp. 123–134, Jan. 2022. doi: 10.1109/JAS.2021.1004287

Citation:

Y. N. Wan, J. H. Qin, X. H. Yu, T. Yang, and Y. Kang, “Price-based residential demand response management in smart grids: A reinforcement learning-based approach,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 1, pp. 123–134, Jan. 2022. doi: 10.1109/JAS.2021.1004287

Citation:

PDF( 1919 KB)

Price-Based Residential Demand Response Management in Smart Grids: A Reinforcement Learning-Based Approach

doi: 10.1109/JAS.2021.1004287

Yanni Wan^1
,,
Jiahu Qin^{1, 2
,
,},
Xinghuo Yu^3
,,
Tao Yang^4
,,
Yu Kang^{5, 6
,}

1.
Department of Automation, University of Science and Technology of China, Hefei 230027, China
2.
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei 230088, China
3.
School of Engineering, RMIT University, VIC 3000, Australia
4.
State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang 110819, China
5.
Department of Automation, State Key Laboratory of Fire Science, Institute of Advanced Technology, University of Science and Technology of China, Hefei 230027, China
6.
Key Laboratory of Technology in Geo-Spatial Information Processing and Application Systems, Chinese Academy of Sciences, Beijing 100190, China

Funds: This work was supported in part by the National Natural Science Foundation of China (61922076, 61725304, 61873252, 61991403, 61991400) and in part by the Australian Research Council Discovery Program (DP200101199)

More Information

Author Bio:
Yanni Wan (Student Member, IEEE) received the B.E. degree in automation from the Ocean University of China in 2016. She is currently pursuing the Ph.D. degree in control science and engineering with the University of Science and Technology of China. Her current research interests include distributed optimization in multi-agent systems, distributed energy management and demand response in smart grid

Jiahu Qin (Senior Member, IEEE) received the first Ph.D. degree in control science and engineering from the Harbin Institute of Technology in 2012, and the second Ph.D. degree in systems and control from the Australian National University, Australia, in 2014. He is currently a Professor with the Department of Automation, University of Science and Technology of China. His current research interests include multiagent systems, cyber-physical systems, and complex dynamical networks

Xinghuo Yu (Fellow, IEEE) received the B.Eng. and M.Eng. degrees in electrical and electronic engineering from the University of Science and Technology of China, in 1982 and 1984, and the Ph.D. degree in control science and engineering from Southeast University in 1988, respectively. He is currently an Associate Deputy Vice-Chancellor and a Distinguished Professor with RMIT University, Australia. His research interests include control systems, complex and intelligent systems, and smart energy systems. Dr. Yu received a number of awards and honors for his contributions, including the 2013 Dr.-Ing. Eugene Mittelmann Achievement Award of the IEEE Industrial Electronics Society, the 2018 M. A. Sargent Medal from Engineers Australia, and the 2018 Australasian AI Distinguished Research Contribution Award from the Australian Computer Society. He is also the Junior Past President of the IEEE Industrial Electronics Society for 2020 and 2021. He has served as an Associate Editor for IEEE Transactions on Automatic Control, IEEE Transactions on Circuits and Systems I: Regular Papers, IEEE Transactions on Industrial Electronics, and IEEE Transactions on Industrial Informatics

Tao Yang (Senior Member, IEEE) received the Ph.D. degree in electrical engineering from Washington State University, USA, in 2012. From 2012 to 2014, he was an ACCESS Post-Doctoral Researcher with the ACCESS Linnaeus Centre, Royal Institute of Technology, Sweden. He then joined the Pacific Northwest National Laboratory as a postdoc, and was promoted to Scientist/Engineer II in 2015. From 2016 to 2019, he was an Assistant Professor at the Department of Electrical Engineering, University of North Texas, USA. He is currently a Professor at the State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University. His research interests include distributed control and optimization with applications to power systems, cyber-physical systems, networked control systems, and multi-agent systems. He received Ralph E. Powe Junior Faculty Enhancement Award (2018) and Best Student Paper award (as an Advisor) of the 14th IEEE International Conference on Control & Automation

Yu Kang (Senior Member, IEEE) received the Dr. Eng. degree in control theory and control engineering from the University of Science and Technology of China in 2005. From 2005 to 2007, he was a Post-Doctoral Fellow with the Academy of Mathematics and Systems Science, Chinese Academy of Sciences. He is currently a Professor with the Department of Automation, University of Science and Technology of China. His current research interests include adaptive/robust control, variable structure control, mobile manipulator, and Markovian jump systems
Corresponding author: Jiahu Qin, e-mail: jhqin@ustc.edu.cn
¹ Since the energy demand and consumption of loads is affected by many factors, the state transition is rather difficult to obtain. Therefore, we next employ a model-free Q-learning method to solve the dynamic retail pricing problem.
Received Date: 2021-03-21
Revised Date: 2021-04-23
Accepted Date: 2021-05-17

Available Online: 2021-07-07

Abstract

Abstract

This paper studies price-based residential demand response management (PB-RDRM) in smart grids, in which non-dispatchable and dispatchable loads (including general loads and plug-in electric vehicles (PEVs)) are both involved. The PB-RDRM is composed of a bi-level optimization problem, in which the upper-level dynamic retail pricing problem aims to maximize the profit of a utility company (UC) by selecting optimal retail prices (RPs), while the lower-level demand response (DR) problem expects to minimize the comprehensive cost of loads by coordinating their energy consumption behavior. The challenges here are mainly two-fold: 1) the uncertainty of energy consumption and RPs; 2) the flexible PEVs’ temporally coupled constraints, which make it impossible to directly develop a model-based optimization algorithm to solve the PB-RDRM. To address these challenges, we first model the dynamic retail pricing problem as a Markovian decision process (MDP), and then employ a model-free reinforcement learning (RL) algorithm to learn the optimal dynamic RPs of UC according to the loads’ responses. Our proposed RL-based DR algorithm is benchmarked against two model-based optimization approaches (i.e., distributed dual decomposition-based (DDB) method and distributed primal-dual interior (PDI)-based method), which require exact load and electricity price models. The comparison results show that, compared with the benchmark solutions, our proposed algorithm can not only adaptively decide the RPs through on-line learning processes, but also achieve larger social welfare within an unknown electricity market environment.
- Demand response management (DRM),
- Markovian decision process (MDP),
- Monte Carlo simulation,
- reinforcement learning (RL),
- smart grid

FullText(HTML)

¹ Since the energy demand and consumption of loads is affected by many factors, the state transition is rather difficult to obtain. Therefore, we next employ a model-free Q-learning method to solve the dynamic retail pricing problem.

References(38)

References

[1]	X. Fang, S. Misra, G. Xue, and D. Yang, “Smart grid–The new and improved power grid: A survey,” IEEE Commun. Surveys Tuts., vol. 14, no. 4, pp. 944–980, Apr. 2012. doi: 10.1109/SURV.2011.101911.00087
[2]	X. Yu and Y. Xue, “Smart grids: A cyber-physical systems perspective,” Proceedings of the IEEE, vol. 104, no. 5, pp. 1058–1070, May 2016. doi: 10.1109/JPROC.2015.2503119
[3]	R. Deng, Z. Yang, M. Chow, and J. Chen, “A survey on demand response in smart grids: Mathematical models and approaches,” IEEE Trans. Ind. Informat., vol. 11, no. 3, pp. 570–582, Jun. 2015. doi: 10.1109/TII.2015.2414719
[4]	L. Ding, L. Y. Wang, G. Y. Yin, W. X. Zheng, and Q. Han, “Distributed energy management for smart grids with an event-triggered communication scheme,” IEEE Trans. Control Syst. Technol., vol. 27, no. 5, pp. 1950–1961, Sept. 2019. doi: 10.1109/TCST.2018.2842208
[5]	H. Hao, C. D. Corbin, K. Kalsi, and R. G. Pratt, “Transactive control of commercial buildings for demand response,” IEEE Trans. Power Syst., vol. 32, no. 1, pp. 774–783, Jan. 2017. doi: 10.1109/TPWRS.2016.2559485
[6]	B. Shen, G. Ghatikar, Z. Lei, J. Li, G. Wikler, and P. Martin, “The role of regulatory reforms, market changes, and technology development to make demand response a viable resource in meeting energy challenges,” Appl. Energy, vol. 130, pp. 814–823, Oct. 2014. doi: 10.1016/j.apenergy.2013.12.069
[7]	J. Qin, Y. Wan, X. Yu, F. Li, and C. Li, “Consensus-based distributed coordination between economic dispatch and demand response,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 3709–3719, Jul. 2019. doi: 10.1109/TSG.2018.2834368
[8]	Y. Hung and G. Michailidis, “Modeling and optimization of time-of-use electricity pricing systems,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 4116–4127, Jul. 2019. doi: 10.1109/TSG.2018.2850326
[9]	R. Deng, Z. Yang, F. Hou, M. Y. Chow, and J. Chen, “Distributed realtime demand response in multiseller-multibuyer smart distribution grid,” IEEE Trans. Power Syst., vol. 30, no. 5, pp. 2364–2374, Sept. 2015. doi: 10.1109/TPWRS.2014.2359457
[10]	Q. Dong, L. Yu, W. Song, J. Yang, Y. Wu, and J. Qi, “Fast distributed demand response algorithm in smart grid,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 2, pp. 280–296, Apr. 2017. doi: 10.1109/JAS.2017.7510529
[11]	H. Hao, D. Wu, J. Lian, and T. Yang, “Optimal coordination of building loads and energy storage for power grid and end user services,” IEEE Trans. Smart Grid, vol. 9, no. 5, pp. 4335–4345, Sept. 2018. doi: 10.1109/TSG.2017.2655083
[12]	R. Carli and M. Dotoli, “Decentralized control for residential energy management of a smart users’ microgrid with renewable energy exchange,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 3, pp. 641–656, May 2019. doi: 10.1109/JAS.2019.1911462
[13]	L. Yu, Y. Sun, Z. B. Xu, et al., “Multi-agent deep reinforcement learning for HVAC control in commercial buildings,” IEEE Trans. Smart Grid, vol. 12, no. 1, pp. 407–419, Jan. 2021. doi: 10.1109/TSG.2020.3011739
[14]	D. Azuatalam, W. L. Lee, F. D. Nijs, and A. Liebman, “Reinforcement learning for whole-building HVAC control and demand response,” Energy and AI, vol. 2, p. 100020, 2020.
[15]	K. Vanthournout, B. Dupont, W. Foubert, and S. Claessens, “An automated residential demand response pilot experiment, based on day-ahead dynamic pricing,” Appl. Energy, vol. 155, pp. 195–203, Oct. 2015. doi: 10.1016/j.apenergy.2015.05.100
[16]	A. Mohsenian-Rad, V. W. S. Wong, J. Jatskevich, R. Schober, and A. Leon-Garcia, “Autonomous demand-side management based on game-theoretic energy consumption scheduling for the future smart grid,” IEEE Trans. Smart Grid, vol. 1, no. 3, pp. 320–331, Dec. 2010. doi: 10.1109/TSG.2010.2089069
[17]	C. Li, C. Liu, K. Deng, X. Yu, and T. Huang, “Data-driven charging strategy of PEVs under transformer aging risk,” IEEE Trans. Control Syst. Technol., vol. 26, no. 4, pp. 1386–1399, Jul. 2018. doi: 10.1109/TCST.2017.2713321
[18]	R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 2011.
[19]	S. Vandael, B. Claessens, D. Ernst, T. Holvoet, and G. Deconinck, “Reinforcement learning of heuristic EV fleet charging in a day-ahead electricity market,” IEEE Trans. Smart Grid, vol. 6, no. 4, pp. 1795–1805, Jul. 2015. doi: 10.1109/TSG.2015.2393059
[20]	B. Kim, Y. Zhang, M. V. D. Schaar, and J. Lee, “Dynamic pricing and energy consumption scheduling with reinforcement learning,” IEEE Trans. on Smart Grid, vol. 7, no. 5, pp. 2187–2198, Sept. 2016. doi: 10.1109/TSG.2015.2495145
[21]	E. Foruzan, L. Soh, and S. Asgarpoor, “Reinforcement learning approach for optimal distributed energy management in a microgrid,” IEEE Trans. Power Syst., vol. 33, no. 5, pp. 5749–5758, Sept. 2018. doi: 10.1109/TPWRS.2018.2823641
[22]	M. Imani and S. F. Ghoreishi, “Scalable inverse reinforcement learning through multifidelity Bayesian optimization,” IEEE Trans. Neural Netw. Learn. Syst., 2021. DOI: 10.1109/TNNLS.2021.3051012
[23]	M. Sadeghi and M. Erol-Kantarci, “Power loss minimization in microgrids using bayesian reinforcement learning with coalition formation, in Proc. IEEE 30th Annu. Int. Symp. Personal, Indoor and Mobile Radio Communications, 2019, 1–6.
[24]	T. Liu, B. Tian, Y. Ai, and F. Wang, “Parallel reinforcement learningbased energy efficiency improvement for a cyber-physical system,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 2, pp. 617–626, Mar. 2020. doi: 10.1109/JAS.2020.1003072
[25]	M. Ahrarinouri, M. Rastegar, and A. R. Seifi, “Multiagent reinforcement learning for energy management in residential buildings,” IEEE Trans. Ind. Informat., vol. 17, no. 1, pp. 659–666, Jan. 2021. doi: 10.1109/TII.2020.2977104
[26]	X. Zhang, D. Biagioni, M. Cai, P. Graf, and S. Rahman, “An edge-cloud integrated solution for buildings demand response using reinforcement learning,” IEEE Trans. Smart Grid, vol. 12, no. 1, pp. 420–431, Jan. 2021. doi: 10.1109/TSG.2020.3014055
[27]	Y. Zhao, Y. Cai, and Q. Song, “Energy control of plug-in hybrid electric vehicles using model predictive control with route preview,” IEEE/CAA J. Autom. Sinica, 2018. DOI: 10.1109/JAS.2017.7510889
[28]	M. Jin, W. Feng, C. Marnay, and C. Spanos, “Microgrid to enable optimal distributed energy retail and end-user demand response,” Appl. Energy, vol. 210, pp. 1321–1335, Jan. 2018. doi: 10.1016/j.apenergy.2017.05.103
[29]	R. Lu, S. H. Hong, and X. Zhang, “A dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach,” Appl. Energy, vol. 220, pp. 220–230, Jun. 2018. doi: 10.1016/j.apenergy.2018.03.072
[30]	M. Yu, S. H. Hong, Y. Ding, and X. Ye, “An incentive-based demand response (DR) model considering composited DR resources,” IEEE Trans. Ind. Electron., vol. 66, no. 2, pp. 1488–1498, Feb. 2019. doi: 10.1109/TIE.2018.2826454
[31]	Y. Wan, J. Qin, F. Li, X. Yu, and Y. Kang, “Game theoretic-based distributed charging strategy for PEVs in a smart charging station,” IEEE Trans. Smart Grid, vol. 12, no. 1, pp. 538–547, Jan. 2021. doi: 10.1109/TSG.2020.3020466
[32]	L. Yu, W. W. Xie, D. Xie, et al., “Deep reinforcement learning for smart home energy management,” IEEE Internet Things J., vol. 7, no. 4, pp. 2751–2762, Apr. 2020. doi: 10.1109/JIOT.2019.2957289
[33]	H. Jahangir, S. S. Gougheri, B. Vatandoust, et al., “Plug-in electric vehicle behavior modeling in energy market: A novel deep learning-based approach with clustering technique,” IEEE Trans. Smart Grid, vol. 11, no. 6, pp. 4738–4748, Nov. 2020. doi: 10.1109/TSG.2020.2998072
[34]	F. Li, J. Qin, and W. X. Zheng, “Distributed Q-learning-based online optimization algorithm for unit commitment and dispatch in smart grid,” IEEE Trans. Cybern., vol. 50, no. 9, pp. 4146–4156, Sept. 2020. doi: 10.1109/TCYB.2019.2921475
[35]	Y. Du and F. Li, “Intelligent multi-microgrid energy management based on deep neural network and model-free reinforcement learning,” IEEE Trans. Smart Grid, vol. 11, no. 2, pp. 1066–1076, Mar. 2020. doi: 10.1109/TSG.2019.2930299
[36]	“Sdge.com. Home-san diego gas & electric.” [Online]. Available: https://www.sdge.com
[37]	“Real-time hourly prices,” Commonwealth Edison Company. [Online]. Available: https://rrtp.comed.com/live-prices/
[38]	S. Koenig and R. G. Simmons, “The effect of representation and knowledge on goal-directed exploration with reinforcement-learning algorithms,” Machine Learning, vol. 22, no. 1, pp. 227C250, 1996.

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(14) / Tables(3)

Get Citation

PDF

XML

Article Metrics

Article views (927) PDF downloads(93)

Highlights

Study the price-based residential demand response management in smart grid considering PEV loads
Model the PB-RDRM from a social perspective, i.e., maximize the weighted sum of UC's profit and loads' cost
Propose a model-free reinforcement learning-based DR algorithm to address the uncertainties

Price-Based Residential Demand Response Management in Smart Grids: A Reinforcement Learning-Based Approach

doi: 10.1109/JAS.2021.1004287

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Highlights

Export File

Citation

Format

Content