A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 10 Issue 1
Jan.  2023

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 7.847, Top 10% (SCI Q1)
    CiteScore: 13.0, Top 5% (Q1)
    Google Scholar h5-index: 64, TOP 7
Turn off MathJax
Article Contents
W. Q. Cao, J. Yan, X. Yang, X. Y. Luo, and  X. P. Guan,  “Communication-aware formation control of AUVs with model uncertainty and fading channel via integral reinforcement learning,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 1, pp. 159–176, Jan. 2023. doi: 10.1109/JAS.2023.123021
Citation: W. Q. Cao, J. Yan, X. Yang, X. Y. Luo, and  X. P. Guan,  “Communication-aware formation control of AUVs with model uncertainty and fading channel via integral reinforcement learning,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 1, pp. 159–176, Jan. 2023. doi: 10.1109/JAS.2023.123021

Communication-Aware Formation Control of AUVs With Model Uncertainty and Fading Channel via Integral Reinforcement Learning

doi: 10.1109/JAS.2023.123021
Funds:  This work was supported in part by the National Natural Science Foundation of China (62222314, 61973263, 61873345, 62033011), the Youth Talent Program of Hebei (BJ2020031), the Distinguished Young Foundation of Hebei Province (F2022203001), the Central Guidance Local Foundation of Hebei Province (226Z3201G), and the Three-Three-Three Foundation of Hebei Province (C20221019)
More Information
  • Most formation approaches of autonomous underwater vehicles (AUVs) focus on the control techniques, ignoring the influence of underwater channel. This paper is concerned with a communication-aware formation issue for AUVs, subject to model uncertainty and fading channel. An integral reinforcement learning (IRL) based estimator is designed to calculate the probabilistic channel parameters, wherein the multivariate probabilistic collocation method with orthogonal fractional factorial design (M-PCM-OFFD) is employed to evaluate the uncertain channel measurements. With the estimated signal-to-noise ratio (SNR), we employ the IRL and M-PCM-OFFD to develop a saturated formation controller for AUVs, dealing with uncertain dynamics and current parameters. For the proposed formation approach, an integrated optimization solution is presented to make a balance between formation stability and communication efficiency. Main innovations lie in three aspects: 1) Construct an integrated communication and control optimization framework; 2) Design an IRL-based channel prediction estimator; 3) Develop an IRL-based formation controller with M-PCM-OFFD. Finally, simulation results show that the formation approach can avoid local optimum estimation, improve the channel efficiency, and relax the dependence of AUV model parameters.


  • loading
  • [1]
    J. Yan, Z. W. Guo, X. Yang, X. Y. Luo, and X. P. Guan, “Finite-time tracking control of autonomous underwater vehicle without velocity measurements,” IEEE Trans. Syst. Man Cybern. Syst., vol. 52, no. 11, pp. 6759–6773, Nov. 2022.
    S. Wang, L. Chen, D. B. Gu, and H. S. Hu, “Cooperative localization of AUVs using moving horizon estimation,” IEEE/CAA J. Autom. Sinica, vol. 1, no. 1, pp. 68–76, Jan. 2014. doi: 10.1109/JAS.2014.7004622
    H. Y. Zhao, J. Yan, X. Y. Luo, and X. P. Guan, “Privacy preserving solution for the asynchronous localization of underwater sensor networks,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 6, pp. 1511–1527, Nov. 2020. doi: 10.1109/JAS.2020.1003312
    A. Vasilijević, D. Nad, F. Mandić, N. Mišković, and Z. Vukić, “Coordinated navigation of surface and underwater marine robotic vehicles for ocean sampling and environmental monitoring,” IEEE/ASME Trans. Mechatron., vol. 22, no. 3, pp. 1174–1184, Jun. 2017. doi: 10.1109/TMECH.2017.2684423
    Z. Y. Zhou, J. C. Liu, and J. Z. Yu, “A survey of underwater multi-robot systems,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 1, pp. 1–18, Jan. 2022. doi: 10.1109/JAS.2021.1004269
    H. P. Li, P. Xie, and W. S. Yan, “Receding horizon formation tracking control of constrained underactuated autonomous underwater vehicles,” IEEE Trans. Ind. Electron., vol. 64, no. 6, pp. 5004–5013, Jun. 2017. doi: 10.1109/TIE.2016.2589921
    C. Suryendu and B. Subudhi, “Formation control of multiple autonomous underwater vehicles under communication delays,” IEEE Trans. Circuits Syst. Express Briefs, vol. 67, no. 12, pp. 3182–3186, Dec. 2020. doi: 10.1109/TCSII.2020.2976955
    X. Li and D. Q. Zhu, “An adaptive SOM neural network method for distributed formation control of a group of AUVs,” IEEE Trans. Ind. Electron., vol. 65, no. 10, pp. 8260–8270, Oct. 2018.
    Y. J. Zhao, Y. Ma, and S. L. Hu, “USV formation and path-following control via deep reinforcement learning with random braking,” IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 12, pp. 5468–5478, Dec. 2021. doi: 10.1109/TNNLS.2021.3068762
    H. Liu, Y. H. Wang, and F. L. Lewis, “Robust distributed formation controller design for a group of unmanned underwater vehicles,” IEEE Trans. Syst. Man Cybern. Syst., vol. 51, no. 2, pp. 1215–1223, Feb. 2021. doi: 10.1109/TSMC.2019.2895499
    Z. Y. Gao and G. Guo, “Fixed-time sliding mode formation control of AUVs based on a disturbance observer,” IEEE/CAA J. Autom. Sin., vol. 7, no. 2, pp. 539–545, Mar. 2020. doi: 10.1109/JAS.2020.1003057
    B. Chen, J. P. Hu, Y. Y. Zhao, and B. K. Ghosh, “Finite-time observer based tracking control of uncertain heterogeneous underwater vehicles using adaptive sliding mode approach,” Neurocomputing, vol. 481, no. 1, pp. 322–332, Apr. 2022.
    P. Millán, L. Orihuela, I. Jurado, and F. Rubio, “Formation control of autonomous underwater vehicles subject to communication delays,” IEEE Trans. Control Syst. Technol., vol. 22, no. 2, pp. 770–777, Mar. 2014. doi: 10.1109/TCST.2013.2262768
    J. Q. Wang, C. Wang, Y. J. Wei, and C. J. Zhang, “Filter-backstepping based neural adaptive formation control of leader-following multiple AUVs in three dimensional space,” Ocean Eng., vol. 201, no. 1, pp. 1–11, Apr. 2020.
    C. Z. Yuan, S. Licht, and H. B. He, “Formation learning control of multiple autonomous underwater vehicles with heterogeneous nonlinear uncertain dynamics,” IEEE Trans. Cybern., vol. 48, no. 10, pp. 2920–2934, Oct. 2018. doi: 10.1109/TCYB.2017.2752458
    N. Gu, D. Wang, Z. H. Peng, T. S. Li, and S. C. Tong, “Model-free containment control of underactuated surface vessels under switching topologies based on guiding vector fields and data-driven neural predictors,” IEEE Trans. Cybern., vol. 52, no. 10, pp. 10843–10854, Oct. 2022. doi: 10.1109/TCYB.2021.3061588,2021
    Y. Zhou, Y. Wan, S. Roy, C. Taylor, C. Wanke, D. Ramamurthy, and J. Fei, “Multivariate probabilistic collection method for effective uncertainty evaluation with application to air traffic flow management,” IEEE Trans. Syst. Man Cybern. Syst., vol. 44, no. 10, pp. 1347–1363, Oct. 2014. doi: 10.1109/TSMC.2014.2310712
    M. S. Liu, Y. Wan, F. L. Lewis, and V. Lopez, “Adaptive optimal control for stochastic multiplayer differential games using on-policy and off-policy reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 12, pp. 5522–5533, Dec. 2020. doi: 10.1109/TNNLS.2020.2969215
    J. F. Xie, Y. Wan, K. Mills, J. J. Filliben, and F. L. Lewis, “A scalable sampling method to high-dimensional uncertainties for optimal and reinforcement learning-based controls,” IEEE Control Syst. Lett., vol. 1, no. 1, pp. 98–103, May 2017. doi: 10.1109/LCSYS.2017.2708598
    J. Yan, X. Li, X. Y. Luo, Y. D. Gong, and X. P. Guan, “Joint localization and tracking for autonomous underwater vehicle: A reinforcement learning based approach,” IET Control Theory Appl., vol. 13, no. 17, pp. 2856–2865, Jan. 2019. doi: 10.1049/iet-cta.2018.6122
    Y. Wan, S. Roy, and B. Lesieutre, “Uncertainty evaluation through mapping identification in intensive dynamic simulations,” IEEE Trans. Syst. Man Cybern. Syst. Humans, vol. 40, no. 5, pp. 1094–1104, Sept. 2010. doi: 10.1109/TSMCA.2010.2044172
    M. K. Long, H. S. Su, and Z. G. Zeng, “Model-free algorithms for containment control of saturated discrete-time multiagent systems via Q-learning method,” IEEE Trans. Syst. Man Cybern. Syst., vol. 52, no. 2, pp. 1308–1316, Feb. 2022.
    B. D. Ning, Q.-L. Han, Z. Y. Zuo, J. Jin, and J. C. Zheng, “Collective behaviors of mobile robots beyond the nearest neighbor rules with switching topology,” IEEE Trans. Cybern., vol. 48, no. 5, pp. 1577–1590, May 2018. doi: 10.1109/TCYB.2017.2708321
    B. D. Ning, Q.-L. Han, and Z. Y. Zuo, “Bipartite consensus tracking for second-order multiagent systems: A time-varying function-based preset-time approach,” IEEE Trans. Autom. Control, vol. 66, no. 6, pp. 2739–2745, Jun. 2021. doi: 10.1109/TAC.2020.3008125
    M. Stojanovic and J. Preisig, “Underwater acoustic communication channels: Propagation models and statistical characterization,” IEEE Commun. Mag., vol. 47, no. 1, pp. 84–89, Jan. 2009. doi: 10.1109/MCOM.2009.4752682
    Y. Yang, Y. Xiao, and T. S. Li, “A survey of autonomous underwater vehicle formation: Performance, formation control, and communication capability,” IEEE Commun. Surv. &Tut., vol. 23, no. 2, pp. 815–841, May 2021.
    H. Li, J. Peng, W. R. Liu, G. Kai, and Z. W. Huang, “A novel communication-aware formation control strategy for dynamical multi-agent systems,” J. Franklin Inst., vol. 352, no. 9, pp. 3701–3715, Sept. 2015. doi: 10.1016/j.jfranklin.2015.04.008
    Y. Yan and Y. Mostofi, “Robotic router formation in realistic communication environments,” IEEE Trans. Rob., vol. 28, no. 4, pp. 810–827, Aug. 2012. doi: 10.1109/TRO.2012.2188163
    U. Ali, H. Cai, Y. Mostofi, and Y. Wardi, “Motion-communication co-optimization with cooperative load transfer in mobile robotics: An optimal control perspective,” IEEE Trans. Control Netw. Syst., vol. 6, no. 2, pp. 621–632, Jun. 2019. doi: 10.1109/TCNS.2018.2863048
    Y. K. Xia, K. Xu, Y. Li, G. H. Xu, and X. B. Xiang, “Improved line-of-sight trajectory tracking control of under-actuated AUV subjects to ocean currents and input saturation,” Ocean Eng., vol. 174, no. 1, pp. 14–30, Jan. 2019.
    R. X. Cui, L. P. Chen, C. G. Yang, and M. Chen, “Extended state observer-based integral sliding mode control for an underwater robot with unknown disturbances and uncertain nonlinearities,” IEEE Trans. Ind. Electron., vol. 64, no. 8, pp. 6785–6795, Aug. 2017. doi: 10.1109/TIE.2017.2694410
    R. X. Cui, C. G. Yang, Y. Li, and S. Sharma, “Adaptive neural network control of AUVs with control input nonlinearities using reinforcement learning,” IEEE Trans. Syst. Man Cybern. Syst., vol. 47, no. 6, pp. 1019–1029, Jun. 2017. doi: 10.1109/TSMC.2016.2645699
    W. Y. Gan, D. Q. Zhu, Z. Hu, X. P. Shi, L. Yang, and Y. S. Chen, “Model predictive adaptive constraint tracking control for underwater vehicles,” IEEE Trans. Ind. Electron, vol. 67, no. 9, pp. 7829–7840, Sept. 2020. doi: 10.1109/TIE.2019.2941132
    J. H. Qin, M. Li, Y. Shi, Q. C. Ma, and W. Zhang, “Optimal synchronization control of multiagent systems with input saturation via off-policy reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 30, no. 1, pp. 85–96, Jan. 2019. doi: 10.1109/TNNLS.2018.2832025
    D. Licea, M. Bonilla, M. Ghogho, S. Lasaulce, and V. Varma, “Communication-aware energy efficient trajectory planning with limited channel knowledge,” IEEE Trans. Rob., vol. 36, no. 2, pp. 431–442, Apr. 2020. doi: 10.1109/TRO.2019.2948801
    Y. Zhang, Z. M. Zhang, L. Chen, and X. H. Wang, “Reinforcement learning-based opportunistic routing protocol for underwater acoustic sensor networks,” IEEE Trans. Veh. Technol., vol. 70, no. 3, pp. 2756–2770, Mar. 2021. doi: 10.1109/TVT.2021.3058282
    A. Goldsmith, Wireless Communication. Cambridge, U. K.: Cambridge Univ. Press, 2005.
    K. Vamvoudakis, H. Modares, B. Kiumarsi, and F. L. Lewis, “Game theory-based control system algorithms with real-time reinforcement learning: How to solve multiplayer games online,” IEEE Control Syst. Mag., vol. 37, no. 1, pp. 33–52, Feb. 2017. doi: 10.1109/MCS.2016.2621461
    J. F. Xie, Y. Wan, K. Mills, J. J. Filliben, Y Lei, and Z. L. Lin, “M-PCM-OFFD: An effective output statistics estimation method for systems of high dimensional uncertainties subject to low-order parameter interactions,” Math. Comput. Simul., vol. 159, no. 1, pp. 93–118, May 2019.
    Z. X. Wang, Y. Zou, Y. Z. Liu, and Z. Y. Meng, “Distributed control algorithm for leader-follower formation tracking of multiple quadrotors: Theory and experiment,” IEEE/ASME Trans. Mechatron., vol. 26, no. 2, pp. 1095–1105, Apr. 2021. doi: 10.1109/TMECH.2020.3017816
    F. Chen and D. V. Dimarogonas, “Leader-follower formation control with prescribed performance guarantees,” IEEE Control Syst. Mag., vol. 8, no. 1, pp. 450–461, Mar. 2021.
    M. Trinh, Q. Tran, D. Vu, P. Nguyen, and H. Ahn, “Robust tracking control of bearing-constrained leader-follower formation,” Automatica, vol. 131, no. 1, pp. 1–7, Jun. 2021.
    Z. X. Liu, Y. B. Li, F.-Y. Wang, and Z. Q. Chen, “Reduced-order observer-based leader-following formation control for discrete-time linear multi-agent systems,” IEEE/CAA J. Autom. Sinica, vol. 8, no. 10, pp. 1715–1723, Oct. 2021. doi: 10.1109/JAS.2020.1003441
    M. Abu-Khalaf and F. L. Lewis, “Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach,” Automatica, vol. 41, no. 5, pp. 779–791, May 2005. doi: 10.1016/j.automatica.2004.11.034
    J. Yan, X. Li, X. Yang, X. Y. Luo, C. C. Hua, and X. P. Guan, “Integrated localization and tracking for AUV with model uncertainties via scalable sampling-based reinforcement learning approach,” IEEE Trans. Syst. Man Cybern., vol. 52, no. 11, pp. 6952–6967, Nov. 2022. doi: 10.1109/TSMC.2021.3129534,2021
    J. Yan, J. Gao, X. Yang, X. Y. Luo, and X. P. Guan, “Position tracking control of remotely operated underwater vehicles with communication delay,” IEEE Trans. Control Syst. Technol., vol. 28, no. 6, pp. 2506–2514, Nov. 2020. doi: 10.1109/TCST.2019.2928488


    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(16)  / Tables(2)

    Article Metrics

    Article views (146) PDF downloads(44) Cited by()


    • Channel prediction, formation control and model uncertainty are considered together
    • An IRL-based estimator with M-PCM-OFFD is designed to predict the SNR of AUVs in positions that they have not yet visited and can effectively avoid failing into local optimum
    • An IRL-based formation controller with input saturation of thrusters is developed to guarantee communication-aware formation control of AUVs
    • Meanwhile, the nominal values of model parameters are not required to be known by the proposed formation controller as compared with the others works


    DownLoad:  Full-Size Img  PowerPoint