Neural-Network-Based Control for Discrete-Time Nonlinear Systems with Input Saturation Under Stochastic Communication Protocol

I. Introduction

OPTIMAL control has been one of the main focuses of control fields due to its wide applications in various emerging industrial systems, such as electrical power systems, industrial control systems, and spacecraft attitude control systems [1]-[7]. It is usually equivalent to solve the well-known Hamilton-Jacobi-Bellman (HJB) equation, which is a critical challenge for nonlinear systems [8]. Fortunately, the adaptive dynamic programming (ADP) algorithm, as the most efficient tool, has been developed to perform various suboptimal control issues with known or unknown system dynamics [9]-[11] by virtue of both its ability of effectively approximating correlation functions and the characteristics of iterative forward transfer. The main idea of ADP algorithms is to utilize two function sequences to iteratively approximate the cost and value functions corresponding to the solution of the HJB equation in a forward-in-time manner [12]. It should be pointed out that the value iteration technology developed in [13], [14] is one of the most important iterative ADP algorithms, and its convergence has also been thoroughly discussed in [15]-[17]. Furthermore, some representative algorithms including heuristic dynamic programming (HDP), dual heuristic dynamic programming (DHP), as well as globalized DHP have been proposed and implemented in various control issues benefiting from the famous actor-critic structure, see [18]-[20]. It is noteworthy that the obtained controller is usually a suboptimal one because of the existence of approximation errors of such a structure, and therefore the corresponding control is also regarded as near-optimal control.

In engineering practice, the actuator saturation is very pervasive due mainly to the facility protection or physical limits of the actuators. If the saturation of the actuator is not considered adequately, the performance of the closed-loop system is often severely damaged [21]. As a result, it is of tremendous significance to survey the influence of the input saturation phenomenon. Under the framework of optimal control, a bounded and invertible one-to-one function in a nonquadratic performance functional is usually exploited to evaluate the cost of saturated inputs and the analytical solution of the optimal controller can be obtained although it is still dependent on the cost functional [8], [22], [23]. Inspired by these work, the near-optimal control for various networked control systems has been investigated and some interesting results have been preliminarily reported in the literature, see [24]-[27], for instance. Near-optimal regulation under the actor-critic framework has been investigated in [26] for discrete-time nonlinear systems subject to quantized effects, where the quantization errors can be eliminated via a dynamic quantizer with adaptive step size. Furthermore, an online policy iteration algorithm has been presented in [28] to learn the optimal solution for a class of unknown constrained-input systems. Obviously, compared with the case without control constraints, the near-optimal control issues subject to constrained-inputs and various network-induced phenomena remain at an infant stage and thus require further research efforts.

On another frontier of research, in the past few years, we have witnessed the persistent development of network technologies, which has been attracting recurring attention on networked control systems [29]. In order to effectively utilize the limited resource or reduce the switching frequency for prolonging the service life of the equipment, only one (or a limited number of) sensor/control node, governed by various protocols, is permitted to get access to the communication network. These protocols include, but not limited to, the round-robin protocol [30], the try-once-discard protocol [31] and the stochastic communication protocol [32], and the event-triggered protocol [33], [34]. There is no doubt that the utilization of these protocols tremendously results in the complexity and the difficulty of both the stability analysis and the design of weight updating rules, which is the main reason why there are sparse results on this topic. Very recently, consensus control with the help of reconstructed dynamics of the local tracking errors has been investigated in [35] for multi-agent systems with event-triggered mechanism and input constraints, where the effect on the local cost from the adopted triggering scheme has been investigated. The critic and actor networks combined with an identifier network have been simultaneously designed in [27] to deal with a constrained-input control issue with unknown drift dynamics and event-triggered communication protocols. Unfortunately, so far, near-optimal control for the discrete-time nonlinear systems subjected to input saturations has not yet been adequately investigated, not to mention the stochastic communication protocol (SCP) is also a concern, which constitutes the motivation of this paper.

The addressed system with unknown nonlinear dynamics is essentially a protocol-induced switching system when SCP is employed to govern the data transmission or update between the controller and the actuator. Usually, SCP can be modeled by a Markov chain and the relative networked control issues can be effectively handled via the switching system theory combined with Lyapunov approaches. It is worth noting that this is a nontrivial topic for optimal control issues due mainly to the challenge of the cost function from such a switch. Recently, two typical approaches have been, respectively, developed in [36] via a combined cost function related to transition probabilities and in [37] via the dynamic programming principle [38]. However, when an identifier is designed to approximate the unknown nonlinear dynamics, there exists a great challenge to disclose the influence on the updating rules of the identifier’s weights and the identification errors. Furthermore, the convergence of the designed ADP algorithm and the practical execution with critic and actor networks should be further inspected. As such, motivated by the above discussions, the focus of this paper is to handle the neural networks (NN)-based near-optimal control problem for a discrete-time nonlinear system subject to constrained-inputs and SCPs. This appears to be nontrivial due to the following essential difficulties: 1) how to design an NN-based identifier under SCPs to estimate system dynamics, 2) how to perform the convergence analysis of the ADP algorithm, and 3) how to disclose the performance of the closed-loop system in the framework of critic and actor networks.

In response to the above discussions, this paper is concerned with the near-optimal control problem for a class of discrete-time nonlinear systems with constrained inputs and SCPs, and hence its main contributions are highlighted as follows: 1) an NN-based identifier with a robust term is presented to approximate the unknown nonlinear system, where novel weight updating rules are constructed by virtue of an additional tunable parameter; 2) a set of conditions are derived to check the stability of both identification error dynamics and updated error dynamics of NN weights; 3) the convergence of proposed value iterative ADP algorithm, which solves the optimal control issue of protocol-induced switching systems with saturation constraints in an off-line way, is profoundly discussed in light of mathematical induction; and 4) an actor-critic NN scheme is employed to perform the addressed near-optimal control issue.

The rest of this paper is formulated as follows: the problem formulation and preliminaries are presented in Section II. For the addressed control issue, four subsections are involved in Section III: an NN-based identification with a robust modification term is designed in Section III-A to identify discrete-time systems with unknown nonlinear dynamics; the value iterative ADP algorithm with convergence analysis is developed in Section III-B; the implementation of ADP algorithm with actor-critic networks in Section III-C, and the performance of closed-loop systems is discussed in Section III-D. Furthermore, a numerical example is given in Section IV to demonstrate the effectiveness of the proposed algorithms. Finally, the conclusion is given in Section V.

Notation: The notation used in this paper is standard. $N$ denotes the set of nonnegative integers. ${\mathbb{R}}^{N}$ denotes the set of all $N$-dimensional real matrices. For the matrix $Q$, $Q^{T}$ and ${\rm tr}${Q} denote the transpose and the trace of $Q$, respectively. ${\rm diag}\{Q_1, Q_2, \ldots ,Q_n\}$ stands for a block-diagonal matrix where the square matrices $Q_i$ are in the corresponding main diagonal blocks. For a vector $x$, $\parallel x\parallel$ denotes the Euclidean norm.

II. Problem Formulation and Preliminaries

In this paper, the investigated networked control system consists of a nonlinear plant, sensors, identifier, controller, as well as actuator. We assume that the system states $x_k$ can be measured directly by sensors and then sent to the controller via shared networks. By using NNs, identifier along with the controller is utilized to realize the approximation of the nonlinear systems based on the received signals. To reduce the communication burden, SCPs are employed to schedule the information transmission between the controller and the actuator.

A System Description

Consider the unknown discrete-time nonlinear system with the following form:

where $x_{k}\in {\mathbb{R}}^{M}$ is the system state directly measured by sensors, $\bar{u}_{k} = [\bar{u}_{1,k},\bar{u}_{2,k},\ldots,\bar{u}_{N,k}]^{T}\in {\mathbb{R}}^{N}$ is the actuator input scheduled by SCPs, and $f(x_{k})$ and $g(x_{k})$ are, respectively, the unknown nonlinear functions with $f(0) = 0$ and $g(0) = 0$. Assume that $f+gu$ is Lipschitz continuous on a set $\Omega_x$ containing the origin. Furthermore, the actuator input $\bar u_{k}$ belongs to $\Omega_u = \{\bar u_{k}| |\bar{u}_{i, k}|\leq \bar u\}$ where $\bar{u}$ is a positive scalar representing the saturation-level of the actuator.

In light of unknown nonlinearities, an NN-based system identifier via $x_k$, which will be designed in the next section, needs to be adopted to obtain the ideal control signal $u_k = [{u}_{1,k},{u}_{2,k},\ldots,{u}_{N,k}]^{T}$. For the convenience of analysis, the saturation constraint can be removed from $\bar{u}_{k}$ into the control signal $u_k$, that is, $u_k\in\Omega_u$. In what follows, SCP scheduling is performed to reduce the switching frequency and improve the communication burden between the controller and the actuator. To model this process, let us introduce the scheduling signal $\xi_{k}\in\{1,2,\ldots,N\}$ to describe the selected element obtaining access to the network at time instant $k$. Under SCPs, $\xi_{k}$, a random variable, is modeled by a Markov chain with the known transition probability

where $p_{ij}\geq0$ for $\forall i,\;j = \{1,2,\ldots,N\}$ and $\sum\nolimits^{N}_{_{j = 1}}p_{ij} = 1$. By means of the above variable, the signal $\bar{u}_k$ received by the actuator is expressed as

where zero-order-holders are utilized in the viewpoint of practical engineering.

The actuator is further denoted as

with $\Phi(i) = {\rm diag}\{\sigma_{i}^{1}, \sigma_{i}^{2},\ldots,\sigma_{i}^{N}\}$ where $\sigma_{i}^{n}\triangleq\sigma(i-n)\in\{0,1\} \; (n = 1,2,\ldots,N)$ is the Kronecker delta function, i.e., $\sigma(i-n)$ is a binary function that equals 1 if $i = n$ and equals 0 otherwise.

Thus, the closed-loop system is as follows:

Remark 1: The main idea of SCP is to assign the access privilege for each node in a random manner. The “random switch” behavior of the node scheduling can be usually characterized by a Markov chain, see the corresponding research in [39]. Obviously, the addressed system (5) is essentially a protocol-induced switching system.

B Design Objective

To quantify the control performance, the associate utility of each scheduling is employed as follows:

where $\ell_{i}(x_{j}, u_j)$ is the cost function, in which $Q_{i}(x_{j})$ is positive and usually a quadratic function, and $S_{i}(u_j)$ is generally a positive nonquadratic function to evaluate the constrained control input. In this paper, $S_{i}(u_k)$ is selected as

where $\tanh(\nu/\bar u)$ stands for the hyperbolic tangent function; $\nu = [\nu_1,\;\nu_2,\ldots,\nu_N]^T$ is an integral vector; and $R_{i}\triangleq {\rm diag}\{r_{i,1}, \; r_{i,2},\; \ldots ,r_{i,N}\}$ is a known positive definite diagonal matrix with appropriate dimension. The operator $\tanh^{-1}(\nu/ \bar{u})$ means

Via the same with the approach in [27], $S_{i}(u_k)$ can also be expressed as

where $\bar{R}_{i}\triangleq[r_{i,1},\; r_{i,2},\; \ldots,r_{i,N}]$.

Remark 2: In the framework of optimal control, the term $S_i(u_k)$ in the associate utility should satisfy the following three conditions: 1) a continuous and positive function for the performance evaluation, 2) a monotonic function for each component, and 3) a derivable function whose derived function should be invertible for the analytic solution of the optimal control law. Obviously, the adopted $S_{i}(u_k)$ (7) is definitely the best choice for the control $u_k$ subject to input saturation.

In order to disclose the effect from statistical characteristic of SCPs, similarly to the scheme in [36], reconstruct the performance index function (6) by embedding the transition probability matrix as follows:

By virtue of the weighted sum technique, a combined performance index is constructed as

where $\lambda_{i}>0$ is the weight vector satisfying $\sum\nolimits_{{i} = 1}^{N}\lambda_{i} = 1$.

Define

where $\Gamma_{i} = \sum\nolimits_{s = 1}^{n}\lambda_sp_{si}>0$. It follows from (6) and (8) that

Before proceeding further, let us introduce the following definition.

Definition 1: A law $u_k$ is an admissible control, if $u_k$ is continuous on the compact set $\Omega_u\subseteq \mathbb{R}^{N}$ and can stabilize the closed-loop system (5) for all $x_0\in\Omega_x$ and $J(x_k)$ is finite $\forall x_k\in\Omega_x$.

The purpose of this paper is to find a suboptimal control law $u_k^{\ast}$ to optimize the combined performance index (9), which consists of the following three aspects:

1) Designing an NN-based identifer to identify the unknown nonlinear dynamics;

2) Developing a value iterative ADP algorithm to solve the optimal control of protocol-induced switching systems with saturation constraints in an off-line way;

3) In light of the obtained value iterative ADP algorithm, proposing an actor-critic NN scheme to perform the addressed near optimal control.

The following assumption is needed in order to reveal the boundedness of developed approximate scheme in sequel.

Assumption 1: The cost function $\ell_i(x,u)$ ($i = 1,2,\;\ldots,\;n_y$) satisfies the following conditions:

1) $\ell_i(x,u)$ is continuously differentiable on $u$ and its derivative is denoted as $\psi_{\ell,i}(u): = {\partial\ell_i(x,u)}/{\partial u}$;

2) The derivative function $\psi_{\ell,i}(u)$ is invertible with its inverse function denoted as $u_i(x) = \psi^{-1}_{\ell,i}\big({\partial\ell_i(x,u)}/{\partial u}\big)$;

3) The inverse function satisfies $\|\psi^{-1}_{\ell,i}(x)\|^2\leq \gamma\|x\|^2$ with a known positive constant $\gamma$.

Note, that the function $\ell_i(x,u)$ is quite general with examples including 1) $\kappa\sum\nolimits_{s = 1}^p\int_0^{u}\tanh^{-1}(\nu/\kappa)R_id\nu$ for nonlinear systems with input constraints; and 2) $x^T{P}_ix+u^TR_iu$ for linear systems.

III. Main Results

Four subsections are embodied in this section, including the design of the NN-based identifer, the iterative ADP algorithm, and the actor-critic NN scheme, as well as the performance analysis of the identification errors, the iterative ADP algorithm and the closed-loop system.

A Identification of Closed-Loop Systems via NNs

In this paper, an NN-based approximator is utilized to identify discrete-time nonlinear systems without the knowledge of system dynamics to solve the optimal control issue. Specifically, to learn the unknown nonlinear functions, a stable adaptive weight updating law is proposed for tuning the nonlinear identifier, and a robust modification term, a function of estimated error and an additional tunable parameter, are also introduced to guarantee asymptotic stability of the proposed nonlinear identification scheme.

To start the development of NN-based identifier, the system dynamic (5) is rewritten as

where $F_{i}(x_k, u_{k})\triangleq f(x_k)+\tilde{g}_{i}(x_k){u}_k$.

According to the universal approximation property of NNs, there exists an NN representation of the function $F_{\xi_k}(x_k, u_{k})$ on a compact set $\Omega_x$. In this paper, a three-layer NN is considered as the function approximation structure, under which the number of neurons in the hidden layer is r, the weight matrix (a predetermined constant matrix) between the input and hidden layers is denoted by $W_{1}$, and the weight matrix between the hidden layer and output layer is denoted as $W_{2,\xi_k}$, which needs to be estimated during the training process. In this case, the closed-loop system (10) is further described as

where $\omega_{k} = W_{1}^{T}[x_{k}^{T}\; u_{k}^{T}]^{T}$ is the hidden layer input, $\phi_x(\omega_{k})$ is the bounded activation function satisfying $\|\phi_x(\omega_{k})\|\leq\phi_{x,m}$, and $\varepsilon_{k}$ is the approximation error salifying a general assumption to be provided as follows.

For the NN represented closed-loop system (11), an identifier is designed to estimate the system state, which is described by

where $\hat{W}^{k}_{2,\xi_k}$ denotes the estimation of the ideal weight matrix $W^{k}_{2,\xi_k}$, and $q_k$ is a robust term to reduce the approximation error.

Define the identification error and the estimated error of weight matrix as follows:

Then, subtracting (11) from (12) obtains the following identification error dynamics:

Considering this error dynamics, the robust term inspired by the work of [40] is constructed as

where $c_2>1$ is a given constant, $\nu_k$ is an additional tunable parameter to be designed subsequently. Therefore, the system dynamics (14) can be further rewritten as

where $\Phi^{k}_{1,i}$ and $\Phi^{k}_{2,i}$ are introduced for brevity in writing.

For adopted communication protocols, $\xi_{k}$, modeled by a Markov chain with the known transition probability, is usually known via the communication coding. To minimize the square residual error $E_{k+1} = ({1}/{2})\tilde{x}_{k+1}^{T} \tilde{x}_{k+1}$, the tuning law of $\hat{W}^{k}_{2,i}$ is given as follows:

and the tuning law of additional tunable parameter $\nu_k$ introduced as

where $\gamma_w>0$ is the NN learning rate, and $\alpha_\nu>0$ and $\gamma_\nu>0$ are the designed parameters.

Remark 3: The proposed updating rule (16) is novel and nontrivial. First, a zero-order holder is adopted to keep the weights of unactivated subsystems. Specifically, it can be found from the second case that the weights are unchanged along with the time $k$. Second, the update of weights is performed only when two successive schedulings are satisfied in order to avoid the fluctuation of weight updates.

The following assumption and lemma are used to prove the convergence of the error dynamics.

Assumption 2: The NN approximation error $\varepsilon_{k}$ is upper bounded by a function of identification error $\tilde{x}_k$, that is

where $\bar\vartheta$ is a known constant.

Lemma 1: For any positive definite matrix $\Pi\in {\mathbb{R}}^{n\times n}$, vectors $x,y\in {\mathbb{R}}^{n}$ and scalar $a>0$, the following inequality is true

Theorem 1: Let the identifier (12) be used to identify the nonlinear system (10), where the parameter updating laws given in (16) and (17) are used tuning the NN weights and the robust modification term, respectively. The estimation error $\tilde{x}_k$ in (14) is asymptotically stable while the weights $\hat{W}^{k}_{2,i}$ and the additional tunable parameter $\nu_k$ are convergent if the learning rate $\gamma_w$ satisfies $6\gamma_w\phi_{x,m}^{2}\leq\theta_1$, and parameters $\gamma_\nu$ and $\alpha_\nu$ satisfy $\gamma_\nu = \gamma_w\phi_{x,m}^{2}$ and

Proof: Consider the following Lyapunov function candidate

Taking the first-order difference of $L^{k}_{1}$ along with the dynamics (15) yields

Similarly, taking the first-order difference of $L^{k}_{2}$ along with the dynamics (16) results into

Noting $\phi_x(\omega_k)\leq\phi_{x,m}$, one has

Furthermore, it is not difficult to calculate that

Denote the first-order difference of $\Delta L^{k}$ as

Considering (22), (24) and (25), the equation (26) can be handled as

Then, considering Assumption 2 and $(\Phi^{k}_{3})^{2}\tilde{x}_{k}^{T}\tilde{x}_{k}\leq\tilde{x}_{k}^{T}\tilde{x}_{k}$, one has

Furthermore, noting

one has

where the scalar $\varepsilon$ belongs to $(0,\;1)$.

Furthermore, select the parameters as $\gamma_\nu = \gamma_w\phi_{x,m}^{2}$, and $6\gamma_w\phi_{x,m}^{2}\leq\theta_1$. Applying Lemma 1, it follows from (27) that

Therefore, one has ${{E}}\{\Delta L^{k}|\xi_k = i,x_k\} < 0$ if the following inequalities hold

which yields

that is, the inequalities (20). ■

Remark 4: It should be pointed out that the approximate error $\varepsilon_k$ of NNs should be dependent on system states and will trend to zero as system states are close to the original point. As such, the feature should be adequately taken into consideration. In this paper, a robust term $q_{k}$ with an additional tunable parameter $\nu_k$, inspired by the work of [40], is employed to improve the system perform while guaranteeing the asymptotic stability. Furthermore, the update of $\hat{W}^{k}_{2,i}$ in (16) is affected by the Markov jump, and therefore a novel Lyapunov function candidate is constructed by adding the term $({1}/{\gamma_w})\sum\nolimits_{s = 1}^N {\rm tr}\left\{(\tilde{W}^{k}_{2,s})^{T}\tilde{W}^{k}_{2,s}\right\}$ to discover the desired condition of identifer dynamics.

B Design of ADP Algorithm

According to the Bellman’s optimality principle, the optimal performance index function $J^{\ast}(x_k)$ satisfies the discrete-time HJB equation

and the corresponding optimal control strategy is given by

Assume that the minimum on the right-hand side of (32) exists and is unique. Taking the first-derivative of the right-hand part, the ideal optimal control $u_k^{\ast}$ is given by

Since the direct solution of the HJB equation for nonlinear systems is computationally intensive, the value iteration algorithm, usually named as an ADP algorithm, needs to be developed in light of the Bellman’s principle of optimality. Initializing the value function $J_0(x_k) = \Sigma(x_k)$, one construct the following iterative algorithm:

and

where $s$ is the iterative step, and $J_s(x_k)$ and $u_s(x_k)$ are used to approximate $J^{\ast}(x_k)$ and $u_k^{\ast}$, respectively, as $s\rightarrow\infty$.

Inspired by [17], [41], we further demonstrate the convergence of the developed scheme with the help of a “functional bound” method.

Lemma 2: Consider the sequences $J_{s}(x_{k})$ and $u_s(x_k)$ introduced by (36) and (35), respectively. Given the initial value function $J_{0}(x)$ for $x\in {\mathbb{R}}^{M+N}$, the value function $J_{s}(x)$ is a monotonically sequence as $s$ increases. Specifically, if $J_{0}(x)\leq J_{1}(x)$, the value function $J_{s}(x)$ is a monotonically nondecreasing sequence as $s$ increases, i.e., $J_{s}(x)\leq J_{s+1}(x)$, otherwise the value function $J_{s}(x)$ is a monotonically decreasing sequence as $s$ increases, i.e., $J_{s}(x)> J_{s+1}(x)$.

Proof: Let us first prove the case of $J_{0}(x)\leq J_{1}(x)$ by mathematical induction. Consider (36) and $J_{0}(x)\leq J_{1}(x)$, one has

Assume that $J_{q-1}(x)\leq J_{q}(x)$ holds when $s = q-1$. Then, for $s = q$, one can conclude that

Therefore, this case is true. Similarly, one can conclude that $J_{s}(x)\geq J_{s+1}(x)$ when $J_{0}(x)\geq J_{1}(x)$.■

Theorem 2: Consider the sequences $J_s(x_k)$ and $u_s(x_k)$ introduced in (36) and (35), respectively. If there exist four constants $\underline{\rho}$, $\bar{\rho}$, $\underline{\varrho}$ and $\bar{\varrho}$ satisfying $0<\underline{\rho}\leq\bar{\rho}$ and $0\leq\underline{\varrho}\leq\bar{\varrho}$ such that

and

hold uniformly, then the iterative value function $J_s(x_k)$ converges to the optimal value $J^{\ast}(x_k)$ as $s\rightarrow\infty$, i.e.,

Proof: To verify this result, we will first prove the following assertion by using the mathematical induction method.

Assertion: Case I: For parameters $0\leq\underline{\varrho}\leq\bar{\varrho}<1$, the iterative value function $J_s(x_k)$ satisfies

Case II: For parameters $0\leq\underline{\varrho}\leq1\leq\bar{\varrho}\leq\infty$, the iterative value function $J_s(x_k)$ satisfies

Case III: For parameters $1\leq\underline{\varrho}\leq\bar{\varrho}\leq\infty$, the iterative value function $J_s(x_k)$ satisfies (40).

Considering the limited space, we only prove the left-hand side of the inequality (40) in Case I and the right-hand side of the inequality (41) in Case II. Furthermore, the proof of Case III is similar to those the first two cases and hence its proof is omitted.

Obviously, the left-hand side of the inequality (40) in Case I holds for $s = 0$. Then, combing with the condition (37), one can derive that

Furthermore, assume that the conclusion holds for $s = q-1$, that is

When $s =q$, combining with the condition (37) again, one has

According to the mathematical induction method, the left-hand side of the inequality (40) holds.

In what follows, let us prove the right-hand side of the inequality (41) in Case II. Obviously, it is not difficult to find that

where the term $\,\bar{\rho}\Gamma^{T}L(x_k,u_k)-J^{\ast}(x_{k+1})$ induced by the condition (37) is added.

Furthermore, assume that the conclusion holds for $s = q-1$, that is

When $s = q$, combining with the condition (37), one has

In light of the mathematical induction method, the right-hand side of the inequality (41) holds.

Combining the above conclusions, we can obtain that this assertion is true. Finally, letting $s\to\infty$, the convergence (39) is easily derived.■

Remark 5: The above theorem discloses the convergence of the developed ADP scheme with the help of a “functional bound” method, which comes from [17], [41]. An assertion has been allocated for the convenience of processing. For the practical application, a terminal condition (or a fixed size number $s^\ast$) is adopted, the related algorithm (i.e., ADP Algorithm 1) is provided as follows.

Algorithm 1 ADP algorithm

Initialization Value $J_0(x_k)=\Sigma(x_k)$ and error threshold $\varpi > 0$. Set $s=0$.

1: while $|J_{s}(x_k)-J_{s-1}(x_k)|>\varpi$ do

2: Solve $u_s(x_k)$ according to (35);

Update the value $J_{s+1}(x_k)$ according to (36);

Set $s=s+1$;

3: end while

4: Output Control strategy $u_{s-1}(x_k).$

C Implementation of ADP Algorithm

Due to the unknown $J_{s}(x_{k+1})$, a approximation structure via NNs is employed to approximate both $J^{\ast}(x_k)$ and $u^{\ast}(x_k)$. Such a structure consists of a critic network and an actor network, which are all chosen as three-layer feed forward NNs and their implementation process is shown in Fig. 1. In light of the above conception, the optimal value function (32) and the control input (34) can be described by the following critic NN and actor NN:

Figure  1.  Neural network structure of the proposed ADP approach.

DownLoad: Full-Size Img PowerPoint

and

with $z_{k} = [x^{T}_k\; u^{T}_k]^{T}$ where $W_{2c}$ and $W_{2a}$ are the ideal wights of designed NNs and bounded, respectively, by two positive scalars $\bar{W}_{2cM}$ and $\bar{W}_{2aM}$, i.e., $\|W_{2c}\| \leq W_{2cM}$ and $\|W_{2a}\|\leq W_{2aM}$; $\theta_c({\textit{z}}_{k})$ and $\theta_a(x_{k})$ are the bounded approximation errors, i.e., $\|\theta_c(z_{k})\|\leq\theta_{cM}$ and $\|\theta_a(x_{k})\|\leq\theta_{aM}$; $W_{1c}$ and $W_{1a}$ are the known weight matrices of between the input layer and hidden layer; and $\phi_{c}(\cdot)$, $\phi_{1a}(\cdot)$ and $\phi_{2a}(\cdot)$ are the activation functions satisfying $\|\phi_{c}(\cdot)\|\leq\phi_{c,m}$, $\|\phi_{1a}(\cdot)\|\leq\phi_{1a,m}$ and $\|\phi_{2a}(\cdot)\|\leq\phi_{2a,m}$.

In order to identify the ideal weight $W_{2c}$, the following approximation is developed by virtue of the ADP algorithm

where ${\textit{z}}_{s,k} = [x^{T}_k\; u^{T}_{s,k}]^{T}$.

Taking the above equation into (36), there is usually

that is

Introduce the gap

and then define the cost function

Minimizing such a function results in the updating rule of the weights of the critic network

where $\varepsilon_c>0$ is the learning rate of the critic network. The weights of the model network are kept unchanged after finished the training process.

In the actor network, $x_k$ is used as the input, while the control input is approximated by

On the other hand, it follows from (34) that the approximated value is also obtained by

Denote $\Upsilon_i = ({1}/{2\bar u})R^{-1}\tilde{g}_{i}^{T}(x_k)$ and then introduce the gap

In what follows, define the cost of this gap

By employing the gradient descent approach again to minimize $e_{a,s}$, one has the updating rule of the weights of the actor network

where $\varepsilon_a> 0$ is the learning rate of the action network.

Defining the estimation errors of weight matrices

one has

and

D Performance Analysis

It is easily seen that the estimation errors of weights in actor and critic networks will inevitably affect the performance of the above ADP algorithm. Thus, it is necessary to prove the boundedness of the critic and actor NN weights.

Theorem 3: Consider the discrete-time Markov jump system (MJS) (5), the critic NN (44) and the actor NN (48). Then, for the fixed time $k$, the weight estimation error $\tilde{W}_{c,s}$ in (52) of the critic NN and the weight estimation error $\tilde{W}_{a,s}$ in (50) of the actor NN are all UUB, if the following conditions for the learning rates are satisfied

Proof: In order to show the boundedness, we introduce a Lyapunov function candidate

In what follows, the proof is similar to the one in literature [42], and therefore its details are omitted, and the corresponding learning rates need to satisfy

Since the excitation functions $\phi_c(\cdot)$, $\phi_{1a}(\cdot)$ and $\phi_{2a}(\cdot)$ are bounded, the ideal learning rates can be obtained.■

Assumption 3: The function $\|g(x_k)\|$ in (1) is bounded, and therefore the function $\|g_{i}(x_k)\|$ is also bounded.

Theorem 4: Let the initial control input be admissible and the initial actor-NN and critic-NN weights be selected from a compact set which includes the ideal weights. The NN weight updating laws (47) and (51) are adopted in an off-line way for the critic network (44) and the actor network (48), and the updating law (50) with (17) is employed in an online way for the identifier (12). Then, the closed-loop system (5) (or (10)) with control law (48) selecting $\hat{u}(x_k) = \phi_{2a}(\hat{W}^{T}_{2a,\infty}\phi_{1a}(W_{1a}^{T}x_{k}))$ is ultimately bounded in mean-square sense if all conditions in Theorems 1 and 3 hold.

Proof: In the framework of identifier-based control, taking the control policy (48) into account, the actual closed-loop system as follows:

where

Obviously, considering the property of activation functions of NNs, one has that $\|\phi_{2a}(\tilde{W}^{T}_{2a}\psi_{k})-\theta_a(x_{k})\|$ is bounded. Furthermore, benefiting from Assumption 3, the additional term $g_{\xi_k}(x_k)(\phi_{2a}(\tilde{W}^{T}_{2a}\psi_{k})-\theta_a(x_{k}))$ is also bounded.

On the other hand, according to the optimal control theory, the policy (43) stabilizes the system (11) (i.e., (10)) on the compact set. With the same approach in [37], it is clear that there exists a constant $H^{\ast}$ such that

By virtue of the input-to-state stability or the similar line in [37], one can conclude that the actual closed-loop system is ultimately bounded in mean-square sense.■

Remark 6: In the above subsections, a set of critic and actor networks are designed to approximate the performance index function sequence $J_{s}(x_{k})$ and the control law sequence $u_s(x_k)$ for the fixed $x_k$, where the updating rules of NN weights are derived via the gradient descent. By means of the well-known Lyapunov stability theory, we obtain the conditions on learning rates of neural networks, under which both the weight error dynamics and the closed-loop systems are bounded stable.

Remark 7: In almost all ADP-based suboptimal control issues for the nonlinear systems, NNs are widely utilized to approximate the unknown nonlinear dynamics as well as the actor and critic functions. Such a structure, named the actor-critic structure, provides the capability of forwarding calculation while avoiding the dimensional disaster. Inspired by the idea in [43], a tuning parameter $q_k$ has been employed in the identification of unknown nonlinear systems to adjust the approximate error $\varepsilon_{k}$. Furthermore, the three-layer feed forward NNs have been adopted to approximate actor and critic functions where the approximation capability is enhanced due to the utilization of a hidden layer.

Remark 8: Up to date, two typical iteration strategies of ADP algorithms are utilized to obtain the desired controller parameter and the associate utility, and they are policy iteration (PI) and value iteration (VI), respectively. One major difference between PI and VI strategies is that PI requires an initial admissible control policy that stabilizes the system states [44]. From a mathematical point of view, the initial admissible control can be regarded as a suboptimal control which requires to solve the nonlinear partial differential equations (PDEs) analytically. To overcome the shortage, a VI-based strategy has been developed in this paper to definitely deal with the control issue with input saturation and communication scheduling.

IV. Illustrative Example

In this section, we use a simulation example to show that the proposed suboptimal control is effective for discrete-time nonlinear systems with input saturation under SCPs.

Consider the following nonlinear system:

where $x_{i,k}$ ($i = 1,2$) stands for the $i$-th element of vector $x_k$ with the initial value $x_0 = [-0.5,\;-0.2]^{T}$, and $\bar{u}_{i,k}$ ($i = 1,2$) is the actuator input scheduled by SCPs, where the scheduling probabilities are $p_{11} = 0.65$ and $p_{22} = 0.6$. Its initial value is $u_0 = [-0.1,0.5]^{T}$ and the saturation level $\bar u$ is 5.

In this example, choose three-layer feedforward NNs in the identifier, the critic network and the action network with structures $4-4-2$, $4-2-1$, and $2-2-2$, respectively. Furthermore, select the activation functions as follows

Thus, the bounds of $\phi_x$, $\phi_{c}$, $\phi_{1a}$ and $\phi_{2a}$ are $\phi_{x,m} = 2$, $\phi_{c,m} = \phi_{1a,m} = 1$ and $\phi_{2a} = \bar{u}$, respectively.

In virtue of Theorem 1, we can employ the learning rate $\gamma_\nu = \gamma_w = 0.1$, and parameters $\alpha_\nu = 0.9$, $c_2 = 1$, and $\nu_{0} = 0.1$ in the tuning law $\hat{W}^{k}_{2,i}$ and the additional tunable parameter $\nu_k$. Furthermore, the weight matrix $W_{1}$ between the input and hidden layers is adopted $1.2I$ and the initial weight matrices $\hat{W}^{0}_{2,i}$ ($i = 1, 2$) in (12) between the hidden layer and output layer are selected as

In what follows, we consider the matrices $Q_1 = Q_2 = 0.5I$ and $R_1 = R_2 = 0.2I$ in the cost function (6) and the corresponding scalars $\lambda_1 = 0.4$ and $\lambda_2 = 0.6$ in the combined performance index (8). Furthermore, the parameters in the updating rules (47) and (51) are chosen as $\varepsilon_a = \varepsilon_c = 0.4$ and for the adopted critic-actor network with the help of Theorem 3, the initial weight matrices of this critic-actor network are selected as

In addition, the weight matrices $W_{1a}$ and $W_{1c}$ between the input and hidden layers are

Training of weight matrices for critic-actor networks is performed in instant $k = 4$ with 200 steps. After being trained, the weights are kept unchanged. The training process is shown in Fig. 2 and their trajectories are convergent, which verifies the effectiveness of developed ADP scheme.

Figure  2.  The iterative trajectories of the weight matrices $\hat{W}_{2a,s}$ and $\hat{W}_{2c,s}$.

DownLoad: Full-Size Img PowerPoint

The simulation results are presented in Figs. 3-7. The state trajectories $x_k$ of the open-loop system are depicted in Fig. 3 to reveal that the open-loop system is divergent. With the help of the designed controller, the state trajectories $x_k$ of the closed-loop system and the corresponding trajectories $\hat{x}_k$ of the identifer are respectively shown in Figs. 4 and 5. For this control issue, the secluded node is shown in Fig. 6, which clearly discloses that the system randomly jumps due to the utilization of different actuator units, and the weight matrices of the identifier $\hat{W}^k_{2i}$ are plotted in Fig. 7, all of which are eventually convergent. It is not difficult to see that the closed-loop system is stable, and therefore the developed control strategy is effective.

Figure  3.  State trajectories $x_k$ of the open-loop system.

DownLoad: Full-Size Img PowerPoint

Figure  7.  The iterative trajectories of the weight matrices $\hat{W}^{k}_{2,i}$.

DownLoad: Full-Size Img PowerPoint

Figure  4.  State trajectories $x_k$ of the closed-loop system.

DownLoad: Full-Size Img PowerPoint

Figure  5.  State trajectories $\hat{x}_k$ of the identifer.

DownLoad: Full-Size Img PowerPoint

Figure  6.  The scheduling of actuator units.

DownLoad: Full-Size Img PowerPoint

V. Conclusions

In this paper, we have developed a suboptimal control strategy in the framework of ADP for a class of unknown nonlinear discrete-time systems subject to input constraints. An identification with robust term based on a three-layer neural network in which the weight update relies on protocol-induced jumps, has been established to approximate nonlinear systems and the corresponding stability has been provided. Then, the value iterative ADP algorithm has been developed to solve the suboptimal control problem with boundedness analysis, and the convergence of iterative algorithm, as well as the boundedness of the estimation errors for critic and actor NN weights, has been analyzed. Furthermore, an actor-critic NN scheme has been developed to approximate the control law and the proposed performance index function and the stability of the closed-loop systems have been discussed. Finally, the numerical simulation result has been utilized to demonstrate the effectiveness of the proposed control scheme.