CN113485099B - Online learning control method of nonlinear discrete time system - Google Patents

Abstract

The invention discloses an online learning control method for a nonlinear discrete time system, which includes a behavioral strategy selection step, an optimal Q-function definition step, an evaluation network and execution network introduction step, an estimation error calculation step, and a final optimal weight calculation step. , when the weights of the evaluation network and the execution network converge, the output of the execution network is the approximation of the optimal controller. This invention does not need to iterate repeatedly between strategy evaluation and strategy improvement, and can realize real-time online learning of the optimal controller; it adopts an off-track strategy learning mechanism, which effectively overcomes the problem of insufficient exploration of the state-policy space of the direct heuristic dynamic programming method. Problem, the execution network and the evaluation network can use any form of activation function. The present invention can realize online learning of the optimal controller without a system model and only requires state data generated by the behavioral strategy.

Description

An online learning control method for nonlinear discrete-time systems

Technical field

The present invention relates to the field of industrial production control, and specifically, to an online learning control method for nonlinear discrete time systems.

Background technique

In the process of industrial production, engineering and technical personnel often need to optimize the design of controllers for control objects such as robots, drones, and unmanned vehicles to meet certain control indicators. Since the above control objects often exhibit strong nonlinearity, the optimization of the controller faces great difficulties. From the perspective of optimal control, obtaining the optimal control controller requires solving the complex Hamilton-Jacobi-Bellman equation (HJB equation), but the HJB equation is a nonlinear partial differential equation and is very difficult to solve. Traditional dynamic programming, variational methods, spectral methods, etc. often face great limitations in practical applications due to their extremely high computational complexity.

Adaptive dynamic programming is a new type of intelligent control algorithm that has emerged in recent years. By integrating reinforcement learning, neural network approximation, dynamic programming and adaptive control technologies, it can achieve online learning of the optimal controller and effectively overcome problems. It solves the problem of high computational complexity of traditional methods. For the optimal control problem of nonlinear discrete-time systems, Jennie Si and Yu-Tsung Wang first proposed a direct heuristic dynamic programming algorithm in the paper "Online learning control by association andreinforcement". This algorithm adopts the basic idea of generalized strategy iteration. By introducing two neural networks (ie, execution network and evaluation network), real-time online learning of the optimal controller and optimal value function can be achieved. After continuous development in recent years, the convergence and stability analysis of the algorithm now also has a certain theoretical basis. Although the direct heuristic dynamic programming algorithm can achieve online adaptive optimal control, this algorithm still has the following shortcomings: 1) This algorithm uses an on-orbit policy (on-policy) learning mechanism, which has the problem of insufficient exploration of the state-policy space. , it is easy to fall into the local optimal solution; 2) The activation functions of the execution network and the evaluation network use the hyperbolic tangent function, and all current convergence and stability analysis results are based on the hyperbolic tangent function. For other types of Activation functions no longer apply.

Therefore, how to overcome the above shortcomings of the above-mentioned direct heuristic dynamic programming method so that the convergence and stability analysis results are no longer limited to the hyperbolic tangent function has become a technical problem that needs to be solved urgently in the existing technology.

Contents of the invention

The purpose of the present invention is to propose an online learning control method for nonlinear discrete time systems, which can have better exploration capabilities for the state-policy space, so that the activation function types of the execution network and the evaluation network can be selected arbitrarily, and are no longer limited to Hyperbolic tangent function; compared with iterative methods such as policy iteration or value iteration, this method can achieve online learning of the optimal controller and does not require a system model, only the state data generated by the behavioral policy.

To achieve this goal, the present invention adopts the following technical solutions:

An online learning control method for nonlinear discrete-time systems, including the following steps:

Behavior strategy selection step S110:

According to the characteristics of the controlled object, existing experience is used to select the behavioral strategy u. The behavioral strategy is the control strategy actually applied to the controlled object in the learning process. Its main function is to generate system status data that needs to be used in the learning process;

Optimal Q-function definition step S120:

Define the optimal Q-function as follows:

Its physical meaning is: at time k, adopt the behavioral strategy u, and at all subsequent moments, adopt the optimal control strategy u ^* , that is, the target strategy. It can be seen from the definition of the optimal Q-function that the above formula can be equivalently expressed as :

best control It can be expressed as:/>

For linear systems, Q ^* (x _k ,u _k ) and are nonlinear functions about (x _k ,u _k ) and x _k respectively;

Evaluate the network and perform the network introduction step S130:

Introduce the evaluation network and execution network to respectively control Q ^* (x _k ,u _k ) and Perform online approximation, and the evaluation network and execution network are neural networks;

The evaluation network is used to learn the optimal Q-function Q ^* (x _k , _uk ), and the execution network is used to learn the optimal controller u ^* . Assume that the number of neural network activation functions in the evaluation network is N _c , and note is the best approximation of the evaluation network to Q ^* (x _k ,u _k ) in the sense of least squares, then/> It can be expressed as:

Among them, W _c is the weight from the hidden layer to the output layer, φ _c (·) is the set of all activation functions in the hidden layer in the evaluation network, To evaluate the weight from the input layer to the hidden layer of the network, where,/> is the weight corresponding to the i-th activation function,/> Indicates the input value of each activation function corresponding to (x _k ,u _k ),/> Represents the input value of the i-th activation function;

Let the number of execution network activation functions be N _a , and note To perform network pairs in a least squares sense/> The best approximation of , then/> It can be expressed as:

The input of the execution network is the system state, where W _a is the weight from the hidden layer to the input layer, φ _a (·) is the set of activation functions of the hidden layer of the execution network, is the weight from the input layer to the hidden layer, where,/> is the weight corresponding to the i-th activation function,/> Represents the input value of each activation function corresponding to x _k ,/> Represents the input value of the i-th activation function. For x _k+1 , there is

Estimation error calculation step S140:

best approximation and/> Instead of the exact values Q ^* (x _k ,u _k ) and/> The following estimation error can be obtained:

in, Indicates that the input is/> When , evaluate the input values of each activation function in the network, that is

Optimal weight calculation step S150:

Online learning is performed on the optimal weight W _c of the evaluation network and the optimal weight W _a of the execution network. It is assumed that at time k, the estimated values of W _c and W _a by the evaluation network and execution network are respectively and/> Among them, l≤k, that is, the learning process should be carried out after the behavioral strategy starts to generate state data, then the output of the execution network at time k can be expressed as:

Before the behavior policy u _k generates the next state x _k+1 , the execution network cannot give an estimate of W _a at time k+1. Therefore, the execution network's estimate of W _a at time k+1 still uses Then the output of the execution network at time k+1 is:

In the same way, when the input is (x _k , _uk ), the output of the evaluation network is:

When the input is When , the output of the evaluation network is:

in, _{Similarly ,} before generating _state So there are:

Substituting estimated values for true values yields the following estimation error:

For the weight of the evaluation network Adjust using gradient descent method,

For the weight of the execution network The importance weighting method is used for training, and the improved gradient descent method is used for training/> Make online adjustments,

When evaluating the weight of a network and the weight of the execution network/> After convergence, the output of the execution network is an approximation of the optimal controller.

Optionally, in step S130 of evaluating the network and executing the network introduction,

For the evaluation network, we will Set to a constant value and only need to adjust the weight from the hidden layer to the output layer;

For the execution network, the Set to a constant value and only adjust the weight from the hidden layer to the output layer.

Optionally, in the optimal weight calculation step S150:

For the weight of the evaluation network, the following gradient descent method is used to adjust the weight:

Among them, α>0 evaluates the learning rate of the network, Δφ _c (k) = φ _c (θ ₂ (k+1))-φ _c (θ ₁ (k)) is the regression vector, φ _c (k) = (1 +Δφ _c (k) ^T Δφ _c (k)) ² is the normalization term;

For the weight of the execution network, the importance weighting method is used for training, and the improved gradient descent method is used to The details of online adjustment are as follows:

Optionally, in the behavior strategy selection step S110, the behavior strategy is: u _k =u′ _k +n _k , where u′ is any feasible control strategy, selected according to the characteristics and experience of the controlled system, n _k To explore noise, n _k contains more, such as sine and cosine signals with enough frequencies or random signals with limited amplitude.

Optionally, the evaluation network and the execution network are feedforward neural networks with a single hidden layer. The inputs of the evaluation network used to approximate the Q-function are state and control inputs. The inputs of the execution network are system states, and the output is Multi-m-dimensional vector.

Optionally, the evaluation network and the execution network only adjust the weight from the hidden layer to the output layer. The weight from the input layer to the hidden layer is randomly generated before the learning process starts and remains unchanged during the learning process.

Optionally, the activation function of the evaluation network and the execution network is one of a hyperbolic tangent function, a sigmoid function, a linear rectifier, and a polynomial function.

The invention further discloses a storage medium for storing computer executable instructions, which is characterized by:

The computer-executable instructions, when executed by the processor, execute the online learning control method of the nonlinear discrete-time system.

The invention has the following advantages:

1. The present invention proposes an online learning control method suitable for general nonlinear discrete-time systems. This method does not require repeated iterations between strategy evaluation and strategy improvement, and can achieve real-time online learning of the optimal controller;

2. The present invention adopts an off-track policy learning mechanism, which effectively overcomes the problem of insufficient exploration of the state-policy space by the direct heuristic dynamic programming method; in addition, the execution network and the evaluation network can use any form of activation function.

3. Compared with the classic direct heuristic dynamic programming method, the online learning method proposed by this patent has better exploration capabilities for the state-policy space, and the activation function types of the execution network and evaluation network can be selected arbitrarily. Limited to the hyperbolic tangent function; compared with iterative methods such as policy iteration or value iteration, this method can achieve online learning of the optimal controller and does not require a system model, only the state data generated by the behavioral policy.

Description of the drawings

Figure 1 is a flow chart of an online learning control method for a nonlinear discrete-time system according to a specific embodiment of the present invention;

Figure 2 is a schematic diagram of an evaluation network of an online learning control method for a nonlinear discrete-time system according to a specific embodiment of the present invention;

Figure 3 is a schematic diagram of an execution network of an online learning control method for a nonlinear discrete-time system according to a specific embodiment of the present invention;

Figure 4 is an algorithm schematic diagram of an online learning control method for a nonlinear discrete-time system according to a specific embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It can be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for convenience of description, only some but not all structures related to the present invention are shown in the drawings.

The present invention first needs to consider the following optimal control problem of nonlinear discrete time system. Consider the following discrete-time system:

x _k+1 =F(x _k ,u _k ),x ₀ =x(0)

Among them, x _k is the system state and u _k is the system input. The system function F(x _k ,u _k ) is in the compact set is Lipschitz continuous and satisfies F(0,0)=0. Assume that the system is stabilized on Ω, that is, there is a control sequence u ₁ ,…,u _k ,… such that x _k →0. In addition, it is assumed that the system function F(x _k , _uk ) is unknown. The goal of optimal control of nonlinear systems is to find a feasible control strategy that stabilizes the system and at the same time minimizes the following value function:

According to the Bellman optimality principle, the optimal control strategy u ^* should satisfy the following Bellman equation:

stx _k+1 =F(x _k ,u _k )

Therefore, the optimal controller u ^* has the following expression:

Putting the above equation into Bellman equation, we get the following HJB equation:

Therefore, referring to Figure 1, an online learning control method for a nonlinear discrete-time system according to the present invention is shown, which includes the following steps:

Behavior strategy selection step S110:

According to the characteristics of the controlled object, existing experience is used to select the behavioral strategy u. The behavioral strategy is the control strategy actually applied to the controlled object in the learning process. Its main function is to generate system status data that needs to be used in the learning process.

After selecting the behavioral strategy, the optimal controller will be learned online.

Optimal Q-function definition step S120:

Define the optimal Q-function as follows:

Its physical meaning is: at time k, adopt the behavioral strategy u, and at all subsequent moments, adopt the optimal control strategy u ^* , that is, the target strategy. It can be seen from the definition of the optimal Q-function that the above formula can be equivalently expressed as :

best control It can be expressed as:/>

For linear systems, Q ^* (x _k ,u _k ) and are nonlinear functions about (x _k ,u _k ) and x _k respectively.

Evaluate the network and perform the network introduction step S130:

Considering that when the number of activation functions in the feedforward neural network is large enough, the feedforward neural network can achieve arbitrary precision approximation of smooth or continuous nonlinear functions, so the evaluation network and execution network are introduced to Q ^* (x _k ,u _k )and Perform online approximation, and the evaluation network and execution network are neural networks;

The evaluation network is used to learn the optimal Q-function Q ^* (x _k , _uk ), and the execution network is used to learn the optimal controller u ^* . Assume that the number of neural network activation functions in the evaluation network is N _c , and note is the best approximation of the evaluation network to Q ^* (x _k ,u _k ) in the sense of least squares, then/> It can be expressed as:

Among them, W _c is the weight from the hidden layer to the output layer, φ _c (·) is the set of all activation functions in the hidden layer in the evaluation network, To evaluate the weight from the input layer to the hidden layer of the network, where,/> is the weight corresponding to the i-th activation function,/> Indicates the input value of each activation function corresponding to (x _k ,u _k ),/> Represents the input value of the i-th activation function;

The present invention will Set to a constant value, so you only need to adjust the weights from the hidden layer to the output layer.

Similarly, let the number of execution network activation functions be N _a , and note To perform network pairs in a least squares sense/> The best approximation of , then/> It can be expressed as:

The input of the execution network is the system state, where W _a is the weight from the hidden layer to the input layer, φ _a (·) is the set of activation functions of the hidden layer of the execution network, is the weight from the input layer to the hidden layer, where,/> is the weight corresponding to the i-th activation function,/> Represents the input value of each activation function corresponding to x _k ,/> Represents the input value of the i-th activation function. For x _k+1 , there is

The present invention will also Set to a constant value and only adjust the weight from the hidden layer to the output layer.

Estimation error calculation step S140:

best approximation and/> Instead of the exact values Q ^* (x _k ,u _k ) and/> The following estimation error can be obtained:

in, Indicates that the input is/> When , evaluate the input values of each activation function in the network, that is

Optimal weight calculation step S150:

Online learning is performed on the optimal weight W _c of the evaluation network and the optimal weight W _a of the execution network. It is assumed that at time k, the estimated values of W _c and W _a by the evaluation network and execution network are respectively and/> Among them, l≤k, that is, the learning process should be carried out after the behavioral strategy starts to generate state data, then the output of the execution network at time k can be expressed as:

Before the behavior policy u _k generates the next state x _k+1 , the execution network cannot give an estimate of W _a at time k+1. Therefore, the execution network's estimate of W _a at time k+1 still uses Then the output of the execution network at time k+1 is:

In the same way, when the input is (x _k , _uk ), the output of the evaluation network is:

When the input is When , the output of the evaluation network is:

in, _{Similarly ,} before generating _state So there are:

Substituting the estimated value for the true value gives the following estimation error:

For the evaluation network, the goal is to make the estimation error e _k tend to 0 through online learning. Therefore, the weight of the evaluation network is adjusted using the following gradient descent method:

Among them, α>0 evaluates the learning rate of the network, Δφ _c (k) = φ _c (θ ₂ (k+1))-φ _c (θ ₁ (k)) is the regression vector, φ _c (k) = (1 +Δφ _c (k) ^T Δφ _c (k)) ² is the normalization term.

For the weight of the execution network The importance weighting method is used for training. The objective function of the execution network is defined as:/> Among them, the prediction error e _a (k) of the execution network is defined as:/> U _c is the desired final objective function. In the present invention, U _c =0, that is, the execution network should be minimized as much as possible during the learning process. Similarly, the following improved gradient descent method is used to // Make online adjustments:

β>0 is the learning rate of the execution network, Φ _a (k) = (1 + φ _a (θ ₄ (k)) ^T φ _a (θ ₄ (k))) ² is the normalization term.

It can be seen from the training process of the evaluation network and the execution network that all state data used in the learning process are generated by the behavioral policy u. When the weight of the evaluation network and the weight of the execution network/> After convergence, the output of the execution network is an approximation of the optimal controller.

For behavioral strategies:

In a specific embodiment, in the online learning process of the optimal controller, all state data used are generated by the behavioral strategy u. In order to ensure that the algorithm has a certain detection capability for the policy space, the state generated by the behavioral strategy The data needs to be rich enough and meet certain continuous excitation conditions to ensure the convergence of the algorithm. The behavioral strategy in the present invention is: u _k =u′ _k +n _k , where u′ is any feasible control strategy, usually selected according to the characteristics and experience of the controlled system, n _k is exploration noise, and n _k can include More, such as sine and cosine signals with sufficient frequencies or random signals with limited amplitude.

For evaluation network and execution network:

Both the evaluation network and the execution network in the present invention adopt a feedforward neural network with a single hidden layer, in which the inputs of the evaluation network used to approximate the Q-function are state and control inputs, and the outputs of both are scalars. The input of the execution network is also the system state, and its output is a multi-m-dimensional vector. During the learning process, the three neural networks only adjust the weights from the hidden layer to the output layer. The weights from the input layer to the hidden layer are randomly generated before the learning process starts and remain unchanged during the learning process. The activation functions of the three hidden layers of the neural network can be selected from commonly used hyperbolic tangent functions, sigmoid functions, linear rectifiers, polynomial functions, etc.

Referring to Figures 2 and 3, schematic diagrams of the evaluation network and the neural network are shown respectively.

Of course, the evaluation network and execution network of the present invention can also be selected as feedforward neural networks with multiple hidden layers, and the weights of all connection layers can also be adjusted during the learning process.

Referring to Figure 4, a schematic diagram of the principle of the online learning control method of the present invention is shown.

The invention further discloses a storage medium for storing computer executable instructions, which is characterized by:

The computer-executable instructions, when executed by the processor, execute the above-mentioned online learning control method of the nonlinear discrete-time system.

The invention has the following advantages:

1. The present invention proposes an online learning control method suitable for general nonlinear discrete-time systems. This method does not require repeated iterations between strategy evaluation and strategy improvement, and can achieve real-time online learning of the optimal controller;

2. The present invention adopts an off-track policy learning mechanism, which effectively overcomes the problem of insufficient exploration of the state-policy space by the direct heuristic dynamic programming method; in addition, the execution network and the evaluation network can use any form of activation function.

3. Compared with the classic direct heuristic dynamic programming method, the online learning method proposed by this patent has better exploration capabilities for the state-policy space, and the activation function types of the execution network and evaluation network can be selected arbitrarily. Limited to the hyperbolic tangent function; compared with iterative methods such as policy iteration or value iteration, this method can achieve online learning of the optimal controller and does not require a system model, only the state data generated by the behavioral policy.

Obviously, those skilled in the art should understand that the above-mentioned units or steps of the present invention can be implemented with a general-purpose computing device. They can be concentrated on a single computing device. Alternatively, they can use programs executable by the computer device. Codes are implemented, so that they can be stored in a storage device and executed by a computing device, or they are separately made into individual integrated circuit modules, or multiple modules or steps among them are made into a single integrated circuit module for implementation. As such, the invention is not limited to any specific combination of hardware and software.

The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be concluded that the specific embodiments of the present invention are limited to this. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, Below, several simple deductions or substitutions can be made, which should all be deemed to belong to the protection scope of the present invention as determined by the submitted claims.

Claims (8)

1. An online learning control method of a nonlinear discrete time system comprises the following steps:

behavior policy selection step S110:

according to the characteristics of the controlled object, an existing experience is utilized to select a behavior strategy u, wherein the behavior strategy is a control strategy which is actually applied to the controlled object in the learning process and is mainly used for generating system state data which is needed in the learning process;

optimal Q-function definition step S120:

the following optimal Q-function is defined:

the physical meaning is as follows: at time k, action policy u is taken, and at all times thereafter, optimal control policy u is taken ^* I.e. target strategy, as defined by the optimal Q-function, the above equation can be equivalently expressed as:

optimal controlCan be expressed as: />

For linear systems, Q ^* (x _k ,u _k) and respectively about (x) _k ,u _k) and x_k Is a nonlinear function of (2);

evaluation of network and execution of network introduction step S130:

introducing evaluation network and execution network to Q respectively ^* (x _k ,u _k) and performing online approximation, wherein the evaluation network and the execution network are neural networks;

the evaluation network is used for learning the optimal Q-function Q ^* (x _k ,u _k ) The execution network is used for learning the optimal controller u ^* Assume that the number of neural network activation functions in the evaluation network is N _c And recordEvaluating network pairs Q in the least squares sense ^* (x _k ,u _k ) Is>Can be expressed as:

wherein ,for hiding layer to output layer weights, +.>For evaluating the set of all activation functions in the hidden layer in the network +.>To evaluate the weight of the network input layer to the hidden layer, wherein +.>Weight corresponding to the ith activation function, < ->Representation (x) _k ,u _k ) Input value of the respective activation function, < ->An input value representing an ith activation function;

let N be the number of executing network activation functions _a And recordExecuting the network pair +.>Is>Can be expressed as:

executive netThe input to the network is the system state, wherein,for hiding layer-to-input layer weights, +.>For executing the set of network hidden layer activation functions, < ->For inputting layer-to-hidden layer weights, wherein +.>Weight corresponding to the ith activation function, < ->Represents x _k Input value of the respective activation function, < ->Representing the input value of the ith activation function, for x _k+1 There is->

An estimation error calculation step S140:

optimal approximation and />Instead of the exact value +.> and />The following estimation errors are available:

wherein ,the representation input is +.>When evaluating the input value of each activation function in the network, i.e

The optimal weight calculation step S150:

optimal weight W for evaluation network _c And performing an optimal weight W of the network _a On-line learning is performed, assuming that at time k, the network is evaluated and the network pair W is executed _c and W_a The estimated values of (a) are respectively and />Wherein l.ltoreq.k, i.e. the learning process is to be performed after the behavior strategy starts to generate state data, the output of the execution network at time k can be expressed as:

in action policy u _k Generating the next state x _k+1 Previously, the execution network has not been able to give the k+1 time versus W _a Thus, the network pair W is performed at time k+1 _a The estimated value of (2) is still adoptedThe output of the execution network at time k+1 is:

similarly, when the input is (x _k ,u _k ) The output of the evaluation network is:

when the input isThe output of the evaluation network is:

wherein ,also in the generation state x _k+1 The evaluation network could not give the k+1 time to W before _c So k+1 time evaluates the network pair W _c The estimated value of (2) is also +.>Thus, there are:

replacing the actual value with the estimated value yields the following estimation error:

weights for evaluation networksThe gradient descent method is adopted for adjustment,

weights for execution networkTraining is performed by importance weighting and modified gradient descent is used for ++>The on-line adjustment is carried out,

when evaluating the weight of a networkAnd performing a weight of the network->After convergence, the output of the execution network is the approximation of the optimal controller.

2. The online learning control method according to claim 1, characterized in that:

in the evaluation network and execution network introduction step S130,

for an evaluation network, it willSetting the weight from the hidden layer to the output layer only needs to be adjusted to be a constant value;

for the execution network, it willSet to a constant value, only the weights of the hidden layer to the output layer are adjusted.

3. The online learning control method according to claim 2, characterized in that:

in the optimum weight calculation step S150:

for evaluating the weight of the network, the following gradient descent method is adopted for adjustment, and the specific steps are as follows:

wherein, alpha is more than 0, the learning rate of the network is evaluated, and delta phi _c (k)＝φ _c (θ ₂ (k+1))-φ _c (θ ₁ (k) Is regression vector phi _c (k)＝(1+Δφ _c (k) ^T Δφ _c (k)) ² Is a normalization term;

for the weight of the execution network, training is performed by adopting an importance weighting method, and an objective function of the execution network is defined as follows:wherein the prediction error e of the network is performed _a (k) The definition is as follows: />U _c U in the present invention is the desired final objective function _c =0, i.e. the execution network is to minimize as much as possible during learningAnd adopts the improved gradient descent method to treat +.>The on-line adjustment is specifically as follows:

beta > 0 is the learning rate of the execution network, phi _a (k)＝(1+φ _a (θ ₄ (k)) ^T φ _a (θ ₄ (k))) ² Is a normalization item.

4. The online learning control method according to claim 3, characterized in that:

in the behavior policy selection step S110, the behavior policy is: u (u) _k ＝u′ _k +n _k Where u' is any one of the possible control strategies, n is selected according to the characteristics and experience of the controlled system _k To explore noise, n _k Is a random signal that contains more, e.g., sufficiently high frequencies of sine, cosine signals or of limited amplitude.

5. The online learning control method according to claim 3, characterized in that:

the evaluation network and the execution network are feedforward neural networks with single hidden layers, the input of the evaluation network for approximating the Q-function is a state and control input, the input of the execution network is a system state, and the output is a multi-m-dimensional vector.

6. The online learning control method of claim 5, wherein:

the evaluation network and the execution network only adjust weights from the hidden layer to the output layer, and the weights from the input layer to the hidden layer are randomly generated before the learning process starts and remain unchanged in the learning process.

7. The online learning control method of claim 5, wherein:

the activation function of the evaluation network and the execution network is one of a hyperbolic tangent function, a Sigmoid function, a linear rectifier, a polynomial function.

8. A storage medium storing computer-executable instructions, characterized by:

the computer executable instructions, when executed by a processor, perform the online learning control method of the nonlinear discrete time system in any one of claims 1-7.