CN102063640B - Robot Behavior Learning Model Based on Utility Difference Network - Google Patents

Abstract

The invention relates to a robot behavior learning model based on a utility differential network, which comprises a utility fitting network unit, a differential signal calculating network unit, a confidence evaluating network unit, an action decision network unit, an action correcting network unit and an action executing unit. The model realizes the offline learning process and the online decision process. The utility fitting network unit calculates and obtains a utility fitting value of a state after action is executed; the differential signal calculating network unit is used for calculating a differential signal; the confidence evaluating network unit outputs the confidence obtained by calculating to the action correcting network unit; the action decision network unit outputs an action selecting function; and the action correcting network unit corrects the action selecting function by utilizing confidence, calculates a probability value selected by each action and outputs the action with largest probability to the action executing unit for executing. The invention can more favorably ensure the completeness of a robot for obtaining environmental knowledge and more favorably ensure the timeliness and effectiveness of robot behavior decision.

Description

Robot Behavior Learning Model Based on Utility Difference Network

technical field

The invention relates to a robot behavior learning model based on a utility difference network, which belongs to one of new applications in the field of artificial intelligence.

Background technique

Robot intelligent behavior generally refers to the process in which a robot performs reasoning and decision-making on the basis of perceiving the surrounding environment to achieve behavioral intelligent decision-making. The establishment of an intelligent behavior decision-making model requires the acquisition, representation and reasoning of knowledge, and the ability to automatically evaluate the pros and cons of robot behavior. At present, the cognitive behavioral model based on reinforcement learning technology has advantages in knowledge acquisition, adaptability to decision-making environment, and reusability, making it the first choice for intelligent behavioral modeling.

The reinforcement learning process requires exploration of the environment. It can be expressed as: In a certain state, the decision maker chooses and executes an action, and then perceives the next environment state and the corresponding reward. Decision makers are not directly told what actions to take under what circumstances, but modify their behavior according to the rewards to win more rewards. Simply put, the reinforcement learning process is the process that allows the decision maker to obtain the best sequence of actions through repeated trials.

At present, the behavioral decision-making of robot reinforcement learning is mostly based on specific knowledge or rules. The disadvantage of this method is that the knowledge acquisition is limited, and the second is that the knowledge acquired by the problem is often empirical and cannot learn new ones in time. The third is that the reasoning process is not real-time.

Contents of the invention

Aiming at the shortcomings of the behavior decision-making of the current robot reinforcement learning, the invention establishes a robot behavior learning model based on utility difference network. The model is an evaluation-based learning system. Through the interaction with the environment, the control rate of the system is automatically generated, and then the control is given to select actions. The robot behavior learning model based on the utility difference network of the present invention solves the problem of limited knowledge acquisition and too strong experience of the general behavior decision-making model, realizes the offline learning process and online decision-making process, and solves the problem of low real-time reasoning process.

并输出给差分信号计算网络单元；差分信号计算网络单元根据输入的效用拟合值

以及根据状态空间向量s_t计算的立即回报函数，进一步计算得到差分信号ΔTD_t，并将该差分信号ΔTD_t输出给效用拟合网络单元、置信度评价网络单元以及动作决策网络单元；效用拟合网络单元利用差分信号ΔTD_t更新效用拟合网络单元中神经网络的权值；置信度评价网络单元利用效用拟合网络单元中神经网络的输入层的输入向量和隐层的输出向量以及差分信号，计算动作决策结果的置信度，并将该置信度输出给动作校正网络单元；动作决策网络单元根据输入的差分信号ΔTD_t与状态空间向量s_t，进行动作的选择学习，输出动作选择函数

给动作校正网络单元，其中j、k为大于0的整数；动作校正网络单元利用输入的置信度，对输入的动作选择函数

进行校正，然后计算校正后的动作的选取概率值，将概率最大的动作输出给动作执行单元执行，该动作执行后的状态空间向量再反馈输入给效用拟合网络单元、差分信号计算网络单元和动作决策网络单元。A robot behavior learning model based on a utility differential network, comprising: a utility fitting network unit, a differential signal calculation network unit, a confidence evaluation network unit, an action decision network unit, an action correction network unit and an action execution unit; The fitting network unit is used to calculate the utility fitting value obtained by the state space vector s _t generated by the action a _t executed by the action execution unit at time t

And output to the differential signal calculation network unit; the differential signal calculation network unit fits the value according to the input utility

And according to the immediate reward function calculated by the state space vector s _t , the difference signal ΔTD _t is further calculated, and the difference signal ΔTD _t is output to the utility fitting network unit, the confidence evaluation network unit and the action decision network unit; utility fitting The network unit uses the differential signal ΔTD _t to update the weight of the neural network in the utility fitting network unit; the confidence evaluation network unit utilizes the input vector of the input layer of the input layer of the neural network in the utility fitting network unit, the output vector of the hidden layer and the differential signal, Calculate the confidence degree of the action decision result, and output the confidence degree to the action correction network unit; the action decision network unit performs action selection learning according to the input differential signal ΔTD _t and the state space vector s _t , and outputs the action selection function

Give the action correction network unit, where j and k are integers greater than 0; the action correction network unit uses the confidence of the input to select a function for the input action

Perform correction, then calculate the selected probability value of the corrected action, output the action with the highest probability to the action execution unit for execution, and then feed back the state space vector after the action is executed to the utility fitting network unit, differential signal calculation network unit and Action Decision Network Unit.

通过动作选择器输出最终选择的动作给动作执行单元执行，执行动作后得到的状态空间向量再输入给动作决策网络单元。The learning model has two processes: an offline learning process and an online decision-making process; in the offline learning process, each of the above-mentioned units must participate, and in the online decision-making process, only the action decision network unit finally obtained by offline learning Participating with the action execution unit, the action decision network unit in the online decision-making process calculates according to the state space vector s _t after the action execution at time t and obtains the output action selection function

The final selected action is output through the action selector to the action execution unit for execution, and the state space vector obtained after the action is executed is then input to the action decision network unit.

Advantage of the present invention and beneficial effect are:

(1) The robot learning model of the present invention does not need to calculate and generate correct actions, but solves the problem of difficult robot knowledge acquisition in the learning environment of action-environment interaction-evaluation. Since this learning model does not need to explicitly specify the environment model, the causal relationship of the environment has been implicit in the specific differential feedback network, which can better ensure the completeness of the robot's acquisition of environmental knowledge;

(2) The offline learning process designed by this model can complete the environmental knowledge learning process before the robot makes a decision, and the online decision-making process can further complete the acquisition of the robot's environmental knowledge. The decision-making at runtime does not carry out exploration and learning activities, and only needs to use the reconstructed The network performs calculation and addition, and this offline and online model design ensures that the robot's behavior decision-making has better real-time performance, and better guarantees the timeliness and effectiveness of the robot's behavior decision-making.

Description of drawings

Fig. 1 is a schematic structural diagram of the offline learning process of the first embodiment of the learning model of the present invention;

Fig. 2 is a schematic diagram of the action decision network flow of the first embodiment of the learning model of the present invention;

Fig. 3 is a schematic diagram of the genetic operator coding structure in the action decision network in the first embodiment of the learning model of the present invention;

Fig. 4 is a schematic diagram of the genetic operator crossover operation in the action decision network in the first embodiment of the learning model of the present invention;

Fig. 5 is a schematic diagram of the online decision-making process in the second embodiment of the learning model of the present invention.

Detailed ways

The present invention will be further described in detail with reference to the accompanying drawings and embodiments. Among them, the first embodiment specifically describes the offline learning process of the learning model of the present invention; the second embodiment describes the online decision-making process.

As shown in FIG. 1 , the learning model of the present invention includes five parts: a utility fitting network unit 11 , a differential signal calculation network unit 12 , a confidence evaluation network unit 13 , an action decision network unit 14 and an action correction network unit 15 . In the off-line learning process of the learning model of the present invention, five parts are all involved.

并输出效用拟合值

给差分信号计算网络单元12，差分信号计算网络单元12输出差分信号ΔTD_t给置信度评价网络单元13和效用拟合网络单元11。效用拟合网络单元11再利用差分信号计算网络单元12输入的差分信号ΔTD_t来不断更新，从而达到真实的效用拟合。The utility fitting network unit 11 is used to calculate the utility fitting value obtained from the different state space vectors s _t generated by the action a _t selected at time t after being executed by the action execution unit 16

and output the utility fitted value

To the differential signal calculation network unit 12 , the differential signal calculation network unit 12 outputs the differential signal ΔTD _t to the confidence evaluation network unit 13 and the utility fitting network unit 11 . The utility fitting network unit 11 utilizes the differential signal ΔTD _t inputted by the differential signal calculation network unit 12 to continuously update, so as to achieve real utility fitting.

以及根据状态空间向量s_t计算的立即回报函数，进一步计算得到差分信号ΔTD_t，并将该差分信号ΔTD_t输出给效用拟合网络单元11、置信度评价网络单元13以及动作决策网络单元14。The differential signal calculation network unit 12 fits the value according to the input utility

And according to the immediate reward function calculated by the state space vector _st , the difference signal ΔTD _t is further calculated, and the difference signal ΔTD _t is output to the utility fitting network unit 11, the confidence evaluation network unit 13 and the action decision network unit 14.

The confidence evaluation network unit 13 uses the input vector of the input layer of the neural network in the utility fitting network unit 11, the output vector of the hidden layer and the difference signal ΔTD _t to calculate the confidence of the action decision result, and outputs the confidence to the action correction The network unit 15 is used for adjusting the action selection.

给动作校正网络单元15，其中j、k为大于0的整数。The action decision network unit 14 uses the hierarchical genetic algorithm to optimize the neural network according to the input differential signal ΔTD _t and the state space vector s _t , realizes action selection learning, and outputs an action selection function

Give the action correction network unit 15, where j and k are integers greater than 0.

进行校正，将概率最大的动作输出。动作执行后的状态空间向量再反馈输入给效用拟合网络单元11、差分信号计算网络单元12和动作决策网络单元14。The action correction network unit 15 uses the confidence of the input to select a function for the input action

Make corrections and output the action with the highest probability. The state space vector after the action is executed is fed back to the utility fitting network unit 11 , the differential signal calculation network unit 12 and the action decision network unit 14 .

Among them, the utility fitting network unit 11 is used to evaluate the utility of the state change caused by a specific behavior, and obtain the utility fitting value, which is composed of a two-layer feedback neural network, as shown in FIG. 1 . The input of the neural network is the state space vector s _t , the activation function of the hidden layer is the Sigmoid function, the output of the neural network is the utility fitting value of the state after the action is executed, and the weight coefficients of the neural network are A, B and C (. The neural network Contains n input vector units, and h hidden layer units, each hidden layer unit accepts n inputs and has n connection weights, and the output unit accepts n+h inputs and has n+h weights. For h The value of can be set by the user, generally it is set to 3, and it is set to 2 in the embodiment of the present invention.

The input vector of the neural network is x _i (t), i=1, 2, 3...n, the function x _i (t) is obtained by normalizing _st , then the output vector of the hidden layer unit is:

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; [</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2,3</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; h</span>}$

a_ij(t)为输入层与隐层的权值A的向量，。效用拟合网络11的输出为对效用的拟合值

它是对输入层和隐层的线性组合：The functions used in the above formula

a _ij (t) is the vector of the weight A of the input layer and the hidden layer,. The output of utility fitting network 11 is the fitted value of utility

It is a linear combination of the input and hidden layers:

$\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; h</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>$

Among them, b _i (t) represents the vector of the weight B of the input layer and the output layer, and c _j (t) represents the vector of the weight C of the hidden layer and the output layer.

The weights A, B, and C of the network are updated using the differential signal ΔTD _t . If the differential signal ΔTD _t is positive, it means that the last action had a positive effect, so the chance of this action being selected should be strengthened. The weight B of the input layer and the output layer and the weight C of the hidden layer and the output layer are updated using the following formula:

_bi (t+1)= _bi (t)+λ·ΔTD _t+1 · _xi (t), i=1, 2, 3...n

c _j (t+1)=c _j (t)+λ·ΔTD _t+1 y _j (t), j=1, 2, 3...h

In the formula, λ is a constant greater than zero, which can be set by the user. The update of the weight A of the input and hidden layer is carried out according to the following formula:

a _ij (t+1)＝a _ij (t)+λ _h ∆TD _t+1 y _j (t) sgn(c _j (t)) x _i (t)

Among them, λ _h is a number greater than zero, which can be set by the user. ΔTD _t+1 represents the difference signal of the state space vector corresponding to the execution of the action at time t+1, and sgn is the following function:

$\left(z\right)=\left\{\begin{array}{cc}1& z>0\\ 0& z=0\\ -1& z<0\end{array}\right.,$ Here z is the vector c _j (t) of weight C.

As shown in Figure 1, the differential signal calculation network unit 12 is based on the fitting utility output by the utility fitting network unit 11 And the immediate reward function R(s _t ) of the state is calculated to obtain the differential signal ΔTD _t . According to the instantaneous difference algorithm, ΔTD _t is obtained by iterative calculation using the following formula:

$\mathrm{<span\; class="notranslate">\; \&Delta;T</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}<span\; class="notranslate">\; \&CenterDot;</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>$

表示t+1时刻动作执行后产生的状态空间向量s_t+1所得到的效用拟合值，

表示t时刻动作执行后产生的状态空间向量s_t所得到的效用拟合值。Among them, R(s _t ) is the immediate evaluation of the state s _t , which is the output of the immediate reward function, and γ is the discount coefficient, which can be set by the user.

Indicates the utility fitting value obtained by the state space vector s _t+1 generated after the execution of the action at time t+1,

Indicates the utility fitting value obtained from the state space vector s _t generated after the execution of the action at time t.

The calculated differential signal ΔTD _t is used to train and update the weight coefficients of the utility fitting network unit 11 and the confidence evaluation network unit 13 . If the differential signal ΔTD _t has a positive effect, this action should be strengthened, and its confidence should also be strengthened, that is, it is more believed that this action should be selected. In addition, the differential signal ΔTD _t is also used to update the weight of the action selection function in the action decision network unit 14, so as to ensure the selection of the optimal action.

As shown in FIG. 1 , when the action decision network unit 14 outputs the action decision function, the confidence evaluation network unit 13 needs to calculate the confidence of the output action, and the confidence is used to adjust the action selection. The input of the confidence evaluation network unit 13 is the state vectors x _i (t) and y _j (t), which are derived from the hidden layer and the output layer of the utility fitting network unit 11 .

Confidence p ₀ (t) is calculated by the following formula:

${\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; h</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>$

Among them, the weights α _i (t) and β _j (t) are updated using the following formula:

α _i (t+1)=α _i (t)+λ _p ΔTD _{t +1} xi (t), i=1, 2, 3...n

β _j (t+1)=β _j (t)+λ _p ΔTD _t+1 y _j (t), j=1, 2, 3...h

Among them, λ _p represents the learning rate, which is a value between 0 and 1, and the experience value is 0.618. Users can set it according to their own experience. From the above formula, it is difficult to ensure that the confidence interval of p ₀ (t) is [0, 1], so the Sigmoid function is introduced to transform p ₀ (t) to obtain p(t), so that the output confidence is the same as Random function probabilities coincide:

$\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; a</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}}$

The confidence correction factor a plays a role in smoothing the learning process. Changing a can change the adjustment range of learning to the environment. If a is too large, the learning system will lose its adjustment function. A suitable a should be set according to prior knowledge Value, a > 0, the value range of a in the present invention is [1, 10].

进行校正，校正过程在动作校正网络单元15里完成。The moderating effect of confidence on action choice reflects the uncertainty of decision-making. It can be seen that as the utility of the state gradually tends to the true value, that is, the increase of ΔTD _t , the confidence p(t) also gradually increases, and the choice of action becomes more and more certain. Reuse the output confidence p (t) to each output action selection function of the action decision network unit 14

Correction is performed, and the correction process is completed in the action correction network unit 15.

分别为第h层的第i个节点的输入和输出，i为每层的节点，其中，第一层节点数为I个，第二层节点数为I*J个，第三层节点数为L个，第四层节点数为K个，I，J，K，L都是正整数。均值m_ij，方差σ_ij分别为第二层中对应x_i(t)输入的第j个节点的高斯隶属函数的位置参数和宽度。Action decision network unit 14 adopts neural network to realize, and it is divided into four layers altogether, as shown in Figure 1, the first layer to the fourth layer are respectively: input layer, fuzzy subset layer, variable node layer and function output layer, Among them, the variable node layer is also called the function fitting layer. Use h=1, 2, 3, 4 to represent the four layers of the network respectively. set up

are the input and output of the i-th node of the h-th layer, i is the node of each layer, wherein, the number of nodes in the first layer is I, the number of nodes in the second layer is I*J, and the number of nodes in the third layer is L, the number of nodes in the fourth layer is K, and I, J, K, L are all positive integers. The mean m _ij and the variance σ _ij are respectively the position parameter and width of the Gaussian membership function of the jth node corresponding to the input of x _i (t) in the second layer.

为：In the input layer of the neural network of the action decision network unit 14, the input quantity is _xi (t) obtained by normalizing the state space vector _st , which represents the situation information of the robot at the input moment. The input of the i-th node of the input layer

for:

$\mathrm{<span\; class="notranslate">\; I</span>}{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="i"\; filenames="HDA0000034814170000011.png,HDA0000034814170000021.png"\; state="\{\{state\}\}">i</figure-callout></span>}}^{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2,3</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; I</span>}$

The fuzzy subset layer is used to fuzzify the input variables of the input layer. The output is the degree of membership for each input vector. Each x _i (t) of the input layer corresponds to J inputs in the fuzzy subset layer, for example, in Figure 1, where J is 2, where each input is a fuzzy subset of x _i (t), The output is the degree of membership of x _i (t) in this fuzzy subset. Its activation function of each node is a Gaussian membership function, and the output is:

${{\mathrm{<span\; class="notranslate">\; Q</span>}}_{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; j</span>}}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2,3</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2,3</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; J</span>}$

为对应于输入x_i(t)的第j个输出，，exp是以自然对数e为底的指数函数，x_j(t)为输入层的第j个节点的输入。in,

is the jth output corresponding to the input x _i (t), exp is an exponential function with natural logarithm e as the base, and x _j (t) is the input of the jth node of the input layer.

In order to satisfy the fitting of the action function, the neural network needs to adjust the output to a certain extent, and the variable node layer is used to realize this adjustment function. The variable node layer realizes the adjustment function through changes in the number of nodes and connection weights. The number of nodes and connection weights are optimized using a hierarchical genetic algorithm, and their number and size are dynamically determined to meet the network's fitting of the action function. , which will be introduced later. The activation function of the variable node layer is a Gaussian function, and its position parameter and width are m _l and σ _l respectively. The number of connections between the second layer and the third layer is also uncertain and needs to be dynamically adjusted during the optimization process, and the connection weights are all 1. The output of the third layer node is:

${\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; (</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; J</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{{\mathrm{<span\; class="notranslate">\; o</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}_{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2,3</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; L</span>}$

The number of nodes is the same as the number of optional actions, and the output of the function output layer is the fitting value of the action function, which is used to calculate the selection probability of each action. The output of the fourth layer node is:

${{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="4"\; label="o\; k"\; filenames="HDA0000034814170000011.png"\; state="\{\{state\}\}">o</figure-callout></span><figure-callout\; id="4"\; label="o\; k"\; filenames="HDA0000034814170000011.png"\; state="\{\{state\}\}">\; </figure-callout>}}_{\mathrm{<span\; class="notranslate">\; </span>}}}^{\mathrm{<span\; class="notranslate">\; 4</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; K</span>}$

Among them, the output O _k ⁴ of the fourth layer is the action selection function

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}^{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; K</span>}$

Every node in the third layer is connected to the fourth layer, ω _lk is the connection weight between the lth node in the third layer and the kth node in the fourth layer, and the connection weight ω _lk also needs to be dynamically adjusted during the optimization process .

Assuming that the first layer of the network has I inputs, and the i-th input has k _i fuzzy divisions in the second layer, then there are k ₁ +k ₂ +...+k _I nodes in the second layer, and the node functions are The membership function of the input for its fuzzy subset. To sum up, the neural network structure that needs to be dynamically adjusted and optimized is: the number of nodes in the third layer, and the number of connections between the second layer and the third layer. The network parameters to be adjusted and optimized are: the position m _ij and width σ _ij of the membership function of the input parameter of the second layer, the position parameter m _l and width σ _l of the Gaussian activation function of the third layer (hidden layer), and the third layer and the fourth layer Layer connection weight ω _lk .

Here, the hybrid hierarchical genetic algorithm is used to optimize the structure and parameters of the neural network in the action decision network. The network structure optimization is to determine the number of nodes in the third layer and the number of connections between the second layer and the third layer. The parameter optimization of the network includes the position parameter m _ij and width σ _ij of the membership function of the input vector, the position parameter m _l and width σ _l of the Gaussian function of the hidden node in the third layer, and the connection weight ω between the third layer and the fourth layer _lk . The hierarchical genetic algorithm is used to optimize and adjust the neural network, so that the network can continuously optimize the action selection function according to the change of the input differential signal in each round of decision-making, so as to realize the action selection function.

进行校正，然后计算得到每个动作选取的概率值，将概率最大的动作输出。The action correction network unit 15 uses the evaluation value output by the confidence evaluation network unit 13, that is, the confidence degree p(t) of the action, to the action selection function output by the action selection network unit 14

Perform correction, and then calculate the probability value of each action selection, and output the action with the highest probability.

为均值，以p(t)为概率生成一个随机函数，作为新的动作选择函数A_j(s_t)。p(t)越小，则A_j(s_t)就越远离

反之，则越靠近以新的A_j(s_t)代替

The calibration process is based on

As the mean value, generate a random function with p(t) as the probability, as the new action selection function A _j (s _t ). The smaller p(t) is, the farther A _j (s _t ) is from

On the contrary, the closer Replace with new A _j (s _t )

The greater the value of the action selection function A _j (st _t ), the greater the probability that the corresponding action a _j will be selected. The formula for calculating the probability of selection is:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; e</span>}}^{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}}{\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}^{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}}$

Then the output is the action with the highest probability value.

In the robot behavior learning model, the action decision network unit 14 also includes four subunits: an encoding unit 141, a population initialization unit 142, a fitness function determination unit 143, and a genetic operation unit 144, as shown in FIG. 2 .

The encoding unit 141 determines the chromosome structure of the genetic algorithm. Hierarchical genetic algorithm is proposed based on the hierarchical structure of biological chromosomes. Genes in chromosomes in organisms can be divided into regulatory genes and structural genes. The role of regulatory genes is to control whether structural genes are activated. Here, referring to this characteristic of biological chromosome genes, the above optimization problem is coded. Each individual in the population consists of two parts that determine the structure and parameters of the network. The genetic structure of population individuals adopts a two-level hierarchical structure encoding, that is, it is realized in two layers according to the gene hierarchy of biological chromosomes, and the upper layer genes realize the encoding of the number of nodes in the third layer and the input membership function of the second layer, that is, the third layer The number of nodes and the parameters m _ij and σ _ij of the second layer input membership function. As shown in Figure 3, the part that controls the number of nodes in the third layer (hidden layer) is called the control gene, and the lower layer is the parameter gene, which realizes the membership function of the nodes in the third layer (hidden layer) and the encoding of network connections. Including the third layer (hidden layer) node membership function parameters m _l and σ _l , the number of connections between the second layer and the third layer, and the connection weight ω _lk between the third layer and the fourth layer.

The number of hidden nodes controlling the gene and the gene representing the network connection of the parameter gene are all encoded in binary, and "0" and "1" are used to represent "no" and "yes" respectively. Other genes that represent membership function parameters and connection weights are coded by real values, that is, expressed by real numbers. The third layer structure is coded as a binary string, one bit represents a node in the third layer, as a control gene, "1" indicates that the node works, and "0" indicates that the node does not work. In this way, the number of "1" in the control gene string is the actual number of hidden layer nodes of the neural network. Among the parameter genes, the second and third layer connection genes are coded in binary, "1" indicates that the corresponding second layer is connected to the third layer, and "0" indicates that the corresponding second layer is not connected to the third layer. The weight genes of the third and fourth layers are encoded by real values, which represent the connection weights of the third layer and the fourth layer.

It can be seen that the control gene controls the number of nodes. If a node is "0", then this node has no connection with the two layers before and after, and accordingly its corresponding parameter genes do not exist. It can be seen that , the parameter gene is controlled by the control gene, if a certain node of the upper control gene does not exist, then the corresponding lower parameter gene will not be activated, which just reflects the control function of the control gene, and this control function can be compared with the network corresponding to the topology. The encoded chromosomes form a population, and they are used to complete evolution.

Further, the population initialization unit 142 initializes the chromosome population. In order to run the genetic algorithm smoothly, a certain number of chromosome individuals need to be generated before, and these individuals should be randomly generated, representing the possibility of various network structures, that is, there should be enough solution space. Appropriate population size is of great significance to the convergence of genetic algorithm. If the population size is too small, it is difficult to obtain satisfactory results. If the population size is too large, the calculation will be complicated. The population size is generally 10-160.

Further, the fitness function unit 143 of the chromosome is determined. The fitness function of the individual is expressed by the individual error and the complexity of the structure, and the complexity of the control network is considered while optimizing the individual error, so as to obtain the optimal network structure. The fitness function of the network has the following form:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; I</span>}$

Among them, E(i), H(i) represent the individual error and structural complexity of the i-th individual, respectively, where:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

H(i)＝1+exp[-c(N _i (0))]

和y_ij为第i个个体的第j个输出和期望输出，其中，期望输出y_ij为期望动作的选择函数

如果期望输出某个动作，则设它的期望值

其他期望动作函数都设为0。N_i(0)为第i个个体的隐层节点为零的数目，c为参数调节因子。其中，b，c为常值，α与β为大于零的常数，α+β＝1。利用这样的适应值函数可保证在优化网络权值的同时得到合适的神经网络结构。

and y _ij are the jth output and expected output of the i-th individual, where the expected output y _ij is the selection function of the expected action

If an action is expected to be output, set its expected value

All other expected action functions are set to 0. N _i (0) is the number of hidden layer nodes of the i-th individual that is zero, and c is the parameter adjustment factor. Wherein, b and c are constant values, α and β are constants greater than zero, and α+β=1. Utilizing such a fitness function can ensure that a suitable neural network structure is obtained while optimizing the network weight.

Further, a genetic operation unit 144 is performed, and the genetic operation includes selection, crossover and mutation. The initial population, after selection, crossover, and mutation, undergoes a round of genetic operations, completes a round of evolution, and obtains a new generation of subpopulations, and repeats this process so that the evolution continues, so that the offspring converge to the optimal population. excellent.

Selection is to select some excellent individuals from the previous generation population and inherit them to the next generation population according to the fitness of individuals and according to certain rules or methods. The algorithm adopts the method of elite selection for selection, that is, according to the size of the fitness value, the best individual in each generation population is retained to the next generation, which ensures the gradual convergence of the algorithm. For individual i, its selection probability is:

${\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}$

Among them, _fi is the fitness of individual i, and N is the number of individuals in the population.

The crossover operation is to randomly exchange the corresponding bits of the genes of two individuals. This process reflects random information exchange, and the purpose is to generate new gene combinations, that is, to generate new individuals. When the evolution reaches a certain level, especially when a population with most of the same individuals appears, new individuals cannot be produced by crossover, and new individuals can only be produced by mutation. Mutation is to change the gene position with a certain probability to increase a new search space, that is to say, mutation increases the characteristics of global optimization. In the process of crossover and mutation, randomness plays an important role. Only random crossover and mutation operations can guarantee the appearance of updated individuals, and this randomness is expressed through the probability of crossover and mutation.

In the process of genetic operation, crossover probability and mutation probability have great influence on the performance of genetic algorithm. If the crossover probability is selected to be large and the mutation probability is selected to be small at the initial stage of the Genetic Algorithm (GA) operation, the convergence speed of the algorithm can be accelerated, which is conducive to searching for the optimal solution. However, as the search progresses, it is necessary to reduce the crossover probability and increase the mutation probability, so that the algorithm is not easy to fall into the local extremum and can search for new solutions.

At the same time, the mutation probability cannot be too large, otherwise the algorithm will be difficult to converge and destroy the genes of the optimal solution. For solutions with high fitness, lower crossover probability and mutation probability should be chosen to make it more likely to enter the next generation; for solutions with lower fitness, higher crossover probability and mutation probability should be chosen so that It is eliminated as soon as possible; when mature convergence occurs, the probability of crossover and mutation should be increased to speed up the generation of new individuals. According to the above selection principles of crossover and mutation probability, an adaptive method of crossover probability and mutation probability is adopted, and its calculation formula is:

${\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\frac{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; avg</span>}}}{\mathrm{<span\; class="notranslate">\; f</span>}}& <span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; avg</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; f</span>}\\ \mathrm{<span\; class="notranslate">\; 0.8</span>}& <span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; avg</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; f</span>}\end{array}\right.$

${\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\frac{\mathrm{<span\; class="notranslate">\; 0.2</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; avg</span>}}}& <span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; avg</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; 0.2</span>}& <span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; avg</span>}}<span\; class="notranslate">\; )</span>\end{array}\right.$

Among them, p _c is the crossover probability, and p _m is the mutation probability. f _max is the maximum fitness in the population, f _avg is the average fitness, f is the greater fitness of the two crossed individuals, and f' is the fitness of the mutant individual.

This method can quickly find the optimal solution when the evolution space is large; it can increase the diversity of the population when it converges to the local optimal solution. It can be seen that the mutation probability of the individual with the greatest fitness is zero, and the probability of crossover and mutation of the individual with greater fitness is very small, which protects the excellent individuals. However, individuals with low fitness have a high probability of crossover and mutation, so it needs to be constantly destroyed.

According to the crossover probability, the crossover operation is performed between the two selected individuals, and the crossover operation is performed on the corresponding parts of the control gene and the parameter gene, as shown in Figure 4. Such a crossover operation can make the corresponding genes of the two chromosomes to be crossed over, and also ensures the corresponding crossover of the binary-coded and real-coded genes. The crossover of the corresponding positions of two chromosomes adopts single-point crossover, randomly selects the same position of two individuals, and performs gene exchange operation at the selected position.

The mutation operation includes the operation of all genes. For the control gene and the binary coded gene in the parameter gene, the bit mutation is used to perform the logical inversion operation, that is, to change "1" into "0" and "0" into "1" ". Gaussian mutation for linear combinations of real-valued coded genes:

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0,1</span>}<span\; class="notranslate">\; )</span>$

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0,1</span>}<span\; class="notranslate">\; )</span>$

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0,1</span>}<span\; class="notranslate">\; )</span>$

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0,1</span>}<span\; class="notranslate">\; )</span>$

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; lk</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0,1</span>}<span\; class="notranslate">\; )</span>$

Among them, α is the evolution rate, f is the fitness of each individual, N(0,1) is a normal distribution random function with expectation 0 and standard deviation 1.

To sum up, the algorithm steps of hierarchical genetic algorithm to realize neural network optimization are as follows:

1. Encode the network structure and parameters according to the hierarchical structure to generate chromosome individuals.

2. Randomly generate 2N initial chromosome populations, and set the evolution algebra to t=0.

3. Calculate the fitness value of each individual, the maximum fitness value and the average fitness value in the population according to the formula.

4. Select N individuals in the population as parents according to the individual selection probability, let t=t+1.

5. Randomly select two individuals from the parent generation, and perform the crossover operation according to the crossover probability. If it crosses, first copy the two bodies, and keep the original body. Perform crossover operation with duplicated individuals to generate two new individuals. Until the crossover of the parent population is completed.

6. Perform mutation operations on all individuals according to the mutation probability.

7. When the fitness of the optimal individual and the fitness of the group reach a given threshold, or reach the maximum evolution algebra, the iterative process of the algorithm converges and the algorithm ends. Otherwise, go to 3 and continue to execute until the end condition is met.

After the optimization is finished, the network structure and parameters of the optimal individual are taken as the decision-making network, and it is used to realize the calculation of action decision-making.

In the action decision network unit 14, a hierarchical genetic algorithm is used to optimize the structure and parameters of the network. After each new situation appears, first use the differential signal ΔTD _t provided by the Temporal-Difference method (TD) to update the parameters of the action selection network in order to obtain more favorable optional actions. Specifically, it uses the differential signal ΔTD _t to update the connection weights of the third layer and the fourth layer in each parameter gene of the chromosome in the population, and then performs genetic operations. In this way, the weight space corresponding to this action function has been updated, and the new weight value of the corresponding action obtained through inheritance should also be larger, which can reflect the learning of this optimal action. The update process of the differential signal for the connection weight is:

是加权系数，是0-1之间的数值，经验值是0.62。Among them, ω _ij is the connection weight of the i-th hidden node in the third layer and the j-th action selection function in the fourth layer,

It is a weighting coefficient, a value between 0-1, and the empirical value is 0.62.

In this embodiment, a hierarchical genetic algorithm is used to train the neural network to realize knowledge learning. It solves the reactive way in which behavior decision-making research in the prior art is mostly based on specific knowledge or rules, and better solves the knowledge acquisition and reasoning decision-making problems of robot behavior decision-making. The subject approaches the completeness of knowledge through interactive learning with the environment , with higher-level learning and reasoning abilities.

通过动作选择器输出最终选择的动作，该动作经动作执行单元16执行后得到的状态空间向量再输入给动作决策网络单元14。Fig. 5 is a schematic diagram of the online decision-making process in the second embodiment of the learning model of the present invention. After offline learning, the finally obtained action decision network unit 14 is optimal, and this action decision network unit 14 is used for real-time online decision-making. Others, such as utility fitting network unit 11 , differential signal calculation network unit 12 , confidence evaluation network unit 13 and action correction network unit 15 are removed in the online decision-making process and are no longer used. The action decision network unit 14 performs calculations according to the state space vector _st after the selected action _at is executed by the action execution unit 16, and obtains the output action selection function

The finally selected action is output through the action selector, and the state space vector obtained after the action is executed by the action execution unit 16 is then input to the action decision network unit 14 .

In this embodiment, the trained neural network is used to make real-time decision-making on the behavior of the robot. The separation of learning process and decision-making process ensures the efficiency of online decision-making and meets the needs of real-time operation.

Claims (1)

1. A modeling apparatus of a robot behavior learning model based on a utility difference network, comprising an action execution unit (16), characterized by further comprising: the device comprises a utility fitting network unit (11), a differential signal calculation network unit (12), a confidence evaluation network unit (13), an action decision network unit (14) and an action correction network unit (15);

the utility fitting network unit (11) is used for calculating action a at the moment t_tA state space vector s generated after execution by the action execution unit (16)_tThe obtained utility fitting value

And output to the differential signal calculation network unit (12); the difference signal computing network unit (12) fits values according to the utility of the input

And according to the state space vector s_tThe calculated immediate return function is further calculated to obtain a differential signal delta TD_tAnd the differential signal Δ TD is converted into a differential signal_tThe confidence coefficient is output to a utility fitting network unit (11), a confidence coefficient evaluation network unit (13) and an action decision network unit (14); the utility fitting network unit (11) uses the differential signal Δ TD_tUpdating the weight of the neural network in the utility fitting network unit (11); the confidence evaluation network unit (13) calculates the confidence of the action decision result by using the input vector of the input layer of the neural network in the utility fitting network unit (11), the output vector of the hidden layer and the differential signal, and outputs the confidence to the action correction network unit (15); the action decision network unit (14) is based on the input differential signal Delta TD_tAnd state space vector s_tPerforming selection learning of the motion and outputting a motion selection function

Giving an action correction network element (15), wherein j, k are integers greater than 0; the action correction network unit (15) selects a function for the action of the input using the confidence of the input

Correcting, calculating the selection probability value of the corrected action, outputting the action with the maximum probability to an action execution unit (16) for execution, and feeding back and inputting the state space vector after the action is executed to a utility fitting network unit (11), a differential signal calculation network unit (12) and an action decision network unitA member (14);

the utility fitting network unit (11) is composed of a neural network, and comprises an input layer, a hidden layer and an output layer, wherein the weight of the neural network is A, B and C, and the input vector x of the input layer of the neural network is_i(t) is the state space vector s generated after the action is executed at time t_tThe hidden layer activation function is a Sigmoid function, and the output of the neural network is a utility fitting value of a state after the action is executed

Wherein, b_i(t) vector representing weight B of input layer and output layer, c_j(t) represents the vector of the weight C of the hidden layer and the output layer, n is the number of units of the input layer, h is the number of units of the hidden layer, y_j(t) is the output vector of the hidden layer unit:

according to a function

Calculation of a_ij(t) is a vector of the weight A of the input layer and the hidden layer; the vector of the weight of the neural network in the utility fitting network unit (11) is specifically updated by using the following formula:

b_i(t+1)＝b_i(t)+λ·ΔTD_t+1·x_i(t)，i＝1,2,3...n

c_j(t+1)＝c_j(t)+λ·ΔTD_t+1·y_j(t)，j＝1,2,3...h

a_ij(t+1)＝a_ij(t)+λ_h·ΔTD_t+1·y_j(t)·sgn(c_j(t))·x_i(t)

wherein λ is a constant greater than zero，λ_hIs a number greater than zero, Δ TD_t+1A differential signal, sgn (c), representing the state space vector generated after the execution of the action corresponding to time t +1_j(t)) is determined according to a function sgn, which is:

$\mathrm{sgn}\left(z\right)=\left\{\begin{array}{cc}1& z>0\\ 0& z=0\\ -1& z<0\end{array}\right.$

the differential signal computing network unit (12) computes a differential signal delta TD according to an instantaneous differential algorithm_t：Wherein R(s)_t) Is to the state space vector s_tIs evaluated immediately, gamma is the discount coefficient,

represents the state space vector s generated after the action is executed at the moment t +1_t+1The resulting utility fit values are then compared to the expected values,

representing the state space vector s generated after the execution of the motion at time t_t(ii) the resulting utility fit value;

the confidence p (t) finally output by the confidence evaluation network unit (13) is as follows:

$p\left(t\right)=\frac{1}{1+e^{-a{p}_0\left(t\right)}},$ $p_0\left(t\right)=\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{\mathrm{\&alpha;}}_i\left(t\right)x_i\left(t\right)+\underset{j=1}{\overset{h}{\mathrm{\&Sigma;}}}{\mathrm{\&beta;}}_j\left(t\right)y_j\left(t\right)$

wherein, the value range of the confidence coefficient correction factor a is [1, 10]]，x_i(t)、y_j(t) are respectively the effectsThe input vector of the neural network in the fitting network unit (11) and the output vector of the hidden layer unit, n and h are the number of the neural network input layer units and the number of the hidden layer units in the utility fitting network unit (11) respectively; weight alpha after action execution corresponding to t +1 moment_i(t +1) and beta_jThe update of (t +1) is as follows:

α_i(t+1)＝α_i(t)+λ_p·ΔTD_t+1·x_i(t)，i＝1,2,3...n

β_j(t+1)＝β_j(t)+λ_p·ΔTD_t+1·y_j(t)，j＝1,2,3...h

wherein λ is_pDenotes a learning rate, and is a value between 0 and 1, Δ TD_t+1A differential signal representing a state space vector generated after the action is executed corresponding to the time t + 1;

the action decision network unit (14) is realized by adopting a neural network, the neural network comprises an input layer, a fuzzy subset layer, a variable node layer and a function output layer, and the input IN of the ith node of the input layer_i ¹Comprises the following steps:

IN_i ¹＝x_i(t)，i＝1,2,3...I

wherein I is the number of nodes in the input layer, x_i(t) is the state space vector s after execution of the action_tNormalizing the obtained vector; the fuzzy subset layer is used for fuzzifying the input of the input layer, and corresponds to the input x_iThe jth output of (t)

Comprises the following steps:

${O_{x_{i}j}}^2=\exp [-{\left(\frac{x_j\left(t\right)-m_{\mathrm{ij}}}{{\mathrm{\&sigma;}}_{\mathrm{ij}}}\right)}^2],i=\mathrm{1,2,3}...I,j=\mathrm{1,2,3}...J$

wherein J is each x of the input layers_i(t) number of inputs corresponding to fuzzy subset layer, m_ijAnd σ_ijPosition parameters and widths, x, of membership functions representing input vectors, respectively_j(t) is the input vector for the jth node of the input layer;

the activation function of the variable node layer is a Gaussian function, and the position parameter and the width of the Gaussian function are m respectively_lAnd σ_lNode output of variable node layer O_l ³Comprises the following steps:

${O_l}^3=\exp [-{\left(\frac{\underset{i=1,j=1}{\overset{I,J}{\mathrm{\&Sigma;}}}{O^2}_{x_{i}j}-m_l}{{\mathrm{\&sigma;}}_l}\right)}^2],l=\mathrm{1,2,3}...L$

wherein L is the number of nodes of the variable node layer; the output of the function output layer is a fitting value to the action function, namely an action selection function

${\hat{A}}_k\left(s_t\right)=\underset{l=1}{\overset{L}{\mathrm{\&Sigma;}}}{\mathrm{\&omega;}}_{\mathrm{lk}}{O_l}^3,k=\mathrm{1,2,3}...K$

Wherein K is the number of nodes of the function output layerCounting; omega_lkThe connection weight of the first node of the third layer and the kth node of the fourth layer; i, J, K and L are all positive integers;

the membership function position parameter m of the input vector_ijAnd width σ_ijPosition parameter m of Gaussian function of variable node layer_lAnd width sigma_lAnd the connection weight of the variable node layer and the function output layer is optimized and adjusted by adopting a hierarchical genetic algorithm;

said action correcting network element (15) to

Taking p (t) as the probability to generate a random function as a new action selection function A for the mean value_j(s_t) Then calculating the probability value P (a)_j|s_t) Outputting the action with the maximum probability value; the formula of the selected probability value is as follows:wherein, a_jFor the jth action, s_tFor obtaining state space vectors after the execution of the actions at time t, A_k(s_t) Selecting a function for the kth action, A_j(s_t) Selecting a function for the jth action;

the modeling device has two processes: an offline learning process and an online decision process; all the units are involved in the offline learning process, only the action decision network unit (14) and the action execution unit (16) which are obtained finally by the offline learning participate in the online decision process, and the action decision network unit (14) in the online decision process executes the action according to the state space vector s generated after the action execution unit (16) executes the action at the moment t_tCalculating and obtaining an output action selection function

By action of selectorAnd outputting the finally selected action to an action execution unit (16) for execution, and inputting the state space vector obtained after the action is executed to an action decision network unit (14).

Images