KR101860258B1 - Supply Chain based on Manufacturing System Optimum Design System and Method Using Markov Decision Process and Machine Learning Algorism - Google Patents

Abstract

인 관계식을 만족하도록 상기 가상의 체인 구조를 구성하고 하기 수학식1을 만족하는 상태가치함수(

)와 하기 수학식2를 만족하는 행동가치함수(

) 및 하기 수학식3을 만족하는 정책(policy,

)을 포함하는 MDP(Markov Decision Process) Factor 생성하는 공급망구성부와, 상기 현장보상값를 입력받는 현장보상값수집부 및, 현재시점에서 상기 현장보상값수집부를 통하여 입력 받은 상기 현장보상값과 상기 MDP Factor를 기반으로 특정한 미래시점에서 모든 상태(S)의 상기 상태가치함수의 총합이 특정한 미래목표값에 수렴하도록 정책예측값과 미래가중치예측값와 MDP Factor예측값 및 현장보상값예측값를 포함하는 공급망예측제어정보를 산출하는 공급망학습부를 포함하는 공급망설계단; 및 상기 공급망예측제어정보를 기반으로 상기 공급망을 제어하는 공급망제어단;을 포함하는 것을 특징으로 하는 마르코프 결정 프로세스와 기계학습 알고리즘을 이용한 제조업 기반의 공급 체인 최적화 설계 시스템:
[수학식]

,

,

,
상기 [수학식]에서

는 미래가중치(discount factor)이고,
R는 현장보상값 이다.
에 관한 것이다. The present invention relates to a manufacturing-based supply chain optimization design system using Markov decision processes and machine learning algorithms.
More specifically, at least one state (S) or behavior (A) contributing to the production and distribution of the finished product from the finished product to the consumer forms a virtual chain structure, and the state (S (S) for producing a site compensation value (R) including the raw material or the semi-finished product using the raw material or the finished goods or the shipment amount information of the finished product according to the state (S) Or transition probability (P) between the behavior (A)

And a state value function (< RTI ID = 0.0 >

) And a behavior value function satisfying the following equation (2)

) And a policy (policy,

A field compensation value collection unit for receiving the field compensation value, and a compensation unit for calculating an on-site compensation value, which is input through the field compensation value collection unit at the current point of time, and the MDP The supply chain prediction control information including the policy prediction value, the future weighted prediction value, the MDP factor predicted value, and the on-site compensation predicted value is calculated so that the sum of the state value functions of all states S at a specific future time point converges to a specific future target value A supply-demanding staircase including a supply-chain learning unit for performing supply-chain learning; And a supply chain control unit for controlling the supply chain based on the supply chain prediction control information. The manufacturing-based supply chain optimization design system using a Markov decision process and a machine learning algorithm,
[Mathematical Expression]

,

,

,
In the above equation,

Is a future discount factor,
R is the field compensation value.
.

Description

Technical Field [0001] The present invention relates to a manufacturing-based supply chain optimization design system and method using a Markov decision process and a machine learning algorithm,

The present invention relates to a manufacturing-based supply chain optimization design system using Markov decision processes and machine learning algorithms.

More specifically, the Markov Decision Process is used to display various SCM structures as a discount factor indicating that the compensation value for behavior changes with time, and by applying deep learning based on reinforcement learning, Based supply chain optimization system using a Markov decision process and a machine learning algorithm that can derive a discount factor linked to the structure and supply chain management policy for the system.

Supply Chain (SC) is a network of organizations and business processes that connect raw materials suppliers, manufacturers' manufacturing plants, finished goods shipment and distribution, retail discount stores and customers to supply products or services from source to consumption. to be.

Supply Chain Management (SCM) is a system that grasps the flow of logistics from the supplier to the producer, distributor, and customer from the viewpoint of one value chain and supports the flow of necessary information smoothly. It is one of the management innovation techniques that aims to achieve the whole process optimization among the components of the supply chain by avoiding sectoral optimization within an enterprise or individual enterprise unit optimization.

As the social structure becomes complicated, new types and functions of products are being developed, and major changes are taking place in the characteristics and functions of the products that have been used in human life.

Zara and UNIQLO, leading apparel companies, have excellent supply chain management, but both brands are choosing opposite supply chain management policies.

UNIQLO maintains its inventory strategy to maintain high inventory levels by selecting products and production strategies for small items of basic items, outsourcing many manufacturing functions, and distributing key customer segments at an even rate. Therefore, efficiency is the top priority in supply chain management.

ZARA, on the other hand, maintains a low inventory level inventory strategy by selecting a variety of products, production strategies for a small quantity of sensitive items, and vertical integration from design to manufacture and sales. do. Therefore, for supply chain management, speed is the top priority.

Apple's global leader in the smartphone industry, according to Gartner, a Gartner researcher in Supply Chain Top 25 in 2013, has a total score of 9.51 out of a total score of 10, McDonald's, a hamburger franchise company. Also, SCM is the world's largest sales distributor, Amazone.com, with a score of only 5.86, and Apple's global market dominance can be attributed not only to product performance and innovation, but also to supply chain management. In particular, given that Tim Cook, currently the CEO of Apple, is an SCM expert, you can see how important supply chain management is in a competitive industry in future industrial ecosystems.

Conventional Supply Chain Management has implemented SCM by applying major policies determined within the enterprise without verification or confirmation, in consideration of actual business environment and current status of manufacturers and distribution structures. In addition, although various SCM policies are used for each industry and each customer, the criteria and authority for selecting them are not properly provided to the employees of the company performing the SCM.

Therefore, the Markov Decision Process (Markov Decision Process), which shows the diversification of various SCM structures according to the behavior as a discount factor over time, and applying deep learning based on reinforcement learning, It is necessary to develop a manufacturing - based supply chain optimization design system that can derive a discount factor linked to supply chain management policy.

Patent Document 10-2016-0071776 Patent Document 10-2016-0063006 Patent Document 10-2009-0003488 Patent No. 10-1573985

)과 미래가중치(

)를 입력하여 각 상태(S)에 따른 보상(R)의 예상치를 도출할 수 있으며, 모든 상태가치함수의 총합이 미리 설정한 특정 목표값에 근접수렴 하도록, 입력으로 사용한 정책(

) 또는 미래가중치(

)를 변화시키면서 공급망 구성을 최적화함으로써, 기업이 목표로 하는 가치에 도달하기 위해서 필요한 비즈니스 전략 수립에 활용할 수 있는 정책(

)과 현재와 미래가치의 중요도를 안배할 수 있는 미래가중치(

)를 도출 할 수 있는 마르코프 결정 프로세스와 기계학습 알고리즘을 이용한 제조업 기반의 공급 체인 최적화 설계 시스템을 제공하는데 목적이 있다.SUMMARY OF THE INVENTION Accordingly, the present invention has been made to solve the above-mentioned problems occurring in the prior art, and it is an object of the present invention to provide a method and apparatus for detecting a state S, a behavior A and a state transition probability P, , And a policy that takes into consideration the internal and external conditions of the company in the supply chain structure thus configured

) And future weights

) Can be input to derive an estimate of the compensation (R) according to each state (S), and the policy used as the input so that the sum of all the state value functions converges to a predetermined target value

) Or future weighting (

) To optimize the supply chain structure, thereby enabling the company to use its business strategies to achieve the desired value

) And future weights (which can assign the importance of present and future values)

), And a manufacturing-based supply chain optimization design system using machine learning algorithms.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are not intended to limit the invention to the precise form disclosed. It can be understood.

인 관계식을 만족하도록 상기 가상의 체인 구조를 구성하고 하기 수학식1을 만족하는 상태가치함수(

)와 하기 수학식2를 만족하는 행동가치함수(

) 및 하기 수학식3을 만족하는 정책(policy,

)을 포함하는 MDP(Markov Decision Process) Factor 생성하는 공급망 구성부와, 상기 현장보상값를 입력 받는 현장보상값 수집부 및, 현재시점에서 상기 현장보상값 수집부를 통하여 입력 받은 상기 현장보상값과 상기 MDP Factor를 기반으로 특정한 미래시점에서 모든 상태(S)의 상기 상태가치함수의 총합이 특정한 미래목표값에 수렴하도록 정책예측값과 미래가중치 예측값와 MDP Factor 예측값 및 현장보상값 예측값를 포함하는 공급망예측제어정보를 산출하는 공급망학습부를 포함하는 공급망설계단; 및 상기 공급망예측제어정보를 기반으로 상기 공급망을 제어하는 공급망제어단;을 포함하는 것을 특징으로 하는 마르코프 결정 프로세스와 기계학습 알고리즘을 이용한 제조업 기반의 공급 체인 최적화 설계 시스템:A first object of the present invention is to provide a method and apparatus for forming a virtual chain structure in which at least one state (S) or behavior (A) contributing to the production and circulation of the finished article from the raw material to the consumer forms a virtual chain- A manufacturing supply chain optimization design system for optimizing a supply chain that generates an on-site compensation value (R) including semi-finished products using the raw materials or the raw materials according to a state (S) or inventory quantity or shipment quantity information of the finished product, ) Or the transition probability (P, Transition Probability) between the actions (A)

And a state value function (< RTI ID = 0.0 >

) And a behavior value function satisfying the following equation (2)

) And a policy (policy,

A field compensation value collection unit for receiving the field compensation value, and a compensation unit for calculating an on-site compensation value, which is input through the field compensation value collection unit at the current point of time, and the MDP The supply chain prediction control information including the policy prediction value, the future weighted prediction value, the MDP factor predicted value, and the on-site compensation predicted value is calculated so that the sum of the state value functions of all states S at a specific future time point converges to a specific future target value A supply-demanding staircase including a supply-chain learning unit for performing supply-chain learning; And a supply chain control unit for controlling the supply chain based on the supply chain prediction control information. The manufacturing-based supply chain optimization design system using a Markov decision process and a machine learning algorithm,

[Mathematical Expression]

,

,

,

,

,

,

In the above equation,

는 미래가중치(discount factor)이고,

Is a future discount factor,

R is the field compensation value.

. ≪ / RTI >

The supply stop step may further include a design stage storage unit that stores the development compensation value, the MDP factor, the future target value, and the supply chain prediction control information in a time series manner.

The supply-demand step includes a design stage evaluation unit that compares the compensation value of the development compensation value stored in the design stage storage unit with the in-situ compensation predictive value to calculate the prediction accuracy of the MDP factor, the future target value, And further comprising:

The design stage storage unit may further include a design stage storage unit display unit for displaying at least one of the development compensation value, the MDP factor, the future target value, the supply chain prediction control information, and the prediction accuracy. can do.

The supply stop step may further include a design stage input part for receiving information from an administrator, and receives the future target value, the on-site compensation value, and an initial value of the MDP factor .

The supply network learning unit may further include a learning unit calculation unit for calculating supply policy prediction control information including a policy prediction value, a future weighting prediction value, an MDP factor prediction value, and a site compensation value prediction value based on a Markov Decision Process or a Bellman Equation Unit and a learning unit display unit for displaying to the manager supply policy predictive control information including the policy predictive value, the future weight predictive value, the MDP factor predictive value and the on-site compensation value predictive value.

The supply chain configuration unit may further include a configuration input unit that receives the chain structure from an administrator.

In addition, the supply chain component includes Vendor Managed Inventory (VMI), Quick Response (QR), Collaborative Planning, Forecasting and Replenishment (CPFR) and Continuous Replenishment Replenishment Program, and Efficient Consumer Response (ECR). The administrator can select at least one of the manufacturing-based SCM chain structures .

인 관계식을 만족하도록 상기 체인 구조를 구성하는 단계(S01); 공급망구성부가, 하기의 수학식1을 만족하는 상태가치함수(

)와 하기의 수학식2를 만족하는 행동가치함수(

) 및 하기의 수학식3을 만족하는 정책(policy,

)을 포함하는 MDP(Markov Decision Process) Factor 생성하는 단계(S02); 공급망학습부가, 현재시점에서 입력 받은 상기 현장보상값과 상기 MDP Factor를 기반으로 특정한 미래시점에서 모든 상태(S)의 상기 상태가치함수의 총합이 특정한 미래목표값에 수렴하도록 정책예측값과 미래가중치예측값와 MDP Factor예측값 및 현장보상값예측값를 포함하는 공급망예측제어정보를 산출하는 단계(S03); 및 공급망제어단은, 상기 공급망예측제어정보를 기반으로 공급망제어부가 상기 공급망을 제어하는 단계(S04);를 포함하는 것을 특징으로 하는 마르코프 결정 프로세스와 기계학습 알고리즘을 이용한 제조업 기반의 공급 체인 최적화 설계 방법:A second object of the present invention is to provide a method and apparatus for forming a virtual chain structure in which at least one state (S) or behavior (A) contributing to the production and distribution of the finished article from the raw material to the consumer forms a virtual chain structure, A manufacturing supply chain optimizing design method for optimizing a supply chain that generates an on-site compensation value using the raw material or the raw material according to a state (S) or an in-product value including an inventory quantity or shipment quantity information of the finished product, The transition probability (P) between state (S) and behavior (A)

Constructing the chain structure to satisfy the in-relational expression (S01); The supply chain constructing unit may include a state value function (

) And a behavior value function satisfying the following expression (2)

) And a policy (policy,

A step (S02) of generating a Markov Decision Process (MDP) Factor including a Markov Decision Process (MDP) Factor; The supply network learning unit may calculate the sum of the state value functions of all states S at a specific future point based on the on-site compensation value and the MDP factor input at the current point in time so as to converge to a specific future target value, (S03) of supply chain prediction control information including an MDP factor prediction value and a site compensation value prediction value; And a supply chain control stage (S04), wherein the supply chain control section controls the supply chain based on the supply chain prediction control information (S04). The manufacturing-based supply chain optimization design using the Markov decision process and machine learning algorithm Way:

[Mathematical Expression]

,

,

,

,

,

,

In the above equation,

는 미래가중치(discount factor)이고,

Is a future discount factor,

R is the field compensation value.

. ≪ / RTI >

The step (S05) includes, between the steps S03 and S04, the on-scene compensation value, the MDP factor, the future target value and the supply-demand forecast control information are stored in the design stage storage unit in a time-series manner .

In addition, after step S04, the design stage evaluation unit compares the on-site compensation value stored in the design stage storage unit with the in-situ compensation value stored in the design stage storage unit, and calculates a prediction accuracy of the MDP factor, the future target value, (Step S06) of calculating the number of times that the image is displayed.

Between step S02 and step S03, the design stage storage unit display unit displays at least one of the on-scene compensation value, the MDP factor, the future target value, the supply chain prediction control information, and the prediction accuracy (S07). ≪ / RTI >

The method further includes the step (S08) of inputting the future target value, the field compensation value, and the initial value of the MDP Factor from the manager through the designing unit input terminal between the steps S02 and S03 .

In step S03, the supply chain prediction control information including the policy prediction value, the future weighted prediction value, the MDP factor predicted value, and the on-site compensation predicted value is calculated using the Markov Decision Process or Bellman Equation .

In addition, the step S01 may further include receiving the chain structure from the manager (S10).

)과 미래가중치(

)를 입력하여 각 상태(S)에 따른 보상(R)의 예상치를 도출할 수 있으며, 모든 상태가치함수의 총합이 미리 설정한 특정 목표값에 근접수렴하도록, 입력으로 사용한 정책(

) 또는 미래가중치(

)를 변화시키면서 공급망 구성을 최적화함으로써, 기업이 목표로하는 가치에 도달하기 위해서 필요한 비즈니스 전략 수립에 활용할 수 있는 정책(

)과 현재와 미래가치의 중요도를 안배할 수 있는 미래가중치(

)를 도출 할 수 있는 장점이 있다.According to an embodiment of the present invention, a supply chain suited to the situation of a specific company is constructed so as to satisfy the characteristics of the state (S), the behavior (A) and the state transition probability (P) specified in the Markov decision process, Policies that take into consideration the internal and external situation of the company in the supply chain

) And future weights

) Can be input to derive an estimate of the compensation (R) according to each state (S), and the policy used as the input so that the sum of all the state value functions converges to a predetermined target value

) Or future weighting (

) To optimize the supply chain structure, thereby enabling the company to use its business strategies to achieve the desired value

) And future weights (which can assign the importance of present and future values)

) Can be derived.

It should be understood, however, that the effects obtained by the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art to which the present invention belongs It will be possible.

1a. A general reinforced learning mechanism conceptual diagram.
1b. Concept of Markov process.
1C. Concept of Markov compensation process.
Figure 2. Example of a supply chain configuration applied to a manufacturing-based supply chain optimization design system using the Markov decision process and machine learning algorithm in accordance with an embodiment of the present invention.
Figure 3. Deriving a policy (π) and future weighted (η) predictions using the supply chain structure applied to the manufacturing-based supply chain optimization design system using the Markov decision process and the machine learning algorithm according to an embodiment of the present invention Calculation method for.
FIG. 4 is a block diagram of a manufacturing-based supply chain optimization design system using a Markov decision process and a machine learning algorithm according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the detailed description of known functions and configurations incorporated herein will be omitted when it may unnecessarily obscure the subject matter of the present invention.

The same reference numerals are used for portions having similar functions and functions throughout the drawings. Throughout the specification, when a part is connected to another part, it includes not only a case where it is directly connected but also a case where the other part is indirectly connected with another part in between. In addition, the inclusion of an element does not exclude other elements, but may include other elements, unless specifically stated otherwise.

Hereinafter, the structure and functions of a manufacturing-based supply chain optimization design system using the Markov decision process and machine learning algorithm according to an embodiment of the present invention will be described.

FIG. 1A shows a conceptual diagram of a general reinforcement learning mechanism. Reinforcement learning is a type of machine learning that is inductive learning. It does not provide accurate output information corresponding to an action. However, evaluation information called reward is a problem given , The optimal behavior of each state can be determined from the relationship between action and reward.

For example, if a production plan for an increase in sales is made with the aim of increasing sales in a production state of any supply chain, if the market demand decreases, the increase in production costs, including material and labor costs, And the increase of the warehouse ratio caused by the increase of the production cost, such as the discontinuation of production or production, such as loss of production may result in loss factors, and eventually the action (Action) Reward can be considered as given.

On the other hand, if market demand increases, profit factors such as sales increase and profit structure improvement may occur, and as a result, the amount of increase in production will be expressed as a positive real value Rewards can be considered given.

Therefore, it is possible to predict the behavior of the future optimally through the experience of past actions and rewards of action.

However, in the actual supply chain management, as shown in FIG. 1A, it is not represented by actions and rewards of a simple structure, and a plurality of states and state transition probabilities constitute a complicated form.

It can be increased or subtracted from the production conditions cited above. Thus, there are two possible behaviors in the production state, each of which can be expressed as a probability. Assuming that the probability of a subtraction is 10%, the probability of being multiplied can be divided by 90%.

Also, if the number of states increases, the state transition must take place at various stages until a reward is given. The reward for the action of increase or decrease in the production state cited above should take into account all the effects of the supply chain on the purchase-production-distribution-sale of the action of the increase or decrease. do.

Therefore, through the reinforcement learning, it is possible to design a supply chain structure that can derive the optimal action by predicting the state reward that is determined by the current action.

Figure 1B shows a conceptual diagram of the Markov Process process.

Referring to FIG. 1B, each state represents a one-year temporal event, and the state of the next step expected in the state of the current state is described as being independent of the past event. That is, state 1 affects state 2 or state 3, but does not affect state 4, and can be defined by the following equation (1).

는 상태b에서 상태a로의 상태 전이확률이다.In Equation (1)

Is the state transition probability from state b to state a.

As mentioned earlier, in a typical real-world supply chain management, there are many state transitions between states and state transitions. When the actual supply chain is constructed by using the Markov process as shown in FIG. 1B, there is an advantage that the relation between the states in the logistics or the flow of funds can be clearly defined. As shown in Equation (1), since the Markov process assumes that the next step state expected in the current step state is not related to the past event, the relation between each state, that is, the state transition probability, The output becomes clear, and the prediction function of the entire process is enabled.

Figure 1C illustrates the Markov compensation process. A Markov compensation process can be configured by applying a reward R and a discount factor? To the Markov process shown in FIG. 1B. The state value function at this time can be defined by the following Equation 2 .

The Markov compensation process shown in FIG. 1C can be applied to the Markov process shown in FIG. 1B by feedback from a supply chain configured to satisfy the Markov process when taking a Markov process to move from one state to another in a Markov process. It includes reward to receive.

FIG. 2 illustrates an example of a supply chain configuration applied to a manufacturing-based supply chain optimization design system using a Markov decision process and a machine learning algorithm according to an embodiment of the present invention.

Figure 2 illustrates an example of a configuration that simulates an actual supply chain structure by including the action and policy ([pi]) in the Markov compensation process of Figure 1C.

In this case, a function that connects state and action can define a policy (π) that is defined as a probability to perform an action when a state is given, and the policy (π) Can be expressed by Equation (3).

Also, the sum of the values that can be expected when the behavior a is taken in the state s and the policy π is followed can be expressed as Equation (4).

At this time, the state value function shown in Equation (2) can be modified to a state value function including a policy as shown in Equation (5).

As shown in FIG. 2, in the manufacturing-based supply chain optimization design system using the Markov decision process and the machine learning algorithm according to an embodiment of the present invention, parts manufacturing, The various stages of inclusion and the various actions such as purchasing, marketing, shipment, process improvement, and development can be interconnected to form a supply chain.

These supply chain configurations can be tailored to the company's own supply chain, depending on industry and customer, internal and external environment and corporate business policies. For example, for the stable supply of parts and raw materials, producers of finished products may include supply guarantee management or prepayment in supply chain management, which share raw material production or parts manufacturing suppliers with finished product production plans or plans. In some cases, actions such as product development or process improvement are included in the supply chain structure.

As mentioned above, the supply chain structure can be determined by various factors. In addition to the industrial environment both inside and outside the company, advanced technological power is universalized and the life pattern, quantitative and qualitative development of consumers are accelerated, Due to the influence of the logistics system, there is a tendency to construct a supply chain most suitable for individual companies.

)를 변수로 하여, 임의의 시점의 임의의 단계에서 취할 수 있는 행동에 따른 보상을 예상할 수 있다.In addition, as described above, as the business environment outside the company becomes more complicated and time becomes more important than ever before, the company clarifies its present and future values and presents its management policies accordingly. . Therefore, the policy (p) and the future weight (p) defined in the supply chain optimization design system according to an embodiment of the present invention as shown in FIG. 2

) As a variable, it is possible to predict the compensation according to an action that can be taken at an arbitrary stage at an arbitrary point in time.

)를 입력하여 각 상태(S)에 따른 보상(R)의 예상치를 도출할 수 있다. 이때, 모든 상태가치함수의 총합이 특정한 미리목표값에 근접수렴하도록, 입력으로 사용한 정책(π) 또는 미래가중치(

)를 변화시키면서 공급망 구성을 최적화함으로써, 기업이 목표로 하는 가치에 도달하기 위해서 필요한 비즈니스 전략 수립에 활용할 수 있는 정책(π)과 현재와 미래가치의 중요도를 안배할 수 있는 미래가중치(η)를 도출 할 수 있다. Therefore, by utilizing the supply chain structure applied to the manufacturing-based supply chain optimization design system using the Markov decision process and the machine learning algorithm according to the embodiment of the present invention shown in FIG. 2, (Π) and future weights (π) that take into account the internal and external situation of the company in the supply chain structure thus constituted.

) Can be input to derive an estimate of the compensation R according to each state S. At this time, the policy (?) Or the future weight (?) Used as the input so that the sum of all the state value functions converge to a specific predetermined target value

(Π), which can be used to establish the business strategy needed to reach the target value, and the future weight (η), which can assign the importance of the present and future value .

FIG. 3 is a flow chart illustrating a method for deriving a policy (π) and a future weight (η) by utilizing a supply chain configuration applied to a manufacturing-based supply chain optimization design system using a Markov decision process and a machine learning algorithm according to an embodiment of the present invention The calculation method is shown.

)의 최적값을 도출하기 위해서는 정책(π)과 미래가중치(

)를 변수로 하되, 각각의 상태(S)가 상호 연결되어있으며, 특히 보상(R)은 특정 단계에서 다음 단계로 이동하는 행동을 취할 때 피드백(feedback)받는 스칼라 실수값이기 때문에, 정책(π)과 미래가중치(

)의 입력 조건에 따라서, 상태(S) 가치 함수가 수렵(convergence)하지 못하고 발산(divergence)는 상황이 도출 될 수 있다. Referring to FIG. 2 and Equation 5, the policy (?) And the future weight (

(Π) and future weights (π)

(R) is a scalar real number value that is feedbacked when taking an action that moves from a specific step to the next step, and therefore the policy (? ) And future weights

), The state (S) value function can not converge and divergence can be derived.

)의 입력값 변경과 계산을 반복함으로써, 정책(π)과 미래가중치(

)의 최적값이 도출될 수 있음을 알 수 있다.Referring to Figures 2 and 3, the supply chain optimization design system according to one embodiment of the present invention knows all the information about the supply chain configuration, including all states S and behavior (A) and state transition probability (Π) and future weights (π) until the sum of the state value functions converges to a specific future target value

) And changing the input value of the policy (?) And the future weight

) Can be derived.

) 정보를 기반으로 하여, 각각의 상태(S)에서 행동과 상태 전이를 선택할 수 있는데, 이때 선택된 행동이나 상태 전이를 취할 때 모든 상태(S)에서 보상(R)이 피드백 된다. These derived policies (π) and future weights

Based on the information, we can choose the behavior and state transition in each state S, where the compensation (R) is fed back in all states (S) when taking the selected behavior or state transition.

)의 예측값은, 다음 단계에서의 피드백되는 상태(S)별 보상(R)값을 이용하여 그 정확도를 검증할 수 있다. 즉, 도출된 입력값인 정책(π)과 미래가중치(

)을 기반으로 각 상태(S)에서 행동(A)를 결정하고, 이에 대한 평가정보가 주어짐으로써, 구성된 공급망의 환경 변수를 학습하여, 정책(π)과 미래가중치(

)의 입력과 보상(R)사이의 함수관계를 귀납적으로 도출할 수 있다. 이러한 학습 방법은 출력에 대한 정보는 제공하지 않지만, 평가정보는 주어지는 문제에 대해 각 상태에서의 행동을 결정하는 강화학습(reinforcement learning)에 해당한다.Therefore, the optimal policy (π) and the future weight (π)

) Can be verified by using the compensation (R) value for each state S to be fed back in the next step. In other words, the derived input values (π) and future weights

(Π) and future weights (π) by learning the environmental variables of the configured supply chain by determining the behavior (A) in each state (S)

) And the compensation R can be derived in an inductive manner. This learning method does not provide information about the output, but the evaluation information corresponds to reinforcement learning that determines behavior in each state for a given problem.

FIG. 4 shows a block diagram of a manufacturing-based supply chain optimization design system using a Markov decision process and a machine learning algorithm according to an embodiment of the present invention.

As shown in FIG. 4, the supply chain optimization design system according to an embodiment of the present invention includes a supply chain configured according to the environment inside and outside the company, a supply-demanding staircase designing each constituent element of the supply chain, And a supply chain control stage that controls the supply chain based on the results derived from the stairs.

2, all the components including the state S of the supply chain can be configured to be suitable for the vendor according to the customer-specific specification by industry. The supply chain component can be expressed as Equation 1 to Equation 4 And is configured so that all configurations of the supply chain satisfy the Markov decision process.

) 예측값을 산출한다. The on-site compensation value collector receives the on-site compensation value, which is a compensation value for each state (S) in the supply chain structure, and the supply chain learning unit receives the optimal policy (?) And the future weight

) Predicted value.

)등을 저장함으로써, 최적화 설계 시스템의 예측 정확도를 평가할 수 있으며, 이를 공급망 관리자에게 표시함으로써 공급망 설계의 효용성을 평가함과 동시에, 관리자로 하여금 설계시스템의 변경 또는 수정을 가능하도록 한다.The design stage storage unit stores the result of the iterative calculation and the predicted policy (π) and the future weight (

) To estimate the predictive accuracy of the optimized design system and display it to the supply chain manager to assess the utility of the supply chain design and enable the manager to modify or modify the design system.

In addition, the Monte-Carlo Reinforcement Learning algorithm may be applied when there is no information on some of the configurations including state (S) and behavior (A) in the designed supply chain structure.

It should be noted that the above-described apparatus and method are not limited to the configurations and methods of the embodiments described above, but the embodiments may be modified so that all or some of the embodiments are selectively combined .

Claims (15)

At least one state S or action A contributing to the production and distribution of the finished product from the finished product using the raw material forms a virtual chain structure, CLAIMS 1. A manufacturing supply chain optimization design system for optimizing a supply chain that generates a site compensation value (R) comprising raw materials or semi-finished products using the raw materials or inventory or shipment information of the finished products,
The transition probability (P) between the state (S) and the behavior (A)

And a state value function (< RTI ID = 0.0 >

) And a behavior value function (

) And a policy (policy,

A field compensation value collection unit for receiving the field compensation value, and a compensation unit for calculating an on-site compensation value, which is input through the field compensation value collection unit at the current point of time, and the MDP The supply chain prediction control information including the policy prediction value, the future weighted prediction value, the MDP factor predicted value, and the on-site compensation predicted value is calculated so that the sum of the state value functions of all states S at a specific future time point converges to a specific future target value A supply-demanding staircase including a supply-chain learning unit for performing supply-chain learning; And
And a supply chain control unit for controlling the supply chain based on the supply chain prediction control information.
[Mathematical Expression]

,

,

,
In the above equation,

Is a future discount factor,
R is the field compensation value.
The method according to claim 1,
Wherein the supply walks further comprise a design stage storage unit for storing the on-site compensation value, the MDP factor, the future target value, and the supply chain prediction control information in a time series manner. Manufacturing - based Supply Chain Optimization Design System.
3. The method of claim 2,
Wherein the supply stop step includes a design stage for calculating the predicted accuracy of the MDP factor, the future target value, and the supply-demand forecast control information by comparing the on-site compensation value stored in the design stage storage unit in a time- And an evaluating unit. The manufacturing-based supply chain optimization design system using the Markov decision process and the machine learning algorithm.
The method of claim 3,
Wherein the design stage storage unit further comprises a design stage storage unit display unit for displaying at least one of the on-site compensation value, the MDP factor, the future target value, the supply chain prediction control information, and the prediction accuracy. Manufacturing - based Supply Chain Optimization Design System Using Decision Process and Machine Learning Algorithm.
The method according to claim 1,
The supply step includes a design stage input unit for receiving information from an administrator,
Wherein the future target value, the on-site compensation value, and the initial value of the MDP factor are input to the manufacturing-based supply chain optimization design system using the Markov decision process and machine learning algorithm.
6. The method of claim 5,
The supply network learning unit may include a learning unit calculation unit for calculating supply-side predictive control information including a policy prediction value, a future weighted prediction value, an MDP factor prediction value, and an on-site compensation value prediction value based on a Markov Decision Process or a Bellman Equation And a learning unit display unit for displaying to the manager supply policy predictive control information including the policy predictive value, the future weight predictive value, the MDP factor predictive value, and the on-site compensation value predictive value. Based supply chain optimization design system.
The method according to claim 1,
Wherein the supply chain constructing unit further comprises a constructing unit input unit for receiving the chain structure from an administrator. The manufacturing-based supply chain optimization designing system using the Markov decision process and the machine learning algorithm.
8. The method of claim 7,
The Supply Chain component includes Vendor Managed Inventory (VMI), Quick Response (QR), Collaborative Planning, Forecasting and Replenishment (CPFR) and Continuous Replenishment Program (CRP) ) And an EMS (Efficient Consumer Response), and the manager can select at least one of the manufacturing basic SCM chain structure Manufacturing - based supply chain optimization design system using Markov decision process and machine learning algorithm.
At least one state (S) or action (A) contributing to the production and distribution of the finished product from the finished product using the raw material to the consumer forms a virtual chain structure, A manufacturing supply chain optimization design method for optimizing a supply chain that generates an on-site compensation value using the raw material or an in-process compensation value including an inventory quantity or a shipment quantity information of the finished product,
The supply chain constructor may determine the transition probability (P) between the state (S) or behavior (A)

Constructing the chain structure to satisfy the in-relational expression (S01);
The supply chain component is a state value function that satisfies the following equation:

) And a behavior value function satisfying the following equation (

) And a policy satisfying the following equation (policy,

A step (S02) of generating a Markov Decision Process (MDP) Factor including a Markov Decision Process (MDP) Factor;
The supply network learning unit may calculate the sum of the state value functions of all states S at a specific future point based on the on-site compensation value and the MDP factor input at the current point in time so as to converge to a specific future target value, (S03) of supply chain prediction control information including an MDP factor prediction value and a site compensation value prediction value; And
(S04) of controlling the supply chain control unit (S04) based on the supply chain predictive control information (S04). The manufacturing-based supply chain optimization design method using the Markov decision process and the machine learning algorithm :
[Mathematical Expression]

,

,

,
In the above equation,

Is a future discount factor,
R is the field compensation value.
10. The method of claim 9,
Between the steps S03 and S04,
Further comprising a step (S05) of storing the on-site compensation value, the MDP factor, the future target value, and the supply-demand forecast control information in a design stage storage unit in a time-series manner (S05) A Manufacturing - Based Supply Chain Optimization Design Method Using.
11. The method of claim 10,
After the step S04,
The design stage evaluation unit calculates the prediction accuracy of the MDP factor, the future target value, and the supply-demand forecast control information by comparing the on-site compensation value stored in the design stage storage unit in a time-series manner with the in- ); And a manufacturing-based supply chain optimization design method using a Markov decision process and a machine learning algorithm.
12. The method of claim 11,
Between step S02 and step S03,
Wherein the design stage storage unit display step further comprises the step of displaying at least one of the on-site compensation value, the MDP factor, the future target value, the supply chain prediction control information and the prediction accuracy (S07) Manufacturing - based Supply Chain Optimization Design Method Using Markov Decision Process and Machine Learning Algorithm.
10. The method of claim 9,
Between step S02 and step S03,
(S08) of inputting the future target value, the on-site compensation value, and an initial value of the MDP Factor from an administrator through an input terminal of a designing unit (S08) A Manufacturing - Based Supply Chain Optimization Design Method Using.
14. The method of claim 13,
In operation S03,
The supply chain prediction control information including the policy prediction value, the future weight prediction value, the MDP factor prediction value and the on-site compensation value prediction value is calculated using the Markov Decision Process or the Bellman Equation. Manufacturing - based Supply Chain Optimization Design Method Using Learning Algorithm.
10. The method of claim 9,
In operation S01,
And receiving the chain structure from the manager (S10). The manufacturing-based supply chain optimization design method using the Markov decision process and the machine learning algorithm.

Images