Reinforcement Leaning Study - 1

 


 This post summarized personal studies. the purpose of this post is to reorganize what I studied. 

And I'm not good at English. so there may be some incorrect information and writing expession. If there's something like that, I would appreciate it if you could let me know my mistake.

 

 

 What is Reinforcement Learning?

 The learning process through trial and error to maximize cumulative return in sequential decision-making problem.

 It is simillar to the way that a human baby learns. babies learn that what is good or what is bad by doing (looks like) random actions.


Difference from Supervised/ Unsupervised Learning.

 Reinforcement learning is differenct from supervised learning. Supervised learning is learning from a supervisor(labeled data). And reinforcement learning is different from Unsupervised Learning. ..



Markov Decision Process

 What is the meaning of 'sequential decision making problem'? We can define these problem with MDP(Markov Decision Process). Before learning more about MDP, it's good to know about Markov Process and Markov Reward Process for understanding MDP.

 Markov Process is a process where state transition occur along the probability table until it reaches end state. It has set of state and transition probability matirx. Each element of transition probability matrix is the probabilty of transitioning from one state to another.

 another important feature of Markov process is 'Makov Property'. it means that only current state (S(t)) affects to right next state (S(t+1)). In other words, the states before the present (t-1, t-2, ...) does not affect the next state(t+1).


Markov Reward Process(MRP) is Markov Process that has 'Reward'. Reward is obtained when a state changes to the next state. In this process, we can think about sum of reward, 'Return'. return G(t) is defined as below.

    G(t) = R(t+1) + γR(t+2) + γ^2R(t+3) + ... (γ  is discount rate. R(t) is reward of a state at time t)

 Return does not consider of the past rewards. Reinforcement learning is learning process that maximize 'Return'. 

  In MRP, we can consider the value of a state. The greater the reward is, the greater the value is. so value of state(V(s)) can be expressed as rewards like below. 

  V(s) = E[G(t) | S(t) = s].   ( E is expression of expectation.)


 Markov Decision Process (MDP) has one more thing, 'Agent'. Agent can 'action' in the evironment. Accordingly, the definition of Transition Probability and Reward are different from MRP. In summary, MDP consists of S (set of state), A (set of action), P (probability matrix), R (reward of state), γ (discount rate).




Reference :

 바닥부터 배우는 강화학습 (노승은)

 Reinforcement Leaning An Introduction (Richard S. Sutton)

Comments

Post a Comment

Popular posts from this blog

Reinforcement Learning Study - 7

Image
Deep Reinforcement Learning  In real world, states or action space is too big to record all information about the model in (value) table. So as to generalize the information, the most powerful generalization tool(function), Neural Network is used. A node is fundamental component of neural network, and each node linearly combines(WX + b) the inputs entering the node and then outputs them by applying a nonlinear function(sigmoid, ReLU etc...).   Value-based agent  value-based learning is method where neural netwrok is used to predict value function. In neural network, 'Loss function' is used to update parameter of neural network. Loss function is defined as the difference between the predicted value and the real value.  In Q-learning, value function is defined  .  so Loss is defined as below. In fact, in the above eqation, because we don't know real value Q, we can't use that equation. So, here we have smart way to solve this problem. That's the expected val...
Read more

the US and British attacks on Houthi and the impact of stock market

Image
the US and British attacks on Houthi and the impact of stock market  The US and UK launched airstrikes on more than a dozen sites used by the Iranian-backed Houthis in Yemen. This was a significant military response to the Houthis’ persistent campaign of drone and missile attacks on commercial ships in the Red Sea. The Houthis, backed by Iran, run most of the west of Yemen and control its Red Sea coastline. The strikes have raised fears of a broader escalation of the conflict in the region. The attacks led to an increase in oil prices. Brent jumped on fears there would be more disruption to shipping, and that the conflict could expand into a broader regional conflagration. Global benchmark Brent was trading around 4% higher. The rise in oil prices was driven by the market’s perception that this is an escalation of the conflict. The impact of rising oil prices on the stock market The impact of rising oil prices on the stock market is complex. An increase in oil prices usually lowers...
Read more

U.S. Stock Outlook 2024

Image
  Let's take a look at the 2024 U.S. stock outlook. To take a look at the 2024 U.S. stock outlook, we'll take a look based on recent announcements from Morgan Stanley and Forbes. What is Morgan Stanley? Morgan Stanley is an American multinational investment banking and financial services company headquartered at 1585 Broadway in Manhattan, New York. Morgan Stanley also offers a range of prospects to help raise, manage, and distribute funds. Morgan Stanley's Outlook Morgan Stanley said 2024 is an overvalued stock, so analysts' estimates of corporate earnings may be too optimistic given the easing of U.S. economic growth. Also, it said the market may be overestimating the number of Fed rate cuts in 2024. On the one hand, the stock market will provide investors with "greater opportunities," it said, "and the market has been efficient at rewarding and punishing the right stocks for the right fundamental reasons." Analysts predict the S&P 500 will end...
Read more