Yarn的状态机框架分析
Yarn的状态机框架分析
文章目录
- Yarn的状态机框架分析
-
- 什么是状态机
- 使用状态机的目的
- yarn中的状态机
- 关于yarn的处理过程
- 状态机举例
- 状态机的核心StateMachineFactory类
-
- InternalStateMachine
- 简化实验
什么是状态机
状态机(State Machine)，是有限状态自动机的简称。简单解释：给定一个状态机，同时给定它的当前状态和输入，那么输出状态是可以明确的运算出来的。
使用状态机的目的
当时实体与其相关的事件越多，它的状态和处理就越复杂，状态机负责合理地组织状态转换的流程，快速找到指定初始状态和事件类型对应的状态转换方法。
yarn中的状态机
YARN将各种处理逻辑抽象成事件和对应事件调度器， 并将每类事件的处理过程分割成多个步骤， 用有限状态机表示。
在Yarn中，App、AppAttempt、Container、Node都可以使用状态机表示。其中，
RMApp:用于维护一个Application的生命周期；RMAppAttempt:用于维护一次尝试运行的生命周期；RMContainer:用于维护一个已分配的资源最小单位Container的生命周期；RMNode:用于维护一个NodeManager的生命周期。
关于yarn的处理过程
yarn中的状态机是与事件驱动模型结合使用的，在Service框架中注册调度处理器，然后在处理器使用状态机。
整个处理过程大致为： 处理请求会作为事件进入系统， 由异步调度器（AsyncDispatcher） 负责传递给相应事件调度器（Event Handler） 。 该事件调度器可能将该事件转发给另外一个事件调度器， 也可能交给一个带有有限状态机的事件处理器， 其处理结果也以事件的形式输出给中央异步调度器。 而新的事件会再次被中央异步调度器转发给下一个事件调度器， 直至处理完成（达到终止条件） 。
状态机的使用流程分为两步：
第一步：Service注册Handler。
第二步：Handler使用状态机。
状态机举例
例如，对于一个应用RMApp而言，RMApp存在一个初始状态，处理事件时，会根据事件类型匹配对应的转换类Transition，将RMApp从初始状态转化成目标状态。RMApp经历的流程为：初始状态–>转换方法–>目标状态，将其所有流程汇总起来，就是状态机。
下图为RMApp状态的转换流程。乍看还是比较复杂的。
各个使用状态机的事件处理类（实现EventHandler接口）依赖状态工厂类StateMachineFactory，完成了状态机的初始化。将状态机的处理流程，通过链表结构TransitionsListNode进行组织起来。以处理RMAppEventType事件来说，在RMActiveServices的serviceInit方法中注册RMAppEventType事件的调度处理器ApplicationEventDispatcher
// Register event handler for RmAppEvents
rmDispatcher.register(RMAppEventType.class,
new ApplicationEventDispatcher(rmContext));
ApplicationEventDispatcher中去让RMApp的实现类RMAppImpl处理事件。
@Override
public void handle(RMAppEvent event) {
ApplicationId appID = event.getApplicationId();
RMApp rmApp = this.rmContext.getRMApps().get(appID);
if (rmApp != null) {
try {
rmApp.handle(event);
} catch (Throwable t) {
LOG.error("Error in handling event type " + event.getType()
+ " for application " + appID, t);
}
}
}
RMAppImpl的handle方法中通过调用状态的的转换方法doTransition来改变RMApp对象状态。
public void handle(RMAppEvent event) {
this.writeLock.lock();
try {
ApplicationId appID = event.getApplicationId();
LOG.debug("Processing event for " + appID + " of type "
+ event.getType());
final RMAppState oldState = getState();
try {
/* keep the master in sync with the state machine */
this.stateMachine.doTransition(event.getType(), event);
} catch (InvalidStateTransitionException e) {
LOG.error("App: " + appID
+ " can't handle this event at current state", e);
onInvalidStateTransition(event.getType(), oldState);
}
// Log at INFO if we're not recovering or not in a terminal state.
// Log at DEBUG otherwise.
if ((oldState != getState()) &&
(((recoveredFinalState == null)) ||
(event.getType() != RMAppEventType.RECOVER))) {
LOG.info(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
getState(), event.getType()));
} else if ((oldState != getState()) && LOG.isDebugEnabled()) {
LOG.debug(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
getState(), event.getType()));
}
} finally {
this.writeLock.unlock();
}
}
状态机的核心StateMachineFactory类
StateMachineFactory类中的stateMachineTable对象就是状态机
它的类型是两层Map：Map，
外层Map的key表示旧状态，内层Map的key表示事件类型；
内层Map的value是Transition接口，OPERAND表示操作对象，STATE表示目的状态，EVENTTYPE表示事件类型，EVENT表示事件。
stateMachineTable起到的作用是：RMAppImpl可能有多种旧状态，每种旧状态可以对应多种事件类型，根据旧状态和要处理事件的类型，就能找到处理这种情形的状态转换方法和目的状态，同时状态转换方法包含对事件的处理。
核心成员
- transitionsListNode：就是将状态机的一个个过渡的ApplicableTransition实现串联为一个链表，每个节点包含一个ApplicableTransition实现及指向下一个节点的引用
- stateMachineTable : 状态拓扑表,为了提高检索状态对应的过渡map而冗余的数据结构，此结构在optimized为真时，通过对transitionsListNode链表进行处理产生
- defaultInitialState：对象创建时，内部有限状态机的默认初始状态。比如：RMAppImpl的内部状态机默认初始状态是RMAppState.NEW。
- optimized：布尔类型，用于标记当前状态机是否需要优化性能，即构建状态拓扑表stateMachineTable。
状态转换
-
元素：当前状态，目标状态，事件，回调函数。
-
简单转换：一个当前状态、 一个目标状态、 一种触发事件、 零/一个回调
简单转化函数
目标状态
执行转换方法
初始状态
触发事件
回调函数
-
多事件转换：一个初始状态、 一个最终状态、 多种事件 、 零/一个回调
多事件转换目标状态执行转换方法初始状态触发事件1触发事件2触发事件3回调函数 -
多状态转换：一个初始状态、 多个最终状态、 一种事件、多个回调
多状态转换执行转换方法回调函数1回调函数2回调函数3初始状态目标状态1目标状态2目标状态3触发事件1// 简单转换 public StateMachineFactory <OPERAND, STATE, EVENTTYPE, EVENT> addTransition(STATE preState, STATE postState, EVENTTYPE eventType) { return addTransition(preState, postState, eventType, null); } // 多事件单一状态转换 public StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> addTransition( STATE preState, STATE postState, Set<EVENTTYPE> eventTypes, SingleArcTransition<OPERAND, EVENT> hook) { StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> factory = null; for (EVENTTYPE event : eventTypes) { if (factory == null) { factory = addTransition(preState, postState, event, hook); } else { factory = factory.addTransition(preState, postState, event, hook); } } return factory; } // 简单转换 public StateMachineFactory <OPERAND, STATE, EVENTTYPE, EVENT> addTransition(STATE preState, STATE postState, EVENTTYPE eventType, SingleArcTransition<OPERAND, EVENT> hook){ return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> (this, new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT> (preState, eventType, new SingleInternalArc(postState, hook))); } // 单事件多状态转换 MultipleInternalArc类中的转换方法会将回调的状态与预存的状态们进行正确性匹配，不匹配将抛出异常。 public StateMachineFactory <OPERAND, STATE, EVENTTYPE, EVENT> addTransition(STATE preState, Set<STATE> postStates, EVENTTYPE eventType, MultipleArcTransition<OPERAND, EVENT, STATE> hook){ return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> (this, new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT> (preState, eventType, new MultipleInternalArc(postStates, hook))); }
InternalStateMachine
状态机的具体实现类，主要成员操作对象operand，当前状态currentState，状态转换的监控器StateTransitionListener（有点类似AOP的处理方式）,主要事件处理方法doTransition。
private class InternalStateMachine
implements StateMachine<STATE, EVENTTYPE, EVENT> {
private final OPERAND operand;
private STATE currentState;
private final StateTransitionListener<OPERAND, EVENT, STATE> listener;
...
...
...
@Override
public synchronized STATE doTransition(EVENTTYPE eventType, EVENT event)
throws InvalidStateTransitionException {
// 前置监听处理
listener.preTransition(operand, currentState, event);
STATE oldState = currentState;
currentState = StateMachineFactory.this.doTransition
(operand, currentState, eventType, event);
// 后驱监听处理
listener.postTransition(operand, oldState, currentState, event);
return currentState;
}
}
简化实验
简化类图
implements
implements
implements
implements
«interface»
StateMachine
+STATE getCurrentState()
+STATE doTransition(EVENTTYPE eventType, EVENT event)
«interface»
SingleArcTransition
+void transition(OPERAND operand, EVENT event)
«interface»
MultipleArcTransition
+STATE transition(OPERAND operand, EVENT event)
«interface»
Transition
STATE doTransition(OPERAND operand, STATE oldState, EVENT event, EVENTTYPE eventType)
«interface»
ApplicableTransition
void apply(StateMachineFactory subject)
ApplicableSingleOrMultipleTransition
final STATE preState
final EVENTTYPE eventType
final Transition transition
ApplicableSingleOrMultipleTransition(STATE preState, EVENTTYPE eventType, Transition transition)
+void apply(StateMachineFactory subject)
SingleInternalArc
- STATE postState;
- SingleArcTransition hook;
+STATE doTransition(OPERAND operand, STATE oldState, EVENT event, EVENTTYPE eventType)
MultipleInternalArc
- Set validPostStates
- MultipleArcTransition hook
+STATE doTransition(OPERAND operand, STATE oldState, EVENT event, EVENTTYPE eventType)
TransitionsListNode
final ApplicableTransition transition
final TransitionsListNode next
+STATE doTransition(OPERAND operand, STATE oldState, EVENT event, EVENTTYPE eventType)
InternalStateMachine
final OPERAND operand
STATE currentState
final StateTransitionListener listener
+synchronized STATE getCurrentState()
+synchronized STATE doTransition(EVENTTYPE eventType, EVENT event)
StateMachineFactory
- final TransitionsListNode transitionsListNode
- Map>> stateMachineTable
final StateTransitionListener listener
- STATE defaultInitialState
- final boolean optimized
+StateMachineFactory addTransition(STATE preState, STATE postState, EVENTTYPE eventType, SingleArcTransition hook)
+StateMachineFactory addTransition(STATE preState, Set postStates, EVENTTYPE eventType, MultipleArcTransition hook)
-STATE doTransition(OPERAND operand, STATE oldState, EVENTTYPE eventType, EVENT event)
事件类型
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public enum MyEventType {
START,
Change1,
Change2
}
任务状态
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public enum MyState {
NEW,
State_1,
State_2,
FINISHED
}
状态机框架相关
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public interface SingleArcTransition<OPERAND, EVENT> {
public void transition(OPERAND operand, EVENT event);
}
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public interface MultipleArcTransition<OPERAND, EVENT, STATE extends Enum<STATE>> {
public STATE transition(OPERAND operand, EVENT event);
}
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public interface ApplicableTransition<OPERAND, STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
void apply(StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> subject);
}
package com.donny.state;
import java.util.HashMap;
import java.util.Map;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public class ApplicableSingleOrMultipleTransition<OPERAND, STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> implements ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> {
final STATE preState;
final EVENTTYPE eventType;
final Transition<OPERAND, STATE, EVENTTYPE, EVENT> transition;
ApplicableSingleOrMultipleTransition
(STATE preState, EVENTTYPE eventType,
Transition<OPERAND, STATE, EVENTTYPE, EVENT> transition) {
this.preState = preState;
this.eventType = eventType;
this.transition = transition;
}
@Override
public void apply
(StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> subject) {
Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>> transitionMap
= subject.getStateMachineTable().get(preState);
if (transitionMap == null) {
transitionMap = new HashMap<EVENTTYPE,
Transition<OPERAND, STATE, EVENTTYPE, EVENT>>();
subject.getStateMachineTable().put(preState, transitionMap);
}
transitionMap.put(eventType, transition);
}
}
package com.donny.state;
import java.util.Set;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public class MultipleInternalArc<OPERAND, STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> implements Transition<OPERAND, STATE, EVENTTYPE, EVENT> {
private Set<STATE> validPostStates;
private MultipleArcTransition<OPERAND, EVENT, STATE> hook; // transition hook
MultipleInternalArc(Set<STATE> postStates,
MultipleArcTransition<OPERAND, EVENT, STATE> hook) {
this.validPostStates = postStates;
this.hook = hook;
}
@Override
public STATE doTransition(OPERAND operand, STATE oldState,
EVENT event, EVENTTYPE eventType)
throws RuntimeException {
STATE postState = hook.transition(operand, event);
if (!validPostStates.contains(postState)) {
throw new RuntimeException("oldState:" + oldState + " ,eventType:" + eventType);
}
return postState;
}
}
public class MyTransition implements SingleArcTransition {
@Override
public void transition(Object o, Object o2) {
System.out.println("do transition");
}
}
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public class SingleInternalArc<OPERAND, STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> implements Transition<OPERAND, STATE, EVENTTYPE, EVENT> {
private STATE postState;
private SingleArcTransition<OPERAND, EVENT> hook; // transition hook
SingleInternalArc(STATE postState,
SingleArcTransition<OPERAND, EVENT> hook) {
this.postState = postState;
this.hook = hook;
}
@Override
public STATE doTransition(OPERAND operand, STATE oldState,
EVENT event, EVENTTYPE eventType) {
if (hook != null) {
hook.transition(operand, event);
}
return postState;
}
}
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public interface Transition<OPERAND, STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
STATE doTransition(OPERAND operand, STATE oldState, EVENT event, EVENTTYPE eventType);
}
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public interface StateMachine<STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
public STATE getCurrentState();
public STATE doTransition(EVENTTYPE eventType, EVENT event)
throws RuntimeException;
}
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public class TransitionsListNode<OPERAND, STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
final ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> transition;
final TransitionsListNode next;
TransitionsListNode
(ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> transition,
TransitionsListNode next) {
this.transition = transition;
this.next = next;
}
}
package com.donny.state;
import java.util.*;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public class StateMachineFactory<OPERAND, STATE extends Enum<STATE>,
EVENTTYPE extends Enum<EVENTTYPE>, EVENT> {
private final TransitionsListNode transitionsListNode;
private Map<STATE, Map<EVENTTYPE,
Transition<OPERAND, STATE, EVENTTYPE, EVENT>>> stateMachineTable;
private STATE defaultInitialState;
private final boolean optimized;
public StateMachineFactory(STATE defaultInitialState) {
this.transitionsListNode = null;
this.defaultInitialState = defaultInitialState;
this.optimized = false;
this.stateMachineTable = null;
}
private StateMachineFactory
(StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> that,
ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT> t) {
this.defaultInitialState = that.defaultInitialState;
this.transitionsListNode
= new TransitionsListNode(t, that.transitionsListNode);
this.optimized = false;
this.stateMachineTable = null;
}
public StateMachineFactory
<OPERAND, STATE, EVENTTYPE, EVENT>
installTopology() {
return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>(this, true);
}
private StateMachineFactory
(StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT> that,
boolean optimized) {
this.defaultInitialState = that.defaultInitialState;
this.transitionsListNode = that.transitionsListNode;
this.optimized = optimized;
if (optimized) {
makeStateMachineTable();
} else {
stateMachineTable = null;
}
}
private void makeStateMachineTable() {
Stack<ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT>> stack =
new Stack<ApplicableTransition<OPERAND, STATE, EVENTTYPE, EVENT>>();
Map<STATE, Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>>>
prototype = new HashMap<STATE, Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>>>();
prototype.put(defaultInitialState, null);
stateMachineTable
= new EnumMap<STATE, Map<EVENTTYPE,
Transition<OPERAND, STATE, EVENTTYPE, EVENT>>>(prototype);
for (TransitionsListNode cursor = transitionsListNode;
cursor != null;
cursor = cursor.next) {
stack.push(cursor.transition);
}
while (!stack.isEmpty()) {
stack.pop().apply(this);
}
}
public Map<STATE, Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>>> getStateMachineTable() {
return stateMachineTable;
}
private STATE doTransition
(OPERAND operand, STATE oldState, EVENTTYPE eventType, EVENT event)
throws RuntimeException {
Map<EVENTTYPE, Transition<OPERAND, STATE, EVENTTYPE, EVENT>> transitionMap
= stateMachineTable.get(oldState);
if (transitionMap != null) {
Transition<OPERAND, STATE, EVENTTYPE, EVENT> transition
= transitionMap.get(eventType);
if (transition != null) {
return transition.doTransition(operand, oldState, event, eventType);
}
}
throw new RuntimeException("RuntimeException: " + oldState + eventType);
}
public StateMachineFactory
<OPERAND, STATE, EVENTTYPE, EVENT>
addTransition(STATE preState, STATE postState,
EVENTTYPE eventType,
SingleArcTransition<OPERAND, EVENT> hook) {
return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>
(this, new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT>
(preState, eventType, new SingleInternalArc(postState, hook)));
}
public StateMachineFactory
<OPERAND, STATE, EVENTTYPE, EVENT>
addTransition(STATE preState, Set<STATE> postStates,
EVENTTYPE eventType,
MultipleArcTransition<OPERAND, EVENT, STATE> hook) {
return new StateMachineFactory<OPERAND, STATE, EVENTTYPE, EVENT>
(this,
new ApplicableSingleOrMultipleTransition<OPERAND, STATE, EVENTTYPE, EVENT>
(preState, eventType, new MultipleInternalArc(postStates, hook)));
}
private synchronized void maybeMakeStateMachineTable() {
if (stateMachineTable == null) {
makeStateMachineTable();
}
}
private class InternalStateMachine
implements StateMachine<STATE, EVENTTYPE, EVENT> {
private final OPERAND operand;
private STATE currentState;
InternalStateMachine(OPERAND operand, STATE initialState) {
this.operand = operand;
this.currentState = initialState;
if (!optimized) {
maybeMakeStateMachineTable();
}
}
@Override
public synchronized STATE getCurrentState() {
return currentState;
}
@Override
public synchronized STATE doTransition(EVENTTYPE eventType, EVENT event)
throws RuntimeException {
STATE oldState = currentState;
currentState = StateMachineFactory.this.doTransition
(operand, currentState, eventType, event);
return currentState;
}
}
public StateMachine<STATE, EVENTTYPE, EVENT> make(OPERAND operand) {
return new InternalStateMachine(operand, defaultInitialState);
}
}
package com.donny.state;
/**
* @author [email protected]
* @description
* @date 2023/9/27
*/
public class Test {
private final StateMachine stateMachine;
private static final StateMachineFactory stateMachineFactory
= new StateMachineFactory(MyState.NEW)
.addTransition(MyState.NEW, MyState.State_1, MyEventType.START, new MyTransition())
.addTransition(MyState.State_1, MyState.State_2, MyEventType.Change1, new MyTransition())
.addTransition(MyState.State_2, MyState.FINISHED, MyEventType.Change2, new MyTransition())
.installTopology();
public Test() {
this.stateMachine = stateMachineFactory.make(this);
}
public static void main(String[] args) {
Test t = new Test();
System.out.println(t.stateMachine.getCurrentState());
Object event = new Object();
t.stateMachine.doTransition(MyEventType.START, event);
System.out.println(t.stateMachine.getCurrentState());
t.stateMachine.doTransition(MyEventType.Change1, event);
System.out.println(t.stateMachine.getCurrentState());
t.stateMachine.doTransition(MyEventType.Change2, event);
System.out.println(t.stateMachine.getCurrentState());
}
}
实验结果
NEW
do transition
State_1
do transition
State_2
do transition
FINISHED