AKKA Cluster 原理及实战

Akka集群支持去中心化的基于P2P的集群服务，没有单点故障（SPOF）问题，它主要是通过Gossip协议和向量时钟（VECTOR CLOCKS）来实现。对于集群成员的状态，Akka提供了一种故障检测机制，能够自动发现出现故障而离开集群的成员节点，通过事件驱动的方式，将状态传播到整个集群的其它成员节点。

关于GOSSIP PROTOCOL和向量时钟（VECTOR CLOCKS）：
akka使用了一种push-pull的gossip变种来减少gossip信息的大小。 在push-pull gossip中使用一个摘要来代表当前版本而非使用实际的值。 如果接受者如果有更新的版本，可以发回任意值。如果接受者拥有过期版本，也可以请求实际值。 Akka使用包含一个vector lock的共享状态来控制版本， 所以Akka中的push-pull gossip变种仅在需要时使用这个版本来推送实际的状态。以1秒为周期，每个节点随机的选择另一个节点进行一轮gossip。如果seen set中的节点少于一半那么集群每秒钟会进行3轮gossip。 这个调整机制可以加速状态改变后早期传播阶段的收敛过程。

gossip节点的选择是随机的但是倾向于选择有可能未看到当前版本状态的节点。 在每次gossip信息交换中且集群未能收敛时，节点使用0.8（可配置）的概率来和没在seen set的节点进行gossip。 否则就跟任意活着的节点进行gossip。这种带有倾向性的选择是为了提高状态变化后期传播阶段的收敛速度。

对包含了多于400个节点的集群（可配置，400是经验值），0.8这个概率应该逐渐降低以避免过多的并发gossip把单个掉队者淹没。 gossip接受者同样有一种机制来保护自己，他会将mailbox中入队时间过长的消息丢弃。

当集群在一个收敛状态时参与gossip的节点仅会向选定的节点发送一个很小的包含了gossip版本的状态消息。 集群状态一发生变化（意味着不再收敛），集群就再次回到有倾向性的gossip。

接受者可以使用向量时钟（VECTOR CLOCKS）来确定状态：
1、它拥有一个较新版本的gossip状态，此时它会将这个状态发回去
2、它拥有的是一个过时的状态，此时它会发回它自己的gossip状态并请求当前状态
3、它拥有的是一个冲突的状态，此时不同的版本会被合并后再发回
4、如果接收者和gossip的版本相同，gossip状态就不再发送或请求

gossip周期性的特性对于状态改变有非常好的批处理效果。例如，短时间内加入集群内同一个节点的一批节点，只会引起一次状态变化。另外，gossip消息会使用protobuf序列化同时再进行gzip压缩来降低负载。

关于SEED NODES：
种子节点是为新节点加入集群而配置的联系点。当一个新节点启动后它会向所有种子节点发消息，然后向第一个应答的种子节点发送join命令。

种子节点的配置值对运行中的集群本身没有任何影响，它仅与有新节点加入集群联系，因为它帮助新节点找到join命令的接受者。 一个新成员可以发送这个命令到任意现有集群节点，不必是种子节点。

关于leader：
在gossip收敛后集群可以决定一个leader。其间并不存在leader选举的过程，只要达到gossip收敛，leader总能被任意的节点确定认出。 leader仅仅是一个角色，任何节点都可以成为leader，leader在各轮收敛也可能改变。 leader是有序节点中可作为leader的第一个节点，leader的member状态可以是up和leaving。

leader负责将集群的成员移入或移出集群，将加入集群的成员状态改为up或将已存在的成员改为removed状态。 leader的动作仅在收到一个包含gossip收敛的新集群状态时被触发。

经过配置，leader可以根据Failure Detector来“auto-down”一个unreachable节点。 这意味着在一个配置的时间内，自动将unreachable节点设置为down。

关于状态转移：
Akka内部为集群成员定义了一组有限状态（6种状态），并给出了一个状态转移矩阵。Akka集群中的每个成员节点，都有可能处于上面的一种状态，在发生某些事件以后，会发生状态转移。需要注意的是，除了Down和Removed状态以外，节点处于其它任何一个状态时都有可能变成Down状态，即节点故障而无法提供服务，而在变成Down状态之前有一个虚拟的Unreachable状态，因为在Gossip收敛过程中，是无法到达或者经由Unreachable状态的节点，这个状态是由Akka实现的故障探测器（Failure Detector）来检测到的。处于Down状态的节点如果想要再次加入Akka集群，需要重新启动，并进入Joining状态，然后才能进行后续状态的转移变化。Akka集群成员节点状态及其转移情况，如下图所示：

节点状态发生转移会触发某个事件，我们可以根据不同类型的事件来进行响应处理。AKKA事件集合如下图所示：

下面代码对上述进行测试，基于现成例子进行修改：

package sample.cluster.simple;

import akka.actor.UntypedActor;
import akka.cluster.Cluster;
import akka.cluster.ClusterEvent;
import akka.cluster.ClusterEvent.LeaderChanged;
import akka.cluster.ClusterEvent.MemberEvent;
import akka.cluster.ClusterEvent.MemberUp;
import akka.cluster.ClusterEvent.MemberRemoved;
import akka.cluster.ClusterEvent.UnreachableMember;
import akka.event.Logging;
import akka.event.LoggingAdapter;

public class SimpleClusterListener extends UntypedActor {
  LoggingAdapter log = Logging.getLogger(getContext().system(), this);
  Cluster cluster = Cluster.get(getContext().system());

  //subscribe to cluster changes
  @Override
  public void preStart() {
    //#subscribe
    cluster.subscribe(getSelf(), ClusterEvent.initialStateAsEvents(), 
        MemberEvent.class, UnreachableMember.class,LeaderChanged.class);

  }

  //re-subscribe when restart
  @Override
  public void postStop() {
    cluster.unsubscribe(getSelf());
  }

  @Override
  public void onReceive(Object message) {
    if (message instanceof MemberUp) {
      MemberUp mUp = (MemberUp) message;
      log.info("!!!!!!!!!!!!!!!Member is Up: {}", mUp.member());
    } else if (message instanceof UnreachableMember) {
      UnreachableMember mUnreachable = (UnreachableMember) message;
      log.info("Member detected as unreachable: {}", mUnreachable.member());

    } else if (message instanceof MemberRemoved) {
      MemberRemoved mRemoved = (MemberRemoved) message;
      log.info("Member is Removed: {}", mRemoved.member());

    } else if (message instanceof MemberEvent) {
      // ignore
    } else if (message instanceof LeaderChanged) {
        LeaderChanged leader = (LeaderChanged) message;
        log.info("##############leader is changed: {}", leader.leader());
    }
    else {
      unhandled(message);
    }

  }
}

可以看到，每个节点都订阅了MemberEvent和LeaderChanged消息，并且分别增加前缀！！！！和####方便观测。启动后集群的日志如下：

[INFO] [09/05/2016 22:13:19.069] [main] [Remoting] Starting remoting
[INFO] [09/05/2016 22:13:19.319] [main] [Remoting] Remoting started; listening on addresses :[akka.tcp://ClusterSystem@127.0.0.1:2551]
[INFO] [09/05/2016 22:13:19.333] [main] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Starting up...
[INFO] [09/05/2016 22:13:19.423] [main] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [09/05/2016 22:13:19.423] [main] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Started up successfully
[INFO] [09/05/2016 22:13:19.438] [ClusterSystem-akka.actor.default-dispatcher-15] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
[INFO] [09/05/2016 22:13:19.441] [ClusterSystem-akka.actor.default-dispatcher-15] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Metrics collection has started successfully
[INFO] [09/05/2016 22:13:19.479] [main] [Remoting] Starting remoting
[INFO] [09/05/2016 22:13:19.500] [main] [Remoting] Remoting started; listening on addresses :[akka.tcp://ClusterSystem@127.0.0.1:2552]
[INFO] [09/05/2016 22:13:19.502] [main] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Starting up...
[INFO] [09/05/2016 22:13:19.506] [main] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Started up successfully
[INFO] [09/05/2016 22:13:19.507] [ClusterSystem-akka.actor.default-dispatcher-3] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
[INFO] [09/05/2016 22:13:19.508] [ClusterSystem-akka.actor.default-dispatcher-3] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Metrics collection has started successfully
[INFO] [09/05/2016 22:13:19.533] [main] [Remoting] Starting remoting
[INFO] [09/05/2016 22:13:19.594] [main] [Remoting] Remoting started; listening on addresses :[akka.tcp://ClusterSystem@127.0.0.1:58413]
[INFO] [09/05/2016 22:13:19.601] [main] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58413] - Starting up...
[INFO] [09/05/2016 22:13:19.630] [main] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58413] - Started up successfully
[INFO] [09/05/2016 22:13:19.658] [ClusterSystem-akka.actor.default-dispatcher-5] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58413] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar
[INFO] [09/05/2016 22:13:19.658] [ClusterSystem-akka.actor.default-dispatcher-5] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58413] - Metrics collection has started successfully
[INFO] [09/05/2016 22:13:19.743] [ClusterSystem-akka.actor.default-dispatcher-17] [akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess] Message [akka.dispatch.sysmsg.Terminate] from Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess#-822989391] to Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess#-822989391] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [09/05/2016 22:13:19.746] [ClusterSystem-akka.actor.default-dispatcher-4] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://ClusterSystem@127.0.0.1:2551] is JOINING, roles []
[INFO] [09/05/2016 22:13:19.756] [ClusterSystem-akka.actor.default-dispatcher-16] [akka://ClusterSystem/user/clusterListener2551] ##############leader is changed: Some(akka.tcp://ClusterSystem@127.0.0.1:2551)
[INFO] [09/05/2016 22:13:20.456] [ClusterSystem-akka.actor.default-dispatcher-15] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://ClusterSystem@127.0.0.1:2551] to [Up]
[INFO] [09/05/2016 22:13:20.457] [ClusterSystem-akka.actor.default-dispatcher-15] [akka://ClusterSystem/user/clusterListener2551] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:2551, status = Up)
[INFO] [09/05/2016 22:13:24.783] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/system/cluster/core/daemon/joinSeedNodeProcess] Message [akka.cluster.InternalClusterAction$InitJoinNack] from Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/system/cluster/core/daemon#1894952621] to Actor[akka://ClusterSystem/system/cluster/core/daemon/joinSeedNodeProcess#-753525179] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [09/05/2016 22:13:24.791] [ClusterSystem-akka.actor.default-dispatcher-16] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://ClusterSystem@127.0.0.1:58413] is JOINING, roles []
[INFO] [09/05/2016 22:13:24.793] [ClusterSystem-akka.actor.default-dispatcher-2] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Node [akka.tcp://ClusterSystem@127.0.0.1:2552] is JOINING, roles []
[INFO] [09/05/2016 22:13:24.942] [ClusterSystem-akka.actor.default-dispatcher-4] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:58413] - Welcome from [akka.tcp://ClusterSystem@127.0.0.1:2551]
[INFO] [09/05/2016 22:13:24.942] [ClusterSystem-akka.actor.default-dispatcher-18] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2552] - Welcome from [akka.tcp://ClusterSystem@127.0.0.1:2551]
[INFO] [09/05/2016 22:13:24.944] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/clusterListener2552] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:2551, status = Up)
[INFO] [09/05/2016 22:13:24.944] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/clusterListener0] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:2551, status = Up)
[INFO] [09/05/2016 22:13:24.945] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/clusterListener2552] ##############leader is changed: Some(akka.tcp://ClusterSystem@127.0.0.1:2551)
[INFO] [09/05/2016 22:13:24.944] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/clusterListener0] ##############leader is changed: Some(akka.tcp://ClusterSystem@127.0.0.1:2551)
[INFO] [09/05/2016 22:13:25.440] [ClusterSystem-akka.actor.default-dispatcher-15] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://ClusterSystem@127.0.0.1:2552] to [Up]
[INFO] [09/05/2016 22:13:25.440] [ClusterSystem-akka.actor.default-dispatcher-15] [Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://ClusterSystem@127.0.0.1:2551] - Leader is moving node [akka.tcp://ClusterSystem@127.0.0.1:58413] to [Up]
[INFO] [09/05/2016 22:13:25.444] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/clusterListener2551] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:2552, status = Up)
[INFO] [09/05/2016 22:13:25.444] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/clusterListener2551] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:58413, status = Up)
[INFO] [09/05/2016 22:13:25.461] [ClusterSystem-akka.actor.default-dispatcher-18] [akka://ClusterSystem/user/clusterListener2552] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:2552, status = Up)
[INFO] [09/05/2016 22:13:25.461] [ClusterSystem-akka.actor.default-dispatcher-18] [akka://ClusterSystem/user/clusterListener2552] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:58413, status = Up)
[INFO] [09/05/2016 22:13:25.517] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/clusterListener0] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:2552, status = Up)
[INFO] [09/05/2016 22:13:25.517] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/clusterListener0] !!!!!!!!!!!!!!!Member is Up: Member(address = akka.tcp://ClusterSystem@127.0.0.1:58413, status = Up)

可以看到，集群中的三个节点都收到了MemberUp和LeaderChange通知。这样，我们便轻松的获取了集群的leader及集群中的成员，方便我们开发无单点故障的分布式服务。