Storm源码阅读（二）：客户端

Clojure

Clojure指南http://java.ociweb.com/mark/clojure/article.html

 

Eclipse插件Counterclockwise

Eclipse提供了专门的Clojure 语言开发插件CounterClockwise，在源代码编辑，代码调试，REPL 支持方面也有独到之处，适合于习惯于Eclipse的开发者使用。

http://code.google.com/p/counterclockwise/wiki/Documentation#Install_Counterclockwise

 

storm源码主要目录结构：

.

|-- bin

|   |-- storm 执行外壳storm

|   `-- to_maven.sh

|-- conf storm的配置

|   |-- defaults

|   `-- storm.yaml.example

|-- storm-core

|   |-- src

|   |  |-- clj     :clojure代码

|   |  |-- dev     : python,ruby测试代码

|   |  |-- jvm     : java代码

|   |  |-- multilang : 提供了python和ruby的storm接口

|   |  |-- py    :

|   |  |-- ui        :公用js，css

 

 

Storm中使用Thrifit进行客户端、服务器端通讯。由genthrift.sh生成的文件包括

jvm/backtype/storm/generated目录

py 目录

 

 

客户端

 

运行一个topology很简单。首先，把你所有的代码以及所依赖的jar打进一个jar包。然后运行storm脚本，格式类似如下：

        stormjar all-my-code.jar backtype.storm.MyTopology arg1 arg2

backtype.storm.MyTopology是你的主类，比如storm.starter.WordCountTopology, 参数是arg1,arg2。storm jar负责连接到Nimbus并且上传jar包。

 

//storm脚本内部实现

"""Syntax: [storm jar topology-jar-path class ...]”””

def jar(jarfile, klass, *args):

    exec_storm_class(

        klass,

        jvmtype="-client",

        extrajars=[jarfile, USER_CONF_DIR, STORM_DIR +"/bin"],

        args=args,

        jvmopts=["-Dstorm.jar="+ jarfile])

 

 

def exec_storm_class(klass, jvmtype="-server", jvmopts=[], extrajars=[],args=[], fork=False):

    global CONFFILE

    all_args = [

        "java", jvmtype, get_config_opts(),

        "-Dstorm.home=" +STORM_DIR,

        "-Djava.library.path=" + confvalue("java.library.path", extrajars),

       "-Dstorm.conf.file=" + CONFFILE,

        "-cp", get_classpath(extrajars),

    ] + jvmopts+ [klass] + list(args)

    print "Running: " +" ".join(all_args)

    if fork:

        os.spawnvp(os.P_WAIT,"java", all_args)

    else:

        os.execvp("java",all_args) # replaces the current process and never returns

 

也就是执行如下命令

Java –Dstorm.home=xxx–Djava.libary.path=xxx –Dstorm.conf.file=xxx –cp xxx jvmopts kclass args

 

在你提交的jar包中，主类一般都会最后调用StormSubmitter.submitTopology。比如strom-starter的WordCountTopology：

TopologyBuilder builder = new TopologyBuilder();

builder.setSpout("spout", newRandomSentenceSpout(), 5);

…

StormSubmitter.submitTopology(args[0], conf, builder.createTopology());

 

 

以这个Storm客户端的入口，跟踪源码。

 

 

//https://github.com/nathanmarz/storm/wiki/Lifecycle-of-a-topology

//Starting atopology

//

 

//StormSubmiter.submitTopology

//读取配置default.yaml,storm.yaml ("storm.conf.file")

Map conf = Utils.readStormConfig();

Conf.putAll(stormConf)

String serConf = JSONValue.toJSONString(stormConf);//json化

if(localNimbus!=null) {

    // local模式，调用localNimbus。

    // localNimbus对象LocalCluster.clj实现类, 实现ILocalCluster 接口

    localNimbus.submitTopology(name,null, serConf, topology);

} else {

    NimbusClientclient = NimbusClient.getConfiguredClient(conf);

    // 检查ClusterInfo是否有同名topology

    …

    //上传jar包，命令中指定的，被赋值为ENV"storm.jar"

    submitJar(conf);

try {

        LOG.info("Submitting topology" +name + "in distributed mode”);

        if(opts!=null) {

          client.getClient().submitTopologyWithOpts(name,submittedJar, serConf,topology, opts);

        } else {

          // this is for backwards compatibility

          client.getClient().submitTopology(name, submittedJar, serConf, topology);                                           

        }

    } catch(InvalidTopologyExceptione) {

…

 

//NimbusClient.getConfiguredClient

    //NImbusClient继承ThriftClient，根据配置的host,port创建Nimbus.Client

    public static NimbusClientgetConfiguredClient(Map conf) {

        try {

            StringnimbusHost = (String) conf.get(Config.NIMBUS_HOST);

            int nimbusPort =Utils.getInt(conf.get(Config.NIMBUS_THRIFT_PORT));

            return new NimbusClient(conf,nimbusHost, nimbusPort);

 

    //NimbusClient构造函数

    publicNimbusClient(Map conf, String host, int port, Integer timeout) throws ex{

        super(conf, host,port, timeout);

        _client = new Nimbus.Client(_protocol);

    }

 

    //NimbusClient的父类ThriftClient：和Thrift的边界

    //super(conf,host, port, timeout)

        //安全配置java.security.auth.login.config

            //constructa transport plugin

           ITransportPlugin transportPlugin = AuthUtils.GetTransportPlugin(storm_conf

, login_conf);

 

        //创建TSocket，

             TSocket socket = newTSocket(host, port);

           _transport = transportPlugin.connect(socket, host);

            _protocol = new TBinaryProtocol(_transport);

 

    //NimbusClient._client为Nimbus$Client对象，继承TServiceClient

    //Nimbus$Client 继承TServiceClient（包含提交待发送的topology）

    //TServiceClient只定义了发送接收接口，sendBase，receiveBase

    //似乎两边通过methodName交流

    //sendBase/receiveBase都要提供methodName

    //Nimbus$Client所有提供的方法最后调用TServiceClient发送接收

    //比如：

 

//submitJar

    //获得上传路径、上传文件块，结束上传

    NimbusClientclient = NimbusClient.getConfiguredClient(conf);

    try {

        String uploadLocation = client.getClient().beginFileUpload();

        BufferFileInputStream is = newBufferFileInputStream(localJar);

        while(true) {

            byte[] toSubmit =is.read();

            if(toSubmit.length==0) break;

            client.getClient().uploadChunk(uploadLocation,ByteBuffer.wrap(toSubmit));

        }

        client.getClient().finishFileUpload(uploadLocation);

    }

 

    public String beginFileUpload()throws org.apache.thrift7.TException

    {

     send_beginFileUpload();

      returnrecv_beginFileUpload();

    }

 

    public voidsend_beginFileUpload() throws org.apache.thrift7.TException {

      beginFileUpload_args args = newbeginFileUpload_args();

      sendBase("beginFileUpload", args);

    }

 

    protected voidsendBase(String methodName,TBase args) throws TException {

      oprot_.writeMessageBegin(new TMessage(methodName,TMessageType.CALL, ++seqid_));

      args.write(oprot_);

      oprot_.writeMessageEnd();

      oprot_.getTransport().flush();

    }

 

    protected void receiveBase(TBase result, String methodName) throws TException{

      TMessage msg = iprot_.readMessageBegin();

      if (msg.seqid != seqid_) {

        throw new TApplicationException(“xxx");

      }

      result.read(iprot_);

      iprot_.readMessageEnd();

    }

 

 

//接下来调用的submitTopology方法也是类似的

    //send_submitTopology

     submitTopology_args args = newsubmitTopology_args();

     args.set_name(name);

     args.set_uploadedJarLocation(uploadedJarLocation);

     args.set_jsonConf(jsonConf);

     args.set_topology(topology);

      sendBase("submitTopology", args);

 

 

看来客户端和服务器端通讯时，走的是通thrift，并且每个方法都有对应的methodName。为此，到backtype.storm.generated.Nimbus中求证一下。

 

Nimubs客户端backtype.storm.generated.Nimbus由thrift产生，包括了两个客户端Client/AsyncClient，分别实现Iface/AsyncIface接口。这两个接口定义的方法还分别有对应的参数、结果类。另外还有一个Processor接口实现类(在Nimubus服务器端调用)

 

Iface/AsyncIface接口实现将调用send_xxx和recv_xxx。send_xxx和recv_xxx分别有对应的Client内部类xxx_args，xxx_result包装，然后往下调用sendBase/receiveBase。接着就交给Thrift(TServiceClient)了。

 

我们可以找到Thrift提供的服务如下：

getClusterInfo ：获取Cluster的ClusterSummary，包括topology，supervisors信息

beginFileUpload ：开始上传文件

uploadChunk ：上传文件块

finishFileUpload ：上传文件结束

submitTopologyWithOpts ：提交topology

submitTopology : …

…