Solr/Lucene分布式搜索,Solr Integrate katta step2

Katta的运行是需要ZooKeeper的,下载:http://archive.apache.org/dist/zookeeper/

解压安装: tar -zxvf zookeeper.${version}.tar.gz -C /opt

复制${ZP_INSTALL}/conf/zoo_sample.cfg并更名为zoo.xfg.

# The number of milliseconds of each tick
tickTime=5000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/data/zookeeper

dataLogDir=/data/zookeeper/log

# the port at which the clients will connect
clientPort=2181

其中具体的含义可以从google上获得,zookeeper我这里是单击模式安装的,正式的环境肯定不能这样.ZooKeeper集群安装请见:http://haoningabc.iteye.com/blog/1312080 我这里是测试开发,单机模式就OK

启动ZooKeeper.从终端切到/opt/zookeeper-3.3.6目录:

bin/zkServer.sh start

可以看到类似

/opt/zookeeper-3.3.6$ bin/zkServer.sh start
JMX enabled by default
Using config: /opt/zookeeper-3.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
/opt/zookeeper-3.3.6$ 

则启动成功.也可以使用bin/zkServer.sh status检查启动的状态.如:

$ bin/zkServer.sh status
JMX enabled by default
Using config: /opt/zookeeper-3.3.6/bin/../conf/zoo.cfg
Mode: standalone
/opt/zookeeper-3.3.6$ 

安装katta,把下载的Katta解压:tar -zxvf katta-0.6.3.tar.gz -C /opt

因为我使用的hadoop是1.0.3的,把hadoop-core-1.0.3.jar copy到/opt/katta-0.6.3,并删除原来的hadoop-core.同时copy的我记得还有几个文件.在一会儿启动Katta Master的时候会发现这些NoClassFoundError异常.

编辑${KATTA_INSTALL}/conf/katta.zk.properties文件,如:

zookeeper.embedded=false

# Comma serperated list of host:port of zookeeper servers used by the zookeeper clients. 
# Make sure the port is accessible from all nodes and use hostnames and not ip addresses.
# If embedded is set to true, the server hostname is compared to the servers list. 
# If found an embedded zookeeper server is started within the master or secondary master jvm. 
zookeeper.servers=localhost:2181


# The root zk node. This changes with every new cluster.
zookeeper.root-path=/katta
#zookeeper client timeout in milliseconds  
zookeeper.timeout=5000
#zookeeper tick time
zookeeper.tick-time=2000
# zookeeper init time limit
zookeeper.init-limit=5
# zookeeper sync limit
zookeeper.sync-limit=2
# zookeeper folder where data are stored
zookeeper.data-dir=./zookeeper-data
# zookeeper folder where log data are stored
zookeeper.log-data-dir=./zookeeper-log-data

从终端切到:/opt/katta-0.6.3,执行:bin/katta startMaster

$ bin/katta startMaster
12/10/18 13:05:41 INFO protocol.InteractionProtocol:350 - zhenqin-K45VM_f5f05eb0-f383-4e2a-87d6-d7a35ac6ba21 starting as master...
12/10/18 13:05:41 INFO protocol.InteractionProtocol:370 - master 'zhenqin-K45VM_f5f05eb0-f383-4e2a-87d6-d7a35ac6ba21' published
12/10/18 13:05:41 INFO upgrade.UpgradeRegistry:52 - version of distribution 0.6.3
12/10/18 13:05:41 INFO upgrade.UpgradeRegistry:53 - version of cluster 0.6.3
12/10/18 13:05:41 INFO master.Master:149 - start managing nodes...
12/10/18 13:05:41 INFO master.Master:177 - found following nodes connected: []
12/10/18 13:05:41 INFO master.OperatorThread:68 - starting...
12/10/18 13:05:41 INFO master.OperatorThread:150 - SAFE MODE: No nodes available or state unstable within the last 20000 ms.
12/10/18 13:05:46 INFO master.OperatorThread:150 - SAFE MODE: No nodes available or state unstable within the last 20000 ms.
12/10/18 13:05:51 INFO master.OperatorThread:150 - SAFE MODE: No nodes available or state unstable within the last 20000 ms.

则Katta的Master启动成功.如果出现错误,则又能缺少jar包,找到这些jar包放入${KATTA_INSTALL}/lib尝试 bin/katta startMaster 直到出现上边的字样则表示master成功启动.

在另一个终端中执行:

$bin/katta listIndices

------------------------------------------------------------------------------------------------------------------------
| Name      | Status   | Replication State | Path                                      | Shards | Entries | Disk Usage |
========================================================================================================================
| solrhome1 | DEPLOYED | UNDERREPLICATED   | file:///media/Study/data/katta/testIndexA | 1      | 27      | 258926     |
------------------------------------------------------------------------------------------------------------------------


1 registered indices

因为我加入了一个Solr Shard所以出现了 11 registered indices.

$ bin/katta listNodes

-------------------------------------------------------------------------
| Name (0/3 connected)  | Start time                     | State        |
=========================================================================
| zhenqin-K45VM:20000   | 星期二, 16 十月 2012 20:15:41 +0800 | DISCONNECTED |
-------------------------------------------------------------------------
| PC-20110225YCAY:20000 | 星期二, 16 十月 2012 13:18:47 +0800 | DISCONNECTED |
-------------------------------------------------------------------------
| zhenqin-K45VM:20001   | 星期一, 15 十月 2012 16:00:19 +0800 | DISCONNECTED |
-------------------------------------------------------------------------

可以看见我这里存在了3个Node节点,你的应该还没有.

$ bin/katta showStructure

+ (/katta)
'-+indicies
  '-+solrhome1
'-+work
  '--node-queues (contents hidden)
  '--flags (contents hidden)
  '--master-queue (contents hidden)
'-+shard-to-nodes
  '-+solrhome1#proxy
'-+master
'-+nodes
  '--metrics (contents hidden)
  '-+metadata
    '-+zhenqin-K45VM:20000
    '-+PC-20110225YCAY:20000
    '-+zhenqin-K45VM:20001
  '-+live
'-+version

其他命令可以从katta的源码中获得.在net.sf.katta.Katta类中

private final static List<Command> COMMANDS = new ArrayList<Command>();
    
COMMANDS.add(START_ZK_COMMAND);
    COMMANDS.add(START_MASTER_COMMAND);
    COMMANDS.add(START_NODE_COMMAND);
    COMMANDS.add(LIST_INDICES_COMMAND);
    COMMANDS.add(LIST_NODES_COMMAND);
    COMMANDS.add(LIST_ERRORS_COMMAND);
    COMMANDS.add(ADD_INDEX_COMMAND);
    COMMANDS.add(REMOVE_INDEX_COMMAND);
    COMMANDS.add(REDEPLOY_INDEX_COMMAND);
    COMMANDS.add(CHECK_COMMAND);
    COMMANDS.add(VERSION_COMMAND);
    COMMANDS.add(SHOW_STRUCTURE_COMMAND);
    COMMANDS.add(START_GUI_COMMAND);
    COMMANDS.add(LOG_METRICS_COMMAND);
    COMMANDS.add(GENERATE_INDEX_COMMAND);
    COMMANDS.add(SEARCH_COMMAND);
    COMMANDS.add(LOADTEST_COMMAND);
    COMMANDS.add(RUN_CLASS_COMMAND);

如startNode的Java代码:

  protected static Command START_NODE_COMMAND = new ProtocolCommand("startNode",
          "[-c <serverClass>] [-p <port number>]", "Starts a local node") {

    private NodeConfiguration _nodeConfiguration;
    private IContentServer _server = null;

    @Override
    protected void parseArguments(ZkConfiguration zkConf, String[] args, Map<String, String> optionMap) {
      _nodeConfiguration = new NodeConfiguration();
      String serverClassName;
      if (optionMap.containsKey("-c")) {
        serverClassName = optionMap.get("-c");
      } else {
        serverClassName = _nodeConfiguration.getServerClassName();
      }
      if (optionMap.containsKey("-p")) {
        String portNumber = optionMap.get("-p");
        _nodeConfiguration.setStartPort(Integer.parseInt(portNumber));
      }

      Class<?> serverClass = loadClass(serverClassName, IContentServer.class);
      try {
        _server = (IContentServer) serverClass.newInstance();
      } catch (Exception e) {
        throw new IllegalStateException("could not create instance of class '" + serverClassName + "': "
                + e.getMessage());
      }
    }

    @Override
    public void execute(ZkConfiguration zkConf, InteractionProtocol protocol) throws Exception {
      final Node node = new Node(protocol, _nodeConfiguration, _server);
      node.start();
      Runtime.getRuntime().addShutdownHook(new Thread() {
        @Override
        public void run() {
          node.shutdown();
        }
      });
      node.join();
    }

  };

step2 is over.