Apache Eagle安装配置指南

Apache Eagle开源没多久，网上关于安装它的文章几乎没有，在此我献上自己的安装步骤供大家参考。

官网安装文档地址：http://goeagle.io/docs/deployment-in-production.html

官网的安装大体步骤都没问题，只是在配置的时候需要注意几点。

1.环境依赖

HBase: 0.98 or later

Storm: 0.9.3 or later

Kafka: 0.8.x or later

Zookeeper: 3.4.5

Java: 1.7.x

Spark,用于用户模型训练

Logstash-1.5.5

2.Kafka配置

启动kafka前需要先配置kafka中的zookeeper地址

./kafka-server-start.sh ../config/server.properties &

创建eagle需要的topic

./kafka-topics.sh --create --zookeeper slave02.mesos:2181,slave03.mesos:2181,slave04.mesos:2181 --replication-factor 1 --partitions 1 --topic Demo_hdfs_audit_log

3.配置eagle

修改bin/eagle-env.sh

修改conf/eagle-service.conf

# eagle configuration
eagle{
        # eagle web service configuration
        service{
                # storage type: ["hbase","jdbc"]
                # default is "hbase"
                storage-type="hbase"

                # hbase configuration: hbase.zookeeper.quorum
                # default is "localhost"
                hbase-zookeeper-quorum="slave02.mesos,slave03.mesos,slave04.mesos"

                # hbase configuration: hbase.zookeeper.property.clientPort
                # default is 2181
                hbase-zookeeper-property-clientPort=2181

                # hbase configuration: zookeeper.znode.parent
                # default is "/hbase"
                zookeeper-znode-parent="/hbase"
        }
}

新建一个conf/ Demo-hdfsAuditLog-application.conf

hadoop@master conf]$ cat Demo-hdfsAuditLog-application.conf 
{
  "envContextConfig" : {
    "env" : "storm",
    "mode" : "cluster",
    "topologyName" : "Demo-hdfsAuditLog-topology",
    "stormConfigFile" : "security-auditlog-storm.yaml",
    "parallelismConfig" : {
      "kafkaMsgConsumer" : 1,
      "hdfsAuditLogAlertExecutor*" : 1
    }
  },
  "dataSourceConfig": {
    "topic" : "Demo_hdfs_audit_log",
    "zkConnection" : "slave02.mesos:2181,slave03.mesos:2181,slave04.mesos:2181",
    "zkConnectionTimeoutMS" : 15000,
    "fetchSize" : 1048586,
    "deserializerClass" : "eagle.security.auditlog.HdfsAuditLogKafkaDeserializer",
    "transactionZKServers" : "slave02.mesos,slave03.mesos,slave04.mesos",
    "transactionZKPort" : 2181,
    "transactionZKRoot" : "/consumers",
    "consumerGroupId" : "eagle.hdfsaudit.consumer",
    "transactionStateUpdateMS" : 2000
  },
  "alertExecutorConfigs" : {
     "hdfsAuditLogAlertExecutor" : {
       "parallelism" : 1,
       "partitioner" : "eagle.alert.policy.DefaultPolicyPartitioner",
       "needValidation" : "true"
     }
  },
  "eagleProps" : {
    "site" : "Demo",
    "dataSource": "hdfsAuditLog",
  	"dataJoinPollIntervalSec" : 30,
    "mailHost" : "tendcloud.com",
    "mailSmtpPort":"25",
    "mailDebug" : "true",
    "eagleService": {
      "host": "master.mesos",
      "port": 9099
      "username": "admin",
      "password": "secret"
    }
  },
  "dynamicConfigSource" : {
  	"enabled" : true,
  	"initDelayMillis" : 0,
  	"delayMillis" : 30000
  }
}

注意配置这里面的eagleService的host

4.启动

创建hbase表

bin/eagle-service-init.sh

启动eagle web service

bin/eagle-service.sh start

引入metadata

bin/eagle-topology-init.sh

5.登陆eagle web

http://ip:9099/eagle-service

用户名admin，密码secret 

Storm中的topology

至于添加policy和site等过程请参见官网。

最后的web页面如下：