otter 配置安装

otter 配置安装

架构：

10.10.5.109:    zookeeper server,manager,node server, destination mysql server ,canal server
10.10.5.111 source mysql server


测试从source mysql 【源库】中，把数据同步到 【目标库】中。

整个otter同步由几部分组成，需要预先进行安装，后续会有专门的篇幅展开介绍

manager
node

otter node依赖于zookeeper进行分布式调度，需要安装一个zookeeper节点或者集群.

1.install zookeeper

wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.12/zookeeper-3.4.12.tar.gz

建立日志目录及数据目录

#mkdir /var/log/zookeeper
#mkdir /opt/zookeeper-3.4.12/data


#cp conf/zoo_sample.cfg zoo.cfg
#vi zoo.cfg
***********************************************************
    # The number of milliseconds of each tick
    tickTime=2000
    # The number of ticks that the initial
    # synchronization phase can take
    initLimit=10
    # The number of ticks that can pass between
    # sending a request and getting an acknowledgement
    syncLimit=5
    # the directory where the snapshot is stored.
    # do not use /tmp for storage, /tmp here is just
    # example sakes.
    dataDir=/opt/zookeeper-3.4.12/data
    dataLogDir=/var/log/zookeeper

    # the port at which the clients will connect
    clientPort=2181

先用一个zoo节点测试。所以只需配置一个节点。

# vi log4j.properties 

    # Define Some Default Values That Can Be Overridden By System Properties
    Zookeeper.Root.Logger=Info, Console
    Zookeeper.Console.Threshold=Info
    Zookeeper.Log.Dir=/Var/Log/Zookeeper


启动zookeeper

# ./zkServer.sh start ../conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: ../conf/zoo.cfg
Starting zookeeper ... STARTED

2.manager
otter 下载URL
https://github.com/alibaba/otter/releases

wget https://github.com/alibaba/otter/releases/download/v4.2.15/manager.deployer-4.2.15.tar.gz
wget https://github.com/alibaba/otter/releases/download/v4.2.15/node.deployer-4.2.15.tar.gz

2.1 安装manager

mkdir /opt/otter/manager/
tar zxvf manager.deployer-4.2.15.tar.gz -C /opt/otter/manager

manager的数据也存储在Mysql中，Mysql 的安装这里不再累述。添加库，用户。

mysql> create database otter DEFAULT CHARACTER SET utf8;
mysql>CREATE USER 'canal'@'%' IDENTIFIED BY 'canal';
mysql>GRANT ALL PRIVILEGES ON `otter`.* TO 'canal'@'%'; 
mysql>flush PRIVILEGES;

初始化otter manager系统表：
wget https://raw.github.com/alibaba/otter/master/manager/deployer/src/main/resources/sql/otter-manager-schema.sql 


mysql> source /opt/otter/manager/otter-manager-schema.sql ;


#修改配置文件
vim ./conf/otter.properties

    ## otter manager domain name
    otter.domainName = 127.0.0.1
    ## otter manager http port
    otter.port = 8080
    ## jetty web config xml
    otter.jetty = jetty.xml

    ## otter manager database config
    otter.database.driver.class.name = com.mysql.jdbc.Driver
    otter.database.driver.url = jdbc:mysql://127.0.0.1:3306/otter
    otter.database.driver.username = canal
    otter.database.driver.password = canal

    ## otter communication port
    otter.communication.manager.port = 1099
    ## otter communication pool size
    otter.communication.pool.size = 10

    ## default zookeeper address
    otter.zookeeper.cluster.default = 127.0.0.1:2181
    ## default zookeeper sesstion timeout = 60s
    otter.zookeeper.sessionTimeout = 60000

#启动
sh ./bin/startup.sh

#查看日志
[root@test05-1 manager]# tail -300f logs/manager.log
2018-08-15 09:54:14.941 [] INFO  com.alibaba.otter.manager.deployer.OtterManagerLauncher - ## start the manager server.
2018-08-15 09:54:24.713 [] INFO  com.alibaba.otter.manager.deployer.JettyEmbedServer - ##Jetty Embed Server is startup!
2018-08-15 09:54:24.713 [] INFO  com.alibaba.otter.manager.deployer.OtterManagerLauncher - ## the manager server is running now ......
^C

2.2 安装node
需要aria 支持，先安装

yum install aria2

node 的配置文件很简单：

vi otter.properties:
otter.manager.address = 127.0.0.1:1099



./bin/startup.sh
启动node 总是报 没有 nid 这个文件，手工加一个：
echo 1 > ../conf/nid

3.canal
wget https://github.com/alibaba/canal/releases/download/v1.0.25/canal.deployer-1.0.25.tar.gz

#修改配置文件
vi conf/canal.properties 
    canal.ip=10.10.5.109
    canal.zkServers=10.10.5.109:2181
    canal.instance.binlog.format = ROW
    canal.instance.global.manager.address = 10.10.5.109:1099


#添加连接主数据库信息，
vi conf/example/instance.properties 

    ## mysql serverId
    canal.instance.mysql.slaveId=1
    # position info
    canal.instance.master.address=10.10.5.111:33306
    canal.instance.master.journal.name=mysql-bin.000012
    canal.instance.master.position=164764

#启动
# bin/startup.sh 
#可以查看日志看到有sql 等信息
#tail -30f logs/canal/canal.log


开始放在 【 10.10.5.111】服务器上运行，运行不起来，可能是内存不够(8G).
后在放到服务器【10.10.5.109】（ 16G）运行起来了。

    [root@spring01 canal]# tail -300f logs/canal/canal.log 
    OpenJDK 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0
    OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
    OpenJDK 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
    OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000700000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)
    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.
    # An error report file with more information is saved as:
    # /opt/canal/bin/hs_err_pid4353.log
    OpenJDK 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0
    OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
    OpenJDK 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
    OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000700000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)
    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.
    # An error report file with more information is saved as:
    # /opt/canal/bin/hs_err_pid4800.log

4.在源数据库中添加数据同步用户

mysql> CREATE USER 'repl'@'%' IDENTIFIED BY '8ni6Ph6yWD';
Query OK, 0 rows affected (0.00 sec)


GRANT ALL PRIVILEGES ON `hyjf_user`.* TO 'repl'@'10.10.5.%';
GRANT ALL PRIVILEGES ON `hyjf_trade`.* TO 'repl'@'10.10.5.%';
GRANT ALL PRIVILEGES ON `hyjf_config`.* TO 'repl'@'10.10.5.%';
GRANT ALL PRIVILEGES ON `hyjf_market`.* TO 'repl'@'10.10.5.%';


mysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> flush PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

5.配置：

一般的配置都比较简单，下面只说明一下我出错的配置：
1.canal 配置，开始理解成是连接到manager 数据库去了。后来修改为连接需要同步的源数据库。
(一个Pipeline 指定一个canal，各个canal 中的数据库地址选择你要同步的数据库源的服务器地址)

2.在配置pipeline时，会有选择canal 的选择，选择上面的canal 名称即可。

所有相关配置请看github:　　　https://github.com/alibaba/otter/wiki/Manager%E4%BD%BF%E7%94%A8%E4%BB%8B%E7%BB%8D

管理账号修改为：
admin/admin

otter manager:
http://10.10.5.109:8080

github

https://github.com/alibaba/otter

下载地址：

https://github.com/alibaba/canal/releases

https://blog.csdn.net/hackerwin7/article/details/38044327