mysql主主复制+nginx负载均衡+zookeeper+kafka+elasticsearch配置

一、服务器配置

192.168.2.247	192.168.2.248	192.168.2.249
jdk	jdk	jdk
tomcat	tomcat
nginx
mysql	mysql
		kafka
		elasticsearch+head
activemq	activemq
redis	redis

二、安装jdk

1、解压 jdk压缩包

[root@247 local]# tar -zxf jdk-8u11-linux-x64.tar.gz

2、添加环境变量

[root@247 local]# vim /etc/profile

3、在最后添加

JAVA_HOME=/usr/local/jdk1.8.0_11/
export JRE_HOME=/usr/local/jdk1.8.0_11/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib

4、编译、查看

[root@247 local]# source /etc/profile
[root@247 local]# java -version
java version "1.8.0_11"
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)

5、其余同理，不再赘述

---

二、安装activemq

1、解压activemq

[root@247 local]# tar -zxf apache-activemq-5.15.9-bin.tar.gz

2、启动activemq

[root@247 apache-activemq-5.15.9]# cd bin
[root@247 bin]# ./activemq start
INFO: Loading '/usr/local/apache-activemq-5.15.9//bin/env'
INFO: Using java '/usr/local/jdk1.8.0_11/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
INFO: pidfile created : '/usr/local/apache-activemq-5.15.9//data/activemq.pid' (pid '1495')

---

三、安装redis

1、解压

[root@247 ~]# tar -zxf redis-5.0.5.tar.gz

2、编译、安装

2.1、编译

[root@247 redis-5.0.5]# make

发现报错：报错，缺少依赖，需要安装gcc

make[3]: Entering directory `/usr/local/redis-5.0.5/deps/hiredis'
gcc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  net.c
make[3]: gcc: Command not found
make[3]: *** [net.o] Error 127
make[3]: Leaving directory `/usr/local/redis-5.0.5/deps/hiredis'
make[2]: *** [hiredis] Error 2
make[2]: Leaving directory `/usr/local/redis-5.0.5/deps'
make[1]: [persist-settings] Error 2 (ignored)
    CC adlist.o
/bin/sh: cc: command not found
make[1]: *** [adlist.o] Error 127
make[1]: Leaving directory `/usr/local/redis-5.0.5/src'
make: *** [all] Error 2

安装gcc依赖

[root@247 redis-5.0.5]# yum install gcc

重新编译，成功

    CC listpack.o
    CC localtime.o
    CC lolwut.o
    CC lolwut5.o
    LINK redis-server
    INSTALL redis-sentinel
    CC redis-cli.o
    LINK redis-cli
    CC redis-benchmark.o
    LINK redis-benchmark
    INSTALL redis-check-rdb
    INSTALL redis-check-aof

Hint: It's a good idea to run 'make test' ;)

make[1]: Leaving directory `/usr/local/redis-5.0.5/src'

2.2、安装

[root@247 redis-5.0.5]# make install PREFIX=/usr/local/redis
cd src && make install
make[1]: Entering directory `/usr/local/redis-5.0.5/src'
    CC Makefile.dep
make[1]: Leaving directory `/usr/local/redis-5.0.5/src'
make[1]: Entering directory `/usr/local/redis-5.0.5/src'

Hint: It's a good idea to run 'make test' ;)

    INSTALL install
    INSTALL install
    INSTALL install
    INSTALL install
    INSTALL install
make[1]: Leaving directory `/usr/local/redis-5.0.5/src'

3、移动配置文件到安装目录下

[root@247 redis-5.0.5]# mkdir /usr/local/redis/etc
[root@247 redis-5.0.5]# mv redis.conf /usr/local/redis/etc

4、修改redis.conf配置文件

# 507行 密码
# requirepass foobared 取消注释，输入密码
requirepass 123   指定密码123

5、配置redis为后台启动

vim /usr/local/redis/etc/redis.conf 
# 将daemonize no 改成daemonize yes

6、将redis加入到开机启动

[root@247 redis-5.0.5]# vim /etc/rc.local
#在最后一行添加
/usr/local/redis/bin/redis-server 

7、添加环境变量

[root@247 redis-5.0.5]# vim /etc/profile
JAVA_HOME=/usr/local/jdk1.8.0_11/
REDIS_HOME=/usr/local/redis/           # 添加REDIS_HOME
export JRE_HOME=/usr/local/jdk1.8.0_11/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$REDIS_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
[root@247 redis-5.0.5]# source /etc/profile

8、开机启动

[root@247 redis-5.0.5]# redis-server /usr/local/redis/etc/redis.conf
16161:C 06 Aug 2020 03:45:13.727 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16161:C 06 Aug 2020 03:45:13.727 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=16161, just started
16161:C 06 Aug 2020 03:45:13.727 # Configuration loaded

---

四、安装tomcat

1、解压

[root@247 local]# tar -zxf apache-tomcat-8.5.45.tar.gz

五、安装mysql

1、安装mysql

1.1、安装mysql源镜像

[root@247 ~]# yum localinstall mysql57-community-release-el7-8.noarch.rpm

1.2、检查mysql源是否安装成功

[root@247 ~]# yum repolist enabled | grep "mysql.*-community.*"
mysql-connectors-community/x86_64       MySQL Connectors Community           165
mysql-tools-community/x86_64            MySQL Tools Community                115
mysql57-community/x86_64                MySQL 5.7 Community Server           444

1.3、修改vim /etc/yum.repos.d/mysql-community.repo源

此处默认使用mysql57，所以这里不做改动

1.4、安装Mysql

[root@247 ~]# yum install mysql-community-server

1.5、启动MySQL服务

[root@247 ~]# systemctl start mysqld
[root@247 ~]# systemctl status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-08-06 04:20:55 EDT; 8s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 16414 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 16364 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 16416 (mysqld)
   CGroup: /system.slice/mysqld.service
           └─16416 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Aug 06 04:20:51 247 systemd[1]: Starting MySQL Server...
Aug 06 04:20:55 247 systemd[1]: Started MySQL Server.

1.6、设置开机启动

[root@247 ~]# systemctl enable mysqld
[root@247 ~]# systemctl daemon-reload

1.7、修改密码策略

此处关闭密码的校验，否则需要输入复杂的密码

[root@247 ~]# vim /etc/my.cnf
# 底部添加配置
validate_password = off

1.8、重新启动mysql服务

[root@247 ~]# systemctl restart mysqld

1.9、查看默认密码

[root@247 ~]# grep 'temporary password' /var/log/mysqld.log
2020-08-06T08:20:53.105154Z 1 [Note] A temporary password is generated for root@localhost: klh(HlZ7PHh_

1.10、使用默认密码登录

[root@247 ~]# mysql -uroot -p
Enter password:

1.11、修改密码

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)

1.12、允许外部用户访问

mysql> grant all privileges on *.* to root@'%' identified by 'root';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

1.13、执行sql文件报错

ERROR 1418 (HY000): This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)

此处报错导致存储函数没有执行，set global log_bin_trust_function_creators=TRUE;

1.14、另一台同理配置

2、配置mysql主主复制

2.1、修改配置文件

[root@247 ~]# vim /etc/my.cnf
# 在底部添加一下参数
server-id=1
innodb_flush_log_at_trx_commit=2
sync_binlog=1
log-bin=mysql-bin-1

配置说明：
设置主服务 的ID (id可以自己随便设置但是要保证和slave的id不一样)
server-id=1
设为1当然是最安全的，但性能也是最差的（相对其他两个参数而言，但不是不能接受）。如果对数据一致性和完整性要求不高，完全可以设为2，如果只最求性能，例如高并发写的日志服务器，设为0来获得更高性能
innodb_flush_log_at_trx_commit=2
开启binlog 志同步功能

sync_binlog=1
binlog 日志文件名
log-bin=mysql-bin-200
这个表示只同步某个库 (如果没有此项，表示同步所有的库)
binlog-do-db=xxxx

2.2、重启mysql

[root@247 ~]# systemctl restart mysqld

2.3、进入mysq输入授权命令

GRANT REPLICATION SLAVE ON *.* to 'root'@'%' identified by '123456';

2.4、查看master节点的状态

mysql> show master status;
+--------------------+----------+--------------+------------------+-------------------+
| File               | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+--------------------+----------+--------------+------------------+-------------------+
| mysql-bin-1.000001 |      437 |              |                  |                   |
+--------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

2.5、修改从节点配置

[root@248 ~]# vim /etc/my.cnf
server-id=201  #和master不一样
innodb_flush_log_at_trx_commit=2 
sync_binlog=1 
log-bin=mysql-bin-201
[root@248 ~]# systemctl restart mysqld

2.6、指定master节点

mysql> change master to master_host='192.168.2.247',master_user='root' ,master_password='123456', master_log_file='mysql-bin-1.000001' ,master_log_pos=437;
Query OK, 0 rows affected, 2 warnings (0.02 sec)

命令说明：
master_host: 主机的ip
master_user : 主机授权的用户.
master_password : 主机授权时候填写的密码
master_log_file : 主机show master status;中的File
master_log_pos: 主机show master status;中的Position.

2.7、启动slave

mysql> start slave;
Query OK, 0 rows affected (0.01 sec)
mysql> show slave status \G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.2.247
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin-1.000001
          Read_Master_Log_Pos: 437
               Relay_Log_File: 248-relay-bin.000002
                Relay_Log_Pos: 322
        Relay_Master_Log_File: mysql-bin-1.000001
             Slave_IO_Running: Yes  #IO线程启动
            Slave_SQL_Running: Yes  #SQL线程启动
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 437
              Relay_Log_Space: 527
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: bdbe37d5-d7bd-11ea-b10d-000c29b348db
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.01 sec)

ERROR:
No query specified

至此，已完成了247和248的主从复制，重复操作，完成248和247的主从复制即可实现主主复制了。

---

六、配置Nginx负载均衡

1、安装nginx

1.1、解压nginx

[root@247 local]# tar -zxf nginx-1.18.0.tar.gz

1.2、安装依赖包

[root@247 nginx-1.18.0]# yum -y install gcc gcc-c++ make libtool zlib zlib-devel openssl openssl-devel pcre pcre-devel

1.3、配置nginx

[root@247 nginx-1.18.0]# ./configure --prefix=/usr/local/nginx

1.4、编译、安装

make && make install

1.5、启动

进入nginx/sbin下，执行nginx
[root@247 nginx]# ./nginx

2、配置负载均衡

2.1、配置nginx.conf

nginx_conf.png

2.2、启动tomcat

247-tomcat.jpg

248-tomcat.jpg

2.3、查看负载均衡是否成功

247-nginx.jpg

---

七、配置Kafka、Elasticsearch、Connect连接器

具体参考这位大神的：
使用Kafka的Connect监听Mysql数据并同步到ElasticSearch-刘宇

1、安装zookeeper

1.1、解压zookeeper

[root@249 ~]# tar -xzvf zookeeper-3.4.14.tar.gz -C /usr/local

1.2、创建快照和日志存储路径

[root@249 zookeeper-3.4.14]# mkdir dataDir
[root@249 zookeeper-3.4.14]# mkdir dataDirLog

1.3、修改配置文件

[root@249 zookeeper-3.4.14]# cd conf/
[root@249 conf]# cp zoo_sample.cfg zoo.cfg
[root@249 conf]# vim zoo.cfg

zoo.cfg.png

1.4、添加zookeeper唯一标识

[root@249 conf]# cd ../dataDir
[root@249 dataDir]# echo "1" > myid

1.5、启动zookeeper

[root@249 dataDir]# cd /usr/local/zookeeper-3.4.14/bin
[root@249 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

2、安装kafka

2.1、解压

[root@249 local]# tar -zxf /root/kafka_2.11-2.2.1.tgz -C ./
[root@249 local]# mv kafka_2.11-2.2.1/ kafka

2.2、修改配置文件

[root@249 local]# cd kafka/
[root@249 kafka]# cd config/
[root@249 config]# vim server.properties
# The id of the broker. This must be set to a unique integer for each broker.
# 此处唯一
broker.id=0
# The address the socket server listens on. It will get the value returned from
listeners=PLAINTEXT://192.168.2.249:9092
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.2.249:2181

2.3、后台启动kafka

[root@249 kafka]# nohup bin/kafka-server-start.sh config/server.properties &
[root@249 kafka]# jps
14386 Kafka
14722 Jps
14359 QuorumPeerMain

3、安装Elasticsearch

3.1、解压

[root@249 ~]# tar -zxf elasticsearch-5.6.8.tar.gz -C /usr/local

3.2、修改配置文件

[root@249 ~]# vim /usr/local/elasticsearch-5.6.8/config/elasticsearch.yml

elasticsearch.yml.png

3.3、新建路径

[root@249 elasticsearch-5.6.8]# mkdir data
[root@249 elasticsearch-5.6.8]# mkdir logs

3.4、创建新用户

[root@249 elasticsearch-5.6.8]# useradd es

3.5、设置目录归属

[root@249 elasticsearch-5.6.8]# chown -R es:es /usr/local/elasticsearch-5.6.8
[root@249 elasticsearch-5.6.8]# ll
total 224
drwxr-xr-x  2 es es   4096 Aug 11 02:27 bin
drwxr-xr-x  2 es es     75 Aug 11 02:43 config
drwxr-xr-x  2 es es      6 Aug 11 02:32 data
drwxr-xr-x  2 es es   4096 Feb 16  2018 lib
-rw-r--r--  1 es es  11358 Feb 16  2018 LICENSE.txt
drwxr-xr-x  2 es es      6 Aug 11 02:33 logs
drwxr-xr-x 13 es es    236 Feb 16  2018 modules
-rw-r--r--  1 es es 194187 Feb 16  2018 NOTICE.txt
drwxr-xr-x  2 es es      6 Feb 16  2018 plugins
-rw-r--r--  1 es es   9490 Feb 16  2018 README.textile

3.6、启动elasticsearch

[root@249 elasticsearch-5.6.8]# su es
[es@249 elasticsearch-5.6.8]$ cd bin
[es@249 bin]$ nohup ./elasticsearch &
[1] 14921
[es@249 bin]$ nohup: ignoring input and appending output to ‘nohup.out’

[es@249 bin]$ jps
14960 Jps
14921 Elasticsearch

3.7、报错

1、

ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

root用户编辑/etc/security/limits.conf，尾部添加这两行，并重启服务器

image.png

2、

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

root用户编辑/etc/sysctl.conf
添加vm.max_map_count=262144

[root@249 bin]# vim /etc/sysctl.conf
[root@249 bin]# sysctl -p
vm.max_map_count = 262144

3.8、测试

[es@249 bin]$ curl  http://192.168.2.249:9200
{
  "name" : "node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "dN7-77B7Qn2yrxXgfKk_cw",
  "version" : {
    "number" : "5.6.8",
    "build_hash" : "688ecce",
    "build_date" : "2018-02-16T16:46:30.010Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

八、配置kafka中的Connect，实现将MySQL数据同步到Elasticsearch中

8.1、在kafka的config文件中配置mysql到kafka的连接器

mysql-kafka-connect.png

8.2、在kafka的config文件中配置kafka到elasticsearch的连接器

image.png

8.3、运行connect

[root@249 bin]# nohup ./connect-standalone.sh ../config/connect-standalone.properties ../config/es-sensorData.properties ../config/sensorData.properties &