剑指数据仓库-MySQL5.7部署

一、MySQL常规部署

• 1.1、MySQL重新部署
• 1.2、MySQL部署遇到的问题（innodb_buffer_pool_size不够)
• 1.3、访问受限（Access denied for user ‘root’@‘localhost’ (using password: YES)）
• 1.4、访问受限（Access denied for user ‘root’@‘localhost’ (using password: NO)）
• 1.5、MySQL的几个命令

二、Dbeaver连接MySQL使用

• 2.1、/etc/hosts文件配置

三、本次课程面试题&&作业

• 3.1、本次课程补充

一、MySQL常规部署

1、一些小伙伴在部署mysql的时候会出现问题，我们要看是哪个用户在哪个目录执行的命令

mysql服务 -->mysql用户来维护
hadoop服务 --> hadoop用户维护

cdh的hdfs服务 --> 由hdfs用户来维护的
hbase服务 --> hbase用户维护

2、MySQL主要有两个版本：MySQL5.6和MySQL5.7，如下目录是J总写的关于MySQL的部署文档：

• https://github.com/Hackeruncle/mysql

注意点：check isnot install；检查是否已经安装过了，别到时候安装到一半才发现问题：

[root@hadoop001 local]# ps -ef|grep mysqld
root     21347 21311  0 12:47 pts/0    00:00:00 grep --color=auto mysqld
[root@hadoop001 local]# rpm -qa|grep -i mysql

3、解压后不要做重命名，最好是做一个软连接，方面以后软件升级也直观的查询到版本号：

1、解压之前先要创建几个必要的目录：arch归档目录、data数据目录、tmp临时目录
[root@hadoop local]# mkdir mysql/arch mysql/data mysql/tmp
[root@hadoop001 local]# tar -xzvf tar xzvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
[root@hadoop001 local]# ln -s mysql-5.7.11-linux-glibc2.5-x86_64/ mysql

//源文件和目标路径在一个文件夹，此处使用相对路径

//也可以使用绝对路径：
[root@hadoop local]# tar -xzvf /home/hadoop/software/mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz

2、解压后需要注意的一点是文件权限（用户、用户组）：
drwxr-xr-x  9  7161 wheel 4096 Feb  1  2016 mysql-5.7.11-linux-glibc2.5-x86_64

4、创建用户组，创建mysqladmin用户并且指定mysqladmin用户所在组

1、创建dba组，dba组的id是101
[root@hadoop001 local]# groupadd -g 101 dba

2、创建mysqladmin用户，-u表示指定mysqlamdin用户的id是514，-g表示指定这个用户的主组是dba组，-G表示把root组追加到dba组中去，-d表示指定mysqladmin的家目录是/usr/local/mysql
[root@hadoop001 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.

[root@hadoop001 local]# id mysqladmin
uid=514(mysqladmin) gid=101(dba) groups=101(dba),0(root)

3、groupadd的命令帮助
[root@hadoop001 local]# groupadd --help
Usage: groupadd [options] GROUP

Options:
  -f, --force                   exit successfully if the group already exists,
                                and cancel -g if the GID is already used
  -g, --gid GID                 use GID for the new group
  -h, --help                    display this help message and exit
  -K, --key KEY=VALUE           override /etc/login.defs defaults
  -o, --non-unique              allow to create groups with duplicate
                                (non-unique) GID
  -p, --password PASSWORD       use this encrypted password for the new group
  -r, --system                  create a system account
  -R, --root CHROOT_DIR         directory to chroot into
  -P, --prefix PREFIX_DIR       directory prefix

4、useradd的命令帮助：
  -g, --gid GROUP               name or ID of the primary group of the new
                                account
  -G, --groups GROUPS           list of supplementary groups of the new
                                account
  -u, --uid UID                 user ID of the new account  
  -d, --home-dir HOME_DIR       home directory of the new account

3、此时mysqladmin用户创建好了，那我们尝试使用su - mysqlamdin进行切换，会发现样式丢失，怎么解决？

1、用户切换样式丢失：
[root@hadoop local]# su - mysqladmin
-bash-4.2$ 

2、解决办法：
copy 环境变量配置文件至mysqladmin用户的home目录中,为了以下步骤配置个人环境变量
[root@hadoop39 local]# cp /etc/skel/.* /usr/local/mysql  ###important
[root@hadoop ~]# su - mysqladmin
Last login: Sun Mar 15 01:36:29 EDT 2020 on pts/0

4、需要创建my.cnf文件：在/etc/my.cnf中系统中已经存在了一个文件，保险操作：cp /etc/my.cnf /etc/my.cnf20200315；做好备份后，gg回到第一行，dG清空文件所有内容，按i键进入insert模式，拷贝如下内容到my.cnf中。

[root@hadoop39 local]# vi /etc/my.cnf
[client]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4

[mysqld]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock

skip-slave-start

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600

# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32 

#isolation level and default engine 
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id  = 1739
basedir     = /usr/local/mysql
datadir     = /usr/local/mysql/data
pid-file     = /usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format = ROW
log_bin_trust_function_creators=1
log-error  = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7

innodb_write_io_threads=16

relay-log  = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info

log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF

# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON

#other logs
#general_log =1
#general_log_file  = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
sync_binlog = 500


#for innodb options 
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M

#根据生产需要，调整pool size 
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp

innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2

innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
default-character-set=utf8mb4

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

• 生产调优改动参数：innodb_buffer_pool_size = 1024m，J总公司生产库一个8G，一个12G；正常放个6G或者8G已经很不错了。

5、修改权限：

1、修改/etc/my.cnf权限
[root@hadoop local]# chown -R mysqladmin:dba /etc/my.cnf
[root@hadoop local]# chmod -R 640 /etc/my.cnf
[root@hadoop local]# ll /etc/my.cnf
-rw-r----- 1 mysqladmin dba 2809 Mar 15 01:26 /etc/my.cnf

//640代表着root用户有读写权限，root用户组有读权限，其它用户用户组没有权限

2、因为创建了软连接，所以快捷方式和mysql文件权限都要进行修正，修正完后再次验证。
chown -R mysqladmin:dba mysql/*
chown -R mysqladmin:dba mysql-5.7.11-linux-glibc2.5-x86_64
[root@hadoop local]# chmod -R 755 mysql
[root@hadoop local]# chmod -R 755 mysql-5.7.11-linux-glibc2.5-x86_64/

6、配置环境变量：

• 目的：切换到mysqladmin用户的时候直接显示当前所在目录

1、配置.bashrc
[root@hadoop39 local]# vi mysql/.bashrc
# .bash_profile
# Get the aliases and functions

if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH


unset USERNAME

#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1

## end

2、记得echo打印路径：
hadoop:mysqladmin:/usr/local/mysql:>echo $MYSQL_BASE
/usr/local/mysql

• 好处：很直观的看出当前用户并且所在的目录是哪个。

7、配置服务及开机自启动？

1、进入到mysql安装目录
[root@hadoop local]# cd /usr/local/mysql

2、将服务文件拷贝到init.d下，并重命名为mysql
[root@hadoop mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql

3、它是一个Shell脚本，赋予其可执行权限
[root@hadoop mysql]# chmod +x /etc/rc.d/init.d/mysql

4、如果以前系统下配置过mysql服务，使用此命令删除
[root@hadoop mysql]# chkconfig --del mysql

5、把mysql添加进系统服务列表
[root@hadoop mysql]# chkconfig --add mysql

6、设置开机自启动，经测试，这一步纯属鸡肋，无用
[root@hadoop mysql]# chkconfig --level 345 mysql on

7、编辑这个文件添加一行：切换为mysqladmin用户的时候执行启动mysql服务？
[root@hadoop mysql]# vi /etc/rc.local
#!/bin/bash
touch /var/lock/subsys/local
/usr/bin/setterm -blank 0 || true

su - mysqladmin -c "/etc/init.d/mysql start --federated"

8、yum安装libaio包及初始化

• CentOS7系统进行初始化提示如下错误：bin/mysqld: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

第一步：
yum install -y libaio

第二步：
hadoop:mysqladmin:/usr/local/mysql:>bin/mysqld \
> --defaults-file=/etc/my.cnf \
> --user=mysqladmin \
> --basedir=/usr/local/mysql/ \
> --datadir=/usr/local/mysql/data/ \
> --initialize

bin/mysqld --defaults-file=/etc/my.cnf --user=mysqladmin --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data/ --initialize

第三步：MySQL5.7是有密码的，查看临时密码方式如下：
hadoop:mysqladmin:/usr/local/mysql:>cat data/hostname.err | grep password
2020-03-15T06:22:58.865006Z 1 [Note] A temporary password is generated for root@localhost: 3Rp#hF/wb7Lm

9、启动MySQL

hadoop:mysqladmin:/usr/local/mysql:>bin/mysqld_safe --defaults-file=/etc/my.cnf &
[2] 16315
hadoop:mysqladmin:/usr/local/mysql:>Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
grep: write error: Broken pipe
2020-03-15T06:37:53.336054Z mysqld_safe Logging to '/usr/local/mysql/data/hostname.err'.
2020-03-15T06:37:53.411984Z mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
2020-03-15T06:37:53.912924Z mysqld_safe mysqld from pid file /usr/local/mysql/data/hostname.pid ended

1.1、MySQL重新部署

• 部署过程中真的出现问题，检查执行的目录和用户，重新部署。

1、删除binlog文件和data文件：

• rm -rf arch/* data/*
• arch是存放binlog文件的，mysql数据库在写一条数据的时候，一份会落到data下，同时会备份一遍存放在binlog下；比如生产上mysql同步到hbase，mysql同步到es，原理就是读取的binlog文件；也可以通过某种方式将binlog文件解析成sql.

1、解读/etc/my.cnf
对于mysql的binlog文件是由哪个决定的呢？
log-bin = /usr/local/mysql/arch/mysql-bin

2、重新执行一遍初始化操作：

hadoop:mysqladmin:/usr/local/mysql:>bin/mysqld --defaults-file=/etc/file --user=mysqladmin --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data/ --initialize
mysqld: [ERROR] Could not open required defaults file: /etc/file
mysqld: [ERROR] Fatal error in defaults handling. Program aborted!
hadoop:mysqladmin:/usr/local/mysql:>bin/mysqld --defaults-file=/etc/my.cnf --user=mysqladmin --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data/ --initialize
hadoop:mysqladmin:/usr/local/mysql:>cat data/hostname.err |grep password
2020-03-15T06:46:09.593936Z 1 [Note] A temporary password is generated for root@localhost: BdqP:fdPv2p)

1.2、MySQL部署遇到的问题（innodb_buffer_pool_size不够)

hadoop:mysqladmin:/usr/local/mysql:>service mysql status
 ERROR! MySQL is not running, but lock file (/var/lock/subsys/mysql) exists
hadoop:mysqladmin:/usr/local/mysql:>service mysql start
Starting MySQL............................................................. ERROR! The server quit without updating PID file (/usr/local/mysql/data/hostname.pid).
hadoop:mysqladmin:/usr/local/mysql:>service mysql start
Starting MySQL... SUCCESS! 


2、实时打印日志：提示内存不够：
tail -f /usr/local/mysql/data/hostname.err

2020-03-15T07:43:01.225919Z 0 [Note] InnoDB: PUNCH HOLE support not available
2020-03-15T07:43:01.226028Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-03-15T07:43:01.226069Z 0 [Note] InnoDB: Uses event mutexes
2020-03-15T07:43:01.226095Z 0 [Note] InnoDB: GCC builtin __sync_synchronize() is used for memory barrier
2020-03-15T07:43:01.226115Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2020-03-15T07:43:01.226165Z 0 [Note] InnoDB: Using Linux native AIO
2020-03-15T07:43:01.227915Z 0 [Note] InnoDB: Number of pools: 1
2020-03-15T07:43:01.228658Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-03-15T07:43:01.275648Z 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
2020-03-15T07:43:01.732071Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-03-15T07:44:00.743863Z 0 [ERROR] InnoDB: Cannot allocate 209715736 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
2020-03-15 03:44:00 0x7f5d1f05e740  InnoDB: Assertion failure in thread 140037929166656 in file ut0ut.cc line 935
InnoDB: Failing assertion: !m_fatal

3、修改了一个参数：
innodb_buffer_pool_size = 1G
InnoDB需要innodb buffer pool中处理缓存

//参考博客：https://www.cnblogs.com/wanbin/p/9530833.html

1.3、访问受限（Access denied for user ‘root’@‘localhost’ (using password: YES)）

• 解决办法：

1、vi /etc/my.cnf
在任意一行中加上：skip-grant-tables     登录是跳过输入密码

2、重启mysql服务：
service mysql restart

3、mysql -uroot -p 提示输出密码的时候直接回车就进入了数据库界面：

4、update更新root用户的密码是960210，修改完后记得刷新权限：
mysql> update user set authentication_string=password('960210') where user='root';
mysql > flush privileges;

5、回到my.cnf把添加的那句话删除，再次重启mysql服务，使用我们修改过的密码尝试登陆：经测试没问题

• 赋予权限：所有地方的主机都能使用root用户+密码来访问这个数据库：

mysql> grant all privileges on *.* to 'root'@'%' identified by '960210';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'to 'root'@'%' identified by '960210'' at line 1
mysql> grant all privileges on *.* to 'root'@'%' identified by '960210';
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.
mysql> alter user root@localhost identified by '960210';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on *.* to 'root'@'%' identified by '960210';
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)

1.4、访问受限（Access denied for user ‘root’@‘localhost’ (using password: NO)）

• 访问受限：提示密码不正确：可能的原因就是密码修改后没有flush privileges进行刷新

1.5、MySQL的几个命令

1、使用数据库

• use mysql;

2、MySQL5.7中的密码字段进行了更新：authentication_string

mysql> select user,authentication_string,host from user;
+-----------+-------------------------------------------+-----------+
| user      | authentication_string                     | host      |
+-----------+-------------------------------------------+-----------+
| root      | *9750FE1AB84430D9F83FBB448F1FF04E965785CA | localhost |
| mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | localhost |
| root      | *9750FE1AB84430D9F83FBB448F1FF04E965785CA | %         |
+-----------+-------------------------------------------+-----------+
3 rows in set (0.00 sec)

• 扩充：在MySQL5.6中是空密码，在MySQL5.6中进行密码更新怎么做呢，update user set password=password(‘960210’) where user=‘root’; 删除空账号的数据：delete from user where user=‘’；
• 注意MySQL中跟用户权限挂钩的，都需要flush privileges;

二、Dbeaver连接MySQL使用

• 连接的前提：

• grant all privileges on . to ‘root’@’%’ identified by ‘960210’;

• flush privileges;

• Dbeaver官网连接下载：https://dbeaver.io/download/

1、如下图所示：

2、连接如下图所示：

• 至此，windows客户端连接dbeaver已经完成了，dbeaver这个工具基本涵盖了大数据开发过程中大部分的数据库连接。

2.1、/etc/hosts文件配置

• 扩充：/etc/hosts文件的第一行、第二行不能动，第三行是内网IP+hostname做映射，做ping的时候会根据hostname去找到ip；生产上跟ip有关的，统一的用hostname代替，比如机器迁移，IP发生变化，就只需要修改hostname。

[root@hadoop ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

144.34.179.161 hadoop
[root@hadoop ~]# ping hadoop
PING hadoop (144.34.179.161) 56(84) bytes of data.
64 bytes from hadoop (144.34.179.161): icmp_seq=1 ttl=64 time=0.097 ms
64 bytes from hadoop (144.34.179.161): icmp_seq=2 ttl=64 time=0.069 ms
64 bytes from hadoop (144.34.179.161): icmp_seq=3 ttl=64 time=0.079 ms
^C
--- hadoop ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2056ms
rtt min/avg/max/mdev = 0.069/0.081/0.097/0.015 ms

比如windows访问，IDEA开发，都是外网IP（虚拟的，可能会发生变化），代码中统一的配置hostname。

‪C:\Windows\System32\drivers\etc\hosts

内网服务器：直连
跳板机、堡垒机：都是做不了ip+hostname映射，这是专门有一台机器暴露，进行转发的。

三、本次课程面试题&&作业

面试题：

• 1、mysql/arch目录下的binlog文件会不会被写满呢？

hadoop:mysqladmin:/usr/local/mysql/arch:>ll -h
total 4.1G
-rw-r----- 1 mysqladmin dba 1.0G Mar 15 04:57 ib_logfile0
-rw-r----- 1 mysqladmin dba 1.0G Mar 15 03:35 ib_logfile1
-rw-r----- 1 mysqladmin dba 1.0G Mar 15 03:36 ib_logfile2
-rw-r----- 1 mysqladmin dba 1.0G Mar 15 03:36 ib_logfile3
-rw-r----- 1 mysqladmin dba  98K Mar 15 03:36 mysql-bin.000001
-rw-r----- 1 mysqladmin dba  177 Mar 15 03:56 mysql-bin.000002
-rw-r----- 1 mysqladmin dba  201 Mar 15 04:00 mysql-bin.000003
-rw-r----- 1 mysqladmin dba  177 Mar 15 04:00 mysql-bin.000004
-rw-r----- 1 mysqladmin dba  177 Mar 15 04:36 mysql-bin.000005
-rw-r----- 1 mysqladmin dba  177 Mar 15 04:36 mysql-bin.000006
-rw-r----- 1 mysqladmin dba  177 Mar 15 04:42 mysql-bin.000007
-rw-r----- 1 mysqladmin dba 1.1K Mar 15 04:57 mysql-bin.000008
-rw-r----- 1 mysqladmin dba  830 Mar 15 05:09 mysql-bin.000009
-rw-r----- 1 mysqladmin dba  351 Mar 15 04:57 mysql-bin.index

2、作业：

• 整理nohup执行脚本
• rundeckB站上的视频写博客
• crontab每隔10秒运行的写法
• 软连接的使用场景
• 二进制部署mysql
• mysql的重新部署
• dbeaver连接mysql

3.1、补充

1、若泽数据github站网址

• https://ruozedata.github.io/archives/

2、生产mysql数据恢复

• https://ruozedata.github.io/2019/05/30/%E4%B8%8A%E6%B5%B7%E6%9F%90%E5%85%AC%E5%8F%B8%E7%9A%84%E7%94%9F%E4%BA%A7MySQL%E7%81%BE%E9%9A%BE%E6%80%A7%E6%8C%BD%E6%95%91/