mysql ndbcluster集群安装以及遇到问题

mysql ndbcluster 安装

1.数据库规划：
操作系统 centos 6.3
ndbcluster 版本：MySQL-Cluster-gpl-7.4.8-1.el6.x86_64
2个管理节点、2个sql节点、2个数据节点（管理节点和sql节点放在同一台服务器上）
如下：
管理节点1 192.168.1.17   
管理节点2 192.168.1.18

sql节点1 192.168.1.17
sql节点2 192.168.1.18
ndbd 节点1 192.168.1.19  
ndbd 节点2 192.168.1.20  

2.操作系统设置：
设置服务主机名称
关闭selinux和防火墙
设置内核参数和最大进程数

下载mysql集群
MySQL-Cluster-gpl-7.4.8-1.el6.x86_64.rpm-bundle.tar
上传到服务器的相应目录下如 /home/mysqlinstall

3.安装管理节点和sql节点：
To check if your system has any RPM version of any MySQL package currently installed, run:
shell> rpm -qa | grep -i mysql
检查
[root@redis1 ~]#  rpm -qa | grep -i mysql
mysql-libs-5.1.61-4.el6.x86_64
如果有删除
shell> rpm -e mysql-libs-5.1.61-4.el6.x86_64
删除报错：
[root@redis1 ~]# rpm -e mysql-libs-5.1.61-4.el6.x86_64
error: Failed dependencies:
        libmysqlclient.so.16()(64bit) is needed by (installed) postfix-2:2.6.6-2.2.el6_1.x86_64
        libmysqlclient.so.16(libmysqlclient_16)(64bit) is needed by (installed) postfix-2:2.6.6-2.2.el6_1.x86_64
        mysql-libs is needed by (installed) postfix-2:2.6.6-2.2.el6_1.x86_64
[root@redis1 ~]# 
解决：
[root@redis1 ~]# rpm -e --nodeps mysql-libs-5.1.61-4.el6.x86_64
[root@redis1 ~]# 

[root@ndbcluster1 mysqlinstall]# rpm -ivh MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64.rpm 
[root@ndbcluster1 mysqlinstall]# rpm -ivh MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64.rpm 

4.安装数据节点：
执行上面步骤
[root@ndb mysqlinstall]# rpm -ivh MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64.rpm 
[root@ndb mysqlinstall]# rpm -ivh MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64.rpm 

5.配置集群
管理节点建立目录
mkdir /var/lib/mysql-cluster
mkdir -p /data/mysql/ndbdata

管理节点配置文件config.ini：
# TCP PARAMETERS
[tcp default]
SendBufferMemory=20M
ReceiveBufferMemory=20M
# Increasing the sizes of these 2 buffers beyond the default values
# helps prevent bottlenecks due to slow disk I/O.
# MANAGEMENT NODE PARAMETERS
[ndb_mgmd default]
DataDir=/data/mysql-cluster
# It is possible to use a different data directory for each management
# server, but for ease of administration it is preferable to be
# consistent.
[ndb_mgmd]
NodeId=1
HostName=192.168.1.17
# NodeId=management-server-A-nodeid
[ndb_mgmd]
NodeId=2
HostName=192.168.1.18
[ndbd default]
NoOfReplicas=2
# Using 2 replicas is recommended to guarantee availability of data;
# using only 1 replica does not provide any redundancy, which means
# that the failure of a single data node causes the entire cluster to
# shut down. We do not recommend using more than 2 replicas, since 2 is
# sufficient to provide high availability, and we do not currently test
# with greater values for this parameter.
LockPagesInMainMemory=1
# On Linux and Solaris systems, setting this parameter locks data node
# processes into memory. Doing so prevents them from swapping to disk,
# which can severely degrade cluster performance.
DataMemory=2000M
IndexMemory=800M
# The values provided for DataMemory and IndexMemory assume 4 GB RAM
# per data node. However, for best results, you should first calculate
# the memory that would be used based on the data you actually plan to
# store (you may find the ndb_size.pl utility helpful in estimating
# this), then allow an extra 20% over the calculated values. Naturally,
# you should ensure that each data node host has at least as much
# physical memory as the sum of these two values.
# ODirect=1
# Enabling this parameter causes NDBCLUSTER to try using O_DIRECT
# writes for local checkpoints and redo logs; this can reduce load on
# CPUs. We recommend doing so when using MySQL Cluster on systems running
# Linux kernel 2.6 or later.
NoOfFragmentLogFiles=600
MaxNoOfConcurrentOperations=100000
MaxNoOfLocalOperations=110000
SchedulerSpinTimer=400
SchedulerExecutionTimer=100
RealTimeScheduler=1
# Setting these parameters allows you to take advantage of real-time scheduling
# of NDB threads to achieve increased throughput when using ndbd. They
# are not needed when using ndbmtd; in particular, you should not set
# RealTimeScheduler for ndbmtd data nodes.
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=200
#DiskCheckpointSpeed=20M
#DiskCheckpointSpeedInRestart=100M
MaxNoOfConcurrentScans=500
RedoBuffer=32M
# CompressedLCP=1
# CompressedBackup=1
# Enabling CompressedLCP and CompressedBackup causes, respectively, local
#checkpoint files and backup files to be compressed, which can result in a space
#savings of up to 50% over noncompressed LCPs and backups.
MaxNoOfLocalScans=1000
MaxNoOfTables=4096
MaxNoOfOrderedIndexes=2048
MaxNoOfExecutionThreads=4
TransactionDeadLockDetectionTimeOut=30000
HeartbeatIntervalDbDb=5000
HeartbeatIntervalDbApi=5000
MaxNoOfConcurrentTransactions=4096
MaxNoOfConcurrentIndexOperations=30000
MaxNoOfFiredTriggers=10000
MaxNoOfAttributes=600000
MaxNoOfUniqueHashIndexes=512
[ndbd]
Nodeid=3
HostName=192.168.1.19
DataDir=/data/mysql/ndbdata
# NodeId=data-node-A-nodeid
#LockExecuteThreadToCPU=1
#LockMaintThreadsToCPU=0
# On systems with multiple CPUs, these parameters can be used to lock NDBCLUSTER
# threads to specific CPUs
[ndbd]
Nodeid=4
HostName=192.168.1.20
DataDir=/data/mysql/ndbdata
# NodeId=data-node-B-nodeid
#LockExecuteThreadToCPU=1
#LockMaintThreadsToCPU=0
# You must have an [ndbd] section for every data node in the cluster;
# each of these sections must include a HostName. Each section may
# optionally include a NodeId for convenience, but in most cases, it is
# sufficient to allow the cluster to allocate node IDs dynamically. If
# you do specify the node ID for a data node, it must be in the range 1
# to 48 inclusive and must be unique among all IDs specified for
# cluster nodes.
# SQL NODE / API NODE PARAMETERS
[mysqld]
Nodeid=5
HostName=192.168.1.17
[mysqld]
Nodeid=6
HostName=192.168.1.18
[mysqld]
Nodeid=7
[mysqld]
Nodeid=8

sql节点和数据节点配置文件/etc/my.cnf
[mysqld]
# Options for mysqld process:
ndbcluster                      #run NDB storage engine
#skip-grant-tables
character_set_server=utf8
lower_case_table_names=1
max_connections=1000
ndb-connectstring=192.168.1.17,192.168.1.18  # location of management server
default-storage-engine=ndbcluster
datadir=/data/mysql
# provide connection string for management server host (default port: 1186)
[ndbd]
connect-string=192.168.1.17,192.168.1.18  # location of management server
#provide connection string for management server host (default port: 1186)
[ndb_mgm]
connect-string=192.168.1.17,192.168.1.18  # location of management server
#provide location of cluster configuration file
[ndb_mgmd]
config-file=/var/lib/mysql-cluster/config.ini

5.启动关闭集群：
启动管理节点
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --initial (第一次启动加--initial修改了配置文件后的启动)

启动数据节点
ndbmtd --initial

启动sql节点
service mysql start

查看集群状态
shell> ndb_mgm -e "SHOW"

关闭管理节点和数据节点
ndb_mgm -e shutdown
关闭sql节点
service mysql stop

安装遇到的问题：
1.权限问题启动mysql报错
[root@sqlmgm2 mysql]# service mysql start
Starting MySQL..The server quit without updating PID file ([FAILED]sql/sqlmgm2.pid).
[root@sqlmgm2 mysql]# 

[root@sqlmgm2 mysql]# ll
total 110644
-rw-rw---- 1 mysql mysql       56 Dec 14 17:13 auto.cnf
-rw-rw---- 1 mysql mysql 12582912 Dec 14 17:39 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Dec 14 17:40 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Dec 14 17:13 ib_logfile1
drwx------ 2 mysql mysql     4096 Dec 14 17:39 mysql
drwx------ 2 root  root      4096 Dec 14 17:39 ndbinfo
drwx------ 2 root  root      4096 Dec 14 17:39 performance_schema
-rw-r----- 1 mysql mysql    30068 Dec 14 17:40 sqlmgm2.err
drwx------ 2 root  root      4096 Dec 14 17:33 test
[root@sqlmgm2 mysql]# chown mysql:mysql -R mysql
[root@sqlmgm2 mysql]# service mysql start
Starting MySQL..                                           [  OK  ]
[root@sqlmgm2 mysql]# 

2.安装rpm包时，报包依赖关系如下：
[root@ndbcluster2 mysqlinstall]# rpm -ivh MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64.rpm 
warning: MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
error: Failed dependencies:
        libnuma.so.1()(64bit) is needed by MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64
        libnuma.so.1(libnuma_1.1)(64bit) is needed by MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64
        libnuma.so.1(libnuma_1.2)(64bit) is needed by MySQL-Cluster-server-gpl-7.4.8-1.el6.x86_64       
解决方法是把安装的依赖的包
[root@ndbcluster2 mysqlinstall]# yum install numactl
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
 * base: mirrors.163.com
 * extras: mirrors.opencas.cn
 * updates: mirrors.opencas.cn
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package numactl.x86_64 0:2.0.9-2.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================
 Package                      Arch                        Version                             Repository                 Size
==============================================================================================================================
Installing:
 numactl                      x86_64                      2.0.9-2.el6                         base                       74 k
Transaction Summary
==============================================================================================================================
Install       1 Package(s)
Total download size: 74 k
Installed size: 171 k
Is this ok [y/N]: y
Downloading Packages:
numactl-2.0.9-2.el6.x86_64.rpm                                                                         |  74 kB     00:00     
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Importing GPG key 0xC105B9DE:
 Userid : CentOS-6 Key (CentOS 6 Official Signing Key)
 Package: centos-release-6-3.el6.centos.9.x86_64 (@anaconda-CentOS-201207061011.x86_64/6.3)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : numactl-2.0.9-2.el6.x86_64                                                                                 1/1 
  Verifying  : numactl-2.0.9-2.el6.x86_64                                                                                 1/1 
Installed:
  numactl.x86_64 0:2.0.9-2.el6                                                                                                
Complete!

3.启动数据库节点无法和管理节点相连
原因由于防火墙和selinux没有关闭
解决关闭防火墙和selinux
一、关闭防火墙
1、重启后永久性生效：
开启：chkconfig iptables on
关闭：chkconfig iptables off
2、即时生效，重启后失效：
开启：service iptables start
关闭：service iptables stop
二、关闭SELinux
vim /etc/selinux/config  # 改为 SELINUX=disabled
# 保存退出，重启服务器
init 6
禁用SeLinux
#永久禁用，需要重启生效。
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config  
# 临时禁用，不需要重启
setenforce 0  

4.root用户不能登录
[root@ndbcluster2 ~]# mysql -uroot -p
Enter password: 
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
[root@ndbcluster2 ~]# mysql -uroot -p
Enter password: 
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

设置--skip-grant-tables跳过授权表认证
service mysqld stop
在配置文件中添加--skip-grant-tables（/etc/my.cnf）
service mysqld start
另外开个SSH连接
[root@localhost ~]# mysql
mysql>use mysql
mysql>update user set password=password("123456") where user="root";
mysql>flush privileges;
mysql>exit
去掉--skip-grant-tables重启mysql,root用户可以用设置的密码登录

设置root用户远程登录
GRANT ALL PRIVILEGES ON *.* TO root@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; 
执行报如下错误，ERROR 1820 (HY000): You must SET PASSWORD before executing this statement
[root@ndbcluster1 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.27-ndb-7.4.8-cluster-gpl
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> 
mysql> 
mysql> GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY 'root' WITH GRANT OPTION;
ERROR 1820 (HY000): You must SET PASSWORD before executing this statement
mysql> GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY 'root001' WITH GRANT OPTION;
ERROR 1820 (HY000): You must SET PASSWORD before executing this statement

执行SET PASSWORD修改成功
mysql> SET PASSWORD = PASSWORD('root001');
Query OK, 0 rows affected (0.02 sec)
mysql>  GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY 'root' WITH GRANT OPTION;
Query OK, 0 rows affected (0.02 sec)
mysql> 
FLUSH PRIVILEGES;

执行完成
远程连接测试，mysql集群节点，发现一个节点可以连接，一个节点不能连接
这个问题纠结很久
原因上面的操作，没有在另外一个节点上执行导致的，mysql集群的用户认证是分开管理的，要两个节点都要执行
索引以后创建用户要两个节点都执行

5.配置两个管理节点时show报错Could not get configuration
[root@ndbcluster2 mysql-cluster]# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: 192.168.1.17:1186
ERROR Message: The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 6 to connect.
Could not get configuration
*  4012: Failed to get configuration
*        The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 6 to connect.
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2 (not connected, accepting connect from 192.168.1.18)
id=3 (not connected, accepting connect from 192.168.1.17)
[ndb_mgmd(MGM)] 2 node(s)
id=1    @192.168.1.17  (mysql-5.6.27 ndb-7.4.8)
id=6    @192.168.1.18  (mysql-5.6.27 ndb-7.4.8)
[mysqld(API)]   2 node(s)
id=4 (not connected, accepting connect from 192.168.1.17)
id=5 (not connected, accepting connect from 192.168.1.18)
两个管理节点同时都启动起来，才能看到状态
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @192.168.1.18  (mysql-5.6.27 ndb-7.4.8, Nodegroup: 0, *)
id=3    @192.168.1.17  (mysql-5.6.27 ndb-7.4.8, Nodegroup: 0)
[ndb_mgmd(MGM)] 2 node(s)
id=1    @192.168.1.17  (mysql-5.6.27 ndb-7.4.8)
id=6    @192.168.1.18  (mysql-5.6.27 ndb-7.4.8)
[mysqld(API)]   2 node(s)
id=4    @192.168.1.17  (mysql-5.6.27 ndb-7.4.8)
id=5    @192.168.1.18  (mysql-5.6.27 ndb-7.4.8)