Greenplum5.4.0分布式集群（4节点）安装与配置
Greenplum5.4.0分布式集群（4节点）安装与配置
1、环境介绍
Vmware虚拟机,4台
操作系统：Centos 7.4
硬件资源配置：
CPU：1
内存：1Gb
磁盘空间：20Gb
操作系统：centos 7.4(64)Greenpoum5.4.0分布式集群,4节点
集群规划，详细如下：
192.168.2.10 gpdb01 主节点
192.168.2.11 gpdb02 主节点切换备份节点
192.168.2.12 gpdb03 数据节点1
192.168.2.13 gpdb04 数据节点2
2、相关软件与下载
系统版本：Liunx7.4（64） 下载地址：https://vault.centos.org/
Greenplum版本：greenplum-db-5.4.0-rhel7-x86_64 下载地址：https://d13k9s5899twdr.cloudfront.net/product_files/Pivotal-Greenplum/338484d1-2991-4595-830d-5fcf22570277?Expires=1608884390&Signature=V8hLv-mkZEZdmOWKw-2O3Dnc-X5-6YA1WyAWsaZtpWk3m%7EfeHu0Zt4Nia4awxun-dIeQxtrB93n0Fqfvfz2OASJroUhxyIyhguijX8LfoQowdDffCbH1CujkBUGACE-iBGgNA%7EphFO8QO-EVZk2aKQF9rDD60492yaxq7cqYWPus1sArLwrRKh3Yip4wDH8mMpGa0u8sg9GvnIByRfgYLZtgbunOTQSfsMhArHEdLyVBQMK%7EZ73cVzLo-u6sup4zV61nBXlg4Fv-3RvUvky%7EDnFWml2rLJu6Uke8OWrHqz7sE8nDOYb-aIBWqOF-xSuoRSYcJKpOlaExbzHSkk9pcg__&Key-Pair-Id=APKAIOIREZ2UWYGOZQDA&filename=338484d1-2991-4595-830d-5fcf22570277
Greenplum管理监控工具：greenplum-cc-web-3.3.1-LINUX-x86_64 下载地址：https://d13k9s5899twdr.cloudfront.net/product_files/Pivotal-Greenplum/df14202a-7506-4533-abf2-68854fdaa043?Expires=1608708252&Signature=Zf1%7EMjxOKmTSrc89f9XW5nOLU65LGiahF23r6uyNDjBZz-35v0i%7EZ2uewUfBywjRSivR3Fn4YFNaAM3v7JMtlQND6rk-EtEOe9HeLv5WY2EhTxKVJ1jThQ4cAj-9BN3-Wqnx0TCXXPWh4hsuM5b9SnFDefmMgWpPcsoPmUtVlu9ZLPrRV8gmjaUY3dRX3KaIrdj7-a7av7EMhGkyqJmHJdpt4JHjuJpbU4Y0EnNz-t2451KyE9v-rf7KCniJLxLJi2zqGB72H9qs9t-RNg%7EyPvZVQWPbCAobWzO3Qvd63W7uSKkbg-9UeuBAISbOMCSYIPsd7%7EBCBHEZE%7E2gSWJvnQ__&Key-Pair-Id=APKAIOIREZ2UWYGOZQDA&filename=df14202a-7506-4533-abf2-68854fdaa043
3、系统环境配置
3.1 安装虚拟机环境部署
安装虚拟机系统
自行百度，参考网上资料：https://www.cnblogs.com/yiyongling/p/11104307.html
初始化配置规划
192.168.2.10 gpdb01 主节点
192.168.2.11 gpdb02 数据节点1
192.168.2.12 gpdb03 数据节点2
192.168.2.13 gpdb04 主节点切换备份节点3.2 各节点间免密配置
编译配置各节点hostname和hosts，参考配置规划
所有节点，执行以下命令，并添加相关内容：
vi /etc/hostname
#根据配置规划，在对应节点添加hostname，如主节点添加：
gpdb01
vi /etc/hosts
#添加以下内容：
192.168.2.10 gpdb01
192.168.2.11 gpdb02
192.168.2.12 gpdb03
192.168.2.13 gpdb04在对应主机按顺序分别执行以下命令,注意执行步骤在各主机上顺序执行
步骤一，生成密钥和公钥
--gpdb_01
#执行步骤一:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
cp ~/.ssh/id_dsa.pub ~/.ssh/id_dsa.pub_10
--gpdb_02
#执行步骤一:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
cp ~/.ssh/id_dsa.pub ~/.ssh/id_dsa.pub_11
--gpdb_03
#执行步骤一:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
cp ~/.ssh/id_dsa.pub ~/.ssh/id_dsa.pub_12
--gpdb_04
#执行步骤一:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
cp ~/.ssh/id_dsa.pub ~/.ssh/id_dsa.pub_13
步骤二，分发各公钥到其它节点
--gpdb_01
#执行步骤二:
scp ~/.ssh/id_dsa.pub_10 root@gpdb02:~/.ssh/
scp ~/.ssh/id_dsa.pub_10 root@gpdb03:~/.ssh/
scp ~/.ssh/id_dsa.pub_10 root@gpdb04:~/.ssh/
--gpdb_02
#执行步骤二:
scp ~/.ssh/id_dsa.pub_11 root@gpdb01:~/.ssh/
scp ~/.ssh/id_dsa.pub_11 root@gpdb03:~/.ssh/
scp ~/.ssh/id_dsa.pub_11 root@gpdb04:~/.ssh/
--gpdb_03
#执行步骤二:
scp ~/.ssh/id_dsa.pub_12 root@gpdb01:~/.ssh/
scp ~/.ssh/id_dsa.pub_12 root@gpdb02:~/.ssh/
scp ~/.ssh/id_dsa.pub_12 root@gpdb04:~/.ssh/
--gpdb_04
#执行步骤二:
scp ~/.ssh/id_dsa.pub_13 root@gpdb01:~/.ssh/
scp ~/.ssh/id_dsa.pub_13 root@gpdb02:~/.ssh/
scp ~/.ssh/id_dsa.pub_13 root@gpdb03:~/.ssh/
步骤三，将各节点的公钥合并，完成免密配置
--gpdb_01
#执行步骤三:
cat ~/.ssh/id_dsa.pub_11 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_12 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_13 >> ~/.ssh/authorized_keys
--gpdb_02
#执行步骤三:
cat ~/.ssh/id_dsa.pub_10 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_12 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_13 >> ~/.ssh/authorized_keys
--gpdb_03
#执行步骤三:
cat ~/.ssh/id_dsa.pub_10 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_11 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_13 >> ~/.ssh/authorized_keys
--gpdb_04
#执行步骤三:
cat ~/.ssh/id_dsa.pub_10 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_11 >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub_12 >> ~/.ssh/authorized_keys3.3 关闭防火墙
各节点执行下以下命令，确保关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
iptables -L -n
systemctl status firewalld3.4 修改内核相关参数
在各节点执行以下命令，并添加相关内容
vi /etc/sysctl.conf
#添加或修改以下参数
kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 2000 4096000 2000 2048
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65535
net.core.netdev_max_backlog = 10000
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
vm.overcommit_memory = 2使配置文件生效
sysctl -p各节点执行命令，并添加修改相关值
vi /etc/security/limits.conf
#添加或修改以下参数
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072 4、Greenplum安装配置
4.1 上传安装文件至/opt/目录并解压
执行命令
unzip greenplum-db-5.4.0-rhel7-x86_64.zip
unzip greenplum-cc-web-3.3.1-LINUX-x86_64.zip
#说明，解压后文件分别为：
greenplum-db-5.4.0-rhel7-x86_64.bin
greenplum-cc-web-3.3.1-LINUX-x86_64.bin 4.2 开始安装
4.2.1 执行安装命令
greenplum安装
./greenplum-db-5.4.0-rhel7-x86_64.bin
#按照提示输入安装目录或者默认安装目录
/opt/greenplum-db
#如不输入默认安装目录：/usr/local/greenplum-db-5.4.0
greenplum web安装
./greenplum-cc-web-3.3.1-LINUX-x86_64.bin
#按照提示输入安装目录或者默认安装目录
/opt/greenplum-cc-web
#如不输入默认安装目录：/usr/local/greenplum-cc-web4.2.2 解压完成后以root身份创建gpadmin用户和组，用来管理greenplum-db
groupadd gpadmin # 创建分组
useradd gpadmin -g gpadmin # 创建用户并分配组
passwd gpadmin # 为gpadmin分配密码 Huawei12#$
#更改安装目录的用户权限
cd /usr/local/
chown -R gpadmin:gpadmin greenplum-db/
chown -R gpadmin:gpadmin greenplum-cc-web/
#创建文件夹
mkdir gpconfigs
chown -R gpadmin:gpadmin gpconfigs4.2.2 创建配置文件
修改greenplum的hostfile_exkeys和hostfile配置文件
vim hostfile_exkeys
#添加以下内容
gpdb01
gpdb02
gpdb03
gpdb04
vi hostfile
#添加以下内容
gpdb01
gpdb03
gpdb04在gpadmin和root用户下添加环境变量
vi ~/.bashrc
#添加以下内容
export LD_LIBRARY_PATH=$GPHOME/lib
export MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/master/gpseg-1
source /usr/local/greenplum-db/greenplum_path.sh
source /usr/local/greenplum-cc-web/gpcc_path.sh
#保存退出，执行下面语句使其生效
source ~/.bashrc 4.2.3 切换到root用户,执行以下命令
gpssh-exkeys -f /usr/local/greenplum-db/gpconfigs/hostfile_exkeys
#拷贝mster节点公钥至各segment节点
gpseginstall -f /usr/local/greenplum-db/gpconfigs/hostfile_exkeys -p gpadmin
#说明
gpssh-exkeys -f hostfile_exkeys将会在master节点生成公私钥，并拷贝至hostfile_exkeys各segment节点，实现后续无密钥登陆gpseginstall -f /opt/greenplum-db/gpconfigs/hostfile_exkeys -p gpadmin使用默认用户名(gpadmin) 密码：gpadmin 在各segment节点安装Greenplum-db4.2.4在master及各segment节点创建数据存储目录
在各节点创建数据存储目录，执行以下命令
mkdir -p /home/gpadmin/gpdata/master
cd /home/gpadmin/
chown -R gpadmin:gpadmin /home/gpadmin/gpdata/master在主节点执行，以下命令：
gpssh -f /usr/local/greenplum-db/gpconfigs/hostfile -e "mkdir -p /home/gpadmin/gpdata/data/primary"
gpssh -f /usr/local/greenplum-db/gpconfigs/hostfile -e "mkdir -p /home/gpadmin/gpdata/data/mirror"
gpssh -f /usr/local/greenplum-db/gpconfigs/hostfile -e "chown -R gpadmin:gpadmin /home/gpadmin/gpdata"
gpssh -f /usr/local/greenplum-db/gpconfigs/hostfile_exkeys -v -e 'ntpd'切换到gpadmin用户下,执行以下命令并添加配置信息
su - gpadmin
cd /usr/local/greenplum-db/
cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config /usr/local/greenplum-db/gpconfigs/gpinitsystem_config
vi /usr/local/greenplum-db/gpconfigs/gpinitsystem_config
###修改以下内容
MASTER_HOSTNAME=gpdb01
PORT_BASE=40000
declare -a DATA_DIRECTORY=(/home/gpadmin/gpdata/data1/primary /home/gpadmin/gpdata/data2/primary )
MASTER_DIRECTORY=/home/gpadmin/gpdata/master
MASTER_PORT=5432
MIRROR_PORT_BASE=50000
REPLICATION_PORT_BASE=41000
MIRROR_REPLICATION_PORT_BASE=51000
declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/gpdata/data1/mirror /home/gpadmin/gpdata/data2/mirror)
DATABASE_NAME=gpdb
MACHINE_LIST_FILE=/usr/local/greenplum-db/gpconfigs/hostfile 4.2.5执行以下命令初始化数据库
gpinitsystem -c /usr/local/greenplum-db/gpconfigs/gpinitsystem_config -h /usr/local/greenplum-db/gpconfigs/hostfile出现下面的日志表示安装成功：
cat /home/gpadmin/gpAdminLogs/gpinitsystem_20201226.log
……………………
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Start Function ERROR_CHK
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Successfully completed update Greenplum superuser password
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-End Function ERROR_CHK
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-End Function SET_GP_USER_PW
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Start Function SCAN_LOG
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Scanning utility log file for any warning messages
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[WARN]:-*******************************************************
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[WARN]:-Scan of log file indicates that some warnings or errors
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[WARN]:-were generated during the array creation
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Please review contents of log file
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-/home/gpadmin/gpAdminLogs/gpinitsystem_20201226.log
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-To determine level of criticality
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-These messages could be from a previous run of the utility
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-that was called today!
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[WARN]:-*******************************************************
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-End Function SCAN_LOG
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Greenplum Database instance successfully created
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-------------------------------------------------------
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-To complete the environment configuration, please
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-update gpadmin .bashrc file with the following
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-1. Ensure that the greenplum_path.sh file is sourced
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-2. Add "export MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/master/gpseg-1"
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:- to access the Greenplum scripts for this instance:
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:- or, use -d /home/gpadmin/gpdata/master/gpseg-1 option for the Greenplum scripts
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:- Example gpstate -d /home/gpadmin/gpdata/master/gpseg-1
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Script log file = /home/gpadmin/gpAdminLogs/gpinitsystem_20201226.log
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-To remove instance, run gpdeletesystem utility
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-To initialize a Standby Master Segment for this Greenplum instance
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Review options for gpinitstandby
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-------------------------------------------------------
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-The Master /home/gpadmin/gpdata/master/gpseg-1/pg_hba.conf post gpinitsystem
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-has been configured to allow all hosts within this new
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-array to intercommunicate. Any hosts external to this
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-new array must be explicitly added to this file
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-Refer to the Greenplum Admin support guide which is
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-located in the /usr/local/greenplum-db/./docs directory
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-------------------------------------------------------
20201226:18:31:04:003485 gpinitsystem:gpdb01:gpadmin-[INFO]:-End Main4.2.6 psql修改数据库密码
psql -d gpdb
gpdb=# alter user gpadmin with password 'gpadmin'
gpdb=# \q
然后，重启greenplumdb集群
gpstop -a
gpstart -a上述gpdb已安装完成
5、安装gpcc
5.1、启动greenplum-db commond center(gpcc)
gpperfmon_install --enable --port 5432 --password gpadmin5.2、重启greenplumdb集群，加载gpperfmon_install采集数据
gpstop -a
gpstart -a5.3、启动web控制台界面
gpcmdr --setup
按照提示输入实例名、监听端口信息等
#Please enter the instance name
gpcc
#Press ENTER to use instance name
gpcc
#Is the master host for the Greenplum Database remote? Yy/Nn (default=N)
N
#What port does the Greenplum Database use? (default=5432)
5432
#Enable kerberos login for this instance? Yy/Nn (default=N)
N
#What port would you like the new web server to use for this instance? (default=28080)
280805.4、启动、停止web控制台
gpcmdr --start #启动
gpcmdr --stop #停止5.5、web登录gpcc控制台
访问地址：http://master:28080/
输入用户名和密码：gpmon/gpadmin6、使用Navicat Premium 12连接数据库，首先添加访问主机IP地址
vim /home/gpadmin/gpdata/master/gpseg-1/pg_hba.conf
#添加以下内容，192.168.2.1是外部访问主机ip
host all all 127.0.0.1/32 md5
host all all 192.168.2.1/32 md57、安装遇到问题及解决
7.1 greenplum初始化失败重装：
cd /home/gpadmin/
rm -rf gp*
#重装执行4.2.4内容，配置信息如无问题，不需修改
#网上有说在主节点的/home/gpadmin/gpAdminLogs下找到back开头的文件执行即可，我没有找到，所以将所有已装文件删除重建greenplum重装报错信息：
20201225:21:33:29:055604 gpinitsystem:gpdb01:gpadmin-[INFO]:-Checking Master host
20201225:21:33:30:055604 gpinitsystem:gpdb01:gpadmin-[WARN]:-Have lock file /tmp/.s.PGSQL.5432.lock but no process running on port 5432
20201225:21:33:30:gpinitsystem:gpdb01:gpadmin-[FATAL]:-Found indication of postmaster process on port 5432 on Master host Script Exiting!这个报错是因为之前的安装任务未正常结束导致，根据提示到对应主机相关目录，找到对应文件删除即可：
cd /tmp
rm -rf /tmp/.s.PGSQL.5432.lock7.2 greenplum安装错误1：
执行命令：gpssh-exkeys -f /usr/local/greenplum-db/gpconfigs/all_hosts
错误信息：
[ERROR hadoop4] authentication check failed:
Warning: Permanently added the RSA host key for IP address '192.168.250.184' to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).解决方法：
vi /etc/selinux/config
#SELINUX=enforcing
SELINUX=disabled
#修改之后reboot7.3 greenplum安装错误2：
执行命令： gpseginstall -f /usr/local/greenplum-db/gpconfigs/hostfile_exkeys -p gpadmin
报错信息:
20201225:02:16:37:012974 gpseginstall:gpdb01:root-[INFO]:-Installation Info:
link_name greenplum-db
binary_path /usr/local/greenplum-db-5.4.0
binary_dir_location /usr/local
binary_dir_name greenplum-db-5.4.0
20201225:02:16:37:012974 gpseginstall:gpdb01:root-[INFO]:-check cluster password access
20201225:02:16:39:012974 gpseginstall:gpdb01:root-[INFO]:-de-duplicate hostnames
20201225:02:16:39:012974 gpseginstall:gpdb01:root-[INFO]:-master hostname: gpdb01
20201225:02:16:40:012974 gpseginstall:gpdb01:root-[INFO]:-check for user gpadmin on cluster
20201225:02:16:41:012974 gpseginstall:gpdb01:root-[INFO]:-add user gpadmin on master
20201225:02:16:41:012974 gpseginstall:gpdb01:root-[INFO]:-add user gpadmin on cluster
20201225:02:16:41:012974 gpseginstall:gpdb01:root-[INFO]:-chown -R gpadmin:gpadmin /usr/local/greenplum-db
20201225:02:16:41:012974 gpseginstall:gpdb01:root-[INFO]:-chown -R gpadmin:gpadmin /usr/local/greenplum-db-5.4.0
20201225:02:16:41:012974 gpseginstall:gpdb01:root-[INFO]:-rm -f /usr/local/greenplum-db-5.4.0.tar; rm -f /usr/local/greenplum-db-5.4.0.tar.gz
20201225:02:16:41:012974 gpseginstall:gpdb01:root-[INFO]:-cd /usr/local; tar cf greenplum-db-5.4.0.tar greenplum-db-5.4.0
20201225:02:16:56:012974 gpseginstall:gpdb01:root-[INFO]:-gzip /usr/local/greenplum-db-5.4.0.tar
20201225:02:18:12:012974 gpseginstall:gpdb01:root-[INFO]:-remote command: mkdir -p /usr/local
20201225:02:18:13:012974 gpseginstall:gpdb01:root-[INFO]:-remote command: rm -rf /usr/local/greenplum-db-5.4.0
20201225:02:18:14:012974 gpseginstall:gpdb01:root-[INFO]:-scp software to remote location
20201225:02:18:14:012974 gpseginstall:gpdb01:root-[ERROR]:-[Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 243, in run
self.cmd.run()
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 711, in run
self.exec_context.execute(self)
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 433, in execute
stdout=subprocess.PIPE, close_fds=True)
File "/usr/local/greenplum-db/ext/python/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/local/greenplum-db/ext/python/lib/python2.7/subprocess.py", line 1235, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
20201225:02:18:14:012974 gpseginstall:gpdb01:root-[ERROR]:-[Errno 12] Cannot allocate memory
Traceback (most recent call last):
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 243, in run
self.cmd.run()
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 711, in run
self.exec_context.execute(self)
File "/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line 433, in execute
stdout=subprocess.PIPE, close_fds=True)
File "/usr/local/greenplum-db/ext/python/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/local/greenplum-db/ext/python/lib/python2.7/subprocess.py", line 1235, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
20201225:02:18:30:012974 gpseginstall:gpdb01:root-[ERROR]:-'NoneType' object has no attribute 'rc'
20201225:02:18:30:012974 gpseginstall:gpdb01:root-[CRITICAL]:-early exit from gpseginstall错误原因：linux内存不足或者交换空间不足
涉及知识
linux 的 overcommit_memory、overcommit_ratio
overcommit_memory 是内核对内存分配的一种策略。
vm.overcommit_memory 共有三种取值，分别为 0， 1， 2
vm.overcommit_memory = 0： 表示内核将检查是否有足够的可用内存供应用进程使用；如果有足够的可用内存，内存申请允许；否则，内存申请失败，并把错误返回给应用进程。
vm.overcommit_memory = 1： 表示内核允许分配所有的物理内存，而不管当前的内存状态如何。
vm.overcommit_memory = 2： 拒绝等于或者大于总可用 swap 大小以及 overcommit_ratio 指定的物理 RAM 比例的内存请求。
overcommit_ratio 默认为 50，为物理内存分配时的比例。**只有当 vm.overcommit_memory = 2 的时候才会生效 **
查看系统 overcommit 信息
# cat /proc/meminfo |grep -i commit
CommitLimit: 90971304 kB
Committed_AS: 64872556 kB
CommitLimit： 最大能分配的内存(个人理解仅仅在vm.overcommit_memory=2时候生效)，具体的值是
SWAP内存大小 + 物理内存 * overcommit_ratio / 100
Committed_AS: 当前已经分配的内存大小
解决方案：（两种）
#将 overcommit_ratio 设置为 90
echo 90 > /proc/sys/vm/overcommit_ratio
#将 vm.overcommit_memory 设置为 1
sudo bash -c "echo vm.overcommit_memory=1 >> /etc/sysctl.conf"
sudo sysctl -p7.3 greenplum安装错误3：
执行命令：gpssh -f /usr/local/greenplum-db/gpconfigs/hostfile_exkeys -v -e 'ntpd'
错误信息：
[WARN] Reference default values as $MASTER_DATA_DIRECTORY/gpssh.conf could not be found
Using delaybeforesend 0.05 and prompt_validation_timeout 1.0
[Reset ...]
[INFO] login gpdb03
[INFO] login gpdb01
[INFO] login gpdb02
[INFO] login gpdb04
[gpdb03] ntpd
[gpdb03] bash: ntpd: command not found...
[gpdb01] ntpd
[gpdb01] bash: ntpd: command not found...
[gpdb02] ntpd
[gpdb02] bash: ntpd: command not found...
[gpdb04] ntpd
[gpdb04] bash: ntpd: command not found...
[gpdb04]
[gpdb04] (process:28148): GLib-ERROR **: creating thread 'gdbus': Error creating thread: Resource temporarily unavailable
[gpdb04] Trace/breakpoint trap (core dumped)
[INFO] completed successfully解决方案：
此错误有两个，一是因进程创建的线程个数超过Ubuntu 限制的每个进程可创建线程个数导致，在终端输入如下命令修改最大值即可：
#在报错的服务器执行：
ulimit -n 4096二是NTP没有安装：
#在报错的服务器执行：
yum install ntp7.3 greenplum安装错误5：
执行命令： gpinitsystem -c /usr/local/greenplum-db/gpconfigs/gpinitsystem_config -h /usr/local/greenplum-db/gpconfigs/hostfile
错误信息：
20201225:17:36:41:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Checking configuration parameters, please wait... 20201225:17:36:42:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Reading Greenplum configuration file /usr/local/greenplum-db/gpconfigs/gpinitsystem_config 20201225:17:36:42:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Locale has not been set in /usr/local/greenplum-db/gpconfigs/gpinitsystem_config, will set to default value 20201225:17:36:42:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Locale set to en_US.utf8 20201225:17:36:42:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 250 20201225:17:36:43:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Checking configuration parameters, Completed 20201225:17:36:43:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Commencing multi-home checks, please wait... ... 20201225:17:36:45:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Configuring build for standard array 20201225:17:36:45:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Commencing multi-home checks, Completed 20201225:17:36:45:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Building primary segment instance array, please wait... ............ 20201225:17:36:59:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Building group mirror array type , please wait... ............ 20201225:17:37:18:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Checking Master host 20201225:17:37:19:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Checking new segment hosts, please wait... ........................ 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Checking new segment hosts, Completed 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Greenplum Database Creation Parameters 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:--------------------------------------- 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master Configuration 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:--------------------------------------- 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master instance name = Greenplum Data Platform 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master hostname = gpdb01 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master port = 5432 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master instance dir = /home/gpadmin/gpdata/master/gpseg-1 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master LOCALE = en_US.utf8 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Greenplum segment prefix = gpseg 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master Database = gpdb 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master connections = 250 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master buffers = 128000kB 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Segment connections = 750 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Segment buffers = 128000kB 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Checkpoint segments = 8 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Encoding = UNICODE 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Postgres param file = Off 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Initdb to be used = /usr/local/greenplum-db/./bin/initdb 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-GP_LIBRARY_PATH is = /usr/local/greenplum-db/./lib 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-HEAP_CHECKSUM is = on 20201225:17:38:10:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Ulimit check = Passed 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Array host connect type = Single hostname per node 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master IP address [1] = ::1 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master IP address [2] = 192.168.122.1 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master IP address [3] = 192.168.2.10 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Master IP address [4] = fe80::1111:6f2:37aa:4397 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Standby Master = Not Configured 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Primary segment # = 4 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Total Database segments = 12 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Trusted shell = ssh 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Number segment hosts = 3 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Mirror port base = 50000 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Replicaton port base = 41000 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Mirror replicaton port base= 51000 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Mirror segment # = 4 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Mirroring config = ON 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Mirroring type = Group 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:---------------------------------------- 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Greenplum Primary Segment Configuration 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:---------------------------------------- 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data1/primary/gpseg0 40000 2 0 41000 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data1/primary/gpseg1 40001 3 1 41001 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data2/primary/gpseg2 40002 4 2 41002 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data2/primary/gpseg3 40003 5 3 41003 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data1/primary/gpseg4 40000 6 4 41000 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data1/primary/gpseg5 40001 7 5 41001 20201225:17:38:11:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data2/primary/gpseg6 40002 8 6 41002 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data2/primary/gpseg7 40003 9 7 41003 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data1/primary/gpseg8 40000 10 8 41000 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data1/primary/gpseg9 40001 11 9 41001 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data2/primary/gpseg10 40002 12 10 41002 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data2/primary/gpseg11 40003 13 11 41003 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:--------------------------------------- 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Greenplum Mirror Segment Configuration 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:--------------------------------------- 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data1/mirror/gpseg0 50000 14 0 51000 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data1/mirror/gpseg1 50001 15 1 51001 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data2/mirror/gpseg2 50002 16 2 51002 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb03 /home/gpadmin/gpdata/data2/mirror/gpseg3 50003 17 3 51003 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data1/mirror/gpseg4 50000 18 4 51000 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data1/mirror/gpseg5 50001 19 5 51001 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data2/mirror/gpseg6 50002 20 6 51002 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb04 /home/gpadmin/gpdata/data2/mirror/gpseg7 50003 21 7 51003 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data1/mirror/gpseg8 50000 22 8 51000 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data1/mirror/gpseg9 50001 23 9 51001 20201225:17:38:12:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data2/mirror/gpseg10 50002 24 10 51002 20201225:17:38:13:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-gpdb01 /home/gpadmin/gpdata/data2/mirror/gpseg11 50003 25 11 51003 Continue with Greenplum creation Yy|Nn (default=N): > y > 20201225:17:40:24:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Building the Master instance database, please wait... > 20201225:17:40:53:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Starting the Master in admin mode > 20201225:17:41:33:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Commencing parallel build of primary segment instances > 20201225:17:41:33:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Spawning parallel processes batch [1], please wait... > ............ > 20201225:17:41:35:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Waiting for parallel processes batch [1], please wait... > ..................................................................................................................... > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:------------------------------------------------ > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Parallel process exit status > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:------------------------------------------------ > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Total processes marked as completed = 6 > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Total processes marked as killed = 0 > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[WARN]:-Total processes marked as failed = 6 <<<<< > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:------------------------------------------------ > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Commencing parallel build of mirror segment instances > 20201225:17:43:58:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Spawning parallel processes batch [1], please wait... > ............ > 20201225:17:44:00:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Waiting for parallel processes batch [1], please wait... > .............................................. > 20201225:17:44:52:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:------------------------------------------------ > 20201225:17:44:52:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Parallel process exit status > 20201225:17:44:52:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:------------------------------------------------ > 20201225:17:44:52:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Total processes marked as completed = 4 > 20201225:17:44:52:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Total processes marked as killed = 0 > 20201225:17:44:52:023714 gpinitsystem:gpdb01:gpadmin-[WARN]:-Total processes marked as failed = 8 <<<<< > 20201225:17:44:52:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:------------------------------------------------ > 20201225:17:44:53:023714 gpinitsystem:gpdb01:gpadmin-[FATAL]:-Errors generated from parallel processes > 20201225:17:44:53:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Dumped contents of status file to the log file > 20201225:17:44:53:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Building composite backout file > 20201225:17:44:53:gpinitsystem:gpdb01:gpadmin-[FATAL]:-Failures detected, see log file /home/gpadmin/gpAdminLogs/gpinitsystem_20201225.log for more detail Script Exiting! > 20201225:17:44:53:023714 gpinitsystem:gpdb01:gpadmin-[WARN]:-Script has left Greenplum Database in an incomplete state > 20201225:17:44:53:023714 gpinitsystem:gpdb01:gpadmin-[WARN]:-Run command /bin/bash /home/gpadmin/gpAdminLogs/backout_gpinitsystem_gpadmin_20201225_173641 to remove these changes > 20201225:17:44:54:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-Start Function BACKOUT_COMMAND > 20201225:17:44:54:023714 gpinitsystem:gpdb01:gpadmin-[INFO]:-End Function BACKOUT_COMMAND
查看错误
FAILED:gpdb04~50000~/home/gpadmin/gpdata/data1/mirror/gpseg4~18~4~51000
FAILED:gpdb04~50001~/home/gpadmin/gpdata/data1/mirror/gpseg5~19~5~51001
FAILED:gpdb04~50002~/home/gpadmin/gpdata/data2/mirror/gpseg6~20~6~51002
FAILED:gpdb04~50003~/home/gpadmin/gpdata/data2/mirror/gpseg7~21~7~51003
FAILED:gpdb01~50000~/home/gpadmin/gpdata/data1/mirror/gpseg8~22~8~51000
FAILED:gpdb01~50001~/home/gpadmin/gpdata/data1/mirror/gpseg9~23~9~51001
FAILED:gpdb01~50003~/home/gpadmin/gpdata/data2/mirror/gpseg11~25~11~51003
FAILED:gpdb01~50002~/home/gpadmin/gpdata/data2/mirror/gpseg10~24~10~51002
20201225:17:44:53:023714 gpinitsystem:gpdb01:gpadmin-[FATAL]:-Errors generated from parallel processes
20201225:17:44:14:038625 gpcreateseg.sh:gpdb01:gpadmin-[FATAL][10]:-Failed remote copy of segment data directory from gpdb04 to gpdb01
20201225:17:44:14:038691 gpcreateseg.sh:gpdb01:gpadmin-[FATAL][11]:-Failed remote copy of segment data directory from gpdb04 to gpdb01原因：权限问题导致
[gpadmin@gpdb01 ~]$ gpssh -f /usr/local/greenplum-db/gpconfigs/hostfile -e "mkdir -p /home/gpadmin/gpdata/data1/primary;mkdir -p /home/gpadmin/gpdata/data2/primary"
[gpdb03] mkdir -p /home/gpadmin/gpdata/data1/primary;mkdir -p /home/gpadmin/gpdata/data2/primary
[gpdb03] mkdir: cannot create directory ‘/home/gpadmin/gpdata/data1’: Permission denied
[gpdb03] mkdir: cannot create directory ‘/home/gpadmin/gpdata/data2’: Permission denied
[gpdb04] mkdir -p /home/gpadmin/gpdata/data1/primary;mkdir -p /home/gpadmin/gpdata/data2/primary
[gpdb04] mkdir: cannot create directory ‘/home/gpadmin/gpdata/data1’: Permission denied
[gpdb04] mkdir: cannot create directory ‘/home/gpadmin/gpdata/data2’: Permission denied
[gpdb01] mkdir -p /home/gpadmin/gpdata/data1/primary;mkdir -p /home/gpadmin/gpdata/data2/primary