GreenPlum介绍
PostgreSql
PostgreSQL is an advanced object-relational database management system that supports an extended subset of the SQL standard, including transactions, foreign keys, subqueries, triggers, user-defined types and functions. This distribution also contains C language bindings.
GreenPlum
The Greenplum Database (GPDB) is an advanced, fully featured, open source data warehouse. It provides powerful and rapid analytics on petabyte scale data volumes. Uniquely geared toward big data analytics, Greenplum Database is powered by the world’s most advanced cost-based query optimizer delivering high analytical query performance on large data volumes.
Greenplum 是基于 PostgreSQL 开发的，语法与 PostgreSQL 几乎一样。
本质上，Greenplum 是一个关系型数据库集群，由多个独立的数据库服务组 合成的逻辑数据库，对于用户来说 Greenplum 与 Oracle、PostgreSQL 并无不同， 都可以通过标准 SQL 语句对数据进行访问。
GreenPlum源码编译安装
以CentOS 6.9为例，下载地址为阿里云镜像。
CentOS-6.9-x86_64-bin-DVD1.iso
CentOS-6.9-x86_64-bin-DVD2.iso
强烈建议以gpadmin为用户名。
准备工作
添加用户root权限
vim 打开 /etc/sudoers 文件 （以root权限打开）
找到以下两行
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
并在下一行添加
gpadmin ALL=(ALL) ALL #注意对齐
保存并返回即可
安装gcc-c++
sudo yum install gcc-c++
安装git
sudo yum install git
安装cmake
cd /home/gpadmin
wget https://cmake.org/files/v3.5/cmake-3.5.2.tar.gz
tar -zxvf cmake-3.5.2.tar.gz
cd cmake-3.5.2
./configure --prefix=/usr/local
make
sudo make install
安装pip
sudo yum -y install epel-release
sudo yum -y install python-pip
pip install --upgrade #pip版本至少为 7.x.x
确保已安装必需的python模块
- psutil
- lockfile (>= 0.9.1)
- paramiko
- setuptools
确保动态链接库共享
添加 /usr/local/lib 和 /usr/local/lib64 到 /etc/ld.so.conf 中
并运行命令
sudo ldconfig
支持 c++11
sudo yum install -y centos-release-scl
sudo yum install -y devtoolset-6-toolchain
echo 'source scl_source enable devtoolset-6' >> ~/.bashrc
source ~/.bashrc
安装依赖
sudo yum -y install gcc git apr bison flex readline gcc-c++ curl-devel bzip2-devel python-devel readline-devel apr-devel libevent-devel openssl-devel perl-ExtUtils-Embed libxml2-devel openldap-devel pam pam-devel
安装GPORCA
clone GPORCA
cd /home/gpadmin
git clone https://github.com/greenplum-db/gporca.git
安装 gp-xerces
git clone https://github.com/greenplum-db/gp-xerces.git
cd gp-xerces
mkdir build
cd build
../configure --prefix=/usr/local
make
sudo make install
安装ninja
cd /home/gpadmin
git clone git://github.com/ninja-build/ninja.git && cd ninja
./configure.py --bootstrap
sudo cp ninja /usr/bin
编译安装GPORCA
cd /home/gpadmin/gporca
cmake -GNinja -H. -Bbuild
sudo ninja install -C build
编译GreenPlum
cd /home/gpadmin
git clone https://github.com/greenplum-db/gpdb.git
cd gpdb
./configure --with-perl --with-python --with-libxml --with-gssapi --prefix=/usr/local/gpdb
# Compile and install
make -j8
sudo make -j8 install
# Bring in greenplum environment into your running shell
source /usr/local/gpdb/greenplum_path.sh
若出现
Greenplum Database installation complete.
则说明安装成功
部署
以在同一台主机上的一个master节点和一个segment节点为例
配置host文件
查看ip
ip addr show eth0
# 选择inet后的ip
修改hosts文件
sudo vim /etc/hosts
添加两行
mdw
sdw1
#此处ip相同
修改network文件
sudo vim /etc/sysconfig/network
#修改 HOSTNAME=mdw
创建主机列表文件和segment文件
cd /home/gpadmin
mkdir conf
cd conf
sudo vim hostlist
添加以下内容
mdw
sdw1
sudo vim seg_hosts
添加以下内容
sdw1
共享内存与网络参数配置
sudo vim /etc/sysctl.conf
对照替换成以下内容
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 1
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.sem = 250 64000 100 512
kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 64000 100 512
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 10000
vm.overcommit_memory = 2
net.ipv4.conf.all.arp_filter = 1
sudo sysctl -p
加入限制参数
sudo vim /etc/security/limits.conf
添加以下内容
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
# 星号不能去掉
节点部署
环境变量设置
source /usr/local/gpdb/greenplum_path.sh
配置免密码登陆
sudo pip install psutil
gpssh-exkeys -f /home/gpadmin/conf/hostlist
创建数据目录
cd /home/gpadmin
mkdir gpdata
cd gpdata
mkdir segmentdata1 masterdata
环境变量配置
vim ~/.bash_profile
添加以下内容
source /usr/local/gpdb/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/home/gpadmin/gpdata/masterdata/gpseg-1
export PGPORT=2345
export PGDATABASE=testDB
source ~/.bash_profile
编写数据库启动参数文件
cd /home/gpadmin/conf
mkdir gpconfigs
cd gpconfigs
vim gpinitsystem_config
添加以下内容
#启动参数文件内容
ARRAY_NAME="Greenplum"
SEG_PREFIX=gpseg
PORT_BASE=40000
declare -a DATA_DIRECTORY=(/home/gpadmin/gpdata/segmentdata1 /home/gpadmin/gpdata/segmentdata1)
MASTER_HOSTNAME=mdw
MASTER_DIRECTORY=/home/gpadmin/gpdata/masterdata
##### Port number for the master instance.
MASTER_PORT=2345
# #### Shell utility used to connect to remote hosts.
TRUSTED_SHELL=/usr/bin/ssh
##### Maximum log file segments between automatic WAL checkpoints.
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
#MIRROR_PORT_BASE=50000
REPLICATION_PORT_BASE=41000
#MIRROR_REPLICATION_PORT_BASE=51000
#declare -a MIRROR_DATA_DIRECTORY=(/home/gpadmin/gpdata/segmentmirror1 #/home/gpadmin/gpdata/segmentmirror)
MACHINE_LIST_FILE=/home/gpadmin/conf/seg_hosts
初始化数据库
gpinitsystem -c /home/gpadmin/conf/gpconfigs/gpinitsystem_config –a
数据库使用
gpstart #开启数据库服务
gpstop #关闭数据库服务
gprecoverseg #恢复启动失败的节点
gpstate -m #查看节点状态
createdb testDB #创建数据库
psql #连接数据库 psql --help查看更多选项
TPC-H测试
运行Greenplum TPC-H测试
参考https://yq.aliyun.com/articles/93?commentId=29
如有兴趣，请访问个人博客YuanZi's Blog