搭建CEPH测试环境

搭建CEPH测试环境

环境

硬件环境：
- 单机（虚拟机）
- CPU 8 cores
- MEM 32G
- DISK 3 x 100G

软件环境：
- Centos7 x64
- 数据盘 /dev/sdb1 /data1 xfs
- 数据盘 /dev/sdc1 /data2 xfs
- 系统接入互联网

CEPH版本
- ceph-deploy v1.5.37
- ceph version 10.2.7

CEPH 架构

CEPH统一地提供了对象、文件和块文件系统，如下图：

本文只搭建了Object Storage文件系统，未实现Block Device和Ceph FS文件系统。主要步骤如下：
- Monitor
- OSDs
- MDS
- RGW
- TEST（put & get file）

系统环境配置

Internet

vi /etc/sysconfig/network-scripts/ifcfg-ens33
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.3.125
DNS1=192.168.3.1
GATEWAY=192.168.3.1

Hostname

sed -i '/HOSTNAME/d' /etc/sysconfig/network
echo "HOSTNAME=cydb" >> /etc/sysconfig/network
cat /etc/sysconfig/network

echo "192.168.3.125 cydb" >> /etc/hosts
cat /etc/hosts

hostname cydb
hostname -f

SSH

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
ssh-copy-id root@cydb

Firewall

systemctl stop firewalld.service
systemctl disable firewalld.service 
systemctl status firewalld.service

SELinux

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
cat /etc/selinux/config
reboot
sestatus

Ojbect文件系统

Ceph安装源

yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
cat > /etc/yum.repos.d/ceph.repo << EOF
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-jewel/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
EOF

cat /etc/yum.repos.d/ceph.repo
yum clean all
yum makecache

安装ceph-deploy

yum install -y ceph-deploy
ceph-deploy --help

部署cluster

mkdir /opt/ceph-cluster
cd /opt/ceph-cluster
ceph-deploy new cydb
echo "osd crush chooseleaf type = 0" >> ceph.conf
echo "osd pool default size = 1" >> ceph.conf
echo "osd journal size = 100" >> ceph.conf
cat ceph.conf

安装ceph包

yum remove -y ceph-release # 如果安装失败，则执行
ceph-deploy install cydb
ceph --version

部署Monitor

配置文件定义了hostname

ceph-deploy mon create-initial
ceph mon stat
ceph mon dump

添加OSDs

chown ceph:ceph /data1
chown ceph:ceph /data2

ceph-deploy osd prepare cydb:/data1
ceph-deploy osd prepare cydb:/data2

ceph-deploy osd activate cydb:/data1
ceph-deploy osd activate cydb:/data2

ceph osd stat
ceph osd tree
ceph osd dump

推送配置文件和密钥文件

ceph-deploy admin cydb
chmod +r /etc/ceph/ceph.client.admin.keyring
ceph status

部署MDS

ceph-deploy mds create cydb
ceph mds stat

创建RGW实例

ceph-deploy rgw create cydb

测试

# create pool
rados mkpool data
# put object
echo Test-data > testfile.txt
cat testfile.txt
rados put test-object-1 testfile.txt --pool=data
# list pool
rados -p data ls
# lookup object location
ceph osd map data test-object-1
# get object
rados get test-object-1 testfile.get.txt --pool=data
cat testfile.get.txt
# del object from pool
rados rm test-object-1 --pool=data

参考

CEPH官网 http://docs.ceph.com/docs/master/
Ubuntu 14.04 单机安装 CEPH http://www.cnblogs.com/YaoDD/p/5217578.html