Ceph 集群监控之Calamari 安装部署

部署环境说明

其中，Calamari Server的版本是从ceph社区百度网盘共享的镜像安装的。

网盘地址是：

http://pan.baidu.com/s/1o6BT8sa#path=%252FCalamari%252FCentos%252FCentOS7.1%252BCalamari1.3

注意：本文在部署calamari之前已经将ceph环境部署好

[root@bgw-os-node151 ~]# ceph -s

    cluster00000000-0000-0000-0000-000000000002

     health HEALTH_OK

     monmap e1: 3 mons at{bgw-os-node151=10.240.216.151:6789/0,bgw-os-node152=10.240.216.152:6789/0,bgw-os-node153=10.240.216.153:6789/0},election epoch 380540, quorum 0,1,2bgw-os-node151,bgw-os-node152,bgw-os-node153

     mdsmap e62: 1/1/1 up{0=bgw-os-node151=up:active}, 2 up:standby

     osdmap e213915: 12 osds: 12 up, 12 in

      pgmap v543085: 384 pgs, 6 pools, 123 MBdata, 488 kobjects

            56185 MB used, 3295 GB / 3350 GBavail

                 384 active+clean

Calamari Server端安装配置

说明：红色部分表示内容    蓝色部分表示命令

• 系统安装（省略）

• CalamariServer安装

• 关闭selinux

# vim /etc/selinux/config

      SELINUX=disabled

    # setenforce 0

• 关闭iptables

# systemctl stop firewalld

# chkconfig firewalld off

• 设置hosts

# cat /etc/hosts        （需要在上述四个服务器上都执行）

1. 10.240.240.200 calamari

2. 10.240.216.151 bgw-os-node151

3. 10.240.216.152 bgw-os-node152

4. 10.240.216.153 bgw-os-node153

5. n   设置yum源

#上传镜像文件（省略）并创建下面的挂载点，挂载镜像文件

# mkdir -p /iso/calamari

# mkdir -p /iso/centos7

# mount -o loop /opt/CentOS-7-x86_64-DVD-1503.iso/iso/centos7/

# mount -o loop /opt/Calamari-1.3-CentOS-7.1-x86-64-20150430.iso/iso/calamari/

# cat /etc/yum.repos.d/CentOS-Base.repo

[base]

name=CentOS-7

baseurl=file:///iso/centos7

gpgcheck=0

        # cat/etc/yum.repos.d/calamari.repo

            [calamari]

name=calamari

baseurl=file:///iso/calamari/

gpgcheck=0

n   安装基本包

# yumupdate

#yum -y install vim bash-completion net-tools pciutils sysstat wget

n   安装Calamari和diamond包

# yum-y install calamari-server diamond

n   初始化

#/usr/bin/calamari-ctl initialize

输入用户名：root（保持默认，也可以更改）

输入密码：zdh1113（随意设置，用于登录，最好一次设置好，修改比较麻烦）

n  配置calamari-client

Calamari-client是用来配置calamari登录、管理所需要的相关页面

# cd ~

# cp/iso/calamari/packages/calamari-clients-20150430.tar.gz .

# tar xf calamari-clients-20150430.tar.gz

# cd calamari-clients/

# cp -r login/dist/opt/calamari/webapp/content/login

# cp -r manage/dist/opt/calamari/webapp/content/manage

# cp -r dashboard/dist/opt/calamari/webapp/content/dashboard

# cp -r admin/dist/opt/calamari/webapp/content/admin

至此，按照社区里奇总分享的镜像和文档进行安装部署都很顺利就能完成。

n   登录

登录时使用上面初始化的用户名和密码即可。登录后出现下面的界面

Ceph端安装配置

这一步是一个难点，通过各种搜索，终于搞定了，过程如下

• 查阅资料

http://tracker.ceph.com/issues/7851

• 安装epel源

#rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

• 上传diamond-3.4.67-0.noarch.rpm

• 安装diamond和salt-minion和 ipvsadm

# yum install diamond-3.4.67-0.noarch.rpm ipvsadmsalt-minion

• 配置cephdeploy.conf

# cat ~/.cephdeploy.conf

#

# ceph-deploy configuration file

#

[ceph-deploy-global]

# Overrides for some of ceph-deploy's globalflags, like verbosity or cluster

# name

[ceph-deploy-install]

# Overrides for some of ceph-deploy's installflags, like version of ceph to

# install

#

# Repositories section

# yum repos:

# [myrepo]

# baseurl = https://user:[email protected]/rhel6

# gpgurl = https://example.org/keys/release.asc

# default = True

# extra-repos = cephrepo  # will install the cephrepo file too

#

# [cephrepo]

# name=ceph repo noarch packages

# baseurl=http://ceph.com/rpm-emperor/el6/noarch

# enabled=1

# gpgcheck=1

# type=rpm-md

# gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

# apt repos:

# [myrepo]

# baseurl = https://user:[email protected]/

# gpgurl = https://example.org/keys/release.asc

# default = True

# extra-repos = cephrepo  # will install the cephrepo file too

#

# [cephrepo]

# baseurl=http://ceph.com/rpm-emperor/el6/noarch

#gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

# 下面是重点配置

[calamari-minion]

name=ceph repo noarch packages

baseurl=http://ceph.com/rpm-emperor/fc19/x86_64

enabled=1

gpgcheck=1

type=repo-md

gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

• 修改salt-minion配置

#cat /etc/salt/minion.d/calamari.conf

master: 10.240.240.200

• 将ceph节点加入calamari （三个ceph节点都要执行）

# ceph-deploy calamari --master 10.240.240.200connect bgw-os-node151

[ceph_deploy.conf][DEBUG] found configuration file at: /root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.9): /usr/bin/ceph-deploycalamari --master 10.240.240.200 connect bgw-os-node151

[bgw-os-node151][DEBUG ]connected to host: bgw-os-node151

[bgw-os-node151][DEBUG ]detect platform information from remote host

[bgw-os-node151][DEBUG ]detect machine type

[ceph_deploy.calamari][INFO  ] Distro info: Red Hat Enterprise LinuxServer 6.5 Santiago

[bgw-os-node151][INFO  ] installing calamari-minion package onbgw-os-node151

[bgw-os-node151][INFO  ] adding custom repository file

[bgw-os-node151][INFO  ] Running command: yum clean all

[bgw-os-node151][DEBUG ]Loaded plugins: product-id, security, subscription-manager

[bgw-os-node151][DEBUG ]Cleaning repos: base calamari-minion epel extras os-base os-ceph.repo os-epel

[bgw-os-node151][DEBUG]               : os-i.repo puppetpuppet.repo update

[bgw-os-node151][DEBUG ]Cleaning up Everything

[bgw-os-node151][WARNIN]This system is not registered to Red Hat Subscription Management. You can usesubscription-manager to register.

[bgw-os-node151][INFO  ] Running command: rpm --importhttps://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc

[bgw-os-node151][DEBUG ]set the contents of repo file to /etc/yum.repos.d/

[bgw-os-node151][DEBUG ]creating config dir: /etc/salt/minion.d

[bgw-os-node151][DEBUG ]creating the calamari salt config: /etc/salt/minion.d/calamari.conf

[bgw-os-node151][INFO  ] Running command: yum -y install salt-minion

[bgw-os-node151][DEBUG ]Loaded plugins: product-id, security, subscription-manager

[bgw-os-node151][WARNIN]No data was received after 300 seconds, disconnecting...

[bgw-os-node151][INFO  ] Running command: chkconfig salt-minion on

[bgw-os-node151][INFO  ] Running command: service salt-minion start

[bgw-os-node151][DEBUG] Starting salt-minion daemon: [  OK  ]

 

上面启动成功后，就可以在web界面上看到ceph节点的信息啦

错误汇总

Ceph端的配置参考了http://my.oschina.net/renguijiayi/blog/307313

1.     将diamond-<version>.noarch.rpm复制到所有的ceph服务器。
使用yum localinstall安装，或者yuminstall python-configobj然后使用rpm -ivh安装。

2.     在所有的ceph服务器上安装salt-minion，创建/etc/salt/minion.d/calamari.conf，内容为：

master: {fqdn}

{fqdn}对应calamari服务器的域名。
启动salt-minion服务：

# service salt-minion restart

3.     在Calamari服务器上配置防火墙和saltstack认证（我这里将防火墙关闭了，省略该步骤） 
防火墙（允许ceph服务器访问salt-master和carbon）：

###salt-master

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 4505 -j ACCEPT

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 4506 -j ACCEPT

### carbon

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 2003 -j ACCEPT

# iptables -A INPUT -m state --state NEW -mtcp -p tcp --dport 2004 -j ACCEPT

# systemctl save firewalld

# systemctl restart firewalld

 

saltstack认证：
当ceph服务器上的salt-minion服务启动之后，会自动向salt-master请求认证。
在Calamari服务器上可以通过下面的命令查看salt-minion密钥的列表：

# salt-key -L

刚刚启动salt-minion服务的ceph服务器会出现在Unaccepted Keys列表之后，要使得Calamari能够通过saltstack管理ceph服务器，需要对这些密钥进行认证：

# salt-key -A

1.     部署完成之后，可以访问calamari

但是出现了下面的错误，并没有看到我们想要的结果

根据上面的提示，我做了下面的操作：

无语了，出错了，于是google了一下，查阅 http://tracker.ceph.com/issues/7851解决。注意根据这个解决的 时候，要记得把salt-minion先stop哦。

#cat /etc/ceph/ceph.conf

[global]

osdcrush location hook = /usr/bin/calamari-crush-location