分布式文件系统Ceph的部署使用

Ceph集群部署

阿里云yum镜像

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

yum makecache

/etc/sudoers中可以添加用户到sudo中

安装前的准备

1. 安装ceph-deploy

1. 安装证书管理

yum install subscription-manager

2. 进行认证

subscription-manager repos --enable=rhel-7-server-extras-rpms

3. 安装并启用企业Linux附加软件包（EPEL）存储库

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

4. 添加yum仓库镜像（163的国内镜像）

cat << EOM > /etc/yum.repos.d/ceph.repo
[ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.163.com/ceph/rpm-luminous/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=http://mirrors.163.com/ceph/keys/release.asc
EOM

5. 更新yum源

sudo yum update

6. 安装ceph-deploy

sudo yum install ceph-deploy

2. 设置Ceph节点

1. 安装NTP

sudo yum install ntp ntpdate ntp-doc

2. 安装SSH Server

sudo yum install openssh-server

3. 创建Ceph-deploy用户

sudo useradd -d /home/{username} -m {username}
sudo passwd {username}

4. 将用户添加到节点中，确保有root的权限（每个节点都需要一个用户）

echo "{username} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/{username}
sudo chmod 0440 /etc/sudoers.d/{username}

5. 将各个节点添加到hosts文件中

vim /etc/hosts

192.168.151.5 ceph-admin
192.168.151.7 ceph-gateway
192.168.151.18 ceph-osd1
192.168.151.9 ceph-osd2

6. 设置免密SSH

生成SSH密钥

ssh-keygen

在.ssh目录下创建config文件

Host 192.168.151.5
    HOSTNAME ceph-admin
    USER wu
    
Host 192.168.151.7
   HOSTNAME ceph-gateway
   USER wu

Host 192.168.151.18
   HOSTNAME ceph-osd1
   USER wu

Host 192.168.151.9
   HOSTNAME ceph-osd2
   USER wu

给config文件权限

chmod 600 config

将公钥复制到各个节点

ssh-copy-id {username}@node1
ssh-copy-id {username}@node2
ssh-copy-id {username}@node3

7. 关闭防火墙

systemctl disable firewalld
systemctl stop firewalld

8. 安装插件

yum install yum-plugin-priorities

2. 集群

1. 在管理节点上面创建文件夹

mkdir my-cluster
cd my-cluster

2. 从头开始

ceph-deploy purge {ceph-node} [{ceph-node}]
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys
rm ceph.*

3. 创建集群

ceph-deploy new {initial-monitor-node(s)}

4. 安装Ceph软件包

ceph-deploy install {ceph-node} [...]

# 如果安装一直有问题，清空/etc/yum.repo，然后将yum源换掉：http://mirrors.ustc.edu.cn/help/epel.html
问题1：[ceph_deploy][ERROR ] RuntimeError: NoSectionError: No section: 'ceph'
解决方法：yum remove ceph-releaserm  /etc/yum.repos.d/ceph.repo.rpmsave

问题2：[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y install ceph ceph-radosgw
解决方法: ceph安装缺少python-werkzeug包安装包
下载地址：http://rpmfind.net/linux/rpm2html/search.php?query=python-werkzeugrpm -ivh python-werkzeug-0.9.1-2.el7.noarch.rpm

问题3：[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph –version
解决方式：ceph1 安装速度过慢，已经超时了，直接手动安装 yum -y install ceph ceph-radosgw

5. 配置网络（ceph.conf）和OSD默认数量

public network = {ip-address}/{bits}
osd pool default size = 2

6. 部署Monitor

ceph-deploy mon create-initial

问题1：admin_socket: exception getting command descriptions: [Errno 2] No such file or direc
解决方式：没有配置public network，配置即可

问题2： RuntimeError: config file /etc/ceph/ceph.conf exists with different content; 
use --overwrite-conf to overwrite [ceph_deploy][ERROR ] GenericError: Failed to configure 1 admin hosts
解决方式：ceph-deploy --overwrite-conf config push ceph-admin

问题3：ceph auth get-or-create for keytype admin returned 1 
配置文件的mon_host和public network不一致
解决方式：编辑ceph.conf配置项的mon_host和public network 一致

7. 拷贝配置信息到ceph节点

ceph-deploy admin ceph-osd1 ceph-osd2 gateway

8. 部署管理节点

ceph-deploy mgr create ceph-admin

9. 添加两个OSD

两个磁盘都必须是没有使用且没有数据的，一台服务器可以有多个身份

ceph-deploy osd create --data /dev/sdb ceph-osd1
ceph-deploy osd create --data /dev/sdb ceph-osd2

如果要在LVM卷上创建OSD，则–data的参数必须是 volume_group/lv_name，而不是卷的块设备的路径

3. 拓展

1. 增加两个监控节点

ceph-deploy mon add {ceph-nodes}
例：
ceph-deploy mon add ceph-osd1
ceph-deploy mon add ceph-osd2

2. 一旦创建了多个监控，ceph会自动同步，形成一个quorum，可以查看状态

ceph quorum_status --format json-pretty

问题：auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,
/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
: (2) No such file or directory
ERROR: missing keyring, cannot use cephx for authentication
librados: client.admin initialization error (2) No such file or directory

解决方式：查看令牌权限不够
sudo chmod +r /etc/ceph/ceph.client.admin.keyring

3. 添加两个manager节点

ceph-deploy mgr create node2 node3

4. 添加RGW

ceph-deploy rgw create gateway

5. 网关的默认端口是7480，如果需要修改则需要在ceph.conf配置文件中修改

[client]
rgw frontends = civetweb port=80

6. 在网页输入 http://gateway:7480 可以看到一个 xml 页面，则表示成功

7. 存储对象

   1. 设置一个object name
   2. 设置一个pool

   ceph osd pool create mypool 8   # 8是该pool的PG的数量
   

   3. 存储对象

   rados put my-obj-1 1.txt --pool=mypool
   

   4. 查看存储的对象

   ceph osd map {poolname} {object-name}
   

   5. 删除对象

   rados rm my-obj-1 --pool=mypool
   

https://docs.ceph.com

使用

1. 创建网关用户

sudo radosgw-admin user create --uid="rgwuser" --display-name="This is rgw test user"

在输出的信息中，记录access_key和secret_key

"access_key": "HZY01FJ0BV68D3J0L9GS",
"secret_key": "JZzn0pk6JsLY6lEMQfKgEIDONjg0P89AiMnlKjM7"

2. 在Java中使用参考S3的接口即可

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectResult;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.util.IOUtils;
import com.amazonaws.util.StringUtils;
import org.springframework.stereotype.Service;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.SocketTimeoutException;
import java.util.ArrayList;
import java.util.List;

@Service
public class CephService {

    protected static AmazonS3 amazonS3;

    private static String accessKey;

    private static String secretKey;

    private static String host;

    static {

        accessKey = "HZY01FJ0BV68D3J0L9GS";
        secretKey = "JZzn0pk6JsLY6lEMQfKgEIDONjg0P89AiMnlKjM7";
        host = "192.168.151.5:7480";

        System.out.println("开始初始化cephp配置..." + accessKey + host + secretKey);

        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        ClientConfiguration clientConfig = new ClientConfiguration();
        clientConfig.setProtocol(Protocol.HTTP);
        amazonS3 = new AmazonS3Client(credentials, clientConfig);
        amazonS3.setEndpoint(host);
        System.out.println("ceph配置初始化成功！");

    }

    public Bucket createBucket(String bucketname) throws SocketTimeoutException {

        System.out.println("开始创建桶：" + bucketname);
        Bucket bucket = amazonS3.createBucket(bucketname);
        System.out.println(amazonS3.toString());

        return bucket;
    }

    public List getBuckets() {

        List<Bucket> buckets = amazonS3.listBuckets();

        for (Bucket bucket : buckets) {
            System.out.println(bucket.getName() + "\t" + bucket.getOwner().toString()
                    + "\t" + StringUtils.fromDate(bucket.getCreationDate()));
        }

        return buckets;
    }

    public List getBucketsContents(String bucketName){
        ObjectListing objects = amazonS3.listObjects(bucketName);
        List list = new ArrayList();
        do {
            for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) {
                String contents = objectSummary.getKey() + "\t"
                        + objectSummary.getSize() + "\t"
                        + StringUtils.fromDate(objectSummary.getLastModified());
                System.out.println(contents);
                list.add(contents);
            }
            objects = amazonS3.listNextBatchOfObjects(objects);
        } while (objects.isTruncated());
        return list;
    }

    public void delBucket(String bucketName){
        amazonS3.deleteBucket(bucketName);
    }

    public boolean newObj(String bucketName, String fileName, InputStream inputStream) throws SocketTimeoutException {

        System.out.println("开始写入数据：" );

        PutObjectResult result = amazonS3.putObject(bucketName, fileName, inputStream, new ObjectMetadata());

        return result.isRequesterCharged();

    }

    public S3ObjectInputStream getObj(String bucketName, String filename) throws IOException {

        S3Object s3Object = amazonS3.getObject(new GetObjectRequest(bucketName, filename));
        S3ObjectInputStream objectContent = s3Object.getObjectContent();
        System.out.println("Content-Type: " + s3Object.getObjectMetadata().getContentType());
        System.out.println("Content-Length: " + s3Object.getObjectMetadata().getContentLength());
        displayTextInputStream(objectContent);

        byte[] bytes = IOUtils.toByteArray(objectContent);
        return objectContent;

//        S3Object object = amazonS3.getObject(new GetObjectRequest(bucketName, filename));
//        S3ObjectInputStream objectContent = object.getObjectContent();
//        return objectContent;

    }

    public void delObj(String bucketName,String filename){
        amazonS3.deleteObject(bucketName, filename);
    }

    public List getFiles(String bucketName) {

        System.out.println("开始查找文件：");

        ObjectListing objects = amazonS3.listObjects(bucketName);
        List list = new ArrayList<>();

        for (S3ObjectSummary objectSummary : objects.getObjectSummaries()) {
            list.add(objectSummary);
        }

        return list;
    }

    private static void displayTextInputStream(InputStream input) throws IOException {
        // Read the text input stream one line at a time and display each line.
        BufferedReader reader = new BufferedReader(new InputStreamReader(input));
        String line = null;
        while ((line = reader.readLine()) != null) {
            System.out.println(line);
        }
        System.out.println();
    }

}