Ceph API微服务开发--cephfs文件系统服务

目标：使用libcephfs库，实现对cephfs文件系统的挂载、目录、文件上传与下载等操作

环境：CentOS 7 / IntelliJ IDEA (因为libcephfs仅支持linux，所以在centos7中安装IDE开发程序)

步骤：libcephfs简介->Linux环境配置->cephfs挂载与目录基本方法实现->文件上传与下载功能开发

1.libcephfs简介

CephFS通过在RADOS基础上增加了MDS(Metadata Server)来提供文件存储，并提供libcephfs库供开发者调用其接口。

libcephfs javadoc：http://docs.ceph.org.cn/api/libcephfs-java/javadoc/

上述网址中列出了库中包含的类及其方法，其中主要的类为CephMount。

maven依赖：

    com.ceph
    libcephfs
    0.80.5

2.Linux环境配置

除了添加maven依赖之外，还需在linux开发机上安装libcephfs相关开发包：

libcephfs1-0.94.9-0.el7.x86_64.rpm

libcephfs_jni1-0.94.9-0.el7.x86_64.rpm

下载地址：http://mirror.neu.edu.cn/centos/7/storage/x86_64/ceph-hammer/

安装：yum -y install libcephfs1-0.94.9-0.el7.x86_64.rpm libcephfs_jni1-0.94.9-0.el7.x86_64.rpm

设置链接：

ln -s /usr/lib64/libcephfs_jni.so.1.0.0 /usr/lib/libcephfs_jni.so.1

ln -s /usr/lib64/libcephfs_jni.so.1.0.0 /usr/lib/libcephfs_jni.so

同时需要安装ceph-common组件：yum -y install ceph-common

3.cephfs挂载与目录基本方法实现

Service代码：以CephMount类为主要操作类，实现cephfs的操作

package com.boe.cloud.service.cephfs.service.impl;
 
import com.ceph.fs.CephMount;
import com.ceph.fs.CephStat;
import org.springframework.stereotype.Service;
 
@Service
public class CephfsServiceImpl implements com.boe.cloud.service.cephfs.service.CephfsService {
 
    private CephMount mount=null;
 
    @Override
    public Boolean mountCephfsByRoot(){
        try {
            this.mount = new CephMount("admin");
            this.mount.conf_set("mon_host", "192.168.0.44;192.168.0.45;192.168.0.46");
            //System.out.println(mount.conf_get("mon_host"));
            this.mount.conf_set("key","AQCxGlBbNBRBGBAAAYW6tv/Z8x2Dz1mnCxwW9w==");
            this.mount.mount("/");
            return true;
        }catch (Exception e){
            e.printStackTrace();
        }
        return false;
    }
 
 
    @Override
    public String[] createDirByPath(String path){
        String[] dirList = null;
        try {
            if (this.mount == null){
                return null;
            }
            this.mount.mkdirs(path, 0777);
            dirList = this.mount.listdir("/");
            return dirList;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }
 
    @Override
    public String[] deleteDirByPath(String path){
        String[] dirList = null;
        try {
            if (this.mount == null){
                return null;
            }
            this.mount.rmdir(path);
            dirList = this.mount.listdir("/");
            return dirList;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }
 
    @Override
    public CephStat getFileStatusByPath(String path){
        CephStat stat = new CephStat();
        try {
            if (this.mount == null){
                return null;
            }
            this.mount.lstat(path, stat);
            return stat;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }
 
    @Override
    public String readFileByPath(String path){
        CephStat stat = new CephStat();
        String context=null;
        try {
            if (this.mount == null){
                return null;
            }
            int fd = this.mount.open(path, CephMount.O_RDONLY, 0);
            this.mount.fstat(fd, stat);
            byte[] buffer = new byte[(int)stat.size];
            this.mount.read(fd, buffer, stat.size, 0);
            context = new String(buffer);
            this.mount.close(fd);
            return context;
        }catch (Exception e){
            e.printStackTrace();
        }
        return null;
    }
}

控制类+swagger实现api调用接口：

package com.boe.cloud.service.cephfs.controller;
 
import com.boe.cloud.service.cephfs.service.impl.CephfsServiceImpl;
import com.ceph.fs.CephStat;
import io.swagger.annotations.ApiOperation;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
 
@RestController
@RequestMapping(value = "/cephfs/api")
public class CephfsController {
 
    private CephfsServiceImpl cephfsService = new CephfsServiceImpl();
 
    @ApiOperation(value = "Mount", notes = "Mount")
    @RequestMapping(value = "/mount", method = RequestMethod.GET)
    public Boolean mountCephFsByRoot(){
        return cephfsService.mountCephfsByRoot();
    }
 
    @ApiOperation(value = "CreateDir", notes = "CreateDir")
    @RequestMapping(value = "/createdir", method = RequestMethod.POST)
    public String[] createDirByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.createDirByPath(path);
    }
 
    @ApiOperation(value = "DeleteDir", notes = "DeleteDir")
    @RequestMapping(value = "/deletedir", method = RequestMethod.DELETE)
    public String[] deleteDirByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.deleteDirByPath(path);
    }
 
    @ApiOperation(value = "FileStatus", notes = "FileStatus")
    @RequestMapping(value = "/getfilestatus", method = RequestMethod.GET)
    public CephStat getFileStatusByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.getFileStatusByPath(path);
    }
 
    @ApiOperation(value = "FileContext", notes = "FileContext")
    @RequestMapping(value = "/getfilecontext", method = RequestMethod.GET)
    public String readFileByPath(@RequestParam(value = "DirPath") String path){
        return cephfsService.readFileByPath(path);
    }
 
}

测试：mount "/"目录，使用listdir查看目录

4.文件上传与下载功能开发

(1)上传功能

使用java文件读写方法读取本地文件，libcephfs中的write方法写文件的形式将指定文件上传至cephfs，简单流程图如下：

实现方法：

    @Override
    public Boolean uploadFileByPath(String filePath, String fileName){
 
        // exit with null if not mount
        if (this.mount == null){
            logger.info("Ceph fs not mount!");
            return null;
        }
 
        // file definition
        char pathChar = File.separatorChar;
        String fileFullName = "";
        Long fileLength = 0l;
        Long uploadedLength = 0l;
        File file = null;
 
        // Io
        FileInputStream fis = null;
 
        // get local file info
        fileFullName = filePath + pathChar + fileName;
        file = new File(fileFullName);
        if (!file.exists()){
            logger.info("File not exist!");
            return false;
        }
        fileLength = file.length();
 
        // get io from local file
        try {
            fis = new FileInputStream(file);
        }catch (FileNotFoundException e){
            logger.info("Read local file failed!");
            e.printStackTrace();
        }
 
        // if file exists or not
        String[] dirList = null;
        Boolean fileExist = false;
        try {
            dirList = this.mount.listdir("/");
            for (String fileInfo : dirList){
                if (fileInfo.equals(fileName)){
                    fileExist = true;
                }
            }
        }catch (FileNotFoundException e){
            logger.info("File Path not exist!");
            e.printStackTrace();
        }
 
        // transfer file by diff pattern
        if (!fileExist){
            try {
                // create file and set mode WRITE
                this.mount.open(fileName, CephMount.O_CREAT, 0);
                int fd = this.mount.open(fileName, CephMount.O_RDWR, 0);
 
                // start transfer
                int length = 0;
                byte[] bytes = new byte[1024];
                while ((length = fis.read(bytes, 0, bytes.length)) != -1){
                    // write
                    this.mount.write(fd, bytes, length, uploadedLength);
 
                    // update length
                    uploadedLength += length;
 
                    // output transfer rate
                    float rate = (float)uploadedLength * 100 / (float)fileLength;
                    String rateValue = (int)rate + "%";
                    System.out.println(rateValue);
 
                    // complete flag
                    if (uploadedLength == fileLength){
                        break;
                    }
                }
                System.out.println("文件传输成功！");
 
                // chmod
                this.mount.fchmod(fd, 0666);
 
                // close
                this.mount.close(fd);
                if (fis != null){
                    fis.close();
                }
                return true;
            }catch (Exception e){
                logger.info("File transfer failed!");
                e.printStackTrace();
            }
        }else if (fileExist){
            try {
                // get file length
                CephStat stat = new CephStat();
                this.mount.stat(fileName, stat);
                uploadedLength = stat.size;
                int fd = this.mount.open(fileName, CephMount.O_RDWR, 0);
 
                // start transfer
                int length = 0;
                byte[] bytes = new byte[1024];
                fis.skip(uploadedLength);
                while ((length = fis.read(bytes, 0, bytes.length)) != -1){
                    // write
                    this.mount.write(fd, bytes, length, uploadedLength);
 
                    // update length
                    uploadedLength += length;
 
                    // output transfer rate
                    float rate = (float)uploadedLength * 100 / (float)fileLength;
                    String rateValue = (int)rate + "%";
                    System.out.println(rateValue);
 
                    // complete flag
                    if (uploadedLength == fileLength){
                        break;
                    }
                }
                System.out.println("断点文件传输成功！");
 
                // chmod
                this.mount.fchmod(fd, 0666);
 
                // close
                this.mount.close(fd);
                if (fis != null){
                    fis.close();
                }
                return true;
            }catch (Exception e){
                logger.info("BreakPoint transfer failed!");
                e.printStackTrace();
            }
        }else {
            try {
                if (fis != null){
                    fis.close();
                }
            }catch (Exception e){
                e.printStackTrace();
            }
            return false;
        }
 
        return false;
    }

测试：

上传本地压缩文件，并在传输过程中手动中断，进入cephfs挂载目录查看：文件实际大小为291M

继续传输直至结束：查看控制台输出

查看：

(2)下载功能

与上传功能类似，不同的是使用libcephfs的read方法

下载功能方法编写：

    @Override
    public Boolean downloadFileByPath(String filePath, String fileName){
 
        // exit with null if not mount
        if (this.mount == null){
            logger.info("Ceph fs not mount!");
            return null;
        }
 
        // file definition
        char pathChar = File.separatorChar;
        String fileFullName = "";
        Long fileLength = 0l;
        Long downloadedLength = 0l;
        File file = null;
 
        // IO
        FileOutputStream fos = null;
        RandomAccessFile raf = null;
 
        // new file object
        fileFullName = filePath + pathChar + fileName;
        file = new File(fileFullName);
 
        // get cephfs file size
        try {
            CephStat stat = new CephStat();
            this.mount.stat(fileName, stat);
            fileLength = stat.size;
        }catch (Exception e){
            logger.info("Fail to get file size.");
            e.printStackTrace();
        }
 
        if (fileLength != 0){
            if (!file.exists()){
                // download file
                int length = 10240;
                byte[] bytes = new byte[length];
                try {
                    int fd = this.mount.open(fileName, CephMount.O_RDONLY, 0);
                    fos = new FileOutputStream(file);
                    float rate = 0;
                    String rateValue = "";
                    while ((fileLength - downloadedLength) >= length  && (this.mount.read(fd, bytes, (long)length, downloadedLength)) != -1){
                        fos.write(bytes, 0, length);
                        fos.flush();
                        downloadedLength += (long)length;
 
                        // output transfer rate
                        rate = (float)downloadedLength * 100 / (float)fileLength;
                        rateValue = (int)rate + "%";
                        System.out.println(rateValue);
 
                        if (downloadedLength == fileLength){
                            break;
                        }
                    }
                    if (downloadedLength != fileLength){
                        this.mount.read(fd, bytes, fileLength-downloadedLength, downloadedLength);
                        fos.write(bytes, 0, (int)(fileLength-downloadedLength));
                        fos.flush();
                        downloadedLength = fileLength;
 
                        // output transfer rate
                        rate = (float)downloadedLength * 100 / (float)fileLength;
                        rateValue = (int)rate + "%";
                        System.out.println(rateValue);
                    }
 
                    System.out.println("Download Success!");
                    fos.close();
                    this.mount.close(fd);
                    return true;
                }catch (Exception e){
                    logger.info("First download fail!");
                    e.printStackTrace();
                }
            }else if (file.exists()){
                // download file
                int length = 10240;
                byte[] bytes = new byte[length];
                Long filePoint = file.length();
                try {
                    int fd = this.mount.open(fileName, CephMount.O_RDONLY, 0);
                    raf = new RandomAccessFile(file,"rw");
                    raf.seek(filePoint);
                    downloadedLength = filePoint;
                    float rate = 0;
                    String rateValue = "";
                    while ((fileLength - downloadedLength) >= length  && (this.mount.read(fd, bytes, (long)length, downloadedLength)) != -1){
                        raf.write(bytes, 0, length);
                        downloadedLength += (long)length;
 
                        // output transfer rate
                        rate = (float)downloadedLength * 100 / (float)fileLength;
                        rateValue = (int)rate + "%";
                        System.out.println(rateValue);
 
                        if (downloadedLength == fileLength){
                            break;
                        }
                    }
                    if (downloadedLength != fileLength){
                        this.mount.read(fd, bytes, fileLength-downloadedLength, downloadedLength);
                        raf.write(bytes, 0, (int)(fileLength-downloadedLength));
                        downloadedLength = fileLength;
 
                        // output transfer rate
                        rate = (float)downloadedLength * 100 / (float)fileLength;
                        rateValue = (int)rate + "%";
                        System.out.println(rateValue);
                    }
 
                    System.out.println("Cut Point Download Success!");
                    raf.close();
                    this.mount.close(fd);
                    return true;
                }catch (Exception e){
                    logger.info("Continue download fail!");
                    e.printStackTrace();
                }
            }else {
                logger.info("Unknown Error!");
                return false;
            }
        }

运行测试：

输入参数

手动中止

继续下载至结束

以上，基本方法开发完成。