Ceph 编译以及构建rpm包

概述

 对于一个ceph开发人员来说编译源码以及打rpm是其必备技能。无论是fix bug还是向社区提交pull request都离不开编译源码。而rpm包又是linux系统发行产品时常用的格式。在官网和其他blog可以很多这方面的介绍，我在这里把自己编译的步奏，以及遇到的坑及解决办法记录一下，仅供新人参考。

编译源码

环境介绍：

• ceph version: Luminous 12.2.10
• 硬件环境：Centos7虚拟机/Centos7 docker container

准备工作

• 安装yum源
  这些源用于安装常用工具，执行install-reps.sh安装以来包使用，列表如下：

-rw-r--r--. 1 root root 2523 Jan  8 07:10 CentOS-Base.repo
-rw-r--r--. 1 root root 1309 Jan  8 08:22 CentOS-CR.repo
-rw-r--r--. 1 root root  649 Nov 23 13:16 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  314 Nov 23 13:16 CentOS-fasttrack.repo
-rw-r--r--. 1 root root  630 Nov 23 13:16 CentOS-Media.repo
-rw-r--r--. 1 root root  916 Jan  8 07:10 CentOS-SCLo-scl.repo
-rw-r--r--. 1 root root  898 Jan  8 07:10 CentOS-SCLo-scl-rh.repo
-rw-r--r--. 1 root root 1331 Nov 23 13:16 CentOS-Sources.repo
-rw-r--r--. 1 root root 5701 Nov 23 13:16 CentOS-Vault.repo
-rw-r--r--. 1 root root  664 Jan  8 07:10 epel.repo
-rw-r--r--. 1 root root  951 Oct  2  2017 epel.repo.rpmnew
-rw-r--r--. 1 root root 1050 Oct  2  2017 epel-testing.repo
-rw-r--r--. 1 root root  155 Jan  8 07:10 mirrors.aliyun.com_epel_7_x86_64_.repo
-rw-r--r--. 1 root root  153 Jan  8 07:10 mirrors.aliyun.com_epel_x86_64_.repo

• 安装git

yum install git

You have enabled checking of packages via GPG keys. This is a good thing.
However, you do not have any GPG public keys installed. You need to download
the keys for packages you wish to install and install them.
You can do that by running the command:
    rpm --import public.gpg.key

从别的机器拷贝/etc/ceph/rpm-gpg 过来,然后执行
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

• 下载12.2.10源码
  git clone -b v12.2.10 --single-branch git://github.com/ceph/ceph.git master

• 更新子模块代码
  git submodule update —init —recursive

下面两步如果环境已经有了切版本合适就可以跳过

• 安装gcc

git clone [https://github.com/gcc-mirror/gcc.git](https://github.com/gcc-mirror/gcc.git)

b.mkdir build ; cd build
../configure --prefix=/usr --disable-multilib 
c.yum install -y gmp-devel libmpc-devel mpfr-devel flex
d.make -j16 > log 2>&1 && make install >log 2>&1 &

• 升级cmake

wget https://cmake.org/files/v3.9/cmake-3.9.2.tar.gz
//解压后自行安装

编译

• 执行install-deps.sh来安装编译时需要的各种依赖包
  sh install-deps.sh

Note:
1.这一步一般要耗费较长时间，所以我们可以执行这一步的时候打开yum的缓存模式，把依赖包缓存在本地，随后把这些rpm包建立一个自己的yum源，这样下次安装的时候就从本地的yum源安装，会大大缩减时间!

[root@xt-53 ceph-12.2.10]# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=1  //置为1，打开缓存模式
debuglevel=2
...

缓存的包会保留在/var/cache/yum/x86_64/7/ 下面
2.如果是容器内编译也可以把安装完依赖包的容器提交到新的镜像，之后用这个镜像启动的容器就不用再安装依赖包了。
3.install-deps.sh带的默认的yum源比较慢，可以更改更快的yum源
install-deps.sh中

$SUDO yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/$VERSION_ID/x86_64/
更改为
$SUDO yum-config-manager --add-repo https://mirrors.aliyun.com/epel/$VERSION_ID/x86_64/

• 执行do_cmake.sh
  sh do_cmake.sh

• 开始make
  make -j16 > log 2>&1 & //线程数等于cpu core的2倍，可以提高编译的速度
  当然我们也可以只编译某一个模块,比如只编译client就可以
  make client > log 2>&1 &

最小测试用例

 当我们更改了代码准备提交到公司内部repo或者社区repo都需要先执行一下最小测试集，看看自己修改的代码有没有影响到别的模块(社区也会进行同样的测试)。

• ctest
  可以单独跑一个测试用例，若不加参数就跑所有的测试，和make一样，可以用-j选项来多线程同时跑。
  例如：

# ctest -R cephfs
Test project /home/ivan/ws/Luminous/code/ceph-12.2.10/build
    Start 42: unittest_libcephfs_config
1/1 Test #42: unittest_libcephfs_config ........   Passed    0.02 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.06 sec

 也可以用-V选项来使运行过程打印到前台

# ctest -V -R client
UpdateCTestConfiguration  from :/home/ivan/ws/Luminous/code/ceph-12.2.10/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/ivan/ws/Luminous/code/ceph-12.2.10/build/DartConfiguration.tcl
Test project /home/ivan/ws/Luminous/code/ceph-12.2.10/build
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 138
    Start 138: unittest_mclock_client_queue

138: Test command: /home/ivan/ws/Luminous/code/ceph-12.2.10/build/bin/unittest_mclock_client_queue
138: Test timeout computed to be: 3600
138: [==========] Running 4 tests from 1 test case.
138: [----------] Global test environment set-up.
138: [----------] 4 tests from MClockClientQueueTest
138: [ RUN      ] MClockClientQueueTest.TestSize
138: [       OK ] MClockClientQueueTest.TestSize (0 ms)
138: [ RUN      ] MClockClientQueueTest.TestEnqueue
138: [       OK ] MClockClientQueueTest.TestEnqueue (0 ms)
138: [ RUN      ] MClockClientQueueTest.TestEnqueueStrict
138: [       OK ] MClockClientQueueTest.TestEnqueueStrict (0 ms)
138: [ RUN      ] MClockClientQueueTest.TestRemoveByClass
138: [       OK ] MClockClientQueueTest.TestRemoveByClass (0 ms)
138: [----------] 4 tests from MClockClientQueueTest (1 ms total)
138:
138: [----------] Global test environment tear-down
138: [==========] 4 tests from 1 test case ran. (1 ms total)
138: [  PASSED  ] 4 tests.
1/1 Test #138: unittest_mclock_client_queue .....   Passed    0.04 sec

The following tests passed:
    unittest_mclock_client_queue

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.08 sec

• make check
  编译以及顺便跑所有的测试用例，可以加-j选项来启动多线程。

构建RPM包

 当我们需要发布产品时就需要把各个模块打成rpm包。我把官网上面的环节再加上自己的特殊需求写了个脚本来实现一键打包功能。
 我们首先需要更改make-disk，该脚本主要用于生成ceph.spec和ceph-*.tar.bz2文件以供后面打rpm包使用。我们主要改变了rpm_version,rpm_release ，是否下载boost库等.

# cat make-dist
#!/bin/sh -e

DOWNLOAD_BOOST=$1

if [ ! -d .git ]; then
    echo "no .git present.  run this from the base dir of the git checkout."
    exit 1
fi

##############################################################
#
# Version is very import, we add a Version.txt to record it 
# There should use own Version.txt instead git tag
#
##############################################################
version=`cat Version.txt | grep VERSION |awk -F '=' '{print $2}'`

[ -z "$version" ] && version=`git describe --match 'v*' | sed 's/^v//'`
outfile="ceph-$version"

echo "version $version"

# update submodules
echo "updating submodules..."
force=$(if git submodule usage 2>&1 | grep --quiet 'update.*--force'; then echo --force ; fi)
if ! git submodule sync || ! git submodule update $force --init --recursive; then
    echo "Error: could not initialize submodule projects"
    echo "  Network connectivity might be required."
    exit 1
fi

download_boost() {
    boost_version=$1
    shift
    boost_md5=$1
    shift
    boost_version_underscore=$(echo $boost_version | sed 's/\./_/g')
    boost_fname=boost_${boost_version_underscore}.tar.bz2
    set +e
    if [ "${DOWNLOAD_BOOST}" == "yes" ];then
        while true; do
            url_base=$1
            shift
            if [ -z $url_base ]; then
                echo "Error: failed to download boost."
                exit
            fi
            url=$url_base/$boost_fname

            wget -c --no-verbose -O $boost_fname $url
            if [ $? != 0 -o ! -e $boost_fname ]; then
                echo "Download of $url failed"
            elif [ $(md5sum $boost_fname | awk '{print $1}') != $boost_md5 ]; then
                echo "Error: failed to download boost: MD5 mismatch."
            else
                break
            fi
        done
    fi
    set -e
    tar xjf $boost_fname -C src \
        --exclude="$boost_version_underscore/libs/*/doc" \
        --exclude="$boost_version_underscore/libs/*/example" \
        --exclude="$boost_version_underscore/libs/*/examples" \
        --exclude="$boost_version_underscore/libs/*/meta" \
        --exclude="$boost_version_underscore/libs/*/test" \
        --exclude="$boost_version_underscore/tools/boostbook" \
        --exclude="$boost_version_underscore/tools/quickbook" \
        --exclude="$boost_version_underscore/tools/auto_index" \
        --exclude='doc' --exclude='more' --exclude='status'
    mv src/boost_${boost_version_underscore} src/boost
    tar cf ${outfile}.boost.tar ${outfile}/src/boost
    rm -rf src/boost
}

# clean out old cruft...
echo "cleanup..."
rm -f $outfile*

# build new tarball
echo "building tarball..."
bin/git-archive-all.sh --prefix ceph-$version/ \
               --verbose \
               --ignore corpus \
               $outfile.tar

# populate files with version strings
echo "including src/.git_version, ceph.spec"

# The command finds the most recent tag that is reachable from a commit. If the tag points to the commit, 
# then only the tag is shown. Otherwise, it suffixes the tag name with the number of additional commits on
# top of the tagged object and the abbreviated object name of the most recent commit. The result is a 
# human-readable" object name which can also be used to identify the commit to other git commands.
(git rev-parse HEAD ; git describe) 2> /dev/null > src/.git_version

# if the version has '-' in it, it has a 'release' part,
# like vX.Y.Z-N-g.  If it doesn't, it's just
# vX.Y.Z.  Handle both, and translate - to . for rpm
# naming rules (the - separates version and release).

if expr index $version '-' > /dev/null; then
    rpm_version=`echo $version | cut -d - -f 1-1`
    rpm_release=`echo $version | cut -d - -f 2- | sed 's/-/./'`
else
    rpm_version=$version
        # rpm style: ceph-"module"-"version"-"commit_id"
        # example: ceph-mds-12.2.10.mh.0-1.ga200519.el7.x86_64.rpm
    rpm_release=`tail -n 1 src/.git_version | cut -d - -f 2- | sed 's/-/./'`
fi

for spec in ceph.spec.in alpine/APKBUILD.in; do
    cat $spec |
        sed "s/@VERSION@/$rpm_version/g" |
        sed "s/@RPM_RELEASE@/$rpm_release/g" |
        sed "s/@TARBALL_BASENAME@/ceph-$version/g" > `echo $spec | sed 's/.in$//'`
done
ln -s . $outfile
tar cvf $outfile.version.tar $outfile/src/.git_version $outfile/ceph.spec $outfile/alpine/APKBUILD
# NOTE: If you change this version number make sure the package is available
# at the three URLs referenced below (may involve uploading to download.ceph.com)
boost_version=1.66.0
    download_boost $boost_version b2dfbd6c717be4a7bb2d88018eaccf75 \
    https://dl.bintray.com/boostorg/release/$boost_version/source \
    https://downloads.sourceforge.net/project/boost/boost/$boost_version \
    https://download.ceph.com/qa
tar --concatenate -f $outfile.all.tar $outfile.version.tar
tar --concatenate -f $outfile.all.tar $outfile.boost.tar
tar --concatenate -f $outfile.all.tar $outfile.tar
mv $outfile.all.tar $outfile.tar
rm $outfile
rm -f $outfile.version.tar
rm -f $outfile.boost.tar

echo "compressing..."
bzip2 -9 $outfile.tar

echo "done."

脚本make-rpm.sh如下：

# cat make-rpm.sh
#!/bin/bash

#echo $@

#######################################################
#
# build Ceph rpm packages
#
# ./make-rpm.sh
#
########################################################

CLEANBUILD="yes"
INSTALL_DEPS="false"
COMMAND=$1
DOWNLOAD_BOOST="false"

# current dir of this script
BASEDIR=`cd $(dirname $0); pwd -P`

function usage()
{
cat < [Flags]

Build ceph and generate rpm packages.

Flags:
    -i,--install-deps      //don't excute install-deps.sh
    -n,--no-cleanbuild     //don't clean build
    --download_boost       // download boost library

    -h,--help              //list help info
EOF
}

function build_rpms()
{
    # we don't need install deps every time
    if [  "$INSTALL_DEPS" == "yes" ];then
        echo "sh ./install-deps.sh"
        #sh ./install-deps.sh
    fi

    if [ "$MODULE" != "" ];then
        echo "compile $MODULE"
    else
        echo "compile all modules"
    fi
    # we only need download boot at first time
    if [ "$DOWNLOAD_BOOST" == "yes" ];then
        echo "make-dist yes"
        ./make-dist yes
    else
        echo "make-dist false"
        ./make-dist false
    fi
    ###########################################
    #
    # Version is very import, we add a Version.txt to record it 
    #
    ###########################################
    version=`cat Version.txt | grep VERSION |awk -F '=' '{print $2}'`

    rm -rf ${BASEDIR}/rpmbuild
    mkdir ${BASEDIR}/rpmbuild && cd ${BASEDIR}/rpmbuild && mkdir BUILD  BUILDROOT  RPMS  SOURCES  SPECS  SRPMS
    cp ${BASEDIR}/ceph-${version}.tar.bz2 ${BASEDIR}/rpmbuild/SOURCES
    tar --strip-components=1 -C ${BASEDIR}/rpmbuild/SPECS/ --no-anchored -xvjf ${BASEDIR}/rpmbuild/SOURCES/ceph-${version}.tar.bz2 "ceph.spec"

    if [ "$CLEANBUILD" == "yes" ];then
        #rpmbuild --define="_topdir "${BASEDIR}"" -ba ${BASEDIR}/rpmbuild/SPECS/ceph.spec --with clean_build
        rpmbuild --define="_topdir "${BASEDIR}"/rpmbuild" -ba ./SPECS/ceph.spec --with clean_build
    else
        #rpmbuild --define '_topdir ./rpmbuild' -ba ${BASEDIR}/rpmbuild/SPECS/ceph.spec
        rpmbuild --define="_topdir "${BASEDIR}"/rpmbuild" -ba ./SPECS/ceph.spec
    fi

    mv ${BASEDIR}/rpmbuild/RPMS/*/* ${BASEDIR}/rpmbuild/
    mv ${BASEDIR}rpmbuild/SRPMS/* ${BASEDIR}/rpmbuild/
}

function make_clean()
{
    echo "clean last build result ..."
    rm -rf rpmbuild
}

function do_work()
{
   if [ "$COMMAND" == "build" ];then
       echo "start building ..."
       build_rpms
   else
       echo "start cleaning"
       make_clean

   fi

}
ARGS=`getopt -o inmc --long install-deps,no-cleanbuild,module:,download_boost,clean -n 'example.sh' -- "$@"`
eval set -- "${ARGS}"

while true
do
    case "$1" in
        -i|--install-deps)
            INSTALL_DEPS="yes"
            echo INSTALL_DEPS $INSTALL_DEPS
            shift
            ;;
        -n|--no-cleanbuild)
            CLEANBUILD="false";
            echo CLEANBULD: $CLEANBUILD;
            shift
            ;;
        --download_boost)
            DOWNLOAD_BOOST="yes";
            echo DOWNLOAD_BOOST $DOWNLOAD_BOOST;
            shift
            ;;
        -c|--clean)
            COMMAND="clean";
            echo COMMAND: $COMMAND;
            shift
            ;;
        --)
            shift;
            break;;
        *) echo "Internal error!;"; exit 1 ;;
    esac
done

if [ $# -lt 1 ];then
    usage
    exit -1
fi

do_work

• 脚本使用方法

# sh make-rpm.sh -h
Usage: ./make-rpm.sh  [Flags]

Build ceph and generate rpm packages.

Flags:
    -i,--install-deps      //don't excute install-deps.sh
    -n,--no-cleanbuild     //don't clean build
    --download_boost       // download boost library

    -h,--help              //list help info

End.

• build rpm 包

// 开始打包,不安装依赖包，且不下载boost库
sh make-rpm.sh build

• 清除掉上次编译的残留文件

sh make-rpm.sh clean
或者
sh make-rpm.sh -c