Kolla-Ansible部署Queens openstack Multinode 环境

Kolla-Ansible部署Queens openstack Multinode 环境
1、准备环境：
• 3个物理机，controller一个，compute151，compute152
• 至少2个网络接口
• 至少8GB主内存、40GB磁盘空间（虚机），x86服务器：32G，1T
• 操作系统：CentOS7 3.10.0-957.1.3.el7.x86_64

2、部署
2.1系统服务配置
启动ntp服务
分别在控制节点，计算节点上执行
systemctl enable ntpd.service && systemctl start ntpd.service && systemctl status ntpd.service
关闭libvirt服务
systemctl stop libvirtd.service && systemctl disable libvirtd.service && systemctl status libvirtd.service
关闭防火墙服务
systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld
修改hosts
3个节点的hosts内容保持一致
vi /etc/hosts 添加
10.10.0.10 controller
10.10.0.11 compute11
10.10.0.12 compute12
配置免密登录
分别在3个节点执行
ssh-keygen
ssh-copy-id -i /root/.ssh/id_rsa.pub root@controller
ssh-copy-id -i /root/.ssh/id_rsa.pub root@compute12
ssh-copy-id -i /root/.ssh/id_rsa.pub root@compute11
安装docker
分别在3个节点执行
从阿里云下载docker的repo文件：
wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
安装docker-ce
yum install -y docker-ce
配置国内镜像
控制节点执行
mkdir -p /etc/docker
vim /etc/docker/daemon.json
{
“registry-mirrors”: [“https://7g5a4z30.mirror.aliyuncs.com”]
}
重启docker服务：systemctl daemon-reload && systemctl enable docker && systemctl restart docker
配置Docker共享挂载
控制节点执行
mkdir -p /etc/systemd/system/docker.service.d
vim /etc/systemd/system/docker.service.d/kolla.conf
[Service]
MountFlags=shared
重启docker服务：systemctl daemon-reload && systemctl enable docker && systemctl restart docker
检查镜像站点配置是否正确
docker pull hello-world
2.2安装依赖软件
安装pip并更新
控制节点执行
yum install epel-release -y
yum install python-pip -y
pip install -U pip
修改pip源
控制节点执行
mkdir ~/.pip
vim ~/.pip/pip.conf
[global]
trusted-host = pypi.douban.com
index-url = http://pypi.douban.com/simple
安装其他依赖包
控制节点执行
yum install python-devel libffi-devel gcc openssl-devel libselinux-python -y
2.3安装配置ansible
控制节点安装：先使用pip安装再使用yum安装，可以防止某些py包版本太低
pip install ansible
yum install ansible -y
在/etc/ansible/ansible.cfg配置文件中添加以下内容：
[defaults]
host_key_checking=False
pipelining=True
forks=100
2.4安装配置kolla-ansible
控制节点安装：使用pip安装kolla-ansible
pip install kolla-ansible
Tips:
（安装此步，会有有些包冲突和版本不兼容问题，可提前卸载或者删除）
删除PyYAML ：rm -rf /usr/lib64/python2.7/site-packages/PyYAML*
也有可能会有其他的包如ipaddress,idna,requests等，均可通过该方法删除
)
复制global.yml和password.yml文件到/etc/kolla目录:
cp -r /usr/share/kolla-ansible/etc_examples/kolla /etc/kolla/
复制all-in-one 和multinode 文件到当前操作目录：
cp /usr/share/kolla-ansible/ansible/inventory/* .

拉取镜像，需要在线下载，文件较多，需要1个小时左右时间
kolla-ansible pull -vvv
Tips:（此操作步骤可省略，已经下载完成，直接用docker load到本地docker中即可）

2.5上传镜像到本地registry仓库：
在控制节点执行
启动registry容器，并将端口映射到4000端口：
docker run -d --name registry --restart=always -p 4000:5000 -v /opt/registry:/var/lib/registry registry:2.6.2
Tips:（registry 的版本需要是2.3版本以上的，官方文档说2.3版本的registry有很多bug。）
修改Docker服务配置，信任本地Registry服务
vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --insecure-registry controller:4000
重新启动docker服务
systemctl daemon-reload && systemctl restart docker
测试registry服务是否正常：
curl -X GET http://kolla:4000/v2/_catalog
{“repositories”:[]}
修改镜像的tag

for i in `docker images|grep -v registry|grep -v R|awk '{print $1}'`;do docker image tag $i:queens controller:4000/$i:queens;done

push到本地库

for i in `docker images|grep controller:4000|awk '{print $1}'`;do docker push $i:queens;done

查看镜像是否上传成功
curl -XGET http://controller:4000/v2/_catalog
{“repositories”:[“cron”,“kolla/centos-source-chrony”,“kolla/centos-source-cron”,“kolla/centos-source-fluentd”,“kolla/centos-source-glance-api”,“kolla/centos-source-haproxy”,“kolla/centos-source-heat-api”,“kolla/centos-source-heat-api-cfn”,“kolla/centos-source-heat-engine”,“kolla/centos-source-horizon”,“kolla/centos-source-keepalived”,“kolla/centos-source-keystone”,“kolla/centos-source-keystone-fernet”,“kolla/centos-source-keystone-ssh”,“kolla/centos-source-kolla-toolbox”,“kolla/centos-source-mariadb”,“kolla/centos-source-memcached”,“kolla/centos-source-neutron-dhcp-agent”,“kolla/centos-source-neutron-l3-agent”,“kolla/centos-source-neutron-metadata-agent”,“kolla/centos-source-neutron-openvswitch-agent”,“kolla/centos-source-neutron-server”,“kolla/centos-source-nova-api”,“kolla/centos-source-nova-compute”,“kolla/centos-source-nova-conductor”,“kolla/centos-source-nova-consoleauth”,“kolla/centos-source-nova-libvirt”,“kolla/centos-source-nova-novncproxy”,“kolla/centos-source-nova-placement-api”,“kolla/centos-source-nova-scheduler”,“kolla/centos-source-nova-ssh”,“kolla/centos-source-openvswitch-db-server”,“kolla/centos-source-openvswitch-vswitchd”,“kolla/centos-source-rabbitmq”]}

2.6修改部署配置文件
修改global.yml配置文件：
示例文件如下做参考：
kolla_base_distro: “centos”
kolla_install_type: “source”
openstack_release: “queens”
node_custom_config: “/etc/kolla/config”
kolla_internal_vip_address: “10.10.0.18”
docker_registry: "controller:4000
docker_namespace: “kolla”
network_interface: “eth0”
api_interface: “{{ network_interface }}”
storage_interface: “{{ network_interface }}”
neutron_external_interface: “eth2”
enable_ceph: “no”
enable_ceph_rgw_keystone: “yes”
glance_enable_rolling_upgrade: “no”
nova_compute_virt_type: “kvm”
ironic_dnsmasq_dhcp_range:
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:

修改部署配置文件muitinode
示例文件如下做参考：
[control]
controller
[network]
controller
[inner-compute]
[external-compute]
compute11
compute12
[compute:children]
inner-compute
external-compute
[monitoring]
controller
[storage]
compute11
compute12

生成随机密码文件
kolla-genpwd

vim /etc/kolla/passwords.yml
keepalived_password: mFbTVxF6XyrrT8NqaN5UpFB098GEXuZ9oQyfQI14 keystone_admin_password: admin123 # 更改此处
keystone_database_password: C4EzIx0zhoFjsG9dA9TBRaZfbFIdT3f9sCe7jGyg
其他的地方默认不用修改，也可以手动修改。

2.7开始部署multinode
带有kolla的引导服务器部署依赖关系
kolla-ansible -i ./multinode bootstrap-servers
对主机执行预部署检查
kolla-ansible -i ./multinode prechecks
执行OpenStack部署
kolla-ansible -i ./multinode deploy
以上操作过程如果没有报错，则表示部署完成，执行如下，生成admin用户的凭证
kolla-ansible post-deploy
3、完成部署后，使用OpenStack
3.1安装基本的OpenStack CLI客户端
pip install python-openstackclient python-glanceclient python-neutronclient
3.2运行脚本创建示例网络，镜像，实例等（也可以不执行初始化，登录进openstack后手动进行初始化操作）
修改 /usr/share/kolla-ansible/init-runonce
示例参考配置如下：

3.3登录openstack
打开浏览器输入10.10.0.18，用户admin ，密码 admin123 可登录openstack_dashboard，如图所示：

3.4 创建openstack 测试实例

4、销毁重装/卸载openstack

假如不需要这个openstack环境了，在控制节点可以执行如下命令进行销毁该环境：
kolla-ansible destroy -i ./multinode --yes-i-really-really-mean-i
Tips：删除前，最好把计算节点的instance实例和相关的网络都删除
TroubleShooting

问题描述1 配置问题

TASK [baremetal : Generate /etc/hosts for all of the nodes] ********************************************************************************************************************************* fatal: [4.0.0.11]: FAILED! => {“failed”: true, “msg”: “the field ‘args’ has an invalid value, which appears to include a variable that is undefined. The error was: ‘dict object’ has no attribute u’ansible_ens3’\n\nThe error appears to have been in ‘/usr/share/kolla-ansible/ansible/roles/baremetal/tasks/pre-install.yml’: line 40, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Generate /etc/hosts for all of the nodes\n ^ here\n”} to retry, use: --limit @/usr/share/kolla-ansible/ansible/kolla-host.retry

解决办法 – 查询下是不是有节点安装python失败，导致没有执行Gather facts操作造成失败 - 查询下是不是有节点的网卡名不对

问题描述2 兼容性问题

初始化OpenStack时失败

解决办法–需要删除ipadress，request, idna等包并重新安装（版本过低下一步客户端安装会出错，原先安装其他包的时候作为依赖包安装的ipaddress无法通过pip删除并升级，只能手动删除再安装最新版本）

cd /usr/lib/python2.7/site-packages/
rm -rf ipaddress*
pip install ipaddress

问题描述3 镜像不存在

TASK [ceph : Generating initial Ceph keyrings and monmap] *************************************************************************************************************************
fatal: [controller]: FAILED! => {“changed”: true, “msg”: “‘Traceback (most recent call last):\n File “/tmp/ansible_kolla_docker_payload_oEIl8B/main.py”, line 881, in main\n result = bool(getattr(dw, module.params.get(\‘action\’))())\n File “/tmp/ansible_kolla_docker_payload_oEIl8B/main.py”, line 672, in start_container\n self.pull_image()\n File “/tmp/ansible_kolla_docker_payload_oEIl8B/main.py”, line 513, in pull_image\n repository=image, tag=tag, stream=True\n File “/usr/lib/python2.7/site-packages/docker/api/image.py”, line 400, in pull\n self._raise_for_status(response)\n File “/usr/lib/python2.7/site-packages/docker/api/client.py”, line 258, in _raise_for_status\n raise create_api_error_from_http_exception(e)\n File “/usr/lib/python2.7/site-packages/docker/errors.py”, line 31, in create_api_error_from_http_exception\n raise cls(e, response=response, explanation=explanation)\nNotFound: 404 Client Error: Not Found (“manifest for controller:4000/kolla/centos-source-ceph-mon:queens not found”)\n’”}

解决办法 –没有找到对应的ceph的镜像需要重新下载镜像，并且push到本地的仓库
拉取镜像 ：docker pull kolla/centos-source-ceph-mon:queens
对镜像重新打tag： docker image tag kolla/centos-source-ceph-mon:queens controller:4000/kolla/centos-source-ceph-mon:queens
把镜像push到本地 仓库：docker push controller:4000/kolla/centos-source-ceph-mon:queens
问题描述4 ceph问题
TASK [ceph : Fetching Ceph keyrings] **********************************************************************************************************************************************
fatal: [controller]: FAILED! => {“msg”: “The conditional check ‘(ceph_files_json.stdout | from_json).changed’ failed. The error was: No JSON object could be decoded”}

解决办法 原因是在删除容器和配置文件后，kolla生成的相关volume是没有删除的。其还存在于/var/lib/docker/volume下。因此当再次构建kolla时，这些已经存在的volume会阻止ceph_mon的启动，会导致上述错误Ceph keyring无法获取而产生的一些错误。因此 删除掉docker volume ls下的卷。再次部署就能够成功的解决问题。
[root@controller ~]# docker volume ls
DRIVER VOLUME NAME
local ceph_mon_config
local haproxy_socket
local keystone_fernet_tokens
local kolla_logs
local mariadb
local rabbitmq
[root@controller ~]# docker volume rm ceph_mon_config

问题描述 mariadb问题
TASK: [mariadb | Creating haproxy mysql user] ********************************* … stdout: localhost | FAILED! => { “changed”: false, “failed”: true, “msg”: “unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1045, “Access denied for user ‘root’@‘mick-workstation’ (using password: YES)”)” } msg: Task failed as maximum retries was encountered
解决办法
docker rm mariadb
rm -rf /var/lib/docker/volumes/mariadb/_data/*
问题描述 首页首次登录问题

解决办法
可以忽略，刷新页面能直接进去
docker-enter 脚本

 #!/bin/sh
 
  if [ -e $(dirname "$0")/nsenter ]; then
    # with boot2docker, nsenter is not in the PATH but it is in the same folder
    NSENTER=$(dirname "$0")/nsenter
  else
    NSENTER=nsenter
  fi
 
  if [ -z "$1" ]; then
    echo "Usage: `basename "$0"` CONTAINER [COMMAND [ARG]...]"
    echo ""
    echo "Enters the Docker CONTAINER and executes the specified COMMAND."
    echo "If COMMAND is not specified, runs an interactive shell in CONTAINER."
  else
    PID=$(docker inspect --format "{{.State.Pid}}" "$1")
    if [ -z "$PID" ]; then
      exit 1
    fi
    shift
 
    OPTS="--target $PID --mount --uts --ipc --net --pid --"
 
    if [ -z "$1" ]; then
      # No command given.
      # Use su to clear all host environment variables except for TERM,
      # initialize the environment variables HOME, SHELL, USER, LOGNAME, PATH,
      # and start a login shell.
      "$NSENTER" $OPTS su - root
    else
      # Use env to clear all host environment variables.
      "$NSENTER" $OPTS env --ignore-environment -- "$@"
    fi
  fi