How to test Heat (by quqi99)

 

作者：张华  发表于：2015-12-19
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

 

(http://blog.csdn.net/quqi99 )

Heat根据配置文件模板(HOT, heat orchestration template)实例化一组符合要求的虚机。也能够在其上对应用软件进行配置与编排。对支持对一个组件部署后的负载均衡进行编排。
Heat 服务包含以下重要的组件：

•     Heat-api组件实现OpenStack天然支持的REST API。该组件通过把API请求经由AMQP传送给Heat engine来处理API请求。
•     Heat-api-cfn组件提供兼容AWS CloudFormation的API，同时也会把API请求通过AMQP转发给heat engine。
•     Heat-engine组件提供Heat最主要的协作功能。

Setting up test environment

  heat:  
    charm: cs:~openstack-charmers-next/heat
    constraints: mem=1G
    num_units: 3
    options:
      vip: 10.5.100.20
  heat-hacluster:
    charm: cs:~openstack-charmers-next/hacluster
    options:
      debug: True

relations:
  - [ heat, mysql ]  
  - [ heat, keystone ]
  - [ heat, rabbitmq-server ]
  - [ heat, heat-hacluster ]

or

./generate-bundle.sh -s bionic -r stein --num-compute 2 --heat
juju deploy ./b/openstack.yaml --overlay ./b/o/neutron-gateway.yaml --overlay ./b/o/heat.yaml

neutron net-list && neutron router-list
nova boot --image trusty --nic net-id=98e10e32-13eb-48ee-b265-4ae0e449b6e5 --flavor 2 i1
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova floating-ip-create
nova floating-ip-associate i1 10.5.150.1

或者使用devstack部署，localrc文件如下

 

#OFFLINE=True
DEST=/home/demo
IPSEC_PACKAGE=strongswan
sudo route del -net 10.0.1.0/24 gw 192.168.101.3
sudo apt-get install openvswitch-switch qemu-kvm libvirt-bin
sudo ovs-vsctl -- --may-exist add-br br-phy
sleep 5

sudo ip addr add 172.16.1.1/24 dev br-phy
#sudo ovs-vsctl -- --may-exist add-port br-phy eth0 -- set interface eth0 type=internal
ENABLED_SERVICES=rabbit,mysql,key,g-api,g-reg
ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch
#ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,neutron,q-lbaas,q-fwaas,q-

Q_ML2_TENANT_NETWORK_TYPE=flat,vlan,gre,vxlan
OVS_BRIDGE_MAPPINGS=physnet1:br-phy

#ENABLED_SERVICES+=,horizon
#ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
VOLUME_BACKING_FILE_SIZE=500M
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=/home/demo/data/swift

HOST_IP=172.16.1.1
SERVICE_HOST=172.16.1.1
MYSQL_HOST=172.16.1.1
RABBIT_HOST=172.16.1.1
GLANCE_HOSTPORT=172.16.1.1:9292
Q_HOST=172.16.1.1

FIXED_RANGE=10.0.1.0/24
FLOATING_RANGE=192.168.101.0/24
Q_FLOATING_ALLOCATION_POOL=start=192.168.101.3,end=192.168.101.100
PUBLIC_NETWORK_GATEWAY=192.168.101.1
NETWORK_GATEWAY=10.0.1.1
PUBLIC_BRIDGE=br-ex
# sudo ovs-vsctl add-port br-ex eth1
OVS_PHYSICAL_BRIDGE=br-phy

DATABASE_USER=root
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_TOKEN=ADMIN
LOGFILE=/home/demo/logs/stack.log
ENABLE_DEBUG_LOG_LEVEL=False
SYSLOG=False
SCREEN_LOGDIR=/home/demo/logs
LOG_COLOR=False
Q_USE_DEBUG_COMMAND=False
APACHE_ENABLED_SERVICES+=keystone
KEYSTONE_TOKEN_FORMAT=UUID
USE_SSL=False
disable_service tls-proxy

ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
CIRROS_VERSION=0.3.4ENABLED_SERVICES+=,heat,h-api,h-api-cfn,h-api-cw,h-eng
CIRROS_VERSION=0.3.4

 

 

 

 

 

使用Heat进行部署

 

heat_template_version: 2013-05-23

description: HOT template for two interconnected VMs with floating ips.

parameters:
  image_id:
    type: string
    description: Image Name
 
  secgroup_id:
    type: string
    description : Id of the security groupe

  public_net:
    type: string
    description: public network id

resources:
  private_net:
    type: OS::Neutron::Net
    properties:
      name: private-net
     
  private_subnet:
    type: OS::Neutron::Subnet
    properties:
      network_id: { get_resource: private_net }
      cidr: 172.16.2.0/24
      gateway_ip: 172.16.2.1
     
  router1:
    type: OS::Neutron::Router
    properties:
      external_gateway_info:
        network: { get_param: public_net }
     
  router1_interface:
    type: OS::Neutron::RouterInterface
    properties:
      router_id: { get_resource: router1 }
      subnet_id: { get_resource: private_subnet }

  server1_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_resource: private_net }
      security_groups: [ get_param: secgroup_id ]
      fixed_ips:
        - subnet_id: { get_resource: private_subnet }
 
  server1_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network_id: { get_param: public_net }
      port_id: { get_resource: server1_port }

  server1:
    type: OS::Nova::Server
    properties:
      name: Server1
      image: { get_param: image_id }
      flavor: m1.tiny
      networks:
        - port: { get_resource: server1_port }
    
  server2_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_resource: private_net }
      security_groups: [ get_param: secgroup_id ]
      fixed_ips:
        - subnet_id: { get_resource: private_subnet }
     
  server2_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network_id: { get_param: public_net }
      port_id: { get_resource: server2_port }
     
  server2:
    type: OS::Nova::Server
    properties:
      name: Server2
      image: { get_param: image_id }
      flavor: m1.tiny
      networks:
        - port: { get_resource: server2_port }
     
outputs:
  server1_private_ip:
    description: Private IP address of server1
    value: { get_attr: [ server1, first_address ] }
  server1_public_ip:
    description: Floating IP address of server1
    value: { get_attr: [ server1_floating_ip, floating_ip_address ] }
  server2_private_ip:
    description: Private IP address of server2
    value: { get_attr: [ server2, first_address ] }
  server2_public_ip:
    description: Floating IP address of server2
    value: { get_attr: [ server2_floating_ip, floating_ip_address ] }

 

 

 

这个ymal文件，大概是需要完成下面的工作

1. 创建一个私有网络
2. 创建一个路由器
3. 连接好外部网络和内部网络
4. 创建两个虚拟机，绑定floating ip

Heat目前支持两种格式的模板，一种是基于JSON格式的CFN模板；另外一种是基于YAML格式的HOT模板。CFN模板主要是为了保持对AWS的兼容性。HOT模板是Heat自有的，资源类型更加丰富，更能体现出Heat特点的模板。一个典型的 HOT 模板由下列元素构成：

• 模板版本：必填字段，指定所对应的模板版本，Heat 会根据版本进行检验。
• 参数列表：选填，指输入参数列表。
• 资源列表：必填，指生成的 Stack 所包含的各种资源。可以定义资源间的依赖关系，比如说生成Port，然后再用port来生成VM。
• 输出列表：选填，指生成的 Stack 暴露出来的信息，可以用来给用户使用，也可以用来作为输入提供给其它的 Stack。

NET_ID=$(nova net-list | awk '/ ext_net / { print $2 }')
SEC_ID=$(nova secgroup-list | awk '/ default / { print $2 }')
heat stack-create -f first-stack.yml -P image_id=cirros -P public_net=$NET_ID -P secgroup_id=$SEC_ID First_Stack

ubuntu@zhhuabj-bastion:~/openstack-charm-testing$ heat stack-list
+--------------------------------------+-------------+-----------------+----------------------+
| id                                   | stack_name  | stack_status    | creation_time        |
+--------------------------------------+-------------+-----------------+----------------------+
| b36bdf67-bd59-401d-ab3b-f7437aa06c30 | First_Stack | CREATE_COMPLETE | 2015-12-19T10:18:49Z |
+--------------------------------------+-------------+-----------------+----------------------+

ubuntu@zhhuabj-bastion:~/openstack-charm-testing$ nova list
+--------------------------------------+---------+--------+------------+-------------+------------------------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks                           |
+--------------------------------------+---------+--------+------------+-------------+------------------------------------+
| 901d5365-01b6-4254-a65f-5177d804d074 | Server1 | ACTIVE | -          | Running     | private-net=172.16.2.3, 10.5.150.4 |
| f1441f2d-4fb2-4560-b0ff-85d0cde4bc45 | Server2 | ACTIVE | -          | Running     | private-net=172.16.2.4, 10.5.150.3 |


ubuntu@juju-zhhuabj-machine-7:~$ ps -ef|grep heat
root      3656     1  0 Dec08 ?        00:05:40 /var/lib/juju/tools/unit-heat-0/jujud unit --data-dir /var/lib/juju --unit-name heat/0 --debug
heat     20395     1  0 Dec08 ?        00:00:10 /usr/bin/python /usr/bin/heat-api --config-file=/etc/heat/heat.conf --log-file=/var/log/heat/heat-api.log
heat     20414     1  0 Dec08 ?        00:00:10 /usr/bin/python /usr/bin/heat-api-cfn --config-file=/etc/heat/heat.conf --log-file=/var/log/heat/heat-api-cfn.log
heat     20437     1  0 Dec08 ?        00:19:54 /usr/bin/python /usr/bin/heat-engine --config-file=/etc/heat/heat.conf --log-file=/var/log/heat/heat-engine.log
 

Heat也能对软件进行配置和部署的编排

Heat 提供了多种资源类型来支持对于软件配置和部署的编排，如下所列：

•     OS::Heat::CloudConfig： VM引导程序启动时的配置，由 OS::Nova::Server 引用
•     OS::Heat::SoftwareConfig：描述软件配置
•     OS::Heat::SoftwareDeployment：执行软件部署
•     OS::Heat::SoftwareDeploymentGroup：对一组 VM 执行软件部署
•     OS::Heat::SoftwareComponent：针对软件的不同生命周期部分，对应描述软件配置
•     OS::Heat::StructuredConfig：和 OS::Heat::SoftwareConfig 类似，但是用 Map 来表述配置
•     OS::Heat::StructuredDeployment：执行 OS::Heat::StructuredConfig 对应的配置
•     OS::Heat::StructuredDeploymentsGroup：对一组 VM 执行 OS::Heat::StructuredConfig 对应的配置

Heat 对负载均衡的编排

负载均衡也是一个很高级应用，它也是由一组不同的资源类型来实现的。资源类型包括：

•     OS::Neutron::Pool：定义资源池，一般可以由 VM 组成
•     OS::Neutron::PoolMember：定义资源池的成员
•     OS::Neutron::HealthMonitor：定义健康监视器，根据自定的协议，比如 TCP 来监控资源的状态，并提供给 OS::Neutron::Pool 来调整请求分发
•     OS::Neutron::LoadBalancer：关联资源池以定义整个负载均衡。

Heat对资源自动伸缩的编排
基础架构的自动伸缩是一个很高级的功能。Heat 提供自动伸缩组 OS::Heat::AutoScalingGroup 和伸缩策略 OS::Heat::ScalingPolicy，结合基于 Ceilometer 的 OS::Ceilometer::Alarm 实现了可以根据各种条件，比如负载，进行资源自动伸缩的功能。

 

如何使用Cirros作为例子镜像

Since cirros images don't currently support multi-part mime user-data, it's necessary to inject the hook script to the image and upload the modified image to glance:

https://github.com/openstack/heat-templates/tree/master/hot/software-config/example-templates/cirros-example
wget http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img
virt-copy-in -a cirros-0.3.2-x86_64-disk.img init.d/heat-deploy-hook /etc/init.d
virt-copy-in -a cirros-0.3.2-x86_64-disk.img rc3.d/S99-heat-deploy-hook /etc/rc3.d
glance image-create --name cirros-0.3.2-sc --disk-format=qcow2 --container-format=bare < cirros-0.3.2-x86_64-disk.img
heat stack-create sc1 -f cirros-hello-world.yaml -P "image=cirros-0.3.2-sc"

或者用cirros 0.3.3：

glance image-create --name "Cirros 0.3.3" --disk-format qcow2 --container-format bare --is-public True --copy http://download.cirros-cloud.net/0.3.3/cirros-0.3.3-x86_64-disk.img
nova flavor-create m1.nano 42 64 0 1

或者在devstack中直接指定cirros的版本

 

CIRROS_VERSION=0.3.4

另一个例子

openstack stack create --parameter vm_name=science --parameter image=cirros --parameter flavor=m1.tiny --parameter net=ext_net -t ./test-stack.yml science-stack
openstack stack list
openstack stack list -f value | awk '{print $2}' | while read STA; do echo "Stack $STA"; openstack stack resource list $STA; openstack stack resource list $STA -f value | awk '{print $1}' | while read RES; do echo "Resource $RES"; openstack stack resource show $STA $RES -f yaml; done; echo;echo;echo; done

[root@laas-pikecoa-course:~]# openstack stack output list science-stack
+------------+--------------+
| output_key | description  |
+------------+--------------+
| private_ip | Allocated IP |
+------------+--------------+
[root@laas-pikecoa-course:~]# openstack stack output show science-stack private_ip
+--------------+--------------+
| Field        | Value        |
+--------------+--------------+
| description  | Allocated IP |
| output_key   | private_ip   |
| output_value | 172.24.4.10  |
+--------------+--------------+
$ openstack stack resource list ec3f904e-7f38-45a2-8619-93be53768b45
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| server_vm     | 29f0961c-90e5-4d02-bf3b-554bc851fdf9 | OS::Nova::Server | CREATE_COMPLETE | 2019-08-16T08:17:53Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
openstack stack resource show ec3f904e-7f38-45a2-8619-93be53768b45 server_vm -f yaml

# cat files/test-stack.yml 
heat_template_version: 2013-05-23

description: >
  This is a heat template that will create a server and attach a volume
parameters:
  vm_name:
    type: string
    label: VM Name
    description: The name to use for the VM
    constraints:
    - length: {min: 2}
      description: VM name must be more then 2 characters, letter and numbers only.
    - allowed_pattern: "[a-zA-Z0-9]+"
      description: VM name must consist of characters and numbers only.
  image:
    type: string
    label: Server Image
    description: |
      Name or UUID of the image to use.  
    default: cirros
  flavor:
    type: string
    label: Server flavor
    description: |
      Name or ID of the flavor to use.  
    default: m1.tiny
  net:
    type: string
    description: |
      Name or ID of the Network to use
  volume:
    type: string
    description: |
      ID of volume to attach to the server

resources:
  server_vm:
    type: OS::Nova::Server
    properties:
      name : {get_param: vm_name}
      image: { get_param: image }
      flavor: { get_param: flavor }
      networks:
      - network: { get_param: net }
      user_data:
        str_replace:
          template: |
            #!/bin/bash
            echo "Hi ${vm_name}"
          params:
            ${vm_password} : {get_param : vm_name}

outputs:
  private_ip:
    description: Allocated IP
    value: {get_attr : [server_vm, first_address]}

Bug

heat创建虚机是并发的, 但dhcp-agent是一个一个处理的, 会产生问题 - https://review.opendev.org/#/c/649580/

当添加sg-logging会看到heat创建虚机更慢. 可以:

1, juju config neutron-api rpc-response-timeout=180 
2, disable anti-affinity check by setting [filter_scheduler]/build_failure_weight_multiplier = 0 
3, disable heartbeat by setting [oslo_messaging_rabbit]heartbeat_timeout_threshold=0
4, or setting heartbeat to 300 by adding  the line '{heartbeat, 300},' in /etc/rabbitmq/rabbitmq.config and [oslo_messaging_rabbit]heartbeat_timeout_threshold=300 in /etc/neutron/neutron.conf 

最终这种慢实际上是由大量的安全组造成的:

Neutron has 3 RPC mechanisms:
1, Plugin RPC, used for messaging beteen neutron -server process and various service agent processes.
2, Callback System, used for in-process communication between core resources and service components. eg: make  service aware of lifecycle events changes for network resource.
3, Messaging Callback System, used for inter-process between core resources and service agents. pls refer [1] for more details.

Topic name format is as follows:
neutron-vo--

There are 10 resources (QosPolicy, Trunk, SubPort, Port, Subnet, Network, SecurityGroup, SecurityGrouprule, Log, PortForwarding) according to - https://github.com/openstack/neutron/blob/stable/stein/neutron/api/rpc/callbacks/resources.py#L38

As for there are multiple queues with the same large number of messages, they seem to be related to SecurityGroup and SecurityGroupRule, not Log:
neutron-vo-SecurityGroupRule-1.0_fanout_d8e73717f6544f22ac38b05cd0adf924	410
neutron-vo-SecurityGroup-1.0_fanout_51ee6075715a4f2d831029def5eb8ead	168

#see the connection num from every clients
tshark -r xxx.pcap |grep AMQP |awk '{arr[$5]++}END{for (a in arr) print a, arr[a]}' |sort -n -k 2 -r 
10.55.12.80 166682 
10.55.12.62 33172 
10.55.12.61 18538 

https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1817877
randomize_allocation_candidates = true

openstack security group list | wc -l 

 

一些命令

openstack stack list -f value | awk '{print $2}' | while read STACK; do echo; echo $STACK; openstack stack resource list $STACK; done

openstack stack list -f value | grep FAILED | awk '{print $2}' | while read STA; do echo "Stack $STA"; openstack stack resource list $STA -f value | grep FAILED | awk '{print $1}' | while read RES; do echo "Resource $RES"; openstack stack resource show $STA $RES; done; done

 

参考

[1] http://docs.openstack.org/developer/heat/template_guide/index.html

[2] http://www.ibm.com/developerworks/cn/cloud/library/1511_zoupx_openstackheat/index.htm

[3] https://github.com/openstack/heat-templates/tree/master/hot/software-config/example-templates/cirros-example