OpenStack HA

一 高可用

最小停机时间

最小的数据丢失

二 可用性级别

三 收益与投入成本

四 如何实现

消除单点（SPOFs）

•  网络冗余，比如：交换机、路由器
•  应用服务冗余和自动服务迁移
•  存储冗余
•  基础设施冗余，比如：电源

五 高可用概念

1 无状态服务（stateless services）

• 请求之间没有依赖
• 比如：Nova API，Nova Scheduler，etc.

2 有状态服务（Stateful services）

• 一次操作需要多个关联请求完成
• 比如：MySQL，Qpid，etc.

3 Active/Passive

• Redundant instances of stateless services are load balanced
• For Stateful services a replacement resource can be brought online.

这种方式是一个节点是主，一个节点是备，主节点挂掉后，备节点起来工作。

4 Active/Active

• Redundant instances of stateless services are load balanced
• Stateful services are managed in such a way that services are redundant, and that all instances have an identical state.
• Updates to one instance of a database would also update all other instances.

这种方式是所有节点都是活动的，没有主备之分，但每个节点的数据库状态必须保持一致，因此开销比较大。

5 Failover

• Migration of a service from the “primary” to the “secondary”

6 Failback

• Migration of service back to the “primary”

六 HA for Openstack

1 Compute HA

• Instance HA

2 Controller HA

• MySQL
• Qpid
• OpenStack APIs (keystone, nova-api etc.)
• Nova, Neutron, Cinder, Swift, and so on

七 Server Evacuation

Without Shared Storage

• The instance will be booted from a new disk, but will preserve the configuration, e.g. id, name, uid, ip...etc.

With Shared Storage

• The instance will be booted from same disk and data will be preserved

八 Instance Migration

九 Application implement

十 HA on Openstack

十一 MySQL HA

• 非常重要的一个组成
• 每个Openstack的组件都用到了数据库
• 需要使用数据库记录相当多的信息

1 Master/Master Replication

2 Pacemaker，Corosync and DRBD

Pacemaker

• high availability and load balancing stack for the Linux platform.
• Interacts with applications through Resource Agents (RA)

Corosync

• Totem single-ring ordering and membership protocol
• UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker.

DRBD (Distributed Replication Block Device)

• Synchronizes Data at the block device
• Uses a journaling system (such as ext3 or ext4)

3 Galera

Synchronous multi-master cluster technology for MySQL/InnoDB

• MySQL patched for wsrep (Write Set REPlication)
• Active/active multi-master topology
• Read and write to any cluster node
• True parallel replication, in row level
• No slave lag or integrity issues

4 比较

十二 Qpid HA

• Pacemaker managed without clustering
• Clustered without pacemaker
• Pacemaker managed with clustering

十三 Qpid和Corosync的工作模式

1 Client连接一个Broker

2 Client failover

十四 Openstack APIs HA

• keystone
• glance-api
• nova-api
• cinder-api
• neutron-api
• Swift-proxy

一般都是用HA with keepalived and HAProxy

十五 Keepalived，HAProxy and VRRP

HAProxy

• Load Balancing and Proxying for HTTP and TCP Applications
• Works over multiple connections
• Used to load balance API services

VRRP (Virtual Router Redundancy Protocol)

• Eliminates SPOF in a static default routed environment

Keepalived

• Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing
• Implements a set of checkers to check service status and to maintain health
• Leverage the VRRP Protocol to remap VIPS in event of failure

十六 Openstack multi-host service

• Nova-schedule
• Nova-compute
• Cinder-schedule

十七 keystone的HA

1 keystone是整个openstack的关键。

2 负责用户管理和权限控制。

3 Openstack每个组件都和keystone有交互。

4 单个keystone压力比较大。

5 单点

6 不同部署

十八 Glance的HA

1 Glance负责整个Openstack的image管理，上传和下载操作多，对后端压力大。

2 Image文件丢失与损坏

3 Glance API接受所有的外部请求，压力比较大

4 Glance的HA部署

十九 Swift，cinder以及Neutron的HA

1 Swift本身就自带HA

2 只需要对多个Proxy-Server做负责均衡即可。

3 Neutron HA功能暂时还不是特别完善。

4 Swift的HA部署

二十 Nova的HA

1 Nova支持MutiHost模式部署

2 可以做HA的包括nova-api和nova-scheduler，其中，nova-scheduler本身支持HA

3 MutilHost模式部署

4 对Nova-api进行负载

5 Nova的HA部署

二十一 Neutron HA

Active/active

• dhcp-agent / openvswitch-agent/neutron-server

support active/ passive

• L3-agent and metadata-agent

二十二 Sample Openstack HA Architecture