nova boot代码流程分析(五)：VM启动从neutron-dhcp-agent获取IP与MAC
1. network和subnet创建代码流程
| [root@jun ~(keystone_user1)]# neutron net-create demo-net [root@jun ~(keystone_user1)]# neutron subnet-create demo-net 1.1.1.0/24 --name demo-subnet --gateway 1.1.1.1 --enable_dhcp true |
这里，我们主要分析上面两个命令的代码流程，我们关注的重点是neutron-server如何下发的命令到neutron-dhcp-agent去创建相应的network的代码流程。
首先neutronclient发送HTTP请求给neutron-server，去执行下面的create函数。
#/neutron/api/v2/base.py:Controller
def create(self, request, body=None, **kwargs):
"""Creates a new instance of the requested entity."""
parent_id = kwargs.get(self._parent_id_name)
self._notifier.info(request.context,
self._resource + '.create.start',
body)
body = Controller.prepare_request_body(request.context, body, True,
self._resource, self._attr_info,
allow_bulk=self._allow_bulk)
action = self._plugin_handlers[self.CREATE]
... ... ...
def notify(create_result):
notifier_method = self._resource + '.create.end'
self._notifier.info(request.context,
notifier_method,
create_result)
self._send_dhcp_notification(request.context,
create_result,
notifier_method)
return create_result
kwargs = {self._parent_id_name: parent_id} if parent_id else {}
if self._collection in body and self._native_bulk:
# plugin does atomic bulk create operations
obj_creator = getattr(self._plugin, "%s_bulk" % action)
objs = obj_creator(request.context, body, **kwargs)
# Use first element of list to discriminate attributes which
# should be removed because of authZ policies
fields_to_strip = self._exclude_attributes_by_policy(
request.context, objs[0])
return notify({self._collection: [self._filter_attributes(
request.context, obj, fields_to_strip=fields_to_strip)
for obj in objs]})
else:
obj_creator = getattr(self._plugin, action)
if self._collection in body:
# Emulate atomic bulk behavior
objs = self._emulate_bulk_create(obj_creator, request,
body, parent_id)
return notify({self._collection: objs})
else:
kwargs.update({self._resource: body})
obj = obj_creator(request.context, **kwargs)
self._send_nova_notification(action, {},
{self._resource: obj})
return notify({self._resource: self._view(request.context,
obj)})
这个create函数实现的功能为：
1. 通过调用core_plugin的core resource所对应的action方法(如create_network)在数据库中保存resource信息。
2. 在保存自身的信息到数据库中后，通知agent去做相应的操作。
其实对于port，network，subnet的create操作都会走到这个create函数。这里我们主要分析创建network(包括subnet)时与neutron-dhcp-agent之间的操作。
在network(或subnet)的信息保存到数据库之后，通过notify函数(create函数内部的函数)通知neutron-dhcp-agent做相应的操作。
network和subnet发送的给neutron-dhcp-agent的方法(notifier_method)分别为network.create.end和subnet.create.end
#/neutron/api/v2/base.py:Controller
def _send_dhcp_notification(self, context, data, methodname):
if cfg.CONF.dhcp_agent_notification:
if self._collection in data:
for body in data[self._collection]:
item = {self._resource: body}
self._dhcp_agent_notifier.notify(context, item, methodname)
else:
self._dhcp_agent_notifier.notify(context, data, methodname)
dhcp_agent_notification为/etc/neuton/neutron.conf配置文件中的参数。
| # Allow sending resource operation notification to DHCP agent # dhcp_agent_notification = True dhcp_agent_notification = True |
#/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py:DhcpAgentNotifyAPI
def notify(self, context, data, method_name):
# data is {'key' : 'value'} with only one key
if method_name not in self.VALID_METHOD_NAMES:
return
obj_type = data.keys()[0]
if obj_type not in self.VALID_RESOURCES:
return
obj_value = data[obj_type]
network_id = None
if obj_type == 'network' and 'id' in obj_value:
network_id = obj_value['id']
elif obj_type in ['port', 'subnet'] and 'network_id' in obj_value:
network_id = obj_value['network_id']
if not network_id:
return
method_name = method_name.replace(".", "_")
if method_name.endswith("_delete_end"):
if 'id' in obj_value:
self._notify_agents(context, method_name,
{obj_type + '_id': obj_value['id']},
network_id)
else:
self._notify_agents(context, method_name, data, network_id)
#/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py:DhcpAgentNotifyAPI
def _notify_agents(self, context, method, payload, network_id):
"""Notify all the agents that are hosting the network."""
# fanout is required as we do not know who is "listening"
no_agents = not utils.is_extension_supported(
self.plugin, constants.DHCP_AGENT_SCHEDULER_EXT_ALIAS)
fanout_required = method == 'network_delete_end' or no_agents
# we do nothing on network creation because we want to give the
# admin the chance to associate an agent to the network manually
cast_required = method != 'network_create_end'
if fanout_required:
self._fanout_message(context, method, payload)
elif cast_required:
admin_ctx = (context if context.is_admin else context.elevated())
network = self.plugin.get_network(admin_ctx, network_id)
agents = self.plugin.get_dhcp_agents_hosting_networks(
context, [network_id])
# schedule the network first, if needed
schedule_required = (
method == 'subnet_create_end' or
method == 'port_create_end' and
not self._is_reserved_dhcp_port(payload['port']))
if schedule_required:
agents = self._schedule_network(admin_ctx, network, agents)
enabled_agents = self._get_enabled_agents(
context, network, agents, method, payload)
for agent in enabled_agents:
self._cast_message(
context, method, payload, agent.host, agent.topic)
在_notify_agents函数可以看出，由于创建network时，fanout_required和cast_required的值为False，所以创建network时，并不会通过neutron-dhcp-agent做什么操作。而创建port或subnet时，则会通过rpc方式执行neutron-dhcp-agent的方法，从而做一些操作。所以下面我们主要分析在创建subnet时，neutron-dhcp-agent做了哪些操作。
#/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py:DhcpAgentNotifyAPI
def _cast_message(self, context, method, payload, host,
topic=topics.DHCP_AGENT):
"""Cast the payload to the dhcp agent running on the host."""
cctxt = self.client.prepare(topic=topic, server=host)
cctxt.cast(context, method, payload=payload)
#/neutron/agent/dhcp/agent.py:DhcpAgent
@utils.synchronized('dhcp-agent')
def subnet_update_end(self, context, payload):
"""Handle the subnet.update.end notification event."""
network_id = payload['subnet']['network_id']
self.refresh_dhcp_helper(network_id)
# Use the update handler for the subnet create event.
subnet_create_end = subnet_update_end
看到/neutron/agent/dhcp/agent.py:DhcpAgent类是不是很熟悉，这正是上一篇文章中分析的在neutron-dhcp-agent服务启动时，创建的DhcpAgent对象。
#/neutron/agent/dhcp/agent.py:DhcpAgent
def refresh_dhcp_helper(self, network_id):
"""Refresh or disable DHCP for a network depending on the current state
of the network.
"""
old_network = self.cache.get_network_by_id(network_id)
if not old_network:
# DHCP current not running for network.
return self.enable_dhcp_helper(network_id)
network = self.safe_get_network_info(network_id)
if not network:
return
old_cidrs = set(s.cidr for s in old_network.subnets if s.enable_dhcp)
new_cidrs = set(s.cidr for s in network.subnets if s.enable_dhcp)
if new_cidrs and old_cidrs == new_cidrs:
self.call_driver('reload_allocations', network)
self.cache.put(network)
elif new_cidrs:
if self.call_driver('restart', network):
self.cache.put(network)
else:
self.disable_dhcp_helper(network.id)
refresh_dhcp_helper函数将对比self.cache中存放的network信息与新创建的network信息，通过对比结果做相应操作。
1. 如果self.cache中没有新创建的network信息，则调用enable_dhcp_helper函数，该函数最终调用driver(dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq)的enable函数。
2. 如果self.cache中有新创建的network信息，则判断old network与new network的cidr是否相等，根据结果判断调用driver中的reload_allocations函数还是restart函数，又或者是disable函数。
对于执行neutron.agent.linux.dhcp.Dnsmasq driver中的相关函数(enable, reload_allocations, restart,disable)，我们在这里本篇文章就不分析了，具体查看《neutron-dhcp-agent服务启动流程》。
PS：在《neutron-dhcp-agent服务启动流程》文章中，我们也有分析，如果创建的subnet并没有enable dhcp，则neutron-dhcp-agent不会为该network创建namespace，创建device以及创建dnsmasq进程。具体代码如下。
#/neutron/agent/dhcp/agent.py:DhcpAgent
def configure_dhcp_for_network(self, network):
if not network.admin_state_up:
return
enable_metadata = self.dhcp_driver_cls.should_enable_metadata(
self.conf, network)
dhcp_network_enabled = False
for subnet in network.subnets:
if subnet.enable_dhcp:
if self.call_driver('enable', network):
dhcp_network_enabled = True
self.cache.put(network)
break
if enable_metadata and dhcp_network_enabled:
for subnet in network.subnets:
if subnet.ip_version == 4 and subnet.enable_dhcp:
self.enable_isolated_metadata_proxy(network)
break
小结一下：
1. 创建network，subnet和port时，首先调用core_plugin提供的函数向数据库写入自身的信息。
2. 写完数据库之后，subnet和port通过rpc方式，执行neutron-dhcp-agent启动时创建的DhcpAgent对象提供的函数(这里创建network时，并不会执行neutron-dhcp-agent所提供的函数方法)。
3. neutron-dhcp-agent对创建subnet的操作可以参考《neutron-dhcp-agent服务启动流程》文章。
对于neutron-dhcp-agent对创建VM时所创建的port的操作我们在下一小节分析。
2. VM启动时从neutron-dhcp-agent获取IP与MAC的代码流程分析
2.1 更新dnsmasq配置文件信息
接到《nova boot代码流程分析(三)：nova与neutron的plugin交互》的创建port的代码流程分析，因为在那篇文章中，主要关注的如何将创建的IP和MAC保存到数据库，并未关注VM实际如何从neutron-dhcp-agent获取IP和MAC的代码流程，所以在这篇文章中，我们将关注VM实际获取IP和MAC的代码流程。
#/nova/network/neutronv2/api.py:API
def _create_port(self, port_client, instance, network_id, port_req_body,
fixed_ip=None, security_group_ids=None,
available_macs=None, dhcp_opts=None):
"""Attempts to create a port for the instance on the given network.
:param port_client: The client to use to create the port.
:param instance: Create the port for the given instance.
:param network_id: Create the port on the given network.
:param port_req_body: Pre-populated port request. Should have the
device_id, device_owner, and any required neutron extension values.
:param fixed_ip: Optional fixed IP to use from the given network.
:param security_group_ids: Optional list of security group IDs to
apply to the port.
:param available_macs: Optional set of available MAC addresses,
from which one will be used at random.
:param dhcp_opts: Optional DHCP options.
:returns: ID of the created port.
:raises PortLimitExceeded: If neutron fails with an OverQuota error.
:raises NoMoreFixedIps: If neutron fails with
IpAddressGenerationFailure error.
"""
try:
if fixed_ip:
port_req_body['port']['fixed_ips'] = [
{'ip_address': str(fixed_ip)}]
port_req_body['port']['network_id'] = network_id
port_req_body['port']['admin_state_up'] = True
port_req_body['port']['tenant_id'] = instance.project_id
if security_group_ids:
port_req_body['port']['security_groups'] = security_group_ids
if available_macs is not None:
if not available_macs:
raise exception.PortNotFree(
instance=instance.uuid)
mac_address = available_macs.pop()
port_req_body['port']['mac_address'] = mac_address
if dhcp_opts is not None:
port_req_body['port']['extra_dhcp_opts'] = dhcp_opts
port_id = port_client.create_port(port_req_body)['port']['id']
LOG.debug('Successfully created port: %s', port_id,
instance=instance)
return port_id
except neutron_client_exc.IpAddressInUseClient:
LOG.warning(_LW('Neutron error: Fixed IP %s is '
'already in use.'), fixed_ip)
msg = _("Fixed IP %s is already in use.") % fixed_ip
raise exception.FixedIpAlreadyInUse(message=msg)
except neutron_client_exc.OverQuotaClient:
LOG.warning(_LW(
'Neutron error: Port quota exceeded in tenant: %s'),
port_req_body['port']['tenant_id'], instance=instance)
raise exception.PortLimitExceeded()
except neutron_client_exc.IpAddressGenerationFailureClient:
LOG.warning(_LW('Neutron error: No more fixed IPs in network: %s'),
network_id, instance=instance)
raise exception.NoMoreFixedIps(net=network_id)
except neutron_client_exc.MacAddressInUseClient:
LOG.warning(_LW('Neutron error: MAC address %(mac)s is already '
'in use on network %(network)s.') %
{'mac': mac_address, 'network': network_id},
instance=instance)
raise exception.PortInUse(port_id=mac_address)
except neutron_client_exc.NeutronClientException:
with excutils.save_and_reraise_exception():
LOG.exception(_LE('Neutron error creating port on network %s'),
network_id, instance=instance)
这部分代码是从《nova boot代码流程分析(三)：nova与neutron的plugin交互》文章中提取出来的代码，该代码是nova-compute通过HTTP请求在neutron中创建port信息。然后到达我们第一小节分析的create函数。
#/neutron/api/v2/base.py:Controller
def create(self, request, body=None, **kwargs):
"""Creates a new instance of the requested entity."""
parent_id = kwargs.get(self._parent_id_name)
self._notifier.info(request.context,
self._resource + '.create.start',
body)
body = Controller.prepare_request_body(request.context, body, True,
self._resource, self._attr_info,
allow_bulk=self._allow_bulk)
action = self._plugin_handlers[self.CREATE]
... ... ...
def notify(create_result):
notifier_method = self._resource + '.create.end'
self._notifier.info(request.context,
notifier_method,
create_result)
self._send_dhcp_notification(request.context,
create_result,
notifier_method)
return create_result
kwargs = {self._parent_id_name: parent_id} if parent_id else {}
if self._collection in body and self._native_bulk:
# plugin does atomic bulk create operations
obj_creator = getattr(self._plugin, "%s_bulk" % action)
objs = obj_creator(request.context, body, **kwargs)
# Use first element of list to discriminate attributes which
# should be removed because of authZ policies
fields_to_strip = self._exclude_attributes_by_policy(
request.context, objs[0])
return notify({self._collection: [self._filter_attributes(
request.context, obj, fields_to_strip=fields_to_strip)
for obj in objs]})
else:
obj_creator = getattr(self._plugin, action)
if self._collection in body:
# Emulate atomic bulk behavior
objs = self._emulate_bulk_create(obj_creator, request,
body, parent_id)
return notify({self._collection: objs})
else:
kwargs.update({self._resource: body})
obj = obj_creator(request.context, **kwargs)
self._send_nova_notification(action, {},
{self._resource: obj})
return notify({self._resource: self._view(request.context,
obj)})
create函数首先创建IP和MAC信息保存到neutron数据库中，具体参考《nova boot代码流程分析(三)：nova与neutron的plugin交互》文章。然后调用notify函数发送port.create.end给neutron-dhcp-agent。最终neutron-dhcp-agent将调用port_create_end函数。
#/neutron/agent/dhcp/agent.py:DhcpAgent
@utils.synchronized('dhcp-agent')
def port_update_end(self, context, payload):
"""Handle the port.update.end notification event."""
updated_port = dhcp.DictModel(payload['port'])
network = self.cache.get_network_by_id(updated_port.network_id)
if network:
driver_action = 'reload_allocations'
if self._is_port_on_this_agent(updated_port):
orig = self.cache.get_port_by_id(updated_port['id'])
# assume IP change if not in cache
old_ips = {i['ip_address'] for i in orig['fixed_ips'] or []}
new_ips = {i['ip_address'] for i in updated_port['fixed_ips']}
if old_ips != new_ips:
driver_action = 'restart'
self.cache.put_port(updated_port)
self.call_driver(driver_action, network)
def _is_port_on_this_agent(self, port):
thishost = utils.get_dhcp_agent_device_id(
port['network_id'], self.conf.host)
return port['device_id'] == thishost
# Use the update handler for the port create event.
port_create_end = port_update_end
具体如何到达该代码流程，可以参考第1小节创建subnet的代码流程。这里将port_update_end函数赋给port_create_end函数，所以最终是调用port_update_end函数。port_update_end函数将比较self.cache中保存的ip信息与最新数据库中相对应的network中对应的ip信息(从neutron-server传递下来的，即payload)。正常流程下，因为我们创建VM会创建新的port信息到数据库中，而neutron-dhcp-agent的self.cache中还保存着上一次更新的network信息，所以导致数据库中port信息与self.cache中保存的port信息不一致，因此将更新self.cache中的port信息，且执行reload_allocations函数，重新更新dnsmasq的配置文件。
#/neutron/agent/linux/dhcp.py:Dnsmasq
def reload_allocations(self):
"""Rebuild the dnsmasq config and signal the dnsmasq to reload."""
# If all subnets turn off dhcp, kill the process.
if not self._enable_dhcp():
self.disable()
LOG.debug('Killing dnsmasq for network since all subnets have '
'turned off DHCP: %s', self.network.id)
return
self._release_unused_leases()
self._spawn_or_reload_process(reload_with_HUP=True)
LOG.debug('Reloading allocations for network: %s', self.network.id)
self.device_manager.update(self.network, self.interface_name)
调用_release_unused_leases函数release在leases文件(/var/lib/neutron/dhcp/43c0e274-28e3-482e-a32b-d783980fc3ed/leases)中未被使用的ip和mac。然后再重新加载dnsmasq进程所需的配置文件，由于加载所需的配置文件我们在《neutron-dhcp-agent服务启动流程》中有分析，所以我们这里主要分析release在leases文件中未被使用的ip和mac。
#/neutron/agent/dhcp/agent.py:DhcpAgent
def _release_unused_leases(self):
filename = self.get_conf_file_name('host')
old_leases = self._read_hosts_file_leases(filename)
new_leases = set()
for port in self.network.ports:
for alloc in port.fixed_ips:
new_leases.add((alloc.ip_address, port.mac_address))
for ip, mac in old_leases - new_leases:
self._release_lease(mac, ip)
#/neutron/agent/dhcp/agent.py:DhcpAgent
def _release_lease(self, mac_address, ip):
"""Release a DHCP lease."""
cmd = ['dhcp_release', self.interface_name, ip, mac_address]
ip_wrapper = ip_lib.IPWrapper(namespace=self.network.namespace)
ip_wrapper.netns.execute(cmd, run_as_root=True)
_release_unused_leases函数读取host文件(/var/lib/neutron/dhcp/43c0e274-28e3-482e-a32b-d783980fc3ed/host)中的被dnsmasq分配的ip和mac信息，将这些信息与数据库中network的ip和mac信息作对比，如果dnsmasq分配的ip和mac信息并未在数据库中，说明此ip和mac信息未被使用，所以调用_release_lease函数对其进行释放。
_release_lease函数是通过dhcp_release命令向dnsmasq进程监听接口ns-xxx发送DHCPRELEASE请求来释放未被使用的ip和mac信息。即删除leases文件未被使用的ip和mac信息。下面是一个测试用例。
开始leases文件信息：
| [root@nova 43c0e274-28e3-482e-a32b-d783980fc3ed]# cat leases 1464597255 fa:16:3e:da:42:50 1.1.1.2 host-1-1-1-2 * 1464597255 fa:16:3e:d0:eb:87 1.1.1.9 host-1-1-1-9 * 1464597255 fa:16:3e:d1:d7:72 1.1.1.1 host-1-1-1-1 * |
假设我们将release红色部分的ip和mac信息，则执行以下操作。
| [root@nova ~]# ip netns exec qdhcp-43c0e274-28e3-482e-a32b-d783980fc3ed dhcp_release ns-f7620da4-39 1.1.1.9 fa:16:3e:d0:eb:87 |
在/var/log/messages文件中，可以看到如下日志。
| May 29 04:38:09 nova dnsmasq-dhcp[3759]: DHCPRELEASE(ns-f7620da4-39) 1.1.1.9 fa:16:3e:d0:eb:87 |
执行dhcp_release命令后的leases文件信息：
| [root@nova 43c0e274-28e3-482e-a32b-d783980fc3ed]# cat leases 1464597255 fa:16:3e:da:42:50 1.1.1.2 host-1-1-1-2 * 1464597255 fa:16:3e:d1:d7:72 1.1.1.1 host-1-1-1-1 * |
可以看出，ip为1.1.1.9相关的信息被删除了。
这是_release_unused_leases函数的功能，而reload_allocations函数剩下的代码则是重新配置dnsmasq的配置文件，假设我们创建VM在数据库中分配的ip为1.1.1.10，那么在host文件中将把该ip及相对应的mac写入进去，待VM启动且发送dhcp discover请求时将该IP和mac分配于它。
2.2 分配ip和mac给VM
在创建VM开始时，首先为VM在数据库中创建port信息，创建port信息完成后，通知neutron-dhcp-agent更新dnsmasq的配置文件(包括host和addn_hosts文件)信息。如下
| [root@nova 43c0e274-28e3-482e-a32b-d783980fc3ed]# cat addn_hosts 1.1.1.1 host-1-1-1-1.openstacklocal host-1-1-1-1 1.1.1.2 host-1-1-1-2.openstacklocal host-1-1-1-2 1.1.1.10 host-1-1-1-10.openstacklocal host-1-1-1-10 [root@nova 43c0e274-28e3-482e-a32b-d783980fc3ed]# cat host fa:16:3e:d1:d7:72,host-1-1-1-1.openstacklocal,1.1.1.1 fa:16:3e:da:42:50,host-1-1-1-2.openstacklocal,1.1.1.2 fa:16:3e:3c:a3:3e,host-1-1-1-10.openstacklocal,1.1.1.10 |
这是在VM启动之前，就已经保存到dnsmasq的配置文件中的信息。
等待VM启动时，发送dhcp discover请求。
VM启动的有关dhcp的log如下(这是dhcp client端)。
| Starting network... udhcpc (v1.20.1) started Sending discover... Sending select for 1.1.1.10... Lease of 1.1.1.10 obtained, lease time 86400 cirros-ds 'net' up at 6.03 |
/var/log/messages的有关dhcp的log如下(这是dhcp server端)。
| May 29 05:05:34 nova dnsmasq-dhcp[4388]: DHCPDISCOVER(ns-f7620da4-39) fa:16:3e:3c:a3:3e May 29 05:05:34 nova dnsmasq-dhcp[4388]: DHCPOFFER(ns-f7620da4-39) 1.1.1.10 fa:16:3e:3c:a3:3e May 29 05:05:34 nova dnsmasq-dhcp[4388]: DHCPREQUEST(ns-f7620da4-39) 1.1.1.10 fa:16:3e:3c:a3:3e May 29 05:05:34 nova dnsmasq-dhcp[4388]: DHCPACK(ns-f7620da4-39) 1.1.1.10 fa:16:3e:3c:a3:3e host-1-1-1-10 |
同时neutron-dhcp-agent更新dnsmasq的leases文件信息。
| [root@nova 43c0e274-28e3-482e-a32b-d783980fc3ed]# cat leases 1464599134 fa:16:3e:3c:a3:3e 1.1.1.10 host-1-1-1-10 01:fa:16:3e:3c:a3:3e 1464598886 fa:16:3e:da:42:50 1.1.1.2 host-1-1-1-2 * 1464598886 fa:16:3e:d1:d7:72 1.1.1.1 host-1-1-1-1 * |
Dhcp client与dhcp server的交互过程如下图所示。
这里创建VM时，从neutron-dhcp-agent获取ip和mac的代码流程便分析完成。总结一下：
1. 创建VM时，nova-compute与neutron的plugin交互，在neutron的数据库中创建VM所需的port信息。
2. neutron数据库中的port信息创建完成后，通知neutron-dhcp-agent去执行port_create_end函数。该函数将数据库中的port中的ip和mac信息加载到dnsmasq所需的配置文件中(包括host和addn_hosts文件)。
3. 在VM启动时，广播dhcp discover请求，当dnsmasq进程的监听接口ns-xxx监听到这种请求时，dnsmasq进程将根据配置文件(host和leases文件)中的内容去判定是否有未分配的ip和mac为请求者进行提供。
4. 最终VM便真实的获取到与保存在数据库中的ip和mac信息。neutron-dhcp-agent只是将所创建VM的ip和mac信息从数据库中获取到自己的配置文件中，然后等到VM启动时，为它提供。因此neutron-dhcp-agent相当于在VM和数据库之间起了个中间桥梁的作用。