解决octavia failed to run
octavia failed to run
前言
如果没有特别标注，所有的命令行都是使用admin账户执行的。
问题描述
通过如下调用octavia的api创建loadbalance
@token = gAAAAABe6y8up_ZdUS43Lfe8AFERncGtZFK6IHrnd-MSgbE5U-xKg-6N04eUV3tNwlKEAiZVNIl6I0NVvjMT97CygiJL-7XFciQHl0vWdd2rTQWtaDR9w04pJtk1DYBi7vjcq0VivngYMrn2MhOJmf3X3Fss9kV2BY9m19v-A6mba9CYVPE_NHY
@endpoint = http://10.8.8.11:9876
###
# create loadbalancer
POST {{endpoint}}/v2.0/lbaas/loadbalancers
X-Auth-Token: {{token}}
Content-Type: application/json
{
"loadbalancer": {
"description": "loadbalancer_created_by_gxj_for_test_purpose",
# "flavor_id": "30ad863a-25dc-425a-8ebe-e29f361cb777",
"admin_state_up": true,
"vip_subnet_id": "3aa77344-3086-4d2c-aa1a-27c997c92840",
"provider": "octavia",
"name": "loadbalancer_created_by_gxj_for_test_purpose"
}
}
调用后成功返回
HTTP/1.0 201 Created
Content-Length: 681
Connection: keep-alive
Content-Type: application/json
Date: Thu, 18 Jun 2020 09:34:14 GMT
keep-alive: timeout=4
Proxy-Connection: keep-alive
Server: WSGIServer/0.1 Python/2.7.12
X-Openstack-Request-Id: req-db7143a3-d2aa-450e-8414-1b2fa257e974
{
"loadbalancer": {
"provider": "octavia",
"flavor_id": "",
"description": "loadbalancer_created_by_gxj_for_test_purpose",
"provisioning_status": "PENDING_CREATE",
"created_at": "2020-06-18T09:34:14",
"admin_state_up": true,
"updated_at": null,
"vip_qos_policy_id": null,
"vip_subnet_id": "3aa77344-3086-4d2c-aa1a-27c997c92840",
"listeners": [],
"vip_port_id": "34f53806-1c12-48a5-8724-d965cb74cf7e",
"vip_network_id": "0c1c8b1d-6515-4e5c-bba2-4a5e3ed80084",
"vip_address": "10.0.0.29",
"pools": [],
"project_id": "55982e2c523149158a6cc839ce2eece2",
"id": "59b14489-3f4f-49dc-8533-29fc50377eb9",
"operating_status": "OFFLINE",
"name": "loadbalancer_created_by_gxj_for_test_purpose"
}
}
但是过一段时间去检查loadbalancer的状态的时候发现是error。
问题排查
大致流程
- octavia大致的workflow(工作流)
- 查看octavia-api的日志
- 查看octavia-controller-worker的日志
octavia大致的workflow
图是官方的
相信参考OpenStack Docs: Octavia v0.5 Component Design
查看octavia-api的日志
通过vi /var/log/octavia/api.log查询loadbalancer的id发现下面的日志
2020-06-18 17:34:14.880 24792 INFO octavia.api.v2.controllers.load_balancer [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Sending created Load Balancer 59b14489-3f4f-49dc-8533-29fc50377eb9 to the handler
通过这边发现来自client的创建请求已经被send出去给controller-worker了，也就是说octavia这边没问题,但是往上面看了点，发现了一些error
2020-06-18 17:34:03.665 24792 DEBUG octavia.common.keystone [req-72f1be5c-a7e3-42bc-9590-032ec02a1b0a - 55982e2c523149158a6cc839ce2eece2 - default default] Request path is / and it does not require keystone authentication process_request /usr/lib/python2.7/dist-packages/octavia/common/keystone.py:67
2020-06-18 17:34:13.943 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension security-group found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:13.950 24792 DEBUG neutronclient.v2_0.client [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Error message: {"message": "The resource could not be found.
\nExtension with alias dns-integration does not exist\n\n", "code": "404 Not Found", "title": "Not Found"} _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
2020-06-18 17:34:13.951 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension dns-integration is not enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:70
2020-06-18 17:34:13.957 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:13.965 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension allowed-address-pairs found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.018 24792 DEBUG octavia.db.repositories [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Checking quota for project: 55982e2c523149158a6cc839ce2eece2 object: <class 'octavia.common.data_models.LoadBalancer'> check_quota_met /usr/lib/python2.7/dist-packages/octavia/db/repositories.py:313
2020-06-18 17:34:14.047 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension security-group found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.052 24792 DEBUG neutronclient.v2_0.client [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Error message: {"message": "The resource could not be found.
\nExtension with alias dns-integration does not exist\n\n", "code": "404 Not Found", "title": "Not Found"} _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
2020-06-18 17:34:14.052 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension dns-integration is not enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:70
2020-06-18 17:34:14.057 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.063 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension allowed-address-pairs found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.070 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension project-id found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
这个error应该是请求的这个资源55982e2c523149158a6cc839ce2eece2不存在。我感觉这是个网络相关的资源于是走了下面的一个命令
root@ctl01:~# openstack network list | grep 55982e2c523149158a6cc839ce2eece2
| 26aef702-6d8e-4384-9c87-5d7c851021ac | HA network tenant 55982e2c523149158a6cc839ce2eece2 | 022fb016-e250-42ad-95e2-8ba4f26350d5
实际上这串uuid是某个租户，想当然往往会出乎意外
于是乎，我尝试了下
root@ctl01:~# openstack project list | grep 55982e2c523149158a6cc839ce2eece2
| 55982e2c523149158a6cc839ce2eece2 | admin |
| 9d99f566b76246d4b72722b9882c245f | 55982e2c523149158a6cc839ce2eece2-b77e26e9-a4a6-4773-a42d-7ef029d |
发现这个貌似应该是admin的project_id. 然而被没有软用，细看日志发现自己漏了最关键的东西nExtension with alias dns-integration does not exist, google下这个应该是neutron dns as a service
查看octavia-controller-worker的日志
用vim 打开controller-worker的日志文件"/var/log/octavia/worker.log"根据上一次创建失败的loadbalancer的id查找有用日志，下面是相关的日志。
1223 2020-06-18 17:34:14.922 3374787 INFO octavia.controller.queue.endpoint [-] Creating load balancer '59b14489-3f4f-49dc-8533-29fc50377eb9'...
1224 2020-06-18 17:34:14.984 3374787 INFO octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB with id ee934942-fc30-44ef-b834-e2dda0cd650f
1225 2020-06-18 17:34:15.031 3374787 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
1226 2020-06-18 17:34:15.032 3374787 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
1227 2020-06-18 17:34:15.032 3374787 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
1228 2020-06-18 17:34:15.032 3374787 INFO octavia.certificates.generator.local [-] No Passphrase found for CA Private Key, not using one.
1229 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver [-] Nova failed to build the instance due to: Network 8a588b4c-06ae-4ba2-ae74-2a 8a6ed687ef could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0): BadRequest: Network 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1230 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver Traceback (most recent call last):
1231 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python2.7/dist-packages/octavia/compute/drivers/nova_driver.py" , line 154, in build
1232 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver availability_zone=CONF.nova.availability_zone
1233 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 1313, i n create
1234 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver return self._boot(response_key, *boot_args, **boot_kwargs)
1235 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 772, in _boot
1236 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver return_raw=return_raw, **kwargs)
1237 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 366, in _crea te
1238 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver resp, body = self.api.client.post(url, body=body)
1239 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 310, in post
1240 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver return self.request(url, 'POST', **kwargs)
1241 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 83, in requ est
1242 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver raise exceptions.from_response(resp, body, url, method)
1243 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver BadRequest: Network 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HT TP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1244 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver
1245 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks [-] Compute create for amphora id: ee934942-fc30-44ef-b834-e2dda0cd650 f failed: ComputeBuildException: Failed to build compute instance due to: Network 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HTTP 400) (Req uest-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1246 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks Traceback (most recent call last):
1247 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks File "/usr/lib/python2.7/dist-packages/octavia/controller/worker/tas ks/compute_tasks.py", line 99, in execute
1248 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks server_group_id=server_group_id)
1249 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks File "/usr/lib/python2.7/dist-packages/octavia/compute/drivers/nova_ driver.py", line 160, in build
1250 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks raise exceptions.ComputeBuildException(fault=e)
1251 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks ComputeBuildException: Failed to build compute instance due to: Networ k 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1250 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks raise exceptions.ComputeBuildException(fault=e)
1251 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks ComputeBuildException: Failed to build compute instance due to: Networ k 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1252 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks
1253 2020-06-18 17:34:15.563 3374787 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-co mpute-create' (2f7f05b0-c7a3-4fa1-94f2-584c70ca20d1) transitioned into state 'FAILURE' from state 'RUNNING'
1254 8 predecessors (most recent first):
1255 Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem ': 'gAAAAABe6zUXLP9ofIXoOlDm1_OLc5arzZh_Bo2aunp_iGB9qiRS8X-O79tIYMY24-zhg-MD8JBpWQsZ9hEQsEFYLKaVKjSKUNX9oRkjeyWLZXeAyKhYJ6GN6tndes2LMSL0GrfHqs27fg2UlYuQ0F cLMpfEKoLXLLc7rqIsHdk1bMTIc0uMAAPE4b4QPPH3G0ta5Mk5uBxrrf5lgNAusjdy4Xygvzu5kXlGXrov50sVdQIINGIbq9GwL6hJYG2SHHz0w2-m0EAG6i_31zDYL76D5gVcCVSt0-pKFOdydOe1am23 Gels0RunQgXhNnQ6yiZJf0L1T9XsrkyFP9u6QmCe5cGF48u0Ddgaa9qOXNuOtLLlWFpkmMQbHOOpDMt96w-3ra149J4R39HMQc8DJdBhE5m-x6XxbmCzojPrFtbO0cyGn3pnkhClPdgsGZtuw5rUFm5SQu Yw76HuGB2vxe5qeN6JVccGHhJJUum4XRZHmgJEV7kkxeYz0JYwM2TUDYOdOxten3Jb55-YoaB4X0dcamdx6InFYJHueG8VH7l0BPk5OjxHzyWsOfj5cFiLlng3MdqWa3Lw_zUYACF9HHkUycHJ1v-NNzWf VCFo-8yaDpb6I1tCtwrr6fBdv0x0O_L0XTRNnHS2uztHneIFcXbAfj0gWz1Y2yhpJtLR69PgOM_Qg6RcAZ1fFv4QBynPJ2Zr1D6wJKgz0yJygHrW4LGYrOdnHPHugIKn38xP68qzBNpg-BCqNsjmepPJvx cJXh9XVrvFKyYkrr7uD5OGuo1yHAXXK4Ur29H9-O4qMZICllHWs7tXa3gKKLqt2jXHYheeIoOwC49bieaLw4RLVuKaQoxrHcqezYeRCtuUo0azZuqP3zdTjO_ywGniw1U0zL5Un5-siO1ukqiadAQOByTT gd7hpjEl6CW85sukEjZRSOpDMikg1uodBN4ewTZa_J-u4ujEJaMwEnnjqepwXpfQ0Qbzq3ZtXq-g1_q9GJttYGEtyJqTioq6YS5jSUXX4d039NjtsLU0b_voEbXBon6bwZdCTnUEnHQ7kNqdA_Zm1oQMZX hbAhRWc_chzXx_75TszeRumBwo5DJbmfxuj4BG7IQgtuY-26gH8TZEVVGDNvfNIKXZD-faWR9UXw5KU3SpKU1QYGM5r1oRypRH0NUka4pk-pJzwBpfWZg3gy1-yFmblyGr441-Pfm2cWSXSP0f1oB2mhNf WJWwKGCKD3f27q5E5ln4guOwPHgg4Ioy3uY8QwfQz_LKlY3UmGr-BMsfi2cfxYru76fAKxUf1V5nL4Bn4Lg2a_pa2y2d2c6v2JBWZZnmwR-yXO1Mxq4e8iock-2FNVlHZuRi2xwNPWn5TqlDaqoxbYmPpD Deonecix7Q2po_TWjFnGHAMZzX0qka26Sm4yd6JKtIdfakhqakTma_EqmxM2L9HkldLbljCkn9Q8tLsiv4G4x1lz8ZSETjNCflZmuGjLUbXBVCEQl2I398j-XOZ4F1MBSc6hppg9_MwzF17i8w29T3PxUd jNjA47e1DCb-tD8NnF43BuLufkOzzeRBs5CBXso9SzOTJMtA74EeVzMsDRaFyjZGdWerfKqNisfpzreM1pgLRaM_4BcqPOg5OcaGHrWHhGUV0ov5dzfkzPqskHS-KeI2nT5G3jl37QGfNleN6oBkOArkJz RJ60Q8xlnqjGex89wClAV_dWKVs7XoVxdOvsOo7eskXCsFxNADcynAGBDxK4CKfnZkcN5cdCwyl15tiLp487XKW-axXo2Oe49QlN6p5DLK-OgBdh4bZdt6jdbx7PvzoXNx2HtELjPkDvplbYz7_bwcZmHk Tbd7h1iktKMSv5f2I4zTD7CtJFGMVaN1kmTclEgh1KkQUVF91VxzB6aJ24ZFqoUsP9F_dB3cSk2kQ1TpvJlhfj00aVuLnOnD6cgaiIWQ_Dn11rqodvRdYGS7lxTmJhiwRSRkQjnIIVHT7En3k1Hymx3lIm 1EgpiQ1lVveaebj440UdN7dPAlQ6iG5XcRE3ZTGzdxAuV7adJZsI-9NNO6mOzv4bcOsYchDxhEgn_eQW0MuKhafe3-yVt8TrjvW0w4r0J0QaEC0ZxrLiIBFtnen7LVkAoQNGv2u5t-tb6fGGCDD9gFNtwK dLMC5TR-EmtGbWJehS4bVPR7URTi7czoI0S3GFSXVsqKFQRXRaVqw9uF8OZZFwRYQYKJQg2YnwqYhywfZdl2mMrLxyGeAOe5WqOzI_KeE1fros_l95FinwTbz-i-mm0mO-VDGKhnFcgk2KYQl-w71ywMgp NKXhgPmLbDFpzButkHFhF618oOc__L4ZvgFczu2YIv-_ZNc2y7NA9939cGUV45hFfzbA8JkmZIgn-80dSVhGe0cmGkeHqsETWFuryHQtA1TWnWng8VrR5GPVSFXvwj4P6IJJsBC9uPHPi8M3pRcN1gdiAq Nzq1Q4I6ht3aNpPEz16n2dJ56tMTxY30PoH_swQI4KyN4LRsrBmFUA9e4vnkA4V05xZxCdhTbmmbDvJt3Ncej06935biSR_oPgOl6A-IoW7-jZvoLxD0dkGkYQ-NrBhiO0Qxx0iCmtVME4xiSH26N3RS_K 8_odfZSSAOme1-UDqJ0soaysnxnZiE2ioA3hgku-Ns9K29ZTqBDDb68GfTg4jPzz3jeNUI8GRSuiKbqAf1et6swZHYwsfg559XIxLnWDdyRacXwic2-mA5QSGzT8uEeYEQfsxYplOGAIO4isLBiCW6hwXW ZcgC8gUpCgbdpUHaxGCxBLrYl8HY62wORzcTkOf95oEQ3oJO9GgWX_gx6LQZh-05FvUnWO_IxGd8-hzvExIZ_DqL3I139Y9vy2590oxRsI4AmdOpuS47_jD0KgDGjj_4qF2bqhvrz2HuYL1eGFVGEyT1jD RwgkvfO6qpWBfs-9-utmKAgUg0dvXj-VrZDlHNUBLuhpk8yJNUM8NHVLjpPrJpbCGbZsLw-9r0nG3U4hf8zJSnk_0npEiTsuuYK0Y_Fz1IRyzwGoYpIc7ORuBzYK3nG_AAxCIGmrI2PMO7FeKIgWaanIB0 VRnRkRf9gRv4XkzQnGPYPYFrt6zDxizklpElVK6oasZqOaVT-Oxpw6Uu5v6UR_EFzTDiycDmRqwJfxIRw_izT3BD4vP4LGtvoexq-V58z4Ob3KRUYSOYh-bO3wueZE7dd5U8R98hWqLPjgc6E76vQrpudk sUa1j_5hRt4jMJPWf3eOPHI54Npdj9Jo1tyfGaNDxw_zOIZRxCpfFunL7w4AGsgXPsRKQIhsuLvKI01W2IX5sJ-dJIbQCfRYxQn_nIM_RLxOCY1lj-aL1Qc3Yn_gECx-dUBqyKoPnTiBVF20qlpOgq-6dA -OUa31nAK240KEAIaIXerai-ucltQsnJMFEwgBJduxJPT7d2IIszqaoBJyFDavYKbgBaA2EWokZCv5ur7-UkXRvdkf9MuvjczwUKdLXIXrlyHW8QXtnJN7BtQSoU3MulPQZkcaGXdUOBHHPi-uB6TM5zzi Gypk1N7F3BvMU0yjkraKYesJy4sCbhFk0WwJOapBVQrmQvsyzcCDpdWUvGqp581Sm9YOJ2toeYRCIT63CaJqdlEEOWBXB3MLgoAJDPkMMorH1VkMPWUhGZRMxy_p9bTVMoj6Kd3-z8d70NjGRRPRmxeDli Dq5oDwSdcslGC0yepxNRgRmel5VNmugWxpbWFjrr-cURMQ0BLlWhj7YUoh_7oPw0-_8NHuSaVxtk85p7bBpXXo-9qGrNKt8nHNoPtTj0uV2q4w9ROAYFNU8iHcsjVItfV9mG3Bl4rrXUZ-MPod3u8XuTVV Fty5f8Qic3JS66LWi8vKt2SbRlto9Ai3iYSpTJa81k_IHHVQWdJ7L7waakSuiGjjKOL37f7kQRNLgtwb5lWYXjLPih39PSdHiRVT_huboQnXfhVrNMAH3eq76BBNhgxGyuZaxSqhmONee7_1LnVA5HZMx7 aAJOBkloG56jZNOvnTqbiDlgGC4neJSm-tiSIiHo4u6KONoSRLH76OL2EiVpTGRhQN3wRQkPqzvDOvUtpwJ46_SbH-cbLc-GPFuz4WlGL6dAtQ7Z9KW9TosOC79l-skMeq2EJyTPgedVoedsA01cPiXTz9 q4xYG3Hi2iBc1No-809jYSg4hzrkTUl_kv4nwOsQwU8ykQxIXr3EYgUiJ7AxLXU4ZYf3lRTXG7drE3a0n_itWG-7oOMKnfxoExo-TjC8cUJ0KUHq-ydZ3VskhhlFkSLiH-Rxs4DxzrNlRQN1q-i1O98eYN kJiOOuP4YJRSF6PF9fJX8-lQIsopvrlsx2-x4Rs_HeObEgQiW0s2RULRE16d8XaJC4ko1Rodx7oW3ZS0olIzqYT1qqhqa4oqoQjGWxkoWmRkPG6nZK4MAZkebGLNNtwyuss77HQ7UrBCidaazKhQEKFPVK dphdNY4SA=', 'amphora_id': u'ee934942-fc30-44ef-b834-e2dda0cd650f'}, 'provides': None}
根据日志的提示，octavia由于没有找到network id 为8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef的网络导致无法调用nova成功创建amphora虚拟机。
根据下面的命令
root@ctl01:~# openstack network list | grep 3aa77344-3086-4d2c-aa1a-27c997c92840
| 0c1c8b1d-6515-4e5c-bba2-4a5e3ed80084 | tent_net | 3aa77344-3086-4d2c-aa1a-27c997c92840 |
我们可以看到本套openstack确实没有这个网络，此外我们传入的数据也没有什么跟着个网络id挂钩的。
那么这个网络id，可以的原因便是以下几种
- octavia-api处理流程中增加的
- octavia-controller-worker中增加的
通过我敏锐的第六感，这两种服务加入这个网络的途径，只有三种：
- 数据库
- 第三方服务
- 配置文件
我们先从配置文件开始，别问我问什么，问就是这个最简单。
配置文件排查
-
大致流程
ps -ef | grep target service name碰下运气看能否找到配置文件- 开启grep大法使用
grep target_id -rn config_folerorgrep target_id config_file
-
开始表演
开始ps
root@cmp002:~# ps -ef | grep octavia-worker
octavia 3381004 1 0 May26 ? 00:00:00 octavia-worker: master process [/usr/bin/octavia-worker --config-file /etc/octavia/octavia_manager.conf --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-worker --log-file /var/log/octavia/worker.log]
octavia 3381139 3381004 0 May26 ? 01:25:35 octavia-worker: ConsumerService worker(0)
root 4066420 4064201 0 09:38 pts/11 00:00:00 grep --color=auto octavia-worker
找到配置文件/etc/octavia/octavia_manager.conf和配置文件夹/etc/octavia/conf.d/common和/etc/octavia/conf.d/octavia-worker
开启grep大法
root@cmp001:~# grep "8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef" -rn /etc/octavia/
/etc/octavia/octavia_manager.conf:521:amp_boot_network_list = 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef
运气太好了，第一次尝试就找到了这个网络id
根据这篇文档OpenStack Docs: Load-Balancer-as-a-Service configuration options
配置amp_boot_network_list的作用是(List) List of networks to attach to the Amphorae. All networks defined in the list will be attached to each amphora.
使用ocatavia用户创建一个网络替代这个网络
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://10.8.8.11:35357/v3
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
export OS_USERNAME=octavia
export OS_PASSWORD=trVvCUyPpHwBNfDppE7ug5we1GRvJyeU
export OS_REGION_NAME=RegionOne
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE="internal"
export OS_CACERT="/etc/ssl/certs/ca-certificates.crt"
root@ctl01:~# openstack network create amp_boot_network_1
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2020-06-19T03:20:52Z |
| description | |
| dns_domain | None |
| id | 3ca482e2-34a5-4722-a4c5-2d7262e00945 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | False |
| is_vlan_transparent | None |
| mtu | 1450 |
| name | amp_boot_network_1 |
| port_security_enabled | True |
| project_id | cb20ced7c1ea4f43a35eb15f0ec23650 |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 45 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | |
| tags | |
| updated_at | 2020-06-19T03:20:52Z |
+---------------------------+--------------------------------------+
创建完网络后更新/etc/octavia/octavia/octavia_manager.conf里面的amp_boot_network_list项
然后重启所有octavia-worker节点上的octavia-worker服务。
创建新的loadbalancer验证是否成功
验证
创建一个loadbalancer，并查看状态
POST {{endpoint}}/v2.0/lbaas/loadbalancers
X-Auth-Token: {{token}}
Content-Type: application/json
{
"loadbalancer": {
"description": "loadbalancer_created_by_gxj_for_test_purpose",
# "flavor_id": "30ad863a-25dc-425a-8ebe-e29f361cb777",
"admin_state_up": true,
"vip_subnet_id": "3aa77344-3086-4d2c-aa1a-27c997c92840",
"provider": "octavia",
"name": "loadbalancer_created_by_gxj_for_test_purpose"
}
}
root@ctl01:~# openstack loadbalancer list | grep load
| 9bae06de-d0b9-4cce-9198-389a18de9fcd | loadbalancer_created_by_gxj_for_test_purpose | 55982e2c523149158a6cc839ce2eece2 | 10.0.0.8 | ACTIVE | octavia |
创建成功，问题解决