Fix the error CERTIFICATE_VERIFY_FAILED (by quqi99)

作者：张华 发表于：2020-07-07
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

问题

在测试rbac时, 下列rbac调用candid时会报CERTIFICATE_VERIFY_FAILED这个错. 在读了rbac的代码之后, 确定rbac是在调用 https://node1.lan:8081/discharge/info 时抛的错.

maas (maas configauth --rbac-url https://node1.lan:5000/ --rbac-service-name maastest) -> rbac with http (http://node1.lan:8081) -> candid with https (https://node1.lan:8081/discharge/info) -> ldap with https

奇怪的是, 我已经运行过这个命令( cp ~/certs/ca.crt /usr/share/ca-certificates/extras/ldap.crt && dpkg-reconfigure ca-certificates )了. 不应该啊.

sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs

curl测试

运行下列三个命令均无问题:

curl https://node1.lan:8081/discharge/info
curl https://node1.lan:8081/discharge/info -k
curl https://node1.lan:8081/discharge/info --cacert ~/certs/ca.crt

python测试 - urllib2.request

下列python代码测试也没问题

sudo bash -c 'cat > test.py' << EOF
#!/usr/bin/env python

import sys
import ssl
try:
    import urllib2 #python2
except:
    import urllib.request as urllib2 #python3

req = urllib2.Request(sys.argv[1], headers={'Bakery-Protocol-Version':'3'})
print(urllib2.urlopen(req).read())
EOF
mkdir /usr/share/ca-certificates/extras
cp ~/certs//ca.crt /usr/share/ca-certificates/extras/ldap.crt && dpkg-reconfigure ca-certificates
python test.py https://node1.lan:8081/discharge/info

python测试 - requests

奇了怪了, 只好继续调试代码, 发现rbac使用了pymacaroons (https://github.com/ecordell/pymacaroons), pymacaroons它又会调用requests (不是urllib2.request)
所以继续写了一个python测试, 问题就重现了:

bash -c 'cat > test2.py' << EOF
#!/bin/env python

import sys
import requests

url = sys.argv[1]
print(requests.get(url).read())
EOF

$ python test2.py https://node1.lan:8081/discharge/info
...
requests.exceptions.SSLError: HTTPSConnectionPool(host='node1.lan', port=8081): Max retries exceeded with url: /discharge/info (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

添加REQUESTS_CA_BUNDLE=/etc/ssl/certs/ldap.pem 问题解决.

$ REQUESTS_CA_BUNDLE=/etc/ssl/certs/ldap.pem  python test2.py https://node1.lan:8081/discharge/info
200

问题解决

改为下列使命启动rbac

REQUESTS_CA_BUNDLE=/etc/ssl/certs/ldap.pem crbs-devserver --debug run -P 5000

或者使用全局ca database

# use the full system database
REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt  crbs-devserver --debug run -P 5000

问题分析

应该是遇到了这个bug - https://stackoverflow.com/questions/42982143/python-requests-how-to-use-system-ca-certificates-debian-ubuntu/42982144#42982144
现在request的代码(https://github.com/psf/requests/blob/master/requests/certs.py)使用使用certifi模块中的where函数来指定ca证书路径.

# grep -r where /bak/work/crbs/.ve/lib/python3.6/site-packages/requests/certs.py
if __name__ == '__main__':
    print(where())

# grep -r 'def where' /bak/work/crbs/.ve/lib/python3.6/site-packages/certifi/core.py -A 3
def where():
    f = os.path.dirname(__file__)
    return os.path.join(f, "cacert.pem")

python-request 2.9.1版本的debian包中有一个01_use-system-ca-certificates.patch修改了ca路径为/etc/ssl/certs/ca-certificates.crt, requests-2.18.4版本开始改用certifi模块中的where函数并修复了这个问题.

# grep -r 'return' /tmp/requests-2.9.1/debian/patches/01_use-system-ca-certificates.patch
-        return os.path.join(os.path.dirname(__file__), 'cacert.pem')
+        return '/etc/ssl/certs/ca-certificates.crt'
 
# grep -r where /tmp/requests-2.18.4/requests/certs.py 
environment, you can change the definition of where() to return a separately
from certifi import where
    print(where())
# grep -r 'def where' /usr/lib/python3/dist-packages/certifi/core.py -A 2
def where():
    return "/etc/ssl/certs/ca-certificates.crt"

但virtualenv环境中没有改:

# grep -r 'def where' /bak/work/crbs/.ve/lib/python3.6/site-packages/certifi/core.py -A 2
def where():
    f = os.path.dirname(__file__)

不可能去修改上游virtualenv模块, 看来只能修改rbac这个应用了.

怎么解决呢

requests module (/snap/xxx-rbac/224/lib/python3.6/site-packages/requests)在使用/snap/xxx-rbac/224/lib/python3.6/site-packages/certifi/cacert.pem. 而不是在使用/var/snap/xxx-rbac/current/conf/certs/ca.pem (snap set xxx-rbac ssl.ca="$(cat $cert_dir/ca.crt)")

(.ve) root@node1:~# grep -r 'import where' /snap/xxx-rbac/224/lib/python3.6/site-packages/requests/certs.py -A 3
from certifi import where
if __name__ == '__main__':
    print(where())
(.ve) root@node1:~# grep -r 'where' /snap/xxx-rbac/224/lib/python3.6/site-packages/certifi/core.py -A 3
def where():
    f = os.path.dirname(__file__)
    return os.path.join(f, 'cacert.pem')
  
(.ve) root@node1:~# ls /snap/xxx-rbac/224/lib/python3.6/site-packages/certifi/cacert.pem
/snap/xxx-rbac/224/lib/python3.6/site-packages/certifi/cacert.pem

所以得使用REQUESTS_CA_BUNDLE变量, 但上面是通过源码运行的 (REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt crbs-devserver --debug run -P 5000), 现在改从snap(systemctl restart snap.xxx-rbac.uwsgi)的方式启动. 但发现这条code path (./crbs/wsgi.py#make_wsgi_app -> ./crbs/setup.py#setup_globals -> ./crbs/snap/_env.py(set REQUESTS_CA_BUNDLE))设置了REQUESTS_CA_BUNDLE, 可它为什么不生效了.

# grep -r 'REQUESTS_CA_BUNDLE' /snap/xxx-rbac/224/lib/python3.6/site-packages/crbs/snap/_env.py
            environ['REQUESTS_CA_BUNDLE'] = str(cert_bundle)

但从snap容器中没查着这个环境变量:

(.ve) root@node1:~# snap run --shell xxx-rbac.uwsgi
root@node1:/root# env |grep REQUESTS_CA_BUNDLE

NOTE: 也可以通过:snap run xxx-rbac.uwsgi --ini /var/snap/xxx-rbac/x4/conf/uwsgi.ini 来在snap container里运行命令，但是snap container里没有python命令:
root@node1:~# snap run canonical-rbac.python
error: cannot find app "python" in "canonical-rbac"

从host机器也没查着:

(.ve) root@node1:~# pidof uwsgi
3898 3897 3896 3895 3894 3893 3892 3891 3679
(.ve) root@node1:~# cat /proc/3898/environ |grep REQUESTS_CA_BUNDLE
(.ve) root@node1:~#

改以只能想着通过pdb ( import pdb;pdb.set_trace() )调试, 但/snap/xxx-rbac/224/lib/python3.6/site-packages/crbs/snap/_env.py是只读的没法添加断点.
另一种可能的调试方法是采用’python3 -m pdb xx.py’的方式交互性调试( 这样, 断点设置在第1行 - https://pymotw.com/3/pdb/ ). snap.xxx-rbac.uwsgi最终得通过(/usr/bin/snap run xxx-rbac.uwsgi) 启动, 最终调用command-uwsgi.wrapper启动, 但似乎我们也无法修改command-uwsgi.wrapper啊它也中只读的.

(.ve) root@node1:~# cat /snap/xxx-rbac/224/command-uwsgi.wrapper 
#!/bin/sh
exec "uwsgi" "--ini" "$SNAP_DATA/conf/uwsgi.ini" "$@"

debug snap

想给snap里的python用pdb加个断点，因为snap的代码目录是只读的，非常难调．用下列方法可以修改源代码．但是仍然无法加 ‘import pdb;pdb.set_trace()’ 到源码，因为pdb需要通过python启动程序而不是通过snap启动程序．所以实际上snap部分的代码还是相当难调．

# https://forum.snapcraft.io/t/make-a-snap-writable-temporarily-for-debugging/16370/3
unsquashfs -d canonical-rbac /var/lib/snapd/snaps/canonical-rbac_224.snap
# modfity file here. eg:  ./canonical-rbac/lib/python3.6/site-packages/crbs/snap/_env.py
snap try canonical-rbac --devmode

问题最终解决

snap都是只读的实在无法调试，最后发现一个现象．直接在浏览器(最好用firefox, 使用chrome时不弹出框)访问rbac portal (https://node1.lan:5000 )并不出现这个错，只是使用＂maas configauth --rbac-url https://node1.lan:5000/ --rbac-service-name maastest＂命令时报这个错．这会不会是maas的问题？尤其是maas是通过snap安装的，而不是通过debian安装的，maas是不是缺少了下 面类似于rbac的用于设置的代码．

# grep -r 'REQUESTS_CA_BUNDLE' /snap/xxx-rbac/224/lib/python3.6/site-packages/crbs/snap/_env.py
            environ['REQUESTS_CA_BUNDLE'] = str(cert_bundle)

通过debian包安装后证明果然是这个问题.

#sudo snap install maas --channel=2.8
#sudo snap install maas-test-db                              #enter shell - maas-test-db.psql
#sudo maas init region+rack --database-uri maas-test-db:///  #http://192.168.99.124:5240/MAAS
sudo add-apt-repository ppa:maas/2.8   #https://launchpad.net/~maas
sudo apt install maas

sudo maas configauth --rbac-url https://node1.lan:5000/ --rbac-service-name maastest  #use external authentication
# sudo maas createadmin   #not use external authentication