通过Helm Chart在Kubernetes集群Self-Hosted私有化部署Sentry（离线）

我们知道，Sentry是非常优秀的应用错误跟踪系统，还可以支持大多数的语言。不论是用在灰度的时候做验证也好，还是生产上用来快速追踪、定位以及提高对错误的治理能力也好，都非常适合，让运维人员不再依赖和等待用户的报错。

在上一篇文章《docker-compose安装sentry 9.1.2不踩坑指南》中我们聊了docker-compose部署Sentry，这次我们尝试下通过Helm Chart的方式在Kubernetes集群私有化部署。

话不多说，我们进入正题。

版本和环境

根据helm官方给出的要求

• Kubernetes 1.4+，我手上的用的是1.14
• helm >= v2.3.0，我的环境是2.14.3
• PV provisoner support，我的集群提供ceph-rbd的StorageClass

Sentry版本在Helm Chart当前是9.1.2

环境的话，是公司内部局域网，而且是无法连接外网的局域网，即我们需要通过离线的方式部署，增加了部署的难度，最最关键是我们还没有chartmusium。而且即使有，比如说通过harbor，也是需要修改对应的Chart镜像改为公司内网镜像的。所以这增加了我们的安装难度，不过没有关系，通过配置可以让我们更加了解Sentry。

如果你的集群环境可以联网在线部署，那么强烈推荐这篇博客，前几天刚发布的：《Sentry实时应用错误跟踪系统在Kubernetes中私有化部署》，会更加适合你。

安装步骤

一、下载Chart包和镜像

既然离线，那么只有自己下载了，金融行业局域网限制得太严了。

1、Sentry chart

https://github.com/helm/charts/tree/master/stable/sentry

2、postgres chart

https://github.com/helm/charts/tree/master/stable/postgresql
不过这个github里的版本已经deprecated了，直接用bitnami账户的
https://github.com/bitnami/charts/tree/master/bitnami/postgresql

3、redis chart

https://github.com/helm/charts/tree/master/stable/redis
和pg一样，这个github里的版本已经deprecated了，直接用bitnami账户的
https://github.com/bitnami/charts/tree/master/bitnami/redis

redis有几种模式，看你自己具体的需求，下载chart包的时候有两个选择，一个是redis chart，一个是redis cluster chart。我们用的是redis chart，在redis里面可以配置单节点模式、主从模式、哨兵模式，我们这里选择默认的主从模式，一个master，2个slave。你可以在values.yaml里面配置你想要的模式。

4、docker镜像

在上述chart文件里面用到的images需要全部下载并导入到局域网镜像仓库。主要包括：

# sentry镜像
docker pull sentry:9.1.2
# postgres镜像，这里使用bitnami的镜像，helm官方指定
docker pull bitnami/postgresql:11.7.0-debian-10-r80
# redis镜像，哨兵模式如果不启用可不下载，这里使用bitnami的镜像，helm官方指定
docker pull bitnami/redis:5.0.8-debian-10-r50
docker pull bitnami/redis-sentinel:5.0.8-debian-10-r41
# 设备管理镜像，用于yum install以及提权，这里使用bitnami的镜像，helm官方指定
docker pull bitnami/minideb:buster
# 指标暴露镜像
docker pull prom/statsd-exporter:v0.10.5

二、搭建内网chartmesum并上传chart包

1、搭建chartmesum

如果你的局域网已经有chartmesum了，这一步请跳过

因我们局域网已经有harbor镜像仓库了，所以我们直接开启harbor的chartmuseum功能即可

harbor是通过docker-compose在k8s集群外搭建的，通过docker-compose命令配置开启chartmuseum功能，注意：harbor 1.6及以上版本才支持，如果是通过helm安装的harbor自己去helm upgrade哈

cd harbor
docker-compose stop
./install.sh --with-chartmuseum

如果你没有harbor，也可以搭建本地的chartmuseum
下载chartmuseum包，然后配置values.yaml文件，然后helm install . --name my-chartmuseum

2、修改chart包默认配置

根据你们的实际需要，调整chart包里面的默认设置，比如用到的镜像仓库和地址，改为内网的harbor仓库地址。记得把README文档也一并更新下，方便其他人查阅及配置。

然后把chart包打包成tar文件，比如redis.tar.gz

3、上传chart包

需要把Sentry、redis、postgres的chart包都上传到内网chartmuseum，上传方式如下：
（1）harbor ui界面上传
登录harbor后选择具体的项目，可以看到Helm Charts的Tab，点击上传按钮选择对应的chart包，需要是tar文件，rar是不支持的，如果chart包有签名的还需要导入Prov文件。

（2）通过Helm CLI上传

首先，检查是否有helm-push插件（直接helm push看提示），如果没有先安装

# 在线安装
helm plugin install https://github.com/chartmuseum/helm-push

# 离线安装
# 先下载，看具体想要的版本可以在管网选择下载
wget https://github.com/chartmuseum/helm-push/releases/download/v0.7.1/helm-push_0.7.1_linux_amd64.tar.gz

# 将文件拷贝并解压到/root/.helm/plugins/下
tar -zxvf helm-push_0.7.1_linux_amd64.tar.gz && rm -rf helm-push_0.7.1_linux_amd64.tar.gz

然后，添加仓库

[root@SYSOPS00613596 ~]# helm repo add harbor http://harbor.****.com.cn/chartrepo --username=admin --password='**********'
[root@SYSOPS00613596 ~]# helm repo list
NAME    URL                                         
harbor  http://harbor.*****.com.cn/chartrepo

这里注意，如果是https的harbor仓库，需要指定ca和cert文件，具体参见harbor官方文档。

最后，上传chart包

# cd helm/
# ls
redis.tar.gz
# helm push redis.tar.gz harbor

三、部署Sentry

做了那么多前奏，终于要进入正题了，这就是离线安装的痛，本来的起点应该是这里才对：

1、检查依赖：postgres和redis

（1）如果按照我们上述的步骤完成了chartmuseum的建立，那么在Sentry的Chart包里面存在requirement.yaml文件，修改repository为内网的chartmuseum地址，检查修改name和version是否在内网chartmuseum中（helm search xxxxx）存在：

dependencies:
  - name: postgresql
    version: 6.5.0
    repository: http://harbor.****.com.cn/chartrepo/****/
    condition: postgresql.enabled
  - name: redis
    version: 9.3.2
    repository: http://harbor.****.com.cn/chartrepo/****/
    condition: redis.enabled

（2）如果内网没有chartmuseum，那么需要手动建pg和redis。

下面的操作主要针对没有统一的chartmuseum的情况

• 如果有chartmuseum，这一步可以忽略
• 如果本身你们已经有外部的postgres和redis了，这一步也可以忽略

先删除requirement.yaml和requirement.lock文件，这两文件是Sentry chart的依赖chart，因为我们没用chartmesium，所以没有用，而是改手工部署的redis和postgres，所以删除掉它。然后开始手工部署redis和postgres：

• （一）部署redis

  • 1.修改values.yaml文件

    • 修改image地址和标签为内网harbor仓库镜像地址，包括redis和minideb的镜像
    • 设置password，也可以不设置，那么就是默认10个随机字符，但如果自己设置，记得不要包含 # 号
    • 数据持久化，包括master和slave
      master.persistence.storageClass: "ceph-rbd"
slave.persistence.storageClass: "ceph-rbd"

  • 2.helm install

    helm install . --debug --name redis --namespace sentry
    

• （二）部署postgres

  • 1.修改values.yaml文件

    • 修改image地址和标签为内网harbor仓库镜像地址，包括postgres和minideb的镜像
    • 设置postgresqlPassword
    • 数据持久化，我们设置存到ceph rbd上
      persistence.storageClass: "ceph-rbd"

  • 2.helm install

    helm install . --debug --name postgres --namespace sentry
    

2、helm install

先dry-run一下，并将部署日志打印到yaml文件中确认都没有问题再部署，这一步很有用，方便做检查

helm install sentry harbor/sentry --debug --dry-run \
 --set email.host=smtp.****.com,email.port=24,email.user=****,email.password=***** \
 --set service.type=ClusterIP,ingress.enabled=true,ingress.hostname=sentry.****.com.cn \
 --set filestore.filesystem.persistence.enabled=true,filestore.filesystem.persistence.storageClass=ceph-rbd \
 --set user.email=*******,user.password=******,serviceAccount.create=false \
 --namespace sentry | sed 'w ../sentryt-dry-run.yml'

参数的话对照chart包的values.yaml文件看就明白了。这里稍微说下，我们是内网k8s集群，采用ingress做集群外部访问，所以service.type更改为ClusterIP，并且专门的namespace，用default的sa就好，所以也没创建新的sa。

仔细看sentry-dry-run文件，没有问题了再去掉–dry-run直接安装，并打印部署日志

helm install sentry harbor/sentry --debug  \
 --set email.host=smtp.****.com,email.port=24,email.user=****,email.password=***** \
 --set service.type=ClusterIP,ingress.enabled=true,ingress.hostname=sentry.****.com.cn \
 --set filestore.filesystem.persistence.enabled=true,filestore.filesystem.persistence.storageClass=ceph-rbd \
 --set user.email=*******,user.password=******,serviceAccount.create=false \
 --namespace sentry --wait | sed 'w ../sentryt-deploy.yml'

这里注意添加--wait参数，因为创建db时间会比较久，所以可能会超出默认等待时长。等待的时候可以不断检查pod状态以及日志看是否有问题。

PS：这里如果一致无法自动创建db，那么需要先进到postgres里面创建sentry db，否则会报错，具体操作看第五（1）

直到看到如下提示就标识安装成功了

NOTES:

1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace sentry -l "app=sentry,role=web" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward --namespace sentry $POD_NAME 8080:9000

2. Log in with

  USER: [email protected]
  Get login password with
    kubectl get secret --namespace sentry sentry -o jsonpath="{.data.user-password}" | base64 --decode

四、登录验证

用在values.yaml中你自己配置的用户名和密码登录验证即可，具体参见上一篇docker-compose部署Sentry
中第四部的登录验证，这里不再赘述。

安装中遇到的问题解决及总结

（1）数据库sentry不存在的问题：database "sentry" does not exist

[root@SYSOPS00613596 ~]# kubectl -n sentry logs sentry-db-init-bqkh7
11:38:41 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
11:38:44 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 162, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 36, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 75, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 30, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/usr/local/lib/python2.7/site-packages/south/management/commands/syncdb.py", line 119, in handle_noargs
    self.sync_apps(apps_needing_sync, app_name_to_app_map, options)
  File "/usr/local/lib/python2.7/site-packages/south/management/commands/syncdb.py", line 174, in sync_apps
    syncdb.Command().execute(**options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
    cursor = connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 162, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 44, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 95, in _cursor
    cursor = super(DatabaseWrapper, self)._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 132, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 115, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 115, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  database "sentry" does not exist

这里一直等待都不行，可能是因为外置数据库的原因，所以我们手动去postgres里面创建了sentry就可以了。

[root@ ~]# kubectl -n sentry exec -it postgresql-postgresql-0  bash
I have no name!@postgresql-postgresql-0:/$ psql --host 127.0.0.1 -U postgres -d postgres -p 5432
Password for user postgres: 
psql (11.7)
Type "help" for help.

postgres=# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(4 rows)

postgres=# create database sentry
postgres=# \q
I have no name!@postgresql-postgresql-0:/$ exit
exit

（2）cron容器报redis无法转换为整形的问题：invalid literal for int() with base 10: ‘redis123!’

[root@SYSOPS00613596 ~]# kubectl -n sentry logs sentry-cron-dd59f9b74-kcrwg
11:38:22 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
11:38:25 [INFO] sentry.plugins.github: apps-not-configured
celery beat v3.1.18 (Cipater) is starting.
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 162, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 73, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 36, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/run.py", line 252, in cron
    **options
  File "/usr/local/lib/python2.7/site-packages/celery/apps/beat.py", line 83, in run
    self.start_scheduler()
  File "/usr/local/lib/python2.7/site-packages/celery/apps/beat.py", line 104, in start_scheduler
    c.reset(self.startup_info(beat)))))
  File "/usr/local/lib/python2.7/site-packages/celery/apps/beat.py", line 127, in startup_info
    conninfo=self.app.connection().as_uri(),
  File "/usr/local/lib/python2.7/site-packages/celery/app/base.py", line 385, in connection
    'BROKER_CONNECTION_TIMEOUT', connect_timeout
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 167, in __init__
    params.update(parse_url(hostname))
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/url.py", line 32, in parse_url
    scheme, host, port, user, password, path, query = _parse_url(url)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/url.py", line 24, in _parse_url
    return (scheme, unquote(parts.hostname or '') or None, parts.port,
  File "/usr/local/lib/python2.7/urlparse.py", line 113, in port
    port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'redis123!'

这里我在redis里面配置的密码原来是’redis123!#%’，但是从报错来看，就是这个密码，而且就是这个#号引发的问题。Google大法搜了一下，应该是celery引发的问题，#在rabbitmq文档里是作为URI保留字符的，我找到了一段解释，可以参考看下：

Unable to use ‘#’ as broker url in celery django - django
I know the # is a problem because obviously if I remove that character from the password it works perfectly
The rabbitmq documentation for the uri refers to RFC3986. They have a section about reserved characters and # is one of them.
If data for a URI component would conflict with a reserved character’s purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed. According to the spec you might replace # with %23 - or just use a password that does not include reserved characters.
Update:
You are right, but the same RFC is followed there, too. Celery (or kombu) uses urllib and urllib tells they want to match the RFC, too. It crashes in this line and there your password gets interpreted as a port. Seems like the # inside of the password enforces the library to interpret the password as the port (which fails, as it is not castable to int). Note that domain and port are separated by the same character : as username and password.
The following illustrates whats going on.Note that everything after # is interpreted as the fragment of your url.

from urllib.parse import urlparse
url = 'amqp://user:pass#1#localhost:5672//'
urlparse(url)

ParseResult(scheme=‘amqp’, netloc=‘user:pass’, path=’’, params=’’, query=’’, fragment=‘1#localhost:5672//’)
See what happens when we remove the #

url = 'amqp://user:pass%231#localhost:5672//'
urlparse(url)

ParseResult(scheme=‘amqp’, netloc=‘user:pass%231#localhost:5672’, path=’//’, params=’’, query=’’, fragment=’’)

urlparse(url).port
5672
urlparse(url).password
'pass%231'

The url can be parsed correctly - but I guess the password is wrong now. Sadly I can not find any sources that describe how to escape something in the password of an URI. But to be honest - this sounds weired. Escape characters in a password? I would recommend to just pick a password without # as this confuses pythons URL parser and most likely other implementations, too.

也就是说#号是在celery django的broker url里面是分割password和host的，如果你在password里面用了#，导致解析出了问题。如果你硬是要用，也可以，使用%23代替

也有issue说这个在python老的版本会引发，在python 2.7.9之后不再出现。参见：ValueError: invalid literal for int() with base 10: ’ ’ #2673

所以解决方案有三个，一是改redis密码，二是将#改为%23，三是升级python或者指定Sentry运行的python版本，。显而易见，方案一简单易行，哈哈。所以把redis密码的#号去掉或者改为别的字符就好了。

（3）和docker-compose下的区别

对比docker-compose部署，我看到的区别没有单独架设smtp和memcached容器，而是采用内置的服务去处理了。

---

参考文章：

1. Sentry官方文档
2. Sentry git仓库
3. Sentry Helm chart
4. redis helm chart
5. postgres helm chart
6. Sentry实时应用错误跟踪系统在Kubernetes中私有化部署