redash迁移

工作中，需要把一台自有实体机上的redash迁移到公司容器中。目标容器系统为centos6。迁移过程如下：

redash基础环境搭建

redis

• 安装redis

cd /opt/xxx/apps
[xxx@set-xxx-xxx02 apps]$ mkdir redis
wget http://download.redis.io/releases/redis-5.0.5.tar.gz
tar -zxvf redis-5.0.5.tar.gz
cd redis-5.0.5
make

• 启动redis

cd redis-5.0.5/src
./redis-server

• 测试redis启动ok

cd redis-5.0.5/src
./redis-cli
127.0.0.1:6379> ping
PONG

PostgreSQL

以下操作为root权限

• 安装PostgreSQL

sudo yum install -y postgresql96-server

• 启动PostgreSQL

pg_ctl -D /opt/xxx/pgdata/ start

若启动报错：

[xxx@set-xxx-xxx02 ~]$ < 2019-07-22 11:33:20.555 CST > FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Permission denied

解决方法：给other用户加写权限

sudo chmod a+w /var/run/postgresql

然后再次启动。
进入psql报错：

psql
psql: FATAL:  database "xxx" does not exist

解决方法： https://stackoverflow.com/questions/17633422/psql-fatal-database-user-does-not-exist

[xxx@set-xxx-xxx02 ~]$ createdb xxx
[xxx@set-xxx-xxx02 ~]$ psql
psql (9.6.14)
Type "help" for help.

xxx=#

修改密码:

xxx=# \password
Enter new password:
Enter it again:
xxx=#

• 可执行文件加入到xxx用户的环境变量，修改/home/xxx/.bash_profile

export PATH=/usr/pgsql-9.6/bin:$PATH

source /home/xxx/.bash_profile

nodejs

• 安装nodejs

cd /opt/xxx/apps
wget https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.xz
tar -xvf node-v10.9.0-linux-x64.tar.xz
mv node-v10.9.0-linux-x64 nodejs
ln -s /opt/xxx/apps/nodejs/bin/npm /usr/local/bin/
ln -s /opt/xxx/apps/nodejs/bin/node /usr/local/bin/

Python2.7

以下操作为root权限

• 安装Python2.7
  https://tecadmin.net/install-python-2-7-on-centos-rhel/

yum install -y gcc openssl-devel bzip2-devel
cd /home/xxx/
wget https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tgz
tar xzf Python-2.7.16.tgz
cd Python-2.7.16
./configure --enable-optimizations
make altinstall

• 安装pip

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python2.7 get-pip.py

• 可执行文件加入到xxx用户的环境变量，修改/home/xxx/.bash_profile

export PATH=/usr/local/bin:/usr/pgsql-9.6/bin:$PATH

source /home/xxx/.bash_profile

• 安装Python隔离环境

pip install virtualenv
virtualenv -p /usr/local/bin/python2.7 current

原redash代码和数据备份，并copy到新机器上

• 数据备份

pg_dump -U postgres -d postgres -f postgres.sql
pg_dump -U postgres -d qa_mgmt -f qa_mgmt.sql
pg_dump -U postgres -d redash -f redash.sql

redash

• 安装python包

source /home/xxx/current/bin/activate
cd /home/xxx/redash-7.0.0
pip install -r requirements.txt -r requirements_dev.txt -r requirements_all_ds.tx

• 安装nodejs包

cd /home/xxx/redash-7.0.0/client
npm install
npm run build

• 还远原数据库数据到新机器上的postgresql中

psql -U xxx -d redash -f redash.sql
psql -U xxx -d qa_mgmt -f qa_mgmt.sql
psql -U xxx -d postgres -f postgres.sql

• 配置.env

REDASH_DATABASE_URL="postgresql://xxx:xxxxxx@localhost/redash"

Supervisor

以下为root权限

• 安装Supervisor

pip install supervisor

• 创建配置文件

echo_supervisord_conf > /etc/supervisord.conf

• 修改配置文件
  修改以下几处：

[unix_http_server]
file=/var/run/supervisor.sock   ; the path to the socket file
[supervisord]
logfile=/var/log/supervisord.log ; main log file; default $CWD/supervisord.log
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket
[include]
files = /etc/supervisor.d/*.conf

• 加写权限

chmod 777 /var/run/supervisor.sock
chmod 777 /var/log/supervisord.log

• 进程配置

mkdir -p /etc/supervisor.d
touch redash.conf

文件内容从原机器的.conf中copy出，主要改以下几个地方：目录替换为新目录，用户替换为xxx

[group:redash]
programs=redash_server,redash_celery,redash_celery_scheduled

[program:redash_server]
#environment=PYTHONPATH=/home/xxx/current/bin:%(ENV_PYTHONPATH)s,PATH=/home/xxx/current/bin:%(ENV_PATH)s
command=/home/xxx/redash-7.0.0/bin/run gunicorn -b 0.0.0.0:5000 --name redash -w 4 --max-requests 1000 redash.wsgi:app
directory=/home/xxx/redash-7.0.0
process_name=redash_server
user=xxx
numprocs=1
autostart=true
autorestart=true
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 20
stdout_logfile = /home/xxx/redash-7.0.0/logs/redash_server_stdout.log

# There are two queue types here: one for ad-hoc queries, and one for the refresh of scheduled queries
# (note that "scheduled_queries" appears only in the queue list of "redash_celery_scheduled").
# The default concurrency level for each is 2 (-c2), you can increase based on your machine's resources.

[program:redash_celery]
#environment=PYTHONPATH=/home/xxx/current/bin:%(ENV_PYTHONPATH)s,PATH=/home/xxx/current/bin:%(ENV_PATH)s
command=/home/xxx/redash-7.0.0/bin/run celery worker --app=redash.worker --beat -c2 -Qqueries,celery --maxtasksperchild=10 -Ofair
directory=/home/xxx/redash-7.0.0
process_name=redash_celery
user=xxx
numprocs=1
autostart=true
autorestart=true
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 20
stdout_logfile = /home/xxx/redash-7.0.0/logs/redash_celery_stdout.log

[program:redash_celery_scheduled]
#environment=PYTHONPATH=/home/xxx/current/bin:%(ENV_PYTHONPATH)s,PATH=/home/xxx/current/bin:%(ENV_PATH)s
command=/home/xxx/redash-7.0.0/bin/run celery worker --app=redash.worker -c2 -Qscheduled_queries --maxtasksperchild=10 -Ofair
directory=/home/xxx/redash-7.0.0
process_name=redash_celery_scheduled
user=xxx
numprocs=1
autostart=true
autorestart=true
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 20
stdout_logfile = /home/xxx/redash-7.0.0/logs/redash_celery_scheduled_stdout.log

启动redash

supervisord -c /etc/supervisord.conf

遇到的问题

• redash启动调试服务 ./bin/run python manage.py run --host=0.0.0.0报错

[2019-07-22 16:06:08,618][PID:5136][DEBUG][redash.query_runner] TreasureData query runner enabled but not supported, not registering. Either disable or install missing dependencies.
Traceback (most recent call last):
  File "manage.py", line 6, in 
    from redash.cli import manager
  File "/home/xxx/redash-7.0.0/redash/__init__.py", line 80, in 
    import_query_runners(settings.QUERY_RUNNERS)
  File "/home/xxx/redash-7.0.0/redash/query_runner/__init__.py", line 271, in import_query_runners
    __import__(runner_import)
  File "/home/xxx/redash-7.0.0/redash/query_runner/sqlite.py", line 2, in 
    import sqlite3
  File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in 
    from dbapi2 import *
  File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 28, in 
    from _sqlite3 import *
ImportError: No module named _sqlite3

解决方法：

https://stackoverflow.com/questions/1210664/no-module-named-sqlite3
(1)yum install -y sqlite-devel
(2)重新编译安装Python2.7

• python sasl包安装失败

yum install -y cyrus-sasl-devel