记一次有关SQLAlchemy V2版本的shcema问题查找

SQLAlchemy 2.0版本已经发布有一段时间了，最近在测试的过程中发现在PostGIS数据库迁移过程中有一些问题，接下来回顾一下整个问题的修复过程，整个过程持续大概一周时间

提出问题

环境准备：

pip install -U alembic==1.11.1 Flask-Migrate==4.0.4 Flask-SQLAlchemy==3.0.5 GeoAlchemy2==0.14.0 SQLAlchemy==2.0.19 psycopg2-binary==2.9.3

docker run -d --name=postgis-test -p 5432:5432 -e POSTGRES_PASSWORD=123456 -e POSTGRES_DB=test  postgis/postgis:12-3.2

测试代码：

from flask import Flask
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy
from geoalchemy2 import Geometry
from geoalchemy2.alembic_helpers import include_object, render_item

app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = "postgresql://postgres:123456@localhost:5432/test"
db = SQLAlchemy(app)

Migrate(app=app, db=db, compare_type=True, include_object=include_object, render_item=render_item)


class Lake(db.Model):
    __tablename__ = 'lake'
    id = db.Column(db.Integer, primary_key=True)
    geom = db.Column(
        Geometry(
            geometry_type='LINESTRING',
            srid=4326,
            spatial_index=True,
        )
    )

测试步骤：

1. flask db init
2. flask db migrate

输出：

(venv) D:\workspace\test\test_sqlalchemy>flask db init
Creating directory D:\workspace\test\test_sqlalchemy\migrations ...  done
Creating directory D:\workspace\test\test_sqlalchemy\migrations\versions ...  done
Generating D:\workspace\test\test_sqlalchemy\migrations\alembic.ini ...  done
Generating D:\workspace\test\test_sqlalchemy\migrations\env.py ...  done
Generating D:\workspace\test\test_sqlalchemy\migrations\README ...  done
Generating D:\workspace\test\test_sqlalchemy\migrations\script.py.mako ...  done
Please edit configuration/connection/logging settings in 'D:\\workspace\\test\\test_sqlalchemy\\migrations\\alembic.ini' before proceeding.

(venv) D:\workspace\test\test_sqlalchemy>flask db migrate
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'lake'
INFO  [alembic.autogenerate.compare] Detected added index 'idx_lake_geom' on '['geom']'
INFO  [alembic.ddl.postgresql] Detected sequence named 'faces_gid_seq' as owned by integer column 'faces(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_faces_countyfp' on 'faces'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_faces_tfid' on 'faces'
INFO  [alembic.autogenerate.compare] Detected removed index 'tiger_faces_the_geom_gist' on 'faces'
INFO  [alembic.autogenerate.compare] Detected removed table 'faces'
INFO  [alembic.ddl.postgresql] Detected sequence named 'addrfeat_gid_seq' as owned by integer column 'addrfeat(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_addrfeat_geom_gist' on 'addrfeat'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_addrfeat_tlid' on 'addrfeat'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_addrfeat_zipl' on 'addrfeat'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_addrfeat_zipr' on 'addrfeat'
INFO  [alembic.autogenerate.compare] Detected removed table 'addrfeat'
INFO  [alembic.autogenerate.compare] Detected removed table 'zip_lookup_base'
INFO  [alembic.ddl.postgresql] Detected sequence named 'pagc_gaz_id_seq' as owned by integer column 'pagc_gaz(id)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed table 'pagc_gaz'
INFO  [alembic.ddl.postgresql] Detected sequence named 'cousub_gid_seq' as owned by integer column 'cousub(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'tige_cousub_the_geom_gist' on 'cousub'
INFO  [alembic.autogenerate.compare] Detected removed table 'cousub'
INFO  [alembic.autogenerate.compare] Detected removed table 'zip_lookup_all'
INFO  [alembic.autogenerate.compare] Detected removed index 'countysub_lookup_name_idx' on 'countysub_lookup'
INFO  [alembic.autogenerate.compare] Detected removed index 'countysub_lookup_state_idx' on 'countysub_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'countysub_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'zip_state_loc'
INFO  [alembic.autogenerate.compare] Detected removed table 'layer'
INFO  [alembic.ddl.postgresql] Detected sequence named 'pagc_lex_id_seq' as owned by integer column 'pagc_lex(id)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed table 'pagc_lex'
INFO  [alembic.autogenerate.compare] Detected removed index 'county_lookup_name_idx' on 'county_lookup'
INFO  [alembic.autogenerate.compare] Detected removed index 'county_lookup_state_idx' on 'county_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'county_lookup'
INFO  [alembic.ddl.postgresql] Detected sequence named 'county_gid_seq' as owned by integer column 'county(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_county' on 'county'
INFO  [alembic.autogenerate.compare] Detected removed table 'county'
INFO  [alembic.autogenerate.compare] Detected removed table 'loader_platform'
INFO  [alembic.ddl.postgresql] Detected sequence named 'bg_gid_seq' as owned by integer column 'bg(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed table 'bg'
INFO  [alembic.ddl.postgresql] Detected sequence named 'zcta5_gid_seq' as owned by integer column 'zcta5(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed table 'zcta5'
INFO  [alembic.autogenerate.compare] Detected removed index 'direction_lookup_abbrev_idx' on 'direction_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'direction_lookup'
INFO  [alembic.ddl.postgresql] Detected sequence named 'addr_gid_seq' as owned by integer column 'addr(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_addr_tlid_statefp' on 'addr'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_addr_zip' on 'addr'
INFO  [alembic.autogenerate.compare] Detected removed table 'addr'
INFO  [alembic.autogenerate.compare] Detected removed table 'state_lookup'
INFO  [alembic.ddl.postgresql] Detected sequence named 'pagc_rules_id_seq' as owned by integer column 'pagc_rules(id)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed table 'pagc_rules'
INFO  [alembic.ddl.postgresql] Detected sequence named 'state_gid_seq' as owned by integer column 'state(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_state_the_geom_gist' on 'state'
INFO  [alembic.autogenerate.compare] Detected removed table 'state'
INFO  [alembic.ddl.postgresql] Detected sequence named 'featnames_gid_seq' as owned by integer column 'featnames(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_featnames_lname' on 'featnames'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_featnames_snd_name' on 'featnames'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_featnames_tlid_statefp' on 'featnames'
INFO  [alembic.autogenerate.compare] Detected removed table 'featnames'
INFO  [alembic.autogenerate.compare] Detected removed table 'loader_lookuptables'
INFO  [alembic.autogenerate.compare] Detected removed table 'tabblock20'
INFO  [alembic.autogenerate.compare] Detected removed table 'geocode_settings_default'
INFO  [alembic.ddl.postgresql] Detected sequence named 'tract_gid_seq' as owned by integer column 'tract(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed table 'tract'
INFO  [alembic.ddl.postgresql] Detected sequence named 'tabblock_gid_seq' as owned by integer column 'tabblock(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed table 'tabblock'
INFO  [alembic.autogenerate.compare] Detected removed index 'secondary_unit_lookup_abbrev_idx' on 'secondary_unit_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'secondary_unit_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'loader_variables'
INFO  [alembic.autogenerate.compare] Detected removed table 'geocode_settings'
INFO  [alembic.ddl.postgresql] Detected sequence named 'place_gid_seq' as owned by integer column 'place(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'tiger_place_the_geom_gist' on 'place'
INFO  [alembic.autogenerate.compare] Detected removed table 'place'
INFO  [alembic.autogenerate.compare] Detected removed index 'street_type_lookup_abbrev_idx' on 'street_type_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'street_type_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'zip_state'
INFO  [alembic.ddl.postgresql] Detected sequence named 'edges_gid_seq' as owned by integer column 'edges(gid)', assuming SERIAL and omitting
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_edges_tlid' on 'edges'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_edges_countyfp' on 'edges'
INFO  [alembic.autogenerate.compare] Detected removed index 'idx_tiger_edges_the_geom_gist' on 'edges'
INFO  [alembic.autogenerate.compare] Detected removed table 'edges'
INFO  [alembic.autogenerate.compare] Detected removed table 'zip_lookup'
INFO  [alembic.autogenerate.compare] Detected removed index 'place_lookup_name_idx' on 'place_lookup'
INFO  [alembic.autogenerate.compare] Detected removed index 'place_lookup_state_idx' on 'place_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'place_lookup'
INFO  [alembic.autogenerate.compare] Detected removed table 'topology'
Generating D:\workspace\test\test_sqlalchemy\migrations\versions\7375114c2e0a_.py ...  done

原因分析

• Postgis 似乎将搜索路径设置为 "$user", public, topology, tiger 而不是默认路径。
• 在检测数据库表时检测了所有schema的变化，往往我们仅需要检测public中的表

解决方案

sqlalchemy和alembic的作者们给了两种方案：
第一种：直接更改Postgis的搜索路径：

https://docs.sqlalchemy.org/en/20/dialects/postgresql.html#setting-alternate-search-paths-on-connect

第二种：使用include_schemas=True和include_name两个配置项：

def include_name(name, type_, parent_names):
    if type_ == "schema":
        return False
    else:
        return True


Migrate(app=app, db=db, compare_type=True, include_object=include_object, render_item=render_item, include_schemas=True, include_name=include_name)

总的来说使用第二种方案更方便使用

整个讨论过程：

https://github.com/sqlalchemy/alembic/discussions/1282