项目背景
项目需要适配国产数据库,以下是适配达梦数据库时遇到的一些问题
配置达梦数据库
通过引入Dm8JdbcDriver18依赖配置数据库driver-class,配置数据库连接信息后,mybatis能进行crud操作,但是某些特殊属性及函数是无法使用的,例如 domain,link等。
依赖:
com.dameng
Dm8JdbcDriver18
8.1.1.49
配置:
driver-class-name: dm.jdbc.driver.DmDriver
url: jdbc:dm://xxxx:x/xx
username: xx
password: xx
type: com.alibaba.druid.pool.DruidDataSource
Flowable集成达梦数据库的使用时遇到的问题
- 自动加载数据源时,无法识别DM数据库类型,采用覆盖源码,设置成MySQL或者Oracle的数据库类型
覆盖org.flowable.common.engine.impl.AbstractEngineConfiguration
修改getDefaultDatabaseTypeMappings,新增达梦数据库类型
databaseTypeMappings.setProperty("DM DBMS", DATABASE_TYPE_ORACLE);
-
自动创建act和flw开头的表,需要将初始化语句修改,手动执行;或者将MySQL或Oracle数据表初始化后,手动导入DM数据库,达梦数据库提供了数据迁移工具
执行过程中遇到数据类型不匹配的问题,例如程序启动时,flowable会通过Liquibase去修改FLW_EV_DATABASECHANGELOGLOCK表数据,由于DM无法转换true/false报错
报错信息:
Caused by: liquibase.exception.DatabaseException: Error executing SQL UPDATE FLW_EV_DATABASECHANGELOGLOCK SET LOCKED = TRUE, LOCKEDBY = 'LAPTOP-RO0A74NR (192.168.6.154)', LOCKGRANTED = '2020-08-06 17:06:10.275' WHERE ID = 1 AND LOCKED = FALSE: 第1 行附近出现错误: 数据类型不匹配
第一步需要创建新的class,继承AbstractJdbcDatabase,并且放在liquibase.database.core包下,这样Liquibase才会识别DM DBMS类型,这里是仿照OracleDatabase的做的
package liquibase.database.core;
import liquibase.CatalogAndSchema;
import liquibase.Scope;
import liquibase.database.AbstractJdbcDatabase;
import liquibase.database.DatabaseConnection;
import liquibase.database.OfflineConnection;
import liquibase.database.jvm.JdbcConnection;
import liquibase.exception.DatabaseException;
import liquibase.exception.UnexpectedLiquibaseException;
import liquibase.exception.ValidationErrors;
import liquibase.executor.ExecutorService;
import liquibase.logging.LogService;
import liquibase.logging.LogType;
import liquibase.statement.DatabaseFunction;
import liquibase.statement.SequenceCurrentValueFunction;
import liquibase.statement.SequenceNextValueFunction;
import liquibase.statement.core.RawCallStatement;
import liquibase.statement.core.RawSqlStatement;
import liquibase.structure.DatabaseObject;
import liquibase.structure.core.Catalog;
import liquibase.structure.core.Index;
import liquibase.structure.core.PrimaryKey;
import liquibase.structure.core.Schema;
import liquibase.util.JdbcUtils;
import liquibase.util.StringUtils;
import java.lang.reflect.Method;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Properties;
import java.util.ResourceBundle;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import static java.util.ResourceBundle.getBundle;
/**
* Encapsulates Oracle database support.
*/
public class DmDatabase extends AbstractJdbcDatabase {
private static final Pattern PROXY_USER = Pattern.compile(".*(?:thin|oci)\\:(.+)/@.*");
public static final String PRODUCT_NAME = "DM DBMS";
private static ResourceBundle coreBundle = getBundle("liquibase/i18n/liquibase-core");
protected final int SHORT_IDENTIFIERS_LENGTH = 30;
protected final int LONG_IDENTIFIERS_LEGNTH = 128;
public static final int ORACLE_12C_MAJOR_VERSION = 12;
private Set reservedWords = new HashSet<>();
private Set userDefinedTypes;
private Map savedSessionNlsSettings;
private Boolean canAccessDbaRecycleBin;
private Integer databaseMajorVersion;
private Integer databaseMinorVersion;
/**
* Default constructor for an object that represents the Oracle Database DBMS.
*/
public DmDatabase() {
super.unquotedObjectsAreUppercased = true;
//noinspection HardCodedStringLiteral
super.setCurrentDateTimeFunction("SYSTIMESTAMP");
// Setting list of Oracle's native functions
//noinspection HardCodedStringLiteral
dateFunctions.add(new DatabaseFunction("SYSDATE"));
//noinspection HardCodedStringLiteral
dateFunctions.add(new DatabaseFunction("SYSTIMESTAMP"));
//noinspection HardCodedStringLiteral
dateFunctions.add(new DatabaseFunction("CURRENT_TIMESTAMP"));
//noinspection HardCodedStringLiteral
super.sequenceNextValueFunction = "%s.nextval";
//noinspection HardCodedStringLiteral
super.sequenceCurrentValueFunction = "%s.currval";
}
@Override
public int getPriority() {
return PRIORITY_DEFAULT;
}
private final void tryProxySessionn(final String url, final Connection con) {
Matcher m = PROXY_USER.matcher(url);
if (m.matches()) {
Properties props = new Properties();
props.put("PROXY_USER_NAME", m.group(1));
try {
Method method = con.getClass().getMethod("openProxySession", int.class, Properties.class);
method.setAccessible(true);
method.invoke(con, 1, props);
} catch (Exception e) {
Scope.getCurrentScope().getLog(getClass()).info(LogType.LOG, "Could not open proxy session on OracleDatabase: " + e.getCause().getMessage());
}
}
}
@Override
public void setConnection(DatabaseConnection conn) {
//noinspection HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral,
// HardCodedStringLiteral
//noinspection HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral,
// HardCodedStringLiteral
reservedWords.addAll(Arrays.asList("GROUP", "USER", "SESSION", "PASSWORD", "RESOURCE", "START", "SIZE", "UID", "DESC", "ORDER")); //more reserved words not returned by driver
Connection sqlConn = null;
if (!(conn instanceof OfflineConnection)) {
try {
/*
* Don't try to call getWrappedConnection if the conn instance is
* is not a JdbcConnection. This happens for OfflineConnection.
* see https://liquibase.jira.com/browse/CORE-2192
*/
if (conn instanceof JdbcConnection) {
sqlConn = ((JdbcConnection) conn).getWrappedConnection();
}
} catch (Exception e) {
throw new UnexpectedLiquibaseException(e);
}
if (sqlConn != null) {
tryProxySessionn(conn.getURL(), sqlConn);
try {
//noinspection HardCodedStringLiteral
reservedWords.addAll(Arrays.asList(sqlConn.getMetaData().getSQLKeywords().toUpperCase().split(",\\s*")));
} catch (SQLException e) {
//noinspection HardCodedStringLiteral
LogService.getLog(getClass()).info(LogType.LOG, "Could get sql keywords on OracleDatabase: " + e.getMessage());
//can not get keywords. Continue on
}
try {
Method method = sqlConn.getClass().getMethod("setRemarksReporting", Boolean.TYPE);
method.setAccessible(true);
method.invoke(sqlConn, true);
} catch (Exception e) {
//noinspection HardCodedStringLiteral
LogService.getLog(getClass()).info(LogType.LOG, "Could not set remarks reporting on OracleDatabase: " + e.getMessage());
//cannot set it. That is OK
}
Statement statement = null;
ResultSet resultSet = null;
try {
statement = sqlConn.createStatement();
//noinspection HardCodedStringLiteral
resultSet = statement.executeQuery("SELECT value FROM v$parameter WHERE name = 'compatible'");
String compatibleVersion = null;
if (resultSet.next()) {
//noinspection HardCodedStringLiteral
compatibleVersion = resultSet.getString("value");
}
if (compatibleVersion != null) {
Matcher majorVersionMatcher = Pattern.compile("(\\d+)\\.(\\d+)\\..*").matcher(compatibleVersion);
if (majorVersionMatcher.matches()) {
this.databaseMajorVersion = Integer.valueOf(majorVersionMatcher.group(1));
this.databaseMinorVersion = Integer.valueOf(majorVersionMatcher.group(2));
}
}
} catch (SQLException e) {
@SuppressWarnings("HardCodedStringLiteral") String message = "Cannot read from v$parameter: " + e.getMessage();
//noinspection HardCodedStringLiteral
LogService.getLog(getClass()).info(LogType.LOG, "Could not set check compatibility mode on OracleDatabase, assuming not running in any sort of compatibility mode: " + message);
} finally {
JdbcUtils.close(resultSet, statement);
}
}
}
super.setConnection(conn);
}
@Override
public String getShortName() {
//noinspection HardCodedStringLiteral
return "dm";
}
@Override
protected String getDefaultDatabaseProductName() {
//noinspection HardCodedStringLiteral
return "dm";
}
@Override
public int getDatabaseMajorVersion() throws DatabaseException {
if (databaseMajorVersion == null) {
return super.getDatabaseMajorVersion();
} else {
return databaseMajorVersion;
}
}
@Override
public int getDatabaseMinorVersion() throws DatabaseException {
if (databaseMinorVersion == null) {
return super.getDatabaseMinorVersion();
} else {
return databaseMinorVersion;
}
}
@Override
public Integer getDefaultPort() {
return 5236;
}
@Override
public String getJdbcCatalogName(CatalogAndSchema schema) {
return null;
}
@Override
public String getJdbcSchemaName(CatalogAndSchema schema) {
return correctObjectName((schema.getCatalogName() == null) ? schema.getSchemaName() : schema.getCatalogName(), Schema.class);
}
@Override
protected String getAutoIncrementClause(final String generationType, final Boolean defaultOnNull) {
if (StringUtils.isEmpty(generationType)) {
return "";
}
String autoIncrementClause = "GENERATED %s AS IDENTITY"; // %s -- [ ALWAYS | BY DEFAULT [ ON NULL ] ]
String generationStrategy = generationType;
if (Boolean.TRUE.equals(defaultOnNull) && generationType.toUpperCase().equals("BY DEFAULT")) {
generationStrategy += " ON NULL";
}
return String.format(autoIncrementClause, generationStrategy);
}
@Override
public String generatePrimaryKeyName(String tableName) {
if (tableName.length() > 27) {
//noinspection HardCodedStringLiteral
return "PK_" + tableName.toUpperCase(Locale.US).substring(0, 27);
} else {
//noinspection HardCodedStringLiteral
return "PK_" + tableName.toUpperCase(Locale.US);
}
}
@Override
public boolean supportsInitiallyDeferrableColumns() {
return true;
}
@Override
public boolean isReservedWord(String objectName) {
return reservedWords.contains(objectName.toUpperCase());
}
@Override
public boolean supportsSequences() {
return true;
}
/**
* Oracle supports catalogs in liquibase terms
*
* @return false
*/
@Override
public boolean supportsSchemas() {
return false;
}
@Override
protected String getConnectionCatalogName() throws DatabaseException {
if (getConnection() instanceof OfflineConnection) {
return getConnection().getCatalog();
}
try {
//noinspection HardCodedStringLiteral
return ExecutorService.getInstance().getExecutor(this).queryForObject(new RawCallStatement("select sys_context( 'userenv', 'current_schema' ) from dual"), String.class);
} catch (Exception e) {
//noinspection HardCodedStringLiteral
LogService.getLog(getClass()).info(LogType.LOG, "Error getting default schema", e);
}
return null;
}
@Override
public boolean isCorrectDatabaseImplementation(DatabaseConnection conn) throws DatabaseException {
return PRODUCT_NAME.equalsIgnoreCase(conn.getDatabaseProductName());
}
@Override
public String getDefaultDriver(String url) {
//noinspection HardCodedStringLiteral
if (url.startsWith("jdbc:dm")) {
return "dm.jdbc.driver.DmDriver";
}
return null;
}
@Override
public String getDefaultCatalogName() {//NOPMD
return (super.getDefaultCatalogName() == null) ? null : super.getDefaultCatalogName().toUpperCase(Locale.US);
}
/**
* Returns an Oracle date literal with the same value as a string formatted using ISO 8601.
*
* Convert an ISO8601 date string to one of the following results:
* to_date('1995-05-23', 'YYYY-MM-DD')
* to_date('1995-05-23 09:23:59', 'YYYY-MM-DD HH24:MI:SS')
*
* Implementation restriction:
* Currently, only the following subsets of ISO8601 are supported:
*
* - YYYY-MM-DD
* - YYYY-MM-DDThh:mm:ss
*
*/
@Override
public String getDateLiteral(String isoDate) {
String normalLiteral = super.getDateLiteral(isoDate);
if (isDateOnly(isoDate)) {
StringBuffer val = new StringBuffer();
//noinspection HardCodedStringLiteral
val.append("TO_DATE(");
val.append(normalLiteral);
//noinspection HardCodedStringLiteral
val.append(", 'YYYY-MM-DD')");
return val.toString();
} else if (isTimeOnly(isoDate)) {
StringBuffer val = new StringBuffer();
//noinspection HardCodedStringLiteral
val.append("TO_DATE(");
val.append(normalLiteral);
//noinspection HardCodedStringLiteral
val.append(", 'HH24:MI:SS')");
return val.toString();
} else if (isTimestamp(isoDate)) {
StringBuffer val = new StringBuffer(26);
//noinspection HardCodedStringLiteral
val.append("TO_TIMESTAMP(");
val.append(normalLiteral);
//noinspection HardCodedStringLiteral
val.append(", 'YYYY-MM-DD HH24:MI:SS.FF')");
return val.toString();
} else if (isDateTime(isoDate)) {
int seppos = normalLiteral.lastIndexOf('.');
if (seppos != -1) {
normalLiteral = normalLiteral.substring(0, seppos) + "'";
}
StringBuffer val = new StringBuffer(26);
//noinspection HardCodedStringLiteral
val.append("TO_DATE(");
val.append(normalLiteral);
//noinspection HardCodedStringLiteral
val.append(", 'YYYY-MM-DD HH24:MI:SS')");
return val.toString();
}
//noinspection HardCodedStringLiteral
return "UNSUPPORTED:" + isoDate;
}
@Override
public boolean isSystemObject(DatabaseObject example) {
if (example == null) {
return false;
}
if (this.isLiquibaseObject(example)) {
return false;
}
if (example instanceof Schema) {
//noinspection HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral
if ("SYSTEM".equals(example.getName()) || "SYS".equals(example.getName()) || "CTXSYS".equals(example.getName()) || "XDB".equals(example.getName())) {
return true;
}
//noinspection HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral
if ("SYSTEM".equals(example.getSchema().getCatalogName()) || "SYS".equals(example.getSchema().getCatalogName()) || "CTXSYS".equals(example.getSchema().getCatalogName()) || "XDB".equals(example.getSchema().getCatalogName())) {
return true;
}
} else if (isSystemObject(example.getSchema())) {
return true;
}
if (example instanceof Catalog) {
//noinspection HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral,HardCodedStringLiteral
if (("SYSTEM".equals(example.getName()) || "SYS".equals(example.getName()) || "CTXSYS".equals(example.getName()) || "XDB".equals(example.getName()))) {
return true;
}
} else if (example.getName() != null) {
//noinspection HardCodedStringLiteral
if (example.getName().startsWith("BIN$")) { //oracle deleted table
boolean filteredInOriginalQuery = this.canAccessDbaRecycleBin();
if (!filteredInOriginalQuery) {
filteredInOriginalQuery = StringUtils.trimToEmpty(example.getSchema().getName()).equalsIgnoreCase(this.getConnection().getConnectionUserName());
}
if (filteredInOriginalQuery) {
return !((example instanceof PrimaryKey) || (example instanceof Index) || (example instanceof
liquibase.statement.UniqueConstraint));
} else {
return true;
}
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("AQ$")) { //oracle AQ tables
return true;
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("DR$")) { //oracle index tables
return true;
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("SYS_IOT_OVER")) { //oracle system table
return true;
} else //noinspection HardCodedStringLiteral,HardCodedStringLiteral
if ((example.getName().startsWith("MDRT_") || example.getName().startsWith("MDRS_")) && example.getName().endsWith("$")) {
// CORE-1768 - Oracle creates these for spatial indices and will remove them when the index is removed.
return true;
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("MLOG$_")) { //Created by materliaized view logs for every table that is part of a materialized view. Not available for DDL operations.
return true;
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("RUPD$_")) { //Created by materialized view log tables using primary keys. Not available for DDL operations.
return true;
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("WM$_")) { //Workspace Manager backup tables.
return true;
} else //noinspection HardCodedStringLiteral
if ("CREATE$JAVA$LOB$TABLE".equals(example.getName())) { //This table contains the name of the Java object, the date it was loaded, and has a BLOB column to store the Java object.
return true;
} else //noinspection HardCodedStringLiteral
if ("JAVA$CLASS$MD5$TABLE".equals(example.getName())) { //This is a hash table that tracks the loading of Java objects into a schema.
return true;
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("ISEQ$$_")) { //System-generated sequence
return true;
} else //noinspection HardCodedStringLiteral
if (example.getName().startsWith("USLOG$")) { //for update materialized view
return true;
} else if (example.getName().startsWith("SYS_FBA")) { //for Flashback tables
return true;
}
}
return super.isSystemObject(example);
}
@Override
public boolean supportsTablespaces() {
return true;
}
@Override
public boolean supportsAutoIncrement() {
// Oracle supports Identity beginning with version 12c
boolean isAutoIncrementSupported = false;
try {
if (getDatabaseMajorVersion() >= 12) {
isAutoIncrementSupported = true;
}
// Returning true will generate create table command with 'IDENTITY' clause, example:
// CREATE TABLE AutoIncTest (IDPrimaryKey NUMBER(19) GENERATED BY DEFAULT AS IDENTITY NOT NULL, TypeID NUMBER(3) NOT NULL, Description NVARCHAR2(50), CONSTRAINT PK_AutoIncTest PRIMARY KEY (IDPrimaryKey));
// While returning false will continue to generate create table command without 'IDENTITY' clause, example:
// CREATE TABLE AutoIncTest (IDPrimaryKey NUMBER(19) NOT NULL, TypeID NUMBER(3) NOT NULL, Description NVARCHAR2(50), CONSTRAINT PK_AutoIncTest PRIMARY KEY (IDPrimaryKey));
} catch (DatabaseException ex) {
isAutoIncrementSupported = false;
}
return isAutoIncrementSupported;
}
// public Set findUniqueConstraints(String schema) throws DatabaseException {
// Set returnSet = new HashSet();
//
// List
第二步由于达梦数据库不能自动转换true/false为1/0,这里我覆盖了com.alibaba.druid.pool.DruidPooledStatement,将其中的executeUpdate和execute方法中待执行的SQL语句做了转换。
@Override
public final boolean execute(String sql) throws SQLException {
checkOpen();
incrementExecuteCount();
transactionRecord(sql);
try {
if (StringUtils.isNotEmpty(sql)){
sql = sql.replace("TRUE", "1");
sql = sql.replace("FALSE", "0");
}
return stmt.execute(sql);
} catch (Throwable t) {
errorCheck(t);
throw checkException(t, sql);
}
}
MySQL迁移达梦数据库遇到的问题
检查后发现建表语句,'CURRENT_TIMESTAMP(3)' 错误,应改为 CURRENT_TIMESTAMP
查询时clob字段无法转换的问题
查询含有属性是clob字段的SQL,在spring mvc转换时会报错,无法将Buffer转换成string
org.springframework.http.converter.HttpMessageConversionException: Type definition error: [simple type, class dm.jdbc.util.Buffer]; nested exception is com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class dm.jdbc.util.Buffer and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: com.gsoft.foa.common.dto.AjaxResult["data"]->java.util.ArrayList[1]->java.util.HashMap["C_CHANGELOG"]->com.alibaba.druid.proxy.jdbc.NClobProxyImpl["rawNClob"]->dm.jdbc.driver.DmdbNClob["K"]->dm.jdbc.driver.DmdbConnection["gX"]->dm.jdbc.c.a["li"])
解决办法: 在数据库连接加上参数clobAsString=true,例如 jdbc:dm://xxx:5236/xx?clobAsString=true