使用 JDBC 将 ClickHouse 连接到外部数据源

使用 JDBC 将 ClickHouse 连接到外部数据源

使用 JDBC Bridge

• 单独新建一个安装JDBC Bridge的目录

[root@only ck]# mkdir /data/ck/ck-jdbc-bridge/
[root@only ck]# ls
ck-jdbc-bridge 

• 切换到JDBC Bridge目录

[root@only ck]# cd /data/ck/ck-jdbc-bridge/
[root@only ck-jdbc-bridge]# pwd
/data/ck/ck-jdbc-bridge

• 下载JDBC Bridge的jar包

[root@only ck]# wget https://github.com/ClickHouse/clickhouse-jdbc-bridge/releases/download/v2.0.7/clickhouse-jdbc-bridge-2.0.7-shaded.jar
[root@only ck-jdbc-bridge]# ll
总用量 10488
-rw-r--r--. 1 root root 10737611 8月  23 22:54 clickhouse-jdbc-bridge-2.0.7-shaded.jar

• 新建配置目录、配置文件

[root@only ck-jdbc-bridge]# cd /data/ck/ck-jdbc-bridge
[root@only ck-jdbc-bridge]# mkdir -p config/datasources
[root@only ck-jdbc-bridge]# touch config/datasources/mysql8.json
[root@only ck-jdbc-bridge]# ls config/datasources
mysql8.json

• 配置文件配置mysql信息

{
  "mysql8": {
  "driverUrls": [
    "https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar"
  ],
  "jdbcUrl": "jdbc:mysql://192.168.52.19:3306",
  "username": "root",
  "password": "javaAndBigdata"
  }
}

• 切换到JDBC Bridge安装目录

[root@only config]# cd /data/ck/ck-jdbc-bridge/
[root@only ck-jdbc-bridge]# pwd
/data/ck/ck-jdbc-bridge
[root@only ck-jdbc-bridge]# ls
clickhouse-jdbc-bridge-2.0.7-shaded.jar  config

• 启动JDBC Bridge服务

[root@only ck-jdbc-bridge]# java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
[root@only ck-jdbc-bridge]# java -jar clickhouse-jdbc-bridge-2.0.7-shaded.jar
八月 30, 2022 10:22:25 下午 ru.yandex.clickhouse.jdbcbridge.core.Utils loadJsonFromFile
信息: Loading JSON from file [config/vertx.json]...
八月 30, 2022 10:22:25 下午 ru.yandex.clickhouse.jdbcbridge.core.Utils loadJsonFromFile
警告: Failed to load JSON from file config/vertx.json
八月 30, 2022 10:22:25 下午 ru.yandex.clickhouse.jdbcbridge.core.Utils loadJsonFromFile
信息: Loading JSON from file [config/server.json]...
八月 30, 2022 10:22:25 下午 ru.yandex.clickhouse.jdbcbridge.core.Utils loadJsonFromFile
警告: Failed to load JSON from file config/server.json
八月 30, 2022 10:22:25 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle registerConfigLoader
信息: Start to monitor configuration file(s) at [config/datasources]
八月 30, 2022 10:22:25 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle registerConfigLoader
信息: Start to monitor configuration file(s) at [config/schemas]
八月 30, 2022 10:22:25 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle registerConfigLoader
信息: Start to monitor configuration file(s) at [config/queries]
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.core.BaseRepository registerType
信息: Registering new type of NamedDataSource: [jdbc] -> [ru.yandex.clickhouse.jdbcbridge.impl.JdbcDataSource]
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.core.BaseRepository registerType
信息: Default type of NamedDataSource is set to [jdbc]
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.core.BaseRepository registerType
信息: Registering new type of NamedDataSource: [script] -> [ru.yandex.clickhouse.jdbcbridge.impl.ScriptDataSource]
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.core.Utils loadJsonFromFile
信息: Loading JSON from file [config/httpd.json]...
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.core.Utils loadJsonFromFile
警告: Failed to load JSON from file config/httpd.json
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle startServer
信息: Starting web server...
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.impl.JsonFileRepository reload
信息: No NamedSchema configuration found
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.impl.JsonFileRepository reload
信息: No NamedQuery configuration found
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle lambda$startServer$0
信息: Server http://0.0.0.0:9019 started in 1299 ms
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.impl.JsonFileRepository reload
信息: Loading NamedDataSource configuration...
八月 30, 2022 10:22:26 下午 ru.yandex.clickhouse.jdbcbridge.core.BaseRepository update
信息: Adding NamedDataSource(id=mysql8)...

• 进入mysql查看数据库及表记录

[root@only ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 625
Server version: 5.7.36-log MySQL Community Server (GPL)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| canalShit          |
| dolphinscheduler   |
| hivedb             |
| kylin              |
| mysql              |
| performance_schema |
| shit               |
| sys                |
+--------------------+
9 rows in set (0.01 sec)

mysql> use shit;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_shit |
+----------------+
| testDateTime   |
| testEnum       |
+----------------+
2 rows in set (0.00 sec)

mysql> select * from testEnum;
+------+
| id   |
+------+
|    1 |
|    2 |
|    3 |
|    4 |
|    5 |
+------+
5 rows in set (0.00 sec)

• 进入clickhouse 客户端

1. 确定clickhouse服务是否开启！
[root@only ~]# systemctl status clickhouse-server.service
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
   Loaded: loaded (/etc/systemd/system/clickhouse-server.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2022-08-29 22:36:46 CST; 23h ago
 Main PID: 20509 (clckhouse-watch)
    Tasks: 205
   Memory: 1022.5M
   CGroup: /system.slice/clickhouse-server.service
           ├─20509 clickhouse-watchdog --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid
           └─20510 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

8月 29 22:36:46 only systemd[1]: Started ClickHouse Server (analytic DBMS for big data).
8月 29 22:36:46 only clickhouse-server[20509]: Processing configuration file '/etc/clickhouse-server/config.xml'.
8月 29 22:36:46 only clickhouse-server[20509]: Logging trace to /var/log/clickhouse-server/clickhouse-server.log
8月 29 22:36:46 only clickhouse-server[20509]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
8月 29 22:36:48 only clickhouse-server[20509]: Processing configuration file '/etc/clickhouse-server/config.xml'.
8月 29 22:36:48 only clickhouse-server[20509]: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
8月 29 22:36:48 only clickhouse-server[20509]: Processing configuration file '/etc/clickhouse-server/users.xml'.
8月 29 22:36:48 only clickhouse-server[20509]: Merging configuration file '/etc/clickhouse-server/users.d/default-password.xml'.
8月 29 22:36:48 only clickhouse-server[20509]: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.

# 进入clickhouse客户端 --port为本机修改的端口
[root@only ~]# clickhouse-client -m --password --port 9006
ClickHouse client version 22.1.3.7 (official build).
Password for user (default):
Connecting to localhost:9006 as user default.
Connected to ClickHouse server version 22.1.3 revision 54455.

only :) show databases;

SHOW DATABASES

Query id: 4e7ab4b4-4219-48fe-8643-ea001a6cdd6e

┌─name───────────────┐
│ INFORMATION_SCHEMA │
│ atomicDatabase     │
│ default            │
│ information_schema │
│ javaAndBigdata     │
│ normalDatabase     │
│ system             │
└────────────────────┘

7 rows in set. Elapsed: 0.002 sec.

• 在clickhouse客户端中使用JDBC Bridge服务来查看mysql的表数据

only :) SELECT * FROM jdbc('mysql8', 'shit', 'testEnum');

SELECT *
FROM jdbc('mysql8', 'shit', 'testEnum')

Query id: eaace3ae-2ab9-4738-b07c-49c6fc2e1272

┌─id─┐
│  1 │
│  2 │
│  3 │
│  4 │
│  5 │
└────┘

5 rows in set. Elapsed: 0.228 sec.

• 同时JDBC Bridge后台日志也会输出日志

八月 30, 2022 10:30:56 下午 ru.yandex.clickhouse.jdbcbridge.core.NamedDataSource executeQuery
信息: Executing query(schema=[shit]):
testEnum

• 简单的其他操作

only :) describe jdbc('mysql8', 'shit', 'testEnum');

DESCRIBE TABLE  jdbc('mysql8', 'shit', 'testEnum')

Query id: 8aedefc9-92cf-4621-9a9b-f0539a06f219

┌─name─┬─type────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
│ id   │ Nullable(Int32) │              │                    │         │                  │                │
└──────┴─────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘

1 rows in set. Elapsed: 0.012 sec.
# 将jdbc('mysql8', 'shit', 'testEnum')看成mysql的表即可！！！

• 日志输出

八月 30, 2022 10:30:56 下午 ru.yandex.clickhouse.jdbcbridge.core.NamedDataSource executeQuery
信息: Executing query(schema=[shit]):
testEnum
八月 30, 2022 10:33:16 下午 ru.yandex.clickhouse.jdbcbridge.JdbcBridgeVerticle handleColumnsInfo
信息: Raw query:
testEnum