用Phoenix / PQS 访问HBase
Apache Phoenix is an open-source SQL skin for Apache HBase that makes it easier to connect to your HBase data through standard JDBC and SQL interfaces. This article describes how to build an environment for accessing hbase through phoenix step by step
1. Install hbase
1) download hbase binary from https://hbase.apache.org/, currently the latest stable release is 2.4.16
2) extract binary
$ tar xvf hbase-2.4.16-bin.tar.gz -C ~/bigdata/
3) configrate hbase-env.sh and hbase-site.xml
$ cd ~/bigdata/hbase-2.4.16/conf
$ diff -u hbase-env.sh.orig hbase-env.sh
# The java implementation to use. Java 1.8+ required.
-# export JAVA_HOME=/usr/java/jdk1.8.0/
+export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home/$ diff -u hbase-site.xml.orig hbase-site.xml
--- hbase-site.xml.orig 2020-01-22 23:10:15.000000000 +0800
+++ hbase-site.xml 2023-03-29 09:00:32.000000000 +0800
@@ -51,4 +51,20 @@
hbase.unsafe.stream.capability.enforce
false
+
+ hbase.rootdir
+ file:/Users/sun_xo/bigdata/hbase-2.4.16/HBase/HFiles
+
+
+ hbase.zookeeper.property.dataDir
+ /Users/sun_xo/bigdata/hbase-2.4.16/zookeeper
+
+
+ phoenix.schema.isNamespaceMappingEnabled
+ true
+
+
+ phoenix.schema.mapSystemTablesToNamespace
+ true
+
4) create relevant dirs
$ cd ~/bigdata/hbase-2.4.16/
$ mkdir -p HBase/HFiles zookeeper
2. try hbase
1) start hbase in standlone mode
$ cd ~/bigdata/hbase-2.4.16/
$ bin/start-hbase.sh
$ netstat -ltpn | grep `jps | grep HMaster | awk '{print $1}'` # show sockets listening by hbase master
tcp6 0 0 127.0.0.1:2181 :::* LISTEN 620817/java
tcp6 0 0 :::16010 :::* LISTEN 620817/java
tcp6 0 0 192.168.55.250:16020 :::* LISTEN 620817/java
tcp6 0 0 :::16030 :::* LISTEN 620817/java
tcp6 0 0 192.168.55.250:16000 :::* LISTEN 620817/java
Note: HBase has a Web UI at http://localhost:16010
2). create a test table by using hbase shell
$ bin/hbase shell
hbase:001:0> create 'test', 'cf'
Created table test
Took 0.6299 seconds
=> Hbase::Table - test
hbase:002:0> put 'test', 'row1', 'cf:a', 'value1'
Took 0.0078 seconds
hbase:003:0> put 'test', 'row2', 'cf:b', 'value2'
Took 0.0020 seconds
hbase:004:0> put 'test', 'row3', 'cf:c', 'value3'
Took 0.0055 seconds
hbase:005:0> scan 'test'
ROW COLUMN+CELL
row1 column=cf:a, timestamp=2023-03-31T14:45:27.074, value=value1
row2 column=cf:b, timestamp=2023-03-31T14:45:29.172, value=value2
row3 column=cf:c, timestamp=2023-03-31T14:53:26.496, value=value3
3 row(s)
Took 0.0416 seconds3. install phoenix
1) download hbase binary from http://archive.apache.org/dist/phoenix/
2) extract binary
$ tar xvf phoenix-hbase-2.4-5.1.1-bin.tar.gz -C ~/bigdata/
3) set environment for phoenix
export HBASE_HOME=~/bigdata/hbase-2.4.16
export PHOENIX_CLASS_PATH=~/phoenix-hbase-2.4-5.1.1-bin
4) distribute phoenix server jar to hbase
$ cd ~/bigdata/phoenix-hbase-2.4-5.1.1-bin
$ cp phoenix-server-hbase-2.4-5.1.1.jar ../hbase-2.4.16/lib/
Note: needs to restart hbase here
4. test phoenix
1) remove local hbase-site.xml and check hbase_conf_path of phoenix
$ rm bin/hbase-site.xml
$ bin/phoenix_utils.py
hbase_conf_path: /home/sunxo/bigdata/hbase-2.4.16/conf
...Note: wrong hbase config will cause sqlline fail with following error:
ERROR 726 (43M10): Inconsistent namespace mapping properties. Ensure that config phoenix.schema.isNamespaceMappingEnabled is consistent on client and server
2) connect to hbase by using sqlline
$ bin/sqlline.py # if there is no exception, connected
0: jdbc:phoenix:> !tables # several system tables have been created in hbase
+-----------+-------------+------------+--------------+---------+-----------+----------------------+
| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE | REMARKS | TYPE_NAME | SELF_REFERENCING_COL |
+-----------+-------------+------------+--------------+---------+-----------+----------------------+
| | SYSTEM | CATALOG | SYSTEM TABLE | | | |
| | SYSTEM | CHILD_LINK | SYSTEM TABLE | | | |
| | SYSTEM | FUNCTION | SYSTEM TABLE | | | |
| | SYSTEM | LOG | SYSTEM TABLE | | | |
| | SYSTEM | MUTEX | SYSTEM TABLE | | | |
| | SYSTEM | SEQUENCE | SYSTEM TABLE | | | |
| | SYSTEM | STATS | SYSTEM TABLE | | | |
| | SYSTEM | TASK | SYSTEM TABLE | | | |
+-----------+-------------+------------+--------------+---------+-----------+----------------------+3) create a view so we can manipulate the table in hbase
0: jdbc:phoenix:> create view "test" (
. . . . . . . .)> "ROW" varchar primary key,
. . . . . . . .)> "cf"."a" varchar,
. . . . . . . .)> "cf"."b" varchar,
. . . . . . . .)> "cf"."c" varchar
. . . . . . . .)> );
>
No rows affected (5.937 seconds)
0: jdbc:phoenix:> select * from "test";
+------+--------+--------+--------+
| ROW | a | b | c |
+------+--------+--------+--------+
| row1 | value1 | | |
| row2 | | value2 | |
| row3 | | | value3 |
+------+--------+--------+--------+
3 rows selected (0.069 seconds)5. Install phoenix query server (PQS)
1) download hbase binary from https://phoenix.apache.org/download.html
2) extract binary
$ tar xvf phoenix-queryserver-6.0.0-bin.tar.gz -C ~/bigdata/
6. test phoenix query server
1) check phoenix_class_path of PQS
$ cd ~/bigdata/phoenix-queryserver-6.0.0
$ bin/phoenix_queryserver_utils.py
phoenix_class_path: /home/sunxo/bigdata/phoenix-hbase-2.4-5.1.1-bin
...Note: if the phoenix_class_path is wrong, start query server wiill fail with following error:
Error: Could not find or load main class org.apache.phoenix.queryserver.server.QueryServer
2) start PQS
$ bin/queryserver.py start
23/03/31 16:35:19 INFO server.Server: Started @622ms
23/03/31 16:35:19 INFO server.HttpServer: Service listening on port 87653) from other terminal, start sqlline-thin to connect to hbase
$ bin/sqlline-thin.py http://localhost:8765
sqlline version 1.9.0
0: jdbc:phoenix:thin:url=http://localhost:876> !tables
+-----------+-------------+------------+--------------+---------+-----------+----------------------+
| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE | REMARKS | TYPE_NAME | SELF_REFERENCING_COL |
+-----------+-------------+------------+--------------+---------+-----------+----------------------+
| | SYSTEM | CATALOG | SYSTEM TABLE | | | |
| | SYSTEM | CHILD_LINK | SYSTEM TABLE | | | |
| | SYSTEM | FUNCTION | SYSTEM TABLE | | | |
| | SYSTEM | LOG | SYSTEM TABLE | | | |
| | SYSTEM | MUTEX | SYSTEM TABLE | | | |
| | SYSTEM | SEQUENCE | SYSTEM TABLE | | | |
| | SYSTEM | STATS | SYSTEM TABLE | | | |
| | SYSTEM | TASK | SYSTEM TABLE | | | |
| | | test | VIEW | | | |
+-----------+-------------+------------+--------------+---------+-----------+----------------------+7. Test from a java client using PQS thin diver
$ cat Phoenix.java
public class Phoenix {
private final Connection conn;
public Phoenix(String hostName, int port) throws SQLException {
String url;
if (port == 2181) {
url = String.format("jdbc:phoenix:%s:%d", hostName, port);
} else if (port == 8765) {
url = String.format("jdbc:phoenix:thin:url=http://%s:%d;serialization=PROTOBUF", hostName, port);
} else {
throw new RuntimeException();
}
Properties properties = new Properties();
properties.setProperty("phoenix.schema.isNamespaceMappingEnabled", "true");
properties.setProperty("phoenix.schema.mapSystemTablesToNamespace", "true");
this.conn = DriverManager.getConnection(url, properties);
}
public void close() throws SQLException {
conn.close();
}
public void test() throws SQLException {
PreparedStatement statement = conn.prepareStatement("select * from \"test\"");
ResultSet rs = statement.executeQuery();
while (rs.next()) {
System.out.println(rs.getString("a"));
System.out.println(rs.getString("b"));
System.out.println(rs.getString("c"));
}
statement.close();
}
public static void main(String[] args) throws SQLException {
// Phoenix db = new Phoenix("localhost", 2181);
Phoenix db = new Phoenix("localhost", 8765);
db.test();
db.close();
System.out.println("closed");
}
}
$ javac Phoenix.java
$ java -cp "phoenix-queryserver-client-6.0.0.jar:." Phoenix
value1
null
null
null
value2
null
null
null
value3
closedReference:
https://hbase.apache.org/
https://phoenix.apache.org/