在调试flink table api 查询 hive数据的时候,真的是遇到很多坑,特别是要hive存储的数据是在腾讯云的cos上,而且我是跨集群查询数据,要解决各种依赖和环境问题,下面的代码和pom.xml 已经调试成功,在本地和集群 on yarn都可以运行,本地的时候需要在idea里面加args为dev,集群 on yarn不用加。
代码
package com.bigdata.etl
import org.apache.flink.table.api.{EnvironmentSettings, TableEnvironment}
import org.apache.flink.table.catalog.hive.HiveCatalog
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
object FlinkTableTest extends App {
//本地idea环境需要设置用户为hadoop
System.setProperty("HADOOP_USER_NAME", "hadoop")
val settings = EnvironmentSettings.newInstance().useBlinkPlanner().build()
println(settings.isStreamingMode)
val stenv = TableEnvironment.create(settings)
//查询默认的元数据
stenv.executeSql("show catalogs").print()
stenv.useCatalog("default_catalog")
stenv.executeSql("show databases").print()
stenv.executeSql("select 1").print()
println("-----fengexian--------------")
val name = "hive"
val defaultDatabase = "odl"
var hiveConfDir = ""
var hadoopConf=""
val hiveVision = "2.3.6"
if (args.size > 0) {
hiveConfDir = "/Users/duzhixin/Documents/flink-hive-conf"
hadoopConf="/Users/duzhixin/Documents/flink-hive-conf"
} else {
hiveConfDir = "/usr/local/service/hive/conf"
}
//在这里把hive,hadoop的配置文件加进去,本地idea环境需要制定hive和hadoop的配置文件,集群上不用制定hadoop的配置
val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir,hadoopConf,hiveVision)
hive.getHiveConf.set("streaming-source.enable ", "true")
stenv.getConfig.getConfiguration.setString("streaming-source.enable ", "true")
stenv.getConfig.getConfiguration.setString("table.exec.hive.infer-source-parallelism.max ", "10000")
stenv.getConfig.getConfiguration.setString("table.exec.hive.infer-source-parallelism ", "true")
stenv.registerCatalog("hive", hive)
// set the HiveCatalog as the current catalog of the session
stenv.useCatalog("hive")
//执行sql,查询hive的元数据
stenv.executeSql("show databases").print()
stenv.executeSql("show tables").print()
stenv.executeSql("select 1 from test.app limit 1").print()
stenv.executeSql("select * from odl.tb_book where dt='2021-06-05' limit 10").print()
stenv.executeSql("select * from odl.dadian where dt='2021-06-05' limit 10").print()
}
现实生活中,有些工作是需要团队中成员依次完成的,这就涉及到了一个顺序问题。现在有T1、T2、T3三个工人,如何保证T2在T1执行完后执行,T3在T2执行完后执行?问题分析:首先问题中有三个实体,T1、T2、T3, 因为是多线程编程,所以都要设计成线程类。关键是怎么保证线程能依次执行完呢?
Java实现过程如下:
public class T1 implements Runnabl
hive在使用having count()是,不支持去重计数
hive (default)> select imei from t_test_phonenum where ds=20150701 group by imei having count(distinct phone_num)>1 limit 10;
FAILED: SemanticExcep
转载源:http://blog.sina.com.cn/s/blog_4f925fc30100rx5l.html
mysql -uroot -p
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
[root@centos var]# service mysql
#mq
xdr.mq.url=tcp://192.168.100.15:61618;
import java.io.IOException;
import java.util.Properties;
public class Test {
String conf = "log4j.properties";
private static final
有一个工程,本来运行是正常的,我想把它移植到另一台PC上,结果报:
java.lang.RuntimeException: Unable to instantiate activity ComponentInfo{com.mobovip.bgr/com.mobovip.bgr.MainActivity}: java.lang.ClassNotFoundException: Didn't f
// 报错如下:
$ git pull origin master
fatal: unable to access 'https://git.xxx.com/': SSL certificate problem: unable to get local issuer ce
rtificate
// 原因:
由于git最新版默认使用ssl安全验证,但是我们是使用的git未设