梦飞天
梦飞天

第69课：SparkSQL通过Hive数据源实战学习笔记

第69课：SparkSQL通过Hive数据源实战学习笔记

本期内容：

1 SparkSQL操作Hive解析

2 SparkSQL操作Hive实战

 

数据源：home/richard/slq/spark/people.txt和/home/richard/slq/spark/peoplescores.txt两个文件。

people.txt的文件内容：

Michael 29

Andy 30

Justin 19

peoplescores.txt文件内容：

Michael 99

Andy 97

Justin 68

注：people.txt和peoplescores.txt的文件内容以Tab键为分隔符。

 

代码：

package com.dt.spark.SparkApps.sql

 

import org.apache.spark.SparkContext

import org.apache.spark.SparkConf

import org.apache.spark.sql.SQLContext

import org.apache.spark.sql.hive.HiveContext

 

// 使用Java的方式开发实战对DataFrame的操作

object SparkSQL2Hive {

  def main(args: Array[String]): Unit = {

    val conf = new SparkConf()  //创建SparkConf对象

    conf.setAppName("SparkSQL2Hive")  //设置应用程序名

    conf.setMaster("spark://slq1:7077")  //设置集群的Master

    val sc = new SparkContext  //创建SparkContext对象，

    

    //在目前企业级大数据Spark开发的时候，绝大多数情况下是采用Hive作为数据仓库

    //Spark提供了HIve的支持功能，Spark通过HiveContext可以直接操作Hive中的数据

    //基于HiveContext我们可以使用sql/hql两种方式才编写SQL语句对Hive进行操作，

    //包括创建表、删除表、往表里导入数据 以及用SQL语法构造 各种SQL语句对表中的数据进行CRUD操作   

    //第二：也可以直接通过saveAsTable的方式把DaraFrame中的数据保存到Hive数据仓库中

    //第三：可以直接通过HiveContext.table方法来直接加载Hive中的表而生成DataFrame

    val hiveContext = new HiveContext(sc)

    hiveContext.sql("use hive")

    hiveContext.sql("DROP TABLE IF EXISTS people")

    hiveContext.sql("CREATE TABLE IF NOT EXISTS people(name STRING,age INT)")

    hiveContext.sql("LOAD DATA LOCAL INPATH '/home/richard/slq/spark/people.txt' INTO TABLE people")

    //把本地数据加载到Hive中（背后实际上发生了数据的拷贝）

    //当然也可以通过LOAD DATA INPATH去获得HDFS等上面的数据 到Hive（此时发生了数据的移动）

    hiveContext.sql("DROP TABLE IF EXISTS peoplescores")

    hiveContext.sql("CREATE TABLE IF NOT EXISTS peoplescores(name STRING,score INT)")

    hiveContext.sql("LOAD DATA LOCAL INPATH '/home/richard/slq/spark/peoplescores.txt' INTO TABLE peoplescores")

    //通过HiveContext使用join直接基于Hive中的两张表进行操作获得大于90分的人的name,age,score

    val resultDF = hiveContext.sql("SELECT pi.name,pi.age,ps.score"   

      + "FROM people pi JOIN peoplescores ps ON pi.name=ps.name WHERE ps.score > 90")

    

      //通过saveAsTable创建一张Hive Managed Table，数据放在什么地方、元数据都是Hive管理的

      //当删除该表时，数据也会一起被删除（磁盘上的数据不再存在）

    hiveContext.sql("DROP TABLE IF EXISTS peopleinformationresult")

    resultDF.saveAsTable("peopleinformationresult")  

    

    //使用HivewContext的Table方法可以直接去读Hive中的Table并生成DaraFrame

    //读取的数据就可以进行机器学习、图计算、各种复杂ETL等操作

    val dataFrameHive = hiveContext.table("peopleinformationresult")

    

    dataFrameHive.show()  

    

  }

}

 

创建一个shell脚本：

/home/richard/spark-1.6.0/bin/spark-submit --class SparkSQLByScala.SparkSQL2Hive --master spark://slq1:7077 /home/richard/slq/spark/160327/SparkSQL2Hive.jar

通过./SparkAppsScala.sh执行。执行时的log如下：

[richard@slq1 160327]$ ./SparkAppsScala.sh

16/04/09 23:44:11 INFO spark.SparkContext: Running Spark version 1.6.0

16/04/09 23:44:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/04/09 23:44:17 INFO spark.SecurityManager: Changing view acls to: richard

16/04/09 23:44:17 INFO spark.SecurityManager: Changing modify acls to: richard

16/04/09 23:44:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(richard); users with modify permissions: Set(richard)

16/04/09 23:44:25 INFO util.Utils: Successfully started service 'sparkDriver' on port 35849.

16/04/09 23:44:30 INFO slf4j.Slf4jLogger: Slf4jLogger started

16/04/09 23:44:31 INFO Remoting: Starting remoting

16/04/09 23:44:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:48550]

16/04/09 23:44:34 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 48550.

16/04/09 23:44:35 INFO spark.SparkEnv: Registering MapOutputTracker

16/04/09 23:44:35 INFO spark.SparkEnv: Registering BlockManagerMaster

16/04/09 23:44:35 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-a5fed17f-9ca9-4356-8433-94184add459c

16/04/09 23:44:35 INFO storage.MemoryStore: MemoryStore started with capacity 517.4 MB

16/04/09 23:44:37 INFO spark.SparkEnv: Registering OutputCommitCoordinator

16/04/09 23:44:40 INFO server.Server: jetty-8.y.z-SNAPSHOT

16/04/09 23:44:40 INFO server.AbstractConnector: Started [email protected]:4040

16/04/09 23:44:40 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.

16/04/09 23:44:40 INFO ui.SparkUI: Started SparkUI at http://192.168.1.121:4040

16/04/09 23:44:41 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-03bc88ca-b3b3-410e-b8f7-10f85be695e3/httpd-ea4628a0-346c-4cf9-a951-092ab9f72f2a

16/04/09 23:44:41 INFO spark.HttpServer: Starting HTTP Server

16/04/09 23:44:41 INFO server.Server: jetty-8.y.z-SNAPSHOT

16/04/09 23:44:41 INFO server.AbstractConnector: Started [email protected]:60023

16/04/09 23:44:41 INFO util.Utils: Successfully started service 'HTTP file server' on port 60023.

16/04/09 23:44:41 INFO spark.SparkContext: Added JAR file:/home/richard/slq/spark/160327/SparkSQL2Hive.jar at http://192.168.1.121:60023/jars/SparkSQL2Hive.jar with timestamp 1460216681926

16/04/09 23:44:42 INFO client.AppClient$ClientEndpoint: Connecting to master spark://slq1:7077...

16/04/09 23:44:45 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160409234445-0002

16/04/09 23:44:45 INFO client.AppClient$ClientEndpoint: Executor added: app-20160409234445-0002/0 on worker-20160409171905-192.168.1.122-40769 (192.168.1.122:40769) with 1 cores

16/04/09 23:44:45 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160409234445-0002/0 on hostPort 192.168.1.122:40769 with 1 cores, 1024.0 MB RAM

16/04/09 23:44:45 INFO client.AppClient$ClientEndpoint: Executor added: app-20160409234445-0002/1 on worker-20160409171920-192.168.1.121-58385 (192.168.1.121:58385) with 1 cores

16/04/09 23:44:45 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160409234445-0002/1 on hostPort 192.168.1.121:58385 with 1 cores, 1024.0 MB RAM

16/04/09 23:44:45 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40241.

16/04/09 23:44:45 INFO netty.NettyBlockTransferService: Server created on 40241

16/04/09 23:44:45 INFO client.AppClient$ClientEndpoint: Executor added: app-20160409234445-0002/2 on worker-20160409171905-192.168.1.123-55110 (192.168.1.123:55110) with 1 cores

16/04/09 23:44:45 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160409234445-0002/2 on hostPort 192.168.1.123:55110 with 1 cores, 1024.0 MB RAM

16/04/09 23:44:46 INFO storage.BlockManagerMaster: Trying to register BlockManager

16/04/09 23:44:46 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.121:40241 with 517.4 MB RAM, BlockManagerId(driver, 192.168.1.121, 40241)

16/04/09 23:44:46 INFO storage.BlockManagerMaster: Registered BlockManager

16/04/09 23:44:47 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160409234445-0002/2 is now RUNNING

16/04/09 23:44:47 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160409234445-0002/0 is now RUNNING

16/04/09 23:44:47 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160409234445-0002/1 is now RUNNING

16/04/09 23:44:52 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0

16/04/09 23:45:16 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slq3:44752) with ID 2

16/04/09 23:45:16 INFO storage.BlockManagerMasterEndpoint: Registering block manager slq3:43259 with 517.4 MB RAM, BlockManagerId(2, slq3, 43259)

16/04/09 23:45:19 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slq2:34148) with ID 0

16/04/09 23:45:19 INFO storage.BlockManagerMasterEndpoint: Registering block manager slq2:42107 with 517.4 MB RAM, BlockManagerId(0, slq2, 42107)

16/04/09 23:45:24 INFO hive.HiveContext: Initializing execution hive, version 1.2.1

16/04/09 23:45:25 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0

16/04/09 23:45:25 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0

16/04/09 23:45:33 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

16/04/09 23:45:34 INFO metastore.ObjectStore: ObjectStore, initialize called

16/04/09 23:45:38 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored

16/04/09 23:45:38 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored

16/04/09 23:45:41 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/04/09 23:45:49 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)

16/04/09 23:45:57 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slq1:42766) with ID 1

16/04/09 23:45:58 INFO storage.BlockManagerMasterEndpoint: Registering block manager slq1:55585 with 517.4 MB RAM, BlockManagerId(1, slq1, 55585)

16/04/09 23:46:15 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

16/04/09 23:46:29 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/04/09 23:46:29 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/04/09 23:46:44 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.

16/04/09 23:46:44 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.

16/04/09 23:46:47 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY

16/04/09 23:46:47 INFO metastore.ObjectStore: Initialized ObjectStore

16/04/09 23:46:49 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0

16/04/09 23:46:51 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException

16/04/09 23:46:54 INFO metastore.HiveMetaStore: Added admin role in metastore

16/04/09 23:46:55 INFO metastore.HiveMetaStore: Added public role in metastore

16/04/09 23:46:56 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty

16/04/09 23:46:58 INFO metastore.HiveMetaStore: 0: get_all_databases

16/04/09 23:46:58 INFO HiveMetaStore.audit: ugi=richard ip=unknown-ip-addr cmd=get_all_databases

16/04/09 23:46:58 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=*

16/04/09 23:46:58 INFO HiveMetaStore.audit: ugi=richard ip=unknown-ip-addr cmd=get_functions: db=default pat=*

16/04/09 23:46:58 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.

16/04/09 23:47:08 INFO session.SessionState: Created local directory: /tmp/e3ba4cf4-de8c-4270-81b1-ce3ea721f144_resources

16/04/09 23:47:08 INFO session.SessionState: Created HDFS directory: /tmp/hive/richard/e3ba4cf4-de8c-4270-81b1-ce3ea721f144

16/04/09 23:47:08 INFO session.SessionState: Created local directory: /tmp/richard/e3ba4cf4-de8c-4270-81b1-ce3ea721f144

16/04/09 23:47:08 INFO session.SessionState: Created HDFS directory: /tmp/hive/richard/e3ba4cf4-de8c-4270-81b1-ce3ea721f144/_tmp_space.db

16/04/09 23:47:10 INFO hive.HiveContext: default warehouse location is /user/hive/warehouse

16/04/09 23:47:10 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.

16/04/09 23:47:11 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0

16/04/09 23:47:11 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0

16/04/09 23:47:18 INFO hive.metastore: Trying to connect to metastore with URI thrift://slq1:9083

16/04/09 23:47:19 INFO hive.metastore: Connected to metastore.

16/04/09 23:47:22 INFO session.SessionState: Created local directory: /tmp/4329319b-2c25-4345-862d-7b35b5dc5d6f_resources

16/04/09 23:47:22 INFO session.SessionState: Created HDFS directory: /tmp/hive/richard/4329319b-2c25-4345-862d-7b35b5dc5d6f

16/04/09 23:47:22 INFO session.SessionState: Created local directory: /tmp/richard/4329319b-2c25-4345-862d-7b35b5dc5d6f

16/04/09 23:47:22 INFO session.SessionState: Created HDFS directory: /tmp/hive/richard/4329319b-2c25-4345-862d-7b35b5dc5d6f/_tmp_space.db

16/04/09 23:47:25 INFO parse.ParseDriver: Parsing command: use hive

16/04/09 23:47:35 INFO parse.ParseDriver: Parse Completed

16/04/09 23:47:42 INFO log.PerfLogger:

16/04/09 23:47:42 INFO log.PerfLogger:

16/04/09 23:47:42 INFO log.PerfLogger:

16/04/09 23:47:43 INFO log.PerfLogger:

16/04/09 23:47:43 INFO parse.ParseDriver: Parsing command: use hive

16/04/09 23:47:51 INFO parse.ParseDriver: Parse Completed

16/04/09 23:47:51 INFO log.PerfLogger:

16/04/09 23:47:51 INFO log.PerfLogger:

16/04/09 23:47:52 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:47:52 INFO log.PerfLogger:

16/04/09 23:47:53 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:47:53 INFO log.PerfLogger:

16/04/09 23:47:53 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:47:53 INFO log.PerfLogger:

16/04/09 23:47:53 INFO ql.Driver: Starting command(queryId=richard_20160409234743_dd603e3c-c274-45d7-b647-95dceb1eb8fe): use hive

16/04/09 23:47:53 INFO log.PerfLogger:

16/04/09 23:47:53 INFO log.PerfLogger:

16/04/09 23:47:53 INFO log.PerfLogger:

16/04/09 23:47:53 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode

16/04/09 23:47:53 INFO log.PerfLogger:

16/04/09 23:47:53 INFO log.PerfLogger:

16/04/09 23:47:54 INFO ql.Driver: OK

16/04/09 23:47:54 INFO log.PerfLogger:

16/04/09 23:47:54 INFO log.PerfLogger:

16/04/09 23:47:54 INFO log.PerfLogger:

16/04/09 23:47:55 INFO parse.ParseDriver: Parsing command: DROP TABLE IF EXISTS people

16/04/09 23:47:55 INFO parse.ParseDriver: Parse Completed

16/04/09 23:47:57 INFO log.PerfLogger:

16/04/09 23:47:57 INFO log.PerfLogger:

16/04/09 23:47:57 INFO log.PerfLogger:

16/04/09 23:47:57 INFO log.PerfLogger:

16/04/09 23:47:57 INFO parse.ParseDriver: Parsing command: DROP TABLE IF EXISTS people

16/04/09 23:47:57 INFO parse.ParseDriver: Parse Completed

16/04/09 23:47:57 INFO log.PerfLogger:

16/04/09 23:47:57 INFO log.PerfLogger:

16/04/09 23:47:58 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:47:58 INFO log.PerfLogger:

16/04/09 23:47:58 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:47:58 INFO log.PerfLogger:

16/04/09 23:47:58 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:47:58 INFO log.PerfLogger:

16/04/09 23:47:58 INFO ql.Driver: Starting command(queryId=richard_20160409234757_4297553f-8ad7-4df2-8109-cfb6166a140b): DROP TABLE IF EXISTS people

16/04/09 23:47:58 INFO log.PerfLogger:

16/04/09 23:47:58 INFO log.PerfLogger:

16/04/09 23:47:58 INFO log.PerfLogger:

16/04/09 23:47:58 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode

16/04/09 23:48:00 INFO log.PerfLogger:

16/04/09 23:48:00 INFO metadata.Hive: Dumping metastore api call timing information for : execution phase

16/04/09 23:48:00 INFO metadata.Hive: Total time spent in this metastore function was greater than 1000ms : dropTable_(String, String, boolean, boolean, boolean, )=1325

16/04/09 23:48:00 INFO log.PerfLogger:

16/04/09 23:48:00 INFO ql.Driver: OK

16/04/09 23:48:00 INFO log.PerfLogger:

16/04/09 23:48:00 INFO log.PerfLogger:

16/04/09 23:48:00 INFO log.PerfLogger:

16/04/09 23:48:01 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS people(name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'

16/04/09 23:48:01 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:01 INFO log.PerfLogger:

16/04/09 23:48:01 INFO log.PerfLogger:

16/04/09 23:48:01 INFO log.PerfLogger:

16/04/09 23:48:01 INFO log.PerfLogger:

16/04/09 23:48:01 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS people(name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'

16/04/09 23:48:01 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:01 INFO log.PerfLogger:

16/04/09 23:48:01 INFO log.PerfLogger:

16/04/09 23:48:01 INFO parse.CalcitePlanner: Starting Semantic Analysis

16/04/09 23:48:02 INFO parse.CalcitePlanner: Creating table hive.people position=27

16/04/09 23:48:02 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:48:02 INFO log.PerfLogger:

16/04/09 23:48:02 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:48:02 INFO log.PerfLogger:

16/04/09 23:48:02 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:48:02 INFO log.PerfLogger:

16/04/09 23:48:02 INFO ql.Driver: Starting command(queryId=richard_20160409234801_1b05e701-b5b9-428d-89bf-9d587b9476f9): CREATE TABLE IF NOT EXISTS people(name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'

16/04/09 23:48:02 INFO log.PerfLogger:

16/04/09 23:48:02 INFO log.PerfLogger:

16/04/09 23:48:02 INFO log.PerfLogger:

16/04/09 23:48:02 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO ql.Driver: OK

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO parse.ParseDriver: Parsing command: LOAD DATA LOCAL INPATH '/home/richard/slq/spark/people.txt' INTO TABLE people

16/04/09 23:48:05 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO parse.ParseDriver: Parsing command: LOAD DATA LOCAL INPATH '/home/richard/slq/spark/people.txt' INTO TABLE people

16/04/09 23:48:05 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:05 INFO log.PerfLogger:

16/04/09 23:48:06 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:48:06 INFO log.PerfLogger:

16/04/09 23:48:06 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:48:06 INFO log.PerfLogger:

16/04/09 23:48:06 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:48:06 INFO log.PerfLogger:

16/04/09 23:48:06 INFO ql.Driver: Starting command(queryId=richard_20160409234805_a604d20e-a275-4000-b35e-6944cece8724): LOAD DATA LOCAL INPATH '/home/richard/slq/spark/people.txt' INTO TABLE people

16/04/09 23:48:06 INFO log.PerfLogger:

16/04/09 23:48:06 INFO log.PerfLogger:

16/04/09 23:48:06 INFO log.PerfLogger:

16/04/09 23:48:06 INFO ql.Driver: Starting task [Stage-0:MOVE] in serial mode

16/04/09 23:48:06 INFO exec.Task: Loading data to table hive.people from file:/home/richard/slq/spark/people.txt

16/04/09 23:48:10 INFO metadata.Hive: Renaming src: file:/home/richard/slq/spark/people.txt, dest: hdfs://slq1:9000/user/hive/warehouse/hive.db/people/people.txt, Status:true

16/04/09 23:48:11 INFO log.PerfLogger:

16/04/09 23:48:11 INFO ql.Driver: Starting task [Stage-1:STATS] in serial mode

16/04/09 23:48:11 INFO exec.StatsTask: Executing stats task

16/04/09 23:48:12 INFO exec.Task: Table hive.people stats: [numFiles=1, totalSize=29]

16/04/09 23:48:12 INFO log.PerfLogger:

16/04/09 23:48:12 INFO metadata.Hive: Dumping metastore api call timing information for : execution phase

16/04/09 23:48:12 INFO metadata.Hive: Total time spent in this metastore function was greater than 1000ms : alter_table_(String, String, Table, boolean, )=1048

16/04/09 23:48:12 INFO log.PerfLogger:

16/04/09 23:48:12 INFO ql.Driver: OK

16/04/09 23:48:12 INFO log.PerfLogger:

16/04/09 23:48:12 INFO log.PerfLogger:

16/04/09 23:48:12 INFO log.PerfLogger:

16/04/09 23:48:12 INFO parse.ParseDriver: Parsing command: DROP TABLE IF EXISTS peoplescores

16/04/09 23:48:12 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO parse.ParseDriver: Parsing command: DROP TABLE IF EXISTS peoplescores

16/04/09 23:48:13 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO ql.Driver: Starting command(queryId=richard_20160409234813_a4049719-c4a9-4b5b-8cee-e0256c5dc075): DROP TABLE IF EXISTS peoplescores

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO log.PerfLogger:

16/04/09 23:48:13 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO ql.Driver: OK

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS peoplescores(name STRING, score INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'

16/04/09 23:48:14 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:14 INFO log.PerfLogger:

16/04/09 23:48:15 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS peoplescores(name STRING, score INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'

16/04/09 23:48:15 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO parse.CalcitePlanner: Starting Semantic Analysis

16/04/09 23:48:15 INFO parse.CalcitePlanner: Creating table hive.peoplescores position=27

16/04/09 23:48:15 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO ql.Driver: Starting command(queryId=richard_20160409234814_fb368874-3ec7-4a99-9c79-edcea30096ed): CREATE TABLE IF NOT EXISTS peoplescores(name STRING, score INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO log.PerfLogger:

16/04/09 23:48:15 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO ql.Driver: OK

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO parse.ParseDriver: Parsing command: LOAD DATA LOCAL INPATH '/home/richard/slq/spark/peoplescores.txt' INTO TABLE peoplescores

16/04/09 23:48:16 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO parse.ParseDriver: Parsing command: LOAD DATA LOCAL INPATH '/home/richard/slq/spark/peoplescores.txt' INTO TABLE peoplescores

16/04/09 23:48:16 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO ql.Driver: Starting command(queryId=richard_20160409234816_1d279f16-8c1d-4f1e-811a-621729bb4d62): LOAD DATA LOCAL INPATH '/home/richard/slq/spark/peoplescores.txt' INTO TABLE peoplescores

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO log.PerfLogger:

16/04/09 23:48:16 INFO ql.Driver: Starting task [Stage-0:MOVE] in serial mode

16/04/09 23:48:16 INFO exec.Task: Loading data to table hive.peoplescores from file:/home/richard/slq/spark/peoplescores.txt

16/04/09 23:48:17 INFO metadata.Hive: Renaming src: file:/home/richard/slq/spark/peoplescores.txt, dest: hdfs://slq1:9000/user/hive/warehouse/hive.db/peoplescores/peoplescores.txt, Status:true

16/04/09 23:48:18 INFO log.PerfLogger:

16/04/09 23:48:18 INFO ql.Driver: Starting task [Stage-1:STATS] in serial mode

16/04/09 23:48:18 INFO exec.StatsTask: Executing stats task

16/04/09 23:48:19 INFO exec.Task: Table hive.peoplescores stats: [numFiles=1, totalSize=29]

16/04/09 23:48:19 INFO log.PerfLogger:

16/04/09 23:48:19 INFO metadata.Hive: Dumping metastore api call timing information for : execution phase

16/04/09 23:48:19 INFO metadata.Hive: Total time spent in this metastore function was greater than 1000ms : alter_table_(String, String, Table, boolean, )=1202

16/04/09 23:48:19 INFO log.PerfLogger:

16/04/09 23:48:19 INFO ql.Driver: OK

16/04/09 23:48:19 INFO log.PerfLogger:

16/04/09 23:48:19 INFO log.PerfLogger:

16/04/09 23:48:19 INFO log.PerfLogger:

16/04/09 23:48:19 INFO parse.ParseDriver: Parsing command: SELECT  pi.name, pi.age, ps.score FROM people pi JOIN peoplescores ps ON pi.name=ps.name WHERE ps.score>90

16/04/09 23:48:19 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:21 INFO parse.ParseDriver: Parsing command: DROP TABLE IF EXISTS peopleinformationresult

16/04/09 23:48:21 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:23 INFO parquet.ParquetRelation: Listing hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult on driver

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO parse.ParseDriver: Parsing command: DROP TABLE IF EXISTS peopleinformationresult

16/04/09 23:48:24 INFO parse.ParseDriver: Parse Completed

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO ql.Driver: Semantic Analysis Completed

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO ql.Driver: Starting command(queryId=richard_20160409234824_664b629d-ca06-4498-911a-611f9034c090): DROP TABLE IF EXISTS peopleinformationresult

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO log.PerfLogger:

16/04/09 23:48:24 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode

16/04/09 23:48:25 INFO log.PerfLogger:

16/04/09 23:48:25 INFO log.PerfLogger:

16/04/09 23:48:25 INFO ql.Driver: OK

16/04/09 23:48:25 INFO log.PerfLogger:

16/04/09 23:48:25 INFO log.PerfLogger:

16/04/09 23:48:25 INFO log.PerfLogger:

16/04/09 23:48:28 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

16/04/09 23:48:30 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 520.3 KB, free 520.3 KB)

16/04/09 23:48:31 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 41.9 KB, free 562.3 KB)

16/04/09 23:48:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.121:40241 (size: 41.9 KB, free: 517.4 MB)

16/04/09 23:48:31 INFO spark.SparkContext: Created broadcast 0 from saveAsTable at SparkSQL2Hive.scala:50

16/04/09 23:48:32 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 520.3 KB, free 1082.6 KB)

16/04/09 23:48:32 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 41.9 KB, free 1124.5 KB)

16/04/09 23:48:32 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.1.121:40241 (size: 41.9 KB, free: 517.3 MB)

16/04/09 23:48:32 INFO spark.SparkContext: Created broadcast 1 from saveAsTable at SparkSQL2Hive.scala:50

16/04/09 23:48:35 INFO mapred.FileInputFormat: Total input paths to process : 1

16/04/09 23:48:36 INFO spark.SparkContext: Starting job: run at ThreadPoolExecutor.java:1142

16/04/09 23:48:36 INFO scheduler.DAGScheduler: Got job 0 (run at ThreadPoolExecutor.java:1142) with 2 output partitions

16/04/09 23:48:36 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (run at ThreadPoolExecutor.java:1142)

16/04/09 23:48:36 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/04/09 23:48:36 INFO scheduler.DAGScheduler: Missing parents: List()

16/04/09 23:48:37 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[13] at run at ThreadPoolExecutor.java:1142), which has no missing parents

16/04/09 23:48:37 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 11.3 KB, free 1135.8 KB)

16/04/09 23:48:37 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.6 KB, free 1141.4 KB)

16/04/09 23:48:37 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.1.121:40241 (size: 5.6 KB, free: 517.3 MB)

16/04/09 23:48:37 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006

16/04/09 23:48:37 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[13] at run at ThreadPoolExecutor.java:1142)

16/04/09 23:48:37 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

16/04/09 23:48:38 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, slq1, partition 0,NODE_LOCAL, 2235 bytes)

16/04/09 23:48:38 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, slq2, partition 1,NODE_LOCAL, 2235 bytes)

16/04/09 23:48:43 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on slq2:42107 (size: 5.6 KB, free: 517.4 MB)

16/04/09 23:48:44 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on slq1:55585 (size: 5.6 KB, free: 517.4 MB)

16/04/09 23:48:52 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on slq2:42107 (size: 41.9 KB, free: 517.4 MB)

16/04/09 23:49:03 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on slq1:55585 (size: 41.9 KB, free: 517.4 MB)

16/04/09 23:49:06 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 28108 ms on slq2 (1/2)

16/04/09 23:49:23 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 45654 ms on slq1 (2/2)

16/04/09 23:49:23 INFO scheduler.DAGScheduler: ResultStage 0 (run at ThreadPoolExecutor.java:1142) finished in 45.689 s

16/04/09 23:49:23 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool

16/04/09 23:49:23 INFO scheduler.DAGScheduler: Job 0 finished: run at ThreadPoolExecutor.java:1142, took 47.639028 s

16/04/09 23:49:27 INFO codegen.GenerateUnsafeProjection: Code generated in 3005.850072 ms

16/04/09 23:49:27 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 600.0 B, free 1142.0 KB)

16/04/09 23:49:27 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 196.0 B, free 1142.2 KB)

16/04/09 23:49:27 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.1.121:40241 (size: 196.0 B, free: 517.3 MB)

16/04/09 23:49:27 INFO spark.SparkContext: Created broadcast 3 from run at ThreadPoolExecutor.java:1142

16/04/09 23:49:29 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id

16/04/09 23:49:29 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id

16/04/09 23:49:29 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id

16/04/09 23:49:29 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap

16/04/09 23:49:29 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition

16/04/09 23:49:29 INFO parquet.ParquetRelation: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter

16/04/09 23:49:29 INFO datasources.DefaultWriterContainer: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter

16/04/09 23:49:30 INFO mapred.FileInputFormat: Total input paths to process : 1

16/04/09 23:49:30 INFO spark.SparkContext: Starting job: saveAsTable at SparkSQL2Hive.scala:50

16/04/09 23:49:30 INFO scheduler.DAGScheduler: Got job 1 (saveAsTable at SparkSQL2Hive.scala:50) with 2 output partitions

16/04/09 23:49:30 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (saveAsTable at SparkSQL2Hive.scala:50)

16/04/09 23:49:30 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/04/09 23:49:30 INFO scheduler.DAGScheduler: Missing parents: List()

16/04/09 23:49:30 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[19] at saveAsTable at SparkSQL2Hive.scala:50), which has no missing parents

16/04/09 23:49:30 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 76.3 KB, free 1218.5 KB)

16/04/09 23:49:30 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 29.0 KB, free 1247.4 KB)

16/04/09 23:49:30 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.1.121:40241 (size: 29.0 KB, free: 517.3 MB)

16/04/09 23:49:30 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006

16/04/09 23:49:30 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[19] at saveAsTable at SparkSQL2Hive.scala:50)

16/04/09 23:49:30 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 2 tasks

16/04/09 23:49:30 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, slq1, partition 0,NODE_LOCAL, 2223 bytes)

16/04/09 23:49:30 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, slq2, partition 1,NODE_LOCAL, 2223 bytes)

16/04/09 23:49:31 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on slq2:42107 (size: 29.0 KB, free: 517.3 MB)

16/04/09 23:49:31 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on slq1:55585 (size: 29.0 KB, free: 517.3 MB)

16/04/09 23:49:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slq2:42107 (size: 41.9 KB, free: 517.3 MB)

16/04/09 23:49:32 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on slq2:42107 (size: 196.0 B, free: 517.3 MB)

16/04/09 23:49:32 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slq1:55585 (size: 41.9 KB, free: 517.3 MB)

16/04/09 23:49:34 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on slq1:55585 (size: 196.0 B, free: 517.3 MB)

16/04/09 23:50:00 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 29926 ms on slq1 (1/2)

16/04/09 23:50:04 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 33310 ms on slq2 (2/2)

16/04/09 23:50:04 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool

16/04/09 23:50:04 INFO scheduler.DAGScheduler: ResultStage 1 (saveAsTable at SparkSQL2Hive.scala:50) finished in 33.317 s

16/04/09 23:50:04 INFO scheduler.DAGScheduler: Job 1 finished: saveAsTable at SparkSQL2Hive.scala:50, took 33.704851 s

16/04/09 23:50:05 INFO hadoop.ParquetFileReader: Initiating action with parallelism: 5

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

16/04/09 23:50:10 INFO datasources.DefaultWriterContainer: Job job_201604092349_0000 committed.

16/04/09 23:50:10 INFO parquet.ParquetRelation: Listing hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult on driver

16/04/09 23:50:10 INFO parquet.ParquetRelation: Listing hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult on driver

16/04/09 23:50:10 INFO parquet.ParquetRelation: Listing hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult on driver

16/04/09 23:50:10 INFO hive.HiveContext$$anon$2: Persisting data source relation `peopleinformationresult` with a single input path into Hive metastore in Hive compatible format. Input path: hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult.

16/04/09 23:50:13 INFO parquet.ParquetRelation: Listing hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult on driver

16/04/09 23:50:13 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 62.4 KB, free 1309.9 KB)

16/04/09 23:50:14 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 19.7 KB, free 1329.6 KB)

16/04/09 23:50:14 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.1.121:40241 (size: 19.7 KB, free: 517.3 MB)

16/04/09 23:50:14 INFO spark.SparkContext: Created broadcast 5 from show at SparkSQL2Hive.scala:57

16/04/09 23:50:15 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

16/04/09 23:50:15 INFO parquet.ParquetRelation: Reading Parquet file(s) from hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult/part-r-00000-3ac34634-5b4a-46b3-badf-903fed7a8e1f.gz.parquet, hdfs://slq1:9000/user/hive/warehouse/hive.db/peopleinformationresult/part-r-00001-3ac34634-5b4a-46b3-badf-903fed7a8e1f.gz.parquet

16/04/09 23:50:15 INFO spark.SparkContext: Starting job: show at SparkSQL2Hive.scala:57

16/04/09 23:50:15 INFO scheduler.DAGScheduler: Got job 2 (show at SparkSQL2Hive.scala:57) with 1 output partitions

16/04/09 23:50:15 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (show at SparkSQL2Hive.scala:57)

16/04/09 23:50:15 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/04/09 23:50:15 INFO scheduler.DAGScheduler: Missing parents: List()

16/04/09 23:50:15 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[24] at show at SparkSQL2Hive.scala:57), which has no missing parents

16/04/09 23:50:16 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 5.7 KB, free 1335.2 KB)

16/04/09 23:50:16 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 3.2 KB, free 1338.4 KB)

16/04/09 23:50:16 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.1.121:40241 (size: 3.2 KB, free: 517.3 MB)

16/04/09 23:50:16 INFO spark.SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006

16/04/09 23:50:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[24] at show at SparkSQL2Hive.scala:57)

16/04/09 23:50:16 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks

16/04/09 23:50:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, slq3, partition 0,NODE_LOCAL, 2316 bytes)

16/04/09 23:50:19 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on slq3:43259 (size: 3.2 KB, free: 517.4 MB)

16/04/09 23:50:27 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on slq3:43259 (size: 19.7 KB, free: 517.4 MB)

16/04/09 23:50:42 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 25922 ms on slq3 (1/1)

16/04/09 23:50:42 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool

16/04/09 23:50:42 INFO scheduler.DAGScheduler: ResultStage 2 (show at SparkSQL2Hive.scala:57) finished in 25.928 s

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Job 2 finished: show at SparkSQL2Hive.scala:57, took 26.103248 s

16/04/09 23:50:42 INFO spark.SparkContext: Starting job: show at SparkSQL2Hive.scala:57

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Got job 3 (show at SparkSQL2Hive.scala:57) with 1 output partitions

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (show at SparkSQL2Hive.scala:57)

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Missing parents: List()

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[24] at show at SparkSQL2Hive.scala:57), which has no missing parents

16/04/09 23:50:42 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 5.7 KB, free 1344.1 KB)

16/04/09 23:50:42 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 3.2 KB, free 1347.3 KB)

16/04/09 23:50:42 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.1.121:40241 (size: 3.2 KB, free: 517.3 MB)

16/04/09 23:50:42 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[24] at show at SparkSQL2Hive.scala:57)

16/04/09 23:50:42 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 1 tasks

16/04/09 23:50:42 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, slq3, partition 1,NODE_LOCAL, 2314 bytes)

16/04/09 23:50:42 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on slq3:43259 (size: 3.2 KB, free: 517.4 MB)

16/04/09 23:50:42 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 512 ms on slq3 (1/1)

16/04/09 23:50:42 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool

16/04/09 23:50:42 INFO scheduler.DAGScheduler: ResultStage 3 (show at SparkSQL2Hive.scala:57) finished in 0.515 s

16/04/09 23:50:42 INFO scheduler.DAGScheduler: Job 3 finished: show at SparkSQL2Hive.scala:57, took 0.728242 s

+-------+---+-----+

|   name|age|score|

+-------+---+-----+

|Michael| 29|   99|

|   Andy| 30|   97|

+-------+---+-----+

 

16/04/09 23:50:43 INFO spark.SparkContext: Invoking stop() from shutdown hook

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}

16/04/09 23:50:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}

16/04/09 23:50:43 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.121:4040

16/04/09 23:50:43 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors

16/04/09 23:50:43 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down

16/04/09 23:50:44 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

16/04/09 23:50:44 INFO storage.MemoryStore: MemoryStore cleared

16/04/09 23:50:44 INFO storage.BlockManager: BlockManager stopped

16/04/09 23:50:44 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

16/04/09 23:50:45 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

16/04/09 23:50:45 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.

16/04/09 23:50:45 INFO spark.SparkContext: Successfully stopped SparkContext

16/04/09 23:50:45 INFO util.ShutdownHookManager: Shutdown hook called

16/04/09 23:50:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-03bc88ca-b3b3-410e-b8f7-10f85be695e3/httpd-ea4628a0-346c-4cf9-a951-092ab9f72f2a

16/04/09 23:50:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-c6f6f72f-b64c-4c83-b97f-aab038b6cefb

16/04/09 23:50:45 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.

16/04/09 23:50:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-03bc88ca-b3b3-410e-b8f7-10f85be695e3

 

 




以上内容是王家林老师DT大数据梦工厂《 IMF传奇行动》第69课的学习笔记。
王家林老师是Spark、Flink、Docker、Android技术中国区布道师。Spark亚太研究院院长和首席专家，DT大数据梦工厂创始人，Android软硬整合源码级专家，英语发音魔术师，健身狂热爱好者。

微信公众账号：DT_Spark

联系邮箱[email protected] 

电话：18610086859

QQ:1740415547

微信号：18610086859  

新浪微博：ilovepains


你可能感兴趣的:(Spark,SparkSQL,Join,jdbc,关系型数据库,hive)

  • 25年大数据开发省赛样题第一套，离线数据处理答案 Tometor 大数据sparkscala
    省赛样题一，数据抽取模块这一模块的作用是从mysql抽取数据到ods层进行指标计算，在题目中要求进行全量抽取，并新增etl-date字段进行分区，日期为比赛前一天importorg.apache.spark.sql.SparkSessionimportjava.util.PropertiesobjectTask1{defmain(args:Array[String]):Unit={valspark
  • Mybatis的基本使用 学c真好玩 mybatis
    MyBatis简介MyBatis用于持久层框架,持久层是对数据库操作的部分，前版本iBatis由Apache软件基金组织进行更名并维护。特点:简化数据库的操作SQL映射灵活(半ORM框架)支持高级映射易于集成维护配置动态SQL缓存机制功能：替代JDBC,JDBC是java中提供的用于操作数据库的技术及方案数据库的连接控制难。连接池SQL语句硬编码。将sql语句存放到xml配置文件中参数传递问题。提
  • 【MySQL基础-3】SQL语言详解：定义、分类、注意事项与注释 AllenBright #MySQLmysqlsql
    SQL（StructuredQueryLanguage，结构化查询语言）是用于管理和操作关系型数据库的标准编程语言。无论是查询数据、插入新记录、更新数据还是删除数据，SQL都是与数据库交互的核心工具。本文将深入探讨SQL语言的定义、分类、注意事项以及注释的使用，帮助你全面掌握这一强大的数据库操作语言。1.什么是SQL语言？SQL是一种专门用于管理关系型数据库的编程语言。它允许用户执行以下操作：查询
  • 如何注册下载欧意 OKBN666 区块链web3
    安卓注册下载链接如下，苹果下载需要更换IDwww.growthouyi.biz/zh-hans/join/48253632
  • python脚本使用Bash指令的一种方式 xz1308579340 python
    print(name)print(read_content(name))infos.append(read_content(name))xml=os.path.join(dir,name)new_xml=new_dir+'/'+str(count)+'.xml'img=xml.replace('xml','png')new_img=new_dir+'/'+str(count)+'.png'wd='
  • Linux 让PHP支持MSSQL hello_simon phplinux
    FreeTDS官方网站：http://www.freetds.org当前版本0.82wgethttp://mirrors.xmu.edu.cn/ubuntu/archive/pool/main/f/freetds/freetds_0.82.orig.tar.gz1.编译FreeTDS#tarzxvffreetds-0.82.tar.gz#cdfreetds-0.82//--with-tdsver=
  • YashanDB归档日志文件管理 数据库
    本文内容来自YashanDB官网，原文内容请见https://doc.yashandb.com/yashandb/23.3/zh/%E6%95%B0%E6%8D%AE%...归档日志文件默认存放在$YASDB\_DATA/archive目录下。Note：本文以查询单机部署中的归档日志文档为例，不同部署形态查询方法相同但$YASDB\_DATA路径不同，具体请以实际为准。$cd/data/yasha
  • YashanDB SWAP 表空间管理 数据库
    数据库操作（例如orderby，hashjoin，统计信息收集等）首先会通过数据库虚拟内存（通过VM_BUFFER_SIZE参数控制）缓存计算的中间结果，但如果虚拟内存不足时，需要通过将虚拟内存交换到SWAP表空间来释放内存，必要时再将内存从SWAP表空间换入。SWAP表空间是非持久化表空间，只用于数据库虚拟内存的换入换出，因此持久化对象（例如表、索引等）不能创建在SWAP表空间。SWAP表空间空
  • Analysis of QFN package bridging phenomenon and suggestions for improvement px5213344 pcb工艺
    1.OverviewofthebridgingphenomenonBridginginQFNpackagesisparticularlycommonbetweentheinnerrowsofsolderjointsindouble-rowQFNs,andrelativelyrareinsingle-rowQFNs.Bridgingoccurswhensolderispressedagainstan
  • Android数据存储:SQLite、Room -风になる- Android基础android
    在Android平台上，集成了一个嵌入式关系型数据库—SQLite，SQLite3支持NULL、INTEGER、REAL（浮点数字）、TEXT(字符串文本)和BLOB(二进制对象)数据类型，虽然它支持的类型只有五种，但实际上sqlite3也接受varchar(n)、char(n)、decimal(p,s)等数据类型，只不过在运算或保存时会转成对应的五种数据类型。SQLite最大的特点是你可以把各种
  • Android 里SQLite和ROOM框架简单介绍 大林不要掉头发 android数据库
    简单的AndroidSQLite使用最简单的SQLite在Android开发中，SQLite是一个轻量级的关系型数据库管理系统，经常用于存储和管理应用程序的数据。如果你刚刚学习Android数据库的使用，你一定要学习SQLite的使用。以下是一个简单的示例，展示了如何在Android应用中创建SQLite数据库、创建表、插入数据以及查询数据。创建SQLite数据库、创建表publicclassDB
  • Linux---sqlite3数据库 磨十三 数据库linuxsqlite
    一、数据库分类1.按数据关系分类类型特点代表产品关系型数据库-使用SQL（结构化查询语言）-数据以行列形式存储，支持事务和复杂查询MySQL、Oracle、SQLite非关系型数据库-无固定表结构（如键值对、文档、图）-高扩展性，适合非结构化数据MongoDB、Redis2.按功能规模分类类型特点代表产品大型数据库高并发、高可用性，支持企业级应用Oracle、DB2中型数据库适用于中小型企业，跨平
  • 为什么会出现redis数据库？redis是什么？ Foolforuuu 数据库redis缓存
    什么是Redis?为什么要用Redis?下面我将从Redis出现的背景、Redis的解决方案个来回答。1、Redis出现的背景互联网的应用越来越多，例如社交网络、电商、实时服务发展的十分迅速，这就导致了传统技术栈（如关系型数据库）面临许多挑战：性能瓶颈：并发、低延迟需求激增（如电商秒杀、社交平台动态刷新）。传统关系型数据库（如MySQL）依赖磁盘IO，无法应对每秒数万次的读写请求，导致响应延迟和扩
  • MongoDB介绍与部署使用 zhangleijiutian mongodb
    MongoDB的功能特性MongoDB是一个高性能、开源、无模式的文档型数据库，设计用于提供高性能、水平可扩性、高可用性和高级可查询性，为Web应用提供可扩展的高性能数据存储解决方案。MongoDB的名称取自单词”humogous”中间的几个字符，是很大、巨大的意思。该数据库由10gen公司开发并提供技术支持，它属于NoSQL数据库家族中的一员，在许多场景下可以用来替代传统关系型数据库或key/v
  • mac M1 安装flutter 小泥人（倪） macosflutter
    M1安装Flutter1-官网下载flutterSDK盘符自己记好https://docs.flutter.dev/release/archive?tab=macos2-控制台配置环境变量open.zshrc/bash_profile//平常在哪配就打开哪个文件3-复制粘贴exportPATH="$PATH:/Users/xnr/Documents/flutter/bin:$PATH"//自己安装
  • Hive高级SQL技巧及实际应用场景 小技工丨 大数据随笔sqlhive数据仓库大数据
    Hive高级SQL技巧及实际应用场景引言ApacheHive是一个建立在Hadoop之上的数据仓库基础设施，它提供了一个用于查询和管理分布式存储中的大型数据集的机制。通过使用类似于SQL（称为HiveQL）的语言，Hive使得数据分析变得更加简单和高效。本文将详细探讨一些Hive高级SQL技巧，并结合实际的应用场景进行说明。HiveSQL的高级使用技巧1.窗口函数描述：窗口函数允许我们在不使用GR
  • Hive SQL 精进系列：字符串拼接的三种常用方式 进一步有进一步的欢喜 hivesqlhadoop
    Hive字符串拼接：三种常用方式深度剖析目录Hive字符串拼接：三种常用方式深度剖析引言一、简洁直观的`||`操作符1.基础语法规则2.丰富多样的示例展示3.优势与局限分析二、规范通用的`CONCAT`函数1.全面的语法解析2.生动的示例说明3.优势与局限剖析三、灵活指定分隔符的`CONCAT_WS`函数1.清晰的语法介绍2.实用的示例演示3.优势与局限探讨四、总结与选择建议引言在Hive数据处理
  • HIVE SQL进阶 Q010910 hivesqlhadoop数据分析
    1.lateralviewexplode：将array或map类型的列拆分成多行数据lateralview：把拆分的单个字段数据与原始表的数据关联上LATERALVIEWEXPLODE(col)table_tempAScol_nametable_temp是因为LATERALVIEWUDTF函数在执行时，会生成一个临时的虚拟表。同时生成的列也需要列名col_name。lateralview的位置在f
  • mysql、oracle、SQLserver之间的区别和优势 噔噔噔噔@ mysqloraclesqlserver
    MySQL、Oracle和SQLServer都是常见的关系型数据库管理系统（RDBMS），它们在某些方面有一些区别和优势。MySQL：MySQL是一种开源的RDBMS，由Oracle公司开发和维护。它具有快速、稳定和易于使用的特点。MySQL适用于中小型应用和网站，它在处理大量简单的查询时表现较好。MySQL对于开发者来说是免费的，但也有商业版提供更多高级功能。Oracle：Oracle是一种商业
  • MySQL常用函数详解及SQL代码示例 chenOnlyOne mysqlsqlandroid
    MySQL常用函数详解及SQL代码示例引言MySQL作为一种广泛使用的关系型数据库管理系统，提供了丰富的内置函数来简化数据查询、处理和转换。掌握这些函数可以大大提高数据库操作的效率和准确性。本文将详细介绍MySQL中一些常用的函数，并配以SQL代码示例，帮助读者更好地理解和应用这些函数。当前日期和时间函数在当前时间（中国北京时间2025年03月11日，星期二）的背景下，我们首先介绍一些常用的日期和
  • 【并查集】 weixin_47868976 python
    并查集（DisjointSetUnion，DSU）是一种用于处理不相交集合的数据结构，主要支持两种操作：查找（Find）和合并（Union）。它在解决连通性问题、图论问题以及动态连通性等问题时非常有用。并查集的基础知识基本概念：集合：并查集维护一组不相交的集合，每个集合有一个代表元素。查找（Find）：查找某个元素所属的集合的代表元素。合并（Union）：将两个集合合并为一个集合。核心思想：路径压
  • 用Python实现字符串反转 程序媛了了 pythonjava前端
    字符串反转代码：#第一种：最简单的切片方法defpythonit():n=input("请输入一段文本：")a=n[::-1]print(a)pythonit()#第二种：列表循环法，利用sort()函数defpython():n=input("请输入一段文本：")list=[]foriinn:list.append(i)list.sort(reverse=True)print("".join(l
  • hive 数字转换字符串_Hive架构及Hive SQL的执行流程解读 weixin_39756416 hive数字转换字符串
    1、Hive产生背景MapReduce编程的不便性HDFS上的文件缺少Schema(表名，名称，ID等，为数据库对象的集合)2、Hive是什么Hive的使用场景是什么？基于Hadoop做一些数据清洗啊(ETL)、报表啊、数据分析可以将结构化的数据文件映射为一张数据库表，并提供类SQL查询功能。Hive是SQL解析引擎，它将SQL语句转译成M/RJob然后在Hadoop执行。由Facebook开源，
  • Hive-基础入门 数据牧马人 hivehadoop数据仓库
    目录第1章Hive基本概念第2章、安装步骤：1、上传jar包至/usr/local/soft2、解压并重命名3、配置环境变量4.配置HIVE文件第3章hive中数据库的操作1、hiveDDL操作2、HiveDML操作3、hive的本地模式4、hive表中数据加载方式5、hive中的流量统计6、hive数据导出第4章hive中的数据类型1.基本数据类型2.复杂数据类型第5章hive中的一个emp案例
  • ClickHouse 通过 *ARRAY JOIN* 结合 Map 类型的内置函数取数值 从未完美过 clickhouse
    在ClickHouse中，可以通过ARRAYJOIN结合Map类型的内置函数，将Map字段的键值对展开为多行数据。以下是具体操作方法和示例：一、使用mapKeys和mapValues展开Map1.核心语法SELECTid,key,valueFROMyour_tableARRAYJOINmapKeys(your_map_column)ASkey,mapValues(your_map_column)A
  • Neo4j之CQL基础 风云诀4 图数据库知识图谱大数据人工智能
    Neo4j之CQL基础文章目录Neo4j之CQL基础一、CQL概念二、CQL简介三、CQL命令使用creatematch+returnwheredeleteremovesetorderbymerge四、CQL函数使用StringAggregationRelationship一、CQL概念关系型数据库的查询语言是SQL，Neo4j图数据库也有自己的查询语言，那就是CQL。CQL全称CypherQue
  • sparkML入门，通俗解释机器学习的框架和算法 Tometor spark-ml机器学习算法回归数据挖掘人工智能scala
    一、机器学习的整体框架（类比烹饪）假设你要做一道菜，机器学习的过程可以类比为：步骤-->烹饪类比-->机器学习对应1.确定目标|想做什么菜（红烧肉/沙拉）|明确任务(分类/回归/聚类)2.准备食材|买菜、洗菜、切菜|数据收集与预处理3.设计食谱|决定烹饪步骤和调料|选择算法和模型设计4.试做并尝味道|调整火候和调味|模型训练与调参5.最终成品|端上桌的菜|模型部署与应用二、机器学习的核心流程1.数
  • Flink 1.17.2 版本用 java 读取 starrocks 小强签名设计 flinkjavapython
    文章目录方法一：使用FlinkJDBC连接器（兼容MySQL协议）方法二：使用StarRocksFlinkConnector（推荐）  在Flink1.17.2中使用Java读取StarRocks数据，可以通过JDBC连接器或StarRocks官方提供的FlinkConnector实现。以下是两种方法的详细步骤：方法一：使用FlinkJDBC连接器（兼容MySQL协议）  StarRocks兼容M
  • ranger集成starrock报错 蘑菇丁 大数据+机器学习+oracle大数据
    org.apache.ranger.plugin.client.HadoopException:initConnection:UnabletoconnecttoStarRocksinstance,pleaseprovidevalidvalueoffield:{jdbc.driverClassName}..com.mysql.cj.jdbc.Driver.可能的原因JDBC驱动缺失：运行环境中没有安
  • windows11 环境 paddleOCR 环境配置 一醉千秋 python+银河麒麟python开发语言
    一、系统环境：操作系统：Windows11专业版显卡：RTX4080CUDA：cuda_12.6.2_560.94_windows.exe、cudnn-windows-x86_64-8.9.7.29_cuda12-archivepython：3.12.8二、配置过程1.设置python国内源pipconfigsetglobal.index-urlhttps://pypi.tuna.tsinghua
  • apache 安装linux windows 墙头上一根草 apacheinuxwindows
    linux安装Apache 有两种方式一种是手动安装通过二进制的文件进行安装，另外一种就是通过yum 安装，此中安装方式，需要物理机联网。以下分别介绍两种的安装方式     通过二进制文件安装Apache需要的软件有apr,apr-util,pcre  1，安装 apr        下载地址：htt
  • fill_parent、wrap_content和match_parent的区别 Cb123456 match_parentfill_parent
    fill_parent、wrap_content和match_parent的区别:   1）fill_parent   设置一个构件的布局为fill_parent将强制性地使构件扩展，以填充布局单元内尽可能多的空间。这跟Windows控件的dockstyle属性大体一致。设置一个顶部布局或控件为fill_parent将强制性让它布满整个屏幕。 2） wrap_conte
  • 网页自适应设计 天子之骄 htmlcss响应式设计页面自适应
    网页自适应设计        网页对浏览器窗口的自适应支持变得越来越重要了。自适应响应设计更是异常火爆。再加上移动端的崛起，更是如日中天。以前为了适应不同屏幕分布率和浏览器窗口的扩大和缩小，需要设计几套css样式，用js脚本判断窗口大小，选择加载。结构臃肿，加载负担较大。现笔者经过一定时间的学习，有所心得，故分享于此，加强交流，共同进步。同时希望对大家有所
  • [sql server] 分组取最大最小常用sql 一炮送你回车库 SQL Server
    --分组取最大最小常用sql--测试环境if OBJECT_ID('tb') is not null drop table tb;gocreate table tb( col1 int, col2 int, Fcount int)insert into tbselect 11,20,1 union allselect 11,22,1 union allselect 1
  • ImageIO写图片输出到硬盘 3213213333332132 javaimage
    package awt; import java.awt.Color; import java.awt.Font; import java.awt.Graphics; import java.awt.image.BufferedImage; import java.io.File; import java.io.IOException; import javax.imagei
  • 自己的String动态数组 宝剑锋梅花香 java动态数组数组
    数组还是好说，学过一两门编程语言的就知道，需要注意的是数组声明时需要把大小给它定下来，比如声明一个字符串类型的数组：String str[]=new String[10];    但是问题就来了，每次都是大小确定的数组，我需要数组大小不固定随时变化怎么办呢？  动态数组就这样应运而生，龙哥给我们讲的是自己用代码写动态数组，并非用的ArrayList 看看字符
  • pinyin4j工具类 darkranger .net
    pinyin4j工具类Java工具类 2010-04-24 00:47:00 阅读69 评论0 字号：大中小 引入pinyin4j-2.5.0.jar包: pinyin4j是一个功能强悍的汉语拼音工具包，主要是从汉语获取各种格式和需求的拼音，功能强悍，下面看看如何使用pinyin4j。 本人以前用AscII编码提取工具，效果不理想，现在用pinyin4j简单实现了一个。功能还不是很完美，
  • StarUML学习笔记----基本概念 aijuans UML建模
    介绍StarUML的基本概念，这些都是有效运用StarUML?所需要的。包括对模型、视图、图、项目、单元、方法、框架、模型块及其差异以及UML轮廓。         模型、视与图（Model, View and Diagram）        &
  • Activiti最终总结 avords Activiti id 工作流
    1、流程定义ID：ProcessDefinitionId，当定义一个流程就会产生。 2、流程实例ID：ProcessInstanceId，当开始一个具体的流程时就会产生，也就是不同的流程实例ID可能有相同的流程定义ID。 3、TaskId，每一个userTask都会有一个Id这个是存在于流程实例上的。 4、TaskDefinitionKey和（ActivityImpl activityId
  • 从省市区多重级联想到的，react和jquery的差别 bee1314 jqueryUIreact
    在我们的前端项目里经常会用到级联的select，比如省市区这样。通常这种级联大多是动态的。比如先加载了省，点击省加载市，点击市加载区。然后数据通常ajax返回。如果没有数据则说明到了叶子节点。   针对这种场景，如果我们使用jquery来实现，要考虑很多的问题，数据部分，以及大量的dom操作。比如这个页面上显示了某个区，这时候我切换省，要把市重新初始化数据，然后区域的部分要从页面
  • Eclipse快捷键大全 bijian1013 javaeclipse快捷键
    Ctrl+1 快速修复(最经典的快捷键,就不用多说了)Ctrl+D: 删除当前行 Ctrl+Alt+↓ 复制当前行到下一行(复制增加)Ctrl+Alt+↑ 复制当前行到上一行(复制增加)Alt+↓ 当前行和下面一行交互位置(特别实用,可以省去先剪切,再粘贴了)Alt+↑ 当前行和上面一行交互位置(同上)Alt+← 前一个编辑的页面Alt+→ 下一个编辑的页面(当然是针对上面那条来说了)Alt+En
  • js 笔记 函数 征客丶 JavaScript
    一、函数的使用 1.1、定义函数变量 var vName = funcation(params){ } 1.2、函数的调用 函数变量的调用：      vName(params); 函数定义时自发调用：(function(params){})(params); 1.3、函数中变量赋值 var a = 'a'; var ff
  • 【Scala四】分析Spark源代码总结的Scala语法二 bit1129 scala
    1. Some操作   在下面的代码中，使用了Some操作：if (self.partitioner == Some(partitioner))，那么Some(partitioner)表示什么含义？首先partitioner是方法combineByKey传入的变量， Some的文档说明：   /** Class `Some[A]` represents existin
  • java 匿名内部类 BlueSkator java匿名内部类
    组合优先于继承 Java的匿名类，就是提供了一个快捷方便的手段，令继承关系可以方便地变成组合关系 继承只有一个时候才能用，当你要求子类的实例可以替代父类实例的位置时才可以用继承。   在Java中内部类主要分为成员内部类、局部内部类、匿名内部类、静态内部类。 内部类不是很好理解，但说白了其实也就是一个类中还包含着另外一个类如同一个人是由大脑、肢体、器官等身体结果组成，而内部类相
  • 盗版win装在MAC有害发热，苹果的东西不值得买，win应该不用 ljy325 游戏applewindowsXPOS
    Mac mini 型号: MC270CH-A RMB:5,688   Apple 对windows的产品支持不好,有以下问题:   1.装完了xp,发现机身很热虽然没有运行任何程序！貌似显卡跑游戏发热一样，按照那样的发热量,那部机子损耗很大,使用寿命受到严重的影响!   2.反观安装了Mac os的展示机，发热量很小，运行了1天温度也没有那么高 &nbs
  • 读《研磨设计模式》-代码笔记-生成器模式-Builder bylijinnan java设计模式
    声明： 本文只为方便我个人查阅和理解，详细的分析以及源代码请移步 原作者的博客http://chjavach.iteye.com/ /** * 生成器模式的意图在于将一个复杂的构建与其表示相分离，使得同样的构建过程可以创建不同的表示（GoF） * 个人理解： * 构建一个复杂的对象，对于创建者（Builder）来说，一是要有数据来源(rawData)，二是要返回构
  • JIRA与SVN插件安装 chenyu19891124 SVNjira
    JIRA安装好后提交代码并要显示在JIRA上，这得需要用SVN的插件才能看见开发人员提交的代码。 1.下载svn与jira插件安装包，解压后在安装包(atlassian-jira-subversion-plugin-0.10.1) 2.解压出来的包里下的lib文件夹下的jar拷贝到(C:\Program Files\Atlassian\JIRA 4.3.4\atlassian-jira\WEB
  • 常用数学思想方法 comsci 工作
      对于搞工程和技术的朋友来讲，在工作中常常遇到一些实际问题，而采用常规的思维方式无法很好的解决这些问题，那么这个时候我们就需要用数学语言和数学工具，而使用数学工具的前提却是用数学思想的方法来描述问题。。下面转帖几种常用的数学思想方法，仅供学习和参考   函数思想 　　把某一数学问题用函数表示出来，并且利用函数探究这个问题的一般规律。这是最基本、最常用的数学方法
  • pl/sql集合类型 daizj oracle集合typepl/sql
    --集合类型 /*   单行单列的数据，使用标量变量   单行多列数据，使用记录   单列多行数据，使用集合（。。。）   *集合：类似于数组也就是。pl/sql集合类型包括索引表（pl/sql table）、嵌套表（Nested Table）、变长数组（VARRAY）等 */ /*     --集合方法 &n
  • [Ofbiz]ofbiz初用 dinguangx 电商ofbiz
    从github下载最新的ofbiz（截止2015-7-13），从源码进行ofbiz的试用 1. 加载测试库 ofbiz内置derby，通过下面的命令初始化测试库 ./ant load-demo (与load-seed有一些区别)   2. 启动内置tomcat ./ant start 或 ./startofbiz.sh 或 java -jar ofbiz.jar &
  • 结构体中最后一个元素是长度为0的数组 dcj3sjt126com cgcc
    在Linux源代码中，有很多的结构体最后都定义了一个元素个数为0个的数组，如/usr/include/linux/if_pppox.h中有这样一个结构体： struct pppoe_tag {     __u16 tag_type;     __u16 tag_len;   &n
  • Linux cp 实现强行覆盖 dcj3sjt126com linux
    发现在Fedora 10 /ubutun 里面用cp -fr src dest，即使加了-f也是不能强行覆盖的，这时怎么回事的呢？一两个文件还好说，就输几个yes吧，但是要是n多文件怎么办，那还不输死人呢？下面提供三种解决办法。 方法一 我们输入alias命令，看看系统给cp起了一个什么别名。 [root@localhost ~]# aliasalias cp=’cp -i’a
  • Memcached(一)、HelloWorld frank1234 memcached
    一、简介 高性能的架构离不开缓存，分布式缓存中的佼佼者当属memcached，它通过客户端将不同的key hash到不同的memcached服务器中，而获取的时候也到相同的服务器中获取，由于不需要做集群同步，也就省去了集群间同步的开销和延迟，所以它相对于ehcache等缓存来说能更好的支持分布式应用，具有更强的横向伸缩能力。 二、客户端 选择一个memcached客户端，我这里用的是memc
  • Search in Rotated Sorted Array II hcx2013 search
    Follow up for "Search in Rotated Sorted Array":What if duplicates are allowed? Would this affect the run-time complexity? How and why? Write a function to determine if a given ta
  • Spring4新特性——更好的Java泛型操作API jinnianshilongnian spring4generic type
    Spring4新特性——泛型限定式依赖注入 Spring4新特性——核心容器的其他改进 Spring4新特性——Web开发的增强 Spring4新特性——集成Bean Validation 1.1(JSR-349)到SpringMVC  Spring4新特性——Groovy Bean定义DSL Spring4新特性——更好的Java泛型操作API  Spring4新
  • CentOS安装JDK liuxingguome centos
    1、行卸载原来的： [root@localhost opt]# rpm -qa | grep java tzdata-java-2014g-1.el6.noarch java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64 java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64 [root@localhost
  • 二分搜索专题2-在有序二维数组中搜索一个元素 OpenMind 二维数组算法二分搜索
    1,设二维数组p的每行每列都按照下标递增的顺序递增。 用数学语言描述如下：p满足 (1),对任意的x1，x2，y，如果x1<x2,则p(x1,y)<p(x2,y); (2),对任意的x，y1,y2, 如果y1<y2,则p(x,y1)<p(x,y2); 2,问题： 给定满足1的数组p和一个整数k，求是否存在x0,y0使得p(x0,y0)=k? 3,算法分析： (
  • java 随机数 Math与Random SaraWon javaMathRandom
    今天需要在程序中产生随机数，知道有两种方法可以使用，但是使用Math和Random的区别还不是特别清楚，看到一篇文章是关于的，觉得写的还挺不错的，原文地址是 http://www.oschina.net/question/157182_45274?sort=default&p=1#answers 产生1到10之间的随机数的两种实现方式： //Math Math.roun
  • oracle创建表空间 tugn oracle
    create temporary tablespace TXSJ_TEMP   tempfile 'E:\Oracle\oradata\TXSJ_TEMP.dbf'   size 32m   autoextend on   next 32m maxsize 2048m   extent m
  • 使用Java8实现自己的个性化搜索引擎 yangshangchuan javasuperword搜索引擎java8全文检索
    需要对249本软件著作实现句子级别全文检索，这些著作均为PDF文件，不使用现有的框架如lucene，自己实现的方法如下： 1、从PDF文件中提取文本，这里的重点是如何最大可能地还原文本。提取之后的文本，一个句子一行保存为文本文件。 2、将所有文本文件合并为一个单一的文本文件，这样，每一个句子就有一个唯一行号。 3、对每一行文本进行分词，建立倒排表，倒排表的格式为：词=包含该词的总行数N=行号
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他