第70课:SparkSQL内置函数解密与实战学习笔记
第70课:SparkSQL内置函数解密与实战学习笔记
本期内容：
1 SparkSQL内置函数解析
2 SparkSQL内置函数实战
SparkSQL的DataFrame引入了大量的内置函数，这些内置函数一般都有CG（CodeGeneration）功能，这样的函数在编译和执行时都会经过高度优化。
问题：SparkSQL操作Hive和Hive on spark一样吗？
=> 不一样。SparkSQL操作Hive只是把Hive当作数据仓库的来源，而计算引擎就是SparkSQL本身。Hive on spark是Hive的子项目，Hive on Spark的核心是把Hive的执行引擎换成Spark。众所周知，目前Hive的计算引擎是Mapreduce，因为性能低下等问题，所以Hive的官方就想替换这个引擎。
SparkSQL操作Hive上的数据叫Spark on Hive，而Hive on Spark依旧是以Hive为核心，只是把计算引擎由MapReduce替换为Spark。
Spark官网上DataFrame 的API Docs：
http://spark.apache.org/docs/latest/api/scala/#org.apache.spark.sql.DataFrame
classDataFrame extends Queryable with Serializable
Experimental
A distributed collection of data organized into named columns.
A DataFrame is equivalent to a relational table in Spark SQL. The following example creates a DataFrame by pointing Spark SQL to a Parquet data set.
val people = sqlContext.read.parquet("...") // in Scala
DataFrame people = sqlContext.read().parquet("...") // in Java
Once created, it can be manipulated using the various domain-specific-language (DSL) functions defined in: DataFrame (this class), Column, and functions.
To select a column from the data frame, use apply method in Scala and col in Java.
val ageCol = people("age") // in Scala
Column ageCol = people.col("age") // in Java
Note that the Column type can also be manipulated through its various functions.
// The following creates a new column that increases everybody's age by 10.
people("age") + 10 // in Scala
people.col("age").plus(10); // in Java
A more concrete example in Scala:
// To create DataFrame using SQLContextval people = sqlContext.read.parquet("...")val department = sqlContext.read.parquet("...")
people.filter("age > 30")
.join(department, people("deptId") === department("id"))
.groupBy(department("name"), "gender")
.agg(avg(people("salary")), max(people("age")))
and in Java:
// To create DataFrame using SQLContext
DataFrame people = sqlContext.read().parquet("...");
DataFrame department = sqlContext.read().parquet("...");
people.filter("age".gt(30))
.join(department, people.col("deptId").equalTo(department("id")))
.groupBy(department.col("name"), "gender")
.agg(avg(people.col("salary")), max(people.col("age")));
以上内容中的join,groupBy,agg都是SparkSQL的内置函数。
SParkl1.5.x以后推出了很多内置函数，据不完全统计，有一百多个内置函数。
下面实战开发一个聚合操作的例子：
package com.dt.spark.sql
import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType}
import org.apache.spark.sql.{Row, SQLContext}
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.sql.functions._
/**
* 使用Scala开发集群运行的Spark WordCount程序
* @author DT大数据梦工厂
* 新浪微博：http://weibo.com/ilovepains/
* Created by hp on 2016/3/28.
*
* 使用Spark SQL中的内置函数对数据进行分析，Spark SQL API不同的是，DataFrame中的内置函数操作的结果是返回一个Column对象，而
* DataFrame天生就是"A distributed collection of data organized into named columns.",这就为数据的复杂分析建立了坚实的基础
* 并提供了极大的方便性，例如说，我们在操作DataFrame的方法中可以随时调用内置函数进行业务需要的处理，这之于我们构建附件的业务逻辑而言是可以
* 极大的减少不必须的时间消耗（基于上就是实际模型的映射），让我们聚焦在数据分析上，这对于提高工程师的生产力而言是非常有价值的
* Spark 1.5.x开始提供了大量的内置函数，例如agg：
* def agg(aggExpr: (String, String), aggExprs: (String, String)*): DataFrame = {
* groupBy().agg(aggExpr, aggExprs : _*)
*}
* 还有max、mean、min、sum、avg、explode、size、sort_array、day、to_date、abs、acros、asin、atan
* 总体上而言内置函数包含了五大基本类型：
* 1，聚合函数，例如countDistinct、sumDistinct等；
* 2，集合函数，例如sort_array、explode等
* 3，日期、时间函数，例如hour、quarter、next_day
* 4,数学函数，例如asin、atan、sqrt、tan、round等；
* 5，开窗函数，例如rowNumber等
* 6，字符串函数，concat、format_number、rexexp_extract
* 7,其它函数，isNaN、sha、randn、callUDF
*
*/
object SparkSQLAgg {
def main (args: Array[String]) {
/**
* 第1步：创建Spark的配置对象SparkConf，设置Spark程序的运行时的配置信息，
* 例如说通过setMaster来设置程序要链接的Spark集群的Master的URL,如果设置
* 为local，则代表Spark程序在本地运行，特别适合于机器配置条件非常差（例如
* 只有1G的内存）的初学者 *
*/
val conf = new SparkConf() //创建SparkConf对象
conf.setAppName("SparkSQLInnerFunctions") //设置应用程序的名称，在程序运行的监控界面可以看到名称
// conf.setMaster("spark://Master:7077") //此时，程序在Spark集群
conf.setMaster("local")
/**
* 第2步：创建SparkContext对象
* SparkContext是Spark程序所有功能的唯一入口，无论是采用Scala、Java、Python、R等都必须有一个SparkContext
* SparkContext核心作用：初始化Spark应用程序运行所需要的核心组件，包括DAGScheduler、TaskScheduler、SchedulerBackend
* 同时还会负责Spark程序往Master注册程序等
* SparkContext是整个Spark应用程序中最为至关重要的一个对象
*/
val sc = new SparkContext(conf) //创建SparkContext对象，通过传入SparkConf实例来定制Spark运行的具体参数和配置信息
val sqlContext = new SQLContext(sc) //构建SQL上下文
//要使用Spark SQL的内置函数，就一定要导入SQLContext下的隐式转换
import sqlContext.implicits._
/**
* 第三步：模拟电商访问的数据，实际情况会比模拟数据复杂很多，最后生成RDD
*/
val userData = Array(
"2016-3-27,001,http://spark.apache.org/,1000",
"2016-3-27,001,http://hadoop.apache.org/,1001",
"2016-3-27,002,http://fink.apache.org/,1002",
"2016-3-28,003,http://kafka.apache.org/,1020",
"2016-3-28,004,http://spark.apache.org/,1010",
"2016-3-28,002,http://hive.apache.org/,1200",
"2016-3-28,001,http://parquet.apache.org/,1500",
"2016-3-28,001,http://spark.apache.org/,1800"
)
val userDataRDD = sc.parallelize(userData) //生成DD分布式集合对象
/**
* 第四步：根据业务需要对数据进行预处理生成DataFrame，要想把RDD转换成DataFrame，需要先把RDD中的元素类型变成Row类型
* 于此同时要提供DataFrame中的Columns的元数据信息描述
*/
val userDataRDDRow = userDataRDD.map(row => {val splited = row.split(",") ;Row(splited(0),splited(1).toInt,splited(2),splited(3).toInt)})
val structTypes = StructType(Array(
StructField("time", StringType, true),
StructField("id", IntegerType, true),
StructField("url", StringType, true),
StructField("amount", IntegerType, true)
))
val userDataDF = sqlContext.createDataFrame(userDataRDDRow,structTypes)
/**
* 第五步：使用Spark SQL提供的内置函数对DataFrame进行操作，特别注意：内置函数生成的Column对象且自定进行CG；
*
*
*/
userDataDF.groupBy("time").agg('time, countDistinct('id))
.map(row=>Row(row(1),row(2))).collect.foreach(println)
userDataDF.groupBy("time").agg('time, sum('amount)).show()
}
}
在Eclipse中运行如下：
16/04/10 23:54:04 INFO TaskSetManager: Finished task 58.0 in stage 6.0 (TID 461) in 18 ms on localhost (59/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:04 INFO Executor: Finished task 59.0 in stage 6.0 (TID 462). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 60.0 in stage 6.0 (TID 463, localhost, partition 61,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO Executor: Running task 60.0 in stage 6.0 (TID 463)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 59.0 in stage 6.0 (TID 462) in 15 ms on localhost (60/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/10 23:54:04 INFO Executor: Finished task 60.0 in stage 6.0 (TID 463). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 61.0 in stage 6.0 (TID 464, localhost, partition 62,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 60.0 in stage 6.0 (TID 463) in 17 ms on localhost (61/199)
16/04/10 23:54:04 INFO Executor: Running task 61.0 in stage 6.0 (TID 464)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:04 INFO Executor: Finished task 61.0 in stage 6.0 (TID 464). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 62.0 in stage 6.0 (TID 465, localhost, partition 63,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO Executor: Running task 62.0 in stage 6.0 (TID 465)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 61.0 in stage 6.0 (TID 464) in 99 ms on localhost (62/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:04 INFO Executor: Finished task 62.0 in stage 6.0 (TID 465). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 63.0 in stage 6.0 (TID 466, localhost, partition 64,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 62.0 in stage 6.0 (TID 465) in 18 ms on localhost (63/199)
16/04/10 23:54:04 INFO Executor: Running task 63.0 in stage 6.0 (TID 466)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:04 INFO Executor: Finished task 63.0 in stage 6.0 (TID 466). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 64.0 in stage 6.0 (TID 467, localhost, partition 65,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 63.0 in stage 6.0 (TID 466) in 16 ms on localhost (64/199)
16/04/10 23:54:04 INFO Executor: Running task 64.0 in stage 6.0 (TID 467)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:04 INFO Executor: Finished task 64.0 in stage 6.0 (TID 467). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 65.0 in stage 6.0 (TID 468, localhost, partition 66,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO Executor: Running task 65.0 in stage 6.0 (TID 468)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 64.0 in stage 6.0 (TID 467) in 18 ms on localhost (65/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:04 INFO Executor: Finished task 65.0 in stage 6.0 (TID 468). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 66.0 in stage 6.0 (TID 469, localhost, partition 67,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 65.0 in stage 6.0 (TID 468) in 47 ms on localhost (66/199)
16/04/10 23:54:04 INFO Executor: Running task 66.0 in stage 6.0 (TID 469)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:04 INFO Executor: Finished task 66.0 in stage 6.0 (TID 469). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 67.0 in stage 6.0 (TID 470, localhost, partition 68,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO Executor: Running task 67.0 in stage 6.0 (TID 470)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 66.0 in stage 6.0 (TID 469) in 17 ms on localhost (67/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:04 INFO Executor: Finished task 67.0 in stage 6.0 (TID 470). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 68.0 in stage 6.0 (TID 471, localhost, partition 69,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO Executor: Running task 68.0 in stage 6.0 (TID 471)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 67.0 in stage 6.0 (TID 470) in 11 ms on localhost (68/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:04 INFO Executor: Finished task 68.0 in stage 6.0 (TID 471). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 69.0 in stage 6.0 (TID 472, localhost, partition 70,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO Executor: Running task 69.0 in stage 6.0 (TID 472)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 68.0 in stage 6.0 (TID 471) in 21 ms on localhost (69/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:04 INFO Executor: Finished task 69.0 in stage 6.0 (TID 472). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 70.0 in stage 6.0 (TID 473, localhost, partition 71,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO Executor: Running task 70.0 in stage 6.0 (TID 473)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 69.0 in stage 6.0 (TID 472) in 15 ms on localhost (70/199)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
16/04/10 23:54:04 INFO Executor: Finished task 70.0 in stage 6.0 (TID 473). 1652 bytes result sent to driver
16/04/10 23:54:04 INFO TaskSetManager: Starting task 71.0 in stage 6.0 (TID 474, localhost, partition 72,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:04 INFO TaskSetManager: Finished task 70.0 in stage 6.0 (TID 473) in 50 ms on localhost (71/199)
16/04/10 23:54:04 INFO Executor: Running task 71.0 in stage 6.0 (TID 474)
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 71.0 in stage 6.0 (TID 474). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 72.0 in stage 6.0 (TID 475, localhost, partition 73,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 71.0 in stage 6.0 (TID 474) in 42 ms on localhost (72/199)
16/04/10 23:54:05 INFO Executor: Running task 72.0 in stage 6.0 (TID 475)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 72.0 in stage 6.0 (TID 475). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 73.0 in stage 6.0 (TID 476, localhost, partition 74,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 72.0 in stage 6.0 (TID 475) in 102 ms on localhost (73/199)
16/04/10 23:54:05 INFO Executor: Running task 73.0 in stage 6.0 (TID 476)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 73.0 in stage 6.0 (TID 476). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 74.0 in stage 6.0 (TID 477, localhost, partition 75,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 73.0 in stage 6.0 (TID 476) in 32 ms on localhost (74/199)
16/04/10 23:54:05 INFO Executor: Running task 74.0 in stage 6.0 (TID 477)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 74.0 in stage 6.0 (TID 477). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 75.0 in stage 6.0 (TID 478, localhost, partition 76,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 74.0 in stage 6.0 (TID 477) in 66 ms on localhost (75/199)
16/04/10 23:54:05 INFO Executor: Running task 75.0 in stage 6.0 (TID 478)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 75.0 in stage 6.0 (TID 478). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 76.0 in stage 6.0 (TID 479, localhost, partition 77,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 75.0 in stage 6.0 (TID 478) in 56 ms on localhost (76/199)
16/04/10 23:54:05 INFO Executor: Running task 76.0 in stage 6.0 (TID 479)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 76.0 in stage 6.0 (TID 479). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 77.0 in stage 6.0 (TID 480, localhost, partition 78,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 76.0 in stage 6.0 (TID 479) in 15 ms on localhost (77/199)
16/04/10 23:54:05 INFO Executor: Running task 77.0 in stage 6.0 (TID 480)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 77.0 in stage 6.0 (TID 480). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 78.0 in stage 6.0 (TID 481, localhost, partition 79,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 78.0 in stage 6.0 (TID 481)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 77.0 in stage 6.0 (TID 480) in 15 ms on localhost (78/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 78.0 in stage 6.0 (TID 481). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 79.0 in stage 6.0 (TID 482, localhost, partition 80,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 78.0 in stage 6.0 (TID 481) in 54 ms on localhost (79/199)
16/04/10 23:54:05 INFO Executor: Running task 79.0 in stage 6.0 (TID 482)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 79.0 in stage 6.0 (TID 482). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 80.0 in stage 6.0 (TID 483, localhost, partition 81,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 80.0 in stage 6.0 (TID 483)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 79.0 in stage 6.0 (TID 482) in 19 ms on localhost (80/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 80.0 in stage 6.0 (TID 483). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 81.0 in stage 6.0 (TID 484, localhost, partition 82,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 81.0 in stage 6.0 (TID 484)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 80.0 in stage 6.0 (TID 483) in 19 ms on localhost (81/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 81.0 in stage 6.0 (TID 484). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 82.0 in stage 6.0 (TID 485, localhost, partition 83,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 82.0 in stage 6.0 (TID 485)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 81.0 in stage 6.0 (TID 484) in 14 ms on localhost (82/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 82.0 in stage 6.0 (TID 485). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 83.0 in stage 6.0 (TID 486, localhost, partition 84,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 82.0 in stage 6.0 (TID 485) in 79 ms on localhost (83/199)
16/04/10 23:54:05 INFO Executor: Running task 83.0 in stage 6.0 (TID 486)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 83.0 in stage 6.0 (TID 486). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 84.0 in stage 6.0 (TID 487, localhost, partition 85,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 84.0 in stage 6.0 (TID 487)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 83.0 in stage 6.0 (TID 486) in 31 ms on localhost (84/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 84.0 in stage 6.0 (TID 487). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 85.0 in stage 6.0 (TID 488, localhost, partition 86,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 84.0 in stage 6.0 (TID 487) in 26 ms on localhost (85/199)
16/04/10 23:54:05 INFO Executor: Running task 85.0 in stage 6.0 (TID 488)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 85.0 in stage 6.0 (TID 488). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 86.0 in stage 6.0 (TID 489, localhost, partition 87,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 86.0 in stage 6.0 (TID 489)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 85.0 in stage 6.0 (TID 488) in 14 ms on localhost (86/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 86.0 in stage 6.0 (TID 489). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 87.0 in stage 6.0 (TID 490, localhost, partition 88,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 86.0 in stage 6.0 (TID 489) in 48 ms on localhost (87/199)
16/04/10 23:54:05 INFO Executor: Running task 87.0 in stage 6.0 (TID 490)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 87.0 in stage 6.0 (TID 490). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 88.0 in stage 6.0 (TID 491, localhost, partition 89,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 88.0 in stage 6.0 (TID 491)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 87.0 in stage 6.0 (TID 490) in 20 ms on localhost (88/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO GenerateMutableProjection: Code generated in 136.381588 ms
16/04/10 23:54:05 INFO Executor: Finished task 88.0 in stage 6.0 (TID 491). 2032 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 89.0 in stage 6.0 (TID 492, localhost, partition 90,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 88.0 in stage 6.0 (TID 491) in 308 ms on localhost (89/199)
16/04/10 23:54:05 INFO Executor: Running task 89.0 in stage 6.0 (TID 492)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 89.0 in stage 6.0 (TID 492). 2032 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 90.0 in stage 6.0 (TID 493, localhost, partition 91,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 90.0 in stage 6.0 (TID 493)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 89.0 in stage 6.0 (TID 492) in 45 ms on localhost (90/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 90.0 in stage 6.0 (TID 493). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 91.0 in stage 6.0 (TID 494, localhost, partition 92,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO Executor: Running task 91.0 in stage 6.0 (TID 494)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 90.0 in stage 6.0 (TID 493) in 24 ms on localhost (91/199)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 91.0 in stage 6.0 (TID 494). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 92.0 in stage 6.0 (TID 495, localhost, partition 93,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 91.0 in stage 6.0 (TID 494) in 14 ms on localhost (92/199)
16/04/10 23:54:05 INFO Executor: Running task 92.0 in stage 6.0 (TID 495)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:05 INFO Executor: Finished task 92.0 in stage 6.0 (TID 495). 1652 bytes result sent to driver
16/04/10 23:54:05 INFO TaskSetManager: Starting task 93.0 in stage 6.0 (TID 496, localhost, partition 94,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:05 INFO TaskSetManager: Finished task 92.0 in stage 6.0 (TID 495) in 19 ms on localhost (93/199)
16/04/10 23:54:05 INFO Executor: Running task 93.0 in stage 6.0 (TID 496)
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:05 INFO Executor: Finished task 93.0 in stage 6.0 (TID 496). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 94.0 in stage 6.0 (TID 497, localhost, partition 95,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 93.0 in stage 6.0 (TID 496) in 28 ms on localhost (94/199)
16/04/10 23:54:06 INFO Executor: Running task 94.0 in stage 6.0 (TID 497)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 94.0 in stage 6.0 (TID 497). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 95.0 in stage 6.0 (TID 498, localhost, partition 96,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 95.0 in stage 6.0 (TID 498)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 94.0 in stage 6.0 (TID 497) in 113 ms on localhost (95/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 95.0 in stage 6.0 (TID 498). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 96.0 in stage 6.0 (TID 499, localhost, partition 97,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 95.0 in stage 6.0 (TID 498) in 42 ms on localhost (96/199)
16/04/10 23:54:06 INFO Executor: Running task 96.0 in stage 6.0 (TID 499)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 96.0 in stage 6.0 (TID 499). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 97.0 in stage 6.0 (TID 500, localhost, partition 98,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 97.0 in stage 6.0 (TID 500)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 96.0 in stage 6.0 (TID 499) in 23 ms on localhost (97/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 97.0 in stage 6.0 (TID 500). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 98.0 in stage 6.0 (TID 501, localhost, partition 99,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 98.0 in stage 6.0 (TID 501)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 97.0 in stage 6.0 (TID 500) in 14 ms on localhost (98/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 98.0 in stage 6.0 (TID 501). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 99.0 in stage 6.0 (TID 502, localhost, partition 100,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 99.0 in stage 6.0 (TID 502)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 98.0 in stage 6.0 (TID 501) in 21 ms on localhost (99/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 99.0 in stage 6.0 (TID 502). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 100.0 in stage 6.0 (TID 503, localhost, partition 101,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 100.0 in stage 6.0 (TID 503)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 99.0 in stage 6.0 (TID 502) in 11 ms on localhost (100/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 100.0 in stage 6.0 (TID 503). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 101.0 in stage 6.0 (TID 504, localhost, partition 102,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 101.0 in stage 6.0 (TID 504)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 100.0 in stage 6.0 (TID 503) in 12 ms on localhost (101/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 101.0 in stage 6.0 (TID 504). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 102.0 in stage 6.0 (TID 505, localhost, partition 103,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 102.0 in stage 6.0 (TID 505)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 101.0 in stage 6.0 (TID 504) in 10 ms on localhost (102/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 102.0 in stage 6.0 (TID 505). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 103.0 in stage 6.0 (TID 506, localhost, partition 104,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 102.0 in stage 6.0 (TID 505) in 42 ms on localhost (103/199)
16/04/10 23:54:06 INFO Executor: Running task 103.0 in stage 6.0 (TID 506)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 103.0 in stage 6.0 (TID 506). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 104.0 in stage 6.0 (TID 507, localhost, partition 105,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 104.0 in stage 6.0 (TID 507)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 103.0 in stage 6.0 (TID 506) in 19 ms on localhost (104/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 104.0 in stage 6.0 (TID 507). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 105.0 in stage 6.0 (TID 508, localhost, partition 106,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 105.0 in stage 6.0 (TID 508)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 104.0 in stage 6.0 (TID 507) in 13 ms on localhost (105/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 105.0 in stage 6.0 (TID 508). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 106.0 in stage 6.0 (TID 509, localhost, partition 107,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 105.0 in stage 6.0 (TID 508) in 16 ms on localhost (106/199)
16/04/10 23:54:06 INFO Executor: Running task 106.0 in stage 6.0 (TID 509)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 106.0 in stage 6.0 (TID 509). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 107.0 in stage 6.0 (TID 510, localhost, partition 108,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 106.0 in stage 6.0 (TID 509) in 71 ms on localhost (107/199)
16/04/10 23:54:06 INFO Executor: Running task 107.0 in stage 6.0 (TID 510)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 107.0 in stage 6.0 (TID 510). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 108.0 in stage 6.0 (TID 511, localhost, partition 109,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 107.0 in stage 6.0 (TID 510) in 12 ms on localhost (108/199)
16/04/10 23:54:06 INFO Executor: Running task 108.0 in stage 6.0 (TID 511)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 108.0 in stage 6.0 (TID 511). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 109.0 in stage 6.0 (TID 512, localhost, partition 110,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 109.0 in stage 6.0 (TID 512)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 108.0 in stage 6.0 (TID 511) in 30 ms on localhost (109/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 109.0 in stage 6.0 (TID 512). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 110.0 in stage 6.0 (TID 513, localhost, partition 111,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 109.0 in stage 6.0 (TID 512) in 48 ms on localhost (110/199)
16/04/10 23:54:06 INFO Executor: Running task 110.0 in stage 6.0 (TID 513)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/10 23:54:06 INFO Executor: Finished task 110.0 in stage 6.0 (TID 513). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 111.0 in stage 6.0 (TID 514, localhost, partition 112,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 111.0 in stage 6.0 (TID 514)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 110.0 in stage 6.0 (TID 513) in 26 ms on localhost (111/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 111.0 in stage 6.0 (TID 514). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 112.0 in stage 6.0 (TID 515, localhost, partition 113,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 111.0 in stage 6.0 (TID 514) in 21 ms on localhost (112/199)
16/04/10 23:54:06 INFO Executor: Running task 112.0 in stage 6.0 (TID 515)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 112.0 in stage 6.0 (TID 515). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 113.0 in stage 6.0 (TID 516, localhost, partition 114,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 113.0 in stage 6.0 (TID 516)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 112.0 in stage 6.0 (TID 515) in 77 ms on localhost (113/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 113.0 in stage 6.0 (TID 516). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 114.0 in stage 6.0 (TID 517, localhost, partition 115,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 113.0 in stage 6.0 (TID 516) in 22 ms on localhost (114/199)
16/04/10 23:54:06 INFO Executor: Running task 114.0 in stage 6.0 (TID 517)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 114.0 in stage 6.0 (TID 517). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 115.0 in stage 6.0 (TID 518, localhost, partition 116,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 114.0 in stage 6.0 (TID 517) in 24 ms on localhost (115/199)
16/04/10 23:54:06 INFO Executor: Running task 115.0 in stage 6.0 (TID 518)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 115.0 in stage 6.0 (TID 518). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 116.0 in stage 6.0 (TID 519, localhost, partition 117,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 116.0 in stage 6.0 (TID 519)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 115.0 in stage 6.0 (TID 518) in 45 ms on localhost (116/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 116.0 in stage 6.0 (TID 519). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 117.0 in stage 6.0 (TID 520, localhost, partition 118,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 117.0 in stage 6.0 (TID 520)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 116.0 in stage 6.0 (TID 519) in 50 ms on localhost (117/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 117.0 in stage 6.0 (TID 520). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 118.0 in stage 6.0 (TID 521, localhost, partition 119,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 118.0 in stage 6.0 (TID 521)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 117.0 in stage 6.0 (TID 520) in 12 ms on localhost (118/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 118.0 in stage 6.0 (TID 521). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 119.0 in stage 6.0 (TID 522, localhost, partition 120,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 118.0 in stage 6.0 (TID 521) in 23 ms on localhost (119/199)
16/04/10 23:54:06 INFO Executor: Running task 119.0 in stage 6.0 (TID 522)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 119.0 in stage 6.0 (TID 522). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 120.0 in stage 6.0 (TID 523, localhost, partition 121,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 120.0 in stage 6.0 (TID 523)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 119.0 in stage 6.0 (TID 522) in 15 ms on localhost (120/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 120.0 in stage 6.0 (TID 523). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 121.0 in stage 6.0 (TID 524, localhost, partition 122,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 121.0 in stage 6.0 (TID 524)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 120.0 in stage 6.0 (TID 523) in 48 ms on localhost (121/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 121.0 in stage 6.0 (TID 524). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 122.0 in stage 6.0 (TID 525, localhost, partition 123,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 122.0 in stage 6.0 (TID 525)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 121.0 in stage 6.0 (TID 524) in 15 ms on localhost (122/199)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/10 23:54:06 INFO Executor: Finished task 122.0 in stage 6.0 (TID 525). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 123.0 in stage 6.0 (TID 526, localhost, partition 124,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 122.0 in stage 6.0 (TID 525) in 14 ms on localhost (123/199)
16/04/10 23:54:06 INFO Executor: Running task 123.0 in stage 6.0 (TID 526)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 13 ms
16/04/10 23:54:06 INFO Executor: Finished task 123.0 in stage 6.0 (TID 526). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 124.0 in stage 6.0 (TID 527, localhost, partition 125,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 123.0 in stage 6.0 (TID 526) in 108 ms on localhost (124/199)
16/04/10 23:54:06 INFO Executor: Running task 124.0 in stage 6.0 (TID 527)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:06 INFO Executor: Finished task 124.0 in stage 6.0 (TID 527). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 125.0 in stage 6.0 (TID 528, localhost, partition 126,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 124.0 in stage 6.0 (TID 527) in 24 ms on localhost (125/199)
16/04/10 23:54:06 INFO Executor: Running task 125.0 in stage 6.0 (TID 528)
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:06 INFO Executor: Finished task 125.0 in stage 6.0 (TID 528). 1652 bytes result sent to driver
16/04/10 23:54:06 INFO TaskSetManager: Starting task 126.0 in stage 6.0 (TID 529, localhost, partition 127,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:06 INFO Executor: Running task 126.0 in stage 6.0 (TID 529)
16/04/10 23:54:06 INFO TaskSetManager: Finished task 125.0 in stage 6.0 (TID 528) in 21 ms on localhost (126/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 126.0 in stage 6.0 (TID 529). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 127.0 in stage 6.0 (TID 530, localhost, partition 128,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 127.0 in stage 6.0 (TID 530)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 126.0 in stage 6.0 (TID 529) in 38 ms on localhost (127/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 127.0 in stage 6.0 (TID 530). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 128.0 in stage 6.0 (TID 531, localhost, partition 129,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 127.0 in stage 6.0 (TID 530) in 41 ms on localhost (128/199)
16/04/10 23:54:07 INFO Executor: Running task 128.0 in stage 6.0 (TID 531)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 128.0 in stage 6.0 (TID 531). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 129.0 in stage 6.0 (TID 532, localhost, partition 130,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 129.0 in stage 6.0 (TID 532)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 128.0 in stage 6.0 (TID 531) in 14 ms on localhost (129/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 129.0 in stage 6.0 (TID 532). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 130.0 in stage 6.0 (TID 533, localhost, partition 131,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 130.0 in stage 6.0 (TID 533)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 129.0 in stage 6.0 (TID 532) in 13 ms on localhost (130/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 130.0 in stage 6.0 (TID 533). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 131.0 in stage 6.0 (TID 534, localhost, partition 132,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 130.0 in stage 6.0 (TID 533) in 14 ms on localhost (131/199)
16/04/10 23:54:07 INFO Executor: Running task 131.0 in stage 6.0 (TID 534)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 131.0 in stage 6.0 (TID 534). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 132.0 in stage 6.0 (TID 535, localhost, partition 133,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 131.0 in stage 6.0 (TID 534) in 50 ms on localhost (132/199)
16/04/10 23:54:07 INFO Executor: Running task 132.0 in stage 6.0 (TID 535)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 132.0 in stage 6.0 (TID 535). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 133.0 in stage 6.0 (TID 536, localhost, partition 134,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 133.0 in stage 6.0 (TID 536)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 132.0 in stage 6.0 (TID 535) in 14 ms on localhost (133/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 133.0 in stage 6.0 (TID 536). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 134.0 in stage 6.0 (TID 537, localhost, partition 135,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 134.0 in stage 6.0 (TID 537)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 133.0 in stage 6.0 (TID 536) in 12 ms on localhost (134/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 134.0 in stage 6.0 (TID 537). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 135.0 in stage 6.0 (TID 538, localhost, partition 136,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 134.0 in stage 6.0 (TID 537) in 46 ms on localhost (135/199)
16/04/10 23:54:07 INFO Executor: Running task 135.0 in stage 6.0 (TID 538)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 135.0 in stage 6.0 (TID 538). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 136.0 in stage 6.0 (TID 539, localhost, partition 137,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 136.0 in stage 6.0 (TID 539)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 135.0 in stage 6.0 (TID 538) in 22 ms on localhost (136/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 136.0 in stage 6.0 (TID 539). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 137.0 in stage 6.0 (TID 540, localhost, partition 138,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 137.0 in stage 6.0 (TID 540)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 136.0 in stage 6.0 (TID 539) in 19 ms on localhost (137/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 137.0 in stage 6.0 (TID 540). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 138.0 in stage 6.0 (TID 541, localhost, partition 139,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 137.0 in stage 6.0 (TID 540) in 15 ms on localhost (138/199)
16/04/10 23:54:07 INFO Executor: Running task 138.0 in stage 6.0 (TID 541)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 138.0 in stage 6.0 (TID 541). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 139.0 in stage 6.0 (TID 542, localhost, partition 140,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 138.0 in stage 6.0 (TID 541) in 50 ms on localhost (139/199)
16/04/10 23:54:07 INFO Executor: Running task 139.0 in stage 6.0 (TID 542)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 139.0 in stage 6.0 (TID 542). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 140.0 in stage 6.0 (TID 543, localhost, partition 141,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 140.0 in stage 6.0 (TID 543)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 139.0 in stage 6.0 (TID 542) in 11 ms on localhost (140/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 140.0 in stage 6.0 (TID 543). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 141.0 in stage 6.0 (TID 544, localhost, partition 142,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 140.0 in stage 6.0 (TID 543) in 12 ms on localhost (141/199)
16/04/10 23:54:07 INFO Executor: Running task 141.0 in stage 6.0 (TID 544)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 141.0 in stage 6.0 (TID 544). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 142.0 in stage 6.0 (TID 545, localhost, partition 143,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 142.0 in stage 6.0 (TID 545)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 141.0 in stage 6.0 (TID 544) in 50 ms on localhost (142/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 142.0 in stage 6.0 (TID 545). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 143.0 in stage 6.0 (TID 546, localhost, partition 144,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 143.0 in stage 6.0 (TID 546)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 142.0 in stage 6.0 (TID 545) in 12 ms on localhost (143/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 143.0 in stage 6.0 (TID 546). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 144.0 in stage 6.0 (TID 547, localhost, partition 145,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 143.0 in stage 6.0 (TID 546) in 11 ms on localhost (144/199)
16/04/10 23:54:07 INFO Executor: Running task 144.0 in stage 6.0 (TID 547)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 144.0 in stage 6.0 (TID 547). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 145.0 in stage 6.0 (TID 548, localhost, partition 146,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 145.0 in stage 6.0 (TID 548)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 144.0 in stage 6.0 (TID 547) in 13 ms on localhost (145/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 145.0 in stage 6.0 (TID 548). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 146.0 in stage 6.0 (TID 549, localhost, partition 147,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 145.0 in stage 6.0 (TID 548) in 26 ms on localhost (146/199)
16/04/10 23:54:07 INFO Executor: Running task 146.0 in stage 6.0 (TID 549)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 146.0 in stage 6.0 (TID 549). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 147.0 in stage 6.0 (TID 550, localhost, partition 148,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 147.0 in stage 6.0 (TID 550)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 146.0 in stage 6.0 (TID 549) in 85 ms on localhost (147/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 147.0 in stage 6.0 (TID 550). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 148.0 in stage 6.0 (TID 551, localhost, partition 149,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 148.0 in stage 6.0 (TID 551)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 147.0 in stage 6.0 (TID 550) in 13 ms on localhost (148/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 148.0 in stage 6.0 (TID 551). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 149.0 in stage 6.0 (TID 552, localhost, partition 150,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 148.0 in stage 6.0 (TID 551) in 17 ms on localhost (149/199)
16/04/10 23:54:07 INFO Executor: Running task 149.0 in stage 6.0 (TID 552)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 149.0 in stage 6.0 (TID 552). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 150.0 in stage 6.0 (TID 553, localhost, partition 151,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 149.0 in stage 6.0 (TID 552) in 21 ms on localhost (150/199)
16/04/10 23:54:07 INFO Executor: Running task 150.0 in stage 6.0 (TID 553)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 150.0 in stage 6.0 (TID 553). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 151.0 in stage 6.0 (TID 554, localhost, partition 152,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 150.0 in stage 6.0 (TID 553) in 136 ms on localhost (151/199)
16/04/10 23:54:07 INFO Executor: Running task 151.0 in stage 6.0 (TID 554)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
16/04/10 23:54:07 INFO Executor: Finished task 151.0 in stage 6.0 (TID 554). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 152.0 in stage 6.0 (TID 555, localhost, partition 153,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 151.0 in stage 6.0 (TID 554) in 17 ms on localhost (152/199)
16/04/10 23:54:07 INFO Executor: Running task 152.0 in stage 6.0 (TID 555)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 152.0 in stage 6.0 (TID 555). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 153.0 in stage 6.0 (TID 556, localhost, partition 154,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 152.0 in stage 6.0 (TID 555) in 22 ms on localhost (153/199)
16/04/10 23:54:07 INFO Executor: Running task 153.0 in stage 6.0 (TID 556)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 153.0 in stage 6.0 (TID 556). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 154.0 in stage 6.0 (TID 557, localhost, partition 155,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 153.0 in stage 6.0 (TID 556) in 22 ms on localhost (154/199)
16/04/10 23:54:07 INFO Executor: Running task 154.0 in stage 6.0 (TID 557)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/10 23:54:07 INFO Executor: Finished task 154.0 in stage 6.0 (TID 557). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 155.0 in stage 6.0 (TID 558, localhost, partition 156,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 155.0 in stage 6.0 (TID 558)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 154.0 in stage 6.0 (TID 557) in 81 ms on localhost (155/199)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:07 INFO Executor: Finished task 155.0 in stage 6.0 (TID 558). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 156.0 in stage 6.0 (TID 559, localhost, partition 157,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 155.0 in stage 6.0 (TID 558) in 37 ms on localhost (156/199)
16/04/10 23:54:07 INFO Executor: Running task 156.0 in stage 6.0 (TID 559)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 156.0 in stage 6.0 (TID 559). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 157.0 in stage 6.0 (TID 560, localhost, partition 158,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 156.0 in stage 6.0 (TID 559) in 37 ms on localhost (157/199)
16/04/10 23:54:07 INFO Executor: Running task 157.0 in stage 6.0 (TID 560)
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:07 INFO Executor: Finished task 157.0 in stage 6.0 (TID 560). 1652 bytes result sent to driver
16/04/10 23:54:07 INFO TaskSetManager: Starting task 158.0 in stage 6.0 (TID 561, localhost, partition 159,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:07 INFO Executor: Running task 158.0 in stage 6.0 (TID 561)
16/04/10 23:54:07 INFO TaskSetManager: Finished task 157.0 in stage 6.0 (TID 560) in 22 ms on localhost (158/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 158.0 in stage 6.0 (TID 561). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 159.0 in stage 6.0 (TID 562, localhost, partition 160,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 159.0 in stage 6.0 (TID 562)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 158.0 in stage 6.0 (TID 561) in 148 ms on localhost (159/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 159.0 in stage 6.0 (TID 562). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 160.0 in stage 6.0 (TID 563, localhost, partition 161,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 160.0 in stage 6.0 (TID 563)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 159.0 in stage 6.0 (TID 562) in 16 ms on localhost (160/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 160.0 in stage 6.0 (TID 563). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 161.0 in stage 6.0 (TID 564, localhost, partition 162,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 161.0 in stage 6.0 (TID 564)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 160.0 in stage 6.0 (TID 563) in 17 ms on localhost (161/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 161.0 in stage 6.0 (TID 564). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 162.0 in stage 6.0 (TID 565, localhost, partition 163,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 161.0 in stage 6.0 (TID 564) in 20 ms on localhost (162/199)
16/04/10 23:54:08 INFO Executor: Running task 162.0 in stage 6.0 (TID 565)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 162.0 in stage 6.0 (TID 565). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 163.0 in stage 6.0 (TID 566, localhost, partition 164,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 163.0 in stage 6.0 (TID 566)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 162.0 in stage 6.0 (TID 565) in 71 ms on localhost (163/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 163.0 in stage 6.0 (TID 566). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 164.0 in stage 6.0 (TID 567, localhost, partition 165,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 163.0 in stage 6.0 (TID 566) in 20 ms on localhost (164/199)
16/04/10 23:54:08 INFO Executor: Running task 164.0 in stage 6.0 (TID 567)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 164.0 in stage 6.0 (TID 567). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 165.0 in stage 6.0 (TID 568, localhost, partition 166,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 165.0 in stage 6.0 (TID 568)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 164.0 in stage 6.0 (TID 567) in 21 ms on localhost (165/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 165.0 in stage 6.0 (TID 568). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 166.0 in stage 6.0 (TID 569, localhost, partition 167,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 166.0 in stage 6.0 (TID 569)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 165.0 in stage 6.0 (TID 568) in 29 ms on localhost (166/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
16/04/10 23:54:08 INFO Executor: Finished task 166.0 in stage 6.0 (TID 569). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 167.0 in stage 6.0 (TID 570, localhost, partition 168,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 166.0 in stage 6.0 (TID 569) in 105 ms on localhost (167/199)
16/04/10 23:54:08 INFO Executor: Running task 167.0 in stage 6.0 (TID 570)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/10 23:54:08 INFO Executor: Finished task 167.0 in stage 6.0 (TID 570). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 168.0 in stage 6.0 (TID 571, localhost, partition 169,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 168.0 in stage 6.0 (TID 571)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 167.0 in stage 6.0 (TID 570) in 31 ms on localhost (168/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 168.0 in stage 6.0 (TID 571). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 169.0 in stage 6.0 (TID 572, localhost, partition 170,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 168.0 in stage 6.0 (TID 571) in 17 ms on localhost (169/199)
16/04/10 23:54:08 INFO Executor: Running task 169.0 in stage 6.0 (TID 572)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 169.0 in stage 6.0 (TID 572). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 170.0 in stage 6.0 (TID 573, localhost, partition 171,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 169.0 in stage 6.0 (TID 572) in 32 ms on localhost (170/199)
16/04/10 23:54:08 INFO Executor: Running task 170.0 in stage 6.0 (TID 573)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 170.0 in stage 6.0 (TID 573). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 171.0 in stage 6.0 (TID 574, localhost, partition 172,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 170.0 in stage 6.0 (TID 573) in 62 ms on localhost (171/199)
16/04/10 23:54:08 INFO Executor: Running task 171.0 in stage 6.0 (TID 574)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 171.0 in stage 6.0 (TID 574). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 172.0 in stage 6.0 (TID 575, localhost, partition 173,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 171.0 in stage 6.0 (TID 574) in 75 ms on localhost (172/199)
16/04/10 23:54:08 INFO Executor: Running task 172.0 in stage 6.0 (TID 575)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 172.0 in stage 6.0 (TID 575). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 173.0 in stage 6.0 (TID 576, localhost, partition 174,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 172.0 in stage 6.0 (TID 575) in 67 ms on localhost (173/199)
16/04/10 23:54:08 INFO Executor: Running task 173.0 in stage 6.0 (TID 576)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 173.0 in stage 6.0 (TID 576). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 174.0 in stage 6.0 (TID 577, localhost, partition 175,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 174.0 in stage 6.0 (TID 577)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 173.0 in stage 6.0 (TID 576) in 15 ms on localhost (174/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 174.0 in stage 6.0 (TID 577). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 175.0 in stage 6.0 (TID 578, localhost, partition 176,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 174.0 in stage 6.0 (TID 577) in 43 ms on localhost (175/199)
16/04/10 23:54:08 INFO Executor: Running task 175.0 in stage 6.0 (TID 578)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:08 INFO Executor: Finished task 175.0 in stage 6.0 (TID 578). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 176.0 in stage 6.0 (TID 579, localhost, partition 177,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO Executor: Running task 176.0 in stage 6.0 (TID 579)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 175.0 in stage 6.0 (TID 578) in 314 ms on localhost (176/199)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 176.0 in stage 6.0 (TID 579). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 177.0 in stage 6.0 (TID 580, localhost, partition 178,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 176.0 in stage 6.0 (TID 579) in 13 ms on localhost (177/199)
16/04/10 23:54:08 INFO Executor: Running task 177.0 in stage 6.0 (TID 580)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:08 INFO Executor: Finished task 177.0 in stage 6.0 (TID 580). 1652 bytes result sent to driver
16/04/10 23:54:08 INFO TaskSetManager: Starting task 178.0 in stage 6.0 (TID 581, localhost, partition 179,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:08 INFO TaskSetManager: Finished task 177.0 in stage 6.0 (TID 580) in 14 ms on localhost (178/199)
16/04/10 23:54:08 INFO Executor: Running task 178.0 in stage 6.0 (TID 581)
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 178.0 in stage 6.0 (TID 581). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 179.0 in stage 6.0 (TID 582, localhost, partition 180,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 178.0 in stage 6.0 (TID 581) in 75 ms on localhost (179/199)
16/04/10 23:54:09 INFO Executor: Running task 179.0 in stage 6.0 (TID 582)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:09 INFO Executor: Finished task 179.0 in stage 6.0 (TID 582). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 180.0 in stage 6.0 (TID 583, localhost, partition 181,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 179.0 in stage 6.0 (TID 582) in 19 ms on localhost (180/199)
16/04/10 23:54:09 INFO Executor: Running task 180.0 in stage 6.0 (TID 583)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:09 INFO Executor: Finished task 180.0 in stage 6.0 (TID 583). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 181.0 in stage 6.0 (TID 584, localhost, partition 182,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 181.0 in stage 6.0 (TID 584)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 180.0 in stage 6.0 (TID 583) in 16 ms on localhost (181/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 181.0 in stage 6.0 (TID 584). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 182.0 in stage 6.0 (TID 585, localhost, partition 183,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 182.0 in stage 6.0 (TID 585)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 181.0 in stage 6.0 (TID 584) in 13 ms on localhost (182/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 8 ms
16/04/10 23:54:09 INFO Executor: Finished task 182.0 in stage 6.0 (TID 585). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 183.0 in stage 6.0 (TID 586, localhost, partition 184,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 183.0 in stage 6.0 (TID 586)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 182.0 in stage 6.0 (TID 585) in 55 ms on localhost (183/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:09 INFO Executor: Finished task 183.0 in stage 6.0 (TID 586). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 184.0 in stage 6.0 (TID 587, localhost, partition 185,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 184.0 in stage 6.0 (TID 587)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 183.0 in stage 6.0 (TID 586) in 12 ms on localhost (184/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 184.0 in stage 6.0 (TID 587). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 185.0 in stage 6.0 (TID 588, localhost, partition 186,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 185.0 in stage 6.0 (TID 588)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 184.0 in stage 6.0 (TID 587) in 12 ms on localhost (185/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 185.0 in stage 6.0 (TID 588). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 186.0 in stage 6.0 (TID 589, localhost, partition 187,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 185.0 in stage 6.0 (TID 588) in 44 ms on localhost (186/199)
16/04/10 23:54:09 INFO Executor: Running task 186.0 in stage 6.0 (TID 589)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:09 INFO Executor: Finished task 186.0 in stage 6.0 (TID 589). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 187.0 in stage 6.0 (TID 590, localhost, partition 188,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 187.0 in stage 6.0 (TID 590)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 186.0 in stage 6.0 (TID 589) in 24 ms on localhost (187/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:09 INFO Executor: Finished task 187.0 in stage 6.0 (TID 590). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 188.0 in stage 6.0 (TID 591, localhost, partition 189,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 187.0 in stage 6.0 (TID 590) in 15 ms on localhost (188/199)
16/04/10 23:54:09 INFO Executor: Running task 188.0 in stage 6.0 (TID 591)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 188.0 in stage 6.0 (TID 591). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 189.0 in stage 6.0 (TID 592, localhost, partition 190,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 189.0 in stage 6.0 (TID 592)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 188.0 in stage 6.0 (TID 591) in 14 ms on localhost (189/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 189.0 in stage 6.0 (TID 592). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 190.0 in stage 6.0 (TID 593, localhost, partition 191,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 190.0 in stage 6.0 (TID 593)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 189.0 in stage 6.0 (TID 592) in 37 ms on localhost (190/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 190.0 in stage 6.0 (TID 593). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 191.0 in stage 6.0 (TID 594, localhost, partition 192,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 190.0 in stage 6.0 (TID 593) in 14 ms on localhost (191/199)
16/04/10 23:54:09 INFO Executor: Running task 191.0 in stage 6.0 (TID 594)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 191.0 in stage 6.0 (TID 594). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 192.0 in stage 6.0 (TID 595, localhost, partition 193,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 192.0 in stage 6.0 (TID 595)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 191.0 in stage 6.0 (TID 594) in 18 ms on localhost (192/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 192.0 in stage 6.0 (TID 595). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 193.0 in stage 6.0 (TID 596, localhost, partition 194,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 193.0 in stage 6.0 (TID 596)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 192.0 in stage 6.0 (TID 595) in 18 ms on localhost (193/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:09 INFO Executor: Finished task 193.0 in stage 6.0 (TID 596). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 194.0 in stage 6.0 (TID 597, localhost, partition 195,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 194.0 in stage 6.0 (TID 597)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 193.0 in stage 6.0 (TID 596) in 34 ms on localhost (194/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/10 23:54:09 INFO Executor: Finished task 194.0 in stage 6.0 (TID 597). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 195.0 in stage 6.0 (TID 598, localhost, partition 196,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 195.0 in stage 6.0 (TID 598)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 194.0 in stage 6.0 (TID 597) in 16 ms on localhost (195/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 195.0 in stage 6.0 (TID 598). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 196.0 in stage 6.0 (TID 599, localhost, partition 197,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO Executor: Running task 196.0 in stage 6.0 (TID 599)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 195.0 in stage 6.0 (TID 598) in 11 ms on localhost (196/199)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/10 23:54:09 INFO Executor: Finished task 196.0 in stage 6.0 (TID 599). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 197.0 in stage 6.0 (TID 600, localhost, partition 198,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 196.0 in stage 6.0 (TID 599) in 11 ms on localhost (197/199)
16/04/10 23:54:09 INFO Executor: Running task 197.0 in stage 6.0 (TID 600)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/10 23:54:09 INFO Executor: Finished task 197.0 in stage 6.0 (TID 600). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Starting task 198.0 in stage 6.0 (TID 601, localhost, partition 199,NODE_LOCAL, 1999 bytes)
16/04/10 23:54:09 INFO TaskSetManager: Finished task 197.0 in stage 6.0 (TID 600) in 32 ms on localhost (198/199)
16/04/10 23:54:09 INFO Executor: Running task 198.0 in stage 6.0 (TID 601)
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/10 23:54:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
16/04/10 23:54:09 INFO Executor: Finished task 198.0 in stage 6.0 (TID 601). 1652 bytes result sent to driver
16/04/10 23:54:09 INFO TaskSetManager: Finished task 198.0 in stage 6.0 (TID 601) in 24 ms on localhost (199/199)
16/04/10 23:54:09 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
16/04/10 23:54:09 INFO DAGScheduler: ResultStage 6 (show at SparkSQLAgg.scala:105) finished in 7.771 s
16/04/10 23:54:09 INFO DAGScheduler: Job 2 finished: show at SparkSQLAgg.scala:105, took 8.252784 s
+---------+---------+-----------+
| time| time|sum(amount)|
+---------+---------+-----------+
|2016-3-27|2016-3-27| 3003|
|2016-3-28|2016-3-28| 6530|
+---------+---------+-----------+
16/04/10 23:54:09 INFO SparkContext: Invoking stop() from shutdown hook
16/04/10 23:54:09 WARN QueuedThreadPool: 3 threads could not be stopped
16/04/10 23:54:09 INFO SparkUI: Stopped Spark web UI at http://192.168.56.1:4040
16/04/10 23:54:09 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/04/10 23:54:10 INFO MemoryStore: MemoryStore cleared
16/04/10 23:54:10 INFO BlockManager: BlockManager stopped
16/04/10 23:54:10 INFO BlockManagerMaster: BlockManagerMaster stopped
16/04/10 23:54:10 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/04/10 23:54:10 INFO SparkContext: Successfully stopped SparkContext
16/04/10 23:54:10 INFO ShutdownHookManager: Shutdown hook called
16/04/10 23:54:10 INFO ShutdownHookManager: Deleting directory C:\Users\think\AppData\Local\Temp\spark-2cf92ebc-75bb-4a98-ad56-2090123af563
以上内容是王家林老师DT大数据梦工厂《 IMF传奇行动》第70课的学习笔记。
王家林老师是Spark、Flink、Docker、Android技术中国区布道师。Spark亚太研究院院长和首席专家，DT大数据梦工厂创始人，Android软硬整合源码级专家，英语发音魔术师，健身狂热爱好者。
微信公众账号：DT_Spark
电话：18610086859
QQ:1740415547
微信号：18610086859
新浪微博：ilovepains