在屋顶听歌

Spark之RDD编程快速入门

注：笔者用的spark1.6.3版本的，计划是用spark1.x版本大致了解一遍spark，然后再用spark2.x正式玩玩。
RDD 基本原理可见RDD 基本原理

一、RDD的简单介绍

RDD是一种只读的、分区的记录集合。
RDD分为两类，transformation(map/filter/groupBy/join等) 与 action(count/ take / collect / first / reduce等)
其中转换得到的RDD是惰性求值的，也就是说，整个转换过程只是记录了转换的轨迹，并不会发生真正的计算，只有遇到行动操作时，才会发生真正的计算

RDD 常用的Transformation：

filter(func)：筛选出满足函数func的元素，并返回一个新的数据集
map(func)：将每个元素传递到函数func中，并将结果返回为一个新的数据集
flatMap(func)：与map()相似，但每个输入元素都可以映射到0或多个输出结果
groupByKey()：应用于(K,V)键值对的数据集时，返回一个新的(K, Iterable)形式的数据集
reduceByKey(func)：应用于(K,V)键值对的数据集时，返回一个新的(K, V)形式的数据集，其中的每个值是将每个key传递到函数func中进行聚合行动操作是真正触发计算的地方。Spark程序执行到行动操作时，才会执行真正的计算，从文件中加载数据，完成一次又一次转换操作，最终,完成行动操作得到结果。

常见的行动操作（Action API）：

count() 返回数据集中的元素个数
collect() 以数组的形式返回数据集中的所有元素
first() 返回数据集中的第一个元素
take(n) 以数组的形式返回数据集中的前n个元素
reduce(func) 通过函数func（输入两个参数并返回一个值）
聚合数据集中的元素
foreach(func) 将数据集中的每个元素传递到函数func中运行

二、RDD的创建

RDD创建方式：

a. 通过parallelize函数把一般数据结构加载为RDD,makeRDD():通过makeRDD函数把一般数据结构加载为RDD
b. 从外部存储创建RDD，例如通过textFile直接加载数据文件为RDD
c. 通过已存在的RDD创建新的RDD

scala> val a = sc.parallelize(List("dog", "cat", "owl", "gnu", "ant"), 2)
scala> a.partitions.size
res24: Int = 2

其中数字2为设定的分区数。
如下没有指定分区，系统会默认给其分配，查看默认给分配了8个分区

scala> val a1 = sc.parallelize(List("dog", "cat", "owl", "gnu", "ant123","123","4321"))
scala> a1.saveAsTextFile("/tmp/a1.txt")
scala> a1.partitions.size
res22: Int = 8

hdfs目录下的结果:

当然也可以指定保存为多少个分区，如下指定保存为一个分区，这时生成的结果只在一个分区中

scala> a1.repartition(1).saveAsTextFile("/tmp/a2.txt")

三、数据准备：

将数据上传至Hdfs中

student.txt

1001 李正明
1002 王一磊
1003 陈志华
1004 张永丽
1005 赵信
1006 古明远
1007 刘浩明
1008 沈彬
1009 李子琪
1010 五嘉栋
1011 柳梦文
1012 钱多多

result_bigdata.txt

1001 大数据基础 90
1002 大数据基础 94
1003 大数据基础 100
1004 大数据基础 99
1005 大数据基础 90
1006 大数据基础 93
1007 大数据基础 100
1008 大数据基础 93
1009 大数据基础 89
1010 大数据基础 78
1011 大数据基础 91
1012 大数据基础 84

result_math.txt

1001 应用数学 90
1002 应用数学 94
1003 应用数学 100
1004 应用数学 100
1005 应用数学 94
1006 应用数学 80
1007 应用数学 90
1008 应用数学 94
1009 应用数学 84
1010 应用数学 86
1011 应用数学 79
1012 应用数学 91

四、编程样例

（1）取出成绩排名前5的学生成绩信息

思路：
先将两个成绩表的RDD中的数据进行转换，每条数据被分成3列，表示学生ID、课程、成绩，分隔符为“ ”，存储为三元组格式，成绩要求转化为Int类型，可以直接通过toInt的转化
通过sortBy对元组中的成绩列降序排序，排序位置是每个元组的第3位的成绩
通过take操作取出每个RDD的前5个值就是成绩排在前5的学生

进入spark-shell

./bin/spark-shell
val math = sc.textFile("/tmp/result_math.txt")
val bigdata = sc.textFile("/tmp/result_bigdata.txt")
val m_math  = math .map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
val m_bigdata = bigdata.map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
val mathResult = m_math.sortBy(_._3,false).take(5)
val bigdataResult = m_bigdata.sortBy(_._3,false).take(5)

（2）找出单科成绩为100的学生ID

在上述的基础中，可以直接使用已存在的RDD
思路：将当科成绩为100的学生取出并通过union方法合并,从合并后的结果中取出学生ID，并将重复的去掉
具体代码如下：

scala> val student_100 = m_math.filter(_._3 == 100).union(m_bigdata.filter(_._3 == 100))
scala> val result = student_100.map(_._1).distinct
scala> val resultArray = result.collect

（3）输出每位学生的总成绩，要求将两个成绩表中学生ID相同的成绩相加。

scala> :paste
val math = sc.textFile("/tmp/result_math.txt")
val bigdata = sc.textFile("/tmp/result_bigdata.txt")
val m_math  = math .map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
val m_bigdata = bigdata.map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
val all_scores = (m_math  union m_bigdata).map(x=>(x._1,x._3)).reduceByKey((x,y) => x+y).collect

（4）计算每个学生的平均成绩

解法一：取得样例三中总成绩all_scores对其进行求平均值便可，如下

scala> val aveScores = all_scores.map(x => (x._1,x._2/2.0))
scala> aveScores.collect
res2: Array[(String, Double)] = Array((1005,92.0), (1012,87.5), (1001,90.0), (1009,86.5), (1002,94.0), (1006,86.5), (1010,82.0), (1003,100.0), (1007,95.0), (1008,93.5), (1011,85.0), (1004,99.5))

解法二：CombineByKey

（5）汇总学生成绩并以文本格式存储在HDFS上，数据汇总为学生ID，姓名，大数据成绩，数学成绩，总分，平均分。

scala> :paste
val math = sc.textFile("/tmp/result_math.txt")
val bigdata = sc.textFile("/tmp/result_bigdata.txt")
val student = sc.textFile("/tmp/student.txt")
val m_student = student.map{x=> val line=x.split(" ");(line(0),line(1))}
val m_math  = math .map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
val m_bigdata = bigdata.map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
val all_scores = (m_math  union m_bigdata).map(x=>(x._1,x._3)).reduceByKey((x,y) => x+y)
val ave_scores = all_scores.map(x => (x._1,x._2/2.0))
val studentInfo = m_student.join(m_bigdata.map(x=>(x._1,x._3))).join(m_math.map(x=>(x._1,x._3))).join(all_scores ).join(ave_scores)
studentInfo.repartition(1).saveAsTextFile("/tmp/student")

保存的结果如下：

[root@dn02 ~]# hdfs dfs -cat /tmp/student/part-00000
(1005,((((赵信,90),94),184),92.0))
(1012,((((钱多多,84),91),175),87.5))
(1001,((((李正明,90),90),180),90.0))
(1009,((((李子琪,89),84),173),86.5))
(1002,((((王一磊,94),94),188),94.0))
(1006,((((古明远,93),80),173),86.5))
(1010,((((五嘉栋,78),86),164),82.0))
(1003,((((陈志华,100),100),200),100.0))
(1007,((((刘浩明,100),90),190),95.0))
(1008,((((沈彬,93),94),187),93.5))
(1011,((((柳梦文,91),79),170),85.0))
(1004,((((张永丽,99),100),199),99.5))

五、编程样例执行全过程

上述样例一全过程：

scala> val math = sc.textFile("/tmp/result_math.txt")
18/08/02 10:38:22 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 336.5 KB, free 1066.5 KB)
18/08/02 10:38:22 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 28.5 KB, free 1095.1 KB)
18/08/02 10:38:22 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:43561 (size: 28.5 KB, free: 511.0 MB)
18/08/02 10:38:22 INFO SparkContext: Created broadcast 2 from textFile at :27
math: org.apache.spark.rdd.RDD[String] = /tmp/result_math.txt MapPartitionsRDD[7] at textFile at :27

scala> val bigdata = sc.textFile("/tmp/result_bigdata.txt")
18/08/02 10:38:36 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 336.5 KB, free 1431.6 KB)
18/08/02 10:38:36 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 28.5 KB, free 1460.1 KB)
18/08/02 10:38:36 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:43561 (size: 28.5 KB, free: 511.0 MB)
18/08/02 10:38:36 INFO SparkContext: Created broadcast 3 from textFile at :27
bigdata: org.apache.spark.rdd.RDD[String] = /tmp/result_bigdata.txt MapPartitionsRDD[9] at textFile at :27

scala> val m_math  = math .map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
m_math: org.apache.spark.rdd.RDD[(String, String, Int)] = MapPartitionsRDD[10] at map at :29

scala> val m_bigdata = bigdata.map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
m_bigdata: org.apache.spark.rdd.RDD[(String, String, Int)] = MapPartitionsRDD[11] at map at :29

scala> val mathResult = m_math.sortBy(_._3,false).take(5)
18/08/02 10:39:07 INFO FileInputFormat: Total input paths to process : 1
18/08/02 10:39:07 INFO SparkContext: Starting job: sortBy at :31
18/08/02 10:39:07 INFO DAGScheduler: Got job 0 (sortBy at :31) with 2 output partitions
18/08/02 10:39:07 INFO DAGScheduler: Final stage: ResultStage 0 (sortBy at :31)
18/08/02 10:39:07 INFO DAGScheduler: Parents of final stage: List()
18/08/02 10:39:07 INFO DAGScheduler: Missing parents: List()
18/08/02 10:39:07 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[14] at sortBy at :31), which has no missing parents
18/08/02 10:39:07 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.1 KB, free 1464.2 KB)
18/08/02 10:39:07 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1466.5 KB)
18/08/02 10:39:07 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:43561 (size: 2.2 KB, free: 511.0 MB)
18/08/02 10:39:07 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1008
18/08/02 10:39:07 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[14] at sortBy at :31)
18/08/02 10:39:07 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
18/08/02 10:39:07 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,ANY, 2152 bytes)
18/08/02 10:39:07 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,ANY, 2152 bytes)
18/08/02 10:39:07 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
18/08/02 10:39:07 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
18/08/02 10:39:07 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:133+134
18/08/02 10:39:07 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:0+133
18/08/02 10:39:07 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
18/08/02 10:39:07 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
18/08/02 10:39:07 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
18/08/02 10:39:07 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
18/08/02 10:39:07 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
18/08/02 10:39:07 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 10:39:07 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2293 bytes result sent to driver
18/08/02 10:39:07 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 2293 bytes result sent to driver
18/08/02 10:39:07 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 152 ms on localhost (1/2)
18/08/02 10:39:07 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 179 ms on localhost (2/2)
18/08/02 10:39:07 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
18/08/02 10:39:07 INFO DAGScheduler: ResultStage 0 (sortBy at :31) finished in 0.199 s
18/08/02 10:39:07 INFO DAGScheduler: Job 0 finished: sortBy at :31, took 0.278488 s
18/08/02 10:39:07 INFO SparkContext: Starting job: take at :31
18/08/02 10:39:07 INFO DAGScheduler: Registering RDD 12 (sortBy at :31)
18/08/02 10:39:07 INFO DAGScheduler: Got job 1 (take at :31) with 1 output partitions
18/08/02 10:39:07 INFO DAGScheduler: Final stage: ResultStage 2 (take at :31)
18/08/02 10:39:07 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)
18/08/02 10:39:07 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1)
18/08/02 10:39:07 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[12] at sortBy at :31), which has no missing parents
18/08/02 10:39:07 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 4.7 KB, free 1471.2 KB)
18/08/02 10:39:07 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.6 KB, free 1473.8 KB)
18/08/02 10:39:07 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:43561 (size: 2.6 KB, free: 511.0 MB)
18/08/02 10:39:07 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1008
18/08/02 10:39:07 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[12] at sortBy at :31)
18/08/02 10:39:07 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
18/08/02 10:39:07 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, partition 0,ANY, 2141 bytes)
18/08/02 10:39:07 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, localhost, partition 1,ANY, 2141 bytes)
18/08/02 10:39:07 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
18/08/02 10:39:07 INFO Executor: Running task 1.0 in stage 1.0 (TID 3)
18/08/02 10:39:07 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:133+134
18/08/02 10:39:07 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:0+133
18/08/02 10:39:07 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 10:39:07 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 2309 bytes result sent to driver
18/08/02 10:39:07 INFO Executor: Finished task 1.0 in stage 1.0 (TID 3). 2309 bytes result sent to driver
18/08/02 10:39:08 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 38 ms on localhost (1/2)
18/08/02 10:39:08 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 39 ms on localhost (2/2)
18/08/02 10:39:08 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
18/08/02 10:39:08 INFO DAGScheduler: ShuffleMapStage 1 (sortBy at :31) finished in 0.042 s
18/08/02 10:39:08 INFO DAGScheduler: looking for newly runnable stages
18/08/02 10:39:08 INFO DAGScheduler: running: Set()
18/08/02 10:39:08 INFO DAGScheduler: waiting: Set(ResultStage 2)
18/08/02 10:39:08 INFO DAGScheduler: failed: Set()
18/08/02 10:39:08 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[16] at sortBy at :31), which has no missing parents
18/08/02 10:39:08 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.4 KB, free 1477.2 KB)
18/08/02 10:39:08 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2021.0 B, free 1479.2 KB)
18/08/02 10:39:08 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:43561 (size: 2021.0 B, free: 511.0 MB)
18/08/02 10:39:08 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1008
18/08/02 10:39:08 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[16] at sortBy at :31)
18/08/02 10:39:08 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
18/08/02 10:39:08 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes)
18/08/02 10:39:08 INFO Executor: Running task 0.0 in stage 2.0 (TID 4)
18/08/02 10:39:08 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
18/08/02 10:39:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms
18/08/02 10:39:08 INFO Executor: Finished task 0.0 in stage 2.0 (TID 4). 1480 bytes result sent to driver
18/08/02 10:39:08 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 62 ms on localhost (1/1)
18/08/02 10:39:08 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
18/08/02 10:39:08 INFO DAGScheduler: ResultStage 2 (take at :31) finished in 0.063 s
18/08/02 10:39:08 INFO DAGScheduler: Job 1 finished: take at :31, took 0.136507 s
mathResult: Array[(String, String, Int)] = Array((1003,应用数学,100), (1004,应用数学,100), (1002,应用数学,94), (1005,应用数学,94), (1008,应用数学,94))

scala> val bigdataResult = m_bigdata.sortBy(_._3,false).take(5)
18/08/02 10:39:17 INFO FileInputFormat: Total input paths to process : 1
18/08/02 10:39:17 INFO SparkContext: Starting job: sortBy at :31
18/08/02 10:39:17 INFO DAGScheduler: Got job 2 (sortBy at :31) with 2 output partitions
18/08/02 10:39:17 INFO DAGScheduler: Final stage: ResultStage 3 (sortBy at :31)
18/08/02 10:39:17 INFO DAGScheduler: Parents of final stage: List()
18/08/02 10:39:17 INFO DAGScheduler: Missing parents: List()
18/08/02 10:39:17 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[19] at sortBy at :31), which has no missing parents
18/08/02 10:39:17 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.2 KB, free 1483.3 KB)
18/08/02 10:39:17 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.3 KB, free 1485.6 KB)
18/08/02 10:39:17 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:43561 (size: 2.3 KB, free: 511.0 MB)
18/08/02 10:39:17 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1008
18/08/02 10:39:17 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 3 (MapPartitionsRDD[19] at sortBy at :31)
18/08/02 10:39:17 INFO TaskSchedulerImpl: Adding task set 3.0 with 2 tasks
18/08/02 10:39:17 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, localhost, partition 0,ANY, 2155 bytes)
18/08/02 10:39:17 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 6, localhost, partition 1,ANY, 2155 bytes)
18/08/02 10:39:17 INFO Executor: Running task 0.0 in stage 3.0 (TID 5)
18/08/02 10:39:17 INFO Executor: Running task 1.0 in stage 3.0 (TID 6)
18/08/02 10:39:17 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:0+151
18/08/02 10:39:17 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:151+152
18/08/02 10:39:17 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 10:39:17 INFO Executor: Finished task 0.0 in stage 3.0 (TID 5). 2348 bytes result sent to driver
18/08/02 10:39:17 INFO Executor: Finished task 1.0 in stage 3.0 (TID 6). 2348 bytes result sent to driver
18/08/02 10:39:17 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 14 ms on localhost (1/2)
18/08/02 10:39:17 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 6) in 15 ms on localhost (2/2)
18/08/02 10:39:17 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
18/08/02 10:39:17 INFO DAGScheduler: ResultStage 3 (sortBy at :31) finished in 0.017 s
18/08/02 10:39:17 INFO DAGScheduler: Job 2 finished: sortBy at :31, took 0.024577 s
18/08/02 10:39:17 INFO SparkContext: Starting job: take at :31
18/08/02 10:39:17 INFO DAGScheduler: Registering RDD 17 (sortBy at :31)
18/08/02 10:39:17 INFO DAGScheduler: Got job 3 (take at :31) with 1 output partitions
18/08/02 10:39:17 INFO DAGScheduler: Final stage: ResultStage 5 (take at :31)
18/08/02 10:39:17 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
18/08/02 10:39:17 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
18/08/02 10:39:17 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[17] at sortBy at :31), which has no missing parents
18/08/02 10:39:17 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.7 KB, free 1490.3 KB)
18/08/02 10:39:17 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.6 KB, free 1492.9 KB)
18/08/02 10:39:17 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:43561 (size: 2.6 KB, free: 511.0 MB)
18/08/02 10:39:17 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1008
18/08/02 10:39:17 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[17] at sortBy at :31)
18/08/02 10:39:17 INFO TaskSchedulerImpl: Adding task set 4.0 with 2 tasks
18/08/02 10:39:17 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 7, localhost, partition 0,ANY, 2144 bytes)
18/08/02 10:39:17 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 8, localhost, partition 1,ANY, 2144 bytes)
18/08/02 10:39:17 INFO Executor: Running task 0.0 in stage 4.0 (TID 7)
18/08/02 10:39:17 INFO Executor: Running task 1.0 in stage 4.0 (TID 8)
18/08/02 10:39:17 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:151+152
18/08/02 10:39:17 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:0+151
18/08/02 10:39:17 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 10:39:17 INFO Executor: Finished task 1.0 in stage 4.0 (TID 8). 2309 bytes result sent to driver
18/08/02 10:39:17 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 8) in 11 ms on localhost (1/2)
18/08/02 10:39:17 INFO Executor: Finished task 0.0 in stage 4.0 (TID 7). 2309 bytes result sent to driver
18/08/02 10:39:17 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 7) in 17 ms on localhost (2/2)
18/08/02 10:39:17 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
18/08/02 10:39:17 INFO DAGScheduler: ShuffleMapStage 4 (sortBy at :31) finished in 0.017 s
18/08/02 10:39:17 INFO DAGScheduler: looking for newly runnable stages
18/08/02 10:39:17 INFO DAGScheduler: running: Set()
18/08/02 10:39:17 INFO DAGScheduler: waiting: Set(ResultStage 5)
18/08/02 10:39:17 INFO DAGScheduler: failed: Set()
18/08/02 10:39:17 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[21] at sortBy at :31), which has no missing parents
18/08/02 10:39:17 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.4 KB, free 1496.3 KB)
18/08/02 10:39:17 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2021.0 B, free 1498.3 KB)
18/08/02 10:39:17 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:43561 (size: 2021.0 B, free: 511.0 MB)
18/08/02 10:39:17 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1008
18/08/02 10:39:17 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[21] at sortBy at :31)
18/08/02 10:39:17 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
18/08/02 10:39:17 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 9, localhost, partition 0,NODE_LOCAL, 1894 bytes)
18/08/02 10:39:17 INFO Executor: Running task 0.0 in stage 5.0 (TID 9)
18/08/02 10:39:17 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
18/08/02 10:39:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
18/08/02 10:39:17 INFO Executor: Finished task 0.0 in stage 5.0 (TID 9). 1505 bytes result sent to driver
18/08/02 10:39:17 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 9) in 6 ms on localhost (1/1)
18/08/02 10:39:17 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
18/08/02 10:39:17 INFO DAGScheduler: ResultStage 5 (take at :31) finished in 0.006 s
18/08/02 10:39:17 INFO DAGScheduler: Job 3 finished: take at :31, took 0.036589 s
bigdataResult: Array[(String, String, Int)] = Array((1003,大数据基础,100), (1007,大数据基础,100), (1004,大数据基础,99), (1002,大数据基础,94), (1006,大数据基础,93))

上述样例二的全过程：

scala> val student_100 = m_math.filter(_._3 == 100).union(m_bigdata.filter(_._3 == 100))
student_100: org.apache.spark.rdd.RDD[(String, String, Int)] = UnionRDD[36] at union at :35

scala> val result = student_100.map(_._1).distinct
result: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[40] at distinct at :37

scala> val resultArray = result.collect
18/08/02 10:55:13 INFO SparkContext: Starting job: collect at :39
18/08/02 10:55:13 INFO DAGScheduler: Registering RDD 38 (distinct at :37)
18/08/02 10:55:13 INFO DAGScheduler: Got job 10 (collect at :39) with 4 output partitions
18/08/02 10:55:13 INFO DAGScheduler: Final stage: ResultStage 15 (collect at :39)
18/08/02 10:55:13 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 14)
18/08/02 10:55:13 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 14)
18/08/02 10:55:13 INFO DAGScheduler: Submitting ShuffleMapStage 14 (MapPartitionsRDD[38] at distinct at :37), which has no missing parents
18/08/02 10:55:13 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 5.7 KB, free 1508.5 KB)
18/08/02 10:55:13 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 3.1 KB, free 1511.6 KB)
18/08/02 10:55:13 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:43561 (size: 3.1 KB, free: 511.0 MB)
18/08/02 10:55:13 INFO SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1008
18/08/02 10:55:13 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 14 (MapPartitionsRDD[38] at distinct at :37)
18/08/02 10:55:13 INFO TaskSchedulerImpl: Adding task set 14.0 with 4 tasks
18/08/02 10:55:13 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 38, localhost, partition 0,ANY, 2250 bytes)
18/08/02 10:55:13 INFO TaskSetManager: Starting task 1.0 in stage 14.0 (TID 39, localhost, partition 1,ANY, 2250 bytes)
18/08/02 10:55:13 INFO TaskSetManager: Starting task 2.0 in stage 14.0 (TID 40, localhost, partition 2,ANY, 2253 bytes)
18/08/02 10:55:13 INFO TaskSetManager: Starting task 3.0 in stage 14.0 (TID 41, localhost, partition 3,ANY, 2253 bytes)
18/08/02 10:55:13 INFO Executor: Running task 0.0 in stage 14.0 (TID 38)
18/08/02 10:55:13 INFO Executor: Running task 1.0 in stage 14.0 (TID 39)
18/08/02 10:55:13 INFO Executor: Running task 3.0 in stage 14.0 (TID 41)
18/08/02 10:55:13 INFO Executor: Running task 2.0 in stage 14.0 (TID 40)
18/08/02 10:55:13 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:0+133
18/08/02 10:55:13 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:151+152
18/08/02 10:55:13 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:133+134
18/08/02 10:55:13 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:0+151
18/08/02 10:55:13 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 10:55:13 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 10:55:13 INFO Executor: Finished task 1.0 in stage 14.0 (TID 39). 2311 bytes result sent to driver
18/08/02 10:55:13 INFO Executor: Finished task 0.0 in stage 14.0 (TID 38). 2311 bytes result sent to driver
18/08/02 10:55:13 INFO Executor: Finished task 2.0 in stage 14.0 (TID 40). 2311 bytes result sent to driver
18/08/02 10:55:13 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 38) in 14 ms on localhost (1/4)
18/08/02 10:55:13 INFO Executor: Finished task 3.0 in stage 14.0 (TID 41). 2311 bytes result sent to driver
18/08/02 10:55:13 INFO TaskSetManager: Finished task 2.0 in stage 14.0 (TID 40) in 14 ms on localhost (2/4)
18/08/02 10:55:13 INFO TaskSetManager: Finished task 1.0 in stage 14.0 (TID 39) in 14 ms on localhost (3/4)
18/08/02 10:55:13 INFO TaskSetManager: Finished task 3.0 in stage 14.0 (TID 41) in 14 ms on localhost (4/4)
18/08/02 10:55:13 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 
18/08/02 10:55:13 INFO DAGScheduler: ShuffleMapStage 14 (distinct at :37) finished in 0.017 s
18/08/02 10:55:13 INFO DAGScheduler: looking for newly runnable stages
18/08/02 10:55:13 INFO DAGScheduler: running: Set()
18/08/02 10:55:13 INFO DAGScheduler: waiting: Set(ResultStage 15)
18/08/02 10:55:13 INFO DAGScheduler: failed: Set()
18/08/02 10:55:13 INFO DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[40] at distinct at :37), which has no missing parents
18/08/02 10:55:13 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 3.3 KB, free 1514.9 KB)
18/08/02 10:55:13 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 1952.0 B, free 1516.8 KB)
18/08/02 10:55:13 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:43561 (size: 1952.0 B, free: 511.0 MB)
18/08/02 10:55:13 INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1008
18/08/02 10:55:13 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 15 (MapPartitionsRDD[40] at distinct at :37)
18/08/02 10:55:13 INFO TaskSchedulerImpl: Adding task set 15.0 with 4 tasks
18/08/02 10:55:13 INFO TaskSetManager: Starting task 2.0 in stage 15.0 (TID 42, localhost, partition 2,NODE_LOCAL, 1894 bytes)
18/08/02 10:55:13 INFO TaskSetManager: Starting task 3.0 in stage 15.0 (TID 43, localhost, partition 3,NODE_LOCAL, 1894 bytes)
18/08/02 10:55:13 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 44, localhost, partition 0,PROCESS_LOCAL, 1894 bytes)
18/08/02 10:55:13 INFO TaskSetManager: Starting task 1.0 in stage 15.0 (TID 45, localhost, partition 1,PROCESS_LOCAL, 1894 bytes)
18/08/02 10:55:13 INFO Executor: Running task 3.0 in stage 15.0 (TID 43)
18/08/02 10:55:13 INFO Executor: Running task 2.0 in stage 15.0 (TID 42)
18/08/02 10:55:13 INFO Executor: Running task 0.0 in stage 15.0 (TID 44)
18/08/02 10:55:13 INFO Executor: Running task 1.0 in stage 15.0 (TID 45)
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 4 blocks
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 4 blocks
18/08/02 10:55:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
18/08/02 10:55:13 INFO Executor: Finished task 1.0 in stage 15.0 (TID 45). 1165 bytes result sent to driver
18/08/02 10:55:13 INFO Executor: Finished task 0.0 in stage 15.0 (TID 44). 1165 bytes result sent to driver
18/08/02 10:55:13 INFO Executor: Finished task 3.0 in stage 15.0 (TID 43). 1172 bytes result sent to driver
18/08/02 10:55:13 INFO Executor: Finished task 2.0 in stage 15.0 (TID 42). 1179 bytes result sent to driver
18/08/02 10:55:13 INFO TaskSetManager: Finished task 1.0 in stage 15.0 (TID 45) in 5 ms on localhost (1/4)
18/08/02 10:55:13 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 44) in 5 ms on localhost (2/4)
18/08/02 10:55:13 INFO TaskSetManager: Finished task 3.0 in stage 15.0 (TID 43) in 7 ms on localhost (3/4)
18/08/02 10:55:13 INFO TaskSetManager: Finished task 2.0 in stage 15.0 (TID 42) in 7 ms on localhost (4/4)
18/08/02 10:55:13 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 
18/08/02 10:55:13 INFO DAGScheduler: ResultStage 15 (collect at :39) finished in 0.006 s
18/08/02 10:55:13 INFO DAGScheduler: Job 10 finished: collect at :39, took 0.037964 s
resultArray: Array[String] = Array(1003, 1007, 1004)

样例三执行的全过程：

scala> :paste
// Entering paste mode (ctrl-D to finish)

val math = sc.textFile("/tmp/result_math.txt")
val bigdata = sc.textFile("/tmp/result_bigdata.txt")
val m_math  = math .map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}
val m_bigdata = bigdata.map{x=>val line=x.split(" ");(line(0),line(1),line(2).toInt)}

val all_scores = (m_math  union m_bigdata).map(x=>(x._1,x._3)).reduceByKey((x,y) => x+y).collect

// Exiting paste mode, now interpreting.

18/08/02 16:35:14 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 336.5 KB, free 336.5 KB)
18/08/02 16:35:14 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 28.5 KB, free 365.0 KB)
18/08/02 16:35:14 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:51245 (size: 28.5 KB, free: 511.1 MB)
18/08/02 16:35:14 INFO SparkContext: Created broadcast 0 from textFile at :27
18/08/02 16:35:14 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 336.5 KB, free 701.5 KB)
18/08/02 16:35:14 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 28.5 KB, free 730.0 KB)
18/08/02 16:35:14 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:51245 (size: 28.5 KB, free: 511.1 MB)
18/08/02 16:35:14 INFO SparkContext: Created broadcast 1 from textFile at :28
18/08/02 16:35:14 INFO FileInputFormat: Total input paths to process : 1
18/08/02 16:35:14 INFO FileInputFormat: Total input paths to process : 1
18/08/02 16:35:14 INFO SparkContext: Starting job: collect at :32
18/08/02 16:35:14 INFO DAGScheduler: Registering RDD 7 (map at :32)
18/08/02 16:35:14 INFO DAGScheduler: Got job 0 (collect at :32) with 4 output partitions
18/08/02 16:35:14 INFO DAGScheduler: Final stage: ResultStage 1 (collect at :32)
18/08/02 16:35:14 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
18/08/02 16:35:14 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
18/08/02 16:35:14 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[7] at map at :32), which has no missing parents
18/08/02 16:35:14 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.2 KB, free 735.2 KB)
18/08/02 16:35:14 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.9 KB, free 738.1 KB)
18/08/02 16:35:14 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:51245 (size: 2.9 KB, free: 511.1 MB)
18/08/02 16:35:14 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1008
18/08/02 16:35:14 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[7] at map at :32)
18/08/02 16:35:14 INFO TaskSchedulerImpl: Adding task set 0.0 with 4 tasks
18/08/02 16:35:14 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,ANY, 2250 bytes)
18/08/02 16:35:14 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,ANY, 2250 bytes)
18/08/02 16:35:14 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,ANY, 2253 bytes)
18/08/02 16:35:14 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,ANY, 2253 bytes)
18/08/02 16:35:14 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
18/08/02 16:35:14 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
18/08/02 16:35:14 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
18/08/02 16:35:14 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
18/08/02 16:35:14 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:0+133
18/08/02 16:35:14 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_math.txt:133+134
18/08/02 16:35:14 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:151+152
18/08/02 16:35:14 INFO HadoopRDD: Input split: hdfs://nn01.bmsoft.com.cn:8020/tmp/result_bigdata.txt:0+151
18/08/02 16:35:14 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
18/08/02 16:35:14 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
18/08/02 16:35:14 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
18/08/02 16:35:14 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
18/08/02 16:35:14 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
18/08/02 16:35:14 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 16:35:14 INFO LineRecordReader: Found UTF-8 BOM and skipped it
18/08/02 16:35:14 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 2256 bytes result sent to driver
18/08/02 16:35:14 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 2256 bytes result sent to driver
18/08/02 16:35:14 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 2256 bytes result sent to driver
18/08/02 16:35:14 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 2256 bytes result sent to driver
18/08/02 16:35:14 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 180 ms on localhost (1/4)
18/08/02 16:35:14 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 183 ms on localhost (2/4)
18/08/02 16:35:14 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 204 ms on localhost (3/4)
18/08/02 16:35:14 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 182 ms on localhost (4/4)
18/08/02 16:35:14 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
18/08/02 16:35:14 INFO DAGScheduler: ShuffleMapStage 0 (map at :32) finished in 0.222 s
18/08/02 16:35:14 INFO DAGScheduler: looking for newly runnable stages
18/08/02 16:35:14 INFO DAGScheduler: running: Set()
18/08/02 16:35:14 INFO DAGScheduler: waiting: Set(ResultStage 1)
18/08/02 16:35:14 INFO DAGScheduler: failed: Set()
18/08/02 16:35:14 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[8] at reduceByKey at :32), which has no missing parents
18/08/02 16:35:14 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 2.7 KB, free 740.8 KB)
18/08/02 16:35:14 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1619.0 B, free 742.3 KB)
18/08/02 16:35:14 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:51245 (size: 1619.0 B, free: 511.1 MB)
18/08/02 16:35:14 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1008
18/08/02 16:35:14 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (ShuffledRDD[8] at reduceByKey at :32)
18/08/02 16:35:14 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks
18/08/02 16:35:14 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, partition 0,NODE_LOCAL, 1894 bytes)
18/08/02 16:35:14 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 5, localhost, partition 1,NODE_LOCAL, 1894 bytes)
18/08/02 16:35:14 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 6, localhost, partition 2,NODE_LOCAL, 1894 bytes)
18/08/02 16:35:14 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 7, localhost, partition 3,NODE_LOCAL, 1894 bytes)
18/08/02 16:35:14 INFO Executor: Running task 1.0 in stage 1.0 (TID 5)
18/08/02 16:35:14 INFO Executor: Running task 0.0 in stage 1.0 (TID 4)
18/08/02 16:35:14 INFO Executor: Running task 2.0 in stage 1.0 (TID 6)
18/08/02 16:35:14 INFO Executor: Running task 3.0 in stage 1.0 (TID 7)
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 7 ms
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 7 ms
18/08/02 16:35:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
18/08/02 16:35:14 INFO Executor: Finished task 2.0 in stage 1.0 (TID 6). 1356 bytes result sent to driver
18/08/02 16:35:14 INFO Executor: Finished task 1.0 in stage 1.0 (TID 5). 1333 bytes result sent to driver
18/08/02 16:35:14 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 6) in 42 ms on localhost (1/4)
18/08/02 16:35:14 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 5) in 44 ms on localhost (2/4)
18/08/02 16:35:14 INFO Executor: Finished task 0.0 in stage 1.0 (TID 4). 1382 bytes result sent to driver
18/08/02 16:35:14 INFO Executor: Finished task 3.0 in stage 1.0 (TID 7). 1356 bytes result sent to driver
18/08/02 16:35:14 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 49 ms on localhost (3/4)
18/08/02 16:35:14 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 7) in 45 ms on localhost (4/4)
18/08/02 16:35:14 INFO DAGScheduler: ResultStage 1 (collect at :32) finished in 0.048 s
18/08/02 16:35:14 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
18/08/02 16:35:14 INFO DAGScheduler: Job 0 finished: collect at :32, took 0.417303 s
math: org.apache.spark.rdd.RDD[String] = /tmp/result_math.txt MapPartitionsRDD[1] at textFile at :27
bigdata: org.apache.spark.rdd.RDD[String] = /tmp/result_bigdata.txt MapPartitionsRDD[3] at textFile at :28
m_math: org.apache.spark.rdd.RDD[(String, String, Int)] = MapPartitionsRDD[4] at map at :29
m_bigdata: org.apache.spark.rdd.RDD[(String, String, Int)] = MapPartitionsRDD[5] at map at :30
all_scores: Array[(String, Int)] = Array((1005,184), (1012,175), (1001,180), (1009,173), (1002,188), (1006,173), (1010,164), (1003,200), (1007,190), (1008,187), (1011,170), (1004,199))

scala>

样例4执行过程：

前面步骤与样例3一样的已省略

scala> val aveScores = all_scores.map(x => (x._1,x._2/2.0))
aveScores: org.apache.spark.rdd.RDD[(String, Double)] = MapPartitionsRDD[10] at map at :37

scala> aveScores.collect
18/08/03 10:44:00 INFO SparkContext: Starting job: collect at :40
18/08/03 10:44:00 INFO DAGScheduler: Got job 2 (collect at :40) with 4 output partitions
18/08/03 10:44:00 INFO DAGScheduler: Final stage: ResultStage 5 (collect at :40)
18/08/03 10:44:00 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
18/08/03 10:44:00 INFO DAGScheduler: Missing parents: List()
18/08/03 10:44:00 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[10] at map at :37), which has no missing parents
18/08/03 10:44:00 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.1 KB, free 750.3 KB)
18/08/03 10:44:00 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1842.0 B, free 752.1 KB)
18/08/03 10:44:00 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:59770 (size: 1842.0 B, free: 511.1 MB)
18/08/03 10:44:00 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1008
18/08/03 10:44:00 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 5 (MapPartitionsRDD[10] at map at :37)
18/08/03 10:44:00 INFO TaskSchedulerImpl: Adding task set 5.0 with 4 tasks
18/08/03 10:44:00 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 12, localhost, partition 0,NODE_LOCAL, 1894 bytes)
18/08/03 10:44:00 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 13, localhost, partition 1,NODE_LOCAL, 1894 bytes)
18/08/03 10:44:00 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 14, localhost, partition 2,NODE_LOCAL, 1894 bytes)
18/08/03 10:44:00 INFO TaskSetManager: Starting task 3.0 in stage 5.0 (TID 15, localhost, partition 3,NODE_LOCAL, 1894 bytes)
18/08/03 10:44:00 INFO Executor: Running task 0.0 in stage 5.0 (TID 12)
18/08/03 10:44:00 INFO Executor: Running task 1.0 in stage 5.0 (TID 13)
18/08/03 10:44:00 INFO Executor: Running task 3.0 in stage 5.0 (TID 15)
18/08/03 10:44:00 INFO Executor: Running task 2.0 in stage 5.0 (TID 14)
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 4 blocks
18/08/03 10:44:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
18/08/03 10:44:00 INFO Executor: Finished task 3.0 in stage 5.0 (TID 15). 1370 bytes result sent to driver
18/08/03 10:44:00 INFO Executor: Finished task 2.0 in stage 5.0 (TID 14). 1370 bytes result sent to driver
18/08/03 10:44:00 INFO Executor: Finished task 0.0 in stage 5.0 (TID 12). 1397 bytes result sent to driver
18/08/03 10:44:00 INFO Executor: Finished task 1.0 in stage 5.0 (TID 13). 1340 bytes result sent to driver
18/08/03 10:44:00 INFO TaskSetManager: Finished task 3.0 in stage 5.0 (TID 15) in 8 ms on localhost (1/4)
18/08/03 10:44:00 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 12) in 12 ms on localhost (2/4)
18/08/03 10:44:00 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID 14) in 10 ms on localhost (3/4)
18/08/03 10:44:00 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 13) in 13 ms on localhost (4/4)
18/08/03 10:44:00 INFO DAGScheduler: ResultStage 5 (collect at :40) finished in 0.013 s
18/08/03 10:44:00 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
18/08/03 10:44:00 INFO DAGScheduler: Job 2 finished: collect at :40, took 0.020852 s
res2: Array[(String, Double)] = Array((1005,92.0), (1012,87.5), (1001,90.0), (1009,86.5), (1002,94.0), (1006,86.5), (1010,82.0), (1003,100.0), (1007,95.0), (1008,93.5), (1011,85.0), (1004,99.5))

注：spark-shell 下同样支持用Tab键进行提示操作。
参考自Spark大数据技术与应用-PPT课件。

你可能感兴趣的:(spark)

nosql数据库技术与应用知识点皆过客，揽星河 NoSQL nosql 数据库大数据数据分析数据结构非关系型数据库
Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
分享一个基于python的电子书数据采集与可视化分析 hadoop电子书数据分析与推荐系统 spark大数据毕设项目（源码、调试、LW、开题、PPT) 计算机源码社 Python项目大数据大数据 python hadoop 计算机毕业设计选题计算机毕业设计源码数据分析 spark毕设
作者：计算机源码社个人简介：本人八年开发经验，擅长Java、Python、PHP、.NET、Node.js、Android、微信小程序、爬虫、大数据、机器学习等，大家有这一块的问题可以一起交流！学习资料、程序开发、技术解答、文档报告如需要源码，可以扫取文章下方二维码联系咨询Java项目微信小程序项目Android项目Python项目PHP项目ASP.NET项目Node.js项目选题推荐项目实战|p
Spark 组件 GraphX、Streaming 叶域大数据 spark spark 大数据分布式
Spark组件GraphX、Streaming一、SparkGraphX1.1GraphX的主要概念1.2GraphX的核心操作1.3示例代码1.4GraphX的应用场景二、SparkStreaming2.1SparkStreaming的主要概念2.2示例代码2.3SparkStreaming的集成2.4SparkStreaming的应用场景SparkGraphX用于处理图和图并行计算。Graph
大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏租房推荐系统 58同城租房爬虫房源推荐系统房价预测系统计算机毕业设计机器学习深度学习人工智能 2401_84572577 程序员大数据 hadoop 人工智能
做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
Spark集群的三种模式 MelodyYN #Spark spark hadoop big data
文章目录1、Spark的由来1.1Hadoop的发展1.2MapReduce与Spark对比2、Spark内置模块3、Spark运行模式3.1Standalone模式部署配置历史服务器配置高可用运行模式3.2Yarn模式安装部署配置历史服务器运行模式4、WordCount案例1、Spark的由来定义：Hadoop主要解决，海量数据的存储和海量数据的分析计算。Spark是一种基于内存的快速、通用、可
Java中的大数据处理框架对比分析省赚客app开发者 java 开发语言
Java中的大数据处理框架对比分析大家好，我是微赚淘客系统3.0的小编，是个冬天不穿秋裤，天冷也要风度的程序猿！今天，我们将深入探讨Java中常用的大数据处理框架，并对它们进行对比分析。大数据处理框架是现代数据驱动应用的核心，它们帮助企业处理和分析海量数据，以提取有价值的信息。本文将重点介绍ApacheHadoop、ApacheSpark、ApacheFlink和ApacheStorm这四种流行的
写出渗透测试信息收集详细流程卿酌南烛_b805
一、扫描域名漏洞：域名漏洞扫描工具有AWVS、APPSCAN、Netspark、WebInspect、Nmap、Nessus、天镜、明鉴、WVSS、RSAS等。二、子域名探测：1、dns域传送漏洞2、搜索引擎查找（通过Google、bing、搜索c段）3、通过ssl证书查询网站：https://myssl.com/ssl.html和https://www.chinassl.net/ssltools
Spark MLlib模型训练—推荐算法 ALS(Alternative Least Squares) 不二人生 Spark ML 实战 spark-ml 推荐算法算法
SparkMLlib模型训练—推荐算法ALS(AlternativeLeastSquares)如果你平时爱刷抖音，或者热衷看电影，不知道有没有过这样的体验：这类影视App你用得越久，它就好像会读心术一样，总能给你推荐对胃口的内容。其实这种迎合用户喜好的推荐，离不开机器学习中的推荐算法。在今天这一讲，我们就结合两个有趣的电影推荐场景，为你讲解SparkMLlib支持的协同过滤与频繁项集算法电影推荐场
Python基础知识进阶之正则表达式_头歌python正则表达式进阶前端陈萨龙程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
分布式离线计算—Spark—基础介绍测试开发abbey 人工智能—大数据
原文作者：饥渴的小苹果原文地址：【Spark】Spark基础教程目录Spark特点Spark相对于Hadoop的优势Spark生态系统Spark基本概念Spark结构设计Spark各种概念之间的关系Executor的优点Spark运行基本流程Spark运行架构的特点Spark的部署模式Spark三种部署方式Hadoop和Spark的统一部署摘要：Spark是基于内存计算的大数据并行计算框架Spar
spark常用命令我是浣熊的微笑 spark
查看报错日志：yarnlogsapplicationIDspark2-submit--masteryarn--classcom.hik.ReadHdfstest-1.0-SNAPSHOT.jar进入$SPARK_HOME目录，输入bin/spark-submit--help可以得到该命令的使用帮助。hadoop@wyy:/app/hadoop/spark100$bin/spark-submit--
spark启动命令学不会又听不懂 spark 大数据分布式
hadoop启动：cd/root/toolssstart-dfs.sh，只需在hadoop01上启动stop-dfs.sh日志查看：cat/root/toolss/hadoop/logs/hadoop-root-datanode-hadoop03.outzookeeper启动：cd/root/toolss/zookeeperbin/zkServer.shstart，三台都要启动bin/zkServ
大数据领域的深度分析——AI是在帮助开发者还是取代他们？阳爱铭大数据与数据中台技术沉淀大数据人工智能后端数据库架构数据库开发 etl工程师 chatgpt
在大数据领域，生成式人工智能（AIGC）的应用正在迅速扩展，改变了数据科学家和开发者的工作方式。本文将从大数据的专业视角，探讨AI工具在这一领域的作用，以及它们是如何帮助开发者而非取代他们的。1.大数据领域的AI工具现状在大数据领域，AI工具已经取得了显著进展，以下是几款主要的AI工具及其功能和实际应用：ApacheSpark+MLlib：ApacheSpark是一个开源的分布式计算系统，广泛用于
大数据新视界 --大数据大厂之 Spark 性能优化秘籍：从配置到代码实践青云交大数据新视界 Spark 性能优化内存分配并行度存储级别 shuffle 减少算法优化代码实践数据读取广播变量数据倾斜 Spark 数据库
亲爱的朋友们，热烈欢迎你们来到青云交的博客！能与你们在此邂逅，我满心欢喜，深感无比荣幸。在这个瞬息万变的时代，我们每个人都在苦苦追寻一处能让心灵安然栖息的港湾。而我的博客，正是这样一个温暖美好的所在。在这里，你们不仅能够收获既富有趣味又极为实用的内容知识，还可以毫无拘束地畅所欲言，尽情分享自己独特的见解。我真诚地期待着你们的到来，愿我们能在这片小小的天地里共同成长，共同进步。本博客的精华专栏：Ja
编程常用命令总结 Yellow0523 Linux BigData 大数据
编程命令大全1.软件环境变量的配置JavaScalaSparkHadoopHive2.大数据软件常用命令Spark基本命令Spark-SQL命令Hive命令HDFS命令YARN命令Zookeeper命令kafka命令Hibench命令MySQL命令3.Linux常用命令Git命令conda命令pip命令查看Linux系统的详细信息查看Linux系统架构(X86还是ARM，两种方法都可)端口号命令L
【面试系列】Spark 高频面试题解答野老杂谈全网最全IT公司面试宝典面试 spark 职场和发展大数据
欢迎来到我的博客，很高兴能够在这里和您见面！欢迎订阅相关专栏：⭐️全网最全IT互联网公司面试宝典：收集整理全网各大IT互联网公司技术、项目、HR面试真题.⭐️AIGC时代的创新与未来：详细讲解AIGC的概念、核心技术、应用领域等内容。⭐️大数据平台建设指南：全面讲解从数据采集到数据可视化的整个过程，掌握构建现代化数据平台的核心技术和方法。⭐️《遇见Python：初识、了解与热恋》：涵盖了Pytho
spark常见面试题爱敲代码的小黑 spark 大数据分布式
文章目录1.Spark的运行流程？2.Spark中的RDD机制理解吗？3.RDD的宽窄依赖4.DAG中为什么要划分Stage？5.Spark程序执行，有时候默认为什么会产生很多task，怎么修改默认task执行个数？6.RDD中reduceBykey与groupByKey哪个性能好，为什么？7.SparkMasterHA主从切换过程不会影响到集群已有作业的运行，为什么？8.SparkMaster使
Spark面试题 golove666 面试题大全 spark 大数据分布式面试
Spark面试题1.Spark基础概念1.1解释Spark是什么以及它的主要特点Spark是什么？Spark的主要特点1.2描述Spark运行时架构和组件主要的Spark架构组件：1.3讲述Spark中的弹性分布式数据集（RDD）和数据帧（DataFrame）弹性分布式数据集（RDD）主要特征：创建和转换：使用场景：数据帧（DataFrame）主要特征：创建和操作：使用场景：RDD与DataFra
图计算：基于SparkGrpahX计算聚类系数妙龄少女郭德纲 Spark 图算法 Scala 聚类数据挖掘机器学习
图计算：基于SparkGrpahX计算聚类系数文章目录图计算：基于SparkGrpahX计算聚类系数一、什么是聚类系数二、基于SparkGraphX的聚类系数代码实现总结一、什么是聚类系数聚类系数（ClusteringCoefficient）是图计算和网络分析中的一个重要概念，用于衡量网络中节点的局部聚集程度。它有助于理解网络中节点之间的紧密程度和网络的结构特性。这是一种用来衡量图中节点聚类程度的
2024年最全使用Python求解方程_python解方程(1)，字节面试官迟到 2401_84569545 程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
Spark运行时架构 tooolik spark 架构大数据
目录一，Spark运行时架构二，YARN集群架构（一）YARN集群主要组件1、ResourceManager-资源管理器2、NodeManager-节点管理器3、Task-任务4、Container-容器5、ApplicationMaster-应用程序管理器6，总结（二）YARN集群中应用程序的执行流程三、SparkStandalone架构（一）client提交方式（二）cluster提交方式四、
使用SparkSql进行表的分析与统计 xingyuan8 大数据 java
背景我们的数据挖掘平台对数据统计有比较迫切的需求，而Spark本身对数据统计已经做了一些工作，希望梳理一下Spark已经支持的数据统计功能，后期再进行扩展。准备数据在参考文献6中下载鸢尾花数据，此处格式为iris.data格式，先将data后缀改为csv后缀（不影响使用，只是为了保证后续操作不需要修改）。数据格式如下：SepalLengthSepalWidthPetalLengthPetalWid
13.Spark Core-Spark中广播变量和累加器 __元昊__
一、前述Spark中因为算子中的真正逻辑是发送到Executor中去运行的，所以当Executor中需要引用外部变量时，需要使用广播变量。累机器相当于统筹大变量，常用于计数，统计。二、具体原理1、广播变量广播变量理解图image注意事项1、能不能将一个RDD使用广播变量广播出去？不能，因为RDD是不存储数据的。可以将RDD的结果广播出去。2、广播变量只能在Driver端定义，不能在Executor
比较Spark与Flink 傲雪凌霜，松柏长青大数据后端 spark flink 大数据
ApacheSpark和ApacheFlink都是目前非常流行的大数据处理引擎，但它们在架构、处理模式、应用场景等方面有一些显著的区别。下面是二者的对比：1.处理模式Spark:主要支持批处理（BatchProcessing），也能通过SparkStreaming处理流式数据，但SparkStreaming本质上是通过微批（micro-batching）的方式处理流数据，延迟相对较高。SparkS
Spark底层逻辑傲雪凌霜，松柏长青大数据后端 spark 大数据
ApacheSpark的底层逻辑可以从其核心概念、组件和执行流程等方面来理解。Spark提供了一个分布式数据处理框架，其底层逻辑基于批处理架构，能够在大规模集群中高效地处理数据。以下是Spark的底层逻辑的详细介绍：1.核心概念Spark的底层基于几个核心概念来实现分布式计算，包括：RDD（ResilientDistributedDataset，弹性分布式数据集）：RDD是Spark最基础的数据抽
Spark - 升级版数据源JDBC2 大猪大猪
在spark的数据源中，只支持Append,Overwrite,ErrorIfExists,Ignore,这几种模式，但是我们在线上的业务几乎全是需要upsert功能的，就是已存在的数据肯定不能覆盖，在mysql中实现就是采用：ONDUPLICATEKEYUPDATE，有没有这样一种实现？官方：不好意思，不提供，dounine：我这有呀，你来用吧。哈哈，为了方便大家的使用我已经把项目打包到mave
PySpark 静听山水 Spark spark
PySpark的本质确实是Python的一个接口层，它允许你使用Python语言来编写ApacheSpark应用程序。通过这个接口，你可以利用Spark强大的分布式计算能力，同时享受Python的易用性和灵活性。1、PySpark的工作原理PySpark的工作原理可以概括为以下几个步骤：编写Python代码：开发者使用Python语法来编写Spark应用程序。这些程序通常涉及创建RDDs（弹性分布
Ubuntu的ssh 请不要问我是谁
安装sshsudoapt-getupdatesudoapt-getinstallopenssh-server检测ssh是否启动sudops-e|grepssh创建root用户sudopasswdroot配置本机无密码ssh登录cd/home/spark0ssh-keygen-trsa-P""cat.ssh/id_rsa.pub>>.ssh/authorized_keyschmod600.ssh/a
2024年大数据最新实时数仓之实时数仓架构(Hudi) 2401_84185556 程序员大数据架构
技术框架Kafka：用于接入数据源；FlinkCDC：如果直接接入业务数据源可以考虑CDC方式，如果通过Kafka缓冲接入业务数据可以忽略;Flink：用于数据ETL，包括接入数据、处理数据及输出数据全链路数据计算任务；Spark：用于数据ETL，包括处理数据及输出数据全链路数据计算任务；Hudi：湖仓一体数据管理框架，用来管理模型数据，包括ODS/DWD/DWS/DIM/ADS等；Doris：O
实时数仓之实时数仓架构(Hudi)(1)，2024年最新熬夜整理华为最新大数据开发笔试题 2401_84181221 程序员架构大数据
+Hudi：湖仓一体数据管理框架，用来管理模型数据，包括ODS/DWD/DWS/DIM/ADS等；+Doris：OLAP引擎，同步数仓结果模型，对外提供数据服务支持；+Hbase：用来存储维表信息，维表数据来源一部分有Flink加工实时写入，另一部分是从Spark任务生产，其主要作用用来支持FlinkETL处理过程中的LookupJoin功能。这里选用Hbase原因主要因为Table的HbaseC
对于规范和实现，你会混淆吗？ yangshangchuan HotSpot
昨晚和朋友聊天，喝了点咖啡，由于我经常喝茶，很长时间没喝咖啡了，所以失眠了，于是起床读JVM规范，读完后在朋友圈发了一条信息： JVM Run-Time Data Areas：The Java Virtual Machine defines various run-time data areas that are used during execution of a program. So
android 网络百合不是茶网络
android的网络编程和java的一样没什么好分析的都是一些死的照着写就可以了,所以记录下来方便查找 , 服务器使用的是TomCat 服务器代码; servlet的使用需要在xml中注册 package servlet; import java.io.IOException; import java.util.Arr
[读书笔记]读法拉第传 comsci 读书笔记
1831年的时候,一年可以赚到1000英镑的人..应该很少的... 要成为一个科学家,没有足够的资金支持,很多实验都无法完成但是当钱赚够了以后....就不能够一直在商业和市场中徘徊......
随机数的产生沐刃青蛟随机数
c++中阐述随机数的方法有两种：一是产生假随机数（不管操作多少次，所产生的数都不会改变）这类随机数是使用了默认的种子值产生的，所以每次都是一样的。 //默认种子 for (int i = 0; i < 5; i++) { cout<<
PHP检测函数所在的文件名 IT独行者 PHP 函数
很简单的功能，用到PHP中的反射机制，具体使用的是ReflectionFunction类，可以获取指定函数所在PHP脚本中的具体位置。创建引用脚本。代码： [php] view plain copy // Filename: functions.php <?php&nbs
银行各系统功能简介文强chu 金融
银行各系统功能简介　业务系统核心业务系统业务功能包括：总账管理、卡系统管理、客户信息管理、额度控管、存款、贷款、资金业务、国际结算、支付结算、对外接口等清分清算系统以清算日期为准，将账务类交易、非账务类交易的手续费、代理费、网络服务费等相关费用，按费用类型计算应收、应付金额，经过清算人员确认后上送核心系统完成结算的过程国际结算系
Python学习1(pip django 安装以及第一个project) 小桔子 python django pip
最近开始学习python,要安装个pip的工具。听说这个工具很强大，安装了它，在安装第三方工具的话so easy!然后也下载了，按照别人给的教程开始安装，奶奶的怎么也安装不上！第一步：官方下载pip-1.5.6.tar.gz, https://pypi.python.org/pypi/pip easy! 第二部：解压这个压缩文件，会看到一个setup.p
php 数组 aichenglong PHP 排序数组循环多维数组
1 php中的创建数组 $product = array('tires','oil','spark');//array()实际上是语言结构而不是函数 2 如果需要创建一个升序的排列的数字保存在一个数组中，可以使用range()函数来自动创建数组 $numbers=range(1,10)//1 2 3 4 5 6 7 8 9 10 $numbers=range(1,10,
安装python2.7 AILIKES python
安装python2.7 1、下载可从 http://www.python.org/进行下载#wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz 2、复制解压 #mkdir -p /opt/usr/python #cp /opt/soft/Python-2
java异常的处理探讨百合不是茶 JAVA异常
//java异常 /* 1，了解java 中的异常处理机制，有三种操作 a,声明异常 b,抛出异常 c,捕获异常 2，学会使用try-catch-finally来处理异常 3，学会如何声明异常和抛出异常 4，学会创建自己的异常 */ //2，学会使用try-catch-finally来处理异常
getElementsByName实例 bijian1013 element
实例1： <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/x
探索JUnit4扩展：Runner bijian1013 java 单元测试 JUnit
参加敏捷培训时，教练提到Junit4的Runner和Rule，于是特上网查一下，发现很多都讲的太理论，或者是举的例子实在是太牵强。多搜索了几下，搜索到两篇我觉得写的非常好的文章。文章地址：http://www.blogjava.net/jiangshachina/archive/20
[MongoDB学习笔记二]MongoDB副本集 bit1129 mongodb
1. 副本集的特性 1)一台主服务器(Primary),多台从服务器(Secondary) 2)Primary挂了之后，从服务器自动完成从它们之中选举一台服务器作为主服务器，继续工作，这就解决了单点故障，因此，在这种情况下，MongoDB集群能够继续工作 3)挂了的主服务器恢复到集群中只能以Secondary服务器的角色加入进来 2
【Spark八十一】Hive in the spark assembly bit1129 assembly
Spark SQL supports most commonly used features of HiveQL. However, different HiveQL statements are executed in different manners: 1. DDL statements (e.g. CREATE TABLE, DROP TABLE, etc.)
Nginx问题定位之监控进程异常退出 ronin47
nginx在运行过程中是否稳定，是否有异常退出过？这里总结几项平时会用到的小技巧。 1. 在error.log中查看是否有signal项，如果有，看看signal是多少。比如，这是一个异常退出的情况： $grep signal error.log 2012/12/24 16:39:56 [alert] 13661#0: worker process 13666 exited on s
No grammar constraints (DTD or XML schema).....两种解决方法 byalias xml
方法一：常用方法关闭XML验证工具栏：windows => preferences => xml => xml files => validation => Indicate when no grammar is specified:选择Ignore即可。方法二：（个人推荐）添加内容如下 <?xml version=
Netty源码学习-DefaultChannelPipeline bylijinnan netty
package com.ljn.channel; /** * ChannelPipeline采用的是Intercepting Filter 模式 * 但由于用到两个双向链表和内部类，这个模式看起来不是那么明显，需要仔细查看调用过程才发现 * * 下面对ChannelPipeline作一个模拟，只模拟关键代码： */ public class Pipeline {
MYSQL数据库常用备份及恢复语句 chicony mysql
备份MySQL数据库的命令，可以加选不同的参数选项来实现不同格式的要求。 mysqldump -h主机 -u用户名 -p密码数据库名 > 文件备份MySQL数据库为带删除表的格式，能够让该备份覆盖已有数据库而不需要手动删除原有数据库。 mysqldump -–add-drop-table -uusername -ppassword databasename > ba
小白谈谈云计算--基于Google三大论文 CrazyMizzz Google 云计算 GFS
之前在没有接触到云计算之前，只是对云计算有一点点模糊的概念，觉得这是一个很高大上的东西，似乎离我们大一的还很远。后来有机会上了一节云计算的普及课程吧，并且在之前的一周里拜读了谷歌三大论文。不敢说理解，至少囫囵吞枣啃下了一大堆看不明白的理论。现在就简单聊聊我对于云计算的了解。我先说说GFS &n
hadoop 平衡空间设置方法 daizj hadoop balancer
在hdfs-site.xml中增加设置balance的带宽，默认只有1M： <property> <name>dfs.balance.bandwidthPerSec</name> <value>10485760</value> <description&g
Eclipse程序员要掌握的常用快捷键 dcj3sjt126com 编程
判断一个人的编程水平，就看他用键盘多，还是鼠标多。用键盘一是为了输入代码（当然了，也包括注释），再有就是熟练使用快捷键。曾有人在豆瓣评《卓有成效的程序员》：“人有多大懒，才有多大闲”。之前我整理了一个程序员图书列表，目的也就是通过读书，让程序员变懒。程序员作为特殊的群体，有的人可以这么懒，懒到事情都交给机器去做，而有的人又可以那么勤奋，每天都孜孜不倦得
Android学习之路 dcj3sjt126com Android学习
转自：http://blog.csdn.net/ryantang03/article/details/6901459 以前有J2EE基础，接触JAVA也有两三年的时间了，上手Android并不困难，思维上稍微转变一下就可以很快适应。以前做的都是WEB项目，现今体验移动终端项目，让我越来越觉得移动互联网应用是未来的主宰。下面说说我学习Android的感受，我学Android首先是看MARS的视
java 遍历Map的四种方法 eksliang java HashMap java 遍历Map的四种方法
转载请出自出处： http://eksliang.iteye.com/blog/2059996 package com.ickes; import java.util.HashMap; import java.util.Iterator; import java.util.Map; import java.util.Map.Entry; /** * 遍历Map的四种方式
【精典】数据库相关相关 gengzg 数据库
package C3P0; import java.sql.Connection; import java.sql.SQLException; import java.beans.PropertyVetoException; import com.mchange.v2.c3p0.ComboPooledDataSource; public class DBPool{
自动补全 huyana_town 自动补全
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml&quo
jquery在线预览PDF文件，打开PDF文件天梯梦 jquery
最主要的是使用到了一个jquery的插件jquery.media.js，使用这个插件就很容易实现了。核心代码 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.
ViewPager刷新单个页面的方法 lovelease android viewpager tag 刷新
使用ViewPager做滑动切换图片的效果时，如果图片是从网络下载的，那么再子线程中下载完图片时我们会使用handler通知UI线程，然后UI线程就可以调用mViewPager.getAdapter().notifyDataSetChanged()进行页面的刷新，但是viewpager不同于listview，你会发现单纯的调用notifyDataSetChanged()并不能刷新页面
利用按位取反（~）从复合枚举值里清除枚举值草料场 enum
以 C# 中的 System.Drawing.FontStyle 为例。如果需要同时有多种效果，如：“粗体”和“下划线”的效果，可以用按位或（|） FontStyle style = FontStyle.Bold | FontStyle.Underline; 如果需要去除 style 里的某一种效果，
Linux系统新手学习的11点建议刘星宇编程工作 linux 脚本
　　随着Linux应用的扩展许多朋友开始接触Linux，根据学习Windwos的经验往往有一些茫然的感觉：不知从何处开始学起。这里介绍学习Linux的一些建议。　　一、从基础开始：常常有些朋友在Linux论坛问一些问题，不过，其中大多数的问题都是很基础的。例如：为什么我使用一个命令的时候，系统告诉我找不到该目录，我要如何限制使用者的权限等问题，这些问题其实都不是很难的，只要了解了 Linu
hibernate dao层应用之HibernateDaoSupport二次封装 wangzhezichuan DAO Hibernate
/** * 方法描述:sql语句查询返回List<Class> * 方法备注: Class 只能是自定义类 * @param calzz * @param sql * @return * 创建人：王川 * 创建时间：Jul