qq_20545159

spark RDD transformation和action操作

spark RDD transformation和action
1.启用spark-shell,使用根目录下的test.txt作为文件的示例
scala> sc
res30: org.apache.spark.SparkContext = org.apache.spark.SparkContext@68fda8

scala> val file = sc.textFile("test.txt")
15/12/09 14:00:40 INFO MemoryStore: ensureFreeSpace(191856) called with curMem=1362582, maxMem=277877882
15/12/09 14:00:40 INFO MemoryStore: Block broadcast_28 stored as values in memory (estimated size 187.4 KB, free 263.5 MB)
15/12/09 14:00:40 INFO BlockManagerInfo: Removed broadcast_27_piece0 on 10.28.23.201:51294 in memory (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:00:40 INFO BlockManagerInfo: Removed broadcast_27_piece0 on 10.28.23.202:50706 in memory (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:00:40 INFO BlockManagerInfo: Removed broadcast_27_piece0 on 10.28.23.201:60179 in memory (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:00:40 INFO MemoryStore: ensureFreeSpace(19750) called with curMem=1549548, maxMem=277877882
15/12/09 14:00:40 INFO MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 19.3 KB, free 263.5 MB)
15/12/09 14:00:40 INFO BlockManagerInfo: Removed broadcast_25_piece0 on 10.28.23.201:60179 in memory (size: 1901.0 B, free: 264.9 MB)
15/12/09 14:00:40 INFO BlockManagerInfo: Removed broadcast_25_piece0 on 10.28.23.203:57813 in memory (size: 1901.0 B, free: 264.9 MB)
15/12/09 14:00:40 INFO BlockManagerInfo: Removed broadcast_24_piece0 on 10.28.23.201:60179 in memory (size: 1901.0 B, free: 264.9 MB)
15/12/09 14:00:41 INFO BlockManagerInfo: Removed broadcast_24_piece0 on 10.28.23.202:50706 in memory (size: 1901.0 B, free: 264.9 MB)
15/12/09 14:00:41 INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on 10.28.23.201:60179 (size: 19.3 KB, free: 264.9 MB)
15/12/09 14:00:41 INFO SparkContext: Created broadcast 28 from textFile at <console>:21
file: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[21] at textFile at <console>:21

2.文件中的内容如下
scala> file.toArray
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
15/12/09 14:02:50 INFO FileInputFormat: Total input paths to process : 1
15/12/09 14:02:50 INFO SparkContext: Starting job: toArray at <console>:24
15/12/09 14:02:50 INFO DAGScheduler: Got job 21 (toArray at <console>:24) with 2 output partitions (allowLocal=false)
15/12/09 14:02:50 INFO DAGScheduler: Final stage: ResultStage 21(toArray at <console>:24)
15/12/09 14:02:50 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:02:50 INFO DAGScheduler: Missing parents: List()
15/12/09 14:02:50 INFO DAGScheduler: Submitting ResultStage 21 (MapPartitionsRDD[21] at textFile at <console>:21), which has no missing parents
15/12/09 14:02:50 INFO MemoryStore: ensureFreeSpace(3112) called with curMem=1558888, maxMem=277877882
15/12/09 14:02:50 INFO MemoryStore: Block broadcast_29 stored as values in memory (estimated size 3.0 KB, free 263.5 MB)
15/12/09 14:02:50 INFO MemoryStore: ensureFreeSpace(1778) called with curMem=1562000, maxMem=277877882
15/12/09 14:02:50 INFO MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 1778.0 B, free 263.5 MB)
15/12/09 14:02:50 INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on 10.28.23.201:60179 (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:02:50 INFO SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:874
15/12/09 14:02:50 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 21 (MapPartitionsRDD[21] at textFile at <console>:21)
15/12/09 14:02:50 INFO TaskSchedulerImpl: Adding task set 21.0 with 2 tasks
15/12/09 14:02:50 INFO TaskSetManager: Starting task 0.0 in stage 21.0 (TID 33, 10.28.23.202, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:02:50 INFO TaskSetManager: Starting task 1.0 in stage 21.0 (TID 34, 10.28.23.201, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:02:50 INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on 10.28.23.202:50706 (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:02:50 INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on 10.28.23.202:50706 (size: 19.3 KB, free: 264.9 MB)
15/12/09 14:02:50 INFO BlockManagerInfo: Added broadcast_29_piece0 in memory on 10.28.23.201:51294 (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:02:50 INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on 10.28.23.201:51294 (size: 19.3 KB, free: 264.9 MB)
15/12/09 14:02:50 INFO TaskSetManager: Finished task 0.0 in stage 21.0 (TID 33) in 91 ms on 10.28.23.202 (1/2)
15/12/09 14:02:50 INFO TaskSetManager: Finished task 1.0 in stage 21.0 (TID 34) in 247 ms on 10.28.23.201 (2/2)
15/12/09 14:02:50 INFO TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool
15/12/09 14:02:50 INFO DAGScheduler: ResultStage 21 (toArray at <console>:24) finished in 0.248 s
15/12/09 14:02:50 INFO DAGScheduler: Job 21 finished: toArray at <console>:24, took 0.290135 s
res31: Array[String] = Array(demo test can you see the best, walking in the sun, hello world, hello spark)

filter,找出包含hello的文字
scala> var filterLine = file.filter(_.contains("hello"))
filterLine: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[22] at filter at <console>:23
scala> filterLine.toArray
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
15/12/09 14:04:40 INFO SparkContext: Starting job: toArray at <console>:26
15/12/09 14:04:40 INFO DAGScheduler: Got job 22 (toArray at <console>:26) with 2 output partitions (allowLocal=false)
15/12/09 14:04:40 INFO DAGScheduler: Final stage: ResultStage 22(toArray at <console>:26)
15/12/09 14:04:40 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:04:40 INFO DAGScheduler: Missing parents: List()
15/12/09 14:04:40 INFO DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[22] at filter at <console>:23), which has no missing parents
15/12/09 14:04:40 INFO MemoryStore: ensureFreeSpace(3336) called with curMem=1563778, maxMem=277877882
15/12/09 14:04:40 INFO MemoryStore: Block broadcast_30 stored as values in memory (estimated size 3.3 KB, free 263.5 MB)
15/12/09 14:04:40 INFO MemoryStore: ensureFreeSpace(1889) called with curMem=1567114, maxMem=277877882
15/12/09 14:04:40 INFO MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 1889.0 B, free 263.5 MB)
15/12/09 14:04:40 INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on 10.28.23.201:60179 (size: 1889.0 B, free: 264.9 MB)
15/12/09 14:04:40 INFO SparkContext: Created broadcast 30 from broadcast at DAGScheduler.scala:874
15/12/09 14:04:40 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 22 (MapPartitionsRDD[22] at filter at <console>:23)
15/12/09 14:04:40 INFO TaskSchedulerImpl: Adding task set 22.0 with 2 tasks
15/12/09 14:04:40 INFO TaskSetManager: Starting task 0.0 in stage 22.0 (TID 35, 10.28.23.203, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:04:40 INFO TaskSetManager: Starting task 1.0 in stage 22.0 (TID 36, 10.28.23.202, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:04:40 INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on 10.28.23.203:57813 (size: 1889.0 B, free: 264.9 MB)
15/12/09 14:04:40 INFO BlockManagerInfo: Added broadcast_30_piece0 in memory on 10.28.23.202:50706 (size: 1889.0 B, free: 264.9 MB)
15/12/09 14:04:40 INFO TaskSetManager: Finished task 1.0 in stage 22.0 (TID 36) in 67 ms on 10.28.23.202 (1/2)
15/12/09 14:04:40 INFO BlockManagerInfo: Added broadcast_28_piece0 in memory on 10.28.23.203:57813 (size: 19.3 KB, free: 264.9 MB)
15/12/09 14:04:40 INFO TaskSetManager: Finished task 0.0 in stage 22.0 (TID 35) in 144 ms on 10.28.23.203 (2/2)
15/12/09 14:04:40 INFO TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool
15/12/09 14:04:40 INFO DAGScheduler: ResultStage 22 (toArray at <console>:26) finished in 0.145 s
15/12/09 14:04:40 INFO DAGScheduler: Job 22 finished: toArray at <console>:26, took 0.176616 s
res32: Array[String] = Array(hello world, hello spark)

Map
scala> filterLine.map(line=>line.split(" "))
res33: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[23] at map at <console>:26
scala> filterLine.map(line=>line.split(" ")) take 4
15/12/09 14:06:32 INFO SparkContext: Starting job: take at <console>:26
15/12/09 14:06:33 INFO DAGScheduler: Got job 23 (take at <console>:26) with 1 output partitions (allowLocal=true)
15/12/09 14:06:33 INFO DAGScheduler: Final stage: ResultStage 23(take at <console>:26)
15/12/09 14:06:33 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:06:33 INFO DAGScheduler: Missing parents: List()
15/12/09 14:06:33 INFO DAGScheduler: Submitting ResultStage 23 (MapPartitionsRDD[24] at map at <console>:26), which has no missing parents
15/12/09 14:06:33 INFO MemoryStore: ensureFreeSpace(3528) called with curMem=1569003, maxMem=277877882
15/12/09 14:06:33 INFO MemoryStore: Block broadcast_31 stored as values in memory (estimated size 3.4 KB, free 263.5 MB)
15/12/09 14:06:33 INFO MemoryStore: ensureFreeSpace(1971) called with curMem=1572531, maxMem=277877882
15/12/09 14:06:33 INFO MemoryStore: Block broadcast_31_piece0 stored as bytes in memory (estimated size 1971.0 B, free 263.5 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Removed broadcast_30_piece0 on 10.28.23.202:50706 in memory (size: 1889.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Removed broadcast_30_piece0 on 10.28.23.201:60179 in memory (size: 1889.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Added broadcast_31_piece0 in memory on 10.28.23.201:60179 (size: 1971.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Removed broadcast_30_piece0 on 10.28.23.203:57813 in memory (size: 1889.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO SparkContext: Created broadcast 31 from broadcast at DAGScheduler.scala:874
15/12/09 14:06:33 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 23 (MapPartitionsRDD[24] at map at <console>:26)
15/12/09 14:06:33 INFO TaskSchedulerImpl: Adding task set 23.0 with 1 tasks
15/12/09 14:06:33 INFO TaskSetManager: Starting task 0.0 in stage 23.0 (TID 37, 10.28.23.201, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:06:33 INFO BlockManagerInfo: Removed broadcast_29_piece0 on 10.28.23.201:60179 in memory (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Removed broadcast_29_piece0 on 10.28.23.201:51294 in memory (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Removed broadcast_29_piece0 on 10.28.23.202:50706 in memory (size: 1778.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Added broadcast_31_piece0 in memory on 10.28.23.201:51294 (size: 1971.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO TaskSetManager: Finished task 0.0 in stage 23.0 (TID 37) in 536 ms on 10.28.23.201 (1/1)
15/12/09 14:06:33 INFO DAGScheduler: ResultStage 23 (take at <console>:26) finished in 0.534 s
15/12/09 14:06:33 INFO TaskSchedulerImpl: Removed TaskSet 23.0, whose tasks have all completed, from pool
15/12/09 14:06:33 INFO DAGScheduler: Job 23 finished: take at <console>:26, took 0.666443 s
15/12/09 14:06:33 INFO SparkContext: Starting job: take at <console>:26
15/12/09 14:06:33 INFO DAGScheduler: Got job 24 (take at <console>:26) with 1 output partitions (allowLocal=true)
15/12/09 14:06:33 INFO DAGScheduler: Final stage: ResultStage 24(take at <console>:26)
15/12/09 14:06:33 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:06:33 INFO DAGScheduler: Missing parents: List()
15/12/09 14:06:33 INFO DAGScheduler: Submitting ResultStage 24 (MapPartitionsRDD[24] at map at <console>:26), which has no missing parents
15/12/09 14:06:33 INFO MemoryStore: ensureFreeSpace(3528) called with curMem=1564387, maxMem=277877882
15/12/09 14:06:33 INFO MemoryStore: Block broadcast_32 stored as values in memory (estimated size 3.4 KB, free 263.5 MB)
15/12/09 14:06:33 INFO MemoryStore: ensureFreeSpace(1971) called with curMem=1567915, maxMem=277877882
15/12/09 14:06:33 INFO MemoryStore: Block broadcast_32_piece0 stored as bytes in memory (estimated size 1971.0 B, free 263.5 MB)
15/12/09 14:06:33 INFO BlockManagerInfo: Added broadcast_32_piece0 in memory on 10.28.23.201:60179 (size: 1971.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO SparkContext: Created broadcast 32 from broadcast at DAGScheduler.scala:874
15/12/09 14:06:33 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 24 (MapPartitionsRDD[24] at map at <console>:26)
15/12/09 14:06:33 INFO TaskSchedulerImpl: Adding task set 24.0 with 1 tasks
15/12/09 14:06:33 INFO TaskSetManager: Starting task 0.0 in stage 24.0 (TID 38, 10.28.23.202, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:06:33 INFO BlockManagerInfo: Added broadcast_32_piece0 in memory on 10.28.23.202:50706 (size: 1971.0 B, free: 264.9 MB)
15/12/09 14:06:33 INFO TaskSetManager: Finished task 0.0 in stage 24.0 (TID 38) in 39 ms on 10.28.23.202 (1/1)
15/12/09 14:06:33 INFO TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool
15/12/09 14:06:33 INFO DAGScheduler: ResultStage 24 (take at <console>:26) finished in 0.039 s
15/12/09 14:06:33 INFO DAGScheduler: Job 24 finished: take at <console>:26, took 0.072027 s
res34: Array[Array[String]] = Array(Array(hello, world), Array(hello, spark))
map的最终输出是Array(array1,array2...)数组

flatMap将数据扁平化
scala> file.flatMap(line=>line.split(" ")) take 20
15/12/09 14:09:07 INFO SparkContext: Starting job: take at <console>:24
15/12/09 14:09:07 INFO DAGScheduler: Got job 26 (take at <console>:24) with 1 output partitions (allowLocal=true)
15/12/09 14:09:07 INFO DAGScheduler: Final stage: ResultStage 26(take at <console>:24)
15/12/09 14:09:07 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:09:07 INFO DAGScheduler: Missing parents: List()
15/12/09 14:09:07 INFO DAGScheduler: Submitting ResultStage 26 (MapPartitionsRDD[26] at flatMap at <console>:24), which has no missing parents
15/12/09 14:09:07 INFO MemoryStore: ensureFreeSpace(3360) called with curMem=1575165, maxMem=277877882
15/12/09 14:09:07 INFO MemoryStore: Block broadcast_34 stored as values in memory (estimated size 3.3 KB, free 263.5 MB)
15/12/09 14:09:07 INFO MemoryStore: ensureFreeSpace(1923) called with curMem=1578525, maxMem=277877882
15/12/09 14:09:07 INFO MemoryStore: Block broadcast_34_piece0 stored as bytes in memory (estimated size 1923.0 B, free 263.5 MB)
15/12/09 14:09:07 INFO BlockManagerInfo: Added broadcast_34_piece0 in memory on 10.28.23.201:60179 (size: 1923.0 B, free: 264.8 MB)
15/12/09 14:09:07 INFO SparkContext: Created broadcast 34 from broadcast at DAGScheduler.scala:874
15/12/09 14:09:07 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 26 (MapPartitionsRDD[26] at flatMap at <console>:24)
15/12/09 14:09:07 INFO TaskSchedulerImpl: Adding task set 26.0 with 1 tasks
15/12/09 14:09:07 INFO TaskSetManager: Starting task 0.0 in stage 26.0 (TID 40, 10.28.23.202, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:09:07 INFO BlockManagerInfo: Added broadcast_34_piece0 in memory on 10.28.23.202:50706 (size: 1923.0 B, free: 264.9 MB)
15/12/09 14:09:07 INFO TaskSetManager: Finished task 0.0 in stage 26.0 (TID 40) in 64 ms on 10.28.23.202 (1/1)
15/12/09 14:09:07 INFO TaskSchedulerImpl: Removed TaskSet 26.0, whose tasks have all completed, from pool
15/12/09 14:09:07 INFO DAGScheduler: ResultStage 26 (take at <console>:24) finished in 0.065 s
15/12/09 14:09:07 INFO DAGScheduler: Job 26 finished: take at <console>:24, took 0.113043 s
15/12/09 14:09:07 INFO SparkContext: Starting job: take at <console>:24
15/12/09 14:09:07 INFO DAGScheduler: Got job 27 (take at <console>:24) with 1 output partitions (allowLocal=true)
15/12/09 14:09:07 INFO DAGScheduler: Final stage: ResultStage 27(take at <console>:24)
15/12/09 14:09:07 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:09:07 INFO DAGScheduler: Missing parents: List()
15/12/09 14:09:07 INFO DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[26] at flatMap at <console>:24), which has no missing parents
15/12/09 14:09:07 INFO MemoryStore: ensureFreeSpace(3360) called with curMem=1580448, maxMem=277877882
15/12/09 14:09:07 INFO MemoryStore: Block broadcast_35 stored as values in memory (estimated size 3.3 KB, free 263.5 MB)
15/12/09 14:09:07 INFO MemoryStore: ensureFreeSpace(1923) called with curMem=1583808, maxMem=277877882
15/12/09 14:09:07 INFO MemoryStore: Block broadcast_35_piece0 stored as bytes in memory (estimated size 1923.0 B, free 263.5 MB)
15/12/09 14:09:07 INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on 10.28.23.201:60179 (size: 1923.0 B, free: 264.8 MB)
15/12/09 14:09:07 INFO SparkContext: Created broadcast 35 from broadcast at DAGScheduler.scala:874
15/12/09 14:09:07 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 27 (MapPartitionsRDD[26] at flatMap at <console>:24)
15/12/09 14:09:07 INFO TaskSchedulerImpl: Adding task set 27.0 with 1 tasks
15/12/09 14:09:07 INFO TaskSetManager: Starting task 0.0 in stage 27.0 (TID 41, 10.28.23.201, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:09:08 INFO BlockManagerInfo: Added broadcast_35_piece0 in memory on 10.28.23.201:51294 (size: 1923.0 B, free: 264.9 MB)
15/12/09 14:09:08 INFO TaskSetManager: Finished task 0.0 in stage 27.0 (TID 41) in 312 ms on 10.28.23.201 (1/1)
15/12/09 14:09:08 INFO TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool
15/12/09 14:09:08 INFO DAGScheduler: ResultStage 27 (take at <console>:24) finished in 0.311 s
15/12/09 14:09:08 INFO DAGScheduler: Job 27 finished: take at <console>:24, took 0.324655 s
res36: Array[String] = Array(demo, test, can, you, see, the, best, walking, in, the, sun, hello, world, hello, spark)

union
scala> val unionLine = file.union(file)
unionLine: org.apache.spark.rdd.RDD[String] = UnionRDD[30] at union at <console>:23

scala> unionLine.toArray
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
15/12/09 14:11:43 INFO SparkContext: Starting job: toArray at <console>:26
15/12/09 14:11:43 INFO DAGScheduler: Got job 31 (toArray at <console>:26) with 4 output partitions (allowLocal=false)
15/12/09 14:11:43 INFO DAGScheduler: Final stage: ResultStage 31(toArray at <console>:26)
15/12/09 14:11:43 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:11:43 INFO DAGScheduler: Missing parents: List()
15/12/09 14:11:43 INFO DAGScheduler: Submitting ResultStage 31 (UnionRDD[30] at union at <console>:23), which has no missing parents
15/12/09 14:11:43 INFO MemoryStore: ensureFreeSpace(3688) called with curMem=1564712, maxMem=277877882
15/12/09 14:11:43 INFO MemoryStore: Block broadcast_39 stored as values in memory (estimated size 3.6 KB, free 263.5 MB)
15/12/09 14:11:43 INFO MemoryStore: ensureFreeSpace(2158) called with curMem=1568400, maxMem=277877882
15/12/09 14:11:43 INFO MemoryStore: Block broadcast_39_piece0 stored as bytes in memory (estimated size 2.1 KB, free 263.5 MB)
15/12/09 14:11:43 INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on 10.28.23.201:60179 (size: 2.1 KB, free: 264.9 MB)
15/12/09 14:11:43 INFO SparkContext: Created broadcast 39 from broadcast at DAGScheduler.scala:874
15/12/09 14:11:43 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 31 (UnionRDD[30] at union at <console>:23)
15/12/09 14:11:43 INFO TaskSchedulerImpl: Adding task set 31.0 with 4 tasks
15/12/09 14:11:43 INFO TaskSetManager: Starting task 0.0 in stage 31.0 (TID 50, 10.28.23.203, PROCESS_LOCAL, 1506 bytes)
15/12/09 14:11:43 INFO TaskSetManager: Starting task 1.0 in stage 31.0 (TID 51, 10.28.23.202, PROCESS_LOCAL, 1506 bytes)
15/12/09 14:11:43 INFO TaskSetManager: Starting task 2.0 in stage 31.0 (TID 52, 10.28.23.201, PROCESS_LOCAL, 1506 bytes)
15/12/09 14:11:43 INFO TaskSetManager: Starting task 3.0 in stage 31.0 (TID 53, 10.28.23.203, PROCESS_LOCAL, 1506 bytes)
15/12/09 14:11:43 INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on 10.28.23.202:50706 (size: 2.1 KB, free: 264.9 MB)
15/12/09 14:11:43 INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on 10.28.23.203:57813 (size: 2.1 KB, free: 264.9 MB)
15/12/09 14:11:43 INFO TaskSetManager: Finished task 1.0 in stage 31.0 (TID 51) in 47 ms on 10.28.23.202 (1/4)
15/12/09 14:11:43 INFO BlockManagerInfo: Added broadcast_39_piece0 in memory on 10.28.23.201:51294 (size: 2.1 KB, free: 264.9 MB)
15/12/09 14:11:43 INFO TaskSetManager: Finished task 0.0 in stage 31.0 (TID 50) in 70 ms on 10.28.23.203 (2/4)
15/12/09 14:11:43 INFO TaskSetManager: Finished task 3.0 in stage 31.0 (TID 53) in 69 ms on 10.28.23.203 (3/4)
15/12/09 14:11:43 INFO TaskSetManager: Finished task 2.0 in stage 31.0 (TID 52) in 89 ms on 10.28.23.201 (4/4)
15/12/09 14:11:43 INFO TaskSchedulerImpl: Removed TaskSet 31.0, whose tasks have all completed, from pool
15/12/09 14:11:43 INFO DAGScheduler: ResultStage 31 (toArray at <console>:26) finished in 0.091 s
15/12/09 14:11:43 INFO DAGScheduler: Job 31 finished: toArray at <console>:26, took 0.109727 s

res43: Array[String] = Array(demo test can you see the best, walking in the sun, hello world, hello spark, demo test can you see the best, walking in the sun, hello world, hello spark)

flatMap
scala> val wordcount = file.flatMap(line=>line.split(" ")).map(word=>(word,1))
wordcount: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[42] at map at <console>:23

scala> wordcount.toArray
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
15/12/09 14:19:21 INFO SparkContext: Starting job: toArray at <console>:26
15/12/09 14:19:21 INFO DAGScheduler: Got job 36 (toArray at <console>:26) with 2 output partitions (allowLocal=false)
15/12/09 14:19:21 INFO DAGScheduler: Final stage: ResultStage 36(toArray at <console>:26)
15/12/09 14:19:21 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:19:21 INFO DAGScheduler: Missing parents: List()
15/12/09 14:19:21 INFO DAGScheduler: Submitting ResultStage 36 (MapPartitionsRDD[42] at map at <console>:23), which has no missing parents
15/12/09 14:19:21 INFO MemoryStore: ensureFreeSpace(3488) called with curMem=1564303, maxMem=277877882
15/12/09 14:19:21 INFO MemoryStore: Block broadcast_44 stored as values in memory (estimated size 3.4 KB, free 263.5 MB)
15/12/09 14:19:21 INFO MemoryStore: ensureFreeSpace(1927) called with curMem=1567791, maxMem=277877882
15/12/09 14:19:21 INFO MemoryStore: Block broadcast_44_piece0 stored as bytes in memory (estimated size 1927.0 B, free 263.5 MB)
15/12/09 14:19:21 INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on 10.28.23.201:60179 (size: 1927.0 B, free: 264.9 MB)
15/12/09 14:19:21 INFO SparkContext: Created broadcast 44 from broadcast at DAGScheduler.scala:874
15/12/09 14:19:21 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 36 (MapPartitionsRDD[42] at map at <console>:23)
15/12/09 14:19:21 INFO TaskSchedulerImpl: Adding task set 36.0 with 2 tasks
15/12/09 14:19:21 INFO TaskSetManager: Starting task 0.0 in stage 36.0 (TID 62, 10.28.23.201, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:19:21 INFO TaskSetManager: Starting task 1.0 in stage 36.0 (TID 63, 10.28.23.203, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:19:21 INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on 10.28.23.201:51294 (size: 1927.0 B, free: 264.9 MB)
15/12/09 14:19:21 INFO BlockManagerInfo: Added broadcast_44_piece0 in memory on 10.28.23.203:57813 (size: 1927.0 B, free: 264.9 MB)
15/12/09 14:19:21 INFO TaskSetManager: Finished task 1.0 in stage 36.0 (TID 63) in 107 ms on 10.28.23.203 (1/2)
15/12/09 14:19:21 INFO TaskSetManager: Finished task 0.0 in stage 36.0 (TID 62) in 133 ms on 10.28.23.201 (2/2)
15/12/09 14:19:21 INFO TaskSchedulerImpl: Removed TaskSet 36.0, whose tasks have all completed, from pool
15/12/09 14:19:21 INFO DAGScheduler: ResultStage 36 (toArray at <console>:26) finished in 0.141 s
15/12/09 14:19:21 INFO DAGScheduler: Job 36 finished: toArray at <console>:26, took 0.198194 s
res53: Array[(String, Int)] = Array((demo,1), (test,1), (can,1), (you,1), (see,1), (the,1), (best,1), (walking,1), (in,1), (the,1), (sun,1), (hello,1), (world,1), (hello,1), (spark,1))

groupByKey
scala> wordcount.groupByKey().toArray
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
15/12/09 14:20:32 INFO SparkContext: Starting job: toArray at <console>:28
15/12/09 14:20:32 INFO DAGScheduler: Registering RDD 42 (map at <console>:23)
15/12/09 14:20:32 INFO DAGScheduler: Got job 37 (toArray at <console>:28) with 2 output partitions (allowLocal=false)
15/12/09 14:20:32 INFO DAGScheduler: Final stage: ResultStage 38(toArray at <console>:28)
15/12/09 14:20:32 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 37)
15/12/09 14:20:32 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 37)
15/12/09 14:20:32 INFO DAGScheduler: Submitting ShuffleMapStage 37 (MapPartitionsRDD[42] at map at <console>:23), which has no missing parents
15/12/09 14:20:32 INFO MemoryStore: ensureFreeSpace(4864) called with curMem=1569718, maxMem=277877882
15/12/09 14:20:32 INFO MemoryStore: Block broadcast_45 stored as values in memory (estimated size 4.8 KB, free 263.5 MB)
15/12/09 14:20:32 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=1574582, maxMem=277877882
15/12/09 14:20:32 INFO MemoryStore: Block broadcast_45_piece0 stored as bytes in memory (estimated size 2.5 KB, free 263.5 MB)
15/12/09 14:20:32 INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on 10.28.23.201:60179 (size: 2.5 KB, free: 264.8 MB)
15/12/09 14:20:32 INFO SparkContext: Created broadcast 45 from broadcast at DAGScheduler.scala:874
15/12/09 14:20:32 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 37 (MapPartitionsRDD[42] at map at <console>:23)
15/12/09 14:20:32 INFO TaskSchedulerImpl: Adding task set 37.0 with 2 tasks
15/12/09 14:20:32 INFO TaskSetManager: Starting task 0.0 in stage 37.0 (TID 64, 10.28.23.202, PROCESS_LOCAL, 1386 bytes)
15/12/09 14:20:32 INFO TaskSetManager: Starting task 1.0 in stage 37.0 (TID 65, 10.28.23.201, PROCESS_LOCAL, 1386 bytes)
15/12/09 14:20:32 INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on 10.28.23.202:50706 (size: 2.5 KB, free: 264.9 MB)
15/12/09 14:20:32 INFO BlockManagerInfo: Added broadcast_45_piece0 in memory on 10.28.23.201:51294 (size: 2.5 KB, free: 264.9 MB)
15/12/09 14:20:32 INFO TaskSetManager: Finished task 0.0 in stage 37.0 (TID 64) in 274 ms on 10.28.23.202 (1/2)
15/12/09 14:20:34 INFO TaskSetManager: Finished task 1.0 in stage 37.0 (TID 65) in 1690 ms on 10.28.23.201 (2/2)
15/12/09 14:20:34 INFO TaskSchedulerImpl: Removed TaskSet 37.0, whose tasks have all completed, from pool
15/12/09 14:20:34 INFO DAGScheduler: ShuffleMapStage 37 (map at <console>:23) finished in 1.691 s
15/12/09 14:20:34 INFO DAGScheduler: looking for newly runnable stages
15/12/09 14:20:34 INFO DAGScheduler: running: Set()
15/12/09 14:20:34 INFO DAGScheduler: waiting: Set(ResultStage 38)
15/12/09 14:20:34 INFO DAGScheduler: failed: Set()
15/12/09 14:20:34 INFO DAGScheduler: Missing parents for ResultStage 38: List()
15/12/09 14:20:34 INFO DAGScheduler: Submitting ResultStage 38 (ShuffledRDD[43] at groupByKey at <console>:25), which is now runnable
15/12/09 14:20:34 INFO MemoryStore: ensureFreeSpace(5384) called with curMem=1577146, maxMem=277877882
15/12/09 14:20:34 INFO MemoryStore: Block broadcast_46 stored as values in memory (estimated size 5.3 KB, free 263.5 MB)
15/12/09 14:20:34 INFO MemoryStore: ensureFreeSpace(2765) called with curMem=1582530, maxMem=277877882
15/12/09 14:20:34 INFO MemoryStore: Block broadcast_46_piece0 stored as bytes in memory (estimated size 2.7 KB, free 263.5 MB)
15/12/09 14:20:34 INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on 10.28.23.201:60179 (size: 2.7 KB, free: 264.8 MB)
15/12/09 14:20:34 INFO SparkContext: Created broadcast 46 from broadcast at DAGScheduler.scala:874
15/12/09 14:20:34 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 38 (ShuffledRDD[43] at groupByKey at <console>:25)
15/12/09 14:20:34 INFO TaskSchedulerImpl: Adding task set 38.0 with 2 tasks
15/12/09 14:20:34 INFO TaskSetManager: Starting task 0.0 in stage 38.0 (TID 66, 10.28.23.203, PROCESS_LOCAL, 1165 bytes)
15/12/09 14:20:34 INFO TaskSetManager: Starting task 1.0 in stage 38.0 (TID 67, 10.28.23.202, PROCESS_LOCAL, 1165 bytes)
15/12/09 14:20:34 INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on 10.28.23.202:50706 (size: 2.7 KB, free: 264.9 MB)
15/12/09 14:20:34 INFO BlockManagerInfo: Added broadcast_46_piece0 in memory on 10.28.23.203:57813 (size: 2.7 KB, free: 264.9 MB)
15/12/09 14:20:34 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.28.23.202:53700
15/12/09 14:20:34 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 162 bytes
15/12/09 14:20:34 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.28.23.203:56943
15/12/09 14:20:34 INFO TaskSetManager: Finished task 1.0 in stage 38.0 (TID 67) in 552 ms on 10.28.23.202 (1/2)
15/12/09 14:20:34 INFO TaskSetManager: Finished task 0.0 in stage 38.0 (TID 66) in 558 ms on 10.28.23.203 (2/2)
15/12/09 14:20:34 INFO TaskSchedulerImpl: Removed TaskSet 38.0, whose tasks have all completed, from pool
15/12/09 14:20:34 INFO DAGScheduler: ResultStage 38 (toArray at <console>:28) finished in 0.558 s
15/12/09 14:20:34 INFO DAGScheduler: Job 37 finished: toArray at <console>:28, took 2.518733 s
res54: Array[(String, Iterable[Int])] = Array((can,CompactBuffer(1)), (best,CompactBuffer(1)), (hello,CompactBuffer(1, 1)), (sun,CompactBuffer(1)), (test,CompactBuffer(1)), (world,CompactBuffer(1)), (walking,CompactBuffer(1)), (spark,CompactBuffer(1)), (you,CompactBuffer(1)), (demo,CompactBuffer(1)), (in,CompactBuffer(1)), (see,CompactBuffer(1)), (the,CompactBuffer(1, 1)))

distinct
scala> wordcount.count()
15/12/09 14:29:52 INFO SparkContext: Starting job: count at <console>:26
15/12/09 14:29:52 INFO DAGScheduler: Got job 48 (count at <console>:26) with 2 output partitions (allowLocal=false)
15/12/09 14:29:52 INFO DAGScheduler: Final stage: ResultStage 58(count at <console>:26)
15/12/09 14:29:52 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:29:52 INFO DAGScheduler: Missing parents: List()
15/12/09 14:29:52 INFO DAGScheduler: Submitting ResultStage 58 (MapPartitionsRDD[42] at map at <console>:23), which has no missing parents
15/12/09 14:29:52 INFO MemoryStore: ensureFreeSpace(3344) called with curMem=1579816, maxMem=277877882
15/12/09 14:29:52 INFO MemoryStore: Block broadcast_65 stored as values in memory (estimated size 3.3 KB, free 263.5 MB)
15/12/09 14:29:52 INFO MemoryStore: ensureFreeSpace(1893) called with curMem=1583160, maxMem=277877882
15/12/09 14:29:52 INFO MemoryStore: Block broadcast_65_piece0 stored as bytes in memory (estimated size 1893.0 B, free 263.5 MB)
15/12/09 14:29:52 INFO BlockManagerInfo: Added broadcast_65_piece0 in memory on 10.28.23.201:60179 (size: 1893.0 B, free: 264.8 MB)
15/12/09 14:29:52 INFO SparkContext: Created broadcast 65 from broadcast at DAGScheduler.scala:874
15/12/09 14:29:52 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 58 (MapPartitionsRDD[42] at map at <console>:23)
15/12/09 14:29:52 INFO TaskSchedulerImpl: Adding task set 58.0 with 2 tasks
15/12/09 14:29:52 INFO TaskSetManager: Starting task 0.0 in stage 58.0 (TID 104, 10.28.23.203, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:29:52 INFO TaskSetManager: Starting task 1.0 in stage 58.0 (TID 105, 10.28.23.201, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:29:52 INFO BlockManagerInfo: Added broadcast_65_piece0 in memory on 10.28.23.201:51294 (size: 1893.0 B, free: 264.9 MB)
15/12/09 14:29:52 INFO BlockManagerInfo: Added broadcast_65_piece0 in memory on 10.28.23.203:57813 (size: 1893.0 B, free: 264.9 MB)
15/12/09 14:29:52 INFO TaskSetManager: Finished task 0.0 in stage 58.0 (TID 104) in 67 ms on 10.28.23.203 (1/2)
15/12/09 14:29:52 INFO TaskSetManager: Finished task 1.0 in stage 58.0 (TID 105) in 69 ms on 10.28.23.201 (2/2)
15/12/09 14:29:52 INFO DAGScheduler: ResultStage 58 (count at <console>:26) finished in 0.083 s
15/12/09 14:29:52 INFO TaskSchedulerImpl: Removed TaskSet 58.0, whose tasks have all completed, from pool
15/12/09 14:29:52 INFO DAGScheduler: Job 48 finished: count at <console>:26, took 0.119783 s
res72: Long = 15
scala> wordcount.distinct.count()
15/12/09 14:30:10 INFO SparkContext: Starting job: count at <console>:26
15/12/09 14:30:10 INFO DAGScheduler: Registering RDD 52 (distinct at <console>:26)
15/12/09 14:30:10 INFO DAGScheduler: Got job 50 (count at <console>:26) with 2 output partitions (allowLocal=false)
15/12/09 14:30:10 INFO DAGScheduler: Final stage: ResultStage 61(count at <console>:26)
15/12/09 14:30:10 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 60)
15/12/09 14:30:10 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 60)
15/12/09 14:30:10 INFO DAGScheduler: Submitting ShuffleMapStage 60 (MapPartitionsRDD[52] at distinct at <console>:26), which has no missing parents
15/12/09 14:30:10 INFO MemoryStore: ensureFreeSpace(4232) called with curMem=1564303, maxMem=277877882
15/12/09 14:30:10 INFO MemoryStore: Block broadcast_67 stored as values in memory (estimated size 4.1 KB, free 263.5 MB)
15/12/09 14:30:10 INFO MemoryStore: ensureFreeSpace(2306) called with curMem=1568535, maxMem=277877882
15/12/09 14:30:10 INFO MemoryStore: Block broadcast_67_piece0 stored as bytes in memory (estimated size 2.3 KB, free 263.5 MB)
15/12/09 14:30:10 INFO BlockManagerInfo: Added broadcast_67_piece0 in memory on 10.28.23.201:60179 (size: 2.3 KB, free: 264.9 MB)
15/12/09 14:30:10 INFO SparkContext: Created broadcast 67 from broadcast at DAGScheduler.scala:874
15/12/09 14:30:10 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 60 (MapPartitionsRDD[52] at distinct at <console>:26)
15/12/09 14:30:10 INFO TaskSchedulerImpl: Adding task set 60.0 with 2 tasks
15/12/09 14:30:10 INFO TaskSetManager: Starting task 0.0 in stage 60.0 (TID 108, 10.28.23.203, PROCESS_LOCAL, 1386 bytes)
15/12/09 14:30:10 INFO TaskSetManager: Starting task 1.0 in stage 60.0 (TID 109, 10.28.23.201, PROCESS_LOCAL, 1386 bytes)
15/12/09 14:30:10 INFO BlockManagerInfo: Added broadcast_67_piece0 in memory on 10.28.23.201:51294 (size: 2.3 KB, free: 264.9 MB)
15/12/09 14:30:10 INFO BlockManagerInfo: Added broadcast_67_piece0 in memory on 10.28.23.203:57813 (size: 2.3 KB, free: 264.9 MB)
15/12/09 14:30:10 INFO TaskSetManager: Finished task 0.0 in stage 60.0 (TID 108) in 75 ms on 10.28.23.203 (1/2)
15/12/09 14:30:10 INFO TaskSetManager: Finished task 1.0 in stage 60.0 (TID 109) in 135 ms on 10.28.23.201 (2/2)
15/12/09 14:30:10 INFO TaskSchedulerImpl: Removed TaskSet 60.0, whose tasks have all completed, from pool
15/12/09 14:30:10 INFO DAGScheduler: ShuffleMapStage 60 (distinct at <console>:26) finished in 0.135 s
15/12/09 14:30:10 INFO DAGScheduler: looking for newly runnable stages
15/12/09 14:30:10 INFO DAGScheduler: running: Set()
15/12/09 14:30:10 INFO DAGScheduler: waiting: Set(ResultStage 61)
15/12/09 14:30:10 INFO DAGScheduler: failed: Set()
15/12/09 14:30:10 INFO DAGScheduler: Missing parents for ResultStage 61: List()
15/12/09 14:30:10 INFO DAGScheduler: Submitting ResultStage 61 (MapPartitionsRDD[54] at distinct at <console>:26), which is now runnable
15/12/09 14:30:10 INFO MemoryStore: ensureFreeSpace(2584) called with curMem=1570841, maxMem=277877882
15/12/09 14:30:10 INFO MemoryStore: Block broadcast_68 stored as values in memory (estimated size 2.5 KB, free 263.5 MB)
15/12/09 14:30:10 INFO MemoryStore: ensureFreeSpace(1530) called with curMem=1573425, maxMem=277877882
15/12/09 14:30:10 INFO MemoryStore: Block broadcast_68_piece0 stored as bytes in memory (estimated size 1530.0 B, free 263.5 MB)
15/12/09 14:30:10 INFO BlockManagerInfo: Added broadcast_68_piece0 in memory on 10.28.23.201:60179 (size: 1530.0 B, free: 264.8 MB)
15/12/09 14:30:10 INFO SparkContext: Created broadcast 68 from broadcast at DAGScheduler.scala:874
15/12/09 14:30:10 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 61 (MapPartitionsRDD[54] at distinct at <console>:26)
15/12/09 14:30:10 INFO TaskSchedulerImpl: Adding task set 61.0 with 2 tasks
15/12/09 14:30:10 INFO TaskSetManager: Starting task 0.0 in stage 61.0 (TID 110, 10.28.23.201, PROCESS_LOCAL, 1165 bytes)
15/12/09 14:30:10 INFO TaskSetManager: Starting task 1.0 in stage 61.0 (TID 111, 10.28.23.203, PROCESS_LOCAL, 1165 bytes)
15/12/09 14:30:10 INFO BlockManagerInfo: Added broadcast_68_piece0 in memory on 10.28.23.201:51294 (size: 1530.0 B, free: 264.9 MB)
15/12/09 14:30:10 INFO BlockManagerInfo: Added broadcast_68_piece0 in memory on 10.28.23.203:57813 (size: 1530.0 B, free: 264.9 MB)
15/12/09 14:30:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 9 to 10.28.23.201:44784
15/12/09 14:30:10 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 9 is 163 bytes
15/12/09 14:30:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 9 to 10.28.23.203:56943
15/12/09 14:30:10 INFO TaskSetManager: Finished task 1.0 in stage 61.0 (TID 111) in 101 ms on 10.28.23.203 (1/2)
15/12/09 14:30:10 INFO TaskSetManager: Finished task 0.0 in stage 61.0 (TID 110) in 103 ms on 10.28.23.201 (2/2)
15/12/09 14:30:10 INFO TaskSchedulerImpl: Removed TaskSet 61.0, whose tasks have all completed, from pool
15/12/09 14:30:10 INFO DAGScheduler: ResultStage 61 (count at <console>:26) finished in 0.103 s
15/12/09 14:30:10 INFO DAGScheduler: Job 50 finished: count at <console>:26, took 0.267361 s
res74: Long = 13

sortByKey
scala> wordcount.sortByKey(true) take 20
15/12/09 14:31:41 INFO SparkContext: Starting job: sortByKey at <console>:26
15/12/09 14:31:41 INFO DAGScheduler: Got job 53 (sortByKey at <console>:26) with 2 output partitions (allowLocal=false)
15/12/09 14:31:41 INFO DAGScheduler: Final stage: ResultStage 64(sortByKey at <console>:26)
15/12/09 14:31:41 INFO DAGScheduler: Parents of final stage: List()
15/12/09 14:31:41 INFO DAGScheduler: Missing parents: List()
15/12/09 14:31:41 INFO DAGScheduler: Submitting ResultStage 64 (MapPartitionsRDD[56] at sortByKey at <console>:26), which has no missing parents
15/12/09 14:31:41 INFO MemoryStore: ensureFreeSpace(3984) called with curMem=1564303, maxMem=277877882
15/12/09 14:31:41 INFO MemoryStore: Block broadcast_71 stored as values in memory (estimated size 3.9 KB, free 263.5 MB)
15/12/09 14:31:41 INFO MemoryStore: ensureFreeSpace(2127) called with curMem=1568287, maxMem=277877882
15/12/09 14:31:41 INFO MemoryStore: Block broadcast_71_piece0 stored as bytes in memory (estimated size 2.1 KB, free 263.5 MB)
15/12/09 14:31:41 INFO BlockManagerInfo: Added broadcast_71_piece0 in memory on 10.28.23.201:60179 (size: 2.1 KB, free: 264.9 MB)
15/12/09 14:31:41 INFO SparkContext: Created broadcast 71 from broadcast at DAGScheduler.scala:874
15/12/09 14:31:41 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 64 (MapPartitionsRDD[56] at sortByKey at <console>:26)
15/12/09 14:31:41 INFO TaskSchedulerImpl: Adding task set 64.0 with 2 tasks
15/12/09 14:31:41 INFO TaskSetManager: Starting task 0.0 in stage 64.0 (TID 116, 10.28.23.202, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:31:41 INFO TaskSetManager: Starting task 1.0 in stage 64.0 (TID 117, 10.28.23.203, PROCESS_LOCAL, 1397 bytes)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_71_piece0 in memory on 10.28.23.202:50706 (size: 2.1 KB, free: 264.9 MB)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_71_piece0 in memory on 10.28.23.203:57813 (size: 2.1 KB, free: 264.9 MB)
15/12/09 14:31:42 INFO TaskSetManager: Finished task 0.0 in stage 64.0 (TID 116) in 86 ms on 10.28.23.202 (1/2)
15/12/09 14:31:42 INFO TaskSetManager: Finished task 1.0 in stage 64.0 (TID 117) in 83 ms on 10.28.23.203 (2/2)
15/12/09 14:31:42 INFO TaskSchedulerImpl: Removed TaskSet 64.0, whose tasks have all completed, from pool
15/12/09 14:31:42 INFO DAGScheduler: ResultStage 64 (sortByKey at <console>:26) finished in 0.087 s
15/12/09 14:31:42 INFO DAGScheduler: Job 53 finished: sortByKey at <console>:26, took 0.124303 s
15/12/09 14:31:42 INFO SparkContext: Starting job: take at <console>:26
15/12/09 14:31:42 INFO DAGScheduler: Registering RDD 42 (map at <console>:23)
15/12/09 14:31:42 INFO DAGScheduler: Got job 54 (take at <console>:26) with 1 output partitions (allowLocal=true)
15/12/09 14:31:42 INFO DAGScheduler: Final stage: ResultStage 66(take at <console>:26)
15/12/09 14:31:42 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 65)
15/12/09 14:31:42 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 65)
15/12/09 14:31:42 INFO DAGScheduler: Submitting ShuffleMapStage 65 (MapPartitionsRDD[42] at map at <console>:23), which has no missing parents
15/12/09 14:31:42 INFO MemoryStore: ensureFreeSpace(4288) called with curMem=1570414, maxMem=277877882
15/12/09 14:31:42 INFO MemoryStore: Block broadcast_72 stored as values in memory (estimated size 4.2 KB, free 263.5 MB)
15/12/09 14:31:42 INFO MemoryStore: ensureFreeSpace(2366) called with curMem=1574702, maxMem=277877882
15/12/09 14:31:42 INFO MemoryStore: Block broadcast_72_piece0 stored as bytes in memory (estimated size 2.3 KB, free 263.5 MB)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_72_piece0 in memory on 10.28.23.201:60179 (size: 2.3 KB, free: 264.8 MB)
15/12/09 14:31:42 INFO SparkContext: Created broadcast 72 from broadcast at DAGScheduler.scala:874
15/12/09 14:31:42 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 65 (MapPartitionsRDD[42] at map at <console>:23)
15/12/09 14:31:42 INFO TaskSchedulerImpl: Adding task set 65.0 with 2 tasks
15/12/09 14:31:42 INFO TaskSetManager: Starting task 0.0 in stage 65.0 (TID 118, 10.28.23.202, PROCESS_LOCAL, 1386 bytes)
15/12/09 14:31:42 INFO TaskSetManager: Starting task 1.0 in stage 65.0 (TID 119, 10.28.23.203, PROCESS_LOCAL, 1386 bytes)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_72_piece0 in memory on 10.28.23.203:57813 (size: 2.3 KB, free: 264.9 MB)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_72_piece0 in memory on 10.28.23.202:50706 (size: 2.3 KB, free: 264.9 MB)
15/12/09 14:31:42 INFO TaskSetManager: Finished task 1.0 in stage 65.0 (TID 119) in 61 ms on 10.28.23.203 (1/2)
15/12/09 14:31:42 INFO TaskSetManager: Finished task 0.0 in stage 65.0 (TID 118) in 67 ms on 10.28.23.202 (2/2)
15/12/09 14:31:42 INFO TaskSchedulerImpl: Removed TaskSet 65.0, whose tasks have all completed, from pool
15/12/09 14:31:42 INFO DAGScheduler: ShuffleMapStage 65 (map at <console>:23) finished in 0.066 s
15/12/09 14:31:42 INFO DAGScheduler: looking for newly runnable stages
15/12/09 14:31:42 INFO DAGScheduler: running: Set()
15/12/09 14:31:42 INFO DAGScheduler: waiting: Set(ResultStage 66)
15/12/09 14:31:42 INFO DAGScheduler: failed: Set()
15/12/09 14:31:42 INFO DAGScheduler: Missing parents for ResultStage 66: List()
15/12/09 14:31:42 INFO DAGScheduler: Submitting ResultStage 66 (ShuffledRDD[57] at sortByKey at <console>:26), which is now runnable
15/12/09 14:31:42 INFO MemoryStore: ensureFreeSpace(2528) called with curMem=1577068, maxMem=277877882
15/12/09 14:31:42 INFO MemoryStore: Block broadcast_73 stored as values in memory (estimated size 2.5 KB, free 263.5 MB)
15/12/09 14:31:42 INFO MemoryStore: ensureFreeSpace(1512) called with curMem=1579596, maxMem=277877882
15/12/09 14:31:42 INFO MemoryStore: Block broadcast_73_piece0 stored as bytes in memory (estimated size 1512.0 B, free 263.5 MB)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_73_piece0 in memory on 10.28.23.201:60179 (size: 1512.0 B, free: 264.8 MB)
15/12/09 14:31:42 INFO SparkContext: Created broadcast 73 from broadcast at DAGScheduler.scala:874
15/12/09 14:31:42 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 66 (ShuffledRDD[57] at sortByKey at <console>:26)
15/12/09 14:31:42 INFO TaskSchedulerImpl: Adding task set 66.0 with 1 tasks
15/12/09 14:31:42 INFO TaskSetManager: Starting task 0.0 in stage 66.0 (TID 120, 10.28.23.201, PROCESS_LOCAL, 1165 bytes)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_73_piece0 in memory on 10.28.23.201:51294 (size: 1512.0 B, free: 264.9 MB)
15/12/09 14:31:42 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to 10.28.23.201:44784
15/12/09 14:31:42 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 10 is 163 bytes
15/12/09 14:31:42 INFO TaskSetManager: Finished task 0.0 in stage 66.0 (TID 120) in 122 ms on 10.28.23.201 (1/1)
15/12/09 14:31:42 INFO TaskSchedulerImpl: Removed TaskSet 66.0, whose tasks have all completed, from pool
15/12/09 14:31:42 INFO DAGScheduler: ResultStage 66 (take at <console>:26) finished in 0.123 s
15/12/09 14:31:42 INFO DAGScheduler: Job 54 finished: take at <console>:26, took 0.218343 s
15/12/09 14:31:42 INFO SparkContext: Starting job: take at <console>:26
15/12/09 14:31:42 INFO DAGScheduler: Got job 55 (take at <console>:26) with 1 output partitions (allowLocal=true)
15/12/09 14:31:42 INFO DAGScheduler: Final stage: ResultStage 68(take at <console>:26)
15/12/09 14:31:42 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 67)
15/12/09 14:31:42 INFO DAGScheduler: Missing parents: List()
15/12/09 14:31:42 INFO DAGScheduler: Submitting ResultStage 68 (ShuffledRDD[57] at sortByKey at <console>:26), which has no missing parents
15/12/09 14:31:42 INFO MemoryStore: ensureFreeSpace(2528) called with curMem=1581108, maxMem=277877882
15/12/09 14:31:42 INFO MemoryStore: Block broadcast_74 stored as values in memory (estimated size 2.5 KB, free 263.5 MB)
15/12/09 14:31:42 INFO MemoryStore: ensureFreeSpace(1512) called with curMem=1583636, maxMem=277877882
15/12/09 14:31:42 INFO MemoryStore: Block broadcast_74_piece0 stored as bytes in memory (estimated size 1512.0 B, free 263.5 MB)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_74_piece0 in memory on 10.28.23.201:60179 (size: 1512.0 B, free: 264.8 MB)
15/12/09 14:31:42 INFO SparkContext: Created broadcast 74 from broadcast at DAGScheduler.scala:874
15/12/09 14:31:42 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 68 (ShuffledRDD[57] at sortByKey at <console>:26)
15/12/09 14:31:42 INFO TaskSchedulerImpl: Adding task set 68.0 with 1 tasks
15/12/09 14:31:42 INFO TaskSetManager: Starting task 0.0 in stage 68.0 (TID 121, 10.28.23.202, PROCESS_LOCAL, 1165 bytes)
15/12/09 14:31:42 INFO BlockManagerInfo: Added broadcast_74_piece0 in memory on 10.28.23.202:50706 (size: 1512.0 B, free: 264.9 MB)
15/12/09 14:31:42 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 10 to 10.28.23.202:53700
15/12/09 14:31:42 INFO TaskSetManager: Finished task 0.0 in stage 68.0 (TID 121) in 55 ms on 10.28.23.202 (1/1)
15/12/09 14:31:42 INFO TaskSchedulerImpl: Removed TaskSet 68.0, whose tasks have all completed, from pool
15/12/09 14:31:42 INFO DAGScheduler: ResultStage 68 (take at <console>:26) finished in 0.055 s
15/12/09 14:31:42 INFO DAGScheduler: Job 55 finished: take at <console>:26, took 0.070686 s
res77: Array[(String, Int)] = Array((best,1), (can,1), (demo,1), (hello,1), (hello,1), (in,1), (see,1), (spark,1), (sun,1), (test,1), (the,1), (the,1), (walking,1), (world,1), (you,1))

你可能感兴趣的:(spark,RDD,transforma)

nosql数据库技术与应用知识点皆过客，揽星河 NoSQL nosql 数据库大数据数据分析数据结构非关系型数据库
Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
分享一个基于python的电子书数据采集与可视化分析 hadoop电子书数据分析与推荐系统 spark大数据毕设项目（源码、调试、LW、开题、PPT) 计算机源码社 Python项目大数据大数据 python hadoop 计算机毕业设计选题计算机毕业设计源码数据分析 spark毕设
作者：计算机源码社个人简介：本人八年开发经验，擅长Java、Python、PHP、.NET、Node.js、Android、微信小程序、爬虫、大数据、机器学习等，大家有这一块的问题可以一起交流！学习资料、程序开发、技术解答、文档报告如需要源码，可以扫取文章下方二维码联系咨询Java项目微信小程序项目Android项目Python项目PHP项目ASP.NET项目Node.js项目选题推荐项目实战|p
Android jni中数组参数的传递方式 lokeyme Andriod android开发 JNI NDK java c语言
1、背景今天调试了一下Androidjni关于Java中调用C代码的程序，发现我的数组参数传递方式不对，导致值传递不正确，我的方法是：C代码，入口函数#include#includejintJava_sony_MedicalRecordDemo_MainActivity_decryptionSuccess(JNIEnv*env,jobjectthiz,jintAttr[]){returnAttr[
Spark 组件 GraphX、Streaming 叶域大数据 spark spark 大数据分布式
Spark组件GraphX、Streaming一、SparkGraphX1.1GraphX的主要概念1.2GraphX的核心操作1.3示例代码1.4GraphX的应用场景二、SparkStreaming2.1SparkStreaming的主要概念2.2示例代码2.3SparkStreaming的集成2.4SparkStreaming的应用场景SparkGraphX用于处理图和图并行计算。Graph
大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏租房推荐系统 58同城租房爬虫房源推荐系统房价预测系统计算机毕业设计机器学习深度学习人工智能 2401_84572577 程序员大数据 hadoop 人工智能
做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
Spark集群的三种模式 MelodyYN #Spark spark hadoop big data
文章目录1、Spark的由来1.1Hadoop的发展1.2MapReduce与Spark对比2、Spark内置模块3、Spark运行模式3.1Standalone模式部署配置历史服务器配置高可用运行模式3.2Yarn模式安装部署配置历史服务器运行模式4、WordCount案例1、Spark的由来定义：Hadoop主要解决，海量数据的存储和海量数据的分析计算。Spark是一种基于内存的快速、通用、可
通过进程Id终止进程好学松鼠 C++进程 C++Windows编程
#include#include//通过进程ID终止进程BOOLTerminateProcessFromID(DWORDdwID){BOOLbRet=FALSE;//打开进程HANDLEhProcess=::OpenProcess(PROCESS_ALL_ACCESS,FALSE,dwID);if(hProcess!=NULL){//终止进程bRet=::TerminateProcess(hPro
Java中的大数据处理框架对比分析省赚客app开发者 java 开发语言
Java中的大数据处理框架对比分析大家好，我是微赚淘客系统3.0的小编，是个冬天不穿秋裤，天冷也要风度的程序猿！今天，我们将深入探讨Java中常用的大数据处理框架，并对它们进行对比分析。大数据处理框架是现代数据驱动应用的核心，它们帮助企业处理和分析海量数据，以提取有价值的信息。本文将重点介绍ApacheHadoop、ApacheSpark、ApacheFlink和ApacheStorm这四种流行的
写出渗透测试信息收集详细流程卿酌南烛_b805
一、扫描域名漏洞：域名漏洞扫描工具有AWVS、APPSCAN、Netspark、WebInspect、Nmap、Nessus、天镜、明鉴、WVSS、RSAS等。二、子域名探测：1、dns域传送漏洞2、搜索引擎查找（通过Google、bing、搜索c段）3、通过ssl证书查询网站：https://myssl.com/ssl.html和https://www.chinassl.net/ssltools
Spark MLlib模型训练—推荐算法 ALS(Alternative Least Squares) 不二人生 Spark ML 实战 spark-ml 推荐算法算法
SparkMLlib模型训练—推荐算法ALS(AlternativeLeastSquares)如果你平时爱刷抖音，或者热衷看电影，不知道有没有过这样的体验：这类影视App你用得越久，它就好像会读心术一样，总能给你推荐对胃口的内容。其实这种迎合用户喜好的推荐，离不开机器学习中的推荐算法。在今天这一讲，我们就结合两个有趣的电影推荐场景，为你讲解SparkMLlib支持的协同过滤与频繁项集算法电影推荐场
app反编译工具及使用じ耐人网络安全
apktool——可以反编译软件的布局文件、图片等资源，方便大家学习一些很好的布局；dex2jar——将apk反编译成java源码（classes.dex转化成jar文件）；jd-gui——查看APK中classes.dex转化成出的jar文件，即源码文件。apktool使用方法解包：java-jarapktool_2.4.1.jarddemo.apk-odemo//其中d后面是要反编译的apk文
Python基础知识进阶之正则表达式_头歌python正则表达式进阶前端陈萨龙程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
分布式离线计算—Spark—基础介绍测试开发abbey 人工智能—大数据
原文作者：饥渴的小苹果原文地址：【Spark】Spark基础教程目录Spark特点Spark相对于Hadoop的优势Spark生态系统Spark基本概念Spark结构设计Spark各种概念之间的关系Executor的优点Spark运行基本流程Spark运行架构的特点Spark的部署模式Spark三种部署方式Hadoop和Spark的统一部署摘要：Spark是基于内存计算的大数据并行计算框架Spar
spark常用命令我是浣熊的微笑 spark
查看报错日志：yarnlogsapplicationIDspark2-submit--masteryarn--classcom.hik.ReadHdfstest-1.0-SNAPSHOT.jar进入$SPARK_HOME目录，输入bin/spark-submit--help可以得到该命令的使用帮助。hadoop@wyy:/app/hadoop/spark100$bin/spark-submit--
spark启动命令学不会又听不懂 spark 大数据分布式
hadoop启动：cd/root/toolssstart-dfs.sh，只需在hadoop01上启动stop-dfs.sh日志查看：cat/root/toolss/hadoop/logs/hadoop-root-datanode-hadoop03.outzookeeper启动：cd/root/toolss/zookeeperbin/zkServer.shstart，三台都要启动bin/zkServ
大数据领域的深度分析——AI是在帮助开发者还是取代他们？阳爱铭大数据与数据中台技术沉淀大数据人工智能后端数据库架构数据库开发 etl工程师 chatgpt
在大数据领域，生成式人工智能（AIGC）的应用正在迅速扩展，改变了数据科学家和开发者的工作方式。本文将从大数据的专业视角，探讨AI工具在这一领域的作用，以及它们是如何帮助开发者而非取代他们的。1.大数据领域的AI工具现状在大数据领域，AI工具已经取得了显著进展，以下是几款主要的AI工具及其功能和实际应用：ApacheSpark+MLlib：ApacheSpark是一个开源的分布式计算系统，广泛用于
大数据新视界 --大数据大厂之 Spark 性能优化秘籍：从配置到代码实践青云交大数据新视界 Spark 性能优化内存分配并行度存储级别 shuffle 减少算法优化代码实践数据读取广播变量数据倾斜 Spark 数据库
亲爱的朋友们，热烈欢迎你们来到青云交的博客！能与你们在此邂逅，我满心欢喜，深感无比荣幸。在这个瞬息万变的时代，我们每个人都在苦苦追寻一处能让心灵安然栖息的港湾。而我的博客，正是这样一个温暖美好的所在。在这里，你们不仅能够收获既富有趣味又极为实用的内容知识，还可以毫无拘束地畅所欲言，尽情分享自己独特的见解。我真诚地期待着你们的到来，愿我们能在这片小小的天地里共同成长，共同进步。本博客的精华专栏：Ja
编程常用命令总结 Yellow0523 Linux BigData 大数据
编程命令大全1.软件环境变量的配置JavaScalaSparkHadoopHive2.大数据软件常用命令Spark基本命令Spark-SQL命令Hive命令HDFS命令YARN命令Zookeeper命令kafka命令Hibench命令MySQL命令3.Linux常用命令Git命令conda命令pip命令查看Linux系统的详细信息查看Linux系统架构(X86还是ARM，两种方法都可)端口号命令L
【面试系列】Spark 高频面试题解答野老杂谈全网最全IT公司面试宝典面试 spark 职场和发展大数据
欢迎来到我的博客，很高兴能够在这里和您见面！欢迎订阅相关专栏：⭐️全网最全IT互联网公司面试宝典：收集整理全网各大IT互联网公司技术、项目、HR面试真题.⭐️AIGC时代的创新与未来：详细讲解AIGC的概念、核心技术、应用领域等内容。⭐️大数据平台建设指南：全面讲解从数据采集到数据可视化的整个过程，掌握构建现代化数据平台的核心技术和方法。⭐️《遇见Python：初识、了解与热恋》：涵盖了Pytho
spark常见面试题爱敲代码的小黑 spark 大数据分布式
文章目录1.Spark的运行流程？2.Spark中的RDD机制理解吗？3.RDD的宽窄依赖4.DAG中为什么要划分Stage？5.Spark程序执行，有时候默认为什么会产生很多task，怎么修改默认task执行个数？6.RDD中reduceBykey与groupByKey哪个性能好，为什么？7.SparkMasterHA主从切换过程不会影响到集群已有作业的运行，为什么？8.SparkMaster使
Spark面试题 golove666 面试题大全 spark 大数据分布式面试
Spark面试题1.Spark基础概念1.1解释Spark是什么以及它的主要特点Spark是什么？Spark的主要特点1.2描述Spark运行时架构和组件主要的Spark架构组件：1.3讲述Spark中的弹性分布式数据集（RDD）和数据帧（DataFrame）弹性分布式数据集（RDD）主要特征：创建和转换：使用场景：数据帧（DataFrame）主要特征：创建和操作：使用场景：RDD与DataFra
图计算：基于SparkGrpahX计算聚类系数妙龄少女郭德纲 Spark 图算法 Scala 聚类数据挖掘机器学习
图计算：基于SparkGrpahX计算聚类系数文章目录图计算：基于SparkGrpahX计算聚类系数一、什么是聚类系数二、基于SparkGraphX的聚类系数代码实现总结一、什么是聚类系数聚类系数（ClusteringCoefficient）是图计算和网络分析中的一个重要概念，用于衡量网络中节点的局部聚集程度。它有助于理解网络中节点之间的紧密程度和网络的结构特性。这是一种用来衡量图中节点聚类程度的
2024年最全使用Python求解方程_python解方程(1)，字节面试官迟到 2401_84569545 程序员 python 学习面试
最后硬核资料：关注即可领取PPT模板、简历模板、行业经典书籍PDF。技术互助：技术群大佬指点迷津，你的问题可能不是问题，求资源在群里喊一声。面试题库：由技术群里的小伙伴们共同投稿，热乎的大厂面试真题，持续更新中。知识体系：含编程语言、算法、大数据生态圈组件（Mysql、Hive、Spark、Flink）、数据仓库、Python、前端等等。网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是
Spark运行时架构 tooolik spark 架构大数据
目录一，Spark运行时架构二，YARN集群架构（一）YARN集群主要组件1、ResourceManager-资源管理器2、NodeManager-节点管理器3、Task-任务4、Container-容器5、ApplicationMaster-应用程序管理器6，总结（二）YARN集群中应用程序的执行流程三、SparkStandalone架构（一）client提交方式（二）cluster提交方式四、
使用SparkSql进行表的分析与统计 xingyuan8 大数据 java
背景我们的数据挖掘平台对数据统计有比较迫切的需求，而Spark本身对数据统计已经做了一些工作，希望梳理一下Spark已经支持的数据统计功能，后期再进行扩展。准备数据在参考文献6中下载鸢尾花数据，此处格式为iris.data格式，先将data后缀改为csv后缀（不影响使用，只是为了保证后续操作不需要修改）。数据格式如下：SepalLengthSepalWidthPetalLengthPetalWid
13.Spark Core-Spark中广播变量和累加器 __元昊__
一、前述Spark中因为算子中的真正逻辑是发送到Executor中去运行的，所以当Executor中需要引用外部变量时，需要使用广播变量。累机器相当于统筹大变量，常用于计数，统计。二、具体原理1、广播变量广播变量理解图image注意事项1、能不能将一个RDD使用广播变量广播出去？不能，因为RDD是不存储数据的。可以将RDD的结果广播出去。2、广播变量只能在Driver端定义，不能在Executor
比较Spark与Flink 傲雪凌霜，松柏长青大数据后端 spark flink 大数据
ApacheSpark和ApacheFlink都是目前非常流行的大数据处理引擎，但它们在架构、处理模式、应用场景等方面有一些显著的区别。下面是二者的对比：1.处理模式Spark:主要支持批处理（BatchProcessing），也能通过SparkStreaming处理流式数据，但SparkStreaming本质上是通过微批（micro-batching）的方式处理流数据，延迟相对较高。SparkS
Spark底层逻辑傲雪凌霜，松柏长青大数据后端 spark 大数据
ApacheSpark的底层逻辑可以从其核心概念、组件和执行流程等方面来理解。Spark提供了一个分布式数据处理框架，其底层逻辑基于批处理架构，能够在大规模集群中高效地处理数据。以下是Spark的底层逻辑的详细介绍：1.核心概念Spark的底层基于几个核心概念来实现分布式计算，包括：RDD（ResilientDistributedDataset，弹性分布式数据集）：RDD是Spark最基础的数据抽
bfs 求解迷宫最短路径问题蒟蒻彧彧搜索
问题描述下图给出了一个迷宫的平面图，其中标记为1的为障碍，标记为0的为可以通行的地方。010000000100001001110000迷宫的入口为左上角，出口为右下角，在迷宫中，只能从一个位置走到这个它的上、下、左、右四个方向之一。对于上面的迷宫，从入口开始，可以按DRRURRDDDR的顺序通过迷宫，一共10步。其中D、U、L、R分别表示向下、向上、向左、向右走。对于下面这个更复杂的迷宫（30行5
Leetcode刷题(第139题)——单词拆分卖菜的小白面试算法 leetcode javascript 算法递归
一、题目给你一个字符串s和一个字符串列表wordDict作为字典。请你判断是否可以利用字典中出现的单词拼接出s。注意：不要求字典中出现的单词全部都使用，并且字典中的单词可以重复使用。二、示例输入:s="leetcode",wordDict=["leet","code"]输出:true解释:返回true因为"leetcode"可以由"leet"和"code"拼接成。输入:s="applepenapp
Spring的注解积累 yijiesuifeng spring 注解
用注解来向Spring容器注册Bean。需要在applicationContext.xml中注册： <context:component-scan base-package=”pagkage1[,pagkage2,…,pagkageN]”/>。如：在base-package指明一个包 <context:component-sc
传感器百合不是茶 android 传感器
android传感器的作用主要就是来获取数据,根据得到的数据来触发某种事件下面就以重力传感器为例; 1,在onCreate中获得传感器服务 private SensorManager sm;// 获得系统的服务 private Sensor sensor;// 创建传感器实例 @Override protected void
[光磁与探测]金吕玉衣的意义 comsci
这是一个古代人的秘密:现在告诉大家信不信由你们: 穿上金律玉衣的人,如果处于灵魂出窍的状态,可以飞到宇宙中去看星星这就是为什么古代
精简的反序打印某个数沐刃青蛟打印
以前看到一些让求反序打印某个数的程序。比如：输入123，输出321。记得以前是告诉你是几位数的，当时就抓耳挠腮，完全没有思路。似乎最后是用到%和/方法解决的。而今突然想到一个简短的方法，就可以实现任意位数的反序打印（但是如果是首位数或者尾位数为0时就没有打印出来了）代码如下： long num, num1=0;
PHP：6种方法获取文件的扩展名 IT独行者 PHP 扩展名
PHP：6种方法获取文件的扩展名 1、字符串查找和截取的方法 1 $extension = substr ( strrchr ( $file , '.' ), 1); 2、字符串查找和截取的方法二 1 $extension = substr
面试111 文强chu 面试
1事务隔离级别有那些，事务特性是什么（问到一次） 2 spring aop 如何管理事务的，如何实现的。动态代理如何实现，jdk怎么实现动态代理的，ioc是怎么实现的，spring是单例还是多例，有那些初始化bean的方式，各有什么区别（经常问） 3 struts默认提供了那些拦截器（一次） 4 过滤器和拦截器的区别（频率也挺高） 5 final，finally final
XML的四种解析方式小桔子 dom jdom dom4j sax
在平时工作中，难免会遇到把 XML 作为数据存储格式。面对目前种类繁多的解决方案，哪个最适合我们呢？在这篇文章中，我对这四种主流方案做一个不完全评测，仅仅针对遍历 XML 这块来测试，因为遍历 XML 是工作中使用最多的（至少我认为）。　　预备　　测试环境：　　AMD 毒龙1.4G OC 1.5G、256M DDR333、Windows2000 Server
wordpress中常见的操作 aichenglong 中文注册 wordpress 移除菜单
1 wordpress中使用中文名注册解决办法 1)使用插件 2)修改wp源代码进入到wp-include/formatting.php文件中找到 function sanitize_user( $username, $strict = false
小飞飞学管理-1 alafqq 管理
项目管理的下午题，其实就在提出问题（挑刺），分析问题，解决问题。今天我随意看下10年上半年的第一题。主要就是项目经理的提拨和培养。结合我自己经历写下心得对于公司选拔和培养项目经理的制度有什么毛病呢？ 1，公司考察，选拔项目经理，只关注技术能力，而很少或没有关注管理方面的经验，能力。 2，公司对项目经理缺乏必要的项目管理知识和技能方面的培训。 3，公司对项目经理的工作缺乏进行指
IO输入输出部分探讨百合不是茶 IO
//文件处理在处理文件输入输出时要引入java.IO这个包； /* 1，运用File类对文件目录和属性进行操作 2，理解流，理解输入输出流的概念 3，使用字节/符流对文件进行读/写操作 4，了解标准的I/O 5，了解对象序列化 */ //1，运用File类对文件目录和属性进行操作 //在工程中线创建一个text.txt
getElementById的用法 bijian1013 element
getElementById是通过Id来设置/返回HTML标签的属性及调用其事件与方法。用这个方法基本上可以控制页面所有标签，条件很简单，就是给每个标签分配一个ID号。返回具有指定ID属性值的第一个对象的一个引用。语法： &n
励志经典语录 bijian1013 励志人生
经典语录1: 哈佛有一个著名的理论：人的差别在于业余时间，而一个人的命运决定于晚上8点到10点之间。每晚抽出2个小时的时间用来阅读、进修、思考或参加有意的演讲、讨论，你会发现，你的人生正在发生改变，坚持数年之后，成功会向你招手。不要每天抱着QQ/MSN/游戏/电影/肥皂剧……奋斗到12点都舍不得休息，看就看一些励志的影视或者文章，不要当作消遣；学会思考人生，学会感悟人生
[MongoDB学习笔记三]MongoDB分片 bit1129 mongodb
MongoDB的副本集(Replica Set)一方面解决了数据的备份和数据的可靠性问题，另一方面也提升了数据的读写性能。MongoDB分片(Sharding)则解决了数据的扩容问题，MongoDB作为云计算时代的分布式数据库，大容量数据存储，高效并发的数据存取，自动容错等是MongoDB的关键指标。本篇介绍MongoDB的切片(Sharding) 1.何时需要分片 &nbs
【Spark八十三】BlockManager在Spark中的使用场景 bit1129 manager
1. Broadcast变量的存储，在HttpBroadcast类中可以知道 2. RDD通过CacheManager存储RDD中的数据，CacheManager也是通过BlockManager进行存储的 3. ShuffleMapTask得到的结果数据，是通过FileShuffleBlockManager进行管理的，而FileShuffleBlockManager最终也是使用BlockMan
yum方式部署zabbix ronin47 yum方式部署zabbix
安装网络yum库#rpm -ivh http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm 通过yum装mysql和zabbix调用的插件还有agent代理#yum install zabbix-server-mysql zabbix-web-mysql mysql-
Hibernate4和MySQL5.5自动创建表失败问题解决方法 byalias J2EE Hibernate4
今天初学Hibernate4，了解了使用Hibernate的过程。大体分为4个步骤： ①创建hibernate.cfg.xml文件 ②创建持久化对象 ③创建*.hbm.xml映射文件 ④编写hibernate相应代码在第四步中，进行了单元测试，测试预期结果是hibernate自动帮助在数据库中创建数据表，结果JUnit单元测试没有问题，在控制台打印了创建数据表的SQL语句，但在数据库中
Netty源码学习-FrameDecoder bylijinnan java netty
Netty 3.x的user guide里FrameDecoder的例子，有几个疑问： 1.文档说：FrameDecoder calls decode method with an internally maintained cumulative buffer whenever new data is received. 为什么每次有新数据到达时，都会调用decode方法？ 2.Dec
SQL行列转换方法 chicony 行列转换
create table tb(终端名称 varchar(10) , CEI分值 varchar(10) , 终端数量 int) insert into tb values('三星' , '0-5' , 74) insert into tb values('三星' , '10-15' , 83) insert into tb values('苹果' , '0-5' , 93)
中文编码测试 ctrain 编码
循环打印转换编码 String[] codes = { "iso-8859-1", "utf-8", "gbk", "unicode" }; for (int i = 0; i < codes.length; i++) { for (int j
hive 客户端查询报堆内存溢出解决方法 daizj hive 堆内存溢出
hive> select * from t_test where ds=20150323 limit 2; OK Exception in thread "main" java.lang.OutOfMemoryError: Java heap space 问题原因： hive堆内存默认为256M 这个问题的解决方法为：修改/us
人有多大懒，才有多大闲 (评论『卓有成效的程序员』) dcj3sjt126com 程序员
卓有成效的程序员给我的震撼很大，程序员作为特殊的群体，有的人可以这么懒，懒到事情都交给机器去做，而有的人又可以那么勤奋，每天都孜孜不倦得做着重复单调的工作。在看这本书之前，我属于勤奋的人，而看完这本书以后，我要努力变成懒惰的人。不要在去庞大的开始菜单里面一项一项搜索自己的应用程序，也不要在自己的桌面上放置眼花缭乱的快捷图标
Eclipse简单有用的配置 dcj3sjt126com eclipse
1、显示行号 Window -- Prefences -- General -- Editors -- Text Editors -- show line numbers 2、代码提示字符 Window ->Perferences，并依次展开 Java -> Editor -> Content Assist，最下面一栏 auto-Activation
在tomcat上面安装solr4.8.0全过程 eksliang Solr solr4.0后的版本安装 solr4.8.0安装
转载请出自出处： http://eksliang.iteye.com/blog/2096478 首先solr是一个基于java的web的应用，所以安装solr之前必须先安装JDK和tomcat，我这里就先省略安装tomcat和jdk了第一步：当然是下载去官网上下载最新的solr版本，下载地址
Android APP通用型拒绝服务、漏洞分析报告 gg163 漏洞 android APP 分析
点评：记得曾经有段时间很多SRC平台被刷了大量APP本地拒绝服务漏洞，移动安全团队爱内测（ineice.com）发现了一个安卓客户端的通用型拒绝服务漏洞，来看看他们的详细分析吧。 0xr0ot和Xbalien交流所有可能导致应用拒绝服务的异常类型时，发现了一处通用的本地拒绝服务漏洞。该通用型本地拒绝服务可以造成大面积的app拒绝服务。针对序列化对象而出现的拒绝服务主要
HoverTree项目已经实现分层 hvt 编程 .net Web C#ASP.ENT
HoverTree项目已经初步实现分层，源代码已经上传到 http://hovertree.codeplex.com请到SOURCE CODE查看。在本地用SQL Server 2008 数据库测试成功。数据库和表请参考：http://keleyi.com/a/bjae/ue6stb42.htmHoverTree是一个ASP.NET 开源项目，希望对你学习ASP.NET或者C#语言有帮助，如果你对
Google Maps API v3: Remove Markers 移除标记天梯梦 google maps api
Simply do the following: I. Declare a global variable: var markersArray = []; II. Define a function: function clearOverlays() { for (var i = 0; i < markersArray.length; i++ )
jQuery选择器总结 lq38366 jquery 选择器
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
基础数据结构和算法六：Quick sort sunwinner Algorithm Quicksort
Quick sort is probably used more widely than any other. It is popular because it is not difficult to implement, works well for a variety of different kinds of input data, and is substantially faster t
如何让Flash不遮挡HTML div元素的技巧_HTML/Xhtml_网页制作刘星宇 html Web
今天在写一个flash广告代码的时候，因为flash自带的链接，容易被当成弹出广告，所以做了一个div层放到flash上面，这样链接都是a触发的不会被拦截，但发现flash一直处于div层上面，原来flash需要加个参数才可以。让flash置于DIV层之下的方法，让flash不挡住飘浮层或下拉菜单，让Flash不档住浮动对象或层的关键参数：wmode=opaque。方法如下：
Mybatis实用Mapper SQL汇总示例 wdmcygah sql mysql mybatis 实用
Mybatis作为一个非常好用的持久层框架，相关资料真的是少得可怜，所幸的是官方文档还算详细。本博文主要列举一些个人感觉比较常用的场景及相应的Mapper SQL写法，希望能够对大家有所帮助。不少持久层框架对动态SQL的支持不足，在SQL需要动态拼接时非常苦恼，而Mybatis很好地解决了这个问题，算是框架的一大亮点。对于常见的场景，例如：批量插入/更新/删除，模糊查询，多条件查询，联表查询，