myboyliu2007
myboyliu2007

spark 官方动手练习二：用spark浏览维基百科的数据

在本章中，我们将首先使用Spark shell以交互方式浏览维基百科的数据。然后，我们会给出一个简要介绍了编写spark的程序。请记住，Spark是建立在Hadoop分布式文件系统（HDFS）之上的开源的计算引擎。

交互数据分析

现在，让我们用spark办对数据集中的进行排序和统计。首先，启动spark shell：
[root@hadoop spark-0.8.0]# spark-shell
Welcome to
      ____              __  
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 0.8.0
      /_/                  


Using Scala version 2.9.3 (Java HotSpot(TM) Client VM, Java 1.6.0_24)
Initializing interpreter...
13/12/30 18:21:53 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/30 18:21:53 INFO server.AbstractConnector: Started [email protected]:40248
Creating SparkContext...
13/12/30 18:22:10 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/30 18:22:10 INFO spark.SparkEnv: Registering BlockManagerMaster
13/12/30 18:22:10 INFO storage.MemoryStore: MemoryStore started with capacity 326.7 MB.
13/12/30 18:22:10 INFO storage.DiskStore: Created local directory at /tmp/spark-local-20131230182210-e790
13/12/30 18:22:10 INFO network.ConnectionManager: Bound socket to port 36326 with id = ConnectionManagerId(hadoop,36326)
13/12/30 18:22:10 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/12/30 18:22:10 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager hadoop:36326 with 326.7 MB RAM
13/12/30 18:22:10 INFO storage.BlockManagerMaster: Registered BlockManager
13/12/30 18:22:10 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/30 18:22:10 INFO server.AbstractConnector: Started [email protected]:36168
13/12/30 18:22:10 INFO broadcast.HttpBroadcast: Broadcast server started at http://192.168.223.135:36168
13/12/30 18:22:10 INFO spark.SparkEnv: Registering MapOutputTracker
13/12/30 18:22:11 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-41e1f8f5-140e-4a28-abec-94031bb75be7
13/12/30 18:22:11 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/30 18:22:11 INFO server.AbstractConnector: Started [email protected]:47315
13/12/30 18:22:11 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/storage/rdd,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/storage,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages/stage,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages/pool,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/stages,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/environment,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/executors,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/json,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
13/12/30 18:22:11 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/,null}
13/12/30 18:22:11 INFO server.AbstractConnector: Started [email protected]:4040
13/12/30 18:22:11 INFO ui.SparkUI: Started Spark Web UI at http://hadoop:4040
Spark context available as sc.
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

1.通过创建一个RDD（弹性分布式数据集）名为输入文件pagecounts。在Spark shell，SparkContext已经为您创建为变量SC。

scala> sc
res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@15978e7


scala> val pagecounts = sc.textFile("/wiki/pagecounts")
13/12/30 18:32:07 WARN util.SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
13/12/30 18:32:08 INFO storage.MemoryStore: ensureFreeSpace(32207) called with curMem=0, maxMem=342526525
13/12/30 18:32:08 INFO storage.MemoryStore: Block broadcast_0 stored as values to memory (estimated size 31.5 KB, free 326.6 MB)
pagecounts: org.apache.spark.rdd.RDD[String] = MappedRDD[1] at textFile at <console>:12

2.让我们看一下数据。您可以使用RDD（弹性分布式数据集）的take()来获取前K个记录。这里，K= 10

scala> pagecounts.take(10)
13/12/30 18:35:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/12/30 18:35:08 WARN snappy.LoadSnappy: Snappy native library not loaded
13/12/30 18:35:08 INFO mapred.FileInputFormat: Total input paths to process : 1
13/12/30 18:35:08 INFO spark.SparkContext: Starting job: take at <console>:15
13/12/30 18:35:08 INFO scheduler.DAGScheduler: Got job 0 (take at <console>:15) with 1 output partitions (allowLocal=true)
13/12/30 18:35:08 INFO scheduler.DAGScheduler: Final stage: Stage 0 (take at <console>:15)
13/12/30 18:35:08 INFO scheduler.DAGScheduler: Parents of final stage: List()
13/12/30 18:35:08 INFO scheduler.DAGScheduler: Missing parents: List()
13/12/30 18:35:08 INFO scheduler.DAGScheduler: Computing the requested partition locally
13/12/30 18:35:08 INFO rdd.HadoopRDD: Input split: file:/wiki/pagecounts/wiki-cout.txt:0+3460512
13/12/30 18:35:08 INFO spark.SparkContext: Job finished: take at <console>:15, took 0.085379768 s
res1: Array[String] = Array(20090507-040000 aa ?page=http://www.stockphotosharing.com/Themes/Images/users_raw/id.txt 3 39267, 20090507-040000 aa Main_Page 7 51309, 20090507-040000 aa Special:Boardvote 1 11631, 20090507-040000 aa Special:Imagelist 1 931, 20090507-040000 aa ?page=http://www.stockphotosharing.com/Themes/Images/users_raw/id.txt 3 39267, 20090507-040000 aa Main_Page 7 51309, 20090507-040000 aa Special:Boardvote 1 11631, 20090507-040000 aa Special:Imagelist 1 931, 20090507-040000 aa ?page=http://www.stockphotosharing.com/Themes/Images/users_raw/id.txt 3 39267, 20090507-040000 aa Main_Page 7 51309)

不幸的是，输出的信息是不可读，因为take()返回一个数组和Scala简单地打印该数组由逗号分隔的每个元素。我们可以让它通过遍历数组来更漂亮打印每一行的每个记录。

scala> pagecounts.take(10).foreach(println)
13/12/30 18:40:20 INFO spark.SparkContext: Starting job: take at <console>:15
13/12/30 18:40:20 INFO scheduler.DAGScheduler: Got job 1 (take at <console>:15) with 1 output partitions (allowLocal=true)
13/12/30 18:40:20 INFO scheduler.DAGScheduler: Final stage: Stage 1 (take at <console>:15)
13/12/30 18:40:20 INFO scheduler.DAGScheduler: Parents of final stage: List()
13/12/30 18:40:20 INFO scheduler.DAGScheduler: Missing parents: List()
13/12/30 18:40:20 INFO scheduler.DAGScheduler: Computing the requested partition locally
13/12/30 18:40:20 INFO rdd.HadoopRDD: Input split: file:/wiki/pagecounts/wiki-cout.txt:0+3460512
13/12/30 18:40:20 INFO spark.SparkContext: Job finished: take at <console>:15, took 0.007811997 s
20090507-040000 aa ?page=http://www.stockphotosharing.com/Themes/Images/users_raw/id.txt 3 39267
20090507-040000 aa Main_Page 7 51309
20090507-040000 aa Special:Boardvote 1 11631
20090507-040000 aa Special:Imagelist 1 931
20090507-040000 aa ?page=http://www.stockphotosharing.com/Themes/Images/users_raw/id.txt 3 39267
20090507-040000 aa Main_Page 7 51309
20090507-040000 aa Special:Boardvote 1 11631
20090507-040000 aa Special:Imagelist 1 931
20090507-040000 aa ?page=http://www.stockphotosharing.com/Themes/Images/users_raw/id.txt 3 39267
20090507-040000 aa Main_Page 7 51309


scala> 

3.让我们看看在这个数据集有多少条记录（由于运行时会预读数据集，所以该命令将需要一段时间）。

scala> pagecounts.count
13/12/30 18:44:34 INFO spark.SparkContext: Starting job: count at <console>:15
13/12/30 18:44:34 INFO scheduler.DAGScheduler: Got job 2 (count at <console>:15) with 1 output partitions (allowLocal=false)
13/12/30 18:44:34 INFO scheduler.DAGScheduler: Final stage: Stage 2 (count at <console>:15)
13/12/30 18:44:34 INFO scheduler.DAGScheduler: Parents of final stage: List()
13/12/30 18:44:34 INFO scheduler.DAGScheduler: Missing parents: List()
13/12/30 18:44:34 INFO scheduler.DAGScheduler: Submitting Stage 2 (MappedRDD[1] at textFile at <console>:12), which has no missing parents
13/12/30 18:44:34 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 2 (MappedRDD[1] at textFile at <console>:12)
13/12/30 18:44:35 INFO local.LocalTaskSetManager: Size of task 0 is 1570 bytes
13/12/30 18:44:35 INFO local.LocalScheduler: Running 0
13/12/30 18:44:35 INFO rdd.HadoopRDD: Input split: file:/wiki/pagecounts/wiki-cout.txt:0+3460512
13/12/30 18:44:35 INFO local.LocalScheduler: Finished 0
13/12/30 18:44:35 INFO scheduler.DAGScheduler: Completed ResultTask(2, 0)
13/12/30 18:44:35 INFO scheduler.DAGScheduler: Stage 2 (count at <console>:15) finished in 0.385 s
13/12/30 18:44:35 INFO spark.SparkContext: Job finished: count at <console>:15, took 0.644667004 s
res3: Long = 61248


scala> 13/12/30 18:44:35 INFO local.LocalScheduler: Remove TaskSet 2.0 from pool 

scala> 
由于我这里使用的数据集很小，执行时间很短。
你可以打开星火Web控制台来查看进度。要做到这一点，打开你喜欢的浏览器，并输入以下URL。


包含http://hadoop：4040


需要注意的是，如果你有一个激活的job或spark shell,此页才可用。
你应该会看到spark的应用状况的web界面，类似于以下内容：

在此界面中的链接允许您跟踪作业的进度和有关其执行情况，包括任务持续时间和缓存统计各种指标。


此外，spark单机集群状态的Web界面显示与整个星火集群信息。要查看此用户界面，浏览到


http://hadoop:8080
由于我的spark是单节点的，所以只能看到Master的状况。

您应该看到类似于以下内容的页面：





4.从上面记得，当我们描述的数据集的格式，第二个字段是“project code”，并包含有关网页的语言信息。例如，project code为“en”表示英文页面。让我们来算出含pagecounts里只有英文网页个数。这可以通过应用一个过滤器功能，以pagecounts来完成。对于每个记录，我们可以通过字段分隔符（即一个空格）分开，并得到了第二区域， 然后用字符串“en”和他做比较。


为了避免在执行我们的RDD任何操作都是从从每个磁盘的读取，我们也将缓存RDD到内存中。这是spark真正开始大放异彩的地方。

scala> val enPages = pagecounts.filter(_.split(" ")(1) == "en").cache
enPages: org.apache.spark.rdd.RDD[String] = FilteredRDD[2] at filter at <console>:14

当您键入此命令，进入spark shell，spark定义RDD，而是因为lazy评估，所以将不计算做。任何下一次调用enPages，spark将读取集群中缓存在内存中的数据集。

5.有多少记录在英文网页？

scala> enPages.count
13/12/30 19:12:54 INFO spark.SparkContext: Starting job: count at <console>:17
13/12/30 19:12:54 INFO scheduler.DAGScheduler: Got job 3 (count at <console>:17) with 1 output partitions (allowLocal=false)
13/12/30 19:12:54 INFO scheduler.DAGScheduler: Final stage: Stage 3 (count at <console>:17)
13/12/30 19:12:54 INFO scheduler.DAGScheduler: Parents of final stage: List()
13/12/30 19:12:54 INFO scheduler.DAGScheduler: Missing parents: List()
13/12/30 19:12:54 INFO scheduler.DAGScheduler: Submitting Stage 3 (FilteredRDD[2] at filter at <console>:14), which has no missing parents
13/12/30 19:12:54 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 3 (FilteredRDD[2] at filter at <console>:14)
13/12/30 19:12:54 INFO local.LocalTaskSetManager: Size of task 1 is 1655 bytes
13/12/30 19:12:54 INFO local.LocalScheduler: Running 1
13/12/30 19:12:54 INFO spark.CacheManager: Cache key is rdd_2_0
13/12/30 19:12:54 INFO spark.CacheManager: Computing partition org.apache.spark.rdd.HadoopPartition@691
13/12/30 19:12:54 INFO rdd.HadoopRDD: Input split: file:/wiki/pagecounts/wiki-cout.txt:0+3460512
13/12/30 19:12:54 INFO storage.MemoryStore: ensureFreeSpace(104) called with curMem=32207, maxMem=342526525
13/12/30 19:12:54 INFO storage.MemoryStore: Block rdd_2_0 stored as values to memory (estimated size 104.0 B, free 326.6 MB)
13/12/30 19:12:54 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on hadoop:36326 (size: 104.0 B, free: 326.7 MB)
13/12/30 19:12:54 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/12/30 19:12:54 INFO local.LocalScheduler: Finished 1
13/12/30 19:12:55 INFO local.LocalScheduler: Remove TaskSet 3.0 from pool 
13/12/30 19:12:55 INFO scheduler.DAGScheduler: Completed ResultTask(3, 0)
13/12/30 19:12:55 INFO scheduler.DAGScheduler: Stage 3 (count at <console>:17) finished in 0.985 s
13/12/30 19:12:55 INFO spark.SparkContext: Job finished: count at <console>:17, took 1.003093064 s
res4: Long = 0


scala> 

第一次运行此命令，类似于我们做的最后一次计算，这将需要2 - 3分钟，而在整个数据spark扫描磁盘上的设置。但是，由于enPages被标记为在上一步“缓存”，如果你再次运行在相同的RDD计算，它应该返回一个数量级速度提高。
13/12/30 19:14:27 INFO spark.SparkContext: Job finished: count at <console>:17, took 0.014903482 s

如果您检查控制台日志观察，你会看到这样的线索，它显示一些数据被添加到缓存中：
13/12/30 19:12:54 INFO storage.MemoryStore: Block rdd_2_0 stored as values to memory (estimated size 104.0 B, free 326.6 MB)

6.让我们尝试一些更华丽的操作。来计算在维基百科里的英文页面中的数据里，所表示的日期范围内的总页面访问量（5月5日至5月7日，2009）的直方图。我们会产生每一行的键值对如下。首先，我们生成一个键值对的每一行，关键是日期（第一域的前八个字符），该值是浏览量该日期数量（第四域）

scala> val enTuples = enPages.map(line => line.split(" "))
enTuples: org.apache.spark.rdd.RDD[Array[java.lang.String]] = MappedRDD[3] at map at <console>:16


scala> val enKeyValuePairs = enTuples.map(line => (line(0).substring(0, 8), line(3).toInt))
enKeyValuePairs: org.apache.spark.rdd.RDD[(java.lang.String, Int)] = MappedRDD[4] at map at <console>:18


scala> 
接下来，我们来shuffle数据集，然后将所有相同的key的数据和分组中并且相加。最后，我们将每个键的数据进行合计。目前正是这种模式下，在spark中有一个称为reduceByKey一个方法。请注意，第二个参数reduceByKey决定reduce使用的数量。默认情况下，星火假定reduce函数是交换律和结合，在所用的mapper阶段会应用combiners。既然我们知道有一个非常有限的在这种情况下，key的数量（因为在我们的数据集里只有3个的特定的日期），让我们只用一个reduce。

scala> enKeyValuePairs.reduceByKey(_+_, 1).collect
13/12/30 19:26:09 INFO spark.SparkContext: Starting job: collect at <console>:21
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Registering RDD 5 (reduceByKey at <console>:21)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Got job 5 (collect at <console>:21) with 1 output partitions (allowLocal=false)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Final stage: Stage 5 (collect at <console>:21)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 6)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Missing parents: List(Stage 6)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Submitting Stage 6 (MapPartitionsRDD[5] at reduceByKey at <console>:21), which has no missing parents
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 6 (MapPartitionsRDD[5] at reduceByKey at <console>:21)
13/12/30 19:26:09 INFO local.LocalTaskSetManager: Size of task 3 is 2013 bytes
13/12/30 19:26:09 INFO local.LocalScheduler: Running 3
13/12/30 19:26:09 INFO spark.CacheManager: Cache key is rdd_2_0
13/12/30 19:26:09 INFO spark.CacheManager: Found partition in cache!
13/12/30 19:26:09 INFO local.LocalScheduler: Finished 3
13/12/30 19:26:09 INFO local.LocalScheduler: Remove TaskSet 6.0 from pool 
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Completed ShuffleMapTask(6, 0)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Stage 6 (reduceByKey at <console>:21) finished in 0.039 s
13/12/30 19:26:09 INFO scheduler.DAGScheduler: looking for newly runnable stages
13/12/30 19:26:09 INFO scheduler.DAGScheduler: running: Set()
13/12/30 19:26:09 INFO scheduler.DAGScheduler: waiting: Set(Stage 5)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: failed: Set()
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Missing parents for Stage 5: List()
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Submitting Stage 5 (MapPartitionsRDD[7] at reduceByKey at <console>:21), which is now runnable
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 5 (MapPartitionsRDD[7] at reduceByKey at <console>:21)
13/12/30 19:26:09 INFO local.LocalTaskSetManager: Size of task 4 is 1918 bytes
13/12/30 19:26:09 INFO local.LocalScheduler: Running 4
13/12/30 19:26:09 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 19:26:09 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  12 ms
13/12/30 19:26:09 INFO local.LocalScheduler: Finished 4
13/12/30 19:26:09 INFO local.LocalScheduler: Remove TaskSet 5.0 from pool 
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Completed ResultTask(5, 0)
13/12/30 19:26:09 INFO scheduler.DAGScheduler: Stage 5 (collect at <console>:21) finished in 0.091 s
13/12/30 19:26:09 INFO spark.SparkContext: Job finished: collect at <console>:21, took 0.323184943 s
res6: Array[(java.lang.String, Int)] = Array()


scala> 

在最后的收集方法将结果从RDD转换到一个数组中。需要注意的是，当我们没有为命令的结果指定一个名称（如var enTuples），一个变量为resN被自动创建。


我们可以结合前面的三个命令为一个：
scala> enPages.map(line => line.split(" ")).map(line => (line(0).substring(0, 8), line(3).toInt)).reduceByKey(_+_, 1).collect
13/12/30 19:42:41 INFO spark.SparkContext: Starting job: collect at <console>:17
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Registering RDD 10 (reduceByKey at <console>:17)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Got job 6 (collect at <console>:17) with 1 output partitions (allowLocal=false)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Final stage: Stage 7 (collect at <console>:17)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 8)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Missing parents: List(Stage 8)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Submitting Stage 8 (MapPartitionsRDD[10] at reduceByKey at <console>:17), which has no missing parents
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 8 (MapPartitionsRDD[10] at reduceByKey at <console>:17)
13/12/30 19:42:41 INFO local.LocalTaskSetManager: Size of task 5 is 2001 bytes
13/12/30 19:42:41 INFO local.LocalScheduler: Running 5
13/12/30 19:42:41 INFO spark.CacheManager: Cache key is rdd_2_0
13/12/30 19:42:41 INFO spark.CacheManager: Found partition in cache!
13/12/30 19:42:41 INFO local.LocalScheduler: Finished 5
13/12/30 19:42:41 INFO local.LocalScheduler: Remove TaskSet 8.0 from pool 
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Completed ShuffleMapTask(8, 0)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Stage 8 (reduceByKey at <console>:17) finished in 0.016 s
13/12/30 19:42:41 INFO scheduler.DAGScheduler: looking for newly runnable stages
13/12/30 19:42:41 INFO scheduler.DAGScheduler: running: Set()
13/12/30 19:42:41 INFO scheduler.DAGScheduler: waiting: Set(Stage 7)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: failed: Set()
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Missing parents for Stage 7: List()
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Submitting Stage 7 (MapPartitionsRDD[12] at reduceByKey at <console>:17), which is now runnable
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 7 (MapPartitionsRDD[12] at reduceByKey at <console>:17)
13/12/30 19:42:41 INFO local.LocalTaskSetManager: Size of task 6 is 1899 bytes
13/12/30 19:42:41 INFO local.LocalScheduler: Running 6
13/12/30 19:42:41 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 19:42:41 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  5 ms
13/12/30 19:42:41 INFO local.LocalScheduler: Finished 6
13/12/30 19:42:41 INFO local.LocalScheduler: Remove TaskSet 7.0 from pool 
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Completed ResultTask(7, 0)
13/12/30 19:42:41 INFO scheduler.DAGScheduler: Stage 7 (collect at <console>:17) finished in 0.017 s
13/12/30 19:42:41 INFO spark.SparkContext: Job finished: collect at <console>:17, took 0.083322742 s
res7: Array[(java.lang.String, Int)] = Array()


scala> 

7.假设我们要发现在属于我们的数据集三天被浏览了超过20万次的页面。从概念上讲，这个任务是类似于前面的查询。但是，考虑到大量的页面（23百万个不同的页面名称），新的任务是非常昂贵的。我们正在做用大量数据的网络shuffle来做gruop by操作。



总括来说，首先我们把每行数据分离到其各自的域。接下来，我们提取的页面名称和页面浏览数的字段。我们通过40 reduce来对key进行reduce操作，这个时候。然后，我们筛选出的网页，少于200,000总浏览量。
scala> enPages.map(l => l.split(" ")).map(l => (l(2), l(3).toInt)).reduceByKey(_+_, 40).filter(x => x._2 > 200000).map(x => (x._2, x._1)).collect.foreach(println)
13/12/30 20:21:56 INFO spark.SparkContext: Starting job: collect at <console>:17
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Registering RDD 22 (reduceByKey at <console>:17)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Got job 8 (collect at <console>:17) with 40 output partitions (allowLocal=false)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Final stage: Stage 11 (collect at <console>:17)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 12)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Missing parents: List(Stage 12)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Submitting Stage 12 (MapPartitionsRDD[22] at reduceByKey at <console>:17), which has no missing parents
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 12 (MapPartitionsRDD[22] at reduceByKey at <console>:17)
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 48 is 2002 bytes
13/12/30 20:21:56 INFO local.LocalScheduler: Running 48
13/12/30 20:21:56 INFO spark.CacheManager: Cache key is rdd_2_0
13/12/30 20:21:56 INFO spark.CacheManager: Found partition in cache!
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 48
13/12/30 20:21:56 INFO local.LocalScheduler: Remove TaskSet 12.0 from pool 
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ShuffleMapTask(12, 0)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Stage 12 (reduceByKey at <console>:17) finished in 0.026 s
13/12/30 20:21:56 INFO scheduler.DAGScheduler: looking for newly runnable stages
13/12/30 20:21:56 INFO scheduler.DAGScheduler: running: Set()
13/12/30 20:21:56 INFO scheduler.DAGScheduler: waiting: Set(Stage 11)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: failed: Set()
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Missing parents for Stage 11: List()
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Submitting Stage 11 (MappedRDD[26] at map at <console>:17), which is now runnable
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Submitting 40 missing tasks from Stage 11 (MappedRDD[26] at map at <console>:17)
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 49 is 1944 bytes
13/12/30 20:21:56 INFO local.LocalScheduler: Running 49
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 49
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 50 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 0)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 50
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 50
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 51 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 1)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 51
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 51
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 52 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 2)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 52
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 52
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 53 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 3)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 53
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 53
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 54 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 4)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 54
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 54
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 55 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 5)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 55
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 55
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 56 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 6)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 56
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 56
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 57 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 7)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 57
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 57
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 58 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 8)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 58
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 58
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 59 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 9)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 59
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 59
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 60 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 10)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 60
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 60
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 61 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 11)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 61
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 61
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 62 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 12)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 62
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 62
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 63 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 13)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 63
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 63
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 64 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 14)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 64
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 64
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 65 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 15)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 65
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  1 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 65
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 66 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 16)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 66
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  2 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 66
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 67 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 17)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 67
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 67
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 68 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 18)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 68
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 68
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 69 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 19)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 69
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 69
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 70 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 20)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 70
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 70
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 71 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 21)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 71
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  19 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 71
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 72 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 22)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 72
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  1 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 72
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 73 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 23)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 73
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 73
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 74 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 24)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 74
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 74
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 75 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 25)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 75
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  1 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 75
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 76 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 26)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 76
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 76
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 77 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 27)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 77
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 77
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 78 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 28)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 78
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  1 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 78
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 79 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 29)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 79
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 79
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 80 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 30)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 80
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 80
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 81 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 31)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 81
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 81
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 32)
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 82 is 1944 bytes
13/12/30 20:21:56 INFO local.LocalScheduler: Running 82
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  1 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 82
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 33)
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 83 is 1944 bytes
13/12/30 20:21:56 INFO local.LocalScheduler: Running 83
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 83
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 84 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 34)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 84
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  1 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 84
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 85 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 35)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 85
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 85
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 86 is 1944 bytes
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 36)
13/12/30 20:21:56 INFO local.LocalScheduler: Running 86
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 86
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 37)
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 87 is 1944 bytes
13/12/30 20:21:56 INFO local.LocalScheduler: Running 87
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 87
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 38)
13/12/30 20:21:56 INFO local.LocalTaskSetManager: Size of task 88 is 1944 bytes
13/12/30 20:21:56 INFO local.LocalScheduler: Running 88
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Getting 0 non-zero-bytes blocks out of 1 blocks
13/12/30 20:21:56 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: Started 0 remote gets in  0 ms
13/12/30 20:21:56 INFO local.LocalScheduler: Finished 88
13/12/30 20:21:56 INFO local.LocalScheduler: Remove TaskSet 11.0 from pool 
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Completed ResultTask(11, 39)
13/12/30 20:21:56 INFO scheduler.DAGScheduler: Stage 11 (collect at <console>:17) finished in 0.434 s
13/12/30 20:21:56 INFO spark.SparkContext: Job finished: collect at <console>:17, took 0.561397556 s


scala> 

对于一个给定问题，没有一成不变的方法来计算reduce的最优数量，您将通过使用不同的值尝试，随着时间的推移会建立一种直觉。


离开spark shell，键入exit提示。

8.您可以通过浏览的Java/ Scala的或Python API文档浏览全部RDD的API。

运行单机的spark 程序
由于时间限制，在本教程中，我们专注于使用Spark特设风格分析。然而，对于许多任务，它更有意义编写一个单机的spark 程序。我们将回到这个在sprak下stream，在那里你会真正写一个独立的spark 下的stream作业。我们不打算讨论如何构建，编译，并在这里单机的spark作业，但在我们继续之前，我们在这里列出有关单机spark的资源,来为你将来回来可以浏览。


首先，在AMI在本教程中，我们已把“模板”项目，为Scala和Java单机spark程序和sprak下stream。spark那些可以在 /root/scala-app-template 和/root/java-app-template的应用程序模板目录中找到（我们将在后面讨论的流媒体的） 。随意浏览这些目录的内容。您还可以找到建立在Java和Scala中运行单机spark程序的spark快速入门指南的一部分的例子。如需更详细信息，请参见Matei Zaharia’s 在第一次AMP训练营中的幻灯片和视频中聊关于独立spark工作。

你可能感兴趣的:(scala,spark,AMP训练营)

  • 绘本讲师训练营【24期】8/21阅读原创《独生小孩》 1784e22615e0
    24016-孟娟《独生小孩》图片发自App今天我想分享一个蛮特别的绘本，讲的是一个特殊的群体，我也是属于这个群体，80后的独生小孩。这是一本中国绘本，作者郭婧，也是一个80厚。全书一百多页，均为铅笔绘制，虽然为黑白色调，但并不显得沉闷。全书没有文字，犹如“默片”，但并不影响读者对该作品的理解，反而显得神秘，梦幻，給读者留下想象的空间。作者在前蝴蝶页这样写到：“我更希望父母和孩子一起分享这本书，使他
  • git常用命令笔记 咩酱-小羊 git笔记
    ###用习惯了idea总是不记得git的一些常见命令，需要用到的时候总是担心旁边站了人~~~记个笔记@_@，告诉自己看笔记不丢人初始化初始化一个新的Git仓库gitinit配置配置用户信息gitconfig--globaluser.name"YourName"gitconfig--globaluser.email"[email protected]"基本操作克隆远程仓库gitclone查看
  • PHP环境搭建详细教程 好看资源平台 前端php
    PHP是一个流行的服务器端脚本语言，广泛用于Web开发。为了使PHP能够在本地或服务器上运行，我们需要搭建一个合适的PHP环境。本教程将结合最新资料，介绍在不同操作系统上搭建PHP开发环境的多种方法，包括Windows、macOS和Linux系统的安装步骤，以及本地和Docker环境的配置。1.PHP环境搭建概述PHP环境的搭建主要分为以下几类：集成开发环境：例如XAMPP、WAMP、MAMP，这
  • 四章-32-点要素的聚合 彩云飘过
    本文基于腾讯课堂老胡的课《跟我学Openlayers--基础实例详解》做的学习笔记，使用的openlayers5.3.xapi。源码见1032.html，对应的官网示例https://openlayers.org/en/latest/examples/cluster.htmlhttps://openlayers.org/en/latest/examples/earthquake-clusters.
  • 展现思维导图魅力，不断挖掘人生宝藏 思维导图讲师Mandy
    第13期最强思维导图训练营已经结束一周了，但是我依旧是感觉所有学员还在努力的学习，这些学员中有教师、学生、白领、公务员、宝妈等等，只要你努力，只要你想改变自己，任何行业，任何岗位都可以参与进来，28天足以让你见成效，在这28天中，我们的学员不仅仅是收获了一枚毕业证，最重要的是让自己的思维方式得到升级，今天的你为自己投资，明天的你就会感谢你今天的付出，我们来听一听来自13期最强思维导图训练营优秀学员
  • nosql数据库技术与应用知识点 皆过客，揽星河 NoSQLnosql数据库大数据数据分析数据结构非关系型数据库
    Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
  • 在Ubuntu中编译含有JSON的文件出现报错 芝麻糊76 Linuxkill_buglinuxubuntujson
    在ubuntu中进行JSON相关学习的时候，我发现了一些小问题，决定与大家进行分享，减少踩坑时候出现不必要的时间耗费截取部分含有JSON部分的代码进行展示char*str="{\"title\":\"JSONExample\",\"author\":{\"name\":\"JohnDoe\",\"age\":35,\"isVerified\":true},\"tags\":[\"json\",\"
  • 渝婧感恩日记第68天 梁渝婧lydia
    1.哇！我真是太幸福啦！感恩奇迹感恩训练营毕业典礼，让我能共振到同学们的喜悦和能量，感谢！感谢！感谢！2.哇！我真是太幸福啦！感恩每天早起，运动3公里！这个星期又做到连续三天，不间断！感谢亲爱的渝婧！你真的是非常的棒！加油，继续坚持！感谢！感谢！感谢！3.哇！我真是太幸福啦！感恩曾正波班主任给我们分享的艾宾浩斯的记忆曲线255学习法，让我蠢蠢欲试，感谢！感谢！感谢！4.哇！我真是太幸福啦！感恩胜利
  • 网络编程基础 记得开心一点啊 网络
    目录♫什么是网络编程♫Socket套接字♪什么是Socket套接字♪数据报套接字♪流套接字♫数据报套接字通信模型♪数据报套接字通讯模型♪DatagramSocket♪DatagramPacket♪实现UDP的服务端代码♪实现UDP的客户端代码♫流套接字通信模型♪流套接字通讯模型♪ServerSocket♪Socket♪实现TCP的服务端代码♪实现TCP的客户端代码♫什么是网络编程网络编程，指网络上
  • 放松的一天 4da9b7687fa0
    20190325总结起床07:20图片发自App睡觉:23:00天气:晴今日任务清单学习·信息·阅读•水滴阅读Day40Alice’sAdventuresinWonderlandChapter6.2图片发自App•BBC跟读训练营Day24图片发自App图片发自App图片发自App•潘多拉口语训练营Day6Wow.Whatabigboy!•文化知识学习今日无•阅读时间地狱健康·饮食·锻炼•饮食目标
  • 博客网站制作教程 2401_85194651 javamaven
    首先就是技术框架：后端：Java+SpringBoot数据库：MySQL前端：Vue.js数据库连接：JPA(JavaPersistenceAPI)1.项目结构blog-app/├──backend/│├──src/main/java/com/example/blogapp/││├──BlogApplication.java││├──config/│││└──DatabaseConfig.java
  • 人到中年的5大恐惧 不想独白的独白
    这一段时间闭关在家，心里越来越没有底。全球疫情，全国疫情，一直在关心和自我调试中。但是，好像还是对自己的未来充满了无所适从。不想去做什么，也没有激情和兴趣去开始什么。人生过半，还有什么可以逆袭或改变的机会呢。不知道做什么的时候，去追剧，做美食，教育孩子，锻炼，花钱进什么什么读书训练营，打卡训练营，微信群，各种分享和共同体的群。但是还是没有任何的起色。就这样了吗。中午并不困，但是到了12点，还是习惯
  • LeetCode 53. Maximum Subarray 枯萎的海风 算法与OJC/C++leetcode
    1.题目描述Findthecontiguoussubarraywithinanarray(containingatleastonenumber)whichhasthelargestsum.Forexample,giventhearray[−2,1,−3,4,−1,2,1,−5,4],thecontiguoussubarray[4,−1,2,1]hasthelargestsum=6.clicktos
  • 【LeetCode】53. Maximum Subarray 墨染百城 LeetCodeleetcode
    问题描述问题链接：https://leetcode.com/problems/maximum-subarray/#/descriptionFindthecontiguoussubarraywithinanarray(containingatleastonenumber)whichhasthelargestsum.Forexample,giventhearray[-2,1,-3,4,-1,2,1,-
  • LeetCode 673. Number of Longest Increasing Subsequence (Java版; Meidum) littlehaes 字符串动态规划算法leetcode数据结构
    welcometomyblogLeetCode673.NumberofLongestIncreasingSubsequence(Java版;Meidum)题目描述Givenanunsortedarrayofintegers,findthenumberoflongestincreasingsubsequence.Example1:Input:[1,3,5,4,7]Output:2Explanatio
  • 股票公众号怎么赚钱的？炒股公众号靠什么赚钱？ 氧惠评测
    股票公众号赚钱的方式主要有以下几种：别指望别人对你好，你没有价值，别人怎么可能会对你好，要知道:人生都是相互的，你要让自己有价值，只要你有价值了，水到自然渠成了。氧惠APP是与以往完全不同的抖客+淘客app！2024全新模式，我的直推也会放到你下面。主打：带货高补贴，深受各位带货团队长喜爱（训练营导师每天出单带货几万单）。注册即可享受高补贴+0撸+捡漏等带货新体验。送万元推广大礼包，教你如何1年做
  • 抖音返利平台有哪些?值得推荐抖音返利app有哪些? 氧惠超好用
    随着抖音电商的崛起，越来越多的用户开始关注抖音返利平台希望通过这些平台在享受购物乐趣的同时，也能获得一定的返利优惠。那么，面对众多的抖音返利平台，哪个返利最高呢？本文将为您深入解析，助您找到最优选择。氧惠APP是与以往完全不同的抖客+淘客app！2024全新模式，我的直推也会放到你下面。主打：带货高补贴，深受各位带货团队长喜爱（训练营导师每天出单带货几万单）。注册即可享受高补贴+0撸+捡漏等带货新
  • Kubernetes 自定义控制器开发 IT回忆录 Kubeneteskubernetes
    目录前言一、CRD二、创建数据库表（Mysql）二、控制器开发1.使用kubernetes的examplecontroller模板2.在controller.go中新增数据表监听方法3.修改tools工具生成资源对象结构体定义这里记录开发k8s控制器的一般方式，controller开发主要使用k8s提供的client-go库进行。前言Controller监听集群内部资源对象的变化，编辑资源对象(增
  • svg图片兼容性和用法优缺点 独行侠_ef93
    svg图片的使用方法第一次来认认真真的研究了下svg图片，之前只是在网上见过，但都是一晃而过也没当回事，最近网站改版看到同事有用到svg格式的图片，想想自己干了几年的重构也没用过，这些细节的知识是应该好好研究研究了。暂时还没研究得完全透切，先记下目前为止所看到的吧不然又给忘了。svg可缩放矢量图形（ScalableVectorGraphics），顾名思义就是任意改变其大小也不会变形，是基于可扩展标
  • 超过晚上9点不打卡和点评 别有余味
    一开营宇彤老师就说到：点评导师们，为了让大家积极练习，你们能及时点评，每天晚上9点前打完卡，写不完作业的，点评官就不点评了。原因:声音学习，不同于演讲不同于写作，可以深夜耕耘，“深耕”。声音就得早早练，如果半夜练习，哪能放的开。邻居会同意吗？家人会同意吗？为了家庭，邻居和谐，深夜不练声。给大家养成好习惯在坚持星球第二期声音训练营打卡第5天，宇彤老师上午又在群里抛出要一个问题：“参加写作，演讲学习，
  • Kafka详细解析与应用分析 芊言芊语 kafka分布式
    Kafka是一个开源的分布式事件流平台（EventStreamingPlatform），由LinkedIn公司最初采用Scala语言开发，并基于ZooKeeper协调管理。如今，Kafka已经被Apache基金会纳入其项目体系，广泛应用于大数据实时处理领域。Kafka凭借其高吞吐量、持久化、分布式和可靠性的特点，成为构建实时流数据管道和流处理应用程序的重要工具。Kafka架构Kafka的架构主要由
  • 分享一个基于python的电子书数据采集与可视化分析 hadoop电子书数据分析与推荐系统 spark大数据毕设项目（源码、调试、LW、开题、PPT) 计算机源码社 Python项目大数据大数据pythonhadoop计算机毕业设计选题计算机毕业设计源码数据分析spark毕设
    作者：计算机源码社个人简介：本人八年开发经验，擅长Java、Python、PHP、.NET、Node.js、Android、微信小程序、爬虫、大数据、机器学习等，大家有这一块的问题可以一起交流！学习资料、程序开发、技术解答、文档报告如需要源码，可以扫取文章下方二维码联系咨询Java项目微信小程序项目Android项目Python项目PHP项目ASP.NET项目Node.js项目选题推荐项目实战|p
  • PAT Advanced 1015. Reversible Primes (C语言实现) OliverLew
    我的PAT系列文章更新重心已移至Github，欢迎来看PAT题解的小伙伴请到GithubPages浏览最新内容。此处文章目前已更新至与GithubPages同步。欢迎star我的repo。题目Areversibleprimeinanynumbersystemisaprimewhose"reverse"inthatnumbersystemisalsoaprime.Forexampleinthedec
  • tf.get_collection() yalesaleng
    此函数有两个参数，key和scope。Args:1.key:Thekeyforthecollection.Forexample,theGraphKeysclasscontainsmanystandardnamesforcollections.2.scope:(Optional.)Ifsupplied,theresultinglistisfilteredtoincludeonlyitemswhose
  • SQLite的入门级项目学习记录（二） 深蓝海拓 SQLite学习笔记sqlite学习数据库
    再补充一些基础知识：并行操作的问题1、可以多游标同时运行SQLite，对于同一个连接sqlite3.connect(db_file)，可以同时创建多个游标，每个游标都是独立的，可以执行各自的SQL命令序列。importsqlite3#创建数据库连接conn=sqlite3.connect('example.db')#创建第一个游标cursor1=conn.cursor()cursor1.execu
  • 大数据真实面试题---SQL The博宇 大数据面试题——SQL大数据mysqlsql数据库bigdata
    视频号数据分析组外包招聘笔试题时间限时45分钟完成。题目根据3张表表结构，写出具体求解的SQL代码（搞笑品类定义：视频分类或者视频创建者分类为“搞笑”）1、表创建语句：createtablet_user_video_action_d(dsint,user_idstring,video_idstring,action_typeint,`timestamp`bigint)rowformatdelimi
  • 图片中的上采样，下采样和通道融合(up-sample, down-sample, channel confusion) 迪三 #图像处理_PyTorch计算机视觉深度学习人工智能
    前言以conv2d为例（即图片），Pytorch中输入的数据格式为tensor，格式为:[N,C,W,H,W]第一维N.代表图片个数，类似一个batch里面有N张图片第二维C.代表通道数，在模型中输入如果为彩色，常用RGB三色图，那么就是3维，即C=3。如果是黑白的，即灰度图，那么只有一个通道，即C=1第三维H.代表图片的高度，H的数量是图片像素的列数第四维W.代表图片的宽度，W的数量是图片像素的
  • Spark 组件 GraphX、Streaming 叶域 大数据sparkspark大数据分布式
    Spark组件GraphX、Streaming一、SparkGraphX1.1GraphX的主要概念1.2GraphX的核心操作1.3示例代码1.4GraphX的应用场景二、SparkStreaming2.1SparkStreaming的主要概念2.2示例代码2.3SparkStreaming的集成2.4SparkStreaming的应用场景SparkGraphX用于处理图和图并行计算。Graph
  • 2023-06-30 151d580c6a8c
    翔龙商学院三天学习后感：经过2023年6月27-29日三天学习后，确实成长很快，给自己的成长又提高一个台阶，感谢集团尤总及各位老师的辛苦再次感谢和感恩，对具体学习的翔龙商学院第十七期TTT训练营温故而知新，回顾一下具体学习情况，打造三师农化师，会议推广师，市场策划师继续落地。1、范永强老师从土壤与农化，李比希的矿物营养学，强酸＜5.5，酸性5.5-6.5，中性6.5-7.5，碱性7.5-8.5，强
  • 意志力日志第二次总结意志力消退：玩物丧志首当其冲 保险经纪人生
    高级注意力不留给游戏意志力训练营已经过去半个月了，慢慢实践也利用起来了，记录时间时间的流逝确实很快，但是意志力的流逝如不知原理很快就会出现大问题。意志力消退罪魁祸首是睡眠饮食，其次就是电子游戏。现在记录了娱乐时间，如果每天的游戏时间超过一小时自己就非常有罪恶感，因为自己在游戏中很是争抢好胜吧，但是高强度的集中注意力消耗了自己仅有的高级注意力，以后打游戏就佛系了，自己最重要的东西要放在心上。运动让生
  • knob UI插件使用 换个号韩国红果果 JavaScriptjsonpknob
    图形是用canvas绘制的 js代码 var paras = { max:800, min:100, skin:'tron',//button type thickness:.3,//button width width:'200',//define canvas width.,canvas height displayInput:'tr
  • Android+Jquery Mobile学习系列(5)-SQLite数据库 白糖_ JQuery Mobile
    目录导航   SQLite是轻量级的、嵌入式的、关系型数据库，目前已经在iPhone、Android等手机系统中使用,SQLite可移植性好，很容易使用，很小，高效而且可靠。   因为Android已经集成了SQLite，所以开发人员无需引入任何JAR包，而且Android也针对SQLite封装了专属的API，调用起来非常快捷方便。   我也是第一次接触S
  • impala-2.1.2-CDH5.3.2 dayutianfei impala
    最近在整理impala编译的东西，简单记录几个要点： 根据官网的信息（https://github.com/cloudera/Impala/wiki/How-to-build-Impala）： 1. 首次编译impala，推荐使用命令： ${IMPALA_HOME}/buildall.sh -skiptests -build_shared_libs -format 2.仅编译BE ${I
  • 求二进制数中1的个数 周凡杨 java算法二进制
    解法一： 对于一个正整数如果是偶数，该数的二进制数的最后一位是 0 ，反之若是奇数，则该数的二进制数的最后一位是 1 。因此，可以考虑利用位移、判断奇偶来实现。   public int bitCount(int x){ int count = 0; while(x!=0){ if(x%2!=0){ /
  • spring中hibernate及事务配置 g21121 Hibernate
    hibernate的sessionFactory配置： <!-- hibernate sessionFactory配置 --> <bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean"> <
  • log4j.properties 使用 510888780 log4j
    log4j.properties 使用 一.参数意义说明 输出级别的种类 ERROR、WARN、INFO、DEBUG ERROR 为严重错误 主要是程序的错误 WARN 为一般警告，比如session丢失 INFO 为一般要显示的信息，比如登录登出 DEBUG 为程序的调试信息 配置日志信息输出目的地 log4j.appender.appenderName = fully.qua
  • Spring mvc-jfreeChart柱图（2） 布衣凌宇 jfreechart
    上一篇中生成的图是静态的，这篇将按条件进行搜索，并统计成图表，左面为统计图，右面显示搜索出的结果。 第一步：导包 第二步；配置web.xml(上一篇有代码) 建BarRenderer类用于柱子颜色 import java.awt.Color; import java.awt.Paint; import org.jfree.chart.renderer.category.BarR
  • 我的spring学习笔记14-容器扩展点之PropertyPlaceholderConfigurer aijuans Spring3
    PropertyPlaceholderConfigurer是个bean工厂后置处理器的实现，也就是BeanFactoryPostProcessor接口的一个实现。关于BeanFactoryPostProcessor和BeanPostProcessor类似。我会在其他地方介绍。 PropertyPlaceholderConfigurer可以将上下文（配置文件）中的属性值放在另一个单独的标准java
  • maven 之 cobertura 简单使用 antlove maventestunitcoberturareport
    1. 创建一个maven项目 2. 创建com.CoberturaStart.java package com; public class CoberturaStart { public void helloEveryone(){ System.out.println("=================================================
  • 程序的执行顺序 百合不是茶 JAVA执行顺序
          刚在看java核心技术时发现对java的执行顺序不是很明白了,百度一下也没有找到适合自己的资料,所以就简单的回顾一下吧   代码如下;     经典的程序执行面试题 //关于程序执行的顺序 //例如： //定义一个基类 public class A(){ public A(
  • 设置session失效的几种方法 bijian1013 web.xmlsession失效监听器
    在系统登录后，都会设置一个当前session失效的时间，以确保在用户长时间不与服务器交互，自动退出登录，销毁session。具体设置很简单，方法有三种：（1）在主页面或者公共页面中加入：session.setMaxInactiveInterval(900);参数900单位是秒，即在没有活动15分钟后，session将失效。这里要注意这个session设置的时间是根据服务器来计算的，而不是客户端。所
  • java jvm常用命令工具 bijian1013 javajvm
    一.概述         程序运行中经常会遇到各种问题，定位问题时通常需要综合各种信息，如系统日志、堆dump文件、线程dump文件、GC日志等。通过虚拟机监控和诊断工具可以帮忙我们快速获取、分析需要的数据，进而提高问题解决速度。 本文将介绍虚拟机常用监控和问题诊断命令工具的使用方法，主要包含以下工具:       &nbs
  • 【Spring框架一】Spring常用注解之Autowired和Resource注解 bit1129 Spring常用注解
    Spring自从2.0引入注解的方式取代XML配置的方式来做IOC之后，对Spring一些常用注解的含义行为一直处于比较模糊的状态，写几篇总结下Spring常用的注解。本篇包含的注解有如下几个： Autowired Resource Component Service Controller Transactional 根据它们的功能、目的，可以分为三组，Autow
  • mysql 操作遇到safe update mode问题 bitray update
        我并不知道出现这个问题的实际原理,只是通过其他朋友的博客,文章得知的一个解决方案,目前先记录一个解决方法,未来要是真了解以后,还会继续补全.     在mysql5中有一个safe update mode,这个模式让sql操作更加安全,据说要求有where条件,防止全表更新操作.如果必须要进行全表操作,我们可以执行 SET
  • nginx_perl试用 ronin47 nginx_perl试用
    因为空闲时间比较多，所以在CPAN上乱翻，看到了nginx_perl这个项目(原名Nginx::Engine)，现在托管在github.com上。地址见：https://github.com/zzzcpan/nginx-perl 这个模块的目的，是在nginx内置官方perl模块的基础上，实现一系列异步非阻塞的api。用connector/writer/reader完成类似proxy的功能（这里
  • java-63-在字符串中删除特定的字符 bylijinnan java
    public class DeleteSpecificChars { /** * Q 63 在字符串中删除特定的字符 * 输入两个字符串，从第一字符串中删除第二个字符串中所有的字符。 * 例如，输入”They are students.”和”aeiou”，则删除之后的第一个字符串变成”Thy r stdnts.” */ public static voi
  • EffectiveJava--创建和销毁对象 ccii 创建和销毁对象
    本章内容： 1. 考虑用静态工厂方法代替构造器 2. 遇到多个构造器参数时要考虑用构建器（Builder模式） 3. 用私有构造器或者枚举类型强化Singleton属性 4. 通过私有构造器强化不可实例化的能力 5. 避免创建不必要的对象 6. 消除过期的对象引用 7. 避免使用终结方法 1. 考虑用静态工厂方法代替构造器     类可以通过
  • [宇宙时代]四边形理论与光速飞行 comsci
       从四边形理论来推论 为什么光子飞船必须获得星光信号才能够进行光速飞行？    一组星体组成星座  向空间辐射一组由复杂星光信号组成的辐射频带，按照四边形-频率假说  一组频率就代表一个时空的入口    那么这种由星光信号组成的辐射频带就代表由这些星体所控制的时空通道，该时空通道在三维空间的投影是一
  • ubuntu server下python脚本迁移数据 cywhoyi pythonKettlepymysqlcx_Oracleubuntu server
    因为是在Ubuntu下，所以安装python、pip、pymysql等都极其方便，sudo apt-get install pymysql， 但是在安装cx_Oracle（连接oracle的模块）出现许多问题，查阅相关资料，发现这边文章能够帮我解决，希望大家少走点弯路。http://www.tbdazhe.com/archives/602 1.安装python 2.安装pip、pymysql
  • Ajax正确但是请求不到值解决方案 dashuaifu Ajaxasync
    Ajax正确但是请求不到值解决方案   解决方案：1 .     async: false ,    2.     设置延时执行js里的ajax或者延时后台java方法！！！！！！！   例如：   $.ajax({     &
  • windows安装配置php+memcached dcj3sjt126com PHPInstallmemcache
    Windows下Memcached的安装配置方法 1、将第一个包解压放某个盘下面，比如在c:\memcached。 2、在终端（也即cmd命令界面）下输入 'c:\memcached\memcached.exe -d install' 安装。 3、再输入： 'c:\memcached\memcached.exe -d start' 启动。（需要注意的: 以后memcached将作为windo
  • iOS开发学习路径的一些建议 dcj3sjt126com ios
    iOS论坛里有朋友要求回答帖子，帖子的标题是： 想学IOS开发高阶一点的东西，从何开始，然后我吧啦吧啦回答写了很多。既然敲了那么多字，我就把我写的回复也贴到博客里来分享，希望能对大家有帮助。欢迎大家也到帖子里讨论和分享，地址：http://bbs.csdn.net/topics/390920759   下面是我回复的内容：   结合自己情况聊下iOS学习建议，
  • Javascript闭包概念 fanfanlovey JavaScript闭包
    1.参考资料 http://www.jb51.net/article/24101.htm http://blog.csdn.net/yn49782026/article/details/8549462 2.内容概述 要理解闭包，首先需要理解变量作用域问题 内部函数可以饮用外面全局变量 var n=999; 　　functio
  • yum安装mysql5.6 haisheng mysql
    1、安装http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm   2、yum install mysql   3、yum install mysql-server   4、vi /etc/my.cnf   添加character_set_server=utf8
  • po/bo/vo/dao/pojo的详介 IT_zhlp80 javaBOVODAOPOJOpo
        JAVA几种对象的解释 PO:persistant object持久对象,可以看成是与数据库中的表相映射的java对象。最简单的PO就是对应数据库中某个表中的一条记录，多个记录可以用PO的集合。PO中应该不包含任何对数据库的操作. VO:value object值对象。通常用于业务层之间的数据传递，和PO一样也是仅仅包含数据而已。但应是抽象出的业务对象,可
  • java设计模式 kerryg java设计模式
    设计模式的分类：    一、 设计模式总体分为三大类： 1、创建型模式（5种）：工厂方法模式，抽象工厂模式，单例模式，建造者模式，原型模式。 2、结构型模式（7种）：适配器模式，装饰器模式，代理模式，外观模式，桥接模式，组合模式，享元模式。 3、行为型模式（11种）：策略模式，模版方法模式，观察者模式，迭代子模式，责任链模式，命令模式，备忘录模式，状态模式，访问者
  • [1]CXF3.1整合Spring开发webservice——helloworld篇 木头.java springwebserviceCXF
    Spring 版本3.2.10 CXF 版本3.1.1 项目采用MAVEN组织依赖jar 我这里是有parent的pom，为了简洁明了，我直接把所有的依赖都列一起了，所以都没version，反正上面已经写了版本 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="ht
  • Google 工程师亲授：菜鸟开发者一定要投资的十大目标 qindongliang1922 工作感悟人生
    身为软件开发者，有什么是一定得投资的？ Google 软件工程师 Emanuel Saringan 整理了十项他认为必要的投资，第一项就是身体健康，英文与数学也都是必备能力吗？来看看他怎么说。（以下文字以作者第一人称撰写）） 你的健康 无疑地，软件开发者是世界上最久坐不动的职业之一。 每天连坐八到十六小时，休息时间只有一点点，绝对会让你的鲔鱼肚肆无忌惮的生长。肥胖容易扩大罹患其他疾病的风险，
  • linux打开最大文件数量1,048,576 tianzhihehe clinux
    File descriptors are represented by the C int type. Not using a special type is often  considered odd, but is, historically, the Unix way. Each Linux process has a maximum number of files th
  • java语言中PO、VO、DAO、BO、POJO几种对象的解释 衞酆夼 javaVOBOPOJOpo
    PO:persistant object持久对象 最形象的理解就是一个PO就是数据库中的一条记录。好处是可以把一条记录作为一个对象处理，可以方便的转为其它对象。可以看成是与数据库中的表相映射的java对象。最简单的PO就是对应数据库中某个表中的一条记录，多个记录可以用PO的集合。PO中应该不包含任何对数据库的操作。 BO:business object业务对象 封装业务逻辑的java对象
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他