bob601450868

Spark代码3之Action：reduce，reduceByKey,sorted,lookup,take,saveAsTextFile

代码：

package LocalSpark

/**
  * Created by xubo on 2016/3/3.
  */

import java.text.SimpleDateFormat
import java.util.Date

import org.apache.spark._
object Action1 {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("Transformation1").setMaster("local")
    val spark = new SparkContext(conf)
    //action
    var a1 = spark.parallelize(List(('a', 1), ('b', 1)))
    var a2 = spark.parallelize(List(('c', 1), ('d', 1)))
    var a3 = spark.parallelize(List(('a', 1), ('b', 1), ('a', 1)))
    var a4 = spark.parallelize(List(('c', 1), ('d', 1), ('b', 1), ('b', 2), ('b', 3), ('a', 1), ('a', 2)))
    for (i <- a4.lookup('a')) println(i)

    //reduce
//    var a5=spark.parallelize(List(1,2,4,3,2,5,7,1,3,4))
    var a5=spark.parallelize(List(1,2,4,3,5))
    println(a5.reduce(_+_))

    //reduceByKey
    var r4=(a4.reduceByKey(_+_))
    for((a,b)<-a4)  println("a4:("+a+","+b+")")
    for((a,b)<-r4)  println("r4:("+a+","+b+")")
//    for(i<-r4)  i.mkString(",")  i
    // sortBy
    val s4=r4.sortBy(_._2)
    for((a,b)<-s4)  println("s4:("+a+","+b+")")

    //take
//    for (j,k)=a4.take(1)
//    println("s4:("+a+","+b+")")
    for((a,b)<-a4.take(1))  println("t4:("+a+","+b+")")

    //SaveAsTextFile
    val iString=new SimpleDateFormat("yyyyMMddHHmmssSSS").format(new Date() )
    val soutput="hdfs://<strong>MasterIP</strong>:9000/output/"+iString;  //换成真实IP
    s4.saveAsTextFile(soutput)

    spark.stop()
  }
}

运行结果：

D:\1win7\java\jdk\bin\java -Didea.launcher.port=7533 "-Didea.launcher.bin.path=D:\1win7\idea\IntelliJ IDEA Community Edition 15.0.4\bin" -Dfile.encoding=UTF-8 -classpath "D:\1win7\java\jdk\jre\lib\charsets.jar;D:\1win7\java\jdk\jre\lib\deploy.jar;D:\1win7\java\jdk\jre\lib\ext\access-bridge-64.jar;D:\1win7\java\jdk\jre\lib\ext\dnsns.jar;D:\1win7\java\jdk\jre\lib\ext\jaccess.jar;D:\1win7\java\jdk\jre\lib\ext\localedata.jar;D:\1win7\java\jdk\jre\lib\ext\sunec.jar;D:\1win7\java\jdk\jre\lib\ext\sunjce_provider.jar;D:\1win7\java\jdk\jre\lib\ext\sunmscapi.jar;D:\1win7\java\jdk\jre\lib\ext\zipfs.jar;D:\1win7\java\jdk\jre\lib\javaws.jar;D:\1win7\java\jdk\jre\lib\jce.jar;D:\1win7\java\jdk\jre\lib\jfr.jar;D:\1win7\java\jdk\jre\lib\jfxrt.jar;D:\1win7\java\jdk\jre\lib\jsse.jar;D:\1win7\java\jdk\jre\lib\management-agent.jar;D:\1win7\java\jdk\jre\lib\plugin.jar;D:\1win7\java\jdk\jre\lib\resources.jar;D:\1win7\java\jdk\jre\lib\rt.jar;D:\1win7\scala;D:\1win7\scala\lib;D:\all\idea\scala2\out\production\scala2;G:\149\spark-assembly-1.5.2-hadoop2.6.0.jar;D:\1win7\scala\lib\scala-actors-migration.jar;D:\1win7\scala\lib\scala-actors.jar;D:\1win7\scala\lib\scala-library.jar;D:\1win7\scala\lib\scala-reflect.jar;D:\1win7\scala\lib\scala-swing.jar;D:\1win7\idea\IntelliJ IDEA Community Edition 15.0.4\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain LocalSpark.Action1
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/03/04 10:50:15 INFO SparkContext: Running Spark version 1.5.2
16/03/04 10:50:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/04 10:50:16 INFO SecurityManager: Changing view acls to: xubo
16/03/04 10:50:16 INFO SecurityManager: Changing modify acls to: xubo
16/03/04 10:50:16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(xubo); users with modify permissions: Set(xubo)
16/03/04 10:50:16 INFO Slf4jLogger: Slf4jLogger started
16/03/04 10:50:16 INFO Remoting: Starting remoting
16/03/04 10:50:17 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:62811]
16/03/04 10:50:17 INFO Utils: Successfully started service 'sparkDriver' on port 62811.
16/03/04 10:50:17 INFO SparkEnv: Registering MapOutputTracker
16/03/04 10:50:17 INFO SparkEnv: Registering BlockManagerMaster
16/03/04 10:50:17 INFO DiskBlockManager: Created local directory at C:\Users\xubo\AppData\Local\Temp\blockmgr-f729c5d8-48fd-4bba-80d6-48f0cc86692c
16/03/04 10:50:17 INFO MemoryStore: MemoryStore started with capacity 730.6 MB
16/03/04 10:50:17 INFO HttpFileServer: HTTP File server directory is C:\Users\xubo\AppData\Local\Temp\spark-52f1804d-6099-46a6-b52e-473d64044637\httpd-525f3e90-1ce9-451a-9abb-f2cbd979599c
16/03/04 10:50:17 INFO HttpServer: Starting HTTP Server
16/03/04 10:50:17 INFO Utils: Successfully started service 'HTTP file server' on port 62812.
16/03/04 10:50:17 INFO SparkEnv: Registering OutputCommitCoordinator
16/03/04 10:50:17 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/03/04 10:50:17 INFO SparkUI: Started SparkUI at http://219.219.220.162:4040
16/03/04 10:50:18 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/03/04 10:50:18 INFO Executor: Starting executor ID driver on host localhost
16/03/04 10:50:18 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 62819.
16/03/04 10:50:18 INFO NettyBlockTransferService: Server created on 62819
16/03/04 10:50:18 INFO BlockManagerMaster: Trying to register BlockManager
16/03/04 10:50:18 INFO BlockManagerMasterEndpoint: Registering block manager localhost:62819 with 730.6 MB RAM, BlockManagerId(driver, localhost, 62819)
16/03/04 10:50:18 INFO BlockManagerMaster: Registered BlockManager
16/03/04 10:50:18 INFO SparkContext: Starting job: lookup at Action1.scala:20
16/03/04 10:50:18 INFO DAGScheduler: Got job 0 (lookup at Action1.scala:20) with 1 output partitions
16/03/04 10:50:18 INFO DAGScheduler: Final stage: ResultStage 0(lookup at Action1.scala:20)
16/03/04 10:50:18 INFO DAGScheduler: Parents of final stage: List()
16/03/04 10:50:18 INFO DAGScheduler: Missing parents: List()
16/03/04 10:50:19 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[5] at lookup at Action1.scala:20), which has no missing parents
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(2536) called with curMem=0, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.5 KB, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1430) called with curMem=2536, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1430.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:62819 (size: 1430.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[5] at lookup at Action1.scala:20)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 2283 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 906 bytes result sent to driver
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 95 ms on localhost (1/1)
16/03/04 10:50:19 INFO DAGScheduler: ResultStage 0 (lookup at Action1.scala:20) finished in 0.116 s
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: Job 0 finished: lookup at Action1.scala:20, took 0.503116 s
1
2
16/03/04 10:50:19 INFO SparkContext: Starting job: reduce at Action1.scala:25
16/03/04 10:50:19 INFO DAGScheduler: Got job 1 (reduce at Action1.scala:25) with 1 output partitions
16/03/04 10:50:19 INFO DAGScheduler: Final stage: ResultStage 1(reduce at Action1.scala:25)
16/03/04 10:50:19 INFO DAGScheduler: Parents of final stage: List()
16/03/04 10:50:19 INFO DAGScheduler: Missing parents: List()
16/03/04 10:50:19 INFO DAGScheduler: Submitting ResultStage 1 (ParallelCollectionRDD[6] at parallelize at Action1.scala:24), which has no missing parents
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1192) called with curMem=3966, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 1192.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(854) called with curMem=5158, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 854.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:62819 (size: 854.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (ParallelCollectionRDD[6] at parallelize at Action1.scala:24)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, PROCESS_LOCAL, 2045 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1031 bytes result sent to driver
16/03/04 10:50:19 INFO DAGScheduler: ResultStage 1 (reduce at Action1.scala:25) finished in 0.031 s
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 31 ms on localhost (1/1)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: Job 1 finished: reduce at Action1.scala:25, took 0.053935 s
15
16/03/04 10:50:19 INFO SparkContext: Starting job: foreach at Action1.scala:29
16/03/04 10:50:19 INFO DAGScheduler: Got job 2 (foreach at Action1.scala:29) with 1 output partitions
16/03/04 10:50:19 INFO DAGScheduler: Final stage: ResultStage 2(foreach at Action1.scala:29)
16/03/04 10:50:19 INFO DAGScheduler: Parents of final stage: List()
16/03/04 10:50:19 INFO DAGScheduler: Missing parents: List()
16/03/04 10:50:19 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[8] at filter at Action1.scala:29), which has no missing parents
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1800) called with curMem=6012, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 1800.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1137) called with curMem=7812, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1137.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:62819 (size: 1137.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[8] at filter at Action1.scala:29)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, PROCESS_LOCAL, 2283 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 915 bytes result sent to driver
a4:(c,1)
a4:(d,1)
a4:(b,1)
a4:(b,2)
a4:(b,3)
a4:(a,1)
a4:(a,2)
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 10 ms on localhost (1/1)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: ResultStage 2 (foreach at Action1.scala:29) finished in 0.011 s
16/03/04 10:50:19 INFO DAGScheduler: Job 2 finished: foreach at Action1.scala:29, took 0.021940 s
16/03/04 10:50:19 INFO SparkContext: Starting job: foreach at Action1.scala:30
16/03/04 10:50:19 INFO DAGScheduler: Registering RDD 3 (parallelize at Action1.scala:19)
16/03/04 10:50:19 INFO DAGScheduler: Got job 3 (foreach at Action1.scala:30) with 1 output partitions
16/03/04 10:50:19 INFO DAGScheduler: Final stage: ResultStage 4(foreach at Action1.scala:30)
16/03/04 10:50:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3)
16/03/04 10:50:19 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 3)
16/03/04 10:50:19 INFO DAGScheduler: Submitting ShuffleMapStage 3 (ParallelCollectionRDD[3] at parallelize at Action1.scala:19), which has no missing parents
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1856) called with curMem=8949, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 1856.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1189) called with curMem=10805, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1189.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:62819 (size: 1189.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 3 (ParallelCollectionRDD[3] at parallelize at Action1.scala:19)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, PROCESS_LOCAL, 2272 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1158 bytes result sent to driver
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 54 ms on localhost (1/1)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: ShuffleMapStage 3 (parallelize at Action1.scala:19) finished in 0.055 s
16/03/04 10:50:19 INFO DAGScheduler: looking for newly runnable stages
16/03/04 10:50:19 INFO DAGScheduler: running: Set()
16/03/04 10:50:19 INFO DAGScheduler: waiting: Set(ResultStage 4)
16/03/04 10:50:19 INFO DAGScheduler: failed: Set()
16/03/04 10:50:19 INFO DAGScheduler: Missing parents for ResultStage 4: List()
16/03/04 10:50:19 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[9] at filter at Action1.scala:30), which is now runnable
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(2648) called with curMem=11994, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 2.6 KB, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1571) called with curMem=14642, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1571.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:62819 (size: 1571.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[9] at filter at Action1.scala:30)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, PROCESS_LOCAL, 1901 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
16/03/04 10:50:19 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/03/04 10:50:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms
r4:(d,1)
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1165 bytes result sent to driver
r4:(a,3)
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 47 ms on localhost (1/1)
r4:(b,6)
16/03/04 10:50:19 INFO DAGScheduler: ResultStage 4 (foreach at Action1.scala:30) finished in 0.048 s
r4:(c,1)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: Job 3 finished: foreach at Action1.scala:30, took 0.152494 s
16/03/04 10:50:19 INFO SparkContext: Starting job: foreach at Action1.scala:34
16/03/04 10:50:19 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 143 bytes
16/03/04 10:50:19 INFO DAGScheduler: Registering RDD 10 (sortBy at Action1.scala:33)
16/03/04 10:50:19 INFO DAGScheduler: Got job 4 (foreach at Action1.scala:34) with 1 output partitions
16/03/04 10:50:19 INFO DAGScheduler: Final stage: ResultStage 7(foreach at Action1.scala:34)
16/03/04 10:50:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 6)
16/03/04 10:50:19 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 6)
16/03/04 10:50:19 INFO DAGScheduler: Submitting ShuffleMapStage 6 (MapPartitionsRDD[10] at sortBy at Action1.scala:33), which has no missing parents
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(3192) called with curMem=16213, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.1 KB, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1860) called with curMem=19405, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1860.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:62819 (size: 1860.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[10] at sortBy at Action1.scala:33)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 5, localhost, PROCESS_LOCAL, 1890 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 6.0 (TID 5)
16/03/04 10:50:19 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/03/04 10:50:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 6.0 (TID 5). 1374 bytes result sent to driver
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 5) in 25 ms on localhost (1/1)
16/03/04 10:50:19 INFO DAGScheduler: ShuffleMapStage 6 (sortBy at Action1.scala:33) finished in 0.025 s
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: looking for newly runnable stages
16/03/04 10:50:19 INFO DAGScheduler: running: Set()
16/03/04 10:50:19 INFO DAGScheduler: waiting: Set(ResultStage 7)
16/03/04 10:50:19 INFO DAGScheduler: failed: Set()
16/03/04 10:50:19 INFO DAGScheduler: Missing parents for ResultStage 7: List()
16/03/04 10:50:19 INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[13] at filter at Action1.scala:34), which is now runnable
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(3128) called with curMem=21265, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.1 KB, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1805) called with curMem=24393, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1805.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:62819 (size: 1805.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[13] at filter at Action1.scala:34)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 7.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 6, localhost, PROCESS_LOCAL, 1901 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 7.0 (TID 6)
16/03/04 10:50:19 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/03/04 10:50:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 7.0 (TID 6). 1165 bytes result sent to driver
s4:(d,1)
s4:(c,1)
s4:(a,3)
s4:(b,6)
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 6) in 19 ms on localhost (1/1)
16/03/04 10:50:19 INFO DAGScheduler: ResultStage 7 (foreach at Action1.scala:34) finished in 0.019 s
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: Job 4 finished: foreach at Action1.scala:34, took 0.073994 s
16/03/04 10:50:19 INFO SparkContext: Starting job: take at Action1.scala:39
16/03/04 10:50:19 INFO DAGScheduler: Got job 5 (take at Action1.scala:39) with 1 output partitions
16/03/04 10:50:19 INFO DAGScheduler: Final stage: ResultStage 8(take at Action1.scala:39)
16/03/04 10:50:19 INFO DAGScheduler: Parents of final stage: List()
16/03/04 10:50:19 INFO DAGScheduler: Missing parents: List()
16/03/04 10:50:19 INFO DAGScheduler: Submitting ResultStage 8 (ParallelCollectionRDD[3] at parallelize at Action1.scala:19), which has no missing parents
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(1224) called with curMem=26198, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 1224.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO MemoryStore: ensureFreeSpace(804) called with curMem=27422, maxMem=766075207
16/03/04 10:50:19 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 804.0 B, free 730.6 MB)
16/03/04 10:50:19 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:62819 (size: 804.0 B, free: 730.6 MB)
16/03/04 10:50:19 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (ParallelCollectionRDD[3] at parallelize at Action1.scala:19)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks
16/03/04 10:50:19 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 7, localhost, PROCESS_LOCAL, 2283 bytes)
16/03/04 10:50:19 INFO Executor: Running task 0.0 in stage 8.0 (TID 7)
16/03/04 10:50:19 INFO Executor: Finished task 0.0 in stage 8.0 (TID 7). 1043 bytes result sent to driver
16/03/04 10:50:19 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 7) in 13 ms on localhost (1/1)
16/03/04 10:50:19 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 
16/03/04 10:50:19 INFO DAGScheduler: ResultStage 8 (take at Action1.scala:39) finished in 0.018 s
16/03/04 10:50:19 INFO DAGScheduler: Job 5 finished: take at Action1.scala:39, took 0.064104 s
t4:(c,1)
16/03/04 10:50:20 INFO BlockManagerInfo: Removed broadcast_6_piece0 on localhost:62819 in memory (size: 1805.0 B, free: 730.6 MB)
16/03/04 10:50:20 INFO ContextCleaner: Cleaned accumulator 7
16/03/04 10:50:20 INFO BlockManagerInfo: Removed broadcast_5_piece0 on localhost:62819 in memory (size: 1860.0 B, free: 730.6 MB)
16/03/04 10:50:20 INFO ContextCleaner: Cleaned accumulator 6
16/03/04 10:50:20 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:62819 in memory (size: 1571.0 B, free: 730.6 MB)
16/03/04 10:50:20 INFO ContextCleaner: Cleaned accumulator 5
16/03/04 10:50:20 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:62819 in memory (size: 1189.0 B, free: 730.6 MB)
16/03/04 10:50:20 INFO ContextCleaner: Cleaned accumulator 4
16/03/04 10:50:20 INFO BlockManagerInfo: Removed broadcast_2_piece0 on localhost:62819 in memory (size: 1137.0 B, free: 730.6 MB)
16/03/04 10:50:20 INFO ContextCleaner: Cleaned accumulator 3
16/03/04 10:50:20 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:62819 in memory (size: 854.0 B, free: 730.6 MB)
16/03/04 10:50:20 INFO ContextCleaner: Cleaned accumulator 2
16/03/04 10:50:20 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:62819 in memory (size: 1430.0 B, free: 730.6 MB)
16/03/04 10:50:20 INFO ContextCleaner: Cleaned accumulator 1
16/03/04 10:50:25 WARN : Your hostname, xubo-PC resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:16c%19, but we couldn't find any external IP address!
16/03/04 10:50:44 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
16/03/04 10:50:44 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
16/03/04 10:50:44 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
16/03/04 10:50:44 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
16/03/04 10:50:44 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
16/03/04 10:50:46 INFO SparkContext: Starting job: saveAsTextFile at Action1.scala:44
16/03/04 10:50:46 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 143 bytes
16/03/04 10:50:46 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 143 bytes
16/03/04 10:50:46 INFO DAGScheduler: Got job 6 (saveAsTextFile at Action1.scala:44) with 1 output partitions
16/03/04 10:50:46 INFO DAGScheduler: Final stage: ResultStage 11(saveAsTextFile at Action1.scala:44)
16/03/04 10:50:46 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 10)
16/03/04 10:50:46 INFO DAGScheduler: Missing parents: List()
16/03/04 10:50:46 INFO DAGScheduler: Submitting ResultStage 11 (MapPartitionsRDD[14] at saveAsTextFile at Action1.scala:44), which has no missing parents
16/03/04 10:50:46 INFO MemoryStore: ensureFreeSpace(128432) called with curMem=2028, maxMem=766075207
16/03/04 10:50:46 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 125.4 KB, free 730.5 MB)
16/03/04 10:50:46 INFO MemoryStore: ensureFreeSpace(43159) called with curMem=130460, maxMem=766075207
16/03/04 10:50:46 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 42.1 KB, free 730.4 MB)
16/03/04 10:50:46 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:62819 (size: 42.1 KB, free: 730.5 MB)
16/03/04 10:50:46 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:861
16/03/04 10:50:46 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 11 (MapPartitionsRDD[14] at saveAsTextFile at Action1.scala:44)
16/03/04 10:50:46 INFO TaskSchedulerImpl: Adding task set 11.0 with 1 tasks
16/03/04 10:50:46 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 8, localhost, PROCESS_LOCAL, 1901 bytes)
16/03/04 10:50:46 INFO Executor: Running task 0.0 in stage 11.0 (TID 8)
16/03/04 10:50:46 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/03/04 10:50:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/03/04 10:50:56 INFO FileOutputCommitter: Saved output of task 'attempt_201603041050_0011_m_000000_8' to hdfs://219.219.220.149:9000/output/20160304105019961/_temporary/0/task_201603041050_0011_m_000000
16/03/04 10:50:56 INFO SparkHadoopMapRedUtil: attempt_201603041050_0011_m_000000_8: Committed
16/03/04 10:50:56 INFO Executor: Finished task 0.0 in stage 11.0 (TID 8). 2080 bytes result sent to driver
16/03/04 10:50:56 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 8) in 9642 ms on localhost (1/1)
16/03/04 10:50:56 INFO TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 
16/03/04 10:50:56 INFO DAGScheduler: ResultStage 11 (saveAsTextFile at Action1.scala:44) finished in 9.643 s
16/03/04 10:50:56 INFO DAGScheduler: Job 6 finished: saveAsTextFile at Action1.scala:44, took 9.742452 s
16/03/04 10:50:56 INFO SparkUI: Stopped Spark web UI at http://219.219.220.162:4040
16/03/04 10:50:56 INFO DAGScheduler: Stopping DAGScheduler
16/03/04 10:50:56 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/03/04 10:50:57 INFO MemoryStore: MemoryStore cleared
16/03/04 10:50:57 INFO BlockManager: BlockManager stopped
16/03/04 10:50:57 INFO BlockManagerMaster: BlockManagerMaster stopped
16/03/04 10:50:57 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/03/04 10:50:57 INFO SparkContext: Successfully stopped SparkContext
16/03/04 10:50:57 INFO ShutdownHookManager: Shutdown hook called
16/03/04 10:50:57 INFO ShutdownHookManager: Deleting directory C:\Users\xubo\AppData\Local\Temp\spark-52f1804d-6099-46a6-b52e-473d64044637
16/03/04 10:50:57 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/03/04 10:50:57 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.

Process finished with exit code 0

你可能感兴趣的:(scala,spark,idea)

git常用命令笔记咩酱-小羊 git 笔记
###用习惯了idea总是不记得git的一些常见命令，需要用到的时候总是担心旁边站了人~~~记个笔记@_@，告诉自己看笔记不丢人初始化初始化一个新的Git仓库gitinit配置配置用户信息gitconfig--globaluser.name"YourName"gitconfig--globaluser.email"[email protected]"基本操作克隆远程仓库gitclone查看
nosql数据库技术与应用知识点皆过客，揽星河 NoSQL nosql 数据库大数据数据分析数据结构非关系型数据库
Nosql知识回顾大数据处理流程数据采集(flume、爬虫、传感器)数据存储(本门课程NoSQL所处的阶段)Hdfs、MongoDB、HBase等数据清洗(入仓)Hive等数据处理、分析(Spark、Flink等)数据可视化数据挖掘、机器学习应用(Python、SparkMLlib等)大数据时代存储的挑战(三高)高并发(同一时间很多人访问)高扩展(要求随时根据需求扩展存储)高效率(要求读写速度快)
spring如何整合druid连接池？惜.己 spring spring junit 数据库 java idea 后端 xml
目录spring整合druid连接池1.新建maven项目2.新建mavenModule3.导入相关依赖4.配置log4j2.xml5.配置druid.xml1)xml中如何引入properties2)下面是配置文件6.准备jdbc.propertiesJDBC配置项解释7.配置druid8.测试spring整合druid连接池1.新建maven项目打开IDE（比如IntelliJIDEA,Ecl
【算法练习】IDEA集成leetcode插件实现快速刷 2401_84102892 2024年程序员学习算法 intellij-idea leetcode
============点击右侧边leetcode->设置->配置地址、用户名、密码、存放目录、文件模板用户名要登录后在账号信息里看模板代码1.codefilename!velocityTool.camelC
idea使用自定义checkstyle.xml配置文件 Gemkey
1.下载插件image.png2.插件安装完后,找到设置中的checkstyle,点击"+",新增自定义规则image.png3.输入描述信息,点击Browse找到对应的文件image.pngimage.png4.可以把active勾上,则使用默认校验规则,点击OK,则可以开始使用自定义规则检测单个文件了image.png
101个浪漫的点子..哈哈有需要可以试试...中英对照~ Hecks 学习心得 IDEA UP Go 音乐网页游戏
Thisisafreebonusversionof101RomanticIdeas.Feelfreetoforwardtoormakecopiesforyourfriends.下面是101个浪漫的点子。可随意转发给你的朋友们IDEA#1点子1Ifyourpartnerisgoingawayforafewdays,tellherthatyouareworriedabouthersoyouhaveor
1.8，69 知行思合一
运气动力学万维钢老师提出的名词，不愧是物理博士出身，good1idea，创意一些新名词。开文引用查理芒格的话：得到你想要的东西，最保险的办法，就是让自己配得上你想要的那个东西。接着举例《绝命毒师》Breakingbad导演文斯·吉里根接受采访说的话，引出成功的经验——运气。我们现在所处的这个社会不管是美国还是中国，大体上都是一个精英社会——人们可以靠天赋和努力去争取财富和地位，而不像历史上那样家庭
svg图片兼容性和用法优缺点独行侠_ef93
svg图片的使用方法第一次来认认真真的研究了下svg图片，之前只是在网上见过，但都是一晃而过也没当回事，最近网站改版看到同事有用到svg格式的图片，想想自己干了几年的重构也没用过，这些细节的知识是应该好好研究研究了。暂时还没研究得完全透切，先记下目前为止所看到的吧不然又给忘了。svg可缩放矢量图形（ScalableVectorGraphics），顾名思义就是任意改变其大小也不会变形，是基于可扩展标
Kafka详细解析与应用分析芊言芊语 kafka 分布式
Kafka是一个开源的分布式事件流平台（EventStreamingPlatform），由LinkedIn公司最初采用Scala语言开发，并基于ZooKeeper协调管理。如今，Kafka已经被Apache基金会纳入其项目体系，广泛应用于大数据实时处理领域。Kafka凭借其高吞吐量、持久化、分布式和可靠性的特点，成为构建实时流数据管道和流处理应用程序的重要工具。Kafka架构Kafka的架构主要由
分享一个基于python的电子书数据采集与可视化分析 hadoop电子书数据分析与推荐系统 spark大数据毕设项目（源码、调试、LW、开题、PPT) 计算机源码社 Python项目大数据大数据 python hadoop 计算机毕业设计选题计算机毕业设计源码数据分析 spark毕设
作者：计算机源码社个人简介：本人八年开发经验，擅长Java、Python、PHP、.NET、Node.js、Android、微信小程序、爬虫、大数据、机器学习等，大家有这一块的问题可以一起交流！学习资料、程序开发、技术解答、文档报告如需要源码，可以扫取文章下方二维码联系咨询Java项目微信小程序项目Android项目Python项目PHP项目ASP.NET项目Node.js项目选题推荐项目实战|p
git 恢复误删的远程分支 Hanfank
需求实际工作上，肯定会有不小心的时候，比如我吧，删除了本地dev分支，intellijidea提示是否将trackedbranch也删掉，结果一不小心将远程分支也删除了，接下来就是我的救赎之旅。查看reflog，找到最后一次commitidgitreflog--date=isoreflog是referencelog的意思，也就是引用log，记录HEAD在各个分支上的移动轨迹。选项--date=is
Spark 组件 GraphX、Streaming 叶域大数据 spark spark 大数据分布式
Spark组件GraphX、Streaming一、SparkGraphX1.1GraphX的主要概念1.2GraphX的核心操作1.3示例代码1.4GraphX的应用场景二、SparkStreaming2.1SparkStreaming的主要概念2.2示例代码2.3SparkStreaming的集成2.4SparkStreaming的应用场景SparkGraphX用于处理图和图并行计算。Graph
【免费】springboot项目申报管理系统|毕业设计|Javaweb项目计算机学姐来啦 springboot ssm java spring boot 课程设计后端毕设毕业设计 java-ee
收藏点赞不迷路关注作者有好处编号：springboot375springboot项目申报管理系统开发语言：Java数据库：MySQL技术：Spring+SpringMVC+MyBatis工具：IDEA/Ecilpse、Navicat、Maven1.万字文档展示(部分)2.系统图片展示第5章系统详细设计5.1管理员功能模块的实现5.1.1项目列表如图5.1显示的就是项目列表页面，此页面提供给管理员的
探索未来，大规模分布式深度强化学习——深入解析IMPALA架构汤萌妮Margaret
探索未来，大规模分布式深度强化学习——深入解析IMPALA架构scalable_agent项目地址:https://gitcode.com/gh_mirrors/sc/scalable_agent在当今的人工智能研究前沿，深度强化学习（DRL）因其在复杂任务中的卓越表现而备受瞩目。本文要介绍的是一个开源于GitHub的重量级项目：“ScalableDistributedDeep-RLwithImp
车载以太网之SOME/IP IT_码农车载以太网车载以太网 SOME/IP
整体介绍SOME/IP(全称为：Scalableservice-OrientedMiddlewarEoverIP)，是运行在车载以太网协议栈基础之上的中间件，或者也可以称为应用层软件。发展历程AUTOSAR4.0-完成宝马SOME/IP消息的初步集成；AUTOSAR4.1-支持SOME/IP-SD及其发布/订阅功能；AUTOSAR4.2-添加transformer用于序列化以及其他相关优化；AUT
大数据毕业设计hadoop+spark+hive知识图谱租房数据分析可视化大屏租房推荐系统 58同城租房爬虫房源推荐系统房价预测系统计算机毕业设计机器学习深度学习人工智能 2401_84572577 程序员大数据 hadoop 人工智能
做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。我先来介绍一下这些东西怎么用，文末抱走。（1）Python所有方向的学习路线（
python基于django/flask的NBA球员大数据分析与可视化python+java+node.js QQ_511008285 python django flask java spring boot 数据分析
前端开发框架:vue.js数据库mysql版本不限后端语言框架支持：1java(SSM/springboot)-idea/eclipse2.Nodejs+Vue.js-vscode3.python(flask/django)--pycharm/vscode4.php(thinkphp/laravel)-hbuilderx数据库工具：Navicat/SQLyog等都可以本文针对NBA球员的大数据进行
Java基于spring boot的国产电影数据分析与可视化python+java+node.js QQ_511008285 java spring boot 数据分析 python django vue.js flask
前端开发框架:vue.js数据库mysql版本不限后端语言框架支持：1java(SSM/springboot)-idea/eclipse2.Nodejs+Vue.js-vscode3.python(flask/django)--pycharm/vscode4.php(thinkphp/laravel)-hbuilderx数据库工具：Navicat/SQLyog等都可以该系统使用进行大数据处理和
idea右侧工具栏找不到maven窗口足球数据分析 intellij-idea
idea没有识别到是maven项目处理方案：右键pom.xml文件,点击"addasmavenproject"
Spark集群的三种模式 MelodyYN #Spark spark hadoop big data
文章目录1、Spark的由来1.1Hadoop的发展1.2MapReduce与Spark对比2、Spark内置模块3、Spark运行模式3.1Standalone模式部署配置历史服务器配置高可用运行模式3.2Yarn模式安装部署配置历史服务器运行模式4、WordCount案例1、Spark的由来定义：Hadoop主要解决，海量数据的存储和海量数据的分析计算。Spark是一种基于内存的快速、通用、可
[附源码]SSM计算机毕业设计游戏账号交易平台JAVA 计算机程序源码 java 游戏 mysql
项目运行环境配置：Jdk1.8+Tomcat7.0+Mysql+HBuilderX（Webstorm也行）+Eclispe（IntelliJIDEA,Eclispe,MyEclispe,Sts都支持）。项目技术：SSM+mybatis+Maven+Vue等等组成，B/S模式+Maven管理等等。环境需要1.运行环境：最好是javajdk1.8，我们在这个平台上运行的。其他版本理论上也可以。2.ID
Java中的大数据处理框架对比分析省赚客app开发者 java 开发语言
Java中的大数据处理框架对比分析大家好，我是微赚淘客系统3.0的小编，是个冬天不穿秋裤，天冷也要风度的程序猿！今天，我们将深入探讨Java中常用的大数据处理框架，并对它们进行对比分析。大数据处理框架是现代数据驱动应用的核心，它们帮助企业处理和分析海量数据，以提取有价值的信息。本文将重点介绍ApacheHadoop、ApacheSpark、ApacheFlink和ApacheStorm这四种流行的
Java 入门基础篇05 - Java的关键字仔仔 v1.0 Java基础 java 开发语言 intellij-idea
什么是关键字？就是被java语言赋予特殊含义的单词。关键字的特点组成关键的字母都是小写。常见关键字class,public,static,void.....。关键字注意事项goto和const是java语言的保留字，关键字在IDEA编译器中有明确的颜色变化。关键字列表ABSTRACTCONTINUEFORNEWSWITCHassertdefaultgotopackagesynchronizedbo
idea自动下载gradle_解决 IDEA 导入 Gradle 项目默认下载 Gradle.zip 失败 weixin_39730671 idea自动下载gradle
这几天打算看看Spring-Framework的源码，于是将Spring-Framework的源码从GitHub拉到了Gitee，然后从Gitee上面下载到本地，几经波折终于下载到本地了。欣喜若狂的打开IDEA，Open了这个刚下载好的源码项目，结果一打开项目，它便自己开始下载起了gradle-6.5.1-bin.zip，下载了老半天都没啥动静，最后还直接给超时失败了。本来以前也不这样啊，我记得之
【idea实用系列7】Gradle构建项目中IDEA插件 BigDataMLApplication gradle idea intellij-idea java ide
【idea实用系列7】Gradle构建项目中IDEA插件文章目录【idea实用系列7】Gradle构建项目中IDEA插件使用任务配置自定义生成的文件合并禁用合并并进行完全覆盖钩入生成生命周期部分重写现有内容修改完全填充的领域对象修改XML表示形式识别附加的测试目录其他事项参考链接使用要使用IDEA插件，请在构建脚本中包含以下内容：plugins{id'idea'}IDEA插件会为项目添加一些任务。
Python+Django毕业设计校园易购二手交易平台（程序+LW+部署） Python、JAVA毕设程序源码课程设计 java mysql
项目运行环境配置：Jdk1.8+Tomcat7.0+Mysql+HBuilderX（Webstorm也行）+Eclispe（IntelliJIDEA,Eclispe,MyEclispe,Sts都支持）。项目技术：SSM+mybatis+Maven+Vue等等组成，B/S模式+Maven管理等等。环境需要1.运行环境：最好是javajdk1.8，我们在这个平台上运行的。其他版本理论上也可以。2.ID
【Gradle】（一）在IDEA中初始化gradle项目及其核心概念解释挥之以墨 #Gradle DevOps java gradle
这里写目录标题1.前言2.项目初始化及核心概念2.1.在IDEA中创建gradle项目2.2.gradle-wrapper2.2.1.gradle-wrapper的作用2.2.2.gradle-wrapper的初始化与配置2.3.setting.gradle2.4.build.gradle2.4.1.plugins2.4.2.版本定义2.4.3.仓库配置2.4.4.依赖管理3.总结1.前言本系列主
写出渗透测试信息收集详细流程卿酌南烛_b805
一、扫描域名漏洞：域名漏洞扫描工具有AWVS、APPSCAN、Netspark、WebInspect、Nmap、Nessus、天镜、明鉴、WVSS、RSAS等。二、子域名探测：1、dns域传送漏洞2、搜索引擎查找（通过Google、bing、搜索c段）3、通过ssl证书查询网站：https://myssl.com/ssl.html和https://www.chinassl.net/ssltools
IDEA测试类启动报 “java: 常量字符串过长” 解决办法无休居士 Java工程常见异常 java intellij-idea ide
目录标题问题描述问题分析解决办法其他办法问题描述问题分析字符串长度过长，导致idea默认使用的javac编译器编译不了。查询资料发现，原因是javac在编译期间，常量字符串最大长度为65534。解决办法Javac编译器改为Eclipse编译器。File->Settings->Build,Execution,Deployment->Compiler->JavaCompiler其他办法如果不是新建pr
Hadoop windows intelij 跑 MR WordCount piziyang12138
一、软件环境我使用的软件版本如下:IntellijIdea2017.1Maven3.3.9Hadoop分布式环境二、创建maven工程打开Idea,file->new->Project,左侧面板选择maven工程。(如果只跑MapReduce创建java工程即可，不用勾选Creatfromarchetype，如果想创建web工程或者使用骨架可以勾选)image.png设置GroupId和Artif
ztree设置禁用节点 3213213333332132 JavaScript ztree json setDisabledNode Ajax
ztree设置禁用节点的时候注意，当使用ajax后台请求数据,必须要设置为同步获取数据，否者会获取不到节点对象，导致设置禁用没有效果。 $(function(){ showTree(); setDisabledNode(); });
JVM patch by Taobao bookjovi java HotSpot
在网上无意中看到淘宝提交的hotspot patch，共四个，有意思，记录一下。 7050685：jsdbproc64.sh has a typo in the package name 7058036：FieldsAllocationStyle=2 does not work in 32-bit VM 7060619：C1 should respect inline and
将session存储到数据库中 dcj3sjt126com sql PHP session
CREATE TABLE sessions ( id CHAR(32) NOT NULL, data TEXT, last_accessed TIMESTAMP NOT NULL, PRIMARY KEY (id) ); <?php /** * Created by PhpStorm. * User: michaeldu * Date
Vector 171815164 vector
public Vector<CartProduct> delCart(Vector<CartProduct> cart, String id) { for (int i = 0; i < cart.size(); i++) { if (cart.get(i).getId().equals(id)) { cart.remove(i);
各连接池配置参数比较 g21121 连接池
排版真心费劲，大家凑合看下吧，见谅~ Druid DBCP C3P0 Proxool 数据库用户名称 Username Username User 数据库密码 Password Password Password 驱动名
[简单]mybatis insert语句添加动态字段 53873039oycg mybatis
mysql数据库,id自增,配置如下： <insert id="saveTestTb" useGeneratedKeys="true" keyProperty="id" parameterType=&
struts2拦截器配置云端月影 struts2拦截器
struts2拦截器interceptor的三种配置方法方法1. 普通配置法 <struts> <package name="struts2" extends="struts-default"> &
IE中页面不居中，火狐谷歌等正常 aijuans IE中页面不居中
问题是首页在火狐、谷歌、所有IE中正常显示，列表页的页面在火狐谷歌中正常，在IE6、7、8中都不中，觉得可能那个地方设置的让IE系列都不认识，仔细查看后发现，列表页中没写HTML模板部分没有添加DTD定义，就是<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3
String,int,Integer,char 几个类型常见转换 antonyup_2006 html sql .net
如何将字串 String 转换成整数 int? int i = Integer.valueOf(my_str).intValue(); int i=Integer.parseInt(str); 如何将字串 String 转换成Integer ? Integer integer=Integer.valueOf(str); 如何将整数 int 转换成字串 String ? 1.
PL/SQL的游标类型百合不是茶显示游标(静态游标)隐式游标游标的更新和删除 %rowtype ref游标(动态游标)
游标是oracle中的一个结果集,用于存放查询的结果; PL/SQL中游标的声明; 1,声明游标 2,打开游标(默认是关闭的); 3,提取数据 4,关闭游标注意的要点:游标必须声明在declare中,使用open打开游标,fetch取游标中的数据,close关闭游标隐式游标:主要是对DML数据的操作隐
JUnit4中@AfterClass @BeforeClass @after @before的区别对比 bijian1013 JUnit4 单元测试
一.基础知识 JUnit4使用Java5中的注解（annotation），以下是JUnit4常用的几个annotation： @Before：初始化方法对于每一个测试方法都要执行一次（注意与BeforeClass区别，后者是对于所有方法执行一次）@After：释放资源对于每一个测试方法都要执行一次（注意与AfterClass区别，后者是对于所有方法执行一次
精通Oracle10编程SQL(12)开发包 bijian1013 oracle 数据库 plsql
/* *开发包 *包用于逻辑组合相关的PL/SQL类型（例如TABLE类型和RECORD类型）、PL/SQL项（例如游标和游标变量）和PL/SQL子程序（例如过程和函数） */ --包用于逻辑组合相关的PL/SQL类型、项和子程序，它由包规范和包体两部分组成 --建立包规范：包规范实际是包与应用程序之间的接口，它用于定义包的公用组件，包括常量、变量、游标、过程和函数等 --在包规
【EhCache二】ehcache.xml配置详解 bit1129 ehcache.xml
在ehcache官网上找了多次，终于找到ehcache.xml配置元素和属性的含义说明文档了，这个文档包含在ehcache.xml的注释中！ ehcache.xml ： http://ehcache.org/ehcache.xml ehcache.xsd ： http://ehcache.org/ehcache.xsd ehcache配置文件的根元素是ehcahe ehcac
java.lang.ClassNotFoundException: org.springframework.web.context.ContextLoaderL 白糖_ java eclipse spring tomcat Web
今天学习spring+cxf的时候遇到一个问题：在web.xml中配置了spring的上下文监听器： <listener> <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class> </listener> 随后启动
angular.element boyitech AngularJS AngularJS API angular.element
angular.element 描述: 包裹着一部分DOM element或者是HTML字符串，把它作为一个jQuery元素来处理。（类似于jQuery的选择器啦）如果jQuery被引入了，则angular.element就可以看作是jQuery选择器，选择的对象可以使用jQuery的函数；如果jQuery不可用，angular.e
java-给定两个已排序序列，找出共同的元素。 bylijinnan java
import java.util.ArrayList; import java.util.Arrays; import java.util.List; public class CommonItemInTwoSortedArray { /** * 题目：给定两个已排序序列，找出共同的元素。 * 1.定义两个指针分别指向序列的开始。 * 如果指向的两个元素
sftp 异常，有遇到的吗？求解 Chen.H java jcraft auth jsch jschexception
com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:460) at com.jcraft.jsch.Session.connect(Session.java:154) at cn.vivame.util.ftp.SftpServerAccess.connec
[生物智能与人工智能]神经元中的电化学结构代表什么? comsci 人工智能
我这里做一个大胆的猜想,生物神经网络中的神经元中包含着一些化学和类似电路的结构,这些结构通常用来扮演类似我们在拓扑分析系统中的节点嵌入方程一样,使得我们的神经网络产生智能判断的能力,而这些嵌入到节点中的方程同时也扮演着"经验"的角色.... 我们可以尝试一下...在某些神经
通过LAC和CID获取经纬度信息 dai_lm lac cid
方法1：用浏览器打开http://www.minigps.net/cellsearch.html，然后输入lac和cid信息(mcc和mnc可以填0)，如果数据正确就可以获得相应的经纬度方法2：发送HTTP请求到http://www.open-electronics.org/celltrack/cell.php?hex=0&lac=<lac>&cid=&
JAVA的困难分析 datamachine java
前段时间转了一篇SQL的文章（http://datamachine.iteye.com/blog/1971896），文章不复杂，但思想深刻，就顺便思考了一下java的不足，当砖头丢出来，希望引点和田玉。 -----------------------------------------------------------------------------------------
小学5年级英语单词背诵第二课 dcj3sjt126com english word
money 钱 paper 纸 speak 讲，说 tell 告诉 remember 记得，想起 knock 敲，击，打 question 问题 number 数字，号码 learn 学会，学习 street 街道 carry 搬运，携带 send 发送，邮寄，发射 must 必须 light 灯，光线，轻的 front
linux下面没有tree命令 dcj3sjt126com linux
centos p安装 yum -y install tree mac os安装 brew install tree 首先来看tree的用法 tree 中文解释：tree 功能说明：以树状图列出目录的内容。语　　法：tree [-aACdDfFgilnNpqstux][-I <范本样式>][-P <范本样式
Map迭代方式，Map迭代，Map循环蕃薯耀 Map循环 Map迭代 Map迭代方式
Map迭代方式，Map迭代，Map循环 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 蕃薯耀 2015年
Spring Cache注解+Redis hanqunfeng spring
Spring3.1 Cache注解依赖jar包：  <dependency> <groupId>org.springframework.data</groupId> <artifactId>spring-data-redis</artifactId>
Guava中针对集合的 filter和过滤功能 jackyrong filter
在guava库中，自带了过滤器(filter)的功能，可以用来对collection 进行过滤，先看例子： @Test public void whenFilterWithIterables_thenFiltered() { List<String> names = Lists.newArrayList("John"
学习编程那点事 lampcy 编程 android PHP html5
一年前的夏天，我还在纠结要不要改行，要不要去学php？能学到真本事吗？改行能成功吗？太多的问题，我终于不顾一切，下定决心，辞去了工作，来到传说中的帝都。老师给的乘车方式还算有效，很顺利的就到了学校，赶巧了，正好学校搬到了新校区。先安顿了下来，过了个轻松的周末，第一次到帝都，逛逛吧！接下来的周一，是我噩梦的开始，学习内容对我这个零基础的人来说，除了勉强完成老师布置的作业外，我已经没有时间和精力去
架构师之流处理---------bytebuffer的mark,limit和flip nannan408 ByteBuffer
1.前言。如题，limit其实就是可以读取的字节长度的意思，flip是清空的意思，mark是标记的意思。 2.例子. 例子代码: String str = "helloWorld"; ByteBuffer buff = ByteBuffer.wrap(str.getBytes()); Sy
org.apache.el.parser.ParseException: Encountered " ":" ": "" at line 1, column 1 Everyday都不同 $转义 el表达式
最近在做Highcharts的过程中，在写js时，出现了以下异常：严重: Servlet.service() for servlet jsp threw exception org.apache.el.parser.ParseException: Encountered " ":" ": "" at line 1,
用Java实现发送邮件到163 tntxia java实现
/* 在java版经常看到有人问如何用javamail发送邮件？如何接收邮件？如何访问多个文件夹等。问题零散，而历史的回复早已经淹没在问题的海洋之中。本人之前所做过一个java项目，其中包含有WebMail功能，当初为用java实现而对javamail摸索了一段时间，总算有点收获。看到论坛中的经常有此方面的问题，因此把我的一些经验帖出来，希望对大家有些帮助。此篇仅介绍用
探索实体类存在的真正意义 java小叶檀 POJO
一. 实体类简述实体类其实就是俗称的POJO,这种类一般不实现特殊框架下的接口，在程序中仅作为数据容器用来持久化存储数据用的 POJO（Plain Old Java Objects）简单的Java对象它的一般格式就是 public class A{ private String id; public Str