【解决】WARN YarnClusterSch…

参考 办法http://blog.sina.com.cn/s/blog_4b1452dd0102wyzo.html解决

解决 WARN YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

后正常的日志：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/hadoop/hadoopTmp/nm-local-dir/filecache/148/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

17/02/26 23:59:50 INFO ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]

17/02/26 23:59:51 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1488181659296_0004_000001

17/02/26 23:59:52 INFO SecurityManager: Changing view acls to: hadoop

17/02/26 23:59:52 INFO SecurityManager: Changing modify acls to: hadoop

17/02/26 23:59:52 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)

17/02/26 23:59:52 INFO ApplicationMaster: Starting the user application in a separate Thread

17/02/26 23:59:52 INFO ApplicationMaster: Waiting for spark context initialization

17/02/26 23:59:52 INFO ApplicationMaster: Waiting for spark context initialization ... 

17/02/26 23:59:53 INFO SparkContext: Running Spark version 1.6.1

17/02/26 23:59:53 INFO SecurityManager: Changing view acls to: hadoop

17/02/26 23:59:53 INFO SecurityManager: Changing modify acls to: hadoop

17/02/26 23:59:53 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)

17/02/26 23:59:53 INFO Utils: Successfully started service 'sparkDriver' on port 46344.

17/02/26 23:59:53 INFO Slf4jLogger: Slf4jLogger started

17/02/26 23:59:53 INFO Remoting: Starting remoting

17/02/26 23:59:53 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:46211]

17/02/26 23:59:53 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 46211.

17/02/26 23:59:53 INFO SparkEnv: Registering MapOutputTracker

17/02/26 23:59:53 INFO SparkEnv: Registering BlockManagerMaster

17/02/26 23:59:53 INFO DiskBlockManager: Created local directory at /home/hadoop/hadoopTmp/nm-local-dir/usercache/hadoop/appcache/application_1488181659296_0004/blockmgr-93703b9e-2d79-4494-a02d-7dfdb349da89

17/02/26 23:59:53 INFO MemoryStore: MemoryStore started with capacity 457.9 MB

17/02/26 23:59:53 INFO SparkEnv: Registering OutputCommitCoordinator

17/02/26 23:59:54 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter

17/02/26 23:59:54 INFO Utils: Successfully started service 'SparkUI' on port 35252.

17/02/26 23:59:54 INFO SparkUI: Started SparkUI at http://192.168.136.135:35252

17/02/26 23:59:54 INFO YarnClusterScheduler: Created YarnClusterScheduler

17/02/26 23:59:54 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43620.

17/02/26 23:59:54 INFO NettyBlockTransferService: Server created on 43620

17/02/26 23:59:54 INFO BlockManagerMaster: Trying to register BlockManager

17/02/26 23:59:54 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.136.135:43620 with 457.9 MB RAM, BlockManagerId(driver, 192.168.136.135, 43620)

17/02/26 23:59:54 INFO BlockManagerMaster: Registered BlockManager

17/02/26 23:59:54 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://[email protected]:46344)

17/02/26 23:59:54 INFO RMProxy: Connecting to ResourceManager at bigmaster/192.168.136.134:8030

17/02/26 23:59:54 INFO YarnRMClient: Registering the ApplicationMaster

17/02/26 23:59:55 INFO YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead

17/02/26 23:59:55 INFO YarnAllocator: Container request (host: Any, capability: )

17/02/26 23:59:55 INFO YarnAllocator: Container request (host: Any, capability: )

17/02/26 23:59:55 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals

17/02/26 23:59:55 INFO AMRMClientImpl: Received new token for : big1:38108

17/02/26 23:59:55 INFO YarnAllocator: Launching container container_1488181659296_0004_01_000002 for on host big1

17/02/26 23:59:55 INFO YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://[email protected]:46344,  executorHostname: big1

17/02/26 23:59:55 INFO ExecutorRunnable: Starting Executor Container

17/02/26 23:59:55 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.

17/02/26 23:59:55 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0

17/02/26 23:59:55 INFO ExecutorRunnable: Setting up ContainerLaunchContext

17/02/26 23:59:55 INFO ExecutorRunnable: Preparing Local resources

17/02/26 23:59:55 INFO ExecutorRunnable: Prepared Local resources Map(asm-3.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/asm-3.1.jar" } size: 43033 timestamp: 1487901101364 type: FILE visibility: PUBLIC, hive-cli-1.2.1.spark.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/hive-cli-1.2.1.spark.jar" } size: 40812 timestamp: 1487901102736 type: FILE visibility: PUBLIC, commons-lang3-3.3.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-lang3-3.3.2.jar" } size: 412739 timestamp: 1487901102129 type: FILE visibility: PUBLIC, spark-streaming-kafka_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-kafka_2.10-1.6.1.jar" } size: 284924 timestamp: 1487901112022 type: FILE visibility: PUBLIC, parquet-format-2.3.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-format-2.3.0-incubating.jar" } size: 387188 timestamp: 1487901104900 type: FILE visibility: PUBLIC, tachyon-client-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-client-0.8.2.jar" } size: 2291167 timestamp: 1487901112419 type: FILE visibility: PUBLIC, jackson-module-scala_2.10-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-module-scala_2.10-2.4.4.jar" } size: 549415 timestamp: 1487901103060 type: FILE visibility: PUBLIC, metrics-core-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-core-3.1.2.jar" } size: 112558 timestamp: 1487901104209 type: FILE visibility: PUBLIC, slf4j-log4j12-1.6.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/slf4j-log4j12-1.6.6.jar" } size: 9711 timestamp: 1487901109149 type: FILE visibility: PUBLIC, pmml-agent-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-agent-1.1.15.jar" } size: 5278 timestamp: 1487901105265 type: FILE visibility: PUBLIC, spark-mllib_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-mllib_2.10-1.6.1.jar" } size: 4991327 timestamp: 1487901111550 type: FILE visibility: PUBLIC, __spark__.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-yarn_2.10-1.6.1.jar" } size: 561734 timestamp: 1487901112191 type: FILE visibility: PUBLIC, flume-ng-core-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/flume-ng-core-1.6.0.jar" } size: 370107 timestamp: 1487901102569 type: FILE visibility: PUBLIC, zookeeper-3.4.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/zookeeper-3.4.6.jar" } size: 792964 timestamp: 1487901112691 type: FILE visibility: PUBLIC, oozie-sharelib-oozie-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.3.0.jar" } size: 54976 timestamp: 1487901096210 type: FILE visibility: PUBLIC, json4s-jackson_2.10-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/json4s-jackson_2.10-3.2.10.jar" } size: 39953 timestamp: 1487901103616 type: FILE visibility: PUBLIC, metrics-graphite-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-graphite-3.1.2.jar" } size: 20852 timestamp: 1487901104241 type: FILE visibility: PUBLIC, parquet-common-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-common-1.7.0.jar" } size: 21575 timestamp: 1487901104860 type: FILE visibility: PUBLIC, commons-lang-2.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-lang-2.4.jar" } size: 261809 timestamp: 1487901102081 type: FILE visibility: PUBLIC, avro-mapred-1.7.7-hadoop2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-mapred-1.7.7-hadoop2.jar" } size: 180736 timestamp: 1487901101452 type: FILE visibility: PUBLIC, spark-streaming-flume-sink_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-flume-sink_2.10-1.6.1.jar" } size: 86038 timestamp: 1487901111931 type: FILE visibility: PUBLIC, spark-launcher_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-launcher_2.10-1.6.1.jar" } size: 65251 timestamp: 1487901111451 type: FILE visibility: PUBLIC, RoaringBitmap-0.5.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/RoaringBitmap-0.5.11.jar" } size: 201928 timestamp: 1487901101180 type: FILE visibility: PUBLIC, ivy-2.4.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/ivy-2.4.0.jar" } size: 1282424 timestamp: 1487901102859 type: FILE visibility: PUBLIC, spark-streaming_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming_2.10-1.6.1.jar" } size: 2052495 timestamp: 1487901112129 type: FILE visibility: PUBLIC, kryo-2.22.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kryo-2.22.jar" } size: 442805 timestamp: 1487901103943 type: FILE visibility: PUBLIC, pyspark.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/hadoop/.sparkStaging/application_1488181659296_0004/pyspark.zip" } size: 283560 timestamp: 1488182389071 type: FILE visibility: PRIVATE, spark-core_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-core_2.10-1.6.1.jar" } size: 11179076 timestamp: 1487901111264 type: FILE visibility: PUBLIC, commons-cli-1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-cli-1.2.jar" } size: 41123 timestamp: 1487901101937 type: FILE visibility: PUBLIC, chill_2.10-0.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/chill_2.10-0.5.0.jar" } size: 221032 timestamp: 1487901101915 type: FILE visibility: PUBLIC, curator-recipes-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-recipes-2.5.0.jar" } size: 244782 timestamp: 1487901102457 type: FILE visibility: PUBLIC, parquet-encoding-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-encoding-1.7.0.jar" } size: 285447 timestamp: 1487901104882 type: FILE visibility: PUBLIC, datanucleus-core-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/datanucleus-core-3.2.10.jar" } size: 1890075 timestamp: 1487901102504 type: FILE visibility: PUBLIC, chill-java-0.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/chill-java-0.5.0.jar" } size: 46715 timestamp: 1487901101855 type: FILE visibility: PUBLIC, parquet-hadoop-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-hadoop-1.7.0.jar" } size: 209622 timestamp: 1487901105134 type: FILE visibility: PUBLIC, commons-dbcp-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-dbcp-1.4.jar" } size: 160519 timestamp: 1487901102014 type: FILE visibility: PUBLIC, janino-2.7.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/janino-2.7.8.jar" } size: 613299 timestamp: 1487901103083 type: FILE visibility: PUBLIC, py4j-0.9-src.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/hadoop/.sparkStaging/application_1488181659296_0004/py4j-0.9-src.zip" } size: 44846 timestamp: 1488182389117 type: FILE visibility: PRIVATE, jackson-mapper-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-mapper-asl-1.9.13.jar" } size: 780664 timestamp: 1487901103036 type: FILE visibility: PUBLIC, log4j-1.2.17.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/log4j-1.2.17.jar" } size: 489884 timestamp: 1487901104102 type: FILE visibility: PUBLIC, py4j.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/py4j.zip" } size: 37562 timestamp: 1487901106810 type: FILE visibility: PUBLIC, jtransforms-2.4.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jtransforms-2.4.0.jar" } size: 764569 timestamp: 1487901103680 type: FILE visibility: PUBLIC, javax.servlet-3.0.0.v201112011016.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/javax.servlet-3.0.0.v201112011016.jar" } size: 200387 timestamp: 1487901103143 type: FILE visibility: PUBLIC, arpack_combined_all-0.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/arpack_combined_all-0.1.jar" } size: 1194003 timestamp: 1487901101335 type: FILE visibility: PUBLIC, jets3t-0.7.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jets3t-0.7.1.jar" } size: 377780 timestamp: 1487901103357 type: FILE visibility: PUBLIC, slf4j-api-1.6.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/slf4j-api-1.6.6.jar" } size: 26176 timestamp: 1487901109117 type: FILE visibility: PUBLIC, breeze_2.10-0.11.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/breeze_2.10-0.11.2.jar" } size: 13689583 timestamp: 1487901101691 type: FILE visibility: PUBLIC, paranamer-2.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/paranamer-2.6.jar" } size: 32806 timestamp: 1487901104802 type: FILE visibility: PUBLIC, parquet-hadoop-bundle-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-hadoop-bundle-1.6.0.jar" } size: 2796935 timestamp: 1487901105193 type: FILE visibility: PUBLIC, pmml-model-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-model-1.1.15.jar" } size: 656969 timestamp: 1487901105320 type: FILE visibility: PUBLIC, parquet-generator-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-generator-1.7.0.jar" } size: 21243 timestamp: 1487901105087 type: FILE visibility: PUBLIC, py4j-0.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/py4j-0.9.jar" } size: 84451 timestamp: 1487901105427 type: FILE visibility: PUBLIC, akka-actor_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-actor_2.10-2.3.11.jar" } size: 2658152 timestamp: 1487901101241 type: FILE visibility: PUBLIC, curator-client-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-client-2.5.0.jar" } size: 68692 timestamp: 1487901102398 type: FILE visibility: PUBLIC, lz4-1.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/lz4-1.3.0.jar" } size: 236880 timestamp: 1487901104134 type: FILE visibility: PUBLIC, calcite-core-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-core-1.2.0-incubating.jar" } size: 3519262 timestamp: 1487901101798 type: FILE visibility: PUBLIC, spark1.py -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/examples/apps/pythonApp/lib/spark1.py" } size: 747 timestamp: 1488182369956 type: FILE visibility: PUBLIC, xz-1.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/xz-1.0.jar" } size: 94672 timestamp: 1487901112637 type: FILE visibility: PUBLIC, jersey-core-1.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jersey-core-1.9.jar" } size: 458739 timestamp: 1487901103294 type: FILE visibility: PUBLIC, snappy-java-1.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/snappy-java-1.1.2.jar" } size: 704708 timestamp: 1487901109212 type: FILE visibility: PUBLIC, calcite-linq4j-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-linq4j-1.2.0-incubating.jar" } size: 442406 timestamp: 1487901101825 type: FILE visibility: PUBLIC, spark-sql_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-sql_2.10-1.6.1.jar" } size: 4031634 timestamp: 1487901111866 type: FILE visibility: PUBLIC, leveldbjni-all-1.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/leveldbjni-all-1.8.jar" } size: 1045744 timestamp: 1487901104014 type: FILE visibility: PUBLIC, jsr305-1.3.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jsr305-1.3.9.jar" } size: 33015 timestamp: 1487901103636 type: FILE visibility: PUBLIC, zkclient-0.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/zkclient-0.3.jar" } size: 64009 timestamp: 1487901112662 type: FILE visibility: PUBLIC, oro-2.0.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/oro-2.0.8.jar" } size: 65261 timestamp: 1487901104787 type: FILE visibility: PUBLIC, spire_2.10-0.7.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spire_2.10-0.7.4.jar" } size: 7253596 timestamp: 1487901112341 type: FILE visibility: PUBLIC, eigenbase-properties-1.1.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/eigenbase-properties-1.1.5.jar" } size: 18482 timestamp: 1487901102525 type: FILE visibility: PUBLIC, commons-codec-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-codec-1.4.jar" } size: 58160 timestamp: 1487901101951 type: FILE visibility: PUBLIC, compress-lzf-1.0.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/compress-lzf-1.0.3.jar" } size: 79845 timestamp: 1487901102289 type: FILE visibility: PUBLIC, oozie-hadoop-utils-hadoop-2-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/oozie-hadoop-utils-hadoop-2-4.3.0.jar" } size: 11851 timestamp: 1487901096191 type: FILE visibility: PUBLIC, opencsv-2.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/opencsv-2.3.jar" } size: 19827 timestamp: 1487901104675 type: FILE visibility: PUBLIC, netty-all-4.0.29.Final.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/netty-all-4.0.29.Final.jar" } size: 2054931 timestamp: 1487901104589 type: FILE visibility: PUBLIC, jetty-util-6.1.26.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jetty-util-6.1.26.jar" } size: 177131 timestamp: 1487901103412 type: FILE visibility: PUBLIC, parquet-jackson-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-jackson-1.7.0.jar" } size: 1048110 timestamp: 1487901105228 type: FILE visibility: PUBLIC, kafka-clients-0.8.2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kafka-clients-0.8.2.1.jar" } size: 324010 timestamp: 1487901103773 type: FILE visibility: PUBLIC, jackson-annotations-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-annotations-2.4.4.jar" } size: 38597 timestamp: 1487901102878 type: FILE visibility: PUBLIC, akka-slf4j_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-slf4j_2.10-2.3.11.jar" } size: 15525 timestamp: 1487901101307 type: FILE visibility: PUBLIC, spark-network-shuffle_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-network-shuffle_2.10-1.6.1.jar" } size: 54995 timestamp: 1487901111636 type: FILE visibility: PUBLIC, jersey-server-1.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jersey-server-1.9.jar" } size: 713089 timestamp: 1487901103320 type: FILE visibility: PUBLIC, spark-graphx_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-graphx_2.10-1.6.1.jar" } size: 653914 timestamp: 1487901111361 type: FILE visibility: PUBLIC, quasiquotes_2.10-2.0.0-M8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/quasiquotes_2.10-2.0.0-M8.jar" } size: 721002 timestamp: 1487901107873 type: FILE visibility: PUBLIC, jackson-core-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-core-2.4.4.jar" } size: 225302 timestamp: 1487901102917 type: FILE visibility: PUBLIC, guava-14.0.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/guava-14.0.1.jar" } size: 2189117 timestamp: 1487901102679 type: FILE visibility: PUBLIC, calcite-avatica-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-avatica-1.2.0-incubating.jar" } size: 258370 timestamp: 1487901101715 type: FILE visibility: PUBLIC, jcl-over-slf4j-1.7.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jcl-over-slf4j-1.7.10.jar" } size: 16617 timestamp: 1487901103255 type: FILE visibility: PUBLIC, scala-compiler-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scala-compiler-2.10.5.jar" } size: 14472629 timestamp: 1487901108551 type: FILE visibility: PUBLIC, spark-streaming-flume_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-flume_2.10-1.6.1.jar" } size: 103377 timestamp: 1487901111992 type: FILE visibility: PUBLIC, metrics-core-2.2.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-core-2.2.0.jar" } size: 82123 timestamp: 1487901104191 type: FILE visibility: PUBLIC, jline-0.9.94.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jline-0.9.94.jar" } size: 87325 timestamp: 1487901103443 type: FILE visibility: PUBLIC, scala-library-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scala-library-2.10.5.jar" } size: 7130772 timestamp: 1487901108857 type: FILE visibility: PUBLIC, jodd-core-3.5.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jodd-core-3.5.2.jar" } size: 427780 timestamp: 1487901103523 type: FILE visibility: PUBLIC, config-1.2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/config-1.2.1.jar" } size: 219554 timestamp: 1487901102333 type: FILE visibility: PUBLIC, xbean-asm5-shaded-4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/xbean-asm5-shaded-4.4.jar" } size: 144660 timestamp: 1487901112620 type: FILE visibility: PUBLIC, oozie-sharelib-spark-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/oozie-sharelib-spark-4.3.0.jar" } size: 22351 timestamp: 1487901104611 type: FILE visibility: PUBLIC, joda-time-2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/joda-time-2.1.jar" } size: 570478 timestamp: 1487901103492 type: FILE visibility: PUBLIC, commons-io-2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-io-2.1.jar" } size: 163151 timestamp: 1487901102059 type: FILE visibility: PUBLIC, json-simple-1.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/json-simple-1.1.jar" } size: 16046 timestamp: 1487901096176 type: FILE visibility: PUBLIC, mesos-0.21.1-shaded-protobuf.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/mesos-0.21.1-shaded-protobuf.jar" } size: 1277883 timestamp: 1487901104167 type: FILE visibility: PUBLIC, jetty-6.1.14.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jetty-6.1.14.jar" } size: 516429 timestamp: 1487901103382 type: FILE visibility: PUBLIC, tachyon-underfs-s3-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-underfs-s3-0.8.2.jar" } size: 505388 timestamp: 1487901112537 type: FILE visibility: PUBLIC, spark-hive_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-hive_2.10-1.6.1.jar" } size: 1315602 timestamp: 1487901111425 type: FILE visibility: PUBLIC, commons-logging-1.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-logging-1.1.jar" } size: 52915 timestamp: 1487901102156 type: FILE visibility: PUBLIC, avro-1.7.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-1.7.7.jar" } size: 436303 timestamp: 1487901101385 type: FILE visibility: PUBLIC, avro-ipc-1.7.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-ipc-1.7.7.jar" } size: 192993 timestamp: 1487901101428 type: FILE visibility: PUBLIC, libthrift-0.9.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/libthrift-0.9.2.jar" } size: 227712 timestamp: 1487901104075 type: FILE visibility: PUBLIC, jackson-core-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-core-asl-1.9.13.jar" } size: 232248 timestamp: 1487901102944 type: FILE visibility: PUBLIC, kafka_2.10-0.8.2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kafka_2.10-0.8.2.1.jar" } size: 3991269 timestamp: 1487901103857 type: FILE visibility: PUBLIC, tachyon-underfs-local-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-underfs-local-0.8.2.jar" } size: 7212 timestamp: 1487901112478 type: FILE visibility: PUBLIC, json4s-ast_2.10-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/json4s-ast_2.10-3.2.10.jar" } size: 83798 timestamp: 1487901103559 type: FILE visibility: PUBLIC, jackson-databind-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-databind-2.4.4.jar" } size: 1076926 timestamp: 1487901102989 type: FILE visibility: PUBLIC, stream-2.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/stream-2.7.0.jar" } size: 174351 timestamp: 1487901112365 type: FILE visibility: PUBLIC, pmml-schema-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-schema-1.1.15.jar" } size: 4560 timestamp: 1487901105354 type: FILE visibility: PUBLIC, spark-catalyst_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-catalyst_2.10-1.6.1.jar" } size: 5251737 timestamp: 1487901109618 type: FILE visibility: PUBLIC, flume-ng-sdk-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/flume-ng-sdk-1.6.0.jar" } size: 120996 timestamp: 1487901102605 type: FILE visibility: PUBLIC, netty-3.8.0.Final.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/netty-3.8.0.Final.jar" } size: 1230201 timestamp: 1487901104515 type: FILE visibility: PUBLIC, commons-compress-1.4.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-compress-1.4.1.jar" } size: 241367 timestamp: 1487901101996 type: FILE visibility: PUBLIC, spark-bagel_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-bagel_2.10-1.6.1.jar" } size: 45033 timestamp: 1487901109239 type: FILE visibility: PUBLIC, jul-to-slf4j-1.7.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jul-to-slf4j-1.7.10.jar" } size: 4725 timestamp: 1487901103728 type: FILE visibility: PUBLIC, commons-httpclient-3.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-httpclient-3.1.jar" } size: 305001 timestamp: 1487901102036 type: FILE visibility: PUBLIC, pyrolite-4.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pyrolite-4.9.jar" } size: 93407 timestamp: 1487901107750 type: FILE visibility: PUBLIC, jansi-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jansi-1.4.jar" } size: 106659 timestamp: 1487901103120 type: FILE visibility: PUBLIC, spark-network-common_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-network-common_2.10-1.6.1.jar" } size: 2352127 timestamp: 1487901111610 type: FILE visibility: PUBLIC, jaxb-core-2.2.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jaxb-core-2.2.7.jar" } size: 221747 timestamp: 1487901103191 type: FILE visibility: PUBLIC, spire-macros_2.10-0.7.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spire-macros_2.10-0.7.4.jar" } size: 79162 timestamp: 1487901112216 type: FILE visibility: PUBLIC, breeze-macros_2.10-0.11.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/breeze-macros_2.10-0.11.2.jar" } size: 118301 timestamp: 1487901101472 type: FILE visibility: PUBLIC, json4s-core_2.10-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/json4s-core_2.10-3.2.10.jar" } size: 584691 timestamp: 1487901103591 type: FILE visibility: PUBLIC, akka-remote_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-remote_2.10-2.3.11.jar" } size: 1352129 timestamp: 1487901101273 type: FILE visibility: PUBLIC, commons-pool-1.5.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-pool-1.5.4.jar" } size: 96221 timestamp: 1487901102268 type: FILE visibility: PUBLIC, commons-math3-3.4.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-math3-3.4.1.jar" } size: 2035066 timestamp: 1487901102223 type: FILE visibility: PUBLIC, commons-net-2.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-net-2.2.jar" } size: 212453 timestamp: 1487901102251 type: FILE visibility: PUBLIC, curator-framework-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-framework-2.5.0.jar" } size: 181215 timestamp: 1487901102421 type: FILE visibility: PUBLIC, servlet-api-2.5-6.1.14.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/servlet-api-2.5-6.1.14.jar" } size: 132368 timestamp: 1487901109008 type: FILE visibility: PUBLIC, httpclient-4.3.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/httpclient-4.3.6.jar" } size: 592008 timestamp: 1487901102794 type: FILE visibility: PUBLIC, tachyon-underfs-hdfs-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-underfs-hdfs-0.8.2.jar" } size: 11079 timestamp: 1487901112450 type: FILE visibility: PUBLIC, spark-unsafe_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-unsafe_2.10-1.6.1.jar" } size: 38627 timestamp: 1487901112163 type: FILE visibility: PUBLIC, unused-1.0.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/unused-1.0.0.jar" } size: 2777 timestamp: 1487901112602 type: FILE visibility: PUBLIC, scalap-2.10.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scalap-2.10.0.jar" } size: 855012 timestamp: 1487901108974 type: FILE visibility: PUBLIC, metrics-json-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-json-3.1.2.jar" } size: 15827 timestamp: 1487901104264 type: FILE visibility: PUBLIC, avro-ipc-1.7.7-tests.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-ipc-1.7.7-tests.jar" } size: 346580 timestamp: 1487901101410 type: FILE visibility: PUBLIC, jaxb-impl-2.2.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jaxb-impl-2.2.7.jar" } size: 919968 timestamp: 1487901103229 type: FILE visibility: PUBLIC, mina-core-2.0.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/mina-core-2.0.4.jar" } size: 644934 timestamp: 1487901104374 type: FILE visibility: PUBLIC, parquet-column-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-column-1.7.0.jar" } size: 917052 timestamp: 1487901104837 type: FILE visibility: PUBLIC, jline-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jline-2.10.5.jar" } size: 164623 timestamp: 1487901103462 type: FILE visibility: PUBLIC, scala-reflect-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scala-reflect-2.10.5.jar" } size: 3206179 timestamp: 1487901108931 type: FILE visibility: PUBLIC, uncommons-maths-1.2.2a.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/uncommons-maths-1.2.2a.jar" } size: 49019 timestamp: 1487901112556 type: FILE visibility: PUBLIC, metrics-jvm-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-jvm-3.1.2.jar" } size: 39280 timestamp: 1487901104279 type: FILE visibility: PUBLIC, core-1.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/core-1.1.2.jar" } size: 164422 timestamp: 1487901102350 type: FILE visibility: PUBLIC, flume-ng-configuration-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/flume-ng-configuration-1.6.0.jar" } size: 56525 timestamp: 1487901102549 type: FILE visibility: PUBLIC, httpcore-4.3.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/httpcore-4.3.3.jar" } size: 282793 timestamp: 1487901102821 type: FILE visibility: PUBLIC, libfb303-0.9.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/libfb303-0.9.2.jar" } size: 313686 timestamp: 1487901104048 type: FILE visibility: PUBLIC, commons-compiler-2.7.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-compiler-2.7.6.jar" } size: 30595 timestamp: 1487901101964 type: FILE visibility: PUBLIC, jaxb-api-2.2.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jaxb-api-2.2.2.jar" } size: 105134 timestamp: 1487901103171 type: FILE visibility: PUBLIC, gson-2.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/gson-2.7.jar" } size: 231952 timestamp: 1487901102630 type: FILE visibility: PUBLIC, spark-repl_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-repl_2.10-1.6.1.jar" } size: 678034 timestamp: 1487901111731 type: FILE visibility: PUBLIC, protobuf-java-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/protobuf-java-2.5.0.jar" } size: 533455 timestamp: 1487901105387 type: FILE visibility: PUBLIC)

17/02/26 23:59:56 INFO ExecutorRunnable: 

===============================================================================

YARN executor launch context:

  env:

    CLASSPATH -> $PWD/*{{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*

    SPARK_LOG_URL_STDERR -> http://big1:8042/node/containerlogs/container_1488181659296_0004_01_000002/hadoop/stderr?start=-4096

    SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1488181659296_0004

    SPARK_YARN_CACHE_FILES_FILE_SIZES -> 561734,44846,283560,747,64009,46715,181215,305001,120996,86038,7253596,285447,225302,38597,2189117,18482,83798,2658152,704708,163151,212453,79162,112558,221032,118301,164422,200387,180736,377780,219554,282793,39280,1352129,2796935,1315602,1277883,261809,570478,2291167,764569,387188,458739,14472629,2777,505388,3519262,283560,79845,1194003,21575,54995,780664,20852,4560,549415,65261,3991269,5251737,68692,82123,13689583,33015,96221,94672,1048110,370107,45033,49019,678034,4725,236880,644934,241367,584691,16617,1045744,43033,313686,2054931,792964,713089,232248,174351,1890075,201928,592008,32806,258370,5278,2352127,93407,132368,87325,412739,56525,533455,15525,26176,227712,21243,22351,656969,2035066,164623,40812,1282424,613299,4031634,231952,106659,30595,221747,160519,1230201,41123,284924,144660,442406,919968,3206179,7212,11179076,4991327,37562,39953,15827,19827,84451,346580,721002,855012,103377,7130772,105134,324010,442805,9711,427780,58160,1076926,516429,244782,489884,653914,436303,917052,2052495,38627,65251,52915,177131,11079,209622,192993,16046,11851,54976,283560,44846

    SPARK_USER -> hadoop

    SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PRIVATE,PRIVATE

    SPARK_YARN_MODE -> true

    SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1487901112191,1488182369912,1488182369939,1488182369956,1487901112662,1487901101855,1487901102421,1487901102036,1487901102605,1487901111931,1487901112341,1487901104882,1487901102917,1487901102878,1487901102679,1487901102525,1487901103559,1487901101241,1487901109212,1487901102059,1487901102251,1487901112216,1487901104209,1487901101915,1487901101472,1487901102350,1487901103143,1487901101452,1487901103357,1487901102333,1487901102821,1487901104279,1487901101273,1487901105193,1487901111425,1487901104167,1487901102081,1487901103492,1487901112419,1487901103680,1487901104900,1487901103294,1487901108551,1487901112602,1487901112537,1487901101798,1487901107813,1487901102289,1487901101335,1487901104860,1487901111636,1487901103036,1487901104241,1487901105354,1487901103060,1487901104787,1487901103857,1487901109618,1487901102398,1487901104191,1487901101691,1487901103636,1487901102268,1487901112637,1487901105228,1487901102569,1487901109239,1487901112556,1487901111731,1487901103728,1487901104134,1487901104374,1487901101996,1487901103591,1487901103255,1487901104014,1487901101364,1487901104048,1487901104589,1487901112691,1487901103320,1487901102944,1487901112365,1487901102504,1487901101180,1487901102794,1487901104802,1487901101715,1487901105265,1487901111610,1487901107750,1487901109008,1487901103443,1487901102129,1487901102549,1487901105387,1487901101307,1487901109117,1487901104075,1487901105087,1487901104611,1487901105320,1487901102223,1487901103462,1487901102736,1487901102859,1487901103083,1487901111866,1487901102630,1487901103120,1487901101964,1487901103191,1487901102014,1487901104515,1487901101937,1487901112022,1487901112620,1487901101825,1487901103229,1487901108931,1487901112478,1487901111264,1487901111550,1487901106810,1487901103616,1487901104264,1487901104675,1487901105427,1487901101410,1487901107873,1487901108974,1487901111992,1487901108857,1487901103171,1487901103773,1487901103943,1487901109149,1487901103523,1487901101951,1487901102989,1487901103382,1487901102457,1487901104102,1487901111361,1487901101385,1487901104837,1487901112129,1487901112163,1487901111451,1487901102156,1487901103412,1487901112450,1487901105134,1487901101428,1487901096176,1487901096191,1487901096210,1488182389071,1488182389117

    PYTHONPATH -> {{PWD}}/pyspark.zip{{PWD}}/py4j-0.9-src.zip

    SPARK_LOG_URL_STDOUT -> http://big1:8042/node/containerlogs/container_1488181659296_0004_01_000002/hadoop/stdout?start=-4096

    SPARK_YARN_CACHE_FILES -> hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-yarn_2.10-1.6.1.jar#__spark__.jar,hdfs://bigmaster:9000/user/oozie/examples/apps/pythonApp/lib/py4j-0.9-src.zip#py4j-0.9-src.zip,hdfs://bigmaster:9000/user/oozie/examples/apps/pythonApp/lib/pyspark.zip#pyspark.zip,hdfs://bigmaster:9000/user/oozie/examples/apps/pythonApp/lib/spark1.py#spark1.py,hdfs://bigmaster:9000/user/oozie/share/lib/spark/zkclient-0.3.jar#zkclient-0.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/chill-java-0.5.0.jar#chill-java-0.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/curator-framework-2.5.0.jar#curator-framework-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-httpclient-3.1.jar#commons-httpclient-3.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/flume-ng-sdk-1.6.0.jar#flume-ng-sdk-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming-flume-sink_2.10-1.6.1.jar#spark-streaming-flume-sink_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spire_2.10-0.7.4.jar#spire_2.10-0.7.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-encoding-1.7.0.jar#parquet-encoding-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-core-2.4.4.jar#jackson-core-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-annotations-2.4.4.jar#jackson-annotations-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/guava-14.0.1.jar#guava-14.0.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/eigenbase-properties-1.1.5.jar#eigenbase-properties-1.1.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/json4s-ast_2.10-3.2.10.jar#json4s-ast_2.10-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/akka-actor_2.10-2.3.11.jar#akka-actor_2.10-2.3.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/snappy-java-1.1.2.jar#snappy-java-1.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-io-2.1.jar#commons-io-2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-net-2.2.jar#commons-net-2.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spire-macros_2.10-0.7.4.jar#spire-macros_2.10-0.7.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-core-3.1.2.jar#metrics-core-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/chill_2.10-0.5.0.jar#chill_2.10-0.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/breeze-macros_2.10-0.11.2.jar#breeze-macros_2.10-0.11.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/core-1.1.2.jar#core-1.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/javax.servlet-3.0.0.v201112011016.jar#javax.servlet-3.0.0.v201112011016.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-mapred-1.7.7-hadoop2.jar#avro-mapred-1.7.7-hadoop2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jets3t-0.7.1.jar#jets3t-0.7.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/config-1.2.1.jar#config-1.2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/httpcore-4.3.3.jar#httpcore-4.3.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-jvm-3.1.2.jar#metrics-jvm-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/akka-remote_2.10-2.3.11.jar#akka-remote_2.10-2.3.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-hadoop-bundle-1.6.0.jar#parquet-hadoop-bundle-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-hive_2.10-1.6.1.jar#spark-hive_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/mesos-0.21.1-shaded-protobuf.jar#mesos-0.21.1-shaded-protobuf.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-lang-2.4.jar#commons-lang-2.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/joda-time-2.1.jar#joda-time-2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-client-0.8.2.jar#tachyon-client-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jtransforms-2.4.0.jar#jtransforms-2.4.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-format-2.3.0-incubating.jar#parquet-format-2.3.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jersey-core-1.9.jar#jersey-core-1.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scala-compiler-2.10.5.jar#scala-compiler-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/unused-1.0.0.jar#unused-1.0.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-underfs-s3-0.8.2.jar#tachyon-underfs-s3-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/calcite-core-1.2.0-incubating.jar#calcite-core-1.2.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pyspark.zip#pyspark.zip,hdfs://bigmaster:9000/user/oozie/share/lib/spark/compress-lzf-1.0.3.jar#compress-lzf-1.0.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/arpack_combined_all-0.1.jar#arpack_combined_all-0.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-common-1.7.0.jar#parquet-common-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-network-shuffle_2.10-1.6.1.jar#spark-network-shuffle_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-mapper-asl-1.9.13.jar#jackson-mapper-asl-1.9.13.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-graphite-3.1.2.jar#metrics-graphite-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pmml-schema-1.1.15.jar#pmml-schema-1.1.15.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-module-scala_2.10-2.4.4.jar#jackson-module-scala_2.10-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/oro-2.0.8.jar#oro-2.0.8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/kafka_2.10-0.8.2.1.jar#kafka_2.10-0.8.2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-catalyst_2.10-1.6.1.jar#spark-catalyst_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/curator-client-2.5.0.jar#curator-client-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-core-2.2.0.jar#metrics-core-2.2.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/breeze_2.10-0.11.2.jar#breeze_2.10-0.11.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jsr305-1.3.9.jar#jsr305-1.3.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-pool-1.5.4.jar#commons-pool-1.5.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/xz-1.0.jar#xz-1.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-jackson-1.7.0.jar#parquet-jackson-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/flume-ng-core-1.6.0.jar#flume-ng-core-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-bagel_2.10-1.6.1.jar#spark-bagel_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/uncommons-maths-1.2.2a.jar#uncommons-maths-1.2.2a.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-repl_2.10-1.6.1.jar#spark-repl_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jul-to-slf4j-1.7.10.jar#jul-to-slf4j-1.7.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/lz4-1.3.0.jar#lz4-1.3.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/mina-core-2.0.4.jar#mina-core-2.0.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-compress-1.4.1.jar#commons-compress-1.4.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/json4s-core_2.10-3.2.10.jar#json4s-core_2.10-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jcl-over-slf4j-1.7.10.jar#jcl-over-slf4j-1.7.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/leveldbjni-all-1.8.jar#leveldbjni-all-1.8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/asm-3.1.jar#asm-3.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/libfb303-0.9.2.jar#libfb303-0.9.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/netty-all-4.0.29.Final.jar#netty-all-4.0.29.Final.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/zookeeper-3.4.6.jar#zookeeper-3.4.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jersey-server-1.9.jar#jersey-server-1.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-core-asl-1.9.13.jar#jackson-core-asl-1.9.13.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/stream-2.7.0.jar#stream-2.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/datanucleus-core-3.2.10.jar#datanucleus-core-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/RoaringBitmap-0.5.11.jar#RoaringBitmap-0.5.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/httpclient-4.3.6.jar#httpclient-4.3.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/paranamer-2.6.jar#paranamer-2.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/calcite-avatica-1.2.0-incubating.jar#calcite-avatica-1.2.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pmml-agent-1.1.15.jar#pmml-agent-1.1.15.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-network-common_2.10-1.6.1.jar#spark-network-common_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pyrolite-4.9.jar#pyrolite-4.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/servlet-api-2.5-6.1.14.jar#servlet-api-2.5-6.1.14.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jline-0.9.94.jar#jline-0.9.94.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-lang3-3.3.2.jar#commons-lang3-3.3.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/flume-ng-configuration-1.6.0.jar#flume-ng-configuration-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/protobuf-java-2.5.0.jar#protobuf-java-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/akka-slf4j_2.10-2.3.11.jar#akka-slf4j_2.10-2.3.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/slf4j-api-1.6.6.jar#slf4j-api-1.6.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/libthrift-0.9.2.jar#libthrift-0.9.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-generator-1.7.0.jar#parquet-generator-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/oozie-sharelib-spark-4.3.0.jar#oozie-sharelib-spark-4.3.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pmml-model-1.1.15.jar#pmml-model-1.1.15.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-math3-3.4.1.jar#commons-math3-3.4.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jline-2.10.5.jar#jline-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/hive-cli-1.2.1.spark.jar#hive-cli-1.2.1.spark.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/ivy-2.4.0.jar#ivy-2.4.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/janino-2.7.8.jar#janino-2.7.8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-sql_2.10-1.6.1.jar#spark-sql_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/gson-2.7.jar#gson-2.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jansi-1.4.jar#jansi-1.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-compiler-2.7.6.jar#commons-compiler-2.7.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jaxb-core-2.2.7.jar#jaxb-core-2.2.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-dbcp-1.4.jar#commons-dbcp-1.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/netty-3.8.0.Final.jar#netty-3.8.0.Final.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-cli-1.2.jar#commons-cli-1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming-kafka_2.10-1.6.1.jar#spark-streaming-kafka_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/xbean-asm5-shaded-4.4.jar#xbean-asm5-shaded-4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/calcite-linq4j-1.2.0-incubating.jar#calcite-linq4j-1.2.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jaxb-impl-2.2.7.jar#jaxb-impl-2.2.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scala-reflect-2.10.5.jar#scala-reflect-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-underfs-local-0.8.2.jar#tachyon-underfs-local-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-core_2.10-1.6.1.jar#spark-core_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-mllib_2.10-1.6.1.jar#spark-mllib_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/py4j.zip#py4j.zip,hdfs://bigmaster:9000/user/oozie/share/lib/spark/json4s-jackson_2.10-3.2.10.jar#json4s-jackson_2.10-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-json-3.1.2.jar#metrics-json-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/opencsv-2.3.jar#opencsv-2.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/py4j-0.9.jar#py4j-0.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-ipc-1.7.7-tests.jar#avro-ipc-1.7.7-tests.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/quasiquotes_2.10-2.0.0-M8.jar#quasiquotes_2.10-2.0.0-M8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scalap-2.10.0.jar#scalap-2.10.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming-flume_2.10-1.6.1.jar#spark-streaming-flume_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scala-library-2.10.5.jar#scala-library-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jaxb-api-2.2.2.jar#jaxb-api-2.2.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/kafka-clients-0.8.2.1.jar#kafka-clients-0.8.2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/kryo-2.22.jar#kryo-2.22.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/slf4j-log4j12-1.6.6.jar#slf4j-log4j12-1.6.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jodd-core-3.5.2.jar#jodd-core-3.5.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-codec-1.4.jar#commons-codec-1.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-databind-2.4.4.jar#jackson-databind-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jetty-6.1.14.jar#jetty-6.1.14.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/curator-recipes-2.5.0.jar#curator-recipes-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/log4j-1.2.17.jar#log4j-1.2.17.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-graphx_2.10-1.6.1.jar#spark-graphx_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-1.7.7.jar#avro-1.7.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-column-1.7.0.jar#parquet-column-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming_2.10-1.6.1.jar#spark-streaming_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-unsafe_2.10-1.6.1.jar#spark-unsafe_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-launcher_2.10-1.6.1.jar#spark-launcher_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-logging-1.1.jar#commons-logging-1.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jetty-util-6.1.26.jar#jetty-util-6.1.26.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-underfs-hdfs-0.8.2.jar#tachyon-underfs-hdfs-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-hadoop-1.7.0.jar#parquet-hadoop-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-ipc-1.7.7.jar#avro-ipc-1.7.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/oozie/json-simple-1.1.jar#json-simple-1.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/oozie/oozie-hadoop-utils-hadoop-2-4.3.0.jar#oozie-hadoop-utils-hadoop-2-4.3.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.3.0.jar#oozie-sharelib-oozie-4.3.0.jar,hdfs://bigmaster:9000/user/hadoop/.sparkStaging/application_1488181659296_0004/pyspark.zip#pyspark.zip,hdfs://bigmaster:9000/user/hadoop/.sparkStaging/application_1488181659296_0004/py4j-0.9-src.zip#py4j-0.9-src.zip

  command:

    {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m '-Dlog4j.configuration=spark-log4j.properties' -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=46344' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir= org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://[email protected]:46344 --executor-id 1 --hostname big1 --cores 1 --app-id application_1488181659296_0004 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr

===============================================================================

      

17/02/26 23:59:56 INFO ContainerManagementProtocolProxy: Opening proxy : big1:38108

17/02/26 23:59:56 INFO AMRMClientImpl: Received new token for : big2:39242

17/02/26 23:59:56 INFO YarnAllocator: Launching container container_1488181659296_0004_01_000003 for on host big2

17/02/26 23:59:56 INFO YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://[email protected]:46344,  executorHostname: big2

17/02/26 23:59:56 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.

17/02/26 23:59:56 INFO ExecutorRunnable: Starting Executor Container

17/02/26 23:59:56 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0

17/02/26 23:59:56 INFO ExecutorRunnable: Setting up ContainerLaunchContext

17/02/26 23:59:56 INFO ExecutorRunnable: Preparing Local resources

17/02/26 23:59:56 INFO ExecutorRunnable: Prepared Local resources Map(asm-3.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/asm-3.1.jar" } size: 43033 timestamp: 1487901101364 type: FILE visibility: PUBLIC, hive-cli-1.2.1.spark.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/hive-cli-1.2.1.spark.jar" } size: 40812 timestamp: 1487901102736 type: FILE visibility: PUBLIC, commons-lang3-3.3.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-lang3-3.3.2.jar" } size: 412739 timestamp: 1487901102129 type: FILE visibility: PUBLIC, spark-streaming-kafka_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-kafka_2.10-1.6.1.jar" } size: 284924 timestamp: 1487901112022 type: FILE visibility: PUBLIC, parquet-format-2.3.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-format-2.3.0-incubating.jar" } size: 387188 timestamp: 1487901104900 type: FILE visibility: PUBLIC, tachyon-client-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-client-0.8.2.jar" } size: 2291167 timestamp: 1487901112419 type: FILE visibility: PUBLIC, jackson-module-scala_2.10-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-module-scala_2.10-2.4.4.jar" } size: 549415 timestamp: 1487901103060 type: FILE visibility: PUBLIC, metrics-core-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-core-3.1.2.jar" } size: 112558 timestamp: 1487901104209 type: FILE visibility: PUBLIC, slf4j-log4j12-1.6.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/slf4j-log4j12-1.6.6.jar" } size: 9711 timestamp: 1487901109149 type: FILE visibility: PUBLIC, pmml-agent-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-agent-1.1.15.jar" } size: 5278 timestamp: 1487901105265 type: FILE visibility: PUBLIC, spark-mllib_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-mllib_2.10-1.6.1.jar" } size: 4991327 timestamp: 1487901111550 type: FILE visibility: PUBLIC, __spark__.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-yarn_2.10-1.6.1.jar" } size: 561734 timestamp: 1487901112191 type: FILE visibility: PUBLIC, flume-ng-core-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/flume-ng-core-1.6.0.jar" } size: 370107 timestamp: 1487901102569 type: FILE visibility: PUBLIC, zookeeper-3.4.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/zookeeper-3.4.6.jar" } size: 792964 timestamp: 1487901112691 type: FILE visibility: PUBLIC, oozie-sharelib-oozie-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.3.0.jar" } size: 54976 timestamp: 1487901096210 type: FILE visibility: PUBLIC, json4s-jackson_2.10-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/json4s-jackson_2.10-3.2.10.jar" } size: 39953 timestamp: 1487901103616 type: FILE visibility: PUBLIC, metrics-graphite-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-graphite-3.1.2.jar" } size: 20852 timestamp: 1487901104241 type: FILE visibility: PUBLIC, parquet-common-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-common-1.7.0.jar" } size: 21575 timestamp: 1487901104860 type: FILE visibility: PUBLIC, commons-lang-2.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-lang-2.4.jar" } size: 261809 timestamp: 1487901102081 type: FILE visibility: PUBLIC, avro-mapred-1.7.7-hadoop2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-mapred-1.7.7-hadoop2.jar" } size: 180736 timestamp: 1487901101452 type: FILE visibility: PUBLIC, spark-streaming-flume-sink_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-flume-sink_2.10-1.6.1.jar" } size: 86038 timestamp: 1487901111931 type: FILE visibility: PUBLIC, spark-launcher_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-launcher_2.10-1.6.1.jar" } size: 65251 timestamp: 1487901111451 type: FILE visibility: PUBLIC, RoaringBitmap-0.5.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/RoaringBitmap-0.5.11.jar" } size: 201928 timestamp: 1487901101180 type: FILE visibility: PUBLIC, ivy-2.4.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/ivy-2.4.0.jar" } size: 1282424 timestamp: 1487901102859 type: FILE visibility: PUBLIC, spark-streaming_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming_2.10-1.6.1.jar" } size: 2052495 timestamp: 1487901112129 type: FILE visibility: PUBLIC, kryo-2.22.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kryo-2.22.jar" } size: 442805 timestamp: 1487901103943 type: FILE visibility: PUBLIC, pyspark.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/hadoop/.sparkStaging/application_1488181659296_0004/pyspark.zip" } size: 283560 timestamp: 1488182389071 type: FILE visibility: PRIVATE, spark-core_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-core_2.10-1.6.1.jar" } size: 11179076 timestamp: 1487901111264 type: FILE visibility: PUBLIC, commons-cli-1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-cli-1.2.jar" } size: 41123 timestamp: 1487901101937 type: FILE visibility: PUBLIC, chill_2.10-0.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/chill_2.10-0.5.0.jar" } size: 221032 timestamp: 1487901101915 type: FILE visibility: PUBLIC, curator-recipes-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-recipes-2.5.0.jar" } size: 244782 timestamp: 1487901102457 type: FILE visibility: PUBLIC, parquet-encoding-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-encoding-1.7.0.jar" } size: 285447 timestamp: 1487901104882 type: FILE visibility: PUBLIC, datanucleus-core-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/datanucleus-core-3.2.10.jar" } size: 1890075 timestamp: 1487901102504 type: FILE visibility: PUBLIC, chill-java-0.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/chill-java-0.5.0.jar" } size: 46715 timestamp: 1487901101855 type: FILE visibility: PUBLIC, parquet-hadoop-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-hadoop-1.7.0.jar" } size: 209622 timestamp: 1487901105134 type: FILE visibility: PUBLIC, commons-dbcp-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-dbcp-1.4.jar" } size: 160519 timestamp: 1487901102014 type: FILE visibility: PUBLIC, janino-2.7.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/janino-2.7.8.jar" } size: 613299 timestamp: 1487901103083 type: FILE visibility: PUBLIC, py4j-0.9-src.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/hadoop/.sparkStaging/application_1488181659296_0004/py4j-0.9-src.zip" } size: 44846 timestamp: 1488182389117 type: FILE visibility: PRIVATE, jackson-mapper-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-mapper-asl-1.9.13.jar" } size: 780664 timestamp: 1487901103036 type: FILE visibility: PUBLIC, log4j-1.2.17.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/log4j-1.2.17.jar" } size: 489884 timestamp: 1487901104102 type: FILE visibility: PUBLIC, py4j.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/py4j.zip" } size: 37562 timestamp: 1487901106810 type: FILE visibility: PUBLIC, jtransforms-2.4.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jtransforms-2.4.0.jar" } size: 764569 timestamp: 1487901103680 type: FILE visibility: PUBLIC, javax.servlet-3.0.0.v201112011016.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/javax.servlet-3.0.0.v201112011016.jar" } size: 200387 timestamp: 1487901103143 type: FILE visibility: PUBLIC, arpack_combined_all-0.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/arpack_combined_all-0.1.jar" } size: 1194003 timestamp: 1487901101335 type: FILE visibility: PUBLIC, jets3t-0.7.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jets3t-0.7.1.jar" } size: 377780 timestamp: 1487901103357 type: FILE visibility: PUBLIC, slf4j-api-1.6.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/slf4j-api-1.6.6.jar" } size: 26176 timestamp: 1487901109117 type: FILE visibility: PUBLIC, breeze_2.10-0.11.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/breeze_2.10-0.11.2.jar" } size: 13689583 timestamp: 1487901101691 type: FILE visibility: PUBLIC, paranamer-2.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/paranamer-2.6.jar" } size: 32806 timestamp: 1487901104802 type: FILE visibility: PUBLIC, parquet-hadoop-bundle-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-hadoop-bundle-1.6.0.jar" } size: 2796935 timestamp: 1487901105193 type: FILE visibility: PUBLIC, pmml-model-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-model-1.1.15.jar" } size: 656969 timestamp: 1487901105320 type: FILE visibility: PUBLIC, parquet-generator-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-generator-1.7.0.jar" } size: 21243 timestamp: 1487901105087 type: FILE visibility: PUBLIC, py4j-0.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/py4j-0.9.jar" } size: 84451 timestamp: 1487901105427 type: FILE visibility: PUBLIC, akka-actor_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-actor_2.10-2.3.11.jar" } size: 2658152 timestamp: 1487901101241 type: FILE visibility: PUBLIC, curator-client-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-client-2.5.0.jar" } size: 68692 timestamp: 1487901102398 type: FILE visibility: PUBLIC, lz4-1.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/lz4-1.3.0.jar" } size: 236880 timestamp: 1487901104134 type: FILE visibility: PUBLIC, calcite-core-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-core-1.2.0-incubating.jar" } size: 3519262 timestamp: 1487901101798 type: FILE visibility: PUBLIC, spark1.py -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/examples/apps/pythonApp/lib/spark1.py" } size: 747 timestamp: 1488182369956 type: FILE visibility: PUBLIC, xz-1.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/xz-1.0.jar" } size: 94672 timestamp: 1487901112637 type: FILE visibility: PUBLIC, jersey-core-1.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jersey-core-1.9.jar" } size: 458739 timestamp: 1487901103294 type: FILE visibility: PUBLIC, snappy-java-1.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/snappy-java-1.1.2.jar" } size: 704708 timestamp: 1487901109212 type: FILE visibility: PUBLIC, calcite-linq4j-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-linq4j-1.2.0-incubating.jar" } size: 442406 timestamp: 1487901101825 type: FILE visibility: PUBLIC, spark-sql_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-sql_2.10-1.6.1.jar" } size: 4031634 timestamp: 1487901111866 type: FILE visibility: PUBLIC, leveldbjni-all-1.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/leveldbjni-all-1.8.jar" } size: 1045744 timestamp: 1487901104014 type: FILE visibility: PUBLIC, jsr305-1.3.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jsr305-1.3.9.jar" } size: 33015 timestamp: 1487901103636 type: FILE visibility: PUBLIC, zkclient-0.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/zkclient-0.3.jar" } size: 64009 timestamp: 1487901112662 type: FILE visibility: PUBLIC, oro-2.0.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/oro-2.0.8.jar" } size: 65261 timestamp: 1487901104787 type: FILE visibility: PUBLIC, spire_2.10-0.7.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spire_2.10-0.7.4.jar" } size: 7253596 timestamp: 1487901112341 type: FILE visibility: PUBLIC, eigenbase-properties-1.1.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/eigenbase-properties-1.1.5.jar" } size: 18482 timestamp: 1487901102525 type: FILE visibility: PUBLIC, commons-codec-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-codec-1.4.jar" } size: 58160 timestamp: 1487901101951 type: FILE visibility: PUBLIC, compress-lzf-1.0.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/compress-lzf-1.0.3.jar" } size: 79845 timestamp: 1487901102289 type: FILE visibility: PUBLIC, oozie-hadoop-utils-hadoop-2-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/oozie-hadoop-utils-hadoop-2-4.3.0.jar" } size: 11851 timestamp: 1487901096191 type: FILE visibility: PUBLIC, opencsv-2.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/opencsv-2.3.jar" } size: 19827 timestamp: 1487901104675 type: FILE visibility: PUBLIC, netty-all-4.0.29.Final.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/netty-all-4.0.29.Final.jar" } size: 2054931 timestamp: 1487901104589 type: FILE visibility: PUBLIC, jetty-util-6.1.26.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jetty-util-6.1.26.jar" } size: 177131 timestamp: 1487901103412 type: FILE visibility: PUBLIC, parquet-jackson-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-jackson-1.7.0.jar" } size: 1048110 timestamp: 1487901105228 type: FILE visibility: PUBLIC, kafka-clients-0.8.2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kafka-clients-0.8.2.1.jar" } size: 324010 timestamp: 1487901103773 type: FILE visibility: PUBLIC, jackson-annotations-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-annotations-2.4.4.jar" } size: 38597 timestamp: 1487901102878 type: FILE visibility: PUBLIC, akka-slf4j_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-slf4j_2.10-2.3.11.jar" } size: 15525 timestamp: 1487901101307 type: FILE visibility: PUBLIC, spark-network-shuffle_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-network-shuffle_2.10-1.6.1.jar" } size: 54995 timestamp: 1487901111636 type: FILE visibility: PUBLIC, jersey-server-1.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jersey-server-1.9.jar" } size: 713089 timestamp: 1487901103320 type: FILE visibility: PUBLIC, spark-graphx_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-graphx_2.10-1.6.1.jar" } size: 653914 timestamp: 1487901111361 type: FILE visibility: PUBLIC, quasiquotes_2.10-2.0.0-M8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/quasiquotes_2.10-2.0.0-M8.jar" } size: 721002 timestamp: 1487901107873 type: FILE visibility: PUBLIC, jackson-core-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-core-2.4.4.jar" } size: 225302 timestamp: 1487901102917 type: FILE visibility: PUBLIC, guava-14.0.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/guava-14.0.1.jar" } size: 2189117 timestamp: 1487901102679 type: FILE visibility: PUBLIC, calcite-avatica-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-avatica-1.2.0-incubating.jar" } size: 258370 timestamp: 1487901101715 type: FILE visibility: PUBLIC, jcl-over-slf4j-1.7.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jcl-over-slf4j-1.7.10.jar" } size: 16617 timestamp: 1487901103255 type: FILE visibility: PUBLIC, scala-compiler-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scala-compiler-2.10.5.jar" } size: 14472629 timestamp: 1487901108551 type: FILE visibility: PUBLIC, spark-streaming-flume_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-flume_2.10-1.6.1.jar" } size: 103377 timestamp: 1487901111992 type: FILE visibility: PUBLIC, metrics-core-2.2.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-core-2.2.0.jar" } size: 82123 timestamp: 1487901104191 type: FILE visibility: PUBLIC, jline-0.9.94.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jline-0.9.94.jar" } size: 87325 timestamp: 1487901103443 type: FILE visibility: PUBLIC, scala-library-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scala-library-2.10.5.jar" } size: 7130772 timestamp: 1487901108857 type: FILE visibility: PUBLIC, jodd-core-3.5.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jodd-core-3.5.2.jar" } size: 427780 timestamp: 1487901103523 type: FILE visibility: PUBLIC, config-1.2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/config-1.2.1.jar" } size: 219554 timestamp: 1487901102333 type: FILE visibility: PUBLIC, xbean-asm5-shaded-4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/xbean-asm5-shaded-4.4.jar" } size: 144660 timestamp: 1487901112620 type: FILE visibility: PUBLIC, oozie-sharelib-spark-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/oozie-sharelib-spark-4.3.0.jar" } size: 22351 timestamp: 1487901104611 type: FILE visibility: PUBLIC, joda-time-2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/joda-time-2.1.jar" } size: 570478 timestamp: 1487901103492 type: FILE visibility: PUBLIC, commons-io-2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-io-2.1.jar" } size: 163151 timestamp: 1487901102059 type: FILE visibility: PUBLIC, json-simple-1.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/json-simple-1.1.jar" } size: 16046 timestamp: 1487901096176 type: FILE visibility: PUBLIC, mesos-0.21.1-shaded-protobuf.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/mesos-0.21.1-shaded-protobuf.jar" } size: 1277883 timestamp: 1487901104167 type: FILE visibility: PUBLIC, jetty-6.1.14.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jetty-6.1.14.jar" } size: 516429 timestamp: 1487901103382 type: FILE visibility: PUBLIC, tachyon-underfs-s3-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-underfs-s3-0.8.2.jar" } size: 505388 timestamp: 1487901112537 type: FILE visibility: PUBLIC, spark-hive_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-hive_2.10-1.6.1.jar" } size: 1315602 timestamp: 1487901111425 type: FILE visibility: PUBLIC, commons-logging-1.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-logging-1.1.jar" } size: 52915 timestamp: 1487901102156 type: FILE visibility: PUBLIC, avro-1.7.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-1.7.7.jar" } size: 436303 timestamp: 1487901101385 type: FILE visibility: PUBLIC, avro-ipc-1.7.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-ipc-1.7.7.jar" } size: 192993 timestamp: 1487901101428 type: FILE visibility: PUBLIC, libthrift-0.9.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/libthrift-0.9.2.jar" } size: 227712 timestamp: 1487901104075 type: FILE visibility: PUBLIC, jackson-core-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-core-asl-1.9.13.jar" } size: 232248 timestamp: 1487901102944 type: FILE visibility: PUBLIC, kafka_2.10-0.8.2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kafka_2.10-0.8.2.1.jar" } size: 3991269 timestamp: 1487901103857 type: FILE visibility: PUBLIC, tachyon-underfs-local-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-underfs-local-0.8.2.jar" } size: 7212 timestamp: 1487901112478 type: FILE visibility: PUBLIC, json4s-ast_2.10-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/json4s-ast_2.10-3.2.10.jar" } size: 83798 timestamp: 1487901103559 type: FILE visibility: PUBLIC, jackson-databind-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-databind-2.4.4.jar" } size: 1076926 timestamp: 1487901102989 type: FILE visibility: PUBLIC, stream-2.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/stream-2.7.0.jar" } size: 174351 timestamp: 1487901112365 type: FILE visibility: PUBLIC, pmml-schema-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-schema-1.1.15.jar" } size: 4560 timestamp: 1487901105354 type: FILE visibility: PUBLIC, spark-catalyst_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-catalyst_2.10-1.6.1.jar" } size: 5251737 timestamp: 1487901109618 type: FILE visibility: PUBLIC, flume-ng-sdk-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/flume-ng-sdk-1.6.0.jar" } size: 120996 timestamp: 1487901102605 type: FILE visibility: PUBLIC, netty-3.8.0.Final.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/netty-3.8.0.Final.jar" } size: 1230201 timestamp: 1487901104515 type: FILE visibility: PUBLIC, commons-compress-1.4.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-compress-1.4.1.jar" } size: 241367 timestamp: 1487901101996 type: FILE visibility: PUBLIC, spark-bagel_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-bagel_2.10-1.6.1.jar" } size: 45033 timestamp: 1487901109239 type: FILE visibility: PUBLIC, jul-to-slf4j-1.7.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jul-to-slf4j-1.7.10.jar" } size: 4725 timestamp: 1487901103728 type: FILE visibility: PUBLIC, commons-httpclient-3.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-httpclient-3.1.jar" } size: 305001 timestamp: 1487901102036 type: FILE visibility: PUBLIC, pyrolite-4.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pyrolite-4.9.jar" } size: 93407 timestamp: 1487901107750 type: FILE visibility: PUBLIC, jansi-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jansi-1.4.jar" } size: 106659 timestamp: 1487901103120 type: FILE visibility: PUBLIC, spark-network-common_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-network-common_2.10-1.6.1.jar" } size: 2352127 timestamp: 1487901111610 type: FILE visibility: PUBLIC, jaxb-core-2.2.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jaxb-core-2.2.7.jar" } size: 221747 timestamp: 1487901103191 type: FILE visibility: PUBLIC, spire-macros_2.10-0.7.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spire-macros_2.10-0.7.4.jar" } size: 79162 timestamp: 1487901112216 type: FILE visibility: PUBLIC, breeze-macros_2.10-0.11.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/breeze-macros_2.10-0.11.2.jar" } size: 118301 timestamp: 1487901101472 type: FILE visibility: PUBLIC, json4s-core_2.10-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/json4s-core_2.10-3.2.10.jar" } size: 584691 timestamp: 1487901103591 type: FILE visibility: PUBLIC, akka-remote_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-remote_2.10-2.3.11.jar" } size: 1352129 timestamp: 1487901101273 type: FILE visibility: PUBLIC, commons-pool-1.5.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-pool-1.5.4.jar" } size: 96221 timestamp: 1487901102268 type: FILE visibility: PUBLIC, commons-math3-3.4.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-math3-3.4.1.jar" } size: 2035066 timestamp: 1487901102223 type: FILE visibility: PUBLIC, commons-net-2.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-net-2.2.jar" } size: 212453 timestamp: 1487901102251 type: FILE visibility: PUBLIC, curator-framework-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-framework-2.5.0.jar" } size: 181215 timestamp: 1487901102421 type: FILE visibility: PUBLIC, servlet-api-2.5-6.1.14.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/servlet-api-2.5-6.1.14.jar" } size: 132368 timestamp: 1487901109008 type: FILE visibility: PUBLIC, httpclient-4.3.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/httpclient-4.3.6.jar" } size: 592008 timestamp: 1487901102794 type: FILE visibility: PUBLIC, tachyon-underfs-hdfs-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-underfs-hdfs-0.8.2.jar" } size: 11079 timestamp: 1487901112450 type: FILE visibility: PUBLIC, spark-unsafe_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-unsafe_2.10-1.6.1.jar" } size: 38627 timestamp: 1487901112163 type: FILE visibility: PUBLIC, unused-1.0.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/unused-1.0.0.jar" } size: 2777 timestamp: 1487901112602 type: FILE visibility: PUBLIC, scalap-2.10.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scalap-2.10.0.jar" } size: 855012 timestamp: 1487901108974 type: FILE visibility: PUBLIC, metrics-json-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-json-3.1.2.jar" } size: 15827 timestamp: 1487901104264 type: FILE visibility: PUBLIC, avro-ipc-1.7.7-tests.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-ipc-1.7.7-tests.jar" } size: 346580 timestamp: 1487901101410 type: FILE visibility: PUBLIC, jaxb-impl-2.2.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jaxb-impl-2.2.7.jar" } size: 919968 timestamp: 1487901103229 type: FILE visibility: PUBLIC, mina-core-2.0.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/mina-core-2.0.4.jar" } size: 644934 timestamp: 1487901104374 type: FILE visibility: PUBLIC, parquet-column-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-column-1.7.0.jar" } size: 917052 timestamp: 1487901104837 type: FILE visibility: PUBLIC, jline-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jline-2.10.5.jar" } size: 164623 timestamp: 1487901103462 type: FILE visibility: PUBLIC, scala-reflect-2.10.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/scala-reflect-2.10.5.jar" } size: 3206179 timestamp: 1487901108931 type: FILE visibility: PUBLIC, uncommons-maths-1.2.2a.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/uncommons-maths-1.2.2a.jar" } size: 49019 timestamp: 1487901112556 type: FILE visibility: PUBLIC, metrics-jvm-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-jvm-3.1.2.jar" } size: 39280 timestamp: 1487901104279 type: FILE visibility: PUBLIC, core-1.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/core-1.1.2.jar" } size: 164422 timestamp: 1487901102350 type: FILE visibility: PUBLIC, flume-ng-configuration-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/flume-ng-configuration-1.6.0.jar" } size: 56525 timestamp: 1487901102549 type: FILE visibility: PUBLIC, httpcore-4.3.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/httpcore-4.3.3.jar" } size: 282793 timestamp: 1487901102821 type: FILE visibility: PUBLIC, libfb303-0.9.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/libfb303-0.9.2.jar" } size: 313686 timestamp: 1487901104048 type: FILE visibility: PUBLIC, commons-compiler-2.7.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-compiler-2.7.6.jar" } size: 30595 timestamp: 1487901101964 type: FILE visibility: PUBLIC, jaxb-api-2.2.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jaxb-api-2.2.2.jar" } size: 105134 timestamp: 1487901103171 type: FILE visibility: PUBLIC, gson-2.7.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/gson-2.7.jar" } size: 231952 timestamp: 1487901102630 type: FILE visibility: PUBLIC, spark-repl_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-repl_2.10-1.6.1.jar" } size: 678034 timestamp: 1487901111731 type: FILE visibility: PUBLIC, protobuf-java-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/protobuf-java-2.5.0.jar" } size: 533455 timestamp: 1487901105387 type: FILE visibility: PUBLIC)

17/02/26 23:59:56 INFO ExecutorRunnable: 

===============================================================================

YARN executor launch context:

  env:

    CLASSPATH -> $PWD/*{{PWD}}{{PWD}}/__spark__.jar$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/*$HADOOP_COMMON_HOME/share/hadoop/common/lib/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/*$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*$HADOOP_YARN_HOME/share/hadoop/yarn/*$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*

    SPARK_LOG_URL_STDERR -> http://big2:8042/node/containerlogs/container_1488181659296_0004_01_000003/hadoop/stderr?start=-4096

    SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1488181659296_0004

    SPARK_YARN_CACHE_FILES_FILE_SIZES -> 561734,44846,283560,747,64009,46715,181215,305001,120996,86038,7253596,285447,225302,38597,2189117,18482,83798,2658152,704708,163151,212453,79162,112558,221032,118301,164422,200387,180736,377780,219554,282793,39280,1352129,2796935,1315602,1277883,261809,570478,2291167,764569,387188,458739,14472629,2777,505388,3519262,283560,79845,1194003,21575,54995,780664,20852,4560,549415,65261,3991269,5251737,68692,82123,13689583,33015,96221,94672,1048110,370107,45033,49019,678034,4725,236880,644934,241367,584691,16617,1045744,43033,313686,2054931,792964,713089,232248,174351,1890075,201928,592008,32806,258370,5278,2352127,93407,132368,87325,412739,56525,533455,15525,26176,227712,21243,22351,656969,2035066,164623,40812,1282424,613299,4031634,231952,106659,30595,221747,160519,1230201,41123,284924,144660,442406,919968,3206179,7212,11179076,4991327,37562,39953,15827,19827,84451,346580,721002,855012,103377,7130772,105134,324010,442805,9711,427780,58160,1076926,516429,244782,489884,653914,436303,917052,2052495,38627,65251,52915,177131,11079,209622,192993,16046,11851,54976,283560,44846

    SPARK_USER -> hadoop

    SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PUBLIC,PRIVATE,PRIVATE

    SPARK_YARN_MODE -> true

    SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1487901112191,1488182369912,1488182369939,1488182369956,1487901112662,1487901101855,1487901102421,1487901102036,1487901102605,1487901111931,1487901112341,1487901104882,1487901102917,1487901102878,1487901102679,1487901102525,1487901103559,1487901101241,1487901109212,1487901102059,1487901102251,1487901112216,1487901104209,1487901101915,1487901101472,1487901102350,1487901103143,1487901101452,1487901103357,1487901102333,1487901102821,1487901104279,1487901101273,1487901105193,1487901111425,1487901104167,1487901102081,1487901103492,1487901112419,1487901103680,1487901104900,1487901103294,1487901108551,1487901112602,1487901112537,1487901101798,1487901107813,1487901102289,1487901101335,1487901104860,1487901111636,1487901103036,1487901104241,1487901105354,1487901103060,1487901104787,1487901103857,1487901109618,1487901102398,1487901104191,1487901101691,1487901103636,1487901102268,1487901112637,1487901105228,1487901102569,1487901109239,1487901112556,1487901111731,1487901103728,1487901104134,1487901104374,1487901101996,1487901103591,1487901103255,1487901104014,1487901101364,1487901104048,1487901104589,1487901112691,1487901103320,1487901102944,1487901112365,1487901102504,1487901101180,1487901102794,1487901104802,1487901101715,1487901105265,1487901111610,1487901107750,1487901109008,1487901103443,1487901102129,1487901102549,1487901105387,1487901101307,1487901109117,1487901104075,1487901105087,1487901104611,1487901105320,1487901102223,1487901103462,1487901102736,1487901102859,1487901103083,1487901111866,1487901102630,1487901103120,1487901101964,1487901103191,1487901102014,1487901104515,1487901101937,1487901112022,1487901112620,1487901101825,1487901103229,1487901108931,1487901112478,1487901111264,1487901111550,1487901106810,1487901103616,1487901104264,1487901104675,1487901105427,1487901101410,1487901107873,1487901108974,1487901111992,1487901108857,1487901103171,1487901103773,1487901103943,1487901109149,1487901103523,1487901101951,1487901102989,1487901103382,1487901102457,1487901104102,1487901111361,1487901101385,1487901104837,1487901112129,1487901112163,1487901111451,1487901102156,1487901103412,1487901112450,1487901105134,1487901101428,1487901096176,1487901096191,1487901096210,1488182389071,1488182389117

    PYTHONPATH -> {{PWD}}/pyspark.zip{{PWD}}/py4j-0.9-src.zip

    SPARK_LOG_URL_STDOUT -> http://big2:8042/node/containerlogs/container_1488181659296_0004_01_000003/hadoop/stdout?start=-4096

    SPARK_YARN_CACHE_FILES -> hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-yarn_2.10-1.6.1.jar#__spark__.jar,hdfs://bigmaster:9000/user/oozie/examples/apps/pythonApp/lib/py4j-0.9-src.zip#py4j-0.9-src.zip,hdfs://bigmaster:9000/user/oozie/examples/apps/pythonApp/lib/pyspark.zip#pyspark.zip,hdfs://bigmaster:9000/user/oozie/examples/apps/pythonApp/lib/spark1.py#spark1.py,hdfs://bigmaster:9000/user/oozie/share/lib/spark/zkclient-0.3.jar#zkclient-0.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/chill-java-0.5.0.jar#chill-java-0.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/curator-framework-2.5.0.jar#curator-framework-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-httpclient-3.1.jar#commons-httpclient-3.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/flume-ng-sdk-1.6.0.jar#flume-ng-sdk-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming-flume-sink_2.10-1.6.1.jar#spark-streaming-flume-sink_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spire_2.10-0.7.4.jar#spire_2.10-0.7.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-encoding-1.7.0.jar#parquet-encoding-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-core-2.4.4.jar#jackson-core-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-annotations-2.4.4.jar#jackson-annotations-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/guava-14.0.1.jar#guava-14.0.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/eigenbase-properties-1.1.5.jar#eigenbase-properties-1.1.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/json4s-ast_2.10-3.2.10.jar#json4s-ast_2.10-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/akka-actor_2.10-2.3.11.jar#akka-actor_2.10-2.3.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/snappy-java-1.1.2.jar#snappy-java-1.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-io-2.1.jar#commons-io-2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-net-2.2.jar#commons-net-2.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spire-macros_2.10-0.7.4.jar#spire-macros_2.10-0.7.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-core-3.1.2.jar#metrics-core-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/chill_2.10-0.5.0.jar#chill_2.10-0.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/breeze-macros_2.10-0.11.2.jar#breeze-macros_2.10-0.11.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/core-1.1.2.jar#core-1.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/javax.servlet-3.0.0.v201112011016.jar#javax.servlet-3.0.0.v201112011016.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-mapred-1.7.7-hadoop2.jar#avro-mapred-1.7.7-hadoop2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jets3t-0.7.1.jar#jets3t-0.7.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/config-1.2.1.jar#config-1.2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/httpcore-4.3.3.jar#httpcore-4.3.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-jvm-3.1.2.jar#metrics-jvm-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/akka-remote_2.10-2.3.11.jar#akka-remote_2.10-2.3.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-hadoop-bundle-1.6.0.jar#parquet-hadoop-bundle-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-hive_2.10-1.6.1.jar#spark-hive_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/mesos-0.21.1-shaded-protobuf.jar#mesos-0.21.1-shaded-protobuf.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-lang-2.4.jar#commons-lang-2.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/joda-time-2.1.jar#joda-time-2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-client-0.8.2.jar#tachyon-client-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jtransforms-2.4.0.jar#jtransforms-2.4.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-format-2.3.0-incubating.jar#parquet-format-2.3.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jersey-core-1.9.jar#jersey-core-1.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scala-compiler-2.10.5.jar#scala-compiler-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/unused-1.0.0.jar#unused-1.0.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-underfs-s3-0.8.2.jar#tachyon-underfs-s3-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/calcite-core-1.2.0-incubating.jar#calcite-core-1.2.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pyspark.zip#pyspark.zip,hdfs://bigmaster:9000/user/oozie/share/lib/spark/compress-lzf-1.0.3.jar#compress-lzf-1.0.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/arpack_combined_all-0.1.jar#arpack_combined_all-0.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-common-1.7.0.jar#parquet-common-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-network-shuffle_2.10-1.6.1.jar#spark-network-shuffle_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-mapper-asl-1.9.13.jar#jackson-mapper-asl-1.9.13.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-graphite-3.1.2.jar#metrics-graphite-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pmml-schema-1.1.15.jar#pmml-schema-1.1.15.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-module-scala_2.10-2.4.4.jar#jackson-module-scala_2.10-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/oro-2.0.8.jar#oro-2.0.8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/kafka_2.10-0.8.2.1.jar#kafka_2.10-0.8.2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-catalyst_2.10-1.6.1.jar#spark-catalyst_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/curator-client-2.5.0.jar#curator-client-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-core-2.2.0.jar#metrics-core-2.2.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/breeze_2.10-0.11.2.jar#breeze_2.10-0.11.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jsr305-1.3.9.jar#jsr305-1.3.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-pool-1.5.4.jar#commons-pool-1.5.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/xz-1.0.jar#xz-1.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-jackson-1.7.0.jar#parquet-jackson-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/flume-ng-core-1.6.0.jar#flume-ng-core-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-bagel_2.10-1.6.1.jar#spark-bagel_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/uncommons-maths-1.2.2a.jar#uncommons-maths-1.2.2a.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-repl_2.10-1.6.1.jar#spark-repl_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jul-to-slf4j-1.7.10.jar#jul-to-slf4j-1.7.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/lz4-1.3.0.jar#lz4-1.3.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/mina-core-2.0.4.jar#mina-core-2.0.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-compress-1.4.1.jar#commons-compress-1.4.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/json4s-core_2.10-3.2.10.jar#json4s-core_2.10-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jcl-over-slf4j-1.7.10.jar#jcl-over-slf4j-1.7.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/leveldbjni-all-1.8.jar#leveldbjni-all-1.8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/asm-3.1.jar#asm-3.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/libfb303-0.9.2.jar#libfb303-0.9.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/netty-all-4.0.29.Final.jar#netty-all-4.0.29.Final.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/zookeeper-3.4.6.jar#zookeeper-3.4.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jersey-server-1.9.jar#jersey-server-1.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-core-asl-1.9.13.jar#jackson-core-asl-1.9.13.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/stream-2.7.0.jar#stream-2.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/datanucleus-core-3.2.10.jar#datanucleus-core-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/RoaringBitmap-0.5.11.jar#RoaringBitmap-0.5.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/httpclient-4.3.6.jar#httpclient-4.3.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/paranamer-2.6.jar#paranamer-2.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/calcite-avatica-1.2.0-incubating.jar#calcite-avatica-1.2.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pmml-agent-1.1.15.jar#pmml-agent-1.1.15.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-network-common_2.10-1.6.1.jar#spark-network-common_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pyrolite-4.9.jar#pyrolite-4.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/servlet-api-2.5-6.1.14.jar#servlet-api-2.5-6.1.14.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jline-0.9.94.jar#jline-0.9.94.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-lang3-3.3.2.jar#commons-lang3-3.3.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/flume-ng-configuration-1.6.0.jar#flume-ng-configuration-1.6.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/protobuf-java-2.5.0.jar#protobuf-java-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/akka-slf4j_2.10-2.3.11.jar#akka-slf4j_2.10-2.3.11.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/slf4j-api-1.6.6.jar#slf4j-api-1.6.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/libthrift-0.9.2.jar#libthrift-0.9.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-generator-1.7.0.jar#parquet-generator-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/oozie-sharelib-spark-4.3.0.jar#oozie-sharelib-spark-4.3.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/pmml-model-1.1.15.jar#pmml-model-1.1.15.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-math3-3.4.1.jar#commons-math3-3.4.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jline-2.10.5.jar#jline-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/hive-cli-1.2.1.spark.jar#hive-cli-1.2.1.spark.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/ivy-2.4.0.jar#ivy-2.4.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/janino-2.7.8.jar#janino-2.7.8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-sql_2.10-1.6.1.jar#spark-sql_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/gson-2.7.jar#gson-2.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jansi-1.4.jar#jansi-1.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-compiler-2.7.6.jar#commons-compiler-2.7.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jaxb-core-2.2.7.jar#jaxb-core-2.2.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-dbcp-1.4.jar#commons-dbcp-1.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/netty-3.8.0.Final.jar#netty-3.8.0.Final.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-cli-1.2.jar#commons-cli-1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming-kafka_2.10-1.6.1.jar#spark-streaming-kafka_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/xbean-asm5-shaded-4.4.jar#xbean-asm5-shaded-4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/calcite-linq4j-1.2.0-incubating.jar#calcite-linq4j-1.2.0-incubating.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jaxb-impl-2.2.7.jar#jaxb-impl-2.2.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scala-reflect-2.10.5.jar#scala-reflect-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-underfs-local-0.8.2.jar#tachyon-underfs-local-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-core_2.10-1.6.1.jar#spark-core_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-mllib_2.10-1.6.1.jar#spark-mllib_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/py4j.zip#py4j.zip,hdfs://bigmaster:9000/user/oozie/share/lib/spark/json4s-jackson_2.10-3.2.10.jar#json4s-jackson_2.10-3.2.10.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/metrics-json-3.1.2.jar#metrics-json-3.1.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/opencsv-2.3.jar#opencsv-2.3.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/py4j-0.9.jar#py4j-0.9.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-ipc-1.7.7-tests.jar#avro-ipc-1.7.7-tests.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/quasiquotes_2.10-2.0.0-M8.jar#quasiquotes_2.10-2.0.0-M8.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scalap-2.10.0.jar#scalap-2.10.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming-flume_2.10-1.6.1.jar#spark-streaming-flume_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/scala-library-2.10.5.jar#scala-library-2.10.5.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jaxb-api-2.2.2.jar#jaxb-api-2.2.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/kafka-clients-0.8.2.1.jar#kafka-clients-0.8.2.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/kryo-2.22.jar#kryo-2.22.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/slf4j-log4j12-1.6.6.jar#slf4j-log4j12-1.6.6.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jodd-core-3.5.2.jar#jodd-core-3.5.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-codec-1.4.jar#commons-codec-1.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jackson-databind-2.4.4.jar#jackson-databind-2.4.4.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jetty-6.1.14.jar#jetty-6.1.14.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/curator-recipes-2.5.0.jar#curator-recipes-2.5.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/log4j-1.2.17.jar#log4j-1.2.17.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-graphx_2.10-1.6.1.jar#spark-graphx_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-1.7.7.jar#avro-1.7.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-column-1.7.0.jar#parquet-column-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-streaming_2.10-1.6.1.jar#spark-streaming_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-unsafe_2.10-1.6.1.jar#spark-unsafe_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/spark-launcher_2.10-1.6.1.jar#spark-launcher_2.10-1.6.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/commons-logging-1.1.jar#commons-logging-1.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/jetty-util-6.1.26.jar#jetty-util-6.1.26.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/tachyon-underfs-hdfs-0.8.2.jar#tachyon-underfs-hdfs-0.8.2.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/parquet-hadoop-1.7.0.jar#parquet-hadoop-1.7.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/spark/avro-ipc-1.7.7.jar#avro-ipc-1.7.7.jar,hdfs://bigmaster:9000/user/oozie/share/lib/oozie/json-simple-1.1.jar#json-simple-1.1.jar,hdfs://bigmaster:9000/user/oozie/share/lib/oozie/oozie-hadoop-utils-hadoop-2-4.3.0.jar#oozie-hadoop-utils-hadoop-2-4.3.0.jar,hdfs://bigmaster:9000/user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.3.0.jar#oozie-sharelib-oozie-4.3.0.jar,hdfs://bigmaster:9000/user/hadoop/.sparkStaging/application_1488181659296_0004/pyspark.zip#pyspark.zip,hdfs://bigmaster:9000/user/hadoop/.sparkStaging/application_1488181659296_0004/py4j-0.9-src.zip#py4j-0.9-src.zip

  command:

    {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m '-Dlog4j.configuration=spark-log4j.properties' -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=46344' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir= org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://[email protected]:46344 --executor-id 2 --hostname big2 --cores 1 --app-id application_1488181659296_0004 --user-class-path file:$PWD/__app__.jar 1> /stdout 2> /stderr

===============================================================================

      

17/02/26 23:59:56 INFO ContainerManagementProtocolProxy: Opening proxy : big2:39242

17/02/27 00:00:00 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (big1:48794) with ID 1

17/02/27 00:00:00 INFO BlockManagerMasterEndpoint: Registering block manager big1:43711 with 511.1 MB RAM, BlockManagerId(1, big1, 43711)

17/02/27 00:00:04 INFO YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (big2:58166) with ID 2

17/02/27 00:00:04 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8

17/02/27 00:00:04 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done

17/02/27 00:00:05 INFO BlockManagerMasterEndpoint: Registering block manager big2:44508 with 511.1 MB RAM, BlockManagerId(2, big2, 44508)

17/02/27 00:00:05 INFO SparkContext: Starting job: reduce at spark1.py:16

17/02/27 00:00:05 INFO DAGScheduler: Got job 0 (reduce at spark1.py:16) with 2 output partitions

17/02/27 00:00:05 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at spark1.py:16)

17/02/27 00:00:05 INFO DAGScheduler: Parents of final stage: List()

17/02/27 00:00:05 INFO DAGScheduler: Missing parents: List()

17/02/27 00:00:05 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[1] at reduce at spark1.py:16), which has no missing parents

17/02/27 00:00:05 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.2 KB, free 5.2 KB)

17/02/27 00:00:05 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.2 KB, free 8.4 KB)

17/02/27 00:00:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.136.135:43620 (size: 3.2 KB, free: 457.9 MB)

17/02/27 00:00:05 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006

17/02/27 00:00:05 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (PythonRDD[1] at reduce at spark1.py:16)

17/02/27 00:00:05 INFO YarnClusterScheduler: Adding task set 0.0 with 2 tasks

17/02/27 00:00:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, big1, partition 0,PROCESS_LOCAL, 2064 bytes)

17/02/27 00:00:05 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, big2, partition 1,PROCESS_LOCAL, 2064 bytes)

17/02/27 00:00:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on big1:43711 (size: 3.2 KB, free: 511.1 MB)

17/02/27 00:00:18 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 12913 ms on big1 (1/2)

17/02/27 00:00:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on big2:44508 (size: 3.2 KB, free: 511.1 MB)

17/02/27 00:00:30 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 25340 ms on big2 (2/2)

17/02/27 00:00:30 INFO DAGScheduler: ResultStage 0 (reduce at spark1.py:16) finished in 25.363 s

17/02/27 00:00:30 INFO YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 

17/02/27 00:00:30 INFO DAGScheduler: Job 0 finished: reduce at spark1.py:16, took 25.556450 s

17/02/27 00:00:30 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0

17/02/27 00:00:30 INFO SparkContext: Invoking stop() from shutdown hook

17/02/27 00:00:31 INFO SparkUI: Stopped Spark web UI at http://192.168.136.135:35252

17/02/27 00:00:31 INFO YarnClusterSchedulerBackend: Shutting down all executors

17/02/27 00:00:31 INFO YarnClusterSchedulerBackend: Asking each executor to shut down

17/02/27 00:00:31 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

17/02/27 00:00:31 INFO MemoryStore: MemoryStore cleared

17/02/27 00:00:31 INFO BlockManager: BlockManager stopped

17/02/27 00:00:31 INFO BlockManagerMaster: BlockManagerMaster stopped

17/02/27 00:00:31 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

17/02/27 00:00:31 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.

17/02/27 00:00:31 INFO SparkContext: Successfully stopped SparkContext

17/02/27 00:00:31 INFO ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED

17/02/27 00:00:31 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.

17/02/27 00:00:31 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.

17/02/27 00:00:32 INFO ApplicationMaster: Deleting staging directory .sparkStaging/application_1488181659296_0004

17/02/27 00:00:33 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.

17/02/27 00:00:35 INFO ShutdownHookManager: Shutdown hook called

17/02/27 00:00:35 INFO ShutdownHookManager: Deleting directory /home/hadoop/hadoopTmp/nm-local-dir/usercache/hadoop/appcache/application_1488181659296_0004/spark-a715955c-2fb3-4fd9-bcf3-3ac78a367553

17/02/27 00:00:35 INFO ShutdownHookManager: Deleting directory /home/hadoop/hadoopTmp/nm-local-dir/usercache/hadoop/appcache/application_1488181659296_0004/spark-a715955c-2fb3-4fd9-bcf3-3ac78a367553/pyspark-ef556577-ce4c-4feb-b2af-e064edca45bf