spark初体验

        现在这个美好的时代,作为一个技术人,如果连Spark的大名都不知道,那显然是说不过去的,话说大数据的时候,必提Hadoop、Spark。我跟进Spark也好些日子,这次用最新搭建的Hadoop实验集群用最新的Spark来体验了一把酸甜苦辣!

        环境信息:

        centos 6.7 64位  

        Hadoop-2.7.2(QJM HA) 

        Spark-1.6.1

        scala 2.10.5

     

        我用的是Standalone模式,hadoop-2.7.2已经安装好,并且支持LZO压缩,Hive-1.2.1运行良好,spark和hadoop都运行在同一个用户和用户组下(hadoop:hadoop),HADOOP_HOME=/home/hadoop/hadoop-2.7.2,spark和Hadoop混合部署,在有hadoop的节点上都部署spark。

        直接下载:http://www.apache.org/dyn/closer.lua/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz,解压缩后做如下变更即可试运行spark了。

      (1)变更spark-env.sh文件内容

#SYSTEM
JAVA_HOME=/usr/local/jdk
SCALA_HOME=/usr/local/scala
HADOOP_HOME=/home/hadoop/hadoop-2.7.2
HADOOP_CONF_DIR=/home/hadoop/hadoop-2.7.2/etc/hadoop
SPARK_DIST_CLASSPATH=$(/home/hadoop/hadoop-2.7.2/bin/hadoop classpath) 
#SPARK_DIST_CLASSPATH=$(hadoop --config /home/hadoop/hadoop-2.7.2/etc/hadoop classpath)
#export SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/home/hadoop/hadoop-2.7.2/share/hadoop/tools/lib/*"
#spark
SPARK_HOME=/home/hadoop/spark
SPARK_MASTER_IP=lrts5
SPARK_WORKER_CORES=4
SPARK_WORKER_INSTANCES=1
SPARK_WORKER_MEMORY=4g
SPARK_EXECUTOR_CORES=1
SPARK_EXECUTOR_MEMORY=1g
#spark
SPARK_WORKER_DIR=/home/hadoop/spark/work
SPARK_LOG_DIR=/home/hadoop/spark/logs
SPARK_PID_DIR=/home/hadoop/spark/pid
#LZO
export SPARK_CLASSPATH=/home/hadoop/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar 
export SPARK_CLASSPATH=$SPARK_CLASSPATH:$CLASSPATH
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$HADOOP_HOME/lib/native
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.7.2/etc/hadoop

      (2)变更slave文件内容

lrts5
lrts6
lrts7
lrts8
      (3)将Hadoop的hdfs-site.xml 和core-site.xml文件复制到spark/conf目录下

      (4)追加如下内容到 spark-defaults.conf文件

spark.files file:///home/hadoop/spark/conf/hdfs-site.xml,file:///home/hadoop/spark/conf/core-site.xml
          如果不加这个,会有如下问题发生:

java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster
    at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:231)
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139)
    at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:510)
    at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:453)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:221)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:172)
    at org.apache.spark.rdd.RDD

      (5)读取hdfs中的lzo文件,并且分片来执行

import org.apache.hadoop.io._
import com.hadoop.mapreduce._
val data = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("hdfs://mycluster/user/hive/warehouse/logs_app_nginx/logdate=20160322/loghost=70/var.log.nginx.access_20160322.log.70.lzo")
data.count()
      (6)读取hdfs中的通配符表示的目录和子目录下文件,并且分片来执行
import org.apache.hadoop.io._
import com.hadoop.mapreduce._
val dirdata = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("hdfs://mycluster/user/hive/warehouse/logs_app_nginx/logdate=20160322/loghost=*/")
dirdata.count()


参考:

http://stackoverflow.com/questions/33174386/accessing-hdfs-ha-from-spark-job-unknownhostexception-error

你可能感兴趣的:(spark初体验)