Spark0.91使用sbt编译Unresolved依赖问题

最近测试一下spark0.91编译hadoop0.20.2-cdh3u5.

使用sbt/sbt assembly来编译发布。

修改hadoop的版本号，这样才能和HDFS通讯哈。

object SparkBuild extends Build {
  // Hadoop version to build against. For example, "1.0.4" for Apache releases, or
  // "2.0.0-mr1-cdh4.2.0" for Cloudera Hadoop. Note that these variables can be set
  // through the environment variables SPARK_HADOOP_VERSION and SPARK_YARN.
  val DEFAULT_HADOOP_VERSION = "0.20.2-cdh3u5"

经常会出现Unresolved Dependency问题，类似如下：

[error] (examples/*:update) sbt.ResolveException: unresolved dependency: commons-lang#commons-lang;2.6: configuration not found in commons-lang#commons-lang;2.6: 'compile'. It was required from org.apache.cassandra#cassandra-all;1.2.6 compile

在网上搜了，说的是依赖关系的问题。所以就在$SPARK_HOME/project/SparkBuild.scala

 def coreSettings = sharedSettings ++ Seq(
    name := "spark-core",
    resolvers ++= Seq(
       "JBoss Repository"     at "http://repository.jboss.org/nexus/content/repositories/releases/",
       "Cloudera Repository"  at "https://repository.cloudera.com/artifactory/cloudera-repos/"
    ),

    libraryDependencies ++= Seq(
        "com.google.guava"         % "guava"            % "14.0.1",
        "com.google.code.findbugs" % "jsr305"           % "1.3.9",
        "log4j"                    % "log4j"            % "1.2.17",
        "org.slf4j"                % "slf4j-api"        % slf4jVersion,
        "org.slf4j"                % "slf4j-log4j12"    % slf4jVersion,
        "commons-daemon"           % "commons-daemon"   % "1.0.10", // workaround for bug HADOOP-9407
        "com.ning"                 % "compress-lzf"     % "1.0.0",
        "org.xerial.snappy"        % "snappy-java"      % "1.0.5",
        "org.spark-project.akka"  %% "akka-remote"      % "2.2.3-shaded-protobuf"  excludeAll(excludeNetty),
        "org.spark-project.akka"  %% "akka-slf4j"       % "2.2.3-shaded-protobuf"  excludeAll(excludeNetty),
        "org.spark-project.akka"  %% "akka-testkit"     % "2.2.3-shaded-protobuf" % "test",
        "net.liftweb"             %% "lift-json"        % "2.5.1"  excludeAll(excludeNetty),
        "it.unimi.dsi"             % "fastutil"         % "6.4.4",
        "colt"                     % "colt"             % "1.2.0",
        "org.apache.mesos"         % "mesos"            % "0.13.0",
        "net.java.dev.jets3t"      % "jets3t"           % "0.7.1",
        "org.apache.derby"         % "derby"            % "10.4.2.0"                     % "test",
        "org.apache.hadoop"        % hadoopClient       % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm, excludeCglib),
        "org.apache.avro"          % "avro"             % "1.7.4",
        "org.apache.avro"          % "avro-ipc"         % "1.7.4" excludeAll(excludeNetty),
        "org.apache.zookeeper"     % "zookeeper"        % "3.4.5" excludeAll(excludeNetty),
        "com.codahale.metrics"     % "metrics-core"     % "3.0.0",
        "com.codahale.metrics"     % "metrics-jvm"      % "3.0.0",
        "com.codahale.metrics"     % "metrics-json"     % "3.0.0",
        "com.codahale.metrics"     % "metrics-graphite" % "3.0.0",
        "com.twitter"             %% "chill"            % "0.3.1",
        "com.twitter"              % "chill-java"       % "0.3.1",
        "com.clearspring.analytics" % "stream"          % "2.5.1",
        "commons-codec" % "commons-codec" % "1.4"
      )
  )

去Maven Repository上面查找commons-lang

http://www.mvnrepository.com/artifact/commons-lang/commons-lang/2.6

找到对应版本。添加到libraryDependencies下面即可，要小心逗号哦。

libraryDependencies += "commons-lang" % "commons-lang" % "2.6"
            

最后sbt/sbt assembly即可编译成功。