1 ./compile.sh ./compile.conf 构建 2.0.15
node-sass install caught in infinite loop
gyp verb tmpdir == cwd automatically will remove dev files after to save disk space
gyp verb command install [ '7.8.0' ]
gyp verb download using dist-url https://npm.taobao.org/dist
gyp verb install input version string "7.8.0"
gyp verb install installing version: 7.8.0
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version not already installed, continuing with install 7.8.0
gyp verb ensuring nodedir is created /data/bigData/dr-elephant-2.0.15/web/node_modules/node-sass/.node-gyp/7.8.0
gyp WARN EACCES user "root" does not have permission to access the dev dir "/data/bigData/dr-elephant-2.0.15/web/node_modules/node-sass/.node-gyp/7.8.0"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/data/bigData/dr-elephant-2.0.15/web/node_modules/node-sass/.node-gyp"
resolve：
# npm -g --unsafe-perm install node-sass
参考
2 ./bin/start.sh app-conf/ 启动报错
$ create index yarn_app_result_i5 on yarn_app_result (job_def_id); （job_def_id varchar(800)）
[42000][1071] Specified key was too long; max key length is 3072 bytes
resolve:
索引长度是有限制的, innodb引擎utf8mb4编码的varchar占4个字节，3072 / 4 = 768.
使用create index yarn_app_result_i5 on yarn_app_result (job_def_id(768));创建索引
或 将字段长度改为768
3 spark 启动historyserver
./sbin/start-history-server.sh
报异常
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/tmp/spark-events Did you configure the correct one through spark.history.fs.logDirectory?
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:214)
at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:160)
at org.apache.spark.deploy.history.FsHistoryProvider.(FsHistoryProvider.scala:156)
at org.apache.spark.deploy.history.FsHistoryProvider.(FsHistoryProvider.scala:78)
... 6 more
Caused by: java.io.FileNotFoundException: File file:/tmp/spark-events does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:204)
... 9 more
将spark.fs.history.logDirectory配置到conf/spark-defaults.conf目录里，发现并不起作用，
查看start-history-server.sh脚本内容，发现这个参数可以作为启动脚本的第一个参数传递：
if [ $# != 0 ]; then
echo "Using command line arguments for setting the log directory is deprecated. Please "
echo "set the spark.history.fs.logDirectory configuration option instead."
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$1"
fi
故使用：
./start-history-server.sh hdfs://localhost:9000/history
4 ./compile.sh ./compile.conf ( dr-elephant-2.0.15)
sbt 0.13.2 spark 1.6.3 scala 2.10.4 OK
sbt 0.13.2 spark 2.2.1 scala 2.10.4
java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-api;1.7.6
at org.apache.ivy.core.resolve.IvyNode.getArtifacts(IvyNode.java:809)
at org.apache.ivy.core.resolve.IvyNode.getSelectedArtifacts(IvyNode.java:786)
at org.apache.ivy.core.report.ResolveReport.setDependencies(ResolveReport.java:235)
at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:235)
at org.apache.ivy.Ivy.resolve(Ivy.java:517)
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:211)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:126)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:125)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
- sbt 1.1.1 spark 1.6.3 scala 2.10.4
java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at sbt.Help$.briefDetail(Command.scala:232)
at sbt.Help$.(Command.scala:214)
at sbt.Help$.(Command.scala)
at sbt.Command$.command$default$2(Command.scala:87)
at sbt.BuiltinCommands$.defaults(Main.scala:363)
at sbt.xMain.run(Main.scala:74)
at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:57)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:77)
at xsbt.boot.Launch$.run(Launch.scala:57)
at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
at xsbt.boot.Launch$.launch(Launch.scala:65)
at xsbt.boot.Launch$.apply(Launch.scala:16)
at xsbt.boot.Boot$.runImpl(Boot.scala:32)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
Error during sbt execution: java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
- sbt: 0.13.7 spark: 2.2.1 scala: 2.11.8 / 2.10.4
java.lang.UnsupportedOperationException: Position.start on class scala.reflect.internal.util.OffsetPosition
at scala.reflect.internal.util.Position.start(Position.scala:114)
at sbt.internals.parser.SbtParser.sbt$internals$parser$SbtParser$$convertStatement$1(SbtParser.scala:142)
at sbt.internals.parser.SbtParser$$anonfun$4.apply(SbtParser.scala:147)
at sbt.internals.parser.SbtParser$$anonfun$4.apply(SbtParser.scala:147)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbt.internals.parser.SbtParser.splitExpressions(SbtParser.scala:147)
at sbt.internals.parser.SbtParser.(SbtParser.scala:60)
at sbt.EvaluateConfigurations$.splitExpressions(EvaluateConfigurations.scala:227)
at sbt.EvaluateConfigurations$.parseConfiguration(EvaluateConfigurations.scala:69)
at sbt.EvaluateConfigurations$.evaluateSbtFile(EvaluateConfigurations.scala:107)
at sbt.Load$.sbt$Load$$loadSettingsFile$1(Load.scala:710)
at sbt.Load$$anonfun$sbt$Load$$memoLoadSettingsFile$1$1.apply(Load.scala:715)
at sbt.Load$$anonfun$sbt$Load$$memoLoadSettingsFile$1$1.apply(Load.scala:714)
at scala.Option.getOrElse(Option.scala:120)
at sbt.Load$.sbt$Load$$memoLoadSettingsFile$1(Load.scala:714)
at sbt.Load$$anonfun$loadFiles$1$2.apply(Load.scala:721)
at sbt.Load$$anonfun$loadFiles$1$2.apply(Load.scala:721)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at sbt.Load$.loadFiles$1(Load.scala:721)
at sbt.Load$.discoverProjects(Load.scala:732)
at sbt.Load$.discover$1(Load.scala:545)
at sbt.Load$.loadTransitive(Load.scala:574)
at sbt.Load$.loadProjects$1(Load.scala:442)
at sbt.Load$.loadUnit(Load.scala:446)
at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:91)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:90)
at sbt.BuildLoader.apply(BuildLoader.scala:140)
at sbt.Load$.loadAll(Load.scala:334)
at sbt.Load$.loadURI(Load.scala:289)
at sbt.Load$.load(Load.scala:285)
at sbt.Load$.load(Load.scala:276)
at sbt.Load$.apply(Load.scala:130)
at sbt.Load$.buildPluginDefinition(Load.scala:819)
at sbt.Load$.buildPlugins(Load.scala:785)
at sbt.Load$.plugins(Load.scala:773)
at sbt.Load$.loadUnit(Load.scala:431)
at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
at sbt.Load$$anonfun$18$$anonfun$apply$11.apply(Load.scala:281)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:91)
at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:90)
at sbt.BuildLoader.apply(BuildLoader.scala:140)
at sbt.Load$.loadAll(Load.scala:334)
at sbt.Load$.loadURI(Load.scala:289)
at sbt.Load$.load(Load.scala:285)
at sbt.Load$.load(Load.scala:276)
at sbt.Load$.apply(Load.scala:130)
at sbt.Load$.defaultLoad(Load.scala:36)
at sbt.BuiltinCommands$.doLoadProject(Main.scala:481)
at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:475)
at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:475)
at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:58)
at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:58)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
at sbt.Command$.process(Command.scala:92)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
at sbt.State$$anon$1.process(State.scala:184)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.MainLoop$.next(MainLoop.scala:98)
at sbt.MainLoop$.run(MainLoop.scala:91)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:70)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:65)
at sbt.Using.apply(Using.scala:24)
at sbt.MainLoop$.runWithNewLog(MainLoop.scala:65)
at sbt.MainLoop$.runAndClearLast(MainLoop.scala:48)
at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:32)
at sbt.MainLoop$.runLogged(MainLoop.scala:24)
at sbt.StandardMain$.runManaged(Main.scala:53)
at sbt.xMain.run(Main.scala:28)
at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:57)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:77)
at xsbt.boot.Launch$.run(Launch.scala:57)
at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
at xsbt.boot.Launch$.launch(Launch.scala:65)
at xsbt.boot.Launch$.apply(Launch.scala:16)
at xsbt.boot.Boot$.runImpl(Boot.scala:32)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
[error] java.lang.UnsupportedOperationException: Position.start on class scala.reflect.internal.util.OffsetPosition