Jar运行问题
运行jar包报错
./bin/hadoop jar ./myapp/HDFSExample.jar
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:314)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:268)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
at java.util.jar.JarVerifier.update(JarVerifier.java:228)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
at java.util.jar.JarFile.getInputStream(JarFile.java:450)
at org.apache.hadoop.util.RunJar.unJar(RunJar.java:101)
at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)
at org.apache.hadoop.util.RunJar.run(RunJar.java:209)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
解决方法：
zip -d newspark-1.0-SNAPSHOT.jar META-INF/*.RSA META-INF/*.DSA META-INF/*.SF
运行jar包找不到主类
[root@guanwh bin]# hadoop jar newspark-1.0-SNAPSHOT.jar
Exception in thread "main" java.lang.ClassNotFoundException: com.jhon.test.TestHive
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
打包时候没有class文件
#运行时报错
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq
at com.jhon.test.TestHive.main(TestHive.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: scala.collection.Seq
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
解决办法：
HADOOP_CLASSPATH=/usr/scala/scala-2.11.8/lib/scala-library.jar hadoop jar original-newspark-1.0-SNAPSHOT.jar com.jhon.test.TestHive
无法加载lib
19/05/22 15:09:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop jar 运行jar包
hadoop jar ***.jar -libjars /usr/lib **.MainClass
问题： java.lang.Exception: java.io.IOException: Unable to initialize any output collector
解决办法：
FileOutputFormat.setOutputPath(job, outPath)
[root@guanwh bin]# hadoop jar mytq.jar com.jhon.weather.MyWeather
19/05/28 12:40:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/05/28 12:40:26 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
19/05/28 12:40:26 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
19/05/28 12:40:26 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/05/28 12:40:26 INFO input.FileInputFormat: Total input paths to process : 1
19/05/28 12:40:27 INFO mapreduce.JobSubmitter: number of splits:1
19/05/28 12:40:27 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local822029151_0001
19/05/28 12:40:27 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
19/05/28 12:40:27 INFO mapreduce.Job: Running job: job_local822029151_0001
19/05/28 12:40:27 INFO mapred.LocalJobRunner: OutputCommitter set in config null
19/05/28 12:40:27 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
19/05/28 12:40:27 INFO mapred.LocalJobRunner: Waiting for map tasks
19/05/28 12:40:27 INFO mapred.LocalJobRunner: Starting task: attempt_local822029151_0001_m_000000_0
19/05/28 12:40:27 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
19/05/28 12:40:27 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/var/data/tq:0+273
19/05/28 12:40:27 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
19/05/28 12:40:27 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
19/05/28 12:40:27 INFO mapred.MapTask: soft limit at 83886080
19/05/28 12:40:27 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
19/05/28 12:40:27 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
19/05/28 12:40:27 WARN mapred.MapTask: Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
java.lang.RuntimeException: java.lang.InstantiationException
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:886)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1001)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InstantiationException
at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 13 more
19/05/28 12:40:27 INFO mapred.LocalJobRunner: map task executor complete.
19/05/28 12:40:27 WARN mapred.LocalJobRunner: job_local822029151_0001
java.lang.Exception: java.io.IOException: Unable to initialize any output collector
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.IOException: Unable to initialize any output collector
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
19/05/28 12:40:28 INFO mapreduce.Job: Job job_local822029151_0001 running in uber mode : false
19/05/28 12:40:28 INFO mapreduce.Job: map 0% reduce 0%
19/05/28 12:40:28 INFO mapreduce.Job: Job job_local822029151_0001 failed with state FAILED due to: NA
19/05/28 12:40:28 INFO mapreduce.Job: Counters: 0