idea工具里运行MapReduce报权限问题的错误

问题描述:程序在eclipse正常运行,在idea里不能运行

报错信息

java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
	at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO.(NativeIO.java:89)
	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
	at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
	at MapReducePhone.PhoneNum.main(PhoneNum.java:32)
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-hp\mapred\staging\hp1455504419\.staging to 0700
	at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
	at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
	at MapReducePhone.PhoneNum.main(PhoneNum.java:32)

在尝试Hadoop的各个版本后依然无法解决,最后看源码才看到问题的原因

在源码FileUtil.class下的490行有下列代码

  private static void checkReturnValue(boolean rv, File p, FsPermission permission) throws IOException {
        if (!rv) {
            throw new IOException("Failed to set permissions of path: " + p + " to " + String.format("%04o", permission.toShort()));
        }
    }
这是一个权限的检查功能,我们要把他注释掉,但源码是不能改动的,所以这里教大家一个取巧的方法

我们任意建一个包名为org.apache.hadoop.fs,类名为FileUtil的类,将源码的代码复制过来删除490行的权限检查。
然后去源码FileUtil.class,在本机所在路径下一般在maven下的org\apache\hadoop\hadoop-core\1.2.1下面有一个名为hadoop-core-1.2.1.jar的jar包,将这个jar包拿到一个新建文件夹名hadoop-core-1.2.1为下解压,找到FileUtil.class文件并删除,将我们建的FileUtil的类复制进去修改后缀名为.class。
再将文件夹hadoop-core-1.2.1压缩,修改压缩后文件的后缀名为.jar,再将这个文件替代我们org\apache\hadoop\hadoop-core\1.2.1文件夹下的jar包。

最后去 idea运行MapReduce,测试成功。

下面这个是我自己修改完成的jar包

https://download.csdn.net/download/weixin_43345864/10836668

你可能感兴趣的:(大数据,Hadoop)