新版本Hadoop安装过程

1、进程差异

新版本Hadoop
SecondaryNameNode  NodeManager   NameNode  DataNode  ResourceManager

旧版本Hadoop

SecondaryNamenode,JobTracker,NameNode,DataNode,TraskTracker五个进程

 

安装新版本Haddop注意事项,安装过程同hadoop低版本

loaded: ssh: Could not resolve hostname loaded: Name or service not known
It's: ssh: Could not resolve hostname It's: Name or service not known
try: ssh: Could not resolve hostname try: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
You: ssh: Could not resolve hostname You: Name or service not known
that: ssh: Could not resolve hostname that: Name or service not known
might: ssh: Could not resolve hostname might: Name or service not known
you: ssh: Could not resolve hostname you: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
to: ssh: Could not resolve hostname to: Name or service not known
highly: ssh: Could not resolve hostname highly: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known

解决方案:

export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/bigdata/hadoop272/lib/native
export HADOOP_OPTS="-Djava.library.path=/opt/bigdata/hadoop272/lib"

 

 

Maven依赖的包一起打包进jar包。使用maven-assembly打成jar后,在没有打成jar包的工程中,是可以正常运行的,通过长时间的观察和分析,发现与hadoop filesystem相关的包有两个,分别是:hadoop-hdfs-2.7.1.jar和hadoop-common-2.7.1.jar,这两个包 的META-INF中的services目录下都有,org.apache.hadoop.fs.FileSystem这个文件。使用Maven-assembly-plugin的时候,会将所有依赖的包unpack,然后在pack,这样就会出现,同样的文件被覆盖的情况

 

在函数中使用了hdfs://IP : port的schema,而在生成的最终jar包中,无法找到这个schema的实现。所以就抛出了

      java.io.IOException: No FileSystem for scheme: hdfs

      解决方案是,在设置hadoop的配置的时候,显示设置这个类:"org.apache.hadoop.hdfs.DistributedFileSystem:

      configuration.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");

 

你可能感兴趣的:(新版本Hadoop安装过程)