mac安装hadoop及配置伪分布式

mac安装hadoop

由于是mac系统，所以这里直接用homebrew安装hadoop，默认安装的位置是/usr/local/Cellar

brew install hadoop

#查看hadoop的信息
brew info hadoop

伪分布式配置

1. 修改core-site.xml

<configuration>
	
	 <property>
        <name>hadoop.tmp.dirname>
        <value>file:/usr/local/Cellar/hadoop/3.2.1/libexec/tmpvalue>
        <description>Abase for other temporary directories.description>
    property>
    <property>
        <name>fs.defaultFSname>
        <value>hdfs://localhost:9000value>
    property>
configuration>

2. 修改hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replicationname>
        <value>1value>
    property>
    <property>
        <name>dfs.namenode.name.dirname>
        <value>file:/usr/local/Cellar/hadoop/3.2.1/libexec/tmp/dfs/namevalue>
    property>
    <property>
        <name>dfs.datanode.data.dirname>
        <value>file:/usr/local/Cellar/hadoop/3.2.1/libexec/tmp/dfs/datavalue>
    property>
configuration>

3. 配置环境变量

查看jave_home

 /usr/libexec/java_home
#输出
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home 

打开terminal，vi ~/.bash_profile，添加如下信息

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home
export HADOOP_HOME=/usr/local/Cellar/hadoop/3.2.1/libexec
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

配置ssh免密登陆

不设置免密登陆运行./sbin/start-dfs.sh启动时会提示以下的错误：

1. 打开系统偏好设置-共享，勾选远程登陆，本地调的话只开仅这些用户即可
   
2. 生成rsa密钥对，中途一直输入回车，或者输入y

ssh-keygen -t rsa -P ""

生成的公钥和私钥默认放在~/.ssh目录下， ls -lha查看该目录下的文件

3. 将公钥的内容写入到authorized_keys文件

cat id_rsa.pub >> ~/.ssh/authorized_keys

4. 验证是否可以免密登陆

ssh localhost

运行hadoop

1. 首次安装初始化namenode，进入/usr/local/Cellar/hadoop/3.2.1/bin

 ./hdfs namenode -format

2. 启动。进入/usr/local/Cellar/hadoop/3.2.1/sbin

./start-dfs.sh

3. 查看是否启动，查看java相关的进程

jps -l

4. 浏览器查看namenode，输入localhost:9870

启动yarn

1. 修改libexec/etc/hadoop/mapred-site.xml

<configuration>
	<property>
		<name>mapreduce.framework.namename>
		<value>yarnvalue>
	property>
configuration>

2. 修改libexec/etc/hadoop/yarn-site.xml

<configuration>
	<property>
			<name>yarn.nodemanager.aux-servicesname>
			<value>mapreduce_shufflevalue>
	property>
	<property>
	       <name>yarn.nodemanager.env-whitelistname>      		
	       <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOMEvalue>
	property>
configuration>

3. 启动yarn，进入/usr/local/Cellar/hadoop/3.2.1/sbin

./start-yarn.sh

浏览器地址栏键入localhost:8088

参考

https://blog.csdn.net/vbirdbest/article/details/88189753
https://blog.csdn.net/pgs1004151212/article/details/104391391