CDH hadoop-hdfs-fuse的安装使用

参考文档： http://www.cloudera.com/documentation/cdh/5-1-x/CDH5-Installation-Guide/cdh5ig_hdfs_mountable.html

挂载HDFS

CDH5包含一个HDFS的接口FUSE（用户空间文件系统），FUSE能让你写一个正常的用户空间应用程序作为一个传统文件系统接口。hadoop-hdfs-fuse包使您能够使用HDFS集群就好像它是一个传统的文件系统在Linux上。假设你有一个工作HDFS集群和知道你的NameNode暴露的主机名和端口。(翻译)   简单总结：意思就是你可以把hdfs作为一个本次挂载的磁盘，进行交换。

下面是具体的安装步骤。

如果你的CDH集群不是rpm安装的，你得先去安装 hadoop-hdfs-fuse所需要依赖的包。具体如下：

找到你所需要的rpm包先进下载：(都是国外的网站，速度比较慢，可以放后台下载)

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/noarch/

http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/x86_64/

  # 构建本地yum源 

  mkdir  -p /data/hadoop_install/cdh5

 curl    http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/noarch/ |awk -F'href= "' '{print $2}'|awk -F'">' '{print $1}' |grep rpm>noarch.list

 curl  http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/x86_64/|awk -F’href'= "' '{print $2}'|awk -F'">' '{print $1}' |grep rpm>rpm.list

然后开启后台 screen -S    下载。

 for i in `cat noarch.list` ; do wget  http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/noarch/$i ;done

  for i in `cat rpm.list` ; do wget  http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.5.1/RPMS/x86_64/$i ; done

下载完成后记得构建依赖关系：

    createrepo .

# 安装httpd服务并配置，并配置虚拟主机（如果有nginx的话，可以略过，nginx配置虚拟主机时记得开启autoindex参数）

#yum -y install httpd 
#vim /etc/httpd/conf.d/cdh_yum.conf

    ServerAdmin [email protected]
    DocumentRoot "/data/hadoop_install/"
    ServerName 192.168.8.222
    ErrorLog "logs/cm-error.log"
    CustomLog "logs/cm-access.log" common
    
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    
 
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    

#/etc/init.d/httpd restart        //重启httpd服务
然后可以通过浏览器查看是否能够访问http://192.168.8.222/cm/5.0.2

如果是Nginx的话，配置如下：

server {
        listen 80;
        server_name 192.168.8.222;
        charset utf-8;
        index index.php index.html index.htm;
        root /data/hadoop_install;
        location /cm/ {
                autoindex on;
                autoindex_exact_size off;
                autoindex_localtime on;
        }
        location /cdh/ {
                autoindex on;
                autoindex_exact_size off;
                autoindex_localtime on;
        }
       access_log  off;
}

#/etc/init.d/nginx restart        //重启nginx服务

#####在需要安装的服务上添加repo文件。例如8.94上。

  [root@hadoop94 ~]# cat /etc/yum.repos.d/c-m.repo 

[cloud-manager-5]

name=Cloud Manager,Version 5.7.1

baseurl=http://192.168.8.183:8800/cdh5

gpgcheck=0

################ 接下来就可以 yum安装了。

 [root@hadoop94 ~]# yum  install hadoop-hdfs-fuse

参考：

To set up and test your mount point in a non-HA installation:

$ mkdir -p 
$ hadoop-fuse-dfs dfs://: 

where namenode_port is the NameNode's RPC port, dfs.namenode.servicerpc-address.

To set up and test your mount point in an HA installation:

$ mkdir -p 
$ hadoop-fuse-dfs dfs:// 

where nameservice_id is the value of fs.defaultFS. In this case the port defined for dfs.namenode.rpc-address.[nameservice ID].[name node ID] is used automatically. See Configuring Software for HDFS HA for more information about these properties.

You can now run operations as if they are on your mount point. Press Ctrl+C to end the  fuse-dfs program, and  umount the partition if it is still mounted.

  Note:

To find its configuration directory, hadoop-fuse-dfs uses the HADOOP_CONF_DIR configured at the time the mount command is invoked.

To clean up your test:

$ umount 

You can now add a permanent HDFS mount which persists through reboots. To add a system mount:

1. Open /etc/fstab and add lines to the bottom similar to these:

   hadoop-fuse-dfs#dfs://:  fuse allow_other,usetrash,rw 2 0

   For example:

   hadoop-fuse-dfs#dfs://localhost:8020 /mnt/hdfs fuse allow_other,usetrash,rw 2 0

2. Test to make sure everything is working properly:

   $ mount 

Your system is now configured to allow you to use the ls command and use that mount point as if it were a normal system disk.

By default, the CDH 5 package installation creates the /etc/default/hadoop-fuse file with a maximum heap size of 128 MB. You can change the JVM minimum and maximum heap size; for example

To change it:

export LIBHDFS_OPTS="-Xms64m -Xmx256m"

Be careful not to set the minimum to a higher value than the maximum.

For more information, see the help for hadoop-fuse-dfs:

$ hadoop-fuse-dfs --help

 测试：

建立和测试您的挂载点non-HA安装:

[root@hadoop94 ~]# hadoop fs -ls /tmp      //查看Hdfs文件系统/tmp目录下文件

[root@hadoop94 ~]# mkdir  -p  /tmp/fusetest    // 建立一个本地文件

[root@hadoop94 tmp]# hadoop-fuse-dfs dfs://192.168.8.94:8022 /tmp/fusetest

INFO /data/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.6.0-cdh5.5.1/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:164 Adding FUSE arg /tmp/fusetest

[root@hadoop94 tmp]# df -h          //查看本地系统的硬盘

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root

                      165G   61G   97G  39% /

tmpfs                 3.9G  8.0K  3.9G   1% /dev/shm

/dev/xvda1            477M   51M  401M  12% /boot

cm_processes          3.9G  3.9M  3.9G   1% /var/run/cloudera-scm-agent/process

fuse_dfs              209G  2.4G  206G   2% /tmp/fuse

fuse_dfs              209G  2.4G  206G   2% /tmp/fusetest

[root@hadoop94 tmp]# cd /tmp/fusetest/

[root@hadoop94 tmp]# touch bbb.txt         //创建一个文件，编辑如下 aaaaaaa

[root@hadoop94 tmp]# vim bbb.txt 

aaaaaaa 

## 在hdfs下挂载后查看：

[root@hadoop94 ~]# hadoop fs -mkdir /tmp/fusetest

[root@hadoop94 ~]# hadoop fs -ls /tmp

[root@hadoop94 ~]# hadoop fs -cat /tmp/bbb.txt

aaaaaaa

[root@hadoop94 ~]# 

针对hdfs ha的挂载没有测试。这个主要是nameserver_id  找正确就可以。

下面是卸载：

[root@hadoop94 ~]# umount /tmp/fusetest

[root@hadoop94 ~]# 

#####可以通过一些参数，添加一个永久HDFS持续通过重新启动。添加一个系统安装:

参考上文。