Alluxio集成MapReduce

前提

在前文已经部署好了alluxio集群以及集成了hdfs
https://www.jianshu.com/p/f417806156cf
https://www.jianshu.com/p/2823bcd1cef7

1. 修改配置文件

• Hadoop的core-site.xml

    
    
        fs.alluxio.impl
        alluxio.hadoop.FileSystem
        The Alluxio FileSystem (Hadoop 1.x and 2.x)
    
    
        fs.AbstractFileSystem.alluxio.impl
        alluxio.hadoop.AlluxioFileSystem
        The Alluxio AbstractFileSystem (Hadoop 2.x)
    

• hadoop的hadoop-env.sh

export HADOOP_CLASSPATH=/opt/alluxio-2.0.1/client/alluxio-2.0.1-client.jar:${HADOOP_CLASSPATH}

2. 拷贝jar包到hadoop

将/opt/alluxio-2.0.1/client/alluxio-2.0.1-client.jar拷贝到hadoop的lib下

3. 将更新后的hadoop配置文件和lib的jar包更新到每一个hadoop集群,重启hadoop和alluxio

4. 检查集成mapreduce是否成功

integration/checker/bin/alluxio-checker.sh mapreduce

5. 运行wordcount示例

准备input文件
./bin/alluxio fs copyFromLocal LICENSE /wordcount/input.txt

./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount -libjars /opt/alluxio-2.0.1/client/alluxio-2.0.1-client.jar alluxio://centos134:19998/wordcount/input.txt alluxio://centos134:19998/wordcount/output3

6. CDH版集成方法

CDH版本可能需要单独编译才能正常使用，编译环境可以参考：https://www.jianshu.com/p/38e9c9eec726

• CDH在管理页面修改core-site.xml属性

  
  
  

• CDH在管理页面修改yarn属性

然后重启相关组件