How to running hadoop mapreduce on tachyon

一、修改Hadoop配置文件

1.修改core-site.xml文件

添加如下属性，让MapReduce作业可以使用Tachyon文件系统作为输入和输出

<property>
 <name>fs.tachyon.impl</name>
 <value>tachyon.hadoop.TFS</value>
</property>

2.配置hadoop-env.sh

在hadoop-env.sh文件开头添加Tachyon客户端jar包路径的环境变量。

export HADOOP_CLASSPATH=/usr/local/tachyon/client/target/tachyon-client-0.5.0-jar-with-dependencies.jar

3.将修改后的配置文件同步到其它节点

[root@node1 hadoop]# scp hadoop-env.shnode2:/usr/local/hadoop/etc/hadoop/
hadoop-env.sh                                                                                                                        100%3499     3.4KB/s   00:00   
[root@node1 hadoop]# scp hadoop-env.shnode3:/usr/local/hadoop/etc/hadoop/
hadoop-env.sh                                                                                                                        100%3499     3.4KB/s   00:00   
[root@node1 hadoop]# scp core-site.xmlnode2:/usr/local/hadoop/etc/hadoop/
core-site.xml                                                                                                                        100% 1421     1.4KB/s   00:00   
[root@node1 hadoop]# scp core-site.xmlnode3:/usr/local/hadoop/etc/hadoop/
core-site.xml                                                                                                                        100% 1421     1.4KB/s   00:00   
[root@node1 hadoop]#

4.启动所有的zookeeper节点

[root@node1 hadoop]# zkServer.sh start
JMX enabled by default
Using config:
/usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node1 hadoop]# ssh node2 zkServer.sh
start
JMX enabled by default
Using config:
/usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node1 hadoop]# ssh node3 zkServer.sh
start
JMX enabled by default
Using config:
/usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@node1 hadoop]#

5.启动hadoop集群

[root@node1 hadoop]# pwd
/usr/local/hadoop
[root@node1 hadoop]# sbin/start-all.sh

6.启动Tachyon集群

[root@node1 hadoop]# tachyon-start.sh all Mount

7.启动node2上的Tachyon Master

[root@node2 ~]# tachyon-start.sh master
Starting master @ node2
[root@node2 ~]#

8.查看进程

[root@node1 conf]# jps
21954 QuorumPeerMain
22398 JournalNode
24120 TachyonWorker
22765 NodeManager
22572 DFSZKFailoverController
22663 ResourceManager
24009 TachyonMaster
24354 Jps
22216 DataNode
22115 NameNode
[root@node1 conf]# ssh node2 jps
15524 NameNode
16538 TachyonWorker
15880 NodeManager
15802 DFSZKFailoverController
16650 Jps
15592 DataNode
15456 QuorumPeerMain
15683 JournalNode
16598 TachyonMaster
[root@node1 conf]# ssh node3 jps
9294 DataNode
9231 QuorumPeerMain
9382 JournalNode
10050 Jps
10007 TachyonWorker
9476 NodeManager
[root@node1 conf]#

二、测试Mapreduce作业

1.上传测试文件到Tachyon

[root@node1 conf]# tachyon tfs copyFromLocal /etc/passwd /passwd
Copied /etc/passwd to /passwd
[root@node1 conf]# tachyon tfs tail /passwd
tp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin

2.执行Mapreduce作业

[root@node1 hadoop]# pwd
/usr/local/hadoop
[root@node1 hadoop]# hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount -libjars
/usr/local/tachyon/client/target/tachyon-client-0.5.0-jar-with-dependencies.jar
tachyon://node1:19998/passwd tachyon://node1:19998/out

3.查看执行完成后的输出结果

[root@node1 hadoop]# tachyon tfs cat/out/part-r-00000
Daemon:/var/cache/rpcbind:/sbin/nologin     1
Daemon:/var/run/pulse:/sbin/nologin    1
IPv4LL       1
NFS  1
SSH:/var/empty/sshd:/sbin/nologin         1
Service     1
Stack:/var/lib/avahi-autoipd:/sbin/nologin      1
System     1
User:/var/ftp:/sbin/nologin     1
User:/var/lib/nfs:/sbin/nologin        2