Alluxio 1.7.0 透明命名机制及统一命名空间的案例与解读 以及Alluxio 1.7.0 HA容错安装部署
Alluxio 1.7.0 使用统一透明命名空间，通过使用其透明命名机制以及挂载API，Alluxio支持在不同存储系统之间对数据进行高效的管理。在Alluxio重新安装的基础上，再次安装Alluxio 1.7.0 HA容错。然后进行Alluxio 1.7.0 透明命名机制及统一命名空间的案例与解读
- 在Alluxio重新安装的基础上，再次安装Alluxio 1.7.0 HA容错
1，在Master节点、Worker1、Worker2节点上分别启动ZooKeeper集群及检查
状态。
root@master:~# zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
root@master:~# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower
root@master:~#
root@worker1:~# zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
root@worker1:~# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: leader
root@worker1:~#
root@worker2:~# zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
root@worker2:~# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower
root@worker2:~# 2) Alluxio Master配置
root@master:~# cd /usr/local/alluxio-1.7.0-hadoop-2.6/conf
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# ls
alluxio-env.sh.template alluxio-site.properties core-site.xml.template log4j.properties masters metrics.properties.template workers
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# mv alluxio-env.sh.template alluxio-env.sh
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# vi alluxio-env.sh
#!/usr/bin/env bash
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#
# Copy this file as alluxio-env.sh and edit it to configure Alluxio for your
# site. This file is sourced to launch Alluxio servers or use Alluxio shell
# commands.
#
# This file is an optional approach to configure Alluxio options by setting the
# following listed environment variables. Note that, setting this file will not
# affect jobs (e.g., Spark job or MapReduce job) that are using Alluxio client
# as a library. Alternatively, it is recommended to create alluxio-site.properties file,
# which supports all the configuration options provided by Alluxio
# (http://www.alluxio.org/documentation/en/Configuration-Settings.html),
# and is respected by both external jobs and Alluxio servers (or shell).
# The directory where log files are stored. (Default: ${ALLUXIO_HOME}/logs).
# ALLUXIO_LOGS_DIR
# Config properties set for Alluxio master, worker and shell. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_JAVA_OPTS
# Config properties set for Alluxio master daemon. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_MASTER_JAVA_OPTS
# Config properties set for Alluxio worker daemon. (Default: "")
# E.g. "-Dalluxio.worker.port=49999" to set worker port, "-Xms2048M -Xmx2048M" to limit the heap size of worker.
# ALLUXIO_WORKER_JAVA_OPTS
# Config properties set for Alluxio shell. (Default: "")
# E.g. "-Dalluxio.user.file.writetype.default=CACHE_THROUGH"
# ALLUXIO_USER_JAVA_OPTS
# Additional classpath entries for Alluxio processes. (Default: "")
# E.g. "/path/to/library1/:/path/to/library2/"
# ALLUXIO_CLASSPATH
export ALLUXIO_JAVA_OPTS+="
-Dalluxio.zookeeper.enabled=true
-Dalluxio.zookeeper.address=192.168.189.1:2181,192.168.189.2:2181,192.168.189.3:2181
-Dalluxio.master.journal.folder=hdfs://master:9000/alluxio/data/journal/"
export ALLUXIO_MASTER_ADDRESS=master
"alluxio-env.sh" 72L, 2356C written
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# 在conf/alluxio-env.sh中，配置ALLUXIO_JAVA_OPTS选项，配置Zookeeper地址及在ALLUXIO_JAVA_OPTS中设置alluxio.master.journal.folder。
root@worker1:~# cd /usr/local/alluxio-1.7.0-hadoop-2.6/conf
root@worker1:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# mv alluxio-env.sh.template alluxio-env.sh
root@worker1:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# vi alluxio-env.sh
#!/usr/bin/env bash
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#
# Copy this file as alluxio-env.sh and edit it to configure Alluxio for your
# site. This file is sourced to launch Alluxio servers or use Alluxio shell
# commands.
#
# This file is an optional approach to configure Alluxio options by setting the
# following listed environment variables. Note that, setting this file will not
# affect jobs (e.g., Spark job or MapReduce job) that are using Alluxio client
# as a library. Alternatively, it is recommended to create alluxio-site.properties file,
# which supports all the configuration options provided by Alluxio
# (http://www.alluxio.org/documentation/en/Configuration-Settings.html),
# and is respected by both external jobs and Alluxio servers (or shell).
# The directory where log files are stored. (Default: ${ALLUXIO_HOME}/logs).
# ALLUXIO_LOGS_DIR
# Config properties set for Alluxio master, worker and shell. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_JAVA_OPTS
# Config properties set for Alluxio master daemon. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_MASTER_JAVA_OPTS
# Config properties set for Alluxio worker daemon. (Default: "")
# E.g. "-Dalluxio.worker.port=49999" to set worker port, "-Xms2048M -Xmx2048M" to limit the heap size of worker.
# ALLUXIO_WORKER_JAVA_OPTS
# Config properties set for Alluxio shell. (Default: "")
# E.g. "-Dalluxio.user.file.writetype.default=CACHE_THROUGH"
# ALLUXIO_USER_JAVA_OPTS
# Additional classpath entries for Alluxio processes. (Default: "")
# E.g. "/path/to/library1/:/path/to/library2/"
# ALLUXIO_CLASSPATH
export ALLUXIO_JAVA_OPTS+="
-Dalluxio.zookeeper.enabled=true
-Dalluxio.zookeeper.address=192.168.189.1:2181,192.168.189.2:2181,192.168.189.3:2181
-Dalluxio.master.journal.folder=hdfs://master:9000/alluxio/data/journal/"
export ALLUXIO_MASTER_ADDRESS=worker1
"alluxio-env.sh" 63L, 2348C written
root@worker1:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# 所有Alluxio Master（Master节点（192.168.189.1）、Worker节点（192.168.189.2））以这种方式配置后(配置Master地址)，都可以启动用于Alluxio的容错。其中一个成为Leader，其余重播日志直到当前Master失效。
3） Alluxio Worker配置。
只要以上参数配置正确，Alluxio Worker就可以咨询ZooKeeper，找到当前应当连接的Master。所以，Alluxio Worker无需设置ALLUXIO_MASTER_ADDRESS。 Worker2（192.168.189.3）、Worker3（192.168.189.4）的conf/alluxio-env.sh配置。
root@worker2:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# ls
alluxio-env.sh.template alluxio-site.properties core-site.xml.template log4j.properties masters metrics.properties.template workers
root@worker2:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# mv alluxio-env.sh.template alluxio-env.sh
root@worker2:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# vi alluxio-env.sh
#!/usr/bin/env bash
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#
# Copy this file as alluxio-env.sh and edit it to configure Alluxio for your
# site. This file is sourced to launch Alluxio servers or use Alluxio shell
# commands.
#
# This file is an optional approach to configure Alluxio options by setting the
# following listed environment variables. Note that, setting this file will not
# affect jobs (e.g., Spark job or MapReduce job) that are using Alluxio client
# as a library. Alternatively, it is recommended to create alluxio-site.properties file,
# which supports all the configuration options provided by Alluxio
# (http://www.alluxio.org/documentation/en/Configuration-Settings.html),
# and is respected by both external jobs and Alluxio servers (or shell).
# The directory where log files are stored. (Default: ${ALLUXIO_HOME}/logs).
# ALLUXIO_LOGS_DIR
# Config properties set for Alluxio master, worker and shell. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_JAVA_OPTS
# Config properties set for Alluxio master daemon. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_MASTER_JAVA_OPTS
# Config properties set for Alluxio worker daemon. (Default: "")
# E.g. "-Dalluxio.worker.port=49999" to set worker port, "-Xms2048M -Xmx2048M" to limit the heap size of worker.
# ALLUXIO_WORKER_JAVA_OPTS
# Config properties set for Alluxio shell. (Default: "")
# E.g. "-Dalluxio.user.file.writetype.default=CACHE_THROUGH"
# ALLUXIO_USER_JAVA_OPTS
# Additional classpath entries for Alluxio processes. (Default: "")
# E.g. "/path/to/library1/:/path/to/library2/"
# ALLUXIO_CLASSPATH
export ALLUXIO_JAVA_OPTS+="
-Dalluxio.zookeeper.enabled=true
-Dalluxio.zookeeper.address=192.168.189.1:2181,192.168.189.2:2181,192.168.189.3:2181"
"alluxio-env.sh" 55L, 2231C written
root@worker2:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# root@worker3:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# mv alluxio-env.sh.template alluxio-env.sh
mv: cannot stat 鈥榓lluxio-env.sh.template鈥 No such file or directory
root@worker3:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# vi alluxio-env.sh
#!/usr/bin/env bash
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#
# Copy this file as alluxio-env.sh and edit it to configure Alluxio for your
# site. This file is sourced to launch Alluxio servers or use Alluxio shell
# commands.
#
# This file is an optional approach to configure Alluxio options by setting the
# following listed environment variables. Note that, setting this file will not
# affect jobs (e.g., Spark job or MapReduce job) that are using Alluxio client
# as a library. Alternatively, it is recommended to create alluxio-site.properties file,
# which supports all the configuration options provided by Alluxio
# (http://www.alluxio.org/documentation/en/Configuration-Settings.html),
# and is respected by both external jobs and Alluxio servers (or shell).
# The directory where log files are stored. (Default: ${ALLUXIO_HOME}/logs).
# ALLUXIO_LOGS_DIR
# Config properties set for Alluxio master, worker and shell. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_JAVA_OPTS
# Config properties set for Alluxio master daemon. (Default: "")
# E.g. "-Dalluxio.master.port=39999"
# ALLUXIO_MASTER_JAVA_OPTS
# Config properties set for Alluxio worker daemon. (Default: "")
# E.g. "-Dalluxio.worker.port=49999" to set worker port, "-Xms2048M -Xmx2048M" to limit the heap size of worker.
# ALLUXIO_WORKER_JAVA_OPTS
# Config properties set for Alluxio shell. (Default: "")
# E.g. "-Dalluxio.user.file.writetype.default=CACHE_THROUGH"
# ALLUXIO_USER_JAVA_OPTS
# Additional classpath entries for Alluxio processes. (Default: "")
# E.g. "/path/to/library1/:/path/to/library2/"
# ALLUXIO_CLASSPATH
export ALLUXIO_JAVA_OPTS+="
-Dalluxio.zookeeper.enabled=true
-Dalluxio.zookeeper.address=192.168.189.1:2181,192.168.189.2:2181,192.168.189.3:2181"
~
"alluxio-env.sh" 54L, 2230C written
root@worker3:/usr/local/alluxio-1.7.0-hadoop-2.6/conf#
5) 重新格式化Alluxio。如果是首次修改底层存储系统为HDFS的话，需要进行格式化。
root@master:~# alluxio format
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all worker nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio formatWorker
Waiting for tasks to finish...
All tasks finished
Formatting Alluxio Master @ master
2018-02-15 07:33:08,389 INFO Configuration - Configuration file /usr/local/alluxio-1.7.0-hadoop-2.6/conf/alluxio-site.properties loaded.
2018-02-15 07:33:09,587 INFO Format - Formatting master journal: hdfs://master:9000/alluxio/data/journal/
2018-02-15 07:33:10,299 INFO UnderFileSystemFactoryRegistry - Loading core UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/lib
2018-02-15 07:33:11,092 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-oss-1.7.0.jar
2018-02-15 07:33:11,215 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-gcs-1.7.0.jar
2018-02-15 07:33:11,350 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-wasb-1.7.0.jar
2018-02-15 07:33:11,712 WARN UnderFileSystemFactoryRegistry - Failed to load jar /usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-wasb-1.7.0.jar: alluxio.underfs.UnderFileSystemFactory: Provider alluxio.underfs.hdfs.HdfsUnderFileSystemFactory not found
2018-02-15 07:33:11,712 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-s3a-1.7.0.jar
2018-02-15 07:33:11,924 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-hdfs-1.7.0.jar
2018-02-15 07:33:12,229 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-swift-1.7.0.jar
2018-02-15 07:33:12,340 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-local-1.7.0.jar
2018-02-15 07:33:12,364 INFO UnderFileSystemFactoryRegistry - Loading extension UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/extensions
2018-02-15 07:33:12,365 INFO UnderFileSystemFactoryRegistry - Loading core UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/lib
2018-02-15 07:33:12,370 INFO UnderFileSystemFactoryRegistry - Loading extension UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/extensions
2018-02-15 07:33:15,592 WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-02-15 07:33:21,724 INFO UnderFileSystemFactoryRegistry - Loading core UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/lib
2018-02-15 07:33:21,727 INFO UnderFileSystemFactoryRegistry - Loading extension UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/extensions
2018-02-15 07:33:21,770 INFO UfsJournal - Formatting hdfs://master:9000/alluxio/data/journal/BlockMaster/v1
2018-02-15 07:33:23,433 WARN HdfsUnderFileSystem - Retry count 1 : Cannot delete /alluxio/data/journal/BlockMaster/v1/_format_1518500879508. Name node is in safe mode.
The reported blocks 1700 needs additional 716 blocks to reach the threshold 0.9990 of total blocks 2418.
The number of live datanodes 3 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3967)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3925)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3909)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:786)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:589)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
提示出错，关闭Hadoop的安全模式，再次格式化。
root@master:~# hadoop dfsadmin -safemode leave
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Safe mode is OFF
root@master:~# alluxio format
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all worker nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio formatWorker
Waiting for tasks to finish...
All tasks finished
Formatting Alluxio Master @ master
2018-02-15 07:34:33,428 INFO Configuration - Configuration file /usr/local/alluxio-1.7.0-hadoop-2.6/conf/alluxio-site.properties loaded.
2018-02-15 07:34:33,566 INFO Format - Formatting master journal: hdfs://master:9000/alluxio/data/journal/
2018-02-15 07:34:33,589 INFO UnderFileSystemFactoryRegistry - Loading core UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/lib
2018-02-15 07:34:33,597 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-oss-1.7.0.jar
2018-02-15 07:34:33,600 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-gcs-1.7.0.jar
2018-02-15 07:34:33,604 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-wasb-1.7.0.jar
2018-02-15 07:34:33,613 WARN UnderFileSystemFactoryRegistry - Failed to load jar /usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-wasb-1.7.0.jar: alluxio.underfs.UnderFileSystemFactory: Provider alluxio.underfs.hdfs.HdfsUnderFileSystemFactory not found
2018-02-15 07:34:33,613 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-s3a-1.7.0.jar
2018-02-15 07:34:33,617 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-hdfs-1.7.0.jar
2018-02-15 07:34:33,626 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-swift-1.7.0.jar
2018-02-15 07:34:33,630 INFO ExtensionsClassLoader - Created ExtensionsClassLoader with jars=file:/usr/local/alluxio-1.7.0-hadoop-2.6/lib/alluxio-underfs-local-1.7.0.jar
2018-02-15 07:34:33,633 INFO UnderFileSystemFactoryRegistry - Loading extension UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/extensions
2018-02-15 07:34:33,635 INFO UnderFileSystemFactoryRegistry - Loading core UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/lib
2018-02-15 07:34:33,637 INFO UnderFileSystemFactoryRegistry - Loading extension UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/extensions
2018-02-15 07:34:33,983 WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-02-15 07:34:34,439 INFO UnderFileSystemFactoryRegistry - Loading core UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/lib
2018-02-15 07:34:34,441 INFO UnderFileSystemFactoryRegistry - Loading extension UFS jars from /usr/local/alluxio-1.7.0-hadoop-2.6/extensions
2018-02-15 07:34:34,483 INFO UfsJournal - Formatting hdfs://master:9000/alluxio/data/journal/BlockMaster/v1
2018-02-15 07:34:34,890 INFO UfsJournal - Formatting hdfs://master:9000/alluxio/data/journal/FileSystemMaster/v1
2018-02-15 07:34:34,908 INFO Format - Formatting complete
root@master:~# 6) 启动Alluxio集群，启动第一个Master节点。
root@master:~# alluxio-start.sh all Mount
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all master nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all worker nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all worker nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-stop.sh worker
Waiting for tasks to finish...
All tasks finished
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all master nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-stop.sh master
Waiting for tasks to finish...
All tasks finished
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all master nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-start.sh master
Waiting for tasks to finish...
All tasks finished
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all worker nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-start.sh worker Mount
Waiting for tasks to finish...
All tasks finished
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all master nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-start.sh proxy
Waiting for tasks to finish...
All tasks finished
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Executing the following command on all worker nodes and logging to /usr/local/alluxio-1.7.0-hadoop-2.6/logs/task.log: /usr/local/alluxio-1.7.0-hadoop-2.6/bin/alluxio-start.sh proxy
Waiting for tasks to finish...
All tasks finished
root@master:~# 各节点jps检查
root@master:~# jps
4326 AlluxioMaster
2808 SecondaryNameNode
4552 AlluxioProxy
4632 Jps
3288 QuorumPeerMain
2603 NameNode
2973 ResourceManager
root@master:~# root@worker1:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# jps
2546 QuorumPeerMain
3412 AlluxioProxy
2181 DataNode
3463 Jps
2346 NodeManagerroot@worker2:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# jps
2549 QuorumPeerMain
2182 DataNode
3622 AlluxioProxy
3512 AlluxioWorker
3689 Jps
2348 NodeManagerroot@worker3:/usr/local/alluxio-1.7.0-hadoop-2.6/conf# jps
2352 NodeManager
3316 AlluxioProxy
3354 Jps
2141 DataNode
3197 AlluxioWorker查看Web Interface界面(http://192.168.189.1:19999/home ，第一个Master的地址)
Alluxio Master多Master节点部署成功！
2. Alluxio透明命名机制及统一命名空间
(一) Alluxio透明命名机制
Alluxio透明命名机制保证了Alluxio和底层存储系统的命名空间是一致的。如图6-23所示。
图 6 - 23Alluxio透明命名机制
当在Alluxio文件系统中创建对象时，可以选择这些对象是否要在底层存储系统中进行持久化。对于需要持久化的对象， Alluxio会保存底层文件系统存储这些对象的文件夹的路径。例如，一个用户在根目录下创建了一个Users目录，其中包含Alice和Bob两个子目录，底层文件系统（如HDFS或S3）也会保存相同的目录结构和命名。类似地，当用户在 Alluxio文件系统中对一个持久化的对象进行重命名或者删除操作时，底层文件系统中对应的对象也会被执行相同的操作。
另外，Alluxio能够搜索到底层文件系统中并非通过Alluxio创建的对象。例如，底层文件系统中包含一个Data文件夹， 其中包含Reports和Sales两个文件，它们都不是通过Alluxio创建的，当它们第一次被访问时（例如用户请求打开文件），Alluxio会自动加载这些对象的元数据。然而在该过程中Alluxio不会加载具体文件数据，若要将其加载到Alluxio， 可以用FileInStream读数据，或者通过AlluxioShell中的load命令进行加载。
(二) Alluxio统一命名空间
Alluxio提供了一个挂载API，通过该API能够在Alluxio中访问多个数据源中的数
据。如图6-24所示。
图 6 - 24
默认情况下，Alluxio文件系统挂载到Alluxio配置中alluxio.underfs.address指定的目录，该目录代表Alluxio 的”primary storage”。另外，用户可以通过挂载API添加和删除数据源。
bool mount(String alluxioPath, String ufsPath);
bool unmount(String alluxioPath);
例如，主存储（”primary storage”）可以是HDFS，其中可以包含用户的文件夹；Data文件夹可能存储在S3文件系统 下，这可以通过mount(alluxio://host:port/Data,s3://bucket/directory)命令实现。
1.5.2 Alluxio透明命名机制及统一命名空间案例实战
在本节示例中，将进行Alluxio透明命名机制及统一命名空间案例实战。这里Alluxio的源代码在${ALLUXIO_HOME}目录中（/usr/local/alluxio-1.7.0-hadoop-2.6），并且Alluxio在集群运行。
1) 先在本地文件系统中创建一个临时目录alluxio-demo。
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6# mkdirtmp
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6# cd tmp
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp#mkdir alluxio-demo
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp#touch alluxio-demo/hello
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp# catalluxio-demo/hello
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp# cdalluxio-demo
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo#pwd
/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo2) 将Linux本地alluxio-demo目录挂载到Alluxio中。
检查挂载后的目录demo已经在Alluxio文件系统中存在。
root@master:~# alluxio fs mount /demo file:///usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Mounted file:///usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo at /demo
root@master:~# alluxio fs lsr /demo
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
WARNING: lsr is deprecated. Please use ls -R instead.
-rw-r--r-- root root 0 PERSISTED 02-15-2018 08:10:42:329 100% /demo/hello
root@master:~# 3) 验证对于不是通过Alluxio创建的对象，当第一次访问它们时，其元数据被加载进入了Alluxio中。
root@master:~# alluxio fs ls /demo/hello
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
-rw-r--r-- root root 0 PERSISTED 02-15-2018 08:10:42:329 100% /demo/hello
root@master:~# 4）在Alluxio中重命名一个文件，并验证在底层文件系统中该文件也被重命名。
root@master:~# alluxio fs mv /demo/hello /demo/world
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Renamed /demo/hello to /demo/world
root@master:~# alluxio fs ls /demo/
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
-rw-r--r-- root root 0 PERSISTED 02-15-2018 08:10:42:329 100% /demo/world
-rw-r--r-- root root 0 TO_BE_PERSISTED 02-15-2018 08:12:52:150 100% /demo/hello2root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo# alluxio fs rm /demo/world
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
/demo/world has been removed
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo# ls
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo#
文件hello3，然后测试验证在Alluxio卸载挂载目录/demo，并确认/demo目录已经在Alluxio文件系统中被移除，但原先的数据依然保存在底层HDFS文件系统中。
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo# touch hello3
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo# alluxio fs ls /demo/hello3
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1 and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
-rw-r--r-- root root 0 PERSISTED 02-15-2018 08:50:07:148 100% /demo/hello3
在Alluxio卸载挂载目录/demo，并确认/demo目录已经在Alluxio文件系统中被移除。
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo# alluxio fs unmount /demo
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Unmounted /demo
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo#alluxio fs ls /demo
ALLUXIO_MASTER_ADDRESS is deprecated since version 1.1and will be remove in version 2.0.
Please use "ALLUXIO_MASTER_HOSTNAME" instead.
Path /demo does not exist
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo#原先的数据hello3依然保存在底层HDFS文件系统中。
root@master:/usr/local/alluxio-1.7.0-hadoop-2.6/tmp/alluxio-demo# ls
hello3