Alluxio安装(以HDFS作为底层文件系统)
上一篇我们介绍了如何通过源码实现 Alluxio编译 的整个过程，接下来，这篇文章，将带你一起把我们编译好的安装包（或者是从官方直接下载的安装包——hadoop版本号需要跟你的基础环境一致）部署安装在我们的服务器上。
一、准备工作
1. 编译好的二进制安装文件包
2. 已安装了JDK_1.7或者以上版本的java运行时环境
3. 运行正常且版本号与我们的Alluxio安装包对应的版本一致的Hadoop基础环境
二、执行配置&安装
1. 先将我们前面编译好的二进制安装包文件上传到服务器上，然后解压缩
[op1@HOST_NAME_01 alluxio]$ ls -l
total 235260
-rw------- 1 op1 op1 240902504 Jan 17 11:59 alluxio-1.4.0.bin.tar.gz
tar -zxvf alluxio-1.4.0.bin.tar.gz
ln -s alluxio current
2. 创建HDFS数据目录，用于存放Alluxio数据文件
[op1@HOST_NAME_01 current]$ hdfs dfs -mkdir -p /alluxio/data_demo
...
[op1@HOST_NAME_01 current]$ hdfs dfs -ls /alluxio/
...
drwxr-xr-x - op1 supergroup 0 2017-01-17 12:25 /alluxio/data_demo
3. 初始化配置
[op1@HOST_NAME_01 alluxio]$ cd current
[op1@HOST_NAME_01 current]$ ./bin/alluxio bootstrapConf HOST_NAME_01 hdfs
/home/op1/alluxio/current/bin/../conf/alluxio-env.sh is created.
mv conf/alluxio-site.properties.template conf/alluxio-site.properties
修改conf/alluxio-site.properties文件，将底层存储系统的地址设置为HDFS namenode的地址以及你想挂载到Alluxio根目录下的HDFS目录。
例如，若你的HDFS namenode是在本地默认端口运行，并且HDFS的根目录已经被映射到Alluxio根目录，则该地址为hdfs://HOST_NAME_01:9000；
若只有/alluxio/data这一个HDFS目录被映射到Alluxio根目录，则该地址为hdfs://HOST_NAME_01:9000/alluxio/data_demo。
[op1@HOST_NAME_01 current]$ vi conf/alluxio-site.properties
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#
# Site specific configuration properties for Alluxio
# Details about all configuration properties http://www.alluxio.org/documentation/en/Configuration-Settings.html
# Common properties
# alluxio.master.hostname=localhost
# alluxio.underfs.address=${alluxio.work.dir}/underFSStorage
alluxio.underfs.address=hdfs://HADOOP_MASTER_HOST_NAME:9000/alluxio/data_demo
# Security properties
# alluxio.security.authorization.permission.enabled=true
# alluxio.security.authentication.type=SIMPLE
# Worker properties
# alluxio.worker.memory.size=1GB
# alluxio.worker.tieredstore.levels=1
# alluxio.worker.tieredstore.level0.alias=MEM
# alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk
# User properties
# alluxio.user.file.readtype.default=CACHE_PROMOTE
# alluxio.user.file.writetype.default=MUST_CACHE
三、启动&确认
1.启动
./bin/alluxio format
./bin/alluxio-start.sh all
至此，一个Alluxio master和一个Alluxio worker已经在本地启动完成，可以在浏览器中访问http://host_ip:19999查看Master Web UI,如下图。
2. 测试&验证
接下来，运行一个官方的示例程序
[op1@HOST_NAME_01 current]$ hdfs dfs -ls /alluxio/data_demo
...
[op1@HOST_NAME_01 current]$ ./bin/alluxio runTests
2017-01-17 12:43:49,794 INFO type (MetricsSystem.java:startSinksFromConfig) - Starting sinks with config: {}.
2017-01-17 12:43:49,885 INFO type (AbstractClient.java:connect) - Alluxio client (version 1.4.0) is trying to connect with FileSystemMasterClient master @ SVR3404HW1288/10.2.26.40:19998
2017-01-17 12:43:49,906 INFO type (AbstractClient.java:connect) - Client registered with FileSystemMasterClient master @ SVR3404HW1288/10.2.26.40:19998
runTest Basic CACHE_PROMOTE MUST_CACHE
2017-01-17 12:43:50,107 INFO type (AbstractClient.java:connect) - Alluxio client (version 1.4.0) is trying to connect with BlockMasterClient master @ SVR3404HW1288/10.2.26.40:19998
2017-01-17 12:43:50,110 INFO type (AbstractClient.java:connect) - Client registered with BlockMasterClient master @ SVR3404HW1288/10.2.26.40:19998
2017-01-17 12:43:50,182 INFO type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.BlockWorkerClientService$Client@109f9bf6
2017-01-17 12:43:50,215 INFO type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.BlockWorkerClientService$Client@5d50b632
2017-01-17 12:43:50,305 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 329 ms.
2017-01-17 12:43:50,414 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 109 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE MUST_CACHE
2017-01-17 12:43:50,435 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_MUST_CACHE took 10 ms.
2017-01-17 12:43:50,449 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_MUST_CACHE took 13 ms.
Passed the test!
runTest Basic CACHE_PROMOTE CACHE_THROUGH
2017-01-17 12:43:50,501 INFO type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.FileSystemWorkerClientService$Client@3018fc1a
2017-01-17 12:43:50,527 INFO type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.FileSystemWorkerClientService$Client@24834237
2017-01-17 12:43:50,845 INFO type (NettyChannelPool.java:createNewResource) - Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true)}, handler: alluxio.client.netty.NettyClient$1@7ce489c6, remoteAddress: svr3404hw1288.hadoop.fat.qa.nt.ctripcorp.com/10.2.26.40:29999).
2017-01-17 12:43:51,199 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 750 ms.
2017-01-17 12:43:51,208 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 9 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE CACHE_THROUGH
2017-01-17 12:43:51,272 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_CACHE_THROUGH took 44 ms.
2017-01-17 12:43:51,281 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_CACHE_THROUGH took 9 ms.
Passed the test!
runTest Basic CACHE_PROMOTE THROUGH
2017-01-17 12:43:51,324 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_THROUGH took 41 ms.
2017-01-17 12:43:51,479 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_THROUGH took 155 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE THROUGH
2017-01-17 12:43:51,535 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_THROUGH took 38 ms.
2017-01-17 12:43:51,583 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_THROUGH took 47 ms.
Passed the test!
runTest Basic CACHE_PROMOTE ASYNC_THROUGH
2017-01-17 12:43:51,619 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_ASYNC_THROUGH took 35 ms.
2017-01-17 12:43:51,630 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_ASYNC_THROUGH took 11 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE ASYNC_THROUGH
2017-01-17 12:43:51,650 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_ASYNC_THROUGH took 11 ms.
2017-01-17 12:43:51,662 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_ASYNC_THROUGH took 12 ms.
Passed the test!
runTest Basic CACHE MUST_CACHE
2017-01-17 12:43:51,677 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_MUST_CACHE took 14 ms.
2017-01-17 12:43:51,690 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_MUST_CACHE took 13 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE MUST_CACHE
2017-01-17 12:43:51,707 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_MUST_CACHE took 8 ms.
2017-01-17 12:43:51,716 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_MUST_CACHE took 9 ms.
Passed the test!
runTest Basic CACHE CACHE_THROUGH
2017-01-17 12:43:51,773 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_CACHE_THROUGH took 56 ms.
2017-01-17 12:43:51,784 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_CACHE_THROUGH took 11 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE CACHE_THROUGH
2017-01-17 12:43:51,844 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_CACHE_THROUGH took 42 ms.
2017-01-17 12:43:51,856 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_CACHE_THROUGH took 11 ms.
Passed the test!
runTest Basic CACHE THROUGH
2017-01-17 12:43:51,906 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_THROUGH took 49 ms.
2017-01-17 12:43:51,948 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_THROUGH took 42 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE THROUGH
2017-01-17 12:43:52,003 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_THROUGH took 36 ms.
2017-01-17 12:43:52,054 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_THROUGH took 51 ms.
Passed the test!
runTest Basic CACHE ASYNC_THROUGH
2017-01-17 12:43:52,070 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_ASYNC_THROUGH took 14 ms.
2017-01-17 12:43:52,080 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_ASYNC_THROUGH took 10 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE ASYNC_THROUGH
2017-01-17 12:43:52,100 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_ASYNC_THROUGH took 12 ms.
2017-01-17 12:43:52,129 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_ASYNC_THROUGH took 28 ms.
Passed the test!
runTest Basic NO_CACHE MUST_CACHE
2017-01-17 12:43:52,144 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_MUST_CACHE took 13 ms.
2017-01-17 12:43:52,152 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_MUST_CACHE took 8 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE MUST_CACHE
2017-01-17 12:43:52,167 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_MUST_CACHE took 8 ms.
2017-01-17 12:43:52,176 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_MUST_CACHE took 8 ms.
Passed the test!
runTest Basic NO_CACHE CACHE_THROUGH
2017-01-17 12:43:52,234 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_CACHE_THROUGH took 56 ms.
2017-01-17 12:43:52,243 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_CACHE_THROUGH took 8 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE CACHE_THROUGH
2017-01-17 12:43:52,303 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_CACHE_THROUGH took 44 ms.
2017-01-17 12:43:52,314 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_CACHE_THROUGH took 10 ms.
Passed the test!
runTest Basic NO_CACHE THROUGH
2017-01-17 12:43:52,362 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_THROUGH took 47 ms.
2017-01-17 12:43:52,394 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_THROUGH took 32 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE THROUGH
2017-01-17 12:43:52,444 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_THROUGH took 34 ms.
2017-01-17 12:43:52,476 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_THROUGH took 32 ms.
Passed the test!
runTest Basic NO_CACHE ASYNC_THROUGH
2017-01-17 12:43:52,492 INFO type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_ASYNC_THROUGH took 15 ms.
2017-01-17 12:43:52,499 INFO type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_ASYNC_THROUGH took 7 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE ASYNC_THROUGH
2017-01-17 12:43:52,517 INFO type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_ASYNC_THROUGH took 10 ms.
2017-01-17 12:43:52,526 INFO type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_ASYNC_THROUGH took 9 ms.
Passed the test!
测试程序运行结束后，就能在hdfs的数据目录下看到该测试程序生成是数据目录和文件
[op1@HOST_NAME_01 current]$ hdfs dfs -ls /alluxio/data_demo
...
Found 1 items
drwxr-xr-x - op1 op1 0 2017-01-17 12:43 /alluxio/data_demo/default_tests_files
在Master的WebUI界面上也能看到上述测试程序生成的数据文件及目录
四、停止Alluxio
./bin/alluxio-stop.sh all