前言
HDFS为管理员提供了针对文件夹的配额控制特性,能够控制
名称配额(指定文件夹下的文件&文件夹总数),或者
空间配额(占用磁盘空间的上限)。
本文探究了HDFS的配额控制特性,记录了各类配额控制场景的实验具体过程。
实验环境基于Apache Hadoop 2.5.0-cdh5.2.0。
欢迎转载,请注明出处:http://blog.csdn.net/u010967382/article/details/44452485
名称配额功能试用
设置名称配额,即当前文件夹下文件和文件夹的最大数量:
casliyang@singlehadoop:~$
hdfs dfsadmin -setQuota 3 /Workspace/quotas/
15/03/18 14:53:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
上传文件:
casliyang@singlehadoop:~$
hdfs dfs -put slf4j-log4j12-1.6.4.jar /Workspace/quotas/
15/03/18 14:54:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
上传文件:
casliyang@singlehadoop:~$
hdfs dfs -put dict.txt /Workspace/quotas/
15/03/18 14:55:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
继续上传文件报错:
casliyang@singlehadoop:~$
hdfs dfs -put examples.desktop /Workspace/quotas/
15/03/18 14:55:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put:
The NameSpace quota (directories and files) of directory /Workspace/quotas is exceeded:
quota=3 file count=4
查看该文件夹情况:
casliyang@singlehadoop:~$ hdfs dfs -ls /Workspace/quotas
15/03/18 17:11:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 3 casliyang supergroup 14 2015-03-18 14:55 /Workspace/quotas/dict.txt
-rw-r--r-- 3 casliyang supergroup 9748 2015-03-18 14:38 /Workspace/quotas/slf4j-log4j12-1.6.4.jar
文件夹下仅仅有两个文件。
查看配额情况:
casliyang@singlehadoop:~$ hdfs dfs
-count -q /Workspace/quotas
15/03/18 16:00:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3 0 none inf 1 2 9762 /Workspace/quotas
注意几个数字的含义依次是:
- 名称配额总量(none代表没设定)
- 名称配额剩余量(inf代表没设定)
- 空间配额总量(none代表没设定)
- 空间配额剩余量(inf代表没设定)
- 文件夹数
- 文件数
- 内容占用空间
- 目标地址
名称配额剩余量的计算公式:
名称配额剩余量 = 名称配额总量 - ( 文件夹数 + 文件数 )
基于上面配额查看结果得知:
名称配额总量=3
文件夹数=1
文件数=2
所以,名称配额剩余量=3-(1+2)=0
所以此时继续上传文件会超出名称配额的限制。
BTW,竟然把根文件夹也计数了!
我们再来建一个没有设定quotas的文件夹,測试下是否也将根文件夹纳入count统计:
casliyang@singlehadoop:~$
hdfs dfs -mkdir /Workspace/quotas1
15/03/18 17:19:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
casliyang@singlehadoop:~$
hdfs dfs -count -q /Workspace/quotas1
15/03/18 17:20:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
none inf none inf
1 0 0 /Workspace/quotas1
上面的实验看出,名称配额和空间配额都没设置,所以都显示none/inf,果然当前根文件夹还是计入统计了,文件夹数为1。
以下通过命令取消
/Workspace/quotas文件夹的名称配额:
casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas
15/03/18 17:28:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3
0 none inf 1 2 9762 /Workspace/quotas
casliyang@singlehadoop:~$
hdfs dfsadmin -clrQuota /Workspace/quotas
15/03/18 17:28:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas
15/03/18 17:28:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
none
inf none inf 1 2 9762 /Workspace/quotas
空间配额功能试用
清空文件夹
/Workspace/quotas:
casliyang@singlehadoop:~$
hdfs dfs -rm -r /Workspace/quotas/*
15/03/18 17:33:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/18 17:33:11 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /Workspace/quotas/dict.txt
15/03/18 17:33:11 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /Workspace/quotas/slf4j-log4j12-1.6.4.jar
casliyang@singlehadoop:~$
hdfs dfs -ls /Workspace/quotas/
15/03/18 17:33:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
casliyang@singlehadoop:~$
hdfs dfs -count -q /Workspace/quotas
15/03/18 17:33:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
none inf
none inf 1 0
0 /Workspace/quotas
为文件夹设定空间限额:
casliyang@singlehadoop:~$
hdfs dfsadmin -setSpaceQuota 8000 /Workspace/quotas
15/03/18 17:36:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas
15/03/18 17:36:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
none inf
8000 8000 1 0
0 /Workspace/quotas
为
/Workspace/quotas文件夹设定空间配额为8000字节,能够看到,空间配额总量8000,空间配额剩余量8000,已用空间配额0。
以下尝试上传一个大小超过8000字节的文件
slf4j-log4j12-1.6.4.jar,该文件大小是9748字节:
casliyang@singlehadoop:~$ ll slf4j-log4j12-1.6.4.jar
-rw-r--r-- 1 casliyang casliyang
9748 Mar 6 14:50 slf4j-log4j12-1.6.4.jar
casliyang@singlehadoop:~$ hdfs dfs -put
slf4j-log4j12-1.6.4.jar /Workspace/quotas
15/03/18 17:40:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/18 17:40:36 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException:
The DiskSpace quota of /Workspace/quotas is exceeded:
quota = 8000 B = 7.81 KB but diskspace consumed = 402653184 B = 384 MB
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyDiskspaceQuota(DirectoryWithQuotaFeature.java:144)
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:154)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1815)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1650)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1625)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:373)
会报错是预料之中,可是最后一句话磁盘空间消耗了384MB,这有点出乎预料,经查阅资料得知:
我们须要设置的空间配额不是文件本身的大小,而是block占用的磁盘的最大值,比方,上面实验中的文件大小是
9748
字节,即9.5k左右,hdfs-site.xml中配置的block大小是128MB,副本数是3,文件能够存储在1个block内,所以需占用的磁盘总量最大值(即block的size)是128*3=384MB!
我们改动文件夹的配额,将其设置为384MB:
casliyang@singlehadoop:~$ hdfs dfsadmin -setSpaceQuota
384m /Workspace/quotas
再次上传文件:
casliyang@singlehadoop:~$ hdfs dfs -put slf4j-log4j12-1.6.4.jar /Workspace/quotas
15/03/19 09:02:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
casliyang@singlehadoop:~$ hdfs dfs -count -q /Workspace/quotas
15/03/19 09:03:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
none inf
402653184 402623940 1 1
9748 /Workspace/quotas
可见,上传成功,
空间配额是402653184
,剩余配额是
402623940
,内容占用空间9748。
这三个数字的计算关系是 402653184 - 9748 *3 = 402623940 !
即使剩余空间显示还剩
402623940
,可是已经无法继续上传不论什么东西了,由于用于存9748文件的3个128MB的block已经撑满了磁盘空间配额!
******吐槽一句,这么个显示模式真心不友好,不明确规则的用户肯定看不懂!****
不死心,再尝试是否还能上传文件,用作实验的是一个非常小的文件,仅仅有
14bytes:
casliyang@singlehadoop:~$ hdfs dfs -put
dict.txt /Workspace/quotas
15/03/19 09:04:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/19 09:04:33 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /Workspace/quotas is exceeded: quota = 402653184 B =
384 MB but diskspace consumed = 402682428 B =
384.03 MB
无法上传,的确是配额满了。
要点总结
- HDFS能够为指定文件夹设置名称配额Name Quotas和空间配额Space Quotas。
- Name Quotas控制指定根文件夹下的全部文件夹和文件数量(详细计算规则见上文),Space Quotas控制指定根文件夹下的全部文件占用空间(详细计算规则见上文)。
- 依据官网说明,名称配额和空间配额的最大值是Long.Max_Value。
- 用途1:结合ACL和Quotas精细化控制用户对HDFS的訪问权限;用途2:控制某些程序模块对HDFS的写权限。