一、MogileFS介绍
MogileFS是一个开源的分布式文件存储系统,由LiveJournal旗下的Danga Interactive公司开发。Danga团队开发了包括 Memcached、MogileFS、Perlbal 等多个知名的开源项目。目前使用MogileFS 的公司非常多,如日本排名先前的几个互联公司及国内的yupoo(又拍)、digg、豆瓣、1号店、大众点评、搜狗和安居客等,分别为所在的组织或公司管理着海量的图片。
MogileFS由3个部分组成:
(1) server:主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker,它通过数据库来保存元数据信息,包括站点domain、class、host等;mogstored是存储节点(store node),它其实是个WebDAV服务,默认监听在7500端口,接受客户端的文件存储请求。在MogileFS安装完后,要运行mogadm工具将所有的store node注册到mogilefsd的数据库里,mogilefsd会对这些节点进行管理和监控。
(2) utils(工具集):主要是MogileFS的一些管理工具,例如mogadm等。
(3) 客户端API:MogileFS的客户端API很多,例如Perl、PHP、Java、Python等,用这个模块可以编写客户端程序,实现文件的备份管理功能等。
存储主机(节点)
这个是 MogileFS 存储文件存放在这些机器上,也是 mogstored 节点,也叫 Storage Server,一台存储主要都要启动一个 mogstored 服务.扩容就是增加这些机器.
设备(device)
一个存储节点,以就是上面的主机,可以有多个 device, 就是用来存放文件的目录(例如挂载的目录),每个设备都有一个设备id,需要在 mogstored 的配置文件中的 docroot 配置的项目 指定的目录下面创建相应的设备的目录,目录名为 $docroot/dev$id,设备是不能删除的.只能将其设备的状态的值置为dead,当一个设备 dead 之后,就真的 dead了,里面的数据也无法恢复了,且这个dead了的设备的 id 也不能再用.
二、扑拓图
10.1.2.180 为tracker节点,并且首先安装mysql服务
10.1.2.200-202,总共3台,作为mogstored存储节点
三、180部署步聚
1、安装相关包
[root@logicserver MogileFS]# ll 总用量 620 -rw-r--r--. 1 root root 1916 6月 3 17:03 MogileFS-Server-2.46-2.el6.noarch.rpm -rw-r--r--. 1 root root 176308 6月 3 17:03 MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm -rw-r--r--. 1 root root 26872 6月 3 17:03 MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm -rw-r--r--. 1 root root 75916 6月 3 17:49 MogileFS-Utils-2.19-1.el6.noarch.rpm -rw-r--r--. 1 root root 5880 6月 3 18:04 Perlbal-1.78-1.el6.noarch.rpm -rw-r--r--. 1 root root 1624 6月 3 18:04 Perlbal-doc-1.78-1.el6.noarch.rpm -rw-r--r--. 1 root root 30312 6月 3 17:49 perl-MogileFS-Client-1.14-1.el6.noarch.rpm -rw-r--r--. 1 root root 25140 6月 3 17:49 perl-Net-Netmask-1.9015-8.el6.noarch.rpm -rw-r--r--. 1 root root 268620 6月 3 17:50 perl-Perlbal-1.78-1.el6.noarch.rpm
[root@logicserver MogileFS]# yum install *.rpm Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: MogileFS-Server noarch 2.46-2.el6 /MogileFS-Server-2.46-2.el6.noarch 0.0 MogileFS-Server-mogilefsd noarch 2.46-2.el6 /MogileFS-Server-mogilefsd-2.46-2.el6.noarch 523 k MogileFS-Server-mogstored noarch 2.46-2.el6 /MogileFS-Server-mogstored-2.46-2.el6.noarch 53 k MogileFS-Utils noarch 2.19-1.el6 /MogileFS-Utils-2.19-1.el6.noarch 170 k Perlbal noarch 1.78-1.el6 /Perlbal-1.78-1.el6.noarch 4.4 k Perlbal-doc noarch 1.78-1.el6 /Perlbal-doc-1.78-1.el6.noarch 0.0 perl-MogileFS-Client noarch 1.14-1.el6 /perl-MogileFS-Client-1.14-1.el6.noarch 79 k perl-Net-Netmask noarch 1.9015-8.el6 /perl-Net-Netmask-1.9015-8.el6.noarch 55 k perl-Perlbal noarch 1.78-1.el6 /perl-Perlbal-1.78-1.el6.noarch 666 k Installing for dependencies: perl-BSD-Resource x86_64 1.29.03-3.el6 base 35 k perl-DBD-MySQL x86_64 4.013-3.el6 base 134 k perl-DBI x86_64 1.609-4.el6 base 705 k perl-Danga-Socket noarch 1.61-5.el6 epel 28 k perl-IO-stringy noarch 2.110-10.1.el6 base 68 k perl-Sys-Syscall noarch 0.23-1.el6 epel 14 k perl-Time-HiRes x86_64 4:1.9721-141.el6 base 49 k Transaction Summary
配置文件
[root@logicserver MogileFS]# cd /etc/mogilefs/ [root@logicserver mogilefs]# ls mogilefsd.conf mogstored.conf
查看安装哪些包
[root@logicserver mogilefs]# rpm -qa | grep -i mogilefs MogileFS-Server-mogstored-2.46-2.el6.noarch MogileFS-Server-mogilefsd-2.46-2.el6.noarch perl-MogileFS-Client-1.14-1.el6.noarch MogileFS-Server-2.46-2.el6.noarch MogileFS-Utils-2.19-1.el6.noarch
2、安装MariaDB
[root@logicserver mogilefs]# vim /etc/yum.repos.d/MariaDB.repo [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/5.5/centos6-amd64 gpkgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB gpgcheck=1
手动导入MariaDB的签署密钥
[root@logicserver mogilefs]# rpm --import https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
开始yum安装并启动
[root@logicserver mogilefs]# yum install MariaDB-server MariaDB-client
[root@logicserver mogilefs]# /etc/init.d/mysql start
3、 启动MogileFS前,设定数据库
3.1在数据前授权
MariaDB [mogilefs]> grant all on mogilefs.* to moguser@'10.1.%.%' indentified by 'mog168';
3.2设定数据库
[root@logicserver ~]# mogdbsetup --dbhost=10.1.2.180 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=168 --dbuser=moguser --dbpass=mog168 This will attempt to setup or upgrade your MogileFS database. It won't destroy existing data. Run with --help for more information. Run with --yes to shut up these prompts. Continue? [N/y]: y Create/Upgrade database name 'mogilefs'? [Y/n]: y Grant all privileges to user 'moguser', connecting from anywhere, to the mogilefs database 'mogilefs'? [Y/n]: y Failed to grant privileges: Access denied for user 'root'@'10.1.%.%' to database 'mogilefs'
注意有错提示Failed to grant privileges: Access denied for user 'root'@'10.1.%.%' to database 'mogilefs'
3.3 去数据库查看,已经创建了,但没有tables
MariaDB [mysql]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mogilefs | | mysql | | performance_schema | | test |
MariaDB [mysql]> use mogilefs; Database changed MariaDB [mogilefs]> show tables; Empty set (0.00 sec)
3.4手动创建用户moguser,并授权mogilefs的所有权限
MariaDB [mogilefs]> grant all on mogilefs.* to moguser@'10.1.%.%' identified by 'mog168'; Query OK, 0 rows affected (0.00 sec) MariaDB [mogilefs]> flush privileges; Query OK, 0 rows affected (0.00 sec)
3.5再返回3.2重新执行
[root@logicserver ~]# mogdbsetup --dbhost=10.1.2.180 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=168 --dbuser=moguser --dbpass=mog168 This will attempt to setup or upgrade your MogileFS database. It won't destroy existing data. Run with --help for more information. Run with --yes to shut up these prompts. Continue? [N/y]: y Create/Upgrade database name 'mogilefs'? [Y/n]: y Grant all privileges to user 'moguser', connecting from anywhere, to the mogilefs database 'mogilefs'? [Y/n]: y Failed to grant privileges: Access denied for user 'root'@'10.1.%.%' to database 'mogilefs' [root@logicserver ~]# mogdbsetup --dbhost=10.1.2.180 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=168 --dbuser=moguser --dbpass=mog168 This will attempt to setup or upgrade your MogileFS database. It won't destroy existing data. Run with --help for more information. Run with --yes to shut up these prompts. Continue? [N/y]: y
3.6去数库查看
MariaDB [mogilefs]> show tables; +----------------------+ | Tables_in_mogilefs | +----------------------+ | checksum | | class | | device | | domain | | file | | file_on | | file_on_corrupt | | file_to_delete | | file_to_delete2 | | file_to_delete_later | | file_to_queue | | file_to_replicate | | fsck_log | | host | | server_settings | | tempfile | | unreachable_fids | +----------------------+ 17 rows in set (0.00 sec)
4、修改主配置文件
[root@logicserver ~]# vim /etc/mogilefs/mogilefsd.conf # Enable daemon mode to work in background and use syslog daemonize = 1 # Where to store the pid of the daemon (must be the same in the init script) pidfile = /var/run/mogilefsd/mogilefsd.pid # Database connection information db_dsn = DBI:mysql:mogilefs:host=10.1.2.180 db_user = moguser db_pass = mog168 # IP:PORT to listen on for mogilefs client requests listen = 10.1.2.180:7001 # Optional, if you don't define the port above. conf_port = 7001 # Number of query workers to start by default. query_jobs = 10 # Number of delete workers to start by default. delete_jobs = 1 # Number of replicate workers to start by default. replicate_jobs = 5 # Number of reaper workers to start by default. # (you don't usually need to increase this) reaper_jobs = 1 # Number of fsck workers to start by default. # (these can cause a lot of load when fsck'ing) #fsck_jobs = 1 # Minimum amount of space to reserve in megabytes # default: 100 # Consider setting this to be larger than the largest file you # would normally be uploading. #min_free_space = 200 # Number of seconds to wait for a storage node to respond. # default: 2 # Keep this low, so busy storage nodes are quickly ignored. #node_timeout = 2 # Number of seconds to wait to connect to a storage node. # default: 2 # Keep this low so overloaded nodes get skipped. #conn_timeout = 2 # Allow replication to use the secondary node get port, # if you have apache or similar configured for GET's #repl_use_get_port = 1
5、确保pid文件属主,属组为mogilefs
[root@logicserver ~]# ls -ld /var/run/mogilefsd drwxr-xr-x. 2 mogilefs mogilefs 4096 10月 8 2013 /var/run/mogilefsd
6、启动mogilefsd并确定监听端口为7001
[root@logicserver ~]# /etc/init.d/mogilefsd start Starting mogilefsd [确定]
[root@xenserver1 data0]# netstat -tlnp | grep 7001
四、mogstored 节点
1、200的主机,同样安装这几个rpn包
MogileFS-Server-mogstored-2.46-2.el6.noarch
MogileFS-Server-mogilefsd-2.46-2.el6.noarch
perl-MogileFS-Client-1.14-1.el6.noarch
MogileFS-Server-2.46-2.el6.noarch
MogileFS-Utils-2.19-1.el6.noarch
2、在/data0下创建数据存放位置,并把属主,属组设好
[root@xenserver1 data0]# mkdir nfs/mogdata -pv
[root@xenserver1 data0]# chown -R mogilefs.mogilefs nfs/
3、编辑配置文件
[root@xenserver1 data0]# vim /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /data0/nfs/mogdata
4、启动mogstored,查看监听端口
[root@xenserver1 data0]# /etc/init.d/mogstored start Starting mogstored [确定]
[root@xenserver1 data0]# netstat -tlnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1227/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1436/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1312/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1553/master tcp 0 0 0.0.0.0:55006 0.0.0.0:* LISTEN 1245/rpc.statd tcp 0 0 :::111 :::* LISTEN 1227/rpcbind tcp 0 0 :::22 :::* LISTEN 1436/sshd tcp 0 0 ::1:631 :::* LISTEN 1312/cupsd tcp 0 0 ::1:25 :::* LISTEN 1553/master tcp 0 0 :::38585 :::* LISTEN 1245/rpc.statd
5、找不到监听端口,没起来
[root@xenserver1 data0]# su mogilefs bash-4.1$ mog mogadm mogdbsetup mogfetch mogfileinfo moglistfids mogrename mogautomount mogdelete mogfiledebug mogilefsd moglistkeys mogstats bash-4.1$ mogstored -c /etc/mogilefs/mogstored.conf Running. ERROR: IO::AIO not installed, so async IO not available. Refusing to run unless you set the environment variable MOGSTORED_RUN_WITHOUT_AIO=1
6、从上面错误显示,依赖关系IO-AIO没解决好
[root@xenserver1 data0]# yum -y install perl-IO-AIO
7、安装,重新启动,查看监听端口
[root@xenserver1 data0]# /etc/init.d/mogstored start Starting mogstored [确定]
[root@xenserver1 data0]# ss -tln | grep -E '(7500|7501)' LISTEN 0 128 *:7500 *:* LISTEN 0 128 *:7501 *:*
五、180主机测试
1、用mogadm
[root@logicserver ~]# mogadm -h Usage: (enter any command prefix, leaving off options, for further help) mogadm check Check the state of the MogileFS world. mogadm stats Show MogileFS system statistics. (DEPRECATED: use mogstats instead) mogadm host ... host add ... Add a host to MogileFS. host delete ... Delete a host. host list List all hosts. host mark ... Change the status of a host. (equivalent to 'modify --status') host modify ... Modify a host's properties. mogadm device ... device add ... Add a device to a host. device list ... List all devices, for each host. device mark ... Mark a device as {alive,dead,down,drain,readonly} device modify ... Modify a device's properties. device summary ... List the summary of devices, for each host. mogadm domain ... domain add ... Add a domain (namespace) domain delete ... Delete a domain. domain list List all hosts. mogadm class ... class add ... Add a file class to a domain. class delete ... Delete a file class from a domain. class list List all classes, for each domain. class modify ... Modify properties of a file class. mogadm slave ... slave add ... Add a slave node for store usage slave delete ... Delete a slave node for store usage slave list List current store slave nodes. slave modify ... Modify a slave node for store usage mogadm fsck ... fsck clearlog Clear the fsck log fsck printlog Display the fsck log fsck reset ... Reset fsck position back to the beginning fsck start Start (or resume) background fsck fsck status Show fsck status fsck stop Stop (pause) background fsck fsck taillog Tail the fsck log mogadm rebalance ... rebalance policy ... Add or adjust the current policy rebalance reset Reset an existing policy rebalance settings Display rebalance settings rebalance start Start a rebalance job rebalance status Show status of current rebalance job rebalance stop Stop a rebalance job rebalance test Show what devices the current policy would match mogadm settings ... settings list List all server settings settings set ... Set server setting 'key' to 'value'.
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 check Checking trackers... 10.1.2.180:7001 ... OK Checking hosts... No devices found on tracker(s).
2、
[root@logicserver ~]# mogstats -h Usage: mogstats --db_dsn="DBI:mysql:mfs:host=mfshost" --db_user="mfs" --db_pass="mfs" --verbose --stats="devices,files" mogstats --stats="all" mogstats [all options in ~/.mogilefs.conf] valid stats: all, delete-queue, devices, domains, fids, files, general-queues, replication, replication-queue
3、没有添加进来,查看主机是为空的
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 host list
3.1查看帮助添加主机
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 host add ERROR: Missing argument 'hostname' Help for 'host-add' command: mogadm host add <hostname> [opts] Add a host to MogileFS. <hostname> Hostname of machine --altip=s Alternate IP that is machine is reachable from --altmask=s Netmask which, when matches client, uses alt IP --getport=i Alternate HTTP port serving readonly traffic --ip=s IP address of machine. --port=i HTTP port of mogstored --status=s One of {alive,down}. Default 'down'.
3.2添加主机10.1.2.200
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 host add 10.1.2.200 --ip=10.1.2.200 --status=alive
3.3在查看主机列表
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 host list 10.1.2.200 [1]: alive IP: 10.1.2.200:7500
4、查看帮助和添加设备列表
4.1查看
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device list 10.1.2.200 [1]: alive used(G) free(G) total(G) weight(%)
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device add ERROR: Missing argument 'hostname' Help for 'device-add' command: mogadm device add <hostname> <devid> [opts] Add a device to a host. <devid> Numeric devid. Never reuse these. <hostname> Hostname to add a device --status=s One of 'alive' or 'down'. Defaults to 'alive'.
4.2添加设备,ID号随便取,由于--status默认是alive,可以不指定了
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device add 10.1.2.200 001
4.3查看
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device list 10.1.2.200 [1]: alive used(G) free(G) total(G) weight(%) dev1: alive 0.000 0.000 0.000 100
5、创建domain为files(名称随取)
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 domain add files
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE
六、在201上同样安装这些rpm包
1、
[root@hadoop3 MogileFS]# yum install *.rpm perl-IO-AIO -y
2、
[root@hadoop3 MogileFS]# vim /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /nfs/mogdata ~
3、
[root@hadoop3 MogileFS]# mkdir /nfs/mogdata -pv
[root@hadoop3 MogileFS]# chown -R moigilefs.mogilefs /nfs/mogdata/
4、
[root@hadoop3 MogileFS]# service mogstored start
七、回到180的主机,把10.1.2.201 也添加进去
1、
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 host add 10.1.2.201 --ip=10.1.2.201 --status=alive
2、
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device add 10.1.2.201 002
3、
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 host list 10.1.2.200 [1]: alive IP: 10.1.2.200:7500 10.1.2.201 [2]: alive IP: 10.1.2.201:7500 [root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device list 10.1.2.200 [1]: alive used(G) free(G) total(G) weight(%) dev1: alive 0.000 0.000 0.000 100 10.1.2.201 [2]: alive used(G) free(G) total(G) weight(%) dev2: alive 0.000 0.000 0.000 100
4、上传文件试试
4.1mogupload 用法
注意这里key,访问mogilefs时,所使用的http所访问的URL
[root@logicserver ~]# mogupload -h Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg'
4.2上传一个文件,有错误提示了
[root@logicserver ~]# mogupload --trackers=10.1.2.180:7001 --domain=files --key='fstab.html' --file='/etc/fstab' Error opening MogileFS file: no_devices No devices found to store file at /usr/bin/mogupload line 83, <Sock_10.1.2.180:7001> line 1.
5.回到200主机,创建/data0/nfs/mogdata/dev1
[root@xenserver1 mogdata]# mkdir dev1
[root@xenserver1 mogdata]# chown -R mogilefs.mogilefs dev1/
重启mogstored服务
6、201主机,创建/data0/nfs/mogdata/dev2
[root@hadoop3 mogdata]# mkdir dev2
[root@hadoop3 mogdata]# chown -R mogilefs.mogilefs dev2/
7、
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device list 10.1.2.200 [1]: alive used(G) free(G) total(G) weight(%) dev1: alive 0.064 328.843 328.907 100 10.1.2.201 [2]: alive used(G) free(G) total(G) weight(%) dev2: alive 0.718 2.932 3.649 100
8、这时没有错示了
[root@logicserver ~]# mogupload --trackers=10.1.2.180:7001 --domain=files --key='fstab.html' --file='/etc/fstab'
9、列出所有key
[root@logicserver ~]# moglistkeys --trackers=10.1.2.180:7001 --domain=files fstab.html
10、
[root@logicserver ~]# mogfileinfo --tracker=10.1.2.180:7001 --domain=files --key='fstab.html' - file: fstab.html class: default devcount: 2 domain: files fid: 4 key: fstab.html length: 899 - http://10.1.2.201:7500/dev2/0/000/000/0000000004.fid - http://10.1.2.200:7500/dev1/0/000/000/0000000004.fid
11、把第10步的URL其中一个复制粘贴到浏览器上,即可访问的到
八、常用操作,这里列出两个,可以查看帮助,有详细介绍
1、关闭201存储节点
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device -h Help for 'device' command: (enter any command prefix, leaving off options, for further help) mogadm device add <hostname> <devid> [opts] Add a device to a host. mogadm device list [opts] List all devices, for each host. mogadm device mark <hostname> <devid> <status> Mark a device as {alive,dead,down,drain,readonly} mogadm device modify <hostname> <devid> [opts] Modify a device's properties. mogadm device summary [opts] List the summary of devices, for each host.
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device mark 10.1.2.201 dev2 down
[root@logicserver ~]# mogadm --trackers=10.1.2.180:7001 device list 10.1.2.200 [1]: alive used(G) free(G) total(G) weight(%) dev1: alive 0.064 328.843 328.907 100 10.1.2.201 [2]: alive used(G) free(G) total(G) weight(%) dev2: down 0.718 2.932 3.649 100
2、下载文件到本地
[root@logicserver tmp]# mogfetch -h Usage: /usr/bin/mogfetch --trackers=host --domain=foo --key='/hello.jpg' --file='./output'
[root@logicserver tmp]# mogfetch --trackers=10.1.2.180 --domain=files --key='fstab.html' --file='./output.txt'
[root@logicserver tmp]# cat output.txt # # /etc/fstab # Created by anaconda on Thu Dec 25 19:10:03 2014 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # UUID=17793804-41a2-4cec-b58d-ffec3766aaf3 / ext4 defaults 1 1 UUID=702f98d6-7f82-4b92-8097-5c1c9498ebda /boot ext4 defaults 1 2 UUID=b47bf314-0653-4f4b-a3ca-5f11a8342377 /data0 ext4 defaults 1 2 UUID=a1172ffe-cc17-4639-9c97-a519fc6f0386 swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0