实战nginx前端反代mogfilefs及负载均衡
===============================================================================
实验描述：
使用Nginx代理请求至tackers，实现通过键就可以访问到文件；
在配置nginx做反代将用户的请求调度至后端的MogileFS，此实验依赖于nginx-mogilefs-module模块，需在编译时加上此模块。
实验环境：
再准备一台CentOS 7的虚拟主机，来作为前端nginx反代服务器，因为要想支持反代MogileFS需要
nginx-mogilefs-module模块，所以，要编译安装nginx；
node1节点仍作为tracker，mogstored和MySQL服务器；node2作为tracker和mogstored节点；
IP地址规划：
node1：192.168.1.112
node2：192.168.1.113
nginx：192.168.1.114
实验拓扑图：
实验步骤如下：
要实现Nginx代理tackers，需要在编译Nginx的时候加入第三方模块nginx_mogilefs_module.
下载地址：http://www.grid.net.ru/nginx/mogilefs.en.html
1.准备开发环境，并解决依赖关系
yum groupinstall "Development Tools" "Server Platfrom Development" -y yum install pcre-devel openssl-devel zlib-devel -y
2.下载编译安装的nginx和第三方模块nginx_mogilefs_module.的程序包，如下：
[root@centos7 nginx-mogilefs]# ls nginx-1.10.0.tar.gz nginx_mogilefs_module-1.0.4.tar.gz
3.编译安装Nginx，并且加入第三方模块nginx_mogilefs_module.
# 解压缩程序包 [root@centos7 nginx-mogilefs]# tar xf nginx-1.10.0.tar.gz [root@centos7 nginx-mogilefs]# tar xf nginx_mogilefs_module-1.0.4.tar.gz [root@centos7 nginx-mogilefs]# ls nginx-1.10.0 nginx-1.10.0.tar.gz nginx_mogilefs_module-1.0.4 nginx_mogilefs_module-1.0.4.tar.gz
创建nginx用户，进入解压之后的nginx-1.10.0目录，执行编译，如下：
# 首先添加nginx用户 [root@centos7 nginx-mogilefs]# useradd -r nginx [root@centos7 nginx-mogilefs]# cd nginx-1.10.0/ [root@centos7 nginx-1.10.0]# ls auto CHANGES CHANGES.ru conf configure contrib html LICENSE man README src [root@centos7 nginx-1.10.0]# ./configure \ --prefix=/usr/local/nginx \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --pid-path=/var/run/nginx/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --user=nginx \ --group=nginx \ --with-http_ssl_module \ --with-http_flv_module \ --with-http_stub_status_module \ --with-http_gzip_static_module \ --http-client-body-temp-path=/var/tmp/nginx/client/ \ --http-proxy-temp-path=/var/tmp/nginx/proxy/ \ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \ --http-scgi-temp-path=/var/tmp/nginx/scgi \ --with-pcre \ --with-debug \ --add-module=/root/nginx-mogilefs/nginx_mogilefs_module-1.0.4 #指明第三方模块的位置即可 # [root@centos7 nginx-1.10.0]# make && make install
注意：
在执行make时提示了一个错误，如下：
通过在网上查询得知，找到nginx目录中/objs/Makefile文件，找到 -Werror 字段，去掉-Werror，重新编译，则问题解决!
编译完成后为nginx提供PATH环境变量
[root@centos7 nginx-1.10.0]# cd /usr/local/nginx/ [root@centos7 nginx]# ls html sbin [root@centos7 nginx]# cd sbin/ [root@centos7 sbin]# ./nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: [emerg] mkdir() "/var/tmp/nginx/client/" failed (2: No such file or directory) nginx: configuration file /etc/nginx/nginx.conf test failed [root@centos7 sbin]# mkdir -p /var/tmp/nginx/client/ # 创建文件 [root@centos7 sbin]# ./nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful #======================================================================================= # 提供环境变量 [root@centos7 ~]# vim /etc/profile.d/nginx.sh export PATH=/usr/local/nginx/sbin:$PATH [root@centos7 ~]# . /etc/profile.d/nginx.sh [root@centos7 ~]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
4.配置nginx可以支持mogilefs模块，在server块中添加location仅使p_w_picpaths向后端代理，编辑nginx的配置文件/etc/nginx/nginx.conf 如下：
5.在node1或node2tracker节点上传文件
[root@centos7 ~]# mogupload --trackers=192.168.1.112 --domain=p_w_picpaths --key='man.jpg' --file='/root/man.jpg' [root@centos7 ~]# mogupload --trackers=192.168.1.113 --domain=p_w_picpaths --key='new.jpg' --file='/root/new.jpg' [root@centos7 ~]# mogfileinfo --trackers=192.168.1.112 --domain=p_w_picpaths --key='man.jpg' - file: man.jpg class: default devcount: 2 domain: p_w_picpaths fid: 14 key: man.jpg length: 3401017 - http://192.168.1.112:7500/dev1/0/000/000/0000000014.fid - http://192.168.1.113:7500/dev3/0/000/000/0000000014.fid [root@centos7 ~]# mogfileinfo --trackers=192.168.1.113 --domain=p_w_picpaths --key='new.jpg' - file: new.jpg class: default devcount: 2 domain: p_w_picpaths fid: 15 key: new.jpg length: 1267947 - http://192.168.1.113:7500/dev3/0/000/000/0000000015.fid - http://192.168.1.112:7500/dev1/0/000/000/0000000015.fid
使用浏览器访问nginx服务器，location后面跟key
注意：
上传文件时， –key不能使用 –key='/man.jpg' ,要使用-key='man.jpg' 因为nginx配置文件中直接定义了 location 为 /p_w_picpaths/ ；如果，要使用-key='/man.jpg'，则在配置文件中的location就要定义为/p_w_picpaths；
由于版本bug使用浏览器上传功能并不能实现，但是删除功能是可以实现的，这里就不再演示，实际生产环境中应用程序都是通过调用API直接联系mogilefs上传的，故此处不必过于纠结；
6.配置upstream，开启负载均衡模式：
# 在http的配置段添加upstream
upstream mogtrackers {
server 192.168.1.112:7001;
server 192.168.1.113:7001;
}
# 修改server配置段中定义的location /p_w_picpaths/
location /p_w_picpaths/ {
mogilefs_tracker mogtrackers;
mogilefs_domain p_w_picpaths;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
模拟node1节点的tracker故障（killall mogilefsd），使用浏览器再次访问nginx服务器，发现也可以正常访问，如下：
nginx 完整的配置文件如下：
[root@centos7 ~]# cat /etc/nginx/nginx.conf
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
upstream mogtrackers {
server 192.168.1.112:7001;
server 192.168.1.113:7001;
}
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
location /p_w_picpaths/ {
mogilefs_tracker mogtrackers;
mogilefs_domain p_w_picpaths;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_name somename alias another.alias;
# location / {
# root html;
# index index.html index.htm;
# }
#}
# HTTPS server
#
#server {
# listen 443 ssl;
# server_name localhost;
# ssl_certificate cert.pem;
# ssl_certificate_key cert.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 5m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# location / {
# root html;
# index index.html index.htm;
# }
#}
}
如上，就是整个Nginx反代MogileFS的整个过程。。。
补充一下mogadm命令的额外指令：
mogadm slave //由于文件元数据信息是保存在MySQL中的，每次请求文件需要读取MySQL一旦请求量太大，MySQL必定会成为性能瓶颈，这时我们要给MySQL做主从，这个指令就是用来添加从节点的，可以实现tracker与主从架构的MySQL交互时，实现读写分离。
mogadm fsck //文件系统检测，一般不要使用，除非集群意外断电，在上线时执行。
mogadm rebalance // 重新平衡，当数据出现热区时使用，在执行之前需要定义平衡策略。
mogadm rebalance start // 启动平衡策略。
mogadm rebalance policy //定义平衡策略。
mogadm rebalance test //用于测试是否出现数据不平衡的问题。
mogadm settings //定义mogilefs工作属性。
mogadm class modify
= //定义文件最小的副本份数 --mindevcount
[root@centos7 ~]# mogadm --help
Usage: (enter any command prefix, leaving off options, for further help)
mogadm check Check the state of the MogileFS world.
mogadm stats Show MogileFS system statistics. (DEPRECATED: use mogstats instead)
mogadm host ...
host add ... Add a host to MogileFS.
host delete ... Delete a host.
host list List all hosts.
host mark ... Change the status of a host. (equivalent to 'modify --status')
host modify ... Modify a host's properties.
mogadm device ...
device add ... Add a device to a host.
device list ... List all devices, for each host.
device mark ... Mark a device as {alive,dead,down,drain,readonly}
device modify ... Modify a device's properties.
device next Show the next available devid.
device summary ... List the summary of devices, for each host.
mogadm domain ...
domain add ... Add a domain (namespace)
domain delete ... Delete a domain.
domain list List all hosts.
mogadm class ...
class add ... Add a file class to a domain.
class delete ... Delete a file class from a domain.
class list List all classes, for each domain.
class modify ... Modify properties of a file class.
mogadm slave ...
slave add ... Add a slave node for store usage
slave delete ... Delete a slave node for store usage
slave list List current store slave nodes.
slave modify ... Modify a slave node for store usage
mogadm fsck ...
fsck clearlog Clear the fsck log
fsck printlog Display the fsck log
fsck reset ... Reset fsck position back to the beginning
fsck start Start (or resume) background fsck
fsck status Show fsck status
fsck stop Stop (pause) background fsck
fsck taillog Tail the fsck log
mogadm rebalance ...
rebalance policy ... Add or adjust the current policy
rebalance reset Reset an existing policy
rebalance settings Display rebalance settings
rebalance start Start a rebalance job
rebalance status Show status of current rebalance job
rebalance stop Stop a rebalance job
rebalance test Show what devices the current policy would match
mogadm settings ...
settings list List all server settings
settings set ... Set server setting 'key' to 'value'.
另附三篇相关博客，在我配置过程中给了我很大帮助：
http://tchuairen.blog.51cto.com/3848118/1543856
http://www.178linux.com/6124
http://www.tianfeiyu.com/?p=1196