分布式存储fastDFS集群+nginx后端配置
一、FastDFS组成部分及其它名词
1、tracker server
跟踪服务器：用来调度来自客户端的请求。且在内存中记录所有存储组和存储服务器的信息状态。
2、storage server
存储服务器：用来存储文件(data)和文件属性(metadata)
3、client
客户端：业务请求发起方，通过专用接口基于TCP协议与tracker以及storage server进行交互
group
组，也可称为卷：同组内上的文件是完全相同的
文件标识
包括两部分：组名和文件名(包含路径)
meta data
文件相关属性：键值对(Key Value Pair)方式
fid
文件标识符：
2.1 环境：
服务器版本：CentOS release 6.3 (Final)
内核版本：Linux YUNWEI_BIGDATA_SERVER_02 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
fdfs版本：FastDFS_v4.05.tar.gz、libfastcommon_v1.01.tar.gz
2.3 编译libfastcommon
cd fast-dep
tar zxf libfastcommon_v1.01.tar.gz
cd libfastcommon-1.0.1
./make.sh
./make.sh install
2.4 编译FastDFS
cd /home/isvision/fastdfs-dep/
tar -zxvf FastDFS_v4.05.tar.gz
cd FastDFS
./make.sh这里如果maike.sh 执行报错注意可能缺乏libevent-devel ,如下所示：
tracker_service.c:253: error: storage size of ‘ev_notify’ isn’t known
tracker_service.c:258: warning: implicit declaration of function ‘event_set’
tracker_service.c:259: error: ‘EV_READ’ undeclared (first use in this function)
tracker_service.c:259: error: (Each undeclared identifier is reported only once
tracker_service.c:259: error: for each function it appears in.)
tracker_service.c:259: error: ‘EV_PERSIST’ undeclared (first use in this function)
tracker_service.c:260: warning: implicit declaration of function ‘event_base_set’
tracker_service.c:266: warning: implicit declaration of function ‘event_add’
tracker_service.c:275: warning: implicit declaration of function ‘event_base_loop’
tracker_service.c:279: warning: implicit declaration of function ‘event_base_free’
tracker_service.c:253: warning: unused variable ‘ev_notify’
make: *** [tracker_service.o] Error 1安装libevent-devel后解决。
继续执行
./make.sh install2.5 检查安装
fdfs_storage、fdfs_trackerd、fdfs_monitor命令都存在说明fastdfs安装已经完成，命令行输入fdfs，然后按tab键会自动跳出。
3、配置
3.1 创建运行时需要的文件夹
Tracker服务器上需要预先创建的文件夹：
mkdir /data/server/fastdfs/{tracker1,tracker2...}
//一般monitor服务和tracker 装在同一台机器上，所以fastdfs_client文件夹也创建在Tracker服务器上。
Storage服务器上需要预先创建的文件夹：
mkdir /data/server/fastdfs/{storage1,storage2...}
我这里是多track，多storage就不一一举例。
3.2 修改配置文件
#step 1. 将FastDFS/conf/下所有的配置文件拷贝到/etc/fdfs/路径下
[root@dzytest FastDFS]# cp -R /home/isvision/fastdfs-dep/FastDFS/conf /* /etc/fdfs/
#step 2. 修改配置文件，根据机器的角色修改相应的conf，比如tracker机器只要修改tracker.conf，storage机器只要修改storage.conf。Monitor角色是一台监控设备，可以放在FastDFS集群的任何一台机器上（需要fdfs_monitor），通常放在tracker机器上即可。
Tracker：
disabled=false
bind_addr=
port=22122
connect_timeout=30
network_timeout=60
base_path=/data/server/fastdfs/tracker1
max_connections=10000
accept_threads=1
work_threads=10
min_buff_size = 8KB
max_buff_size = 128KB
store_lookup=0
store_group=group1
store_server=2
store_path=0
download_server=0
reserved_storage_space = 10%
log_level=info
run_by_group=yw
run_by_user=dhom
allow_hosts=*
sync_log_buff_interval = 10
check_active_interval = 120
thread_stack_size = 64KB
storage_ip_changed_auto_adjust = true
storage_sync_file_max_delay = 86400
storage_sync_file_max_time = 300
use_trunk_file = false
slot_min_size = 256
slot_max_size = 16MB
trunk_file_size = 64MB
trunk_create_file_advance = false
trunk_create_file_time_base = 02:00
trunk_create_file_interval = 86400
trunk_create_file_space_threshold = 20G
trunk_init_check_occupying = false
trunk_init_reload_from_binlog = false
trunk_compress_binlog_min_interval = 0
use_storage_id = false
storage_ids_filename = storage_ids.conf
id_type_in_filename = ip
store_slave_file_use_link = false
rotate_error_log = false
error_log_rotate_time=00:00
rotate_error_log_size = 0
log_file_keep_days = 0
use_connection_pool = false
connection_pool_max_idle_time = 3600
http.server_port=8080
http.check_alive_interval=30
http.check_alive_type=tcp
http.check_alive_uri=/status.htmlgroup_name=组名 (提示：组名的名字尽量以英文和数据组成，比如：group0,group1)
tracker_server=IP:22122 (提示：IP请输入部署tracker服务器的地址)
storage：
disabled=false
group_name=group1
bind_addr=0.0.0.0
client_bind=true
port=23001
connect_timeout=30
network_timeout=60
heart_beat_interval=30
stat_report_interval=60
base_path=/data/server/fastdfs/storage1
max_connections=10000
buff_size = 256KB
accept_threads=1
work_threads=4
disk_rw_separated = true
disk_reader_threads = 1
disk_writer_threads = 1
sync_wait_msec=50
sync_interval=0
sync_start_time=00:00
sync_end_time=23:59
write_mark_file_freq=500
store_path_count=1
store_path0=/data/server/fastdfs/storage1
subdir_count_per_path=256
tracker_server=192.168.165.13:22122 #track所在的内外IP
tracker_server=192.168.165.2:22122 #track所在的内外IP
log_level=debug
run_by_group=
run_by_user=
allow_hosts=*
file_distribute_path_mode=0
file_distribute_rotate_count=100
fsync_after_written_bytes=0
sync_log_buff_interval=10
sync_binlog_buff_interval=10
sync_stat_file_interval=300
thread_stack_size=512KB
upload_priority=10
if_alias_prefix=
check_file_duplicate=0
file_signature_method=hash
key_namespace=FastDFS
keep_alive=0
use_access_log = false
rotate_access_log = false
access_log_rotate_time=00:00
rotate_error_log = false
error_log_rotate_time=00:00
rotate_access_log_size = 0
rotate_error_log_size = 0
log_file_keep_days = 0
file_sync_skip_invalid_record=false
use_connection_pool = false
connection_pool_max_idle_time = 3600
http.domain_name=
http.server_port=88884、运行
4.1 启动程序
Ø 根据机器角色启动相应的程序(服务启动无先后顺利，但是最先启动tracker,后再启动storage)
tracker程序启动命令:
fdfs_trackerd /etc/fdfs/tracker.conf
storage程序启动命令:
fdfs_storaged /etc/fdfs/storage.conf
Ø 检查程序是否启动
查看端口以及进程运行状态即可
查看运行状态
通过monitor可以查看整个FastDFS系统的状态:
fdfs_monitor /etc/fdfs/client.conf
查看单个服务器的状态
ssh name@ip
tracker:
cat /data/server/fastdfs/tracker1/logs/trackerd.log
storage:
cat /data/server/fastdfs/storage1/logs/storaged.log
例如：
5、测试
方法2:（请在部署tracker服务器上执行！）
首先配置/etc/fdfs/client.conf文件，修改其中的tracker服务器ip。
用如下的命令进行测试:
fdfs_upload_file /etc/fdfs/client.conf 要上传的文件
例如：
fdfs_upload_file /etc/fdfs/client.conf /home/isvision/fastdfs-dep.zip
返回的结果：
group0/M00/00/00/wKhztFXKrV-APnOHABPE0v0bfWo251.zip
实际放入数据的地址：
/data/server/fastdfs/storage1/data/00/00
后端nginx配置：
server {
listen 2000;
server_name 192.168.12.28;
access_log /data/logs/nginx/access.end.log main;
error_log /data/logs/nginx/error.end.log;
location /group1/M00/ {
alias /data/server/fastdfs/storage1/data/;
}
location ~ /group1/M00/(.+)_(\d+)x(\d+)\.(.*)$ {
set $w $2;
set $h $3;
rewrite /group1/M00/(.+)_(\d+)x(\d+)\.(.*)$ /img1/$1.$4 last;
}
location /img1/ {
alias /data/server/fastdfs/storage1/data/;
image_filter resize $w $h;
image_filter_buffer 5M;
}
location /group2/M00/ {
alias /data/server/fastdfs/storage2/data/;
}
location ~ /group2/M00/(.+)_(\d+)x(\d+)\.(.*)$ {
set $w $2;
set $h $3;
rewrite /group2/M00/(.+)_(\d+)x(\d+)\.(.*)$ /img2/$1.$4 last;
}
。
排错一：
2017-12-20 16:45:39] INFO - local_host_ip_count: 2, 127.0.0.1 172.18.126.33
[2017-12-20 16:45:49] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:45:58] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:02] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:06] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:14] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:15] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:16] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:26] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:35] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:38] INFO - file: tracker_relationship.c, line: 383, selecting leader…
[2017-12-20 16:46:42] INFO - file: tracker_relationship.c, line: 383, selecting leader…
track一直在选举，找不到leader
解决方案 ：
停掉所有storage，重启所有track
排错二：
status 5 != 0
[2017-03-23 13:49:13] INFO - file: tracker_mem.c, line: 4213, sys files loaded from tracker server 192.168.10.78:22122
[2017-03-23 13:49:13] ERROR - file: tracker_mem.c, line: 596, in the file "/fastdfs/tracker/data/storage_groups_new.dat", item "group_count" is not found
[2017-03-23 13:49:13] ERROR - file: tracker_mem.c, line: 4297, get sys files from other trackers fail, errno: 2
[2017-03-23 13:49:19] ERROR - file: tracker_proto.c, line: 48, server: 192.168.10.78:22122, response status 5 != 0
[2017-03-23 13:49:19] INFO - file: tracker_mem.c, line: 4213, sys files loaded from tracker server 192.168.10.78:22122
[2017-03-23 13:49:19] ERROR - file: tracker_mem.c, line: 596, in the file "/fastdfs/tracker/data/storage_groups_new.dat", item "group_count" is not found
[2017-03-23 13:49:19] ERROR - file: tracker_mem.c, line: 4297, get sys files from other trackers fail, errno: 2
[2017-03-23 13:49:20] ERROR - file: tracker_proto.c, line: 48, server: 192.168.10.78:22122, response status 5 != 0
[2017-03-23 13:49:20] INFO - file: tracker_mem.c, line: 4213, sys files loaded from tracker server 192.168.10.78:22122
[2017-03-23 13:49:20] ERROR - file: tracker_mem.c, line: 596, in the file "/fastdfs/tracker/data/storage_groups_new.dat", item "group_count" is not found
[2017-03-23 13:49:20] ERROR - file: tracker_mem.c, line: 4297, get sys files from other trackers fail, errno: 2
。
解决方案：
是多个tracker data数据不一致，没有自动同步。
删掉新加入节点的/data目录，拷贝当前Leader tracker目录下的 /data文件 过去即可（.pid不要拷贝）