分布式存储fastDFS集群+nginx后端配置

一、FastDFS组成部分及其它名词

1、tracker server

跟踪服务器：用来调度来自客户端的请求。且在内存中记录所有存储组和存储服务器的信息状态。

2、storage server

存储服务器：用来存储文件(data)和文件属性(metadata)

3、client

客户端：业务请求发起方，通过专用接口基于TCP协议与tracker以及storage server进行交互

group

组，也可称为卷：同组内上的文件是完全相同的

文件标识

包括两部分：组名和文件名(包含路径)

meta data

文件相关属性：键值对(Key Value Pair)方式

fid

文件标识符：

2.1 环境：

服务器版本：CentOS release 6.3 (Final)

内核版本：Linux YUNWEI_BIGDATA_SERVER_02 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

fdfs版本：FastDFS_v4.05.tar.gz、libfastcommon_v1.01.tar.gz

2.3 编译libfastcommon

    cd fast-dep

tar zxf libfastcommon_v1.01.tar.gz

cd libfastcommon-1.0.1

./make.sh

./make.sh install

2.4 编译FastDFS

cd   /home/isvision/fastdfs-dep/

tar -zxvf  FastDFS_v4.05.tar.gz         

cd FastDFS

./make.sh

这里如果maike.sh 执行报错注意可能缺乏libevent-devel ,如下所示：

tracker_service.c:253: error: storage size of ‘ev_notify’ isn’t known
tracker_service.c:258: warning: implicit declaration of function ‘event_set’
tracker_service.c:259: error: ‘EV_READ’ undeclared (first use in this function)
tracker_service.c:259: error: (Each undeclared identifier is reported only once
tracker_service.c:259: error: for each function it appears in.)
tracker_service.c:259: error: ‘EV_PERSIST’ undeclared (first use in this function)
tracker_service.c:260: warning: implicit declaration of function ‘event_base_set’
tracker_service.c:266: warning: implicit declaration of function ‘event_add’
tracker_service.c:275: warning: implicit declaration of function ‘event_base_loop’
tracker_service.c:279: warning: implicit declaration of function ‘event_base_free’
tracker_service.c:253: warning: unused variable ‘ev_notify’
make: *** [tracker_service.o] Error 1

安装libevent-devel后解决。

继续执行

./make.sh install

2.5 检查安装

fdfs_storage、fdfs_trackerd、fdfs_monitor命令都存在说明fastdfs安装已经完成，命令行输入fdfs，然后按tab键会自动跳出。

3、配置

3.1 创建运行时需要的文件夹

Tracker服务器上需要预先创建的文件夹：

mkdir  /data/server/fastdfs/{tracker1,tracker2...}

       //一般monitor服务和tracker 装在同一台机器上，所以fastdfs_client文件夹也创建在Tracker服务器上。

Storage服务器上需要预先创建的文件夹：

mkdir  /data/server/fastdfs/{storage1,storage2...}

我这里是多track，多storage就不一一举例。

3.2 修改配置文件

#step 1. 将FastDFS/conf/下所有的配置文件拷贝到/etc/fdfs/路径下

[root@dzytest FastDFS]# cp -R /home/isvision/fastdfs-dep/FastDFS/conf /*  /etc/fdfs/

#step 2. 修改配置文件，根据机器的角色修改相应的conf，比如tracker机器只要修改tracker.conf，storage机器只要修改storage.conf。Monitor角色是一台监控设备，可以放在FastDFS集群的任何一台机器上（需要fdfs_monitor），通常放在tracker机器上即可。

Tracker：

disabled=false
bind_addr=
port=22122
connect_timeout=30
network_timeout=60
base_path=/data/server/fastdfs/tracker1
max_connections=10000
accept_threads=1
work_threads=10
min_buff_size = 8KB
max_buff_size = 128KB
store_lookup=0
store_group=group1
store_server=2
store_path=0
download_server=0
reserved_storage_space = 10%
log_level=info
run_by_group=yw
run_by_user=dhom
allow_hosts=*
sync_log_buff_interval = 10
check_active_interval = 120
thread_stack_size = 64KB
storage_ip_changed_auto_adjust = true
storage_sync_file_max_delay = 86400
storage_sync_file_max_time = 300
use_trunk_file = false 
slot_min_size = 256
slot_max_size = 16MB
trunk_file_size = 64MB
trunk_create_file_advance = false
trunk_create_file_time_base = 02:00
trunk_create_file_interval = 86400
trunk_create_file_space_threshold = 20G
trunk_init_check_occupying = false
trunk_init_reload_from_binlog = false
trunk_compress_binlog_min_interval = 0
use_storage_id = false
storage_ids_filename = storage_ids.conf
id_type_in_filename = ip
store_slave_file_use_link = false
rotate_error_log = false
error_log_rotate_time=00:00
rotate_error_log_size = 0
log_file_keep_days = 0
use_connection_pool = false
connection_pool_max_idle_time = 3600
http.server_port=8080
http.check_alive_interval=30
http.check_alive_type=tcp
http.check_alive_uri=/status.html

group_name=组名    (提示：组名的名字尽量以英文和数据组成，比如：group0,group1)

tracker_server=IP:22122  (提示：IP请输入部署tracker服务器的地址)

storage：

disabled=false
group_name=group1
bind_addr=0.0.0.0
client_bind=true
port=23001
connect_timeout=30
network_timeout=60
heart_beat_interval=30
stat_report_interval=60
base_path=/data/server/fastdfs/storage1
max_connections=10000
buff_size = 256KB
accept_threads=1
work_threads=4
disk_rw_separated = true
disk_reader_threads = 1
disk_writer_threads = 1
sync_wait_msec=50
sync_interval=0
sync_start_time=00:00
sync_end_time=23:59
write_mark_file_freq=500
store_path_count=1
store_path0=/data/server/fastdfs/storage1
subdir_count_per_path=256
tracker_server=192.168.165.13:22122   #track所在的内外IP
tracker_server=192.168.165.2:22122    #track所在的内外IP
log_level=debug
run_by_group=
run_by_user=
allow_hosts=*
file_distribute_path_mode=0
file_distribute_rotate_count=100
fsync_after_written_bytes=0
sync_log_buff_interval=10
sync_binlog_buff_interval=10
sync_stat_file_interval=300
thread_stack_size=512KB
upload_priority=10
if_alias_prefix=
check_file_duplicate=0
file_signature_method=hash
key_namespace=FastDFS
keep_alive=0
use_access_log = false
rotate_access_log = false
access_log_rotate_time=00:00
rotate_error_log = false
error_log_rotate_time=00:00
rotate_access_log_size = 0
rotate_error_log_size = 0
log_file_keep_days = 0
file_sync_skip_invalid_record=false
use_connection_pool = false
connection_pool_max_idle_time = 3600
http.domain_name=
http.server_port=8888

4、运行

4.1  启动程序

Ø 根据机器角色启动相应的程序(服务启动无先后顺利，但是最先启动tracker,后再启动storage)

tracker程序启动命令:

fdfs_trackerd /etc/fdfs/tracker.conf

storage程序启动命令:

fdfs_storaged /etc/fdfs/storage.conf

Ø 检查程序是否启动

查看端口以及进程运行状态即可

 

查看运行状态

通过monitor可以查看整个FastDFS系统的状态:

fdfs_monitor /etc/fdfs/client.conf

查看单个服务器的状态

ssh name@ip

tracker:

cat /data/server/fastdfs/tracker1/logs/trackerd.log

storage: 

cat /data/server/fastdfs/storage1/logs/storaged.log

例如：

5、测试

方法2:（请在部署tracker服务器上执行！）

首先配置/etc/fdfs/client.conf文件，修改其中的tracker服务器ip。

用如下的命令进行测试:

fdfs_upload_file /etc/fdfs/client.conf  要上传的文件

例如：

fdfs_upload_file /etc/fdfs/client.conf  /home/isvision/fastdfs-dep.zip

返回的结果：

group0/M00/00/00/wKhztFXKrV-APnOHABPE0v0bfWo251.zip

实际放入数据的地址：

/data/server/fastdfs/storage1/data/00/00

后端nginx配置：

server {

listen 2000;

server_name 192.168.12.28;

access_log /data/logs/nginx/access.end.log main;

error_log /data/logs/nginx/error.end.log;

location /group1/M00/ {

alias /data/server/fastdfs/storage1/data/;

}

location ~ /group1/M00/(.+)_(\d+)x(\d+)\.(.*)$ {

set $w $2;

set $h $3;

rewrite /group1/M00/(.+)_(\d+)x(\d+)\.(.*)$ /img1/$1.$4 last;

}

location /img1/ {

alias /data/server/fastdfs/storage1/data/;

image_filter resize $w $h;

image_filter_buffer 5M;

}

location /group2/M00/ {

alias /data/server/fastdfs/storage2/data/;

}

location ~ /group2/M00/(.+)_(\d+)x(\d+)\.(.*)$ {

set $w $2;

set $h $3;

rewrite /group2/M00/(.+)_(\d+)x(\d+)\.(.*)$ /img2/$1.$4 last;

}

。

排错一：

2017-12-20 16:45:39] INFO - local_host_ip_count: 2, 127.0.0.1 172.18.126.33 

[2017-12-20 16:45:49] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:45:58] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:02] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:06] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:14] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:15] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:16] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:26] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:35] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:38] INFO - file: tracker_relationship.c, line: 383, selecting leader… 

[2017-12-20 16:46:42] INFO - file: tracker_relationship.c, line: 383, selecting leader…

track一直在选举，找不到leader

解决方案 ：

停掉所有storage，重启所有track

排错二：

status 5 != 0

[2017-03-23 13:49:13] INFO - file: tracker_mem.c, line: 4213, sys files loaded from tracker server 192.168.10.78:22122

[2017-03-23 13:49:13] ERROR - file: tracker_mem.c, line: 596, in the file "/fastdfs/tracker/data/storage_groups_new.dat", item "group_count" is not found

[2017-03-23 13:49:13] ERROR - file: tracker_mem.c, line: 4297, get sys files from other trackers fail, errno: 2

[2017-03-23 13:49:19] ERROR - file: tracker_proto.c, line: 48, server: 192.168.10.78:22122, response status 5 != 0

[2017-03-23 13:49:19] INFO - file: tracker_mem.c, line: 4213, sys files loaded from tracker server 192.168.10.78:22122

[2017-03-23 13:49:19] ERROR - file: tracker_mem.c, line: 596, in the file "/fastdfs/tracker/data/storage_groups_new.dat", item "group_count" is not found

[2017-03-23 13:49:19] ERROR - file: tracker_mem.c, line: 4297, get sys files from other trackers fail, errno: 2

[2017-03-23 13:49:20] ERROR - file: tracker_proto.c, line: 48, server: 192.168.10.78:22122, response status 5 != 0

[2017-03-23 13:49:20] INFO - file: tracker_mem.c, line: 4213, sys files loaded from tracker server 192.168.10.78:22122

[2017-03-23 13:49:20] ERROR - file: tracker_mem.c, line: 596, in the file "/fastdfs/tracker/data/storage_groups_new.dat", item "group_count" is not found

[2017-03-23 13:49:20] ERROR - file: tracker_mem.c, line: 4297, get sys files from other trackers fail, errno: 2

。

解决方案：

是多个tracker   data数据不一致，没有自动同步。

删掉新加入节点的/data目录，拷贝当前Leader tracker目录下的 /data文件 过去即可（.pid不要拷贝）