fastdfs图片服务器搭建
一. 准备环境:
| 服务器角色 | IP |
| image fastdfs_tracker_server | 10.1.11.35 |
| 10.1.11.36 | |
| image fastdfs_storage_server | 10.1.11.35 |
| 10.1.11.36 | |
| image nginx_with_fdfs_mod_server | 10.1.11.35 |
| 10.1.11.36 | |
| news fastdfs_tracker_server | 10.1.12.51 |
| 10.1.12.52 | |
| news fastdfs_storage_server | 10.1.12.51 |
| 10.1.12.52 | |
| news nginx_with_fdfs_mod_server | 10.1.12.51 |
| 10.1.12.52 | |
| tenginx | 10.1.11.22 |
客户端通过调用java client api的上传接口完成文件上传.
内部流程是:
1) client api请求tracker server,
2) tracker server根据storage server的连接数,磁盘空间,负载等计算选择一台storage server将其ip返回client api
3) client api向storage server发送上传请求, 完成文件上传
4) storage server向client api返回file id(storage server本地的文件逻辑路径,如:
M00/00/FB/CgELI1XqgSuAWgRfAAD7wSFTiOA15.jpeg,其中:
M00: fastdfs中的pathid
00/FB: 在M00/下创建的两级目录, 这里为每个path创建了256*256个目录
CgELI1XqgSuAWgRfAAD7wSFTiOA15.jpeg: fastdfs根据文件创建时间,大小,storageserver ip,crc32,随机数组合生成文件名)
1) client发送http请求tengine
2) tengine按照设定负载均衡策略将请求代理到后端带fastdfs模块的nginx服务器
3) nginx检查本地是否存在http请求的资源,有则直接返回client, 没有则调用fastdfs的下载api向指定的storage server获取资源后再返回client
下载fastdfs安装包和依赖:
FastDFS_v5.05.tar.gz ------>fastdfs服务,包括tracker,storage
fastdfs-nginx-module_v1.16.tar.gz ------>fastdfs的nginx模块
libfastcommon-master.zip ------>fastdfs的依赖库
nginx-1.8.0.tar.gz ------>nginx
二. 安装部署:
这里使用shell脚本完成一键安装:
首先将fastdfs所需的软件包打包成如下目录:
[root@test fastdfs_install_pkg]# tree -L 1 . . ├── client.conf ├── fastdfs-nginx-module_v1.16.tar.gz ├── FastDFS_v5.05.tar.gz ├── http.conf ├── install_fastdfs.sh ├── libfastcommon-master.zip ├── mime.types ├── mod_fastdfs.conf ├── nginx-1.8.0.tar.gz ├── nginx.conf ├── README.txt ├── storage.conf └── tracker.conf
其中, 里面几个.conf文件用户需要根据自身的部署环境进行相应的修改.
在需要部署fastdfs的所有节点下, cd到fastdfs_install_pkg目录下,执行./install_fastdfs.sh即可完成安装.以下是./install_fastdfs.sh的内容:
[root@test fastdfs_install_pkg]# cat install_fastdfs.sh
#!/bin/bash
#
#set -e
#fastdfs安装包,依赖包所在目录
base_src_path=/usr/local/src/fastdfs_install_pkg
required1=libfastcommon-master.zip
required2=FastDFS_v5.05.tar.gz
required3=nginx-1.8.0.tar.gz
required4=fastdfs-nginx-module_v1.16.tar.gz
#指定nginx的安装路径,启动用户和组
nginx_base_dir=/usr/local/nginx
nginx_user=www
nginx_group=www
#指定fastdfs启动的用户和组
fdfs_user=fdfs
fdfs_group=fdfs
#指定fastdfs的数据目录路径
fdfs_data_path=/data/fdfs
#指定storage组名
storage_groupname=group1
#指定storage路径名
storage_pathname=("html1" "html2" "html3" "html4" "html5")
index=0
errors=()
#check required
cd $base_src_path
[ -f $required1 ] || (echo "$require1 is require, but not found" && exit 1)
[ -f $required2 ] || (echo "$require2 is require, but not found" && exit 1)
[ -f $required3 ] || (echo "$require3 is require, but not found" && exit 1)
#install libfastcommon
function install_libfastcommon() {
cd $base_src_path
unzip -qo $required1
cd ${required1%%.*}
./make.sh
./make.sh install
ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
}
#install fastdfs
function install_fastdfs() {
cd $base_src_path
id -g $fdfs_group &>/dev/null || groupadd $fdfs_group
id -u $fdfs_user &>/dev/null || useradd -M -r -g $fdfs_group -s /sbin/nologin $fdfs_user
mkdir -p $fdfs_data_path/{tracker,storage,client}
for pathname in ${storage_pathname[@]}; do
mkdir -p $fdfs_data_path/storage/${storage_groupname}/${pathname}
done
chown -R ${fdfs_user}:${fdfs_group} $fdfs_data_path
tar xf $required2
cd ${required2%%_*}
./make.sh
./make.sh install
mkdir -p /etc/fdfs/sample && /bin/mv /etc/fdfs/*.sample /etc/fdfs/sample
/bin/cp ${base_src_path}/*.conf /etc/fdfs
}
#install nginx with fdfs_mod
function install_nginx_with_fdfsmod() {
cd $base_src_path
rpm -q pcre-devel &>/dev/null || yum -y install pcre-devel
id -g $nginx_group || groupadd $nginx_group
id -u $nginx_user || useradd -M -r -g $nginx_group -s /sbin/nologin $nginx_user
tar xf $required3
tar xf $required4
sed -i '/CORE_INCS/ s#=.*#=\"\$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon\"#' ${required4%%_*}/src/config
cd ${required3%%.tar.*}
./configure --prefix=/usr/local/nginx --with-pcre --with-http_ssl_module --add-module=../${required4%%_*}/src/ --user=${nginx_user} --group=${nginx_group}
make -j `cat /proc/cpuinfo | grep processor| wc -l` && make install
/bin/cp ${base_src_path}/nginx.conf /usr/local/nginx/conf
}
function usage() {
echo "Usage: ./$(basename $0) [ fastdfs ] [ nginx ] [ libfastcommon ]"
exit 1
}
if [ $# -eq 0 ]; then
install_libfastcommon
install_fastdfs
install_nginx_with_fdfsmod
[ $? -eq 0 ] && echo -e "...\nfinish" && exit 0
exit 1
fi
for arg in $@; do
case $arg in
"libfastcommon")
install_libfastcommon;;
"fastdfs")
install_fastdfs;;
"nginx")
install_nginx_with_fdfsmod;;
"-h"|"--help")
usage;;
*)
errors[${index}]=$arg
index=$[ ${index} + 1 ]
esac
done
if [ ${#errors[@]} -ne 0 ]; then
for err in ${errors[@]}; do
echo -e "Error: unknown argument $err"
done
usage
fi
三. 配置文件说明
tracker.conf:
[root@test fastdfs_install_pkg]# cat tracker.conf disabled=false bind_addr= #服务端口 port=22122 connect_timeout=5 network_timeout=5 #tracker的数据目录路径 base_path=/data/fdfs/tracker max_connections=1536 accept_threads=1 work_threads=24 store_lookup=2 store_group=group1 store_server=0 store_path=0 download_server=0 reserved_storage_space = 4G log_level=debug #运行用户和组 run_by_group=fdfs run_by_user=fdfs allow_hosts=* sync_log_buff_interval = 10 check_active_interval = 120 thread_stack_size = 64KB storage_ip_changed_auto_adjust = true storage_sync_file_max_delay = 86400 storage_sync_file_max_time = 300 use_trunk_file = false slot_min_size = 256 slot_max_size = 16MB trunk_file_size = 64MB trunk_create_file_advance = false trunk_create_file_time_base = 02:00 trunk_create_file_interval = 86400 trunk_create_file_space_threshold = 20G trunk_init_check_occupying = false trunk_init_reload_from_binlog = false trunk_compress_binlog_min_interval = 0 use_storage_id = false storage_ids_filename = storage_ids.conf id_type_in_filename = ip store_slave_file_use_link = false rotate_error_log = false error_log_rotate_time=00:00 rotate_error_log_size = 0 log_file_keep_days = 30 use_connection_pool = true connection_pool_max_idle_time = 3600 http.server_port=8080 http.check_alive_interval=30 http.check_alive_type=tcp http.check_alive_uri=/status.html
storage.conf:
[root@test fastdfs_install_pkg]# cat storage.conf disabled=false #指定存储组的名称 group_name=group1 bind_addr= client_bind=true #服务端口 port=23000 connect_timeout=30 network_timeout=30 heart_beat_interval=30 stat_report_interval=60 #storage节点的数据目录 base_path=/data/fdfs/storage max_connections=1536 buff_size = 256KB accept_threads=1 work_threads=24 disk_rw_separated = true disk_reader_threads = 10 disk_writer_threads = 1 sync_wait_msec=50 sync_interval=0 sync_start_time=00:00 sync_end_time=23:59 write_mark_file_freq=500 #指定存储path的个数和路径 store_path_count=5 store_path0=/data/fdfs/storage/group1/image0 store_path1=/data/fdfs/storage/group1/image1 store_path2=/data/fdfs/storage/group1/image2 store_path3=/data/fdfs/storage/group1/image3 store_path4=/data/fdfs/storage/group1/image4 #指定每个存储path中每个目录层级下创建的目录数 subdir_count_per_path=256 #指定所有的tracker server ip和端口 tracker_server=10.1.12.51:22122 tracker_server=10.1.12.52:22122 log_level=debug #指定运行用户和组 run_by_group=fdfs run_by_user=fdfs allow_hosts=* file_distribute_path_mode=0 file_distribute_rotate_count=100 fsync_after_written_bytes=0 sync_log_buff_interval=10 sync_binlog_buff_interval=10 sync_stat_file_interval=300 thread_stack_size=512KB upload_priority=10 if_alias_prefix= check_file_duplicate=0 file_signature_method=hash key_namespace=FastDFS keep_alive=0 use_access_log = false rotate_access_log = false access_log_rotate_time=00:00 rotate_error_log = false error_log_rotate_time=00:00 rotate_access_log_size = 0 rotate_error_log_size = 0 log_file_keep_days = 0 file_sync_skip_invalid_record=false use_connection_pool = true connection_pool_max_idle_time = 3600 http.domain_name= http.server_port=8888
client.conf :
[root@test fastdfs_install_pkg]# cat client.conf connect_timeout=5 network_timeout=5 #指定client的数据目录 base_path=/data/fdfs/client #指定tracker server的ip和端口 tracker_server=10.1.11.35:22122 tracker_server=10.1.11.36:22122 log_level=info use_connection_pool = true connection_pool_max_idle_time = 3600 load_fdfs_parameters_from_tracker=false use_storage_id = false storage_ids_filename = storage_ids.conf http.tracker_server_port=80
mod_fastdfs.conf :
[root@test fastdfs_install_pkg]# egrep -v '^$|^#' mod_fastdfs.conf connect_timeout=2 network_timeout=30 base_path=/tmp load_fdfs_parameters_from_tracker=false storage_sync_file_max_delay = 86400 use_storage_id = false storage_ids_filename = storage_ids.conf #指定tracker server的ip tracker_server=192.168.124.43:22122 tracker_server=192.168.124.44:22122 #指定storage server的端口,storage group,path storage_server_port=23000 group_name=vol1 url_have_group_name = false store_path_count=5 store_path0=/data/fdfs/storage/group1/image0 store_path1=/data/fdfs/storage/group1/image1 store_path2=/data/fdfs/storage/group1/image2 store_path3=/data/fdfs/storage/group1/image3 store_path4=/data/fdfs/storage/group1/image4 log_level=debug log_filename= response_mode=proxy if_alias_prefix= flv_support = true flv_extension = flv
nginx.conf:
[root@test fastdfs_install_pkg]# egrep -v '^[[:space:]]*#|^$' nginx.conf
user www www;
worker_processes 24;
pid logs/nginx.pid;
events {
worker_connections 51200;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 8090;
server_name localhost;
expires 5m;
add_header Cache-Control 300;
location /group1/M00 {
root /data/fdfs/storage/group1/image0/data;
ngx_fastdfs_module;
}
location /group1/M01 {
root /data/fdfs/storage/group1/image1/data;
ngx_fastdfs_module;
}
location /group1/M02 {
root /data/fdfs/storage/group1/image2/data;
ngx_fastdfs_module;
}
location /group1/M03 {
root /data/fdfs/storage/group1/image3/data;
ngx_fastdfs_module;
}
location /group1/M04 {
root /data/fdfs/storage/group1/image4/data;
ngx_fastdfs_module;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
tengine中的fastdfs代理配置:
upstream部分:
upstream imagefdfs {
check interval=500 rise=2 fall=5 timeout=3000 type=tcp;
server 10.1.11.35:8090 max_fails=10 fail_timeout=10s;
server 10.1.11.36:8090 max_fails=10 fail_timeout=10s;
}
vhost和url location配置部分:
server
{
listen 80;
server_name image.test.com;
index index.html index.htm index.php;
charset utf-8;
expires 5m;
add_header Cache-Control 300;
location ~ ^/group[0-9]+/M[0-9]+/.*\.(jpeg|jpg|gif|png)$ {
proxy_pass http://imagefdfs;
}
access_log /data/nginx/logs/image.access.log;
}
四. 服务启动运行:
通过前面的安装, fastdfs的配置文件都位于/etc/fdfs目录下:
[root@test fdfs]# tree .
/etc/fdfs
├── client.conf
├── http.conf
├── mime.types
├── mod_fastdfs.conf
├── sample
│ ├── client.conf.sample
│ ├── storage.conf.sample
│ └── tracker.conf.sample
├── storage.conf
└── tracker.conf
启动trackerd server:
$(which fdfs_trackerd) /etc/tracker.conf start
启动storage的server:
$(which fdfs_storaged) /etc/storage.conf start
将其做成sysv启动脚本如下:
/etc/init.d/trackerd
[root@test fdfs]# cat /etc/init.d/fdfs_trackerd
#!/bin/bash
#
# fdfs_trackerd Starts fdfs_trackerd
#
#
# chkconfig: 2345 99 01
# description: FastDFS tracker server
### BEGIN INIT INFO
# Provides: $fdfs_trackerd
### END INIT INFO
# Source function library.
. /etc/init.d/functions
PRG=/usr/local/bin/fdfs_trackerd
CONF=/etc/fdfs/tracker.conf
if [ ! -f $PRG ]; then
echo "file $PRG does not exist!"
exit 2
fi
if [ ! -f /usr/bin/stop.sh ]; then
echo "file /usr/bin/stop.sh does not exist!"
exit 2
fi
if [ ! -f /usr/bin/restart.sh ]; then
echo "file /usr/bin/restart.sh does not exist!"
exit 2
fi
if [ ! -f $CONF ]; then
echo "file $CONF does not exist!"
exit 2
fi
CMD="$PRG $CONF"
RETVAL=0
start() {
echo -n $"Starting FastDFS tracker server: "
$CMD &
RETVAL=$?
echo
return $RETVAL
}
stop() {
/usr/bin/stop.sh $CMD
RETVAL=$?
return $RETVAL
}
rhstatus() {
status fdfs_trackerd
}
restart() {
/usr/bin/restart.sh $CMD &
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart|reload)
restart
;;
condrestart)
restart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit 1
esac
exit $?
/etc/init.d/fdfs_storaged
[root@test fdfs]# cat /etc/init.d/fdfs_storaged
#!/bin/bash
#
# fdfs_storaged Starts fdfs_storaged
#
#
# chkconfig: 2345 99 01
# description: FastDFS storage server
### BEGIN INIT INFO
# Provides: $fdfs_storaged
### END INIT INFO
# Source function library.
. /etc/init.d/functions
PRG=/usr/local/bin/fdfs_storaged
CONF=/etc/fdfs/storage.conf
if [ ! -f $PRG ]; then
echo "file $PRG does not exist!"
exit 2
fi
if [ ! -f /usr/bin/stop.sh ]; then
echo "file /usr/bin/stop.sh does not exist!"
exit 2
fi
if [ ! -f /usr/bin/restart.sh ]; then
echo "file /usr/bin/restart.sh does not exist!"
exit 2
fi
if [ ! -f $CONF ]; then
echo "file $CONF does not exist!"
exit 2
fi
CMD="$PRG $CONF"
RETVAL=0
start() {
echo -n $"Starting FastDFS storage server: "
$CMD &
RETVAL=$?
echo
return $RETVAL
}
stop() {
/usr/bin/stop.sh $CMD
RETVAL=$?
return $RETVAL
}
rhstatus() {
status fdfs_storaged
}
restart() {
/usr/bin/restart.sh $CMD &
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart|reload)
restart
;;
condrestart)
restart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit 1
esac
exit $?
chmod +x /etc/init.d/{fdfs_storaged,fdfs_trackerd}
添加为开机自动启动:
chkconfig --add fdfs_storaged
chkconfig --add fdfs_trackerd
chkconfig fdfs_storaged on
chkconfig fdfs_trackerd on
五. 测试fastdfs
上传文件:
Usage: fdfs_upload_file <config_file> <local_filename> [storage_ip:port] [store_path_index] [root@test fdfs]# fdfs_upload_file /etc/fdfs/client.conf /etc/inittab group1/M00/55/1E/CgELJFZSoqCAAj2KAAADdDn5EZ87778834
监控fastdfs状态:
[root@test fdfs]# fdfs_monitor /etc/fdfs/client.conf list