FastDFS安装配置过程

由于网站使用nfs共享方式保存用户上传的图片，附件等资料，然后通过apache下载的方式供用户访问，在网站架构初期，使用这种简单的方式实现了静态 资源的读写分离，但随着网站数据量的增加，图片服务器渐渐成为整个网站的短板，缘次催生了使用fastfds的想法，故而先进行一番简单的测试！在开始之 前还是先来看看fastfds的介绍信息：

FastDFS is an open source high performance distributed file system (DFS). It's major functions include: file storing, file syncing and file accessing, and design for high capacity and load balancing.

FastDFS is an open source high performance distributed file system. It's major functions include: file storing, file syncing and file accessing (file uploading and file downloading), and it can resolve the high capacity and load balancing problem. FastDFS should meet the requirement of the website whose service based on files such as photo sharing site and vidio sharing site.

FastDFS has two roles: tracker and storage. The tracker takes charge of scheduling and load balancing for file access. The storage store files and it's function is file management including: file storing, file syncing, providing file access interface. It also manage the meta data which are attributes representing as key value pair of the file. For example: width=1024, the key is "width" and the value is "1024".

The tracker and storage contain one or more servers. The servers in the tracker or storage cluster can be added to or removed from the cluster by any time without affecting the online services. The servers in the tracker cluster are peer to peer.

The storarge servers organizing by the file volume/group to obtain high capacity. The storage system contains one or more volumes whose files are independent among these volumes. The capacity of the whole storage system equals to the sum of all volumes' capacity. A file volume contains one or more storage servers whose files are same among these servers. The servers in a file volume backup each other, and all these servers are load balancing. When adding a storage server to a volume, files already existing in this volume are replicated to this new server automatically, and when this replication done, system will switch this server online to providing storage services. When the whole storage capacity is insufficiency, you can add one or more volumes to expand the storage capacity. To do this, you need to add one or more storage servers.

The identification of a file is composed of two parts: the volume name and the file name.

大意为：
fastdfs是一个开源的，高性能的的分布式文件系统，他主要的功能包括：文件存储，同步和访问，设计基于高可用和负载均衡，fastfd非常适用于基于文件服务的站点，例如图片分享和视频分享网站

fastfds有两个角色：跟踪服务和存储服务，跟踪服务控制，调度文件以负载均衡的方式访问；存储服务包括：文件存储，文件同步，提供文件访问接口，同时以key value的方式管理文件的元数据

跟踪和存储服务可以由1台或者多台服务器组成，同时可以动态的添加，删除跟踪和存储服务而不会对在线的服务产生影响，在集群中，tracker服务是对等的

存储系统由一个或多个卷组成，卷与卷之间的文件是相互独立的，所有卷的文件容量累加就是整个存储系统中的文件容量。一个卷可以由一台或多台存储服务 器组成，一个卷下的存储服务器中的文件都是相同的，卷中的多台存储服务器起到了冗余备份和负载均衡的作用。在卷中增加服务器时，同步已有的文件由系统自动 完成，同步完成后，系统自动将新增服务器切换到线上提供服务。当存储空间不足或即将耗尽时，可以动态添加卷。只需要增加一台或多台服务器，并将它们配置为 一个新的卷，这样就扩大了存储系统的容量。

下面几张图可以清楚的说明fastfds的架构和文件上传和下载流程等：

安装过程：

    由于FastDFS内部绑定了libevent作为http服务器，所以必须先安装libevent。如果已经安装了libevent，请确认安装路径是 /usr，因为FastDFS在编译源程序时，需要到此目录下查找一些依赖文件，否则编译会出错。如果不是，建议首先卸载libevent，然后安装到 /usr下。

下载最新的libevent包并安装

wget https://sourceforge.net/projects/levent/files/libevent/libevent-2.0/libevent-2.0.22-stable.tar.gz

1、解压

tar -zxvf libevent-2.0.22-stable.tar.gz

2、进入解压目录编译和安装

cd cd libevent-2.0.22-stable

./configure --prefix=/usr/local/libevent/

make && make install

安装FastDFS：5.5版本编译出现错误，建议使用5.2or5.3版本。。。

    wget http://sourceforge.net/projects/fastdfs/files/FastDFS%20Server%20Source%20Code/FastDFS%20Server%20with%20PHP%20Extension%20Source%20Code%20V5.03/FastDFS_v5.03.tar.gz

1、解压

tar -zxvf tar zxvf FastDFS_v5.05.tar.gz

2、进入解压目录修改make.sh中TARGET_PREFIX=你要安装的目录

cd FastDFS

vim make.sh

3、编译和安装

./make.sh

./make.sh install

安装完成后如果出现ln -fs ....即表示安装成功，安装成功后的配置文件在/etc/fdfs

安装FastDHT：

    wget http://sourceforge.net/projects/fastdht/files/FastDHT%20server%20and%20php%20ext/FastDHT%20Server%20Source%20Code%20V1.23/FastDHT_v1.23.tar.gz

1、解压

tar -zxvf FastDHT_v1.23.tar.gz

2、进入解压目录修改mak.sh中的TARGET_PREFIX=你的安装目录

3、编译和安装

./make.sh

./make.sh install

安装完成后如果出现ln -fs ....即表示安装成功，安装成功后的配置文件在/etc/fdht

安装至此结束。。。

修改配置文件

    vim /etc/fdfs/tracker.conf

 1.base_path=/data/fastdfs

 2.#改成你想要的http端口，将来http下载文件的端口就是他了

 3.http.server_port=8080 -> http.server_port=8090

#http支持

 4.##include http.conf -> #include http.conf

#默认4GB，如果空间不足会报no free space的异常，无法正常启动，建议暂时调小

 5.reserved_storage_space = 4GB -> reserved_storage_space = 1GB

#tracker server对storage server供服务的端口，使用默认的即可，也可以自定义

 6.port=22122

B、将http.conf文件拷贝到/etc/fdfs目录下，执行

7. cp %FastDFS%/conf/http.conf /etc/fdfs/

注：为了支持http，必须将这个文件拷贝到此目录，否则无法启动，报param http.XXX not exist or is empty类似的错误

启动：

    /usr/local/fdfs/bin/fdfs_trackerd /etc/fdfs/tracker.conf

报错：

ERROR - file: ../common/process_ctrl.c, line: 189, "/home/yuqing/fastdfs" can't be accessed, error info: No such file or directory

解决：

[root@localhost FastDHT]# echo '/usr/local/libevent/include/' >> /etc/ld.so.conf[root@localhost FastDHT]# echo '/usr/local/libevent/lib/' >> /etc/ld.so.conf
[root@localhost FastDHT]# ldconfig 

依然报错：

[2015-05-29 07:36:03] ERROR - file: ../common/process_ctrl.c, line: 189, "/home/yuqing/fastdfs" can't be accessed, error info: No such file or directory

解决：

mkdir -p /home/yuqing/fastdfs

这回正常了，查看启动成功与否

[root@localhost FastDHT]# ps -ef |grep track |grep -v 'grep'
root     37018     1  0 07:38 ?        00:00:00 /usr/local/fdfs/bin/fdfs_trackerd /etc/fdfs/tracker.conf

过程中的错误排查及解决

一、文件上传后下载提示404

排查思路：

            location ~* /group1/M00/ {
                          root /data/fastdfs/data;
                          ngx_fastdfs_module;
                 }

一直以为是我的都location写的有问题，怎么修改都不对。后来在网上一番查找，发现要把mod_fastdfs.conf配置文件中的“url_have_group_name = false”改成true。更改后重启服务后提示400。。。。。

二、文件上传后访问url下载提示400

排查思路：查看nginx的错误日志，发现错误提示：[2015-06-10 16:19:21] ERROR - file: /usr/local/src/fastdfs-nginx-module/src//common.c, line: 870, file: /home/yuqing/fastdfs/data/00/00/wKgK51V38rGADHyEAAAIofjyE_89461992 not exist

我在安装的时候数据目录明明指定的是/data/fastdfs，为啥会跑到默认路径去呢。。。于是查找tracker、storage、和mod_fastdfs.cof。发现mod_fastdfs.conf中的store_path0的路径没改。。于是修改成我指定的/data/fastdfs，然后重启tracker和storage后便可以正常下载文件了。

三、storage.log报错：ERROR - file: tracker_proto.c, line: 48, server: 10.171.22.124:22122, response status 2 != 0

排查思路：

在tracker上使用fdfs_monitor /etc/fdfs/client.conf 命令查看storage的状态是offline

fdfs_monitor /etc/fdfs/client.conf delete group1 storage的IP地址。    #先在tracker上删除storage的信息。

rm -rf /data/fastdfs/data        #删除storage的数据目录。

service fdfs_storaged restart    #重启storage

在tracker上再次使用fdfs_monitor查看storage的状态，显示active。上传图片，正常了。

四、最近在公司生产环境上配置FastDFS出现的问题，网页无法打开。

nginx错误日志：

2015/07/01 12:52:13 [notice] 4481#0: signal 3 (SIGQUIT) received, shutting down
2015/07/01 12:52:13 [notice] 4481#0: exit
2015/07/01 12:52:16 [notice] 12303#0: using the "epoll" event method
2015/07/01 12:52:16 [notice] 12303#0: nginx/1.6.3
2015/07/01 12:52:16 [notice] 12303#0: built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
2015/07/01 12:52:16 [notice] 12303#0: OS: Linux 2.6.32-431.23.3.el6.x86_64
2015/07/01 12:52:16 [notice] 12303#0: getrlimit(RLIMIT_NOFILE): 65535:65535
2015/07/01 12:52:16 [notice] 12304#0: start worker processes
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12305
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12306
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12307
ngx_http_fastdfs_process_init pid=12305
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12309
ngx_http_fastdfs_process_init pid=12306
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12310
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12311
ngx_http_fastdfs_process_init pid=12307
ngx_http_fastdfs_process_init pid=12309
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12312
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
2015/07/01 12:52:16 [notice] 12304#0: start worker process 12313
ngx_http_fastdfs_process_init pid=12310
ngx_http_fastdfs_process_init pid=12311
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12306 exited with code 2
2015/07/01 12:52:16 [alert] 12304#0: worker process 12306 exited with fatal code 2 and cannot be respawned
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12305 exited with code 2
2015/07/01 12:52:16 [alert] 12304#0: worker process 12305 exited with fatal code 2 and cannot be respawned
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12309 exited with code 2
2015/07/01 12:52:16 [alert] 12304#0: worker process 12309 exited with fatal code 2 and cannot be respawned
ngx_http_fastdfs_process_init pid=12313
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12307 exited with code 2
2015/07/01 12:52:16 [alert] 12304#0: worker process 12307 exited with fatal code 2 and cannot be respawned
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
ngx_http_fastdfs_process_init pid=12312
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12310 exited with code 2
2015/07/01 12:52:16 [alert] 12304#0: worker process 12310 exited with fatal code 2 and cannot be respawned
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12311 exited with code 2
2015/07/01 12:52:16 [alert] 12304#0: worker process 12311 exited with fatal code 2 and cannot be respawned
[2015-07-01 12:52:16] ERROR - file: ../storage/trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /logs
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12313 exited with code 2
2015/07/01 12:52:16 [alert] 12304#0: worker process 12313 exited with fatal code 2 and cannot be respawned
2015/07/01 12:52:16 [notice] 12304#0: signal 29 (SIGIO) received
2015/07/01 12:52:16 [notice] 12304#0: signal 17 (SIGCHLD) received
2015/07/01 12:52:16 [notice] 12304#0: worker process 12312 exited with code 2

2015/07/01 12:52:16 [alert] 12304#0: worker process 12312 exited with fatal code 2 and cannot be respawned

问题解决：

    根据报错信息提示的找不到/logs。查找报错文件trunk_shared.c的177行发现

        logError("file: "__FILE__", line: %d, " \
                        "\"%s\" can't be accessed, error info: %s", \
                        __LINE__, STRERROR(errno), g_fdfs_base_path);
                return errno != 0 ? errno : ENOENT;

    在vim中查找g_fdfs_base_path的变量，查找不到。storaged的nginx启动是要加载mod_fastdfs.conf的配置文件，于是猜想启动不了肯定和这个配置文件有关系，去mod_fastdfs.conf查找发现base_path定义的路径是/logs，检查发现，根下并没有logs目录。于是更改到/tmp下。重启nginx问题解决。

本文出自 “初心、始终” 博客，谢绝转载！