FastDFS 分布式集群搭建详解
文章目录
-
-
-
- 前言
- 1、整体架构
- 2、安装配置FastDFS集群
-
- 2.1 配置tracker
- 2.2 配置storage
- 3、启动集群
- 4、查看集群情况
- 5、nginx配置
-
- 5.1 配置storage的四台机器的nginx
- 5.2 配置tracker的两台机器的nginx
- 5.3 配置统一入口
-
-
前言
阅读本文章之前请先看上一篇单机版FastDFS安装配置详解，所以本文有些内容我就一笔带过，传送门：《FastDFS 环境搭建及使用详解》
1、整体架构
大佬超这次使用了7个服务器节点，其中2个跟踪器节点，4个存储节点，需要注意的是 nginx 有三层，第一层使用了一个 nginx 做统一入口进行负载均衡，第二层 tracker 有两个 nginx 做负载均衡，第三层 storage 有四个 nginx 用于访问FastDFS上的文件，服务器节点信息如下
| 节点 | 角色 | 所属组 | nginx |
|---|---|---|---|
| 192.168.122.1 | nginx | ||
| 192.168.122.22 | tracker | nginx1 | |
| 192.168.122.23 | tracker | nginx2 | |
| 192.168.122.24 | storage1 | Group1 | nginx1 |
| 192.168.122.25 | storage2 | Group1 | nginx2 |
| 192.168.122.26 | storage3 | Group2 | nginx3 |
| 192.168.122.27 | storage4 | Group2 | nginx4 |
2、安装配置FastDFS集群
每个节点先按照前面单机版的步骤安装FastDFS。然后进行集群配置，如下
2.1 配置tracker
192.168.122.22 和 192.168.122.23 节点
vi /etc/fdfs/tracker.conf
#启用配置文件，默认启用配置文件（默认值）
disabled=false
#设置tracker的端口号，通常采用22122这个默认端口 （默认值）
port=22122
#设置tracker的数据文件和日志目录
base_path=/opt/fastdfs/tracker
#设置http端口号，（默认值）
http.server_port=8080
需要注意的是，我们文件上传默认会负载均衡，也就是tracker里如下默认配置
# the method for selecting group to upload files
# 0: round robin
# 1: specify group
# 2: load balance, select the max free space group to upload file
store_lookup = 2
# which group to upload file
# when store_lookup set to 1, must set store_group to the group name
store_group = group2
store_lookup，该值默认是2（负载均衡策略，磁盘空间优先），0（轮询策略），1（指定group进行上传下载操作，也就是store_group的值），只有指定为1时store_group才会起作用。
2.2 配置storage
配置另外四台 storage，只有 group_name 不一样
192.168.122.24 和 192.168.122.25 节点
vi /etc/fdfs/storage.conf
#启用配置文件（默认值）
disabled=false
#组名，根据实际情况修改 (默认值)
group_name=group1
#设置storage的端口号，默认是23000，同一个组的storage端口号必须一致 (默认值)
port=23000
#设置storage数据文件和日志目录
base_path=/opt/fastdfs/storage
#存储路径个数，需要和store_path个数匹配 （默认值）
store_path_count=1
#实际文件存储路径
store_path0=/opt/fastdfs/storage/files
#tracker 服务器的 IP地址和端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#设置 http 端口号 (默认值)
http.server_port=8888
192.168.122.26 和 192.168.122.27 节点
vi /etc/fdfs/storage.conf
#启用配置文件（默认值）
disabled=false
#组名，根据实际情况修改 (默认值)
group_name=group2
#设置storage的端口号，默认是23000，同一个组的storage端口号必须一致 (默认值)
port=23000
#设置storage数据文件和日志目录
base_path=/opt/fastdfs/storage
#存储路径个数，需要和store_path个数匹配 （默认值）
store_path_count=1
#实际文件存储路径
store_path0=/opt/fastdfs/storage/files
#tracker 服务器的 IP地址和端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#设置 http 端口号 (默认值)
http.server_port=8888
3、启动集群
192.168.122.22、192.168.122.23：
#启动tracker
fdfs_trackerd /etc/fdfs/tracker.conf
#重新启动tracker
fdfs_trackerd /etc/fdfs/tracker.conf restart
192.168.122.24、192.168.122.25、192.168.122.26、192.168.122.27：
#启动storage
fdfs_storaged /etc/fdfs/storage.conf
#重新启动storage
fdfs_storaged /etc/fdfs/storage.conf restart
4、查看集群情况
fdfs_monitor /etc/fdfs/storage.conf
结果如下
[root@mongodb-1 logs]# fdfs_monitor /etc/fdfs/storage.conf
server_count=2, server_index=0
tracker server is 192.168.122.22:22122
group count: 2
Group 1:
group name = group1
disk total space = 51,175 MB
disk free space = 47,554 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
Storage 1:
id = 192.168.122.24
ip_addr = 192.168.122.24 ACTIVE
http domain =
version = 6.11.
join time = 2024-01-29 18:12:37
up time = 2024-01-29 18:12:37
total storage = 51,175 MB
free storage = 49,349 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id = 192.168.122.25
if_trunk_server = 0
connection.alloc_count = 64
connection.current_count = 1
connection.max_count = 2
total_upload_count = 0
success_upload_count = 0
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 0
success_set_meta_count = 0
total_delete_count = 0
success_delete_count = 0
total_download_count = 0
success_download_count = 0
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 0
success_upload_bytes = 0
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 0
success_download_bytes = 0
total_sync_in_bytes = 0
success_sync_in_bytes = 0
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 0
success_file_open_count = 0
total_file_read_count = 0
success_file_read_count = 0
total_file_write_count = 0
success_file_write_count = 0
last_heart_beat_time = 2024-01-30 10:09:11
last_source_update = 1970-01-01 08:00:00
last_sync_update = 1970-01-01 08:00:00
last_synced_timestamp = 1970-01-01 08:00:00
Storage 2:
id = 192.168.122.25
ip_addr = 192.168.122.25 ACTIVE
http domain =
version = 6.11.
join time = 2024-01-29 18:12:36
up time = 2024-01-29 18:12:36
total storage = 51,175 MB
free storage = 47,554 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id =
if_trunk_server = 0
connection.alloc_count = 64
connection.current_count = 1
connection.max_count = 2
total_upload_count = 0
success_upload_count = 0
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 0
success_set_meta_count = 0
total_delete_count = 0
success_delete_count = 0
total_download_count = 0
success_download_count = 0
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 0
success_upload_bytes = 0
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 0
success_download_bytes = 0
total_sync_in_bytes = 0
success_sync_in_bytes = 0
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 0
success_file_open_count = 0
total_file_read_count = 0
success_file_read_count = 0
total_file_write_count = 0
success_file_write_count = 0
last_heart_beat_time = 2024-01-30 10:09:09
last_source_update = 1970-01-01 08:00:00
last_sync_update = 1970-01-01 08:00:00
last_synced_timestamp = 1970-01-01 08:00:00
Group 2:
group name = group2
disk total space = 51,175 MB
disk free space = 48,108 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0
Storage 1:
id = 192.168.122.26
ip_addr = 192.168.122.26 ACTIVE
http domain =
version = 6.11.
join time = 2024-01-29 18:12:33
up time = 2024-01-29 18:12:33
total storage = 51,175 MB
free storage = 48,118 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id =
if_trunk_server = 0
connection.alloc_count = 64
connection.current_count = 1
connection.max_count = 4
total_upload_count = 13
success_upload_count = 13
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 13
success_set_meta_count = 13
total_delete_count = 0
success_delete_count = 0
total_download_count = 0
success_download_count = 0
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 151530173
success_upload_bytes = 151530173
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 0
success_download_bytes = 0
total_sync_in_bytes = 272774924
success_sync_in_bytes = 272774924
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 61
success_file_open_count = 61
total_file_read_count = 0
success_file_read_count = 0
total_file_write_count = 1657
success_file_write_count = 1657
last_heart_beat_time = 2024-01-30 10:09:06
last_source_update = 2024-01-30 10:01:12
last_sync_update = 2024-01-30 10:02:35
last_synced_timestamp = 2024-01-30 10:02:35 (0s delay)
Storage 2:
id = 192.168.122.27
ip_addr = 192.168.122.27 ACTIVE
http domain =
version = 6.11.
join time = 2024-01-24 10:29:49
up time = 2024-01-29 18:12:28
total storage = 51,175 MB
free storage = 48,108 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id =
if_trunk_server = 0
connection.alloc_count = 64
connection.current_count = 1
connection.max_count = 3
total_upload_count = 24
success_upload_count = 24
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 24
success_set_meta_count = 24
total_delete_count = 0
success_delete_count = 0
total_download_count = 1
success_download_count = 1
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 272773872
success_upload_bytes = 272773872
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 29
success_download_bytes = 29
total_sync_in_bytes = 151530717
success_sync_in_bytes = 151530717
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 51
success_file_open_count = 51
total_file_read_count = 1
success_file_read_count = 1
total_file_write_count = 1646
success_file_write_count = 1646
last_heart_beat_time = 2024-01-30 10:09:00
last_source_update = 2024-01-30 10:02:35
last_sync_update = 2024-01-30 10:01:13
last_synced_timestamp = 2024-01-30 10:01:12 (0s delay)
5、nginx配置
需要在每个节点都安装nginx，但是作用不是一样的，storage上的nginx是用于访问具体的文件，并且需要使用fastdfs模块，tracker上的nginx是用于负载均衡，不需要使用fastdfs模块，而最外层nginx则是统一入口，用于负载均衡到tracker上的nginx。
5.1 配置storage的四台机器的nginx
安装nginx所需依赖
#安装nginx所需依赖
yum -y install pcre pcre-devel
yum -y install zlib zlib-devel
yum -y install openssl openssl-devel
解压 nginx 和 fastdfs-nginx-module
cd /usr/local/fastdfs
tar -zxvf nginx-1.19.10.tar.gz
unzip fastdfs-nginx-module-master.zip
解压后进入 nginx 目录编译安装 nginx，并添加 fastdfs-nginx-module，我这里为了和原有Nginx进行区分，新安装的Nginx取名为nginx_fdfs
# 创建nginx安装目录
mkdir -p /usr/local/nginx_fdfs
cd /usr/local/fastdfs/nginx-1.19.10
# 配置，指定安装路径，添加模块
./configure --prefix=/usr/local/nginx_fdfs --add-module=/usr/local/fastdfs/fastdfs-nginx-module-master/src
# 编译
make
# 安装
make install
访问配置
把 fastdfs-nginx-module-master/src 目录下的 mod_fastdfs.conf 拷贝到 /etc/fdfs 目录下，这样才能正常启动 Nginx
cp /usr/local/fastdfs/fastdfs-nginx-module-master/src/mod_fastdfs.conf /etc/fdfs/
在 /opt/fastdfs/ 目录下创建 nginx_mod 目录
mkdir -p /opt/fastdfs/nginx_mod
修改 mod_fastdfs.conf 配置文件
vi /etc/fdfs/mod_fastdfs.conf
#日志目录
base_path=/opt/fastdfs/nginx_mod
#tracker服务器的IP地址以及端口号
tracker_server=192.168.122.22:22122
tracker_server=192.168.122.23:22122
#storage服务器的端口号(默认值)
storage_server_port=23000
#组名
group_name = group1
#文件url中是否有 group 名
url_have_group_name = true
#存储路径
store_path0=/opt/fastdfs/storage/files
#组数量
group_count = 2
[group1]
group_name=group1
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs/storage/files
[group2]
group_name=group2
storage_server_port=23000
store_path_count=1
store_path0=/opt/fastdfs/storage/files
注：group_name根据实际组名变化，这里192.168.122.24和192.168.122.25为group1，192.168.122.26和192.168.122.27为group2。
修改 nginx.conf 配置文件
vi /usr/local/nginx_fdfs/conf/nginx.conf
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
# 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求，用fastdfs的Nginx模块进行转发
location ~ /group[1-9]/M0[0-9] {
ngx_fastdfs_module;
}
}
启动nginx
# 测试
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -t
# 启动
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf
# 停止
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s stop
# 重载
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s reload
现在四个storage节点的图片都能进行访问，例如
http://192.168.122.24/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.25/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.26/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.27/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
5.2 配置tracker的两台机器的nginx
192.168.122.22 和 192.168.122.23 节点
安装nginx所需依赖
#安装nginx所需依赖
yum -y install pcre pcre-devel
yum -y install zlib zlib-devel
yum -y install openssl openssl-devel
解压 nginx
cd /usr/local/fastdfs
tar -zxvf nginx-1.19.10.tar.gz
解压后进入 nginx 目录编译安装 nginx，这里不需要 fastdfs-nginx-module
# 创建nginx安装目录
mkdir -p /usr/local/nginx_fdfs
cd /usr/local/fastdfs/nginx-1.19.10
# 配置，指定安装路径，这里不需要添加模块
./configure --prefix=/usr/local/nginx_fdfs
# 编译
make
# 安装
make install
修改 nginx.conf 配置文件
vi /usr/local/nginx_fdfs/conf/nginx.conf
upstream fastdfs_group_server{
server 192.168.122.24:80;
server 192.168.122.25:80;
server 192.168.122.26:80;
server 192.168.122.27:80;
}
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
# 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求
location ~ /group[1-9]/M0[0-9] {
proxy_pass http://fastdfs_group_server;
}
}
启动nginx
# 测试
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -t
# 启动
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf
# 停止
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s stop
# 重载
/usr/local/nginx_fdfs/sbin/nginx -c /usr/local/nginx_fdfs/conf/nginx.conf -s reload
现在通过tracker上的nginx可以访问到四台storage的文件，例如
http://192.168.122.22/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
http://192.168.122.23/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
5.3 配置统一入口
这里tracker有两个nginx，我们在最外层再增加一个nginx负载均衡，也就是我这里的192.168.122.1节点，配置如下
upstream fastdfs_group_server{
server 192.168.122.22:80;
server 192.168.122.23:80;
}
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
# 拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求
location ~ /group[1-9]/M0[0-9] {
proxy_pass http://fastdfs_group_server;
}
}
现在通过统一入口上的nginx可以访问到文件，例如
http://192.168.122.1/group2/M00/00/00/wKh6GmW6AWyAL2tYAAAJ3w30duA26.jpeg
至此，集群搭建完成，我辛苦了。
熬夜写的，觉得有帮助来个赞吧！！！
更多技术干货，请持续关注程序员大佬超。
原创不易，转载请务必注明出处。