Django搭建FastDFS分布式文件存储系统
前言
django admin的ImageField默认会把文件存到settings.py配置的MEDIA_ROOT目录下，所以存储图片的数量受限于django服务器的硬盘大小。为了支持海量的图片存储，更易于拓展存储空间，使用FastDFS分布式存储系统。如果用户上传的文件重复(文件指纹一样)，那么系统只有存储一份数据。
环境：
- 云服务器下的ubuntu16.04
工具：
- docker
- pip
- pycharm
- python3.6
docker安装FastFDS
-
获取镜像
sudo docker image pull delron/fastdfs也可以直接使用镜像备份文件
sudo docker load i 文件路径/fastdfs_docker.tar
-
运行tracker
将fastDFS tracker运行目录映射到本机的 /var/fdfs/tracker目录中
sudo docker run -dti --network=host --name tracker -v /var/fdfs/tracker:/var/fdfs delron/fastdfs tracker-
运行storage
将fastDFS storage运行目录映射到本机的/var/fdfs/storage目录中
sudo docker run -dti --network=host --name storage -e TRACKER_SERVER=192.168.211.132:22122 -v /var/fdfs/storage:/var/fdfs delron/fastdfs storage
这里要注意的是，TRACKER_SERVER后面跟的ip地址需要填写自己的。我使用的是阿里云的服务器，所以我这里填写自己的公网ip。如果你是自己本地的linux，那就填写自己的ip，可通过ifconfig查看，但要注意的是，这里不要填成本地回环地址127.0.0.1
- 查看容器是否已经启动
sudo docker ps -a
注意：如果无法重新运行，可以删除/var/fdfs/storage/data目录下的fdfs_storaged.pid 文件，然后重新运行storage。
安装FastDFS的Python客户端
为了可以让django可以使用FastDFS系统，我们需要一个可以使用python语言调用FastDFS系统的库，我们可以直接从github上下载下来，再通过pip安装。
- 使用pip安装
pip install fdfs_client-py-master.zip
pip install mutagen
pip install requests
- 在django下配置fastdfs的配置文件
在项目/utils目录下新建fastdfs目录，新建client.conf配置文件
# connect timeout in seconds
# default value is 30s
connect_timeout=30
# network timeout in seconds
# default value is 30s
network_timeout=60
# the base path to store log files
#base_path=FastDFS客户端存放日志文件的目录
# tracker_server can ocur more than once, and tracker_server format is
# "host:port", host can be hostname or ip address
tracker_server=172.17.0.1:22122
#standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level=info
# if use connection pool
# default value is false
# since V4.05
use_connection_pool = false
# connections whose the idle time exceeds this time will be closed
# unit: second
# default value is 3600
# since V4.05
connection_pool_max_idle_time = 3600
# if load FastDFS parameters from tracker server
# since V4.05
# default value is false
load_fdfs_parameters_from_tracker=false
# if use storage ID instead of IP address
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# default value is false
# since V4.05
use_storage_id = false
# specify storage ids filename, can use relative or absolute path
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V4.05
storage_ids_filename = storage_ids.conf
#HTTP settings
http.tracker_server_port=80
#use "#include" directive to include HTTP other settiongs
##include http.conf
tracker_server后面的ip设置成自己的。
端口转发的设置
需要进行的端口有三个，分别是tracker_serverd的端口22122、storage_serverd的端口23000、以及最后生成图片地址的端口8888.
测试
进入django项目的shell，进行下面的测试。下面表示测试成功。
(wl_dj) bd@pyvip:~/project_wl$ python manage.py shell
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from fdfs_client.client import Fdfs_client
>>> FDFS_Client = Fdfs_client('utils/fastdfs/client.conf')
>>> ret = FDFS_Client.upload_by_filename('media/2018.png')
getting connection
>>> ret
{'Status': 'Upload successed.', 'Remote file_id': 'group1/M00/00/00/rBMb_lzeUjSATnH1AAfh_rrm7jw829.png', 'Group name': 'group1', 'Uploaded size': '504.00KB', 'Local file name': 'media/2018.png', 'Storage IP': '106.14.148.168'}
django项目中的配置
- settings配置
# fastdfs服务的站点
FASTDFS_SERVER_DOMAIN = "http://106.14.148.168:8888/"- view配置
from django.http import JsonResponse,Http404
class Image_upload(View):
def post(self,request):
"""
url: "/admin/news/images/"
验证：
1、图片对象是否为空
2、请求内容是否是图片格式
3、图片的后缀是否规范
:param request:
:return:
"""
image_file = request.FILES.get('image_file')
if not image_file:
return Http404('图片不存在')
if image_file.content_type not in ('image/jpeg', 'image/png', 'image/gif'):
return Http404('不能上传非图片文件')
try:
image_ext_name = image_file.name.split('.')[-1]
except Exception as e:
image_ext_name = 'jpg'
try:
upload_res = FDFS_Client.upload_by_buffer(image_file.read(), file_ext_name=image_ext_name)
except Exception as e:
return Http404('图片上传异常')
if upload_res.get('Status') != 'Upload successed.':
return Http404('图片上传到服务器失败')
image_name = upload_res.get('Remote file_id')
image_url = settings.FASTDFS_SERVER_DOMAIN + image_name
return JsonResponse({'image_url': image_url,'errmsg':'图片上传成功'})- 路由配置
path('news/images/',views.Image_upload.as_view(),name = 'image_updown'),- 前端js
let $upload_to_server = $("#upload-news-thumbnail"); //上传图片的input框的id
$upload_to_server.change(function () {
let file = this.files[0]; // 获取文件
let oFormData = new FormData(); // 创建一个 FormData
oFormData.append("image_file", file); // 把文件添加进去
// 发送请求
$.ajax({
url: "/admin/news/images/",
method: "POST",
data: oFormData,
processData: false, // 定义文件的传输
contentType: false,
})
.done(function (res) {
if (res.errno === "0") {
// 更新标签成功
message.showSuccess("图片上传成功");
let sImageUrl = res["image_url"];
// console.log(thumbnailUrl);
$thumbnailUrl.val('');
$thumbnailUrl.val(sImageUrl);
} else {
message.showError(res.errmsg)
}
})
.fail(function () {
message.showError('服务器超时，请重试！');
});
});