seaweedfs 调研
简介
SeaweedFS是一个简单并且高度可扩展的分布式文件系统，可以存储数十亿的文件并且快速获得文件，特别适合于有效处理小文件概念：
- Node 系统抽象的节点，抽象为DataCenter、Rack、DataNode
- DataCenter 数据中心，对应现实中的不同机房
- Rack 机架，对应现实中的机柜
- Datanode 存储节点，用于管理、存储逻辑卷
- Volume 逻辑卷，存储的逻辑结构，逻辑卷下存储Needle
- Needle 逻辑卷中的Object，对应存储的文件
- Collection 文件集，可以分布在多个逻辑卷上
备份策略
weed-fs提供了若干种replication策略(rack – 机架，一个逻辑上的概念)：
000 no replication, just one copy
001 replicate once on the same rack
010 replicate once on a different rack in the same data center
100 replicate once on a different data center
200 replicate twice on two other different data center
110 replicate once on a different rack, and once on a different data center
集群模式：注意选择的备份策略，不同的策略需要不同的数据中心机器。否则会报“Cannot grow volume group! No enough data node found” 错误
master: earth139, earth140, earth141(三台机器)
./weed master -ip 10.141.160.93 -port 9333 -mdir ./weedfs_master -peers 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -defaultReplication 100 >master.log 2>&1 &
./weed master -ip 10.141.160.94 -port 9333 -mdir ./weedfs_master -peers 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -defaultReplication 100 >master.log 2>&1 &
./weed master -ip 10.141.160.97 -port 9333 -mdir ./weedfs_master -peers 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -defaultReplication 100 >master.log 2>&1 &volume:earth139, earth140, earth141
./weed volume -ip 10.141.160.93 -port 9222 -dir ./weedfs_volume -mserver 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -dataCenter dc1 -rack rack1 >volume.log 2>&1 &
./weed volume -ip 10.141.160.94 -port 9222 -dir ./weedfs_volume -mserver 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -dataCenter dc1 -rack rack1 >volume.log 2>&1 &
./weed volume -ip 10.141.160.97 -port 9222 -dir ./weedfs_volume -mserver 10.141.160.93:9333,10.141.160.94:9333,10.141.160.97:9333 -dataCenter dc2 -rack rack1 >volume.log 2>&1 &搭建好后，浏览器打开master节点 ip:port看到
filer: 文件服务
./weed scaffold -config filer -output="." // 生成 filer.toml 文件
./weed filer -master 10.141.160.93:9333,10.141.160.94:9333,10.141.160.95:9333 -port 8888浏览器访问 http://10.141.160.93:8888/
也可以mount filer到某个目录进行使用
./weed mount -filer="10.141.160.93:8888" -dir="./mountp_seaweed/"
filer server都是工作在standalone模式下的。可使用redis或Cassandra作为后端，同步，需要在
存文件
curl -F [email protected] http://10.141.160.93:9333/submit
{"eTag":"8ffdca8b","fid":"1,02fd956ac2","fileName":"master.log","fileUrl":"10.141.160.94:9222/1,02fd956ac2","size":1165}
也可以请求一个fid，再传
curl -X POST http://10.141.160.93:9333/dir/assign
{"fid":"2,0194c2cb41","url":"10.141.160.94:9222","publicUrl":"10.141.160.94:9222","count":1}
curl -X PUT -F [email protected] http://10.141.160.94:9222/2,0194c2cb41
获取
curl http://10.141.160.94:9222/1,02fd956ac2
或者浏览器打开 http://10.141.160.94:9222/1,02fd956ac2
删除
curl -X DELETE http://10.141.160.94:9222/1,02fd956ac2
垃圾回收
一般删除后，卷的大小并不会马上更新，当然可手动发起更新大小操作
curl "http://10.141.160.93:9333/vol/vacuum"
参考
https://tonybai.com/2015/08/22/intro-of-using-weedfs/
https://blog.csdn.net/DPnice/article/details/84990050
https://www.jianshu.com/p/8adda9b8856e
https://cloud.tencent.com/developer/article/1363021
https://blog.yasking.org/a/weed-fs.html
https://github.com/bingoohuang/blog/issues/57