最近在使用Golang做了一个网盘项目(学习),文件存储一直保存在本地(各厂商提供的oss贵),所以就在思考怎么来处理这些文件,类似的方案很对hdfs、fastdfs,但这其中MinIO是最近几年比较火热的一个项目,所以尝试使用这个试一试。
MinIO的安装特别简单,大家可以前去官网按照步骤完成,注意以下几点即可
minio server ~/minio --console-address :9090
目前不要随便乱修改,按照原始的方案
mkdir minio-api
cd minio-api
go mod init v1
创建一个main.go,在这个函数中我们首先创建一个用于初始化MinIOClient的函数,该函数细节如下:
func InitMinioClient() *minio.Client {
// 基本的配置信息
endpoint := "172.16.59.130:9000"
accessKeyID := "IdoSKNGz7evlQXVVqGJF"
secretAccessKey := "s4hnwC9yWOsU8TTmODFcMcw0TdExa4GsTpGzibEc"
// 初始化一个minio客户端对象
minioClient, err := minio.New(endpoint, &minio.Options{
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
})
if err != nil {
log.Fatalf("初始化MinioClient错误:%s", err.Error())
}
return minioClient
}
有几点基本的注意事项,首先是基本的配置信息你需要更改为你自己的,一般端口都为9000(注意不是9090),针对这个地方的accessKeyID和secretAccessKey的创建如下图:
然后我们构建一个main函数,在这个main函数中,我们首先调用初始化客户端的函数InitMinioClient,然后我们在main实现一个简单的上传文件的demo
func main() {
// 创建客户端
minioClient := InitMinioClient()
// bucket名称
bucketName := "mypic"
ctx := context.Background()
// 创建这个bucket
err := minioClient.MakeBucket(ctx, bucketName, minio.MakeBucketOptions{})
if err != nil {
// 检测这个bucket是否已经存在
exists, errBucketExists := minioClient.BucketExists(ctx, bucketName)
if errBucketExists == nil && exists {
log.Printf("We already own %s\n", bucketName)
} else {
log.Fatalln(err)
}
} else {
log.Printf("Successfully created %s\n", bucketName)
}
// 需要上传文件的基本信息
objectName := "头像.jpg"
filePath := "image"
contentType := "multipart/form-data"
fPath := filepath.Join(filePath, objectName)
fileInfo, err := os.Stat(fPath)
if err == os.ErrNotExist {
log.Printf("%s目标文件不存在", fPath)
}
f, err := os.Open(fPath)
if err != nil {
return
}
uploadInfo, err := minioClient.PutObject(ctx, bucketName, objectName, f, fileInfo.Size(), minio.PutObjectOptions{ContentType: contentType})
if err != nil {
log.Fatalln(err)
}
log.Printf("Successfully uploaded %s of size %d\n", objectName, uploadInfo.Size)
}
上传文件即成功,然后打开网页页面刷新,出现下面页面即可,同时你打开你的服务器可以发现你上传的文件。
观察上面的代码,至少包括三个API,包括创建Bucket、检测Bucket是否存在、上传文件到指定的Bucket。为了更好的研究这些API,现在我从Bucket开始研究一下常用的API。
API接口
New(endpoint string, opts *Options) (*Client, error)
初步观察这个接口,在结合我们上面的示例,可以发现:
func InitMinioClient() *minio.Client {
// 基本的配置信息
endpoint := "172.16.59.130:9000"
accessKeyID := "IdoSKNGz7evlQXVVqGJF"
secretAccessKey := "s4hnwC9yWOsU8TTmODFcMcw0TdExa4GsTpGzibEc"
// 初始化一个minio客户端对象
minioClient, err := minio.New(endpoint, &minio.Options{
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
})
if err != nil {
log.Fatalf("初始化MinioClient错误:%s", err.Error())
}
return minioClient
}
你自己可以尝试:创建一个bucket01和bucket02
MakeBucket(ctx context.Context, bucketName string, opts MakeBucketOptions)
前两个参数不作过多的介绍,非常的清晰(所谓的BuketName,你可以简单理解为就是你文件保存的文件夹),在这里我们详细得介绍一哈MakeBucketOptions(minio.MakeBucketOptions),其主要是用于指定Bucket的一些选项比如说Region
(这个地方的Region表示在哪里创建你的Bucket,默认为us-east-1
,其它的一些选项可以参考官网文档,在这里需要注意,如果你是自己服务器搭建,而不是使用他所提供的存储服务,其实你是不用指定这个,就默认值问题也不大),一个完整的示例如下(确保你事先已经有这样的minioClient
):
err = minioClient.MakeBucket(context.Background(),minio.MakeBucketOptions{Region: "us-east-1", ObjectLocking: true})
if err != nil {
fmt.Println(err)
return
}
fmt.Println("Successfully created mybucket.")
ListBuckets(ctx context.Context) ([]BucketInfo, error)
这个接口非常明了,返回的是一个BucketInfo的slice,在这里我们可以遍历该元素然后获取到对应的BucketInfo对象,直接看下面的demo:
func ListBuckets(minioClient *minio.Client) {
bucketInfos, err := minioClient.ListBuckets(context.Background())
if err != nil {
fmt.Println("List Buckets err:", err.Error())
return
}
for index, bucketInfo := range bucketInfos {
fmt.Printf("List Bucket No {%d}----filename{%s}-----createTime{%s}\n", index+1, bucketInfo.Name, bucketInfo.CreationDate.Format("2006-01-02 15:04:05"))
}
}
输出为:
List Bucket No {1}----filename{bucket01}-----createTime{2023-08-18 04:03:18}
List Bucket No {2}----filename{bucket02}-----createTime{2023-08-18 03:54:42}
注意:bucket是不支持修改名称的,如果你要修改名称,一般是新建一个bucket然后讲原来需要改名的bucket的内容移到新建的一个bucket。
BucketExists(ctx context.Context, bucketName string) (found bool, err error)
检查Bucket是否存在,查看下面demo:
func CheckBuckets(minioClient *minio.Client) {
bucketName01 := "bucket01"
bucketName02 := "bucket03"
isExist, err := minioClient.BucketExists(context.Background(), bucketName01)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName01, err.Error())
return
}
if isExist {
fmt.Printf("%s exists!\n", bucketName01)
} else {
fmt.Printf("%s not exists!\n", bucketName01)
}
isExist, err = minioClient.BucketExists(context.Background(), bucketName02)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName02, err.Error())
return
}
if isExist {
fmt.Printf("%s exists!\n", bucketName02)
} else {
fmt.Printf("%s not exists!\n", bucketName02)
}
}
输出:
bucket01 exists!
bucket03 not exists!
RemoveBucket(ctx context.Context, bucketName string) error
删除名称为bucketName的bucket,删除之前建议先检查该bucket是否存在,查看下面的demo:
func RemoveBucket(minioClient *minio.Client) {
bucketName01 := "bucket01"
isExist, err := minioClient.BucketExists(context.Background(), bucketName01)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName01, err.Error())
return
}
if isExist {
fmt.Printf("%s exists! Start delete....\n", bucketName01)
// 开始删除逻辑
err = minioClient.RemoveBucket(context.Background(), bucketName01)
if err != nil {
fmt.Printf("Fail to remove %s:%s\n", bucketName01, err.Error())
return
}
fmt.Printf("Success to remove %s\n", bucketName01)
} else {
fmt.Printf("%s not exists!\n", bucketName01)
}
}
输出为:
bucket01 exists! Start delete…
Success to remove bucket01
ListObjects(ctx context.Context, bucketName string, opts ListObjectsOptions) <-chan ObjectInfo
展示一个Object中的所有对象,注意返回的是一个channel。在查看下面的demo,请参考之前的内容先上传一些对象:
func ListObjects(minioClient *minio.Client) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
bucketName := "bucket02"
opts := minio.ListObjectsOptions{
Prefix: "头",
Recursive: true,
}
objectCh := minioClient.ListObjects(ctx, bucketName, opts)
for obj := range objectCh {
fmt.Printf("Name:%s\tSize:%d\tMD5:%s\tModifiedTime:%s\n",
obj.Key, obj.Size, obj.ETag, obj.LastModified.Format("2006-01-02 03:04:05"))
}
}
输出为:
Name:头像.jpg Size:739938 MD5:10bf76e379cd8f381791c6924f33dcd6 ModifiedTime:2023-08-18 05:22:34
注意,在这里Prefix就是所有Object的的前缀。
针对Bucket的操作就到这里,但是minio还提供其余操作,比喻设置tag等等,上面所列举的是常用的一些操作
GetObject(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (*Object, error)
返回一个数据流对象,注意是一个数据流,所以要写入到一个具体的对象中,详见下面demo的使用:
func GetObjects(minioClient *minio.Client) {
bucketName := "bucket02"
objectName := "头像.jpg"
object, err := minioClient.GetObject(context.Background(), bucketName, objectName, minio.GetObjectOptions{})
if err != nil {
fmt.Println(err)
return
}
defer func(object *minio.Object) {
err := object.Close()
if err != nil {
fmt.Println(err)
return
}
}(object)
localFile, err := os.Create("image/local-file.jpg")
if err != nil {
fmt.Println(err)
return
}
defer func(localFile *os.File) {
err := localFile.Close()
if err != nil {
return
}
}(localFile)
if _, err = io.Copy(localFile, object); err != nil {
fmt.Println(err)
return
}
}
注意这个地方文件的写入。
PutObject(ctx context.Context, bucketName, objectName string, reader io.Reader, objectSize int64,opts PutObjectOptions) (info UploadInfo, err error)
将一个文件放入到bucket中,详细见下面的操作,其实之前我们的demo中已经有了这个操作:
func PutObjects(minioClient *minio.Client) {
bucketName := "bucket02"
// 检查bucket是否存在
isExist, err := minioClient.BucketExists(context.Background(), bucketName)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName, err.Error())
return
}
if !isExist {
fmt.Printf("%s not exists!\n", bucketName)
}
// 对象信息
objectName := "头像.jpg"
filePath := "image"
contentType := "multipart/form-data"
fPath := filepath.Join(filePath, objectName)
// 读取对象流
fileInfo, err := os.Stat(fPath)
if err == os.ErrNotExist {
log.Printf("%s目标文件不存在", fPath)
}
f, err := os.Open(fPath)
if err != nil {
log.Printf("%s打开目标文件", fPath)
return
}
// 上传文件
uploadInfo, err := minioClient.PutObject(context.Background(), bucketName,
objectName, f, fileInfo.Size(),
minio.PutObjectOptions{ContentType: contentType})
if err != nil {
log.Fatalln(err)
return
}
log.Printf("Successfully uploaded %s of size %d\n", objectName, uploadInfo.Size)
}
最后这个文件会被放在你minio服务器上的minio下,你可以去该文件下查看
CopyObject(ctx context.Context, dst CopyDestOptions, src CopySrcOptions) (UploadInfo, error)
将一个文件复制到另一个bucket中,详见下面demo:
func CopyObjects(minioClient *minio.Client) {
// Source object
srcOpts := minio.CopySrcOptions{
Bucket: "bucket02",
Object: "头像.jpg",
}
// Destination object
dstOpts := minio.CopyDestOptions{
Bucket: "bucket01",
Object: "图片.jpg",
}
// copy
uploadInfo, err := minioClient.CopyObject(context.Background(), dstOpts, srcOpts)
if err != nil {
fmt.Println(err)
return
}
fmt.Println("Successfully copied object:", uploadInfo)
}
输出为:
Successfully copied object: {bucket01 图片.jpg 10bf76e379cd8f381791c6924f33dcd6 0 2023-08-18 08:144.773 +0000 UTC 0001-01-01 00:00:00 +0000 UTC }
StatObject(ctx context.Context, bucketName, objectName string, opts StatObjectOptions) (ObjectInfo, error)
返回一个object的元数据,demo如下:
func StateObjects(minioClient *minio.Client) {
ObjInfo, err := minioClient.StatObject(context.Background(), "bucket02", "头像.jpg", minio.StatObjectOptions{})
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("LastModified:%s\tETag:%s\tContentType:%s\tSize:%d\n",
ObjInfo.LastModified.Format("2006-01-02 03:04:05"),
ObjInfo.ETag, ObjInfo.ContentType, ObjInfo.Size)
}
输出为:
LastModified:2023-08-18 05:22:34 ETag:10bf76e379cd8f381791c6924f33dcd6 ContentType:multipart/form-data Size:739938
RemoveObject(ctx context.Context, bucketName, objectName string, opts minio.RemoveObjectOptions) error
删除一个object,注意下面demo:
func RemoveObject(minioClient *minio.Client) {
opts := minio.RemoveObjectOptions{}
err := minioClient.RemoveObject(context.Background(), "bucket01", "图片.jpg", opts)
if err != nil {
fmt.Println(err)
return
}
}
RemoveObjects(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, opts RemoveObjectsOptions) <-chan RemoveObjectError
批量删除一个bucket中的object,关键就是构造这里的objectsCh,详细见demo:
func RemoveObjects(minioClient *minio.Client) {
objectsCh := make(chan minio.ObjectInfo)
// 注意一般不要自己来构造,直接选择从bucket中查询,查询到的对象放入objectsCh
for object := range minioClient.ListObjects(context.Background(), "bucket02", minio.ListObjectsOptions{}) {
if object.Key == "头像.jpg" {
objectsCh <- object
}
}
defer close(objectsCh)
// 删除
for rErr := range minioClient.RemoveObjects(context.Background(), "bucket02", objectsCh, minio.RemoveObjectsOptions{}) {
fmt.Println("Delete err:", rErr.Err.Error())
}
}
FPutObject(ctx context.Context, bucketName, objectName, filePath, opts PutObjectOptions) (info UploadInfo, err error)
该demo如下:
func UploadLargeFileObjects(minioClient *minio.Client) {
uploadInfo, err := minioClient.FPutObject(context.Background(), "bucket02", "test.csv", "data", minio.PutObjectOptions{
ContentType: "application/csv",
})
if err != nil {
fmt.Println(err)
return
}
fmt.Println("Successfully uploaded object: ", uploadInfo)
}
在MinIO中,PutObject和FPutObject主要有以下几点区别:
FGetObject(ctx context.Context, bucketName, objectName, filePath string, opts GetObjectOptions) error
下载大文件,详见下面demo:
func DownloadLargeFileObjects(minioClient *minio.Client) {
err := minioClient.FGetObject(context.Background(), "bucket02", "test.csv", "/tmp/myobject", minio.GetObjectOptions{})
if err != nil {
fmt.Println(err)
return
}
}
基本上的操作就是这样,当然这里还可以需要设置一些策略设置就不在这里详讲了,可以去官网查看.
完整代码:
package main
import (
"context"
"fmt"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
"io"
"log"
"os"
"path/filepath"
)
func InitMinioClient() *minio.Client {
// 基本的配置信息
endpoint := "172.16.59.129:9000"
accessKeyID := "6b6U1MlseU8h9dDkACNj"
secretAccessKey := "Hf6NSEHjXwHiApoymYR0yktNUkbZwt3MYYRmXOgT"
// 初始化一个minio客户端对象
minioClient, err := minio.New(endpoint, &minio.Options{
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
})
if err != nil {
log.Fatalf("初始化MinioClient错误:%s", err.Error())
}
return minioClient
}
func main() {
// 创建客户端
minioClient := InitMinioClient()
// Make a new bucket called mymusic.
// bucketName := "mypic"
//ctx := context.Background()
//err := minioClient.MakeBucket(ctx, bucketName, minio.MakeBucketOptions{})
//if err != nil {
// // Check to see if we already own this bucket (which happens if you run this twice)
// exists, errBucketExists := minioClient.BucketExists(ctx, bucketName)
// if errBucketExists == nil && exists {
// log.Printf("We already own %s\n", bucketName)
// } else {
// log.Fatalln(err)
// }
//} else {
// log.Printf("Successfully created %s\n", bucketName)
//}
//
Upload the zip file
//objectName := "头像.jpg"
//filePath := "image"
//contentType := "multipart/form-data"
//fPath := filepath.Join(filePath, objectName)
//fileInfo, err := os.Stat(fPath)
//if err == os.ErrNotExist {
// log.Printf("%s目标文件不存在", fPath)
//}
//
//f, err := os.Open(fPath)
//if err != nil {
// return
//}
//
//uploadInfo, err := minioClient.PutObject(ctx, bucketName, objectName, f, fileInfo.Size(), minio.PutObjectOptions{ContentType: contentType})
//if err != nil {
// log.Fatalln(err)
//}
//
//log.Printf("Successfully uploaded %s of size %d\n", objectName, uploadInfo.Size)
// ListBuckets(minioClient)
//CheckBuckets(minioClient)
//RemoveBucket(minioClient)
//UploadObjToBucket(minioClient)
//ListObjects(minioClient)
//GetObjects(minioClient)
//CopyObjects(minioClient)
//StateObjects(minioClient)
//RemoveObject(minioClient)
RemoveObjects(minioClient)
}
func ListBuckets(minioClient *minio.Client) {
bucketInfos, err := minioClient.ListBuckets(context.Background())
if err != nil {
fmt.Println("List Buckets err:", err.Error())
return
}
for index, bucketInfo := range bucketInfos {
fmt.Printf("List Bucket No {%d}----filename{%s}-----createTime{%s}\n", index+1, bucketInfo.Name, bucketInfo.CreationDate.Format("2006-01-02 15:04:05"))
}
}
func CheckBuckets(minioClient *minio.Client) {
bucketName01 := "bucket01"
bucketName02 := "bucket03"
isExist, err := minioClient.BucketExists(context.Background(), bucketName01)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName01, err.Error())
return
}
if isExist {
fmt.Printf("%s exists!\n", bucketName01)
} else {
fmt.Printf("%s not exists!\n", bucketName01)
}
isExist, err = minioClient.BucketExists(context.Background(), bucketName02)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName02, err.Error())
return
}
if isExist {
fmt.Printf("%s exists!\n", bucketName02)
} else {
fmt.Printf("%s not exists!\n", bucketName02)
}
}
func RemoveBucket(minioClient *minio.Client) {
bucketName01 := "bucket01"
isExist, err := minioClient.BucketExists(context.Background(), bucketName01)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName01, err.Error())
return
}
if isExist {
fmt.Printf("%s exists! Start delete....\n", bucketName01)
// 开始删除逻辑
err = minioClient.RemoveBucket(context.Background(), bucketName01)
if err != nil {
fmt.Printf("Fail to remove %s:%s\n", bucketName01, err.Error())
return
}
fmt.Printf("Success to remove %s\n", bucketName01)
} else {
fmt.Printf("%s not exists!\n", bucketName01)
}
}
func ListObjects(minioClient *minio.Client) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
bucketName := "bucket02"
opts := minio.ListObjectsOptions{
Prefix: "头",
Recursive: true,
}
objectCh := minioClient.ListObjects(ctx, bucketName, opts)
for obj := range objectCh {
fmt.Printf("Name:%s\tSize:%d\tMD5:%s\tModifiedTime:%s\n",
obj.Key, obj.Size, obj.ETag, obj.LastModified.Format("2006-01-02 03:04:05"))
}
}
func GetObjects(minioClient *minio.Client) {
bucketName := "bucket02"
objectName := "头像.jpg"
object, err := minioClient.GetObject(context.Background(), bucketName, objectName, minio.GetObjectOptions{})
if err != nil {
fmt.Println(err)
return
}
defer func(object *minio.Object) {
err := object.Close()
if err != nil {
fmt.Println(err)
return
}
}(object)
localFile, err := os.Create("image/local-file.jpg")
if err != nil {
fmt.Println(err)
return
}
defer func(localFile *os.File) {
err := localFile.Close()
if err != nil {
return
}
}(localFile)
if _, err = io.Copy(localFile, object); err != nil {
fmt.Println(err)
return
}
}
func PutObjects(minioClient *minio.Client) {
bucketName := "bucket02"
// 检查bucket是否存在
isExist, err := minioClient.BucketExists(context.Background(), bucketName)
if err != nil {
fmt.Printf("Check %s err:%s", bucketName, err.Error())
return
}
if !isExist {
fmt.Printf("%s not exists!\n", bucketName)
}
// 对象信息
objectName := "头像.jpg"
filePath := "image"
contentType := "multipart/form-data"
fPath := filepath.Join(filePath, objectName)
// 读取对象流
fileInfo, err := os.Stat(fPath)
if err == os.ErrNotExist {
log.Printf("%s目标文件不存在", fPath)
}
f, err := os.Open(fPath)
if err != nil {
log.Printf("%s打开目标文件", fPath)
return
}
// 上传文件
uploadInfo, err := minioClient.PutObject(context.Background(), bucketName,
objectName, f, fileInfo.Size(),
minio.PutObjectOptions{ContentType: contentType})
if err != nil {
log.Fatalln(err)
return
}
log.Printf("Successfully uploaded %s of size %d\n", objectName, uploadInfo.Size)
}
func CopyObjects(minioClient *minio.Client) {
// Source object
srcOpts := minio.CopySrcOptions{
Bucket: "bucket02",
Object: "头像.jpg",
}
// Destination object
dstOpts := minio.CopyDestOptions{
Bucket: "bucket01",
Object: "图片.jpg",
}
// copy
uploadInfo, err := minioClient.CopyObject(context.Background(), dstOpts, srcOpts)
if err != nil {
fmt.Println(err)
return
}
fmt.Println("Successfully copied object:", uploadInfo)
}
func StateObjects(minioClient *minio.Client) {
ObjInfo, err := minioClient.StatObject(context.Background(), "bucket02", "头像.jpg", minio.StatObjectOptions{})
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("LastModified:%s\tETag:%s\tContentType:%s\tSize:%d\n",
ObjInfo.LastModified.Format("2006-01-02 03:04:05"),
ObjInfo.ETag, ObjInfo.ContentType, ObjInfo.Size)
}
func RemoveObject(minioClient *minio.Client) {
opts := minio.RemoveObjectOptions{}
err := minioClient.RemoveObject(context.Background(), "bucket01", "图片.jpg", opts)
if err != nil {
fmt.Println(err)
return
}
}
func RemoveObjects(minioClient *minio.Client) {
objectsCh := make(chan minio.ObjectInfo)
// 注意一般不要自己来构造,直接选择从bucket中查询,查询到的对象放入objectsCh
for object := range minioClient.ListObjects(context.Background(), "bucket02", minio.ListObjectsOptions{}) {
if object.Key == "头像.jpg" {
objectsCh <- object
}
}
defer close(objectsCh)
// 删除
for rErr := range minioClient.RemoveObjects(context.Background(), "bucket02", objectsCh, minio.RemoveObjectsOptions{}) {
fmt.Println("Delete err:", rErr.Err.Error())
}
}
func UploadLargeFileObjects(minioClient *minio.Client) {
uploadInfo, err := minioClient.FPutObject(context.Background(), "bucket02", "test.csv", "data", minio.PutObjectOptions{
ContentType: "application/csv",
})
if err != nil {
fmt.Println(err)
return
}
fmt.Println("Successfully uploaded object: ", uploadInfo)
}
func DownloadLargeFileObjects(minioClient *minio.Client) {
err := minioClient.FGetObject(context.Background(), "bucket02", "test.csv", "/tmp/myobject", minio.GetObjectOptions{})
if err != nil {
fmt.Println(err)
return
}
}