关于GlusterFS分布式文件系统工作原理及相关概述请参考博文:GlusterFS分布式文件系统概述
一、搭建GlusterFS分布式文件系统群集
此案例相关软件包请访问:https://pan.baidu.com/s/1xazQAkR2Lm31njwu88Z9qA
提取码:qdiy
部署前准备工作:
在所有节点上执行以下操作,添加30G新硬盘,通过fdisk分区、mkfs格式化,创建相应的挂载目录,并将格式化的磁盘挂载到相应的目录中。最后根据实际情况,可以选择修改/etc/fstab配合文件,使其永久生效。具体配置可以参考博文:centos7磁盘文件系统管理(一)
1、所有节点基础配置
[root@centos01 ~]# fdisk /dev/sdb
命令(输入 m 获取帮助):n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
分区号 (1-4,默认 1):
起始 扇区 (2048-62914559,默认为 2048):
将使用默认值 2048
Last 扇区, +扇区 or +size{K,M,G} (2048-62914559,默认为 62914559):+5G
分区 1 已设置为 Linux 类型,大小设为 5 GiB
命令(输入 m 获取帮助):n
Partition type:
p primary (1 primary, 0 extended, 3 free)
e extended
Select (default p): p
分区号 (2-4,默认 2):
起始 扇区 (10487808-62914559,默认为 10487808):
将使用默认值 10487808
Last 扇区, +扇区 or +size{K,M,G} (10487808-62914559,默认为 62914559):+5G
分区 2 已设置为 Linux 类型,大小设为 5 GiB
命令(输入 m 获取帮助):n
Partition type:
p primary (2 primary, 0 extended, 2 free)
e extended
Select (default p): p
分区号 (3,4,默认 3):
起始 扇区 (20973568-62914559,默认为 20973568):
将使用默认值 20973568
Last 扇区, +扇区 or +size{K,M,G} (20973568-62914559,默认为 62914559):+5G
分区 3 已设置为 Linux 类型,大小设为 5 GiB
命令(输入 m 获取帮助):n
Partition type:
p primary (3 primary, 0 extended, 1 free)
e extended
Select (default e): e
已选择分区 4
起始 扇区 (31459328-62914559,默认为 31459328):
将使用默认值 31459328
Last 扇区, +扇区 or +size{K,M,G} (31459328-62914559,默认为 62914559):
将使用默认值 62914559
分区 4 已设置为 Extended 类型,大小设为 15 GiB
命令(输入 m 获取帮助):n
All primary partitions are in use
添加逻辑分区 5
起始 扇区 (31461376-62914559,默认为 31461376):
将使用默认值 31461376
Last 扇区, +扇区 or +size{K,M,G} (31461376-62914559,默认为 62914559):+5G
分区 5 已设置为 Linux 类型,大小设为 5 GiB
命令(输入 m 获取帮助):n
All primary partitions are in use
添加逻辑分区 6
起始 扇区 (41949184-62914559,默认为 41949184):
将使用默认值 41949184
Last 扇区, +扇区 or +size{K,M,G} (41949184-62914559,默认为 62914559):+5G
分区 6 已设置为 Linux 类型,大小设为 5 GiB
命令(输入 m 获取帮助):n
All primary partitions are in use
添加逻辑分区 7
起始 扇区 (52436992-62914559,默认为 52436992):
将使用默认值 52436992
Last 扇区, +扇区 or +size{K,M,G} (52436992-62914559,默认为 62914559):
将使用默认值 62914559
分区 7 已设置为 Linux 类型,大小设为 5 GiB
命令(输入 m 获取帮助):p
磁盘 /dev/sdb:32.2 GB, 32212254720 字节,62914560 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理):512 字节 / 512 字节
I/O 大小(最小/最佳):512 字节 / 512 字节
磁盘标签类型:dos
磁盘标识符:0x0d5bf394
设备 Boot Start End Blocks Id System
/dev/sdb1 2048 10487807 5242880 83 Linux
/dev/sdb2 10487808 20973567 5242880 83 Linux
/dev/sdb3 20973568 31459327 5242880 83 Linux
/dev/sdb4 31459328 62914559 15727616 5 Extended
/dev/sdb5 31461376 41947135 5242880 83 Linux
/dev/sdb6 41949184 52434943 5242880 83 Linux
/dev/sdb7 52436992 62914559 5238784 83 Linux
命令(输入 m 获取帮助):w
The partition table has been altered!
Calling ioctl() to re-read partition table.
正在同步磁盘。
[root@centos01 ~]# fdisk -l
磁盘 /dev/sda:85.9 GB, 85899345920 字节,167772160 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理):512 字节 / 512 字节
I/O 大小(最小/最佳):512 字节 / 512 字节
磁盘标签类型:dos
磁盘标识符:0x000ab7bb
设备 Boot Start End Blocks Id System
/dev/sda1 * 2048 411647 204800 83 Linux
/dev/sda2 411648 8800255 4194304 82 Linux swap / Solaris
/dev/sda3 8800256 167772159 79485952 83 Linux
磁盘 /dev/sdb:32.2 GB, 32212254720 字节,62914560 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理):512 字节 / 512 字节
I/O 大小(最小/最佳):512 字节 / 512 字节
磁盘标签类型:dos
磁盘标识符:0x0d5bf394
设备 Boot Start End Blocks Id System
/dev/sdb1 2048 10487807 5242880 83 Linux
/dev/sdb2 10487808 20973567 5242880 83 Linux
/dev/sdb3 20973568 31459327 5242880 83 Linux
/dev/sdb4 31459328 62914559 15727616 5 Extended
/dev/sdb5 31461376 41947135 5242880 83 Linux
/dev/sdb6 41949184 52434943 5242880 83 Linux
/dev/sdb7 52436992 62914559 5238784 83 Linux
[root@centos01 ~]# mkfs.xfs /dev/sdb1
[root@centos01 ~]# mkfs.xfs /dev/sdb2
[root@centos01 ~]# mkfs.xfs /dev/sdb3
[root@centos01 ~]# mkfs.xfs /dev/sdb5
[root@centos01 ~]# mkfs.xfs /dev/sdb6
[root@centos01 ~]# mkfs.xfs /dev/sdb7
[root@centos01 ~]# mkdir /sdb{1..7}
[root@centos01 ~]# echo "/dev/sdb1 /sdb1 xfs defaults 0 0" >> /etc/fstab
[root@centos01 ~]# echo "/dev/sdb2 /sdb2 xfs defaults 0 0" >> /etc/fstab
[root@centos01 ~]# echo "/dev/sdb3 /sdb3 xfs defaults 0 0" >> /etc/fstab
[root@centos01 ~]# echo "/dev/sdb5 /sdb5 xfs defaults 0 0" >> /etc/fstab
[root@centos01 ~]# echo "/dev/sdb6 /sdb6 xfs defaults 0 0" >> /etc/fstab
[root@centos01 ~]# echo "/dev/sdb7 /sdb7 xfs defaults 0 0" >> /etc/fstab
[root@centos01 ~]# scp /etc/fstab [email protected]:/etc/
[root@centos01 ~]# scp /etc/fstab [email protected]:/etc/
[root@centos01 ~]# scp /etc/fstab [email protected]:/etc/
[root@centos01 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.10 centos01
192.168.100.20 centos02
192.168.100.30 centos03
192.168.100.40 centos04
[root@centos01 ~]# scp /etc/hosts [email protected]:/etc/
[root@centos01 ~]# scp /etc/hosts [email protected]:/etc/
[root@centos01 ~]# scp /etc/hosts [email protected]:/etc/
[root@centos01 ~]# scp /etc/hosts [email protected]:/etc/
[root@centos01 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
/dev/sda3 xfs 76G 3.6G 73G 5% /
devtmpfs devtmpfs 474M 0 474M 0% /dev
tmpfs tmpfs 489M 0 489M 0% /dev/shm
tmpfs tmpfs 489M 7.0M 482M 2% /run
tmpfs tmpfs 489M 0 489M 0% /sys/fs/cgroup
/dev/sdb7 xfs 5.0G 33M 5.0G 1% /sdb7
/dev/sdb2 xfs 5.0G 33M 5.0G 1% /sdb2
/dev/sdb5 xfs 5.0G 33M 5.0G 1% /sdb5
/dev/sdb6 xfs 5.0G 33M 5.0G 1% /sdb6
/dev/sdb1 xfs 5.0G 33M 5.0G 1% /sdb1
/dev/sdb3 xfs 5.0G 33M 5.0G 1% /sdb3
/dev/sda1 xfs 197M 136M 61M 70% /boot
tmpfs tmpfs 98M 0 98M 0% /run/user/0
2、centos01安装glusterFS
[root@centos01 ~]# mount /dev/cdrom /mnt/
mount: /dev/sr0 写保护,将以只读方式挂载
[root@centos01 ~]# rm -rf /etc/yum.repos.d/CentOS-*
[root@centos01 ~]# cd /mnt/
[root@centos01 mnt]# ls
gfsrepo
[root@centos01 mnt]# vim /etc/yum.repos.d/local.repo
[local]
name=centos
baseurl=file:///mnt/gfsrepo
enabled=1
gpgcheck=0
[root@centos01 ~]# yum -y install glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma
[root@centos01 ~]# systemctl start glusterd
[root@centos01 ~]# systemctl enable glusterd
[root@centos01 ~]# scp /etc/yum.repos.d/local.repo [email protected]:/etc/yum.repos.d/
[root@centos01 ~]# scp /etc/yum.repos.d/local.repo [email protected]:/etc/yum.repos.d/
[root@centos01 ~]# scp /etc/yum.repos.d/local.repo [email protected]:/etc/yum.repos.d/
[root@centos01 ~]# scp /etc/yum.repos.d/local.repo [email protected]:/etc/yum.repos.d/
至此,centos01配置完成了,centos02~04重复centos01的配置即可,自行配置。centos05只需要按照客户端依赖程序即可
3、创建glusterFS群集
(接下来的所有配置,在任意一台节点上执行即可,我这里在centos01上执行)
[root@centos01 ~]# gluster peer probe centos02
[root@centos01 ~]# gluster peer probe centos03
[root@centos01 ~]# gluster peer probe centos04
[root@centos01 ~]# gluster peer status
Number of Peers: 3
Hostname: centos02
Uuid: 4cc7469c-9d8b-4fce-9d5c-f0132f4faab9
State: Peer in Cluster (Connected)
Hostname: centos03
Uuid: 200c060d-9ad6-4e61-9065-2637599f3865
State: Peer in Cluster (Connected)
Hostname: centos04
Uuid: e548bbe0-5796-4ef2-a990-017379f96e08
State: Peer in Cluster (Connected)
二、创建各种类型的卷
1、创建分布式卷
[root@centos01 ~]# gluster volume create FBS-Volume centos01:/sdb1 centos02:/sdb1 force
[root@centos01 ~]# gluster volume info FBS-Volume
Volume Name: FBS-Volume
Type: Distribute
Volume ID: e57ab3d2-ab2a-43fe-b035-1a5b0e290db2
Status: Created
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb1
Brick2: centos02:/sdb1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@centos01 ~]# gluster volume start FBS-Volume
[root@centos05 ~]# mkdir /FBS-Volume
[root@centos05 ~]# mount -t glusterfs 192.168.100.10:FBS-Volume /FBS-Volume
[root@centos05 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
192.168.100.10:FBS-Volume fuse.glusterfs 10G 65M 10G 1% /FBS-Volume
2、创建条带卷
[root@centos01 ~]# gluster volume create TD-Volume stripe 2 centos01:/sdb2 centos02:/sdb2 force
[root@centos01 ~]# gluster volume info TD-Volume
Volume Name: TD-Volume
Type: Stripe
Volume ID: c39cb0ee-ae37-4e87-9847-e331cdac2b14
Status: Created
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb2
Brick2: centos02:/sdb2
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@centos01 ~]# gluster volume start TD-Volume
[root@centos05 ~]# mkdir /TD-Volume
[root@centos05 ~]# mount -t glusterfs 192.168.100.10:TD-Volume /TD-Volume
[root@centos05 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
192.168.100.10:FBS-Volume fuse.glusterfs 10G 65M 10G 1% /FBS-Volume
192.168.100.10:TD-Volume fuse.glusterfs 10G 65M 10G 1% /TD-Volume
3、创建复制卷
[root@centos01 ~]# gluster volume create FZ-Volume replica 2 centos01:/sdb3 centos02:/sdb3 force
[root@centos01 ~]# gluster volume info FZ-Volume
Volume Name: FZ-Volume
Type: Replicate
Volume ID: bd1059ef-8822-428e-9e00-1da97f54411a
Status: Created
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb3
Brick2: centos02:/sdb3
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@centos01 ~]# gluster volume start FZ-Volume
[root@centos05 ~]# mkdir /FZ-Volume
[root@centos05 ~]# mount -t glusterfs 192.168.100.10:FZ-Volume /FZ-Volume
[root@centos05 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
192.168.100.10:FBS-Volume fuse.glusterfs 10G 65M 10G 1% /FBS-Volume
192.168.100.10:TD-Volume fuse.glusterfs 10G 65M 10G 1% /TD-Volume
192.168.100.10:FZ-Volume fuse.glusterfs 5.0G 33M 5.0G 1% /FZ-Volume
4、创建分布式条带卷
[root@centos01 ~]# gluster volume create FBSTD-Volume stripe 2 transport tcp
centos01:/sdb5 centos02:/sdb05 centos01:/sdb6 centos02:/sdb6 force
[root@centos01 ~]# gluster volume info FBSTD-Volume
Volume Name: FBSTD-Volume
Type: Distributed-Stripe
Volume ID: c6fd3205-bc95-4d22-bfab-15442b367587
Status: Created
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb5
Brick2: centos02:/sdb05
Brick3: centos01:/sdb6
Brick4: centos02:/sdb6
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@centos01 ~]# gluster volume start FBSTD-Volume
[root@centos05 ~]# mkdir /FBSTD-Volume
[root@centos05 ~]# mount -t glusterfs 192.168.100.10:FBSTD-Volume /FBSTD-Volume/
[root@centos05 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
192.168.100.10:FBS-Volume fuse.glusterfs 10G 65M 10G 1% /FBS-Volume
192.168.100.10:TD-Volume fuse.glusterfs 10G 65M 10G 1% /TD-Volume
192.168.100.10:FZ-Volume fuse.glusterfs 5.0G 33M 5.0G 1% /FZ-Volume
192.168.100.10:FBSTD-Volume fuse.glusterfs 91G 3.7G 88G 5% /FBSTD-Volume
5、创建分布式复制卷
[root@centos01 ~]# gluster volume create FBSFZ-Volume replica 2 transport tcp
centos01:/sdb7 centos02:/sdb7 centos03:/sdb1 centos04:/sdb1 force
[root@centos01 ~]# gluster volume info FBSFZ-Volume
Volume Name: FBSFZ-Volume
Type: Distributed-Replicate
Volume ID: 4be9bc3a-e91b-46ae-b2c8-f80251b8a001
Status: Created
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb7
Brick2: centos02:/sdb7
Brick3: centos03:/sdb1
Brick4: centos04:/sdb1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@centos01 ~]# gluster volume start FBSFZ-Volume
[root@centos05 ~]# mkdir /FBSFZ-Volume
[root@centos05 ~]# mount -t glusterfs 192.168.100.10:FBSFZ-Volume /FBSFZ-Volume
[root@centos05 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
192.168.100.10:FBS-Volume fuse.glusterfs 10G 65M 10G 1% /FBS-Volume
192.168.100.10:TD-Volume fuse.glusterfs 10G 65M 10G 1% /TD-Volume
192.168.100.10:FZ-Volume fuse.glusterfs 5.0G 33M 5.0G 1% /FZ-Volume
192.168.100.10:FBSTD-Volume fuse.glusterfs 91G 3.7G 88G 5% /FBSTD-Volume
192.168.100.10:FBSFZ-Volume fuse.glusterfs 10G 65M 10G 1% /FBSFZ-Volume
三、设置自动挂载glusterFS文件系统
[root@centos05 ~]# vim /etc/fstab
192.168.100.10:FBS-Volume /FBS-Volume glusterfs defaults,_netdev 0 0
192.168.100.10:TD-Volume /TD-Volume glusterfs defaults,_netdev 0 0
192.168.100.10:FZ-Volume /FZ-Volume glusterfs defaults,_netdev 0 0
192.168.100.10:FBSTD-Volume /FBSTD-Volume glusterfs defaults,_netdev 0 0
192.168.100.10:FBSFZ-Volume /FBSFZ-Volume glusterfs defaults,_netdev 0 0
[root@centos05 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
192.168.100.10:FZ-Volume fuse.glusterfs 5.0G 33M 5.0G 1% /FZ-Volume
192.168.100.10:FBS-Volume fuse.glusterfs 10G 65M 10G 1% /FBS-Volume
192.168.100.10:FBSTD-Volume fuse.glusterfs 91G 3.7G 88G 5% /FBSTD-Volume
192.168.100.10:TD-Volume fuse.glusterfs 10G 65M 10G 1% /TD-Volume
192.168.100.10:FBSFZ-Volume fuse.glusterfs 10G 65M 10G 1% /FBSFZ-Volume
四、glusterFS相关命令
[root@centos01 ~]# gluster volume list
FBS-Volume
FBSFZ-Volume
FBSTD-Volume
FZ-Volume
TD-Volume
[root@centos01 ~]# gluster volume status
Status of volume: FBS-Volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick centos01:/sdb1 49152 0 Y 2413
Brick centos02:/sdb1 49152 0 Y 2231
Task Status of Volume FBS-Volume
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: FBSFZ-Volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick centos01:/sdb7 49157 0 Y 3585
Brick centos02:/sdb7 49157 0 Y 3119
Brick centos03:/sdb1 49152 0 Y 2877
Brick centos04:/sdb1 49152 0 Y 2746
Self-heal Daemon on localhost N/A N/A Y 3605
Self-heal Daemon on centos03 N/A N/A Y 2897
Self-heal Daemon on centos02 N/A N/A Y 3139
Self-heal Daemon on centos04 N/A N/A Y 2766
Task Status of Volume FBSFZ-Volume
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: FBSTD-Volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick centos01:/sdb5 49155 0 Y 3393
Brick centos02:/sdb05 49155 0 Y 2889
Brick centos01:/sdb6 49156 0 Y 3412
Brick centos02:/sdb6 49156 0 Y 2908
Task Status of Volume FBSTD-Volume
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: FZ-Volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick centos01:/sdb3 49154 0 Y 3214
Brick centos02:/sdb3 49154 0 Y 2787
Self-heal Daemon on localhost N/A N/A Y 3605
Self-heal Daemon on centos03 N/A N/A Y 2897
Self-heal Daemon on centos02 N/A N/A Y 3139
Self-heal Daemon on centos04 N/A N/A Y 2766
Task Status of Volume FZ-Volume
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: TD-Volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick centos01:/sdb2 49153 0 Y 3003
Brick centos02:/sdb2 49153 0 Y 2667
Task Status of Volume TD-Volume
------------------------------------------------------------------------------
There are no active volume tasks
[root@centos01 ~]# gluster volume info
Volume Name: FBS-Volume
Type: Distribute
Volume ID: e57ab3d2-ab2a-43fe-b035-1a5b0e290db2
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb1
Brick2: centos02:/sdb1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
Volume Name: FBSTD-Volume
Type: Distributed-Stripe
Volume ID: c6fd3205-bc95-4d22-bfab-15442b367587
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb5
Brick2: centos02:/sdb05
Brick3: centos01:/sdb6
Brick4: centos02:/sdb6
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
Volume Name: FZ-Volume
Type: Replicate
Volume ID: bd1059ef-8822-428e-9e00-1da97f54411a
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb3
Brick2: centos02:/sdb3
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
Volume Name: TD-Volume
Type: Stripe
Volume ID: c39cb0ee-ae37-4e87-9847-e331cdac2b14
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: centos01:/sdb2
Brick2: centos02:/sdb2
Options Reconfigured:
auth.allow: 192.168.100.10,192.168.100.20,192.168.100.50
transport.address-family: inet
nfs.disable: on
[root@centos01 ~]# gluster volume stop FBSFZ-Volume
[root@centos01 ~]# gluster volume delete FBSFZ-Volume
[root@centos01 ~]# gluster volume set TD-Volume auth.allow
192.168.100.10,192.168.100.20,192.168.100.50
[root@centos05 ~]# mount -t glusterfs 192.168.100.10:TD-Volume /TD-Volume
[root@centos05 ~]# df -Th
文件系统 类型 容量 已用 可用 已用% 挂载点
192.168.100.10:FZ-Volume fuse.glusterfs 5.0G 33M 5.0G 1% /FZ-Volume
192.168.100.10:FBS-Volume fuse.glusterfs 10G 65M 10G 1% /FBS-Volume
192.168.100.10:FBSTD-Volume fuse.glusterfs 91G 3.7G 88G 5% /FBSTD-Volume
192.168.100.10:FBSFZ-Volume fuse.glusterfs 10G 65M 10G 1% /FBSFZ-Volume
tmpfs tmpfs 98M 0 98M 0% /run/user/0
192.168.100.10:TD-Volume fuse.glusterfs 10G 65M 10G 1% /TD-Volume
———————— 本文至此结束,感谢阅读 ————————