一、前言
redhat官网给出不同应用场景时,其推荐的Ceph集群服务器硬件配置(包括CPU/Memory/Disk/Network)。仅作为服务器配置选择时参考,不做推荐使用。
场景包含如下:
场景:一侧重IOPS的(IOPS低延迟),例如实时性要求高的,但数据量不大的。比如订单的生成。
场景二:侧重Throughput的 (吞吐量优先),吞吐量大,但IOPS延时性要求适当。例如,实时流。
场景三:侧重容量和价格Cost/Capacity的(大存储量),例如大文件的存储。
二、硬件选型标准
1. 这里主要讲IOPS,Throughput, Cost/Capacity 三种不同场景的应用方案在硬件选型时的标准:
SOLUTIONS |
CPU |
RAM |
Networking |
OSD Media |
OSDs |
Journal Media |
Controller/HBA |
Vendor(i.e.) |
Example Uses |
IOPS-OPTIMIZED |
10 cores per NVMe SSD, assuming a 2 GHz CPU. |
16GB baseline, plus 2GB per OSD |
10 Gigabit Ethernet (GbE) per 12 OSDs (each for client- and cluster-facing networks) |
High-performance, high-endurance enterprise NVMe SSDs |
Four per NVMe SSD |
High-performance, high-endurance enterprise NVMe SSD, co-located with OSDs |
Native PCIe bus |
SuperMICRO(SYS-5038MR-OSD006P) |
Database |
THROUGHPUT-OPTIMIZED |
0.5 cores per HDD, assuming a 2 GHz CPU |
16GB baseline, plus 2GB per OSD |
10 GbE per 12 OSDs (each for client- and cluster-facing networks) |
7,200 RPM enterprise HDDs |
One per HDD |
High-endurance, high-performance enterprise serial-attached SCSI (SAS) or NVMe SSDs; OSD-to-journal ratio: 4-5:1 for an SSD journal, or 12-18:1 for an NVMe journal |
Host bus adapter (HBA): just a bunch of disks (JBOD) |
SuperMICRO(SSG-6028R-OSD072P) |
Streaming media |
COST/CAPACITY-OPTIMIZED |
0.5 cores per HDD, assuming a 2 GHz CPU |
16GB baseline, plus 2GB per OSD |
10 GbE per 12 OSDs (each for client- and cluster-facing networks) |
7,200 RPM enterprise HDDs |
One per HDD |
Co-located on the HDD |
HBA:JBOD |
SuperMICRO(SSG-6048R-OSD216P) |
Archive |
Reference: https://www.redhat.com/en/resources/resources-red-hat-ceph-storage-hardware-selection-guide-html |
|
2. SuperMicro推荐的Ceph服务器机型:
块和文件存储类服务器:OSD 服务器 (CPU负载低)
Category |
Block and File OSD Node |
|
 |
 |
 |
Server SKU |
SSG-6019P-ACR12L+ |
SSG-5029P-E1CTR12L |
SSG-6029P-E1CR12L |
Profile |
Density Optimized |
Cost Optimized |
Compute Optimized |
Form Factor |
1U |
2U |
2U |
CPU |
Dual Intel® Xeon® Scalable Processors (8C/16T each) |
Dual Intel® Xeon® Scalable Processors (12C/24T each) |
Dual Intel® Xeon® Scalable Processors (8C/16T each) |
Memory |
96GB DDR4 RDIMM |
HBA |
Broadcom 3216 (IT mode) |
Broadcom 3008 (IT mode) |
Broadcom 3008 (IT mode) |
Power |
600W Redundant Power Supplies |
800W Redundant Power Supplies |
1200W Redundant Power Supplies |
NIC |
Dual-Port 25G Mellanox (SFP+ compatibility) via AOC |
Storage |
Type |
Qty |
Form Factor |
Type |
Qty |
Form Factor |
Type |
Qty |
Form Factor |
Caching |
Optional NVMe |
Up to 2 |
2.5" 7mm |
Optional NVMe |
1 |
AIC |
Optional NVMe |
1 |
AIC |
Capacity |
6-12 TB 7K RPM |
12 |
3.5" SAS3 HDD |
6-12 TB 7K RPM |
12 |
3.5" SAS3 HDD |
6-12 TB 7K RPM |
12 |
3.5" SAS3 HDD |
Drive (OS) |
2x 2.5" 7mm SATA SSD |
2x 2.5" SATA SSD |
2x 2.5" SATA SSD |
媒体类媒体流和媒体仓库服务器:OSD服务器 (CPU负载适中)
Category |
Media OSD Node (File and Object) |
|
 |
 |
Server SKU |
SSG-6029P-E1CR24L |
SSG-6049P-E1CR45L SSG-6049P-E1CR45L+ |
Profile |
Media Streaming |
Media Repository |
Form Factor |
1U |
2U |
CPU |
Dual Intel® Xeon® Scalable Processors (12C/24T each) |
Dual Intel® Xeon® Scalable Processors (12C/24T each) |
Memory |
192GB DDR4 RDIMM |
384GB DDR4 RDIMM |
HBA |
Broadcom 3008 (IT mode) |
Broadcom 3008 (IT mode)(L model) Broadcom 3216 (IT mode)(L+ model) |
Power |
1600W Redundant Power Supplies |
1600W Redundant Power Supplies |
NIC |
SIOM-based Dual-Port 25G Mellanox (SFP+ compatibility) + Dual-Port 10G RJ45 |
Storage |
Type |
Qty |
Form Factor |
Type |
Qty |
Form Factor |
Caching |
Optional NVMe |
Up to 2 |
AIC |
Optional NVMe |
Up to 6 |
2.5" U.2 |
Capacity |
6-12 TB 7K RPM |
24 |
3.5" SAS3 HDD |
6-12 TB 7K RPM |
45 |
3.5" SAS3 HDD |
Drive (OS) |
2x 2.5" SATA SSD |
2x 2.5" SATA SSD |
媒体类存档服务器:OSD服务器(CPU负载稍高)
Category |
Media OSD Node (File and Object) |
|
 |
Server SKU |
SSG-6049P-E1CR60L SSG-6049P-E1CR60L+ |
Profile |
Archive (minimum recommended deployment size of 10+ OSD Nodes) |
Form Factor |
4U |
CPU |
Dual Intel® Xeon® Scalable Processors (14C/28T each) |
Memory |
384GB DDR4 RDIMM |
HBA |
Broadcom 3008 (IT mode)(L model) Broadcom 3616 (IT mode)(L+ model) |
Power |
2000W Redundant Power Supplies |
NIC |
SIOM-based Dual-Port 25G Mellanox (SFP+ compatibility) + Dual-Port 10G RJ45 |
Storage |
Type |
Qty |
Form Factor |
Caching |
Optional NVMe |
Up to 6 |
2.5" U.2 |
Capacity |
6-12 TB 7K RPM |
60 |
3.5" SAS3 HDD |
Drive (OS) |
2x 2.5" SATA SSD |
Monitor/网关服务器 (CPU负载适中)
Category |
Monitor Node / Gateway |
|
 |
Server SKU |
SYS-1019P-WTR |
Profile |
Flash Performance (minimum recommended deployment size of 3 Monitor Nodes) |
Form Factor |
1U |
CPU |
Single Intel® Xeon® Scalable Processor (12C/24T each) |
Memory |
96GB DDR4 RDIMM |
HBA |
Broadcom 3008 (IT mode)(L model) |
Power |
500W Redundant Power Supplies |
NIC |
Dual-Port 25G Mellanox (SFP+ compatibility) via AOC |
Storage |
Type |
Qty |
Form Factor |
Caching |
N/A |
- |
- |
Capacity |
N/A |
- |
- |
Drive (OS) |
2x 2.5" SATA SSD |
美光固态高速服务器:(CPU负载高)
Category |
Micron Solid Ready |
|
 |
 |
Reference BOM Check with PM for details |
SYS-1028U-MON50M |
SYS-1029U-OSD100M |
Server SKU |
SYS-1028U-TNRT+ |
SYS-1029U-TN10RT |
Profile |
Monitor Node |
OSD Node |
Form Factor |
1U |
1U |
CPU |
Dual Intel® Xeon® Scalable Processors (10C/20T each) |
Dual Intel® Xeon® Scalable Processors (24C/48T each) |
Memory |
128GB DDR4 RDIMM |
384GB DDR4 RDIMM |
HBA |
Broadcom 3008 (IT mode) |
N/A |
Power |
750W Redundant Power Supplies |
1000W Redundant Power Supplies |
NIC |
2x Single-Port 50GbE QSFP28 (Mellanox) |
2x Dual-Port 100GbE QSFP (Mellanox) |
Storage |
Type |
Qty |
Form Factor |
Type |
Qty |
Form Factor |
Caching |
N/A |
- |
- |
Optional NVMe |
- |
- |
Capacity |
N/A |
- |
- |
6.4 TB NVMe |
10 |
U.2 NVMe |
Drive (OS) |
1x SATADOM |
1x M.2 SATA SSD |
三、结论:
可以看到高负载的CPU服务器需求集中在(美光)固态高速服务器 和 媒体类存档服务器(OSD服务器)。由于没有高负载计算过,存储服务器总体来说对CPU要求不高。所以如果你的CPU配置高,CPU负载不高,可以用如下服务器配置优化方案:
- 配置CEPH的副本模式为 EC(Erase Code)来使用更多的CPU资源,减少磁盘IO,而不是配置Replicated。
- 使用低延迟高吞吐率的NVME SSD 盘 和 增加 磁盘数
- 使用Cache Tier 和 增加Cache size: 使用Cache Tier 和 增加 Cache size
- 如果x86服务器功耗和价格不符合你的需求,可以考虑采购arm服务器,在存储领域,arm服务器的性价比很高。
Refer to: https://www.supermicro.org.cn/en/solutions/red-hat-ceph