前言
已经实施过2次Hadoop大数据平台的基础环境搭建，对于硬盘的RAID模式都不约而同选择了JBOD，毕竟大数据对于文件系统的管理本身就是基于分布式，由至少3份副本来保障数据可靠性，只是不做RAID后的SAS硬盘读写性能就有点惨不忍睹。这次Hadoop大数据平台工具选择的是Cloudera，大家感兴趣的化也可以浏览下它的官网。本篇文章主要介绍如何实现在不停机状态下在线更换JBOD硬盘。
利用StorCLI命令不停机在线更换JBOD硬盘
更新历史
2016年12月31日 - 初稿
阅读原文 - https://wsgzao.github.io/post/jbod/
扩展阅读
Cloudera - http://www.cloudera.com/
JBOD原理
JBOD （ Just a Bunch Of Disks ）不是标准的 RAID 等级，它通常用来表示一个没有控制软件提供协调控制的磁盘集合。 JBOD 将多个物理磁盘串联起来，提供一个巨大的逻辑磁盘。 JBOD 的数据存放机制是由第一块磁盘开始按顺序往后存储，当前磁盘存储空间用完后，再依次往后面的磁盘存储数据。 JBOD 存储性能完全等同于单块磁盘，而且也不提供数据安全保护。它只是简单提供一种扩展存储空间的机制， JBOD 可用存储容量等于所有成员磁盘的存储空间之和。目前 JBOD 常指磁盘柜，而不论其是否提供 RAID 功能。
RAID磁盘阵列配置和调优小结 - https://wsgzao.github.io/post/raid/
参考文档和工具
StorCLI_RefMan_revf
https://www.broadcom.com/support/download-search/?dk=StorCLI
StorCLI (Command Line) Utility for Storage Management v1.07.07 on Linux
https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5092950
使用方式
#查询硬件RAID卡信息
[root@localhost Linux]# cat /proc/scsi/scsi |more
Attached devices:
Host: scsi0 Channel: 02 Id: 00 Lun: 00
Vendor: IBM Model: ServeRAID M5210 Rev: 4.62
Type: Direct-Access ANSI SCSI revision: 05
#解压安装storcli
unzip ibm_utl_sraidmr_storcli-1.07.07_linux_32-64.zip
rpm -Uvh storcli-1.07.07-1.noarch.rpm
#设置软连接
ln -s /opt/MegaRAID/storcli/storcli64 /bin/storcli
ln -s /opt/MegaRAID/storcli/storcli64 /sbin/storcli
#Raid数量及当前状态
[root@localhost Linux]# storcli show ctrlcount
Status Code = 0
Status = Success
Description = None
Controller Count = 2
#查询虚拟磁盘大小，raid结构等
[root@localhost Linux]# storcli /c0 /v0 show
Controller = 0
Status = Success
Description = None
Virtual Drives :
==============
-----------------------------------------------------------
DG/VD TYPE State Access Consist Cache sCC Size Name
-----------------------------------------------------------
0/0 RAID1 Optl RW Yes RWTD - 278.465 GB
-----------------------------------------------------------
Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|B=Blocked|Consist=Consistent|
R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency
#可以看出此为RAID1，一个虚拟磁盘，大小为278.465 GB
#查看control0的物理硬盘
[root@localhost Linux]# storcli /c0 /eall /sall show
Controller = 0
Status = Success
Description = Show Drive Information Succeeded.
Drive Information :
=================
-------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
-------------------------------------------------------------------------
252:16 8 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U
252:17 9 Onln 0 278.465 GB SAS HDD N Y 512B ST300MP0035 X U
-------------------------------------------------------------------------
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded
#可以看到总计有两颗硬盘，每一颗大小278.465GB，是否在线等
#查看control1的物理硬盘
[root@localhost Linux]# storcli /c1 /eall /sall show
Controller = 1
Status = Success
Description = Show Drive Information Succeeded.
Drive Information :
=================
------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
------------------------------------------------------------------------
8:0 23 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:1 24 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:2 22 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:3 19 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:4 13 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:5 14 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:6 18 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:7 12 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:8 10 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:9 16 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:10 11 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:11 21 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:12 15 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
8:13 9 JBOD - 1.0810 TB SAS HDD N Y 512B ST1200MM0018 X U
------------------------------------------------------------------------
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded
#可以看到总计有两颗硬盘，每一颗大小1.0810 TB，是否在线等
#设置硬盘状态
storcli /c1/e8/s0 show all
storcli /c1/e8/s0 start locate
storcli /c1/e8/s0 set offline
storcli /c1/e8/s0 set online
storcli /c1/e8/s0 set jbod
StorCLI Help
#查看StorCLI帮助信息
[root@localhost Linux]# storcli -h
Storage Command Line Tool Ver 1.07.07 Nov 15, 2013
(c)Copyright 2013, LSI Corporation, All Rights Reserved.
storcli -v
storcli -h| -help| ?
storcli -h| -help| ? legacy
storcli show
storcli show all
storcli show ctrlcount
storcli /cx add vd r[0|1|5|6|00|10|50|60]
[Size=,,..|all] [name=,..]
drives=e:s|e:s-x|e:s-x,y,e:s-x,y,z [PDperArray=x][SED]
[pdcache=on|off|default][pi][DimmerSwitch(ds)=default|automatic(auto)|
none|maximum(max)|MaximumWithoutCaching(maxnocache)]
[wt|wb|awb][nora|ra] [direct|cached] [cachevd]
[Strip=<8|16|32|64|128|256|512|1024>] [AfterVd=X]
[Spares = [e:]s|[e:]s-x|[e:]s-x,y] [force][ExclusiveAccess]
storcli /cx add vd each r0 [name=,..] [drives=e:s|e:s-x|e:s-x,y]
[SED] [pdcache=on|off|default][pi] [DimmerSwitch(ds)=default|automatic(auto)|
none|maximum(max)|MaximumWithoutCaching(maxnocache)]
[wt|wb|awb] [nora|ra] [direct|cached]
[Strip=<8|16|32|64|128|256|512|1024>] [ExclusiveAccess]
storcli /cx add VD cachecade|nytrocache r[aid][0,1,10, 1EC]
drives = [e:]s|[e:]s-x|[e:]s-x,y [WT| WB] [assignvds = 0,1,2] [BOOTVOLSIZE=x]
storcli /cx/ex show all
storcli /cx/ex show status
storcli /cx/vx del [cachecade] [force]
storcli /cx delete events
storcli /cx show events [[type= ] filter=]
file=
storcli /cx show eventloginfo
storcli /cx show health [all]
storcli /cx delete securitykey
storcli /cx set securitykey=xxxxxxxx {passphrase=xxxx} {keyid=xxx}
storcli /cx start Diag Duration=
storcli /cx set securitykey keyid=xxx
storcli /cx compare securitykey=xxxxxxxxxx
storcli /cx set termlog[=on|off|offthisboot]
storcli /cx show termlog [type=config|contents]
storcli /cx delete termlog
storcli /cx Compare factorysettings file=
storcli /cx Compare factorysettings raw file=
storcli /cx set securitykey=xxxxxxxx oldsecuritykey=xxxxxxxx
{passphrase=xxxx} {keyid=xxx}
storcli /cx/dx show
storcli /cx/dall show cachecade
storcli /cx/dx show all
storcli /cx show freespace
storcli /cx/fall show [all] [securityKey = xxx]
storcli /cx/fall del|delete [securityKey = xxx]
storcli /cx/fall import [preview] [securityKey = xxx]
storcli /cx/vx set ssdcaching=on|off
storcli /cx/dx set security=on
storcli /cx/vx show expansion
storcli /cx show fshinting
storcli /cx/vx set fshinting=
storcli /cx/vx expand Size= [expandarray]
storcli /cx[/ex]/sx show
storcli /cx[/ex]/sx show all
storcli /cx[/ex]/sx start rebuild
storcli /cx[/ex]/sx stop rebuild
storcli /cx[/ex]/sx pause rebuild
storcli /cx[/ex]/sx resume rebuild
storcli /cx[/ex]/sx show rebuild
storcli /cx[/ex]/sx show health
storcli /cx[/ex]/sx start copyback target=e:s
storcli /cx[/ex]/sx stop copyback
storcli /cx[/ex]/sx pause copyback
storcli /cx[/ex]/sx resume copyback
storcli /cx[/ex]/sx show copyback
storcli /cx[/ex]/sx start initialization
storcli /cx[/ex]/sx stop initialization
storcli /cx[/ex]/sx show initialization
storcli /cx[/ex]/sx start locate
storcli /cx[/ex]/sx stop locate
storcli /cx[/ex]/sx show securitykey keyid
storcli /cx[/ex]/sx add hotsparedrive [DGs=] [enclaffinity]
[nonrevertible]
storcli /cx[/ex]/sx delete hotsparedrive
storcli /cx[/ex]/sx spinup
storcli /cx[/ex]/sx spindown
storcli /cx[/ex]/sx set online
storcli /cx[/ex]/sx set offline
storcli /cx[/ex]/sx set missing
storcli /cx[/ex]/sx set jbod
storcli /cx[/ex]/sx set good [force]
storcli /cx[/ex]/sx insert dg=A array=B row=C
storcli /cx/vx set emulationType=0|1
storcli /cx/vx set wrcache=WT|WB|AWB
storcli /cx/vx set rdcache=RA|NoRA
storcli /cx/vx set iopolicy=Cached|Direct
storcli /cx/vx set accesspolicy=RW|RO|Blocked|RmvBlkd
storcli /cx/vx set pdcache=On|Off|Default
storcli /cx/vx set name=
storcli /cx/vx set HostAccess=ExclusiveAccess|SharedAccess
storcli /cx/vx set ds=Default|Auto|None|Max|MaxNoCache
storcli /cx/vx set autobgi=On|Off
storcli /cx/vx set pi=Off
storcli /cx/vx show
storcli /cx/vx show all
storcli /cx/vx show init
storcli /cx/vx show cc
storcli /cx/vx show erase
storcli /cx/vx show migrate
storcli /cx/vx show bgi
storcli /cx/vx show autobgi
storcli /cx/vx show trim
storcli /cx set consistencycheck|cc=[off|seq|conc] [delay=value]
starttime=yyyy/mm/dd hh] [excludevd=x-y,z]
storcli /cx show cc|consistencycheck
storcli /cx show ocr
storcli /cx set ocr=
storcli /cx/vx start init[Full][Force]
storcli /cx/vx start erase [simple|normal|thorough|standard][patternA=]
[patternB=]
storcli /cx/vx start cc[Force]
storcli /cx/vx start migrate type=raidx [option=add|remove
drives=[e:]s|[e:]s-x|[e:]s-x,y] [Force]
storcli /cx/vx stop init
storcli /cx/vx stop erase
storcli /cx/vx stop cc
storcli /cx/vx stop bgi
storcli /cx/vx pause cc
storcli /cx/vx pause bgi
storcli /cx/vx resume cc
storcli /cx/vx resume bgi
storcli /cx show
storcli /cx show all
storcli /cx show preservedcache
storcli /cx/vx delete preservedcache[force]
storcli /cx[/ex]/sx download src= [satabridge] [mode= 5|7]
storcli /cx/ex download src= [forceActivate]
storcli /cx[/ex]/sx secureerase [force]
storcli /cx[/ex]/sx start erase [simple| normal| thorough | standard| threepass]
[patternA=] [patternB=]
storcli /cx[/ex]/sx stop erase
storcli /cx[/ex]/sx show erase
storcli /cx[/ex]/sx show rawdata pageaddr= file=
storcli /cx[/ex]/sx set bootdrive=
storcli /cx/vx set bootdrive=
storcli /cx show bootdrive
storcli /cx show bootwithpinnedcache
storcli /cx set bootwithpinnedcache=
storcli /cx show activityforlocate
storcli /cx set activityforlocate=
storcli /cx show copyback
storcli /cx set copyback= type=smartssd|smarthdd|all
storcli /cx show jbod
storcli /cx set jbod=
storcli /cx set autorebuild=
storcli /cx show autorebuild
storcli /cx show cachebypass
storcli /cx set cachebypass=
storcli /cx show usefdeonlyencrypt
storcli /cx set usefdeonlyencrypt=
storcli /cx show prcorrectunconfiguredareas
storcli /cx set prcorrectunconfiguredareas=
storcli /cx show batterywarning
storcli /cx set batterywarning=
storcli /cx show abortcconerror
storcli /cx set abortcconerror=
storcli /cx show ncq
storcli /cx set ncq=
storcli /cx show maintainpdfailhistory
storcli /cx set maintainpdfailhistory=
storcli /cx show restorehotspare
storcli /cx set restorehotspare=
storcli /cx set bios [state=] [Mode=] [abs=]
storcli /cx show bios
storcli /cx show alarm
storcli /cx set alarm=
storcli /cx show foreignautoimport
storcli /cx set foreignautoimport=
storcli /cx show directpdmapping
storcli /cx set directpdmapping=
storcli /cx show rebuildrate
storcli /cx set rebuildrate=
storcli /cx show loadbalancemode
storcli /cx set loadbalancemode=
storcli /cx show eghs
storcli /cx set eghs [state=] [eug=] [smarter=]
storcli /cx show cacheflushint
storcli /cx set cacheflushint=
storcli /cx show prrate
storcli /cx set prrate=
storcli /cx show ccrate
storcli /cx set ccrate=
storcli /cx show bgirate
storcli /cx set bgirate =
storcli /cx show dpm
storcli /cx set dpm =
storcli /cx show sgpioforce
storcli /cx set sgpioforce =
storcli /cx set supportssdpatrolread =
storcli /cx show migraterate
storcli /cx set migraterate=
storcli /cx show spinupdrivecount
storcli /cx show wbsupport
storcli /cx set spinupdrivecount=
storcli /cx show spinupdelay
storcli /cx set spinupdelay=
storcli /cx show coercion
storcli /cx set coercion=
storcli /cx show smartpollinterval
storcli /cx set smartpollinterval=
storcli /cx show eccbucketsize
storcli /cx set eccbucketsize=
storcli /cx show eccbucketleakrate
storcli /cx set eccbucketleakrate=
storcli /cx show backplane
storcli /cx set backplane mode= expose=
storcli /cx show perfmode
storcli /cx set perfmode= [maxflushlines= numiostoorder=]
storcli /cx show perfmodevalues
storcli /cx show pi
storcli /cx set pi [state=] [import=]
storcli /cx show time
storcli /cx set time=
storcli /cx show ds
storcli /cx set ds=OFF type=1|2|3|4
storcli /cx set ds=ON type=1|2 [properties]
storcli /cx set ds=ON type=3|4 DefaultLdType= [properties]
storcli /cx set ds [properties]
storcli /cx show safeid
storcli /cx show rehostinfo
storcli /cx show pci
storcli /cx show ASO
storcli /cx set aso key= preview
storcli /cx set aso key=
storcli /cx set aso transfertovault
storcli /cx set aso rehostcomplete
storcli /cx set aso deactivatetrialkey
storcli /cx set factory defaults
storcli /cx download file= [fwtype=] [ResetNow] [nosigchk]
[noverchk]
storcli /cx flush|flushcache
storcli /cx/px show
storcli /cx/px show all
storcli /cx/px set linkspeed=0|1.5|3|6|12
storcli /cx/bbu show
storcli /cx/bbu set SCAPVPD file= VPDPage=
storcli /cx/bbu show all
storcli /cx/bbu show status
storcli /cx/bbu show properties
storcli /cx/bbu show learn
storcli /cx/bbu show gasgauge Offset=xxxx Numbytes=n
storcli /cx/bbu start learn
storcli /cx/bbu show modes
storcli /cx/bbu set [ learnDelayInterval= | bbuMode=
|learnStartTime=[DDD HH | off] | autolearnmode= | powermode=sleep | writeaccess=sealed ]
storcli /cx/cv show
storcli /cx/cv show all
storcli /cx/cv show status
storcli /cx/cv show learn
storcli /cx/cv start learn
storcli /cx show securitykey keyid
storcli /cx start patrolread
storcli /cx stop patrolread
storcli /cx pause patrolread
storcli /cx resume patrolread
storcli /cx show patrolRead
storcli /cx show powermonitoringinfo
storcli /cx set patrolread = {{on mode= }|{off}}
storcli /cx set patrolread [starttime=< yyyy/mm/dd hh>]
[maxconcurrentpd =] [includessds=]
[uncfgareas=on|off]
storcli /cx set patrolread delay =
storcli /cx[/ex]/sx show diag paniclog [Query] | [ExtractSlot=x]
| [EraseSlot=x] [file=filepath]
storcli /cx[/ex]/sx show diag smartlog [file=filepath]
storcli /cx[/ex]/sx show diag errorlog [file=filepath]
storcli /cx del Nytrocache [force]
storcli /cx show badblocks
storcli /cx flasherase
storcli /cx shutdown
storcli /cx/mx set mode=
storcli /cx/mx show
storcli /cx transform iMR
storcli /cx restart
storcli /cx/vx show BBMT
storcli /cx/vx delete BBMT
storcli /xd add type=CacheDev
[force]
storcli /xd add type=VirtualDrive
storcli /xd delete type=
storcli /xd show type= [state=]
storcli /xd show
storcli /xd show perfmon
storcli /xd/wd show
storcli /xd/wd show safeid
storcli /xd set online [Vds=comma separated VD ids]
storcli /xd/wdX apply key=value
storcli /cx[/ex]/sx start format [thorough]
storcli /cx show dequeuelog file=
Note:Use 'page=[x]'as the last option in all the commands to set the page break.
X=lines per page. E.g. 'storcli help page=10'