记一次DRBD Unknown故障处理过程

配置drbd过程出现Primary/Unknown 故障，最后通过如下方式解决。

1， 节点状态查看

(1) 主节点状态

[root@app1 drbd.d]# cat /proc/drbd    
version: 8.4.3 (api:1/proto:86-101)    
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00    
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----    
    ns:0 nr:0 dw:0 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:604    
[root@app1 drbd.d]#

(2) 从节点状态

[root@app2 ~]# cat /proc/drbd    
version: 8.4.3 (api:1/proto:86-101)    
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00    
0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown   r-----    
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:548    
[root@app2 ~]#

2. 这里确认以主节点的数据为准，重新同步到从节点

(1) 停止app2 drbd服务

[root@app2 ~]# service drbd stop  
Stopping all DRBD resources: .    
[root@app2 ~]#

(2) 重新初始化元数据

[root@app2 ~]# drbdadm create-md data  
You want me to create a v08 style flexible-size internal meta data block.    
There appears to be a v08 flexible-size internal meta data block    
already in place on /dev/sdb1 at byte offset 5364318208    
Do you really want to overwrite the existing v08 meta-data?    
[need to type 'yes' to confirm] yes

Writing meta data...  
md_offset 5364318208    
al_offset 5364285440    
bm_offset 5364121600

Found ext3 filesystem  
     5238400 kB data area apparently used    
     5238400 kB left usable by current configuration

Even though it looks like this would place the new meta data into  
unused space, you still need to confirm, as this is only a guess.

Do you want to proceed?  
[need to type 'yes' to confirm] yes

initializing activity log  
NOT initializing bitmap    
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory    
New drbd meta data block successfully created.    
lk_bdev_save(/var/lib/drbd/drbd-minor-0.lkbd) failed: No such file or directory

(3) 启动drbd服务

[root@app2 ~]# service drbd start  
Starting DRBD resources: [    
     create res: data    
   prepare disk: data    
    adjust disk: data    
     adjust net: data    
]    
..........    
***************************************************************    
DRBD's startup script waits for the peer node(s) to appear.    
- In case this node was already a degraded cluster before the    
   reboot the timeout is 0 seconds. [degr-wfc-timeout]    
- If the peer was available before the reboot the timeout will    
   expire after 0 seconds. [wfc-timeout]    
   (These values are for resource 'data'; 0 sec -> wait forever)    
To abort waiting enter 'yes' [  15]:se

.  
[root@app2 ~]# cat /proc/drbd   
version: 8.4.3 (api:1/proto:86-101)    
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00    
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----    
    ns:0 nr:5238400 dw:5238400 dr:0 al:0 bm:320 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0    
[root@app2 ~]#

3. app1主节点下

(1) 主节点状态正常了

[root@app1 ~]# cat /proc/drbd    
version: 8.4.3 (api:1/proto:86-101)    
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00    
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r-----    
    ns:0 nr:0 dw:0 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:604

(2) 重启drbd之后，数据重新同步到从节点

[root@app1 ~]# service drbd reload  
Reloading DRBD configuration: .    
[root@app1 ~]# cat /proc/drbd    
version: 8.4.3 (api:1/proto:86-101)    
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00    
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n-    
    ns:176816 nr:0 dw:0 dr:180896 al:0 bm:10 lo:4 pe:2 ua:8 ap:0 ep:1 wo:d oos:5063296    
        [>....................] sync'ed:  3.4% (4944/5112)M    
        finish: 0:00:57 speed: 87,552 (87,552) K/sec    
[root@app1 ~]# cat /proc/drbd    
version: 8.4.3 (api:1/proto:86-101)    
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00    
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n-    
    ns:3541004 nr:0 dw:0 dr:3545760 al:0 bm:215 lo:2 pe:4 ua:6 ap:0 ep:1 wo:d oos:1700480    
        [============>.......] sync'ed: 67.6% (1660/5112)M    
        finish: 0:00:23 speed: 71,780 (69,368) K/sec    
[root@app1 ~]# cat /proc/drbd    
version: 8.4.3 (api:1/proto:86-101)    
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by gardner@, 2013-11-29 12:28:00    
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----    
    ns:5238400 nr:0 dw:0 dr:5239072 al:0 bm:320 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0    
[root@app1 ~]#