drbd脑裂彻底解决

转载自：http://blog.itpub.net/18455/viewspace-1129709/

发现由于启动顺序错误（通过snapshot恢复功能）。好像造成mysql服务器用的drbd设备的脑裂。

在选定的非主节点上执行
drbdadm secondary mysql
drbdadm -- --discard-my-data connect mysql
主节点上执行
drbdadm connect mysql　

总是不成功　
发现两台机器确实进行过连接，但是不能正确的决定关系。
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: Handshake successful: Agreed network protocol version 101
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: Agreed to support TRIM on protocol level
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: conn( WFConnection -> WFReportParams )
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: Starting asender thread (from drbd_r_mysql [30139])
Mar 26 09:58:42 hacontroller2 kernel: block drbd0: drbd_sync_handshake:
Mar 26 09:58:42 hacontroller2 kernel: block drbd0: self 5C25C478F3ACE6BE:28E2D98C86D01F89:076A48FD642993B9:076948FD642993B9 bits:75 flags:0
Mar 26 09:58:42 hacontroller2 kernel: block drbd0: peer C596BFE91962551F:076948FD642993B8 F5C06A38B650A2D4:F5BF6A38B650A2D5 bits:4985 flags:0
Mar 26 09:58:42 hacontroller2 kernel: block drbd0: uuid_compare()=-1000 by rule 100 Mar 26 09:58:42 hacontroller2 kernel: block drbd0: Unrelated data, aborting!
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: conn( WFReportParams -> Disconnecting )
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: error receiving ReportState, e: -5 l: 0!
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: asender terminated
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: Terminating drbd_a_mysql
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: Connection closed
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: conn( Disconnecting -> StandAlone )
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: receiver terminated
Mar 26 09:58:42 hacontroller2 kernel: drbd mysql: Terminating drbd_r_mysql

察看上面红字。

最后通过以下方法进行解决。
在secondary接点重新初始化数据。
　　
drbdadm detach mysql
dd if=/dev/zero bs=1M count=100 of=/dev/sdb1
drbdadm down mysql
drbdadm create-md mysql

在主节点上执行
drbdadm connect mysql

后发现开始同步。　
[root@hacontroller1 python]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:195604 nr:0 dw:784 dr:206856 al:6 bm:11 lo:0 pe:3 ua:0 ap:0 ep:1 wo:f oos:10287976
        [>....................] sync'ed:  2.0% (10044/10236)M
        finish: 0:17:08 speed: 9,988 (8,820) K/sec
 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:484 nr:8 dw:492 dr:841 al:4 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0