怎么搭建Heartbeat+DRBD+MySQL

2024-04-02 19:55

短信预约 -IT技能 免费直播动态提醒

本篇内容介绍了“怎么搭建Heartbeat+DRBD+MySQL”的有关知识，在实际案例的操作过程中，不少人都会遇到这样的困境，接下来就让小编带领大家学习一下如何处理这些情况吧！希望大家仔细阅读，能够学有所成！

DRBD是一个用软件实现的、无共享的、服务器之间镜像块设备内容的存储复制解决方案。
DRBD的核心功能通过Linux的内核实现，最接近系统的IO栈，但它不能神奇地添加上层的功能比如检测到EXT3文件系统的崩溃。
DRBD的位置处于文件系统以下，比文件系统更加靠近操作系统内核及IO栈。
DRBD相当于给磁盘做了一个网络的raid1。
heartbeat （Linux-HA）的工作原理：heartbeat最核心的包括两个部分，心跳监测部分和资源接管部分，心跳监测可以通过网络链路和串口进行，而且支持冗余链路，它们之间相互发送报文来告诉对方自己当前的状态，如果在指定的时间内未收到对方发送的报文，那么就认为对方失效，这时需启动资源接管模块来接管运行在对方主机上的资源或者服务。
DRBD Protocol参数：

Protocol A @数据一旦写入磁盘并发送到网络中就认为完成了写入操作

Protocol B @收到接收确认就认为完成了写入操作。

Protocol C @收到写入确认就认为完成了写入操作。

################################################

暗紫色文字都是log或命令输出，红色部分为文件内容。

################################################

vm1 10.0.0.11 主
vm2 10.0.0.12 从
VIP: 10.0.0.20

以下操作vm1 vm2都要做：
1 软件安装：
wget http://elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
ls
rpm -ivh elrepo-release-6-6.el6.elrepo.noarch.rpm
yum search drbd
yum install drbd84 kmod-drbd84
yum install heartbeat

2 创建分区
[root@vm1 ~]# fdisk -l

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00076c33

Device Boot    Start       End    Blocks Id  System
/dev/sda1 *          1       26    204800 83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2             26       157    1048576 82  Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3          157       2611 19717120 83  Linux

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9aa21f28

Device Boot    Start       End    Blocks Id  System

[root@vm1 ~]# fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
      switch off the mode (command 'c') and change display units to
      sectors (command 'u').

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130):
Using default value 130

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@vm1 ~]# partprobe
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
[root@vm1 ~]#

3 创建DRBD配置文件
[root@vm1 ~]# cat /etc/drbd.conf

global { usage-count yes; }
common { syncer { rate 10M; } }
resource db {
      protocol C;
      net {
      }
      on vm1 {
            device /dev/drbd1;
            disk    /dev/sdb1;
            address 10.0.0.11:7789;
            meta-disk  internal;
      }
      on vm2 {
            device /dev/drbd1;
            disk    /dev/sdb1;
            address 10.0.0.12:7789;
            meta-disk  internal;
      }
}

以上操作vm1 vm2都要做。

4 创建DBRD资源
[root@vm1 ~]# drbdadm create-md db  ##db是DRBD配置文件中的resource名字
You want me to create a v08 style flexible-size internal meta data block.
There appears to be a v08 flexible-size internal meta data block
already in place on /dev/sdb1 at byte offset 1069248512

Do you really want to overwrite the existing meta-data?
[need to type 'yes' to confirm] yes

md_offset 1069248512
al_offset 1069215744
bm_offset 1069182976

Found ext3 filesystem
   1044124 kB data area apparently used
   1044124 kB left usable by current configuration

Even though it looks like this would place the new meta data into
unused space, you still need to confirm, as this is only a guess.

Do you want to proceed?
[need to type 'yes' to confirm] yes

initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11
[root@vm1 ~]#

[root@vm2 ~]# drbdadm create-md db
You want me to create a v08 style flexible-size internal meta data block.
There appears to be a v08 flexible-size internal meta data block
already in place on /dev/sdb1 at byte offset 1069248512

Do you really want to overwrite the existing meta-data?
[need to type 'yes' to confirm] yes

md_offset 1069248512
al_offset 1069215744
bm_offset 1069182976

Found ext3 filesystem
   1044124 kB data area apparently used
   1044124 kB left usable by current configuration

Even though it looks like this would place the new meta data into
unused space, you still need to confirm, as this is only a guess.

Do you want to proceed?
[need to type 'yes' to confirm] yes

initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
[root@vm2 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11
[root@vm2 ~]#

5 启动DRBD
[root@vm1 ~]# /etc/init.d/drbd start
Starting DRBD resources: [
   create res: db
prepare disk: db
adjust disk: db
   adjust net: db
]
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- If this node was already a degraded cluster before the
reboot, the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot, the timeout
is 0 seconds. [wfc-timeout]
(These values are for resource 'db'; 0 sec -> wait forever)
To abort waiting enter 'yes' [  12]: yes

.
[root@vm1 ~]# ls /dev/drbd*
/dev/drbd1

/dev/drbd:
by-disk  by-res
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s
ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:1044124

[root@vm2 ~]#  /etc/init.d/drbd start
Starting DRBD resources: [
   create res: db
prepare disk: db
adjust disk: db
   adjust net: db
]
.
[root@vm2 ~]# ls /dev/drbd*
/dev/drbd1

/dev/drbd:
by-disk  by-res

[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:105528 nr:0 dw:0 dr:106200 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:938596
      [=>..................] sync'ed: 10.2% (938596/1044124)K
      finish: 0:00:44 speed: 21,104 (21,104) K/sec

6 设置vm1为primary
[root@vm1 ~]# drbdsetup /dev/drbd1 primary --force=yes
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:105528 nr:0 dw:0 dr:106200 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:938596
      [=>..................] sync'ed: 10.2% (938596/1044124)K
      finish: 0:00:44 speed: 21,104 (21,104) K/sec
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1044124 nr:0 dw:0 dr:1044796 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

7 建立文件系统，并挂载。
[root@vm1 ~]# mkfs.ext4 /dev/drbd1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65280 inodes, 261031 blocks
13051 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8160 inodes per group
Superblock backups stored on blocks:
      32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@vm1 ~]#
[root@vm1 ~]# mount /dev/drbd1 /drbd/
[root@vm1 ~]# df -h
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda3       19G  9.7G  7.9G  56% /
tmpfs          246M    0  246M 0% /dev/shm
/dev/sda1    190M 53M  128M  30% /boot
/dev/drbd1    988M  1.3M  936M 1% /drbd

8 挂载测试；
vm1上：
[root@vm1 ~]# cd /drbd/
[root@vm1 drbd]# touch aa
[root@vm1 drbd]# umount /drbd/
umount: /drbd: device is busy.
      (In some cases useful info about processes that use
      the device is found by lsof(8) or fuser(1))
[root@vm1 drbd]# cd
[root@vm1 ~]# cd
[root@vm1 ~]# umount /drbd/
[root@vm1 ~]# drbdadm secondary db
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:1077352 nr:0 dw:33228 dr:1045509 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm1 ~]#

切换到vm2上验证：
[root@vm2 dev]# drbdadm primary db
[root@vm2 dev]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:1077352 dw:1077352 dr:672 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm2 dev]# mkdir /drbd
[root@vm2 dev]# mount /dev/drbd1 /drbd/
[root@vm2 dev]# cd /drbd/
[root@vm2 drbd]# ls
aa  lost+found
[root@vm2 drbd]#

9 切换回vm1:
切换，将vm1设为主：

[root@vm2 ~]# umount /drbd/
[root@vm2 ~]# drbdadm secondary db
[root@vm2 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:36 nr:1077352 dw:1077388 dr:1025 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@vm1 ~]# drbdadm primary db
[root@vm1 ~]# mount /dev/drbd1 /drbd/
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:1077356 nr:36 dw:33268 dr:1046530 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

10 配置heartbeat.
[root@vm1 ~]# cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d/
[root@vm1 ~]# cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d/
[root@vm1 ~]# cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d/
[root@vm1 ~]# chmod 600 /etc/ha.d/authkeys

VIP 10.0.0.20

设置通信秘钥
[root@vm1 ~]# cat /etc/ha.d/authkeys  | egrep -v "^#"
auth 3
3 md5 Hello!

创建heartbeat配置文件
[root@vm1 ~]# grep -v "^#" /etc/ha.d/ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility    local0
keepalive 2
deadtime 5
warntime 5
initdead 120
ucast eth0 10.0.0.12
auto_failback off
node vm1
node vm2
ping  10.0.0.1

创建mysql管理脚本，仅为简单说明问题
[root@vm1 ~]# cat /etc/ha.d/resource.d/manage_mysql
#!/bin/bash

case  $1 in

'start')

/usr/local/mysql/bin/mysqld_safe --defaults-file=/drbd/mysql3307/my.cnf &
;;

'stop')
if [[ `ps -ef | grep mysqld | wc -l` <  2 ]] ;  then
  exit
else
  mysqladmin -S /tmp/mysql3307.sock  shutdown
fi
;;

*)
;;
esac

[root@vm1 ~]# chmod +x /etc/ha.d/resource.d/manage_mysql
定义heartbeat管理资源
[root@vm1 ~]# grep -v "^#" /etc/ha.d/haresources
vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql

[root@vm1 ~]# scp -r /etc/ha.d/ vm2:/etc

修改vm2的/etc/ha.d/ha.cf内容如下
[root@vm2 ha.d]# grep -v "^#" /etc/ha.d/ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility    local0
keepalive 2
deadtime 5
warntime 5
initdead 120
ucast eth0 10.0.0.11
auto_failback off
node vm1
node vm2
ping  10.0.0.1

11 启动验证
先启动DRBD
[root@vm1 ~]# /etc/init.d/drbd start
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:1452 nr:1148 dw:2600 dr:26342 al:9 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@vm2 resource.d]# /etc/init.d/drbd start
[root@vm2 resource.d]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:1412 nr:6112 dw:7524 dr:24838 al:9 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

启动heartbeat：
[root@vm1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:  Resource is stopped
Done.
[root@vm2 ~]#  /etc/init.d/heartbeat start
Starting High-Availability services: INFO:  Resource is stopped
Done.

检查log
[root@vm1 ~]# tail -20f /var/log/ha-log
Apr 11 17:19:58 vm1 heartbeat: [7734]: info: Pacemaker support: false
Apr 11 17:19:58 vm1 heartbeat: [7734]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Apr 11 17:19:58 vm1 heartbeat: [7734]: info: **************************
Apr 11 17:19:58 vm1 heartbeat: [7734]: info: Configuration validated. Starting heartbeat 3.0.4
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: heartbeat: version 3.0.4
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Heartbeat generation: 1460298765
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: bound send socket to device: eth0
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: bound receive socket to device: eth0
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.12
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: glib: ping heartbeat started.
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Local status now set to: 'up'
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Link 10.0.0.1:10.0.0.1 up.
Apr 11 17:19:58 vm1 heartbeat: [7735]: info: Status update for node 10.0.0.1: status ping
Apr 11 17:20:36 vm1 heartbeat: [7735]: info: Link vm2:eth0 up.
Apr 11 17:20:36 vm1 heartbeat: [7735]: info: Status update for node vm2: status up
harc(default)[7747]: 2016/04/11_17:20:36 info: Running /etc/ha.d//rc.d/status status
Apr 11 17:20:37 vm1 heartbeat: [7735]: info: Comm_now_up(): updating status to active
Apr 11 17:20:37 vm1 heartbeat: [7735]: info: Local status now set to: 'active'
Apr 11 17:20:37 vm1 heartbeat: [7735]: info: Status update for node vm2: status active
harc(default)[7768]: 2016/04/11_17:20:37 info: Running /etc/ha.d//rc.d/status status
Apr 11 17:20:48 vm1 heartbeat: [7735]: info: remote resource transition completed.
Apr 11 17:20:48 vm1 heartbeat: [7735]: info: remote resource transition completed.
Apr 11 17:20:48 vm1 heartbeat: [7735]: info: Initial resource acquisition complete (T_RESOURCES(us))
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[7821]:    2016/04/11_17:20:48 INFO:  Resource is stopped
Apr 11 17:20:48 vm1 heartbeat: [7785]: info: Local Resource acquisition completed.
harc(default)[7887]: 2016/04/11_17:20:49 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[7887]: 2016/04/11_17:20:49 received ip-request-resp IPaddr::10.0.0.20 OK yes
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[7938]:    2016/04/11_17:20:49 INFO:  Resource is stopped
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 start
IPaddr(IPaddr_10.0.0.20)[8034]: 2016/04/11_17:20:49 INFO: Adding inet address 10.0.0.20/24 with broadcast address 10.0.0.255 to device eth0
IPaddr(IPaddr_10.0.0.20)[8034]: 2016/04/11_17:20:49 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.0.0.20)[8034]: 2016/04/11_17:20:49 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.0.20 eth0 10.0.0.20 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[8020]:    2016/04/11_17:20:49 INFO:  Success
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/drbddisk db start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[8162]:    2016/04/11_17:20:49 INFO:  Resource is stopped
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 start
Filesystem(Filesystem_/dev/drbd1)[8249]:       2016/04/11_17:20:49 INFO: Running start for /dev/drbd1 on /drbd
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[8241]:    2016/04/11_17:20:49 INFO:  Success
ResourceManager(default)[7910]: 2016/04/11_17:20:49 info: Running /etc/ha.d/resource.d/manage_mysql  start

[root@vm2 ~]# tail -20f /var/log/ha-log
Apr 11 17:20:36 vm2 heartbeat: [16250]: info: Pacemaker support: false
Apr 11 17:20:36 vm2 heartbeat: [16250]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Apr 11 17:20:36 vm2 heartbeat: [16250]: info: **************************
Apr 11 17:20:36 vm2 heartbeat: [16250]: info: Configuration validated. Starting heartbeat 3.0.4
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: heartbeat: version 3.0.4
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Heartbeat generation: 1460351868
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: bound send socket to device: eth0
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: bound receive socket to device: eth0
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: set SO_REUSEPORT(w)
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ucast: started on port 694 interface eth0 to 10.0.0.11
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: glib: ping heartbeat started.
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: G_main_add_TriggerHandler: Added signal manual handler
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Local status now set to: 'up'
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Link 10.0.0.1:10.0.0.1 up.
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Status update for node 10.0.0.1: status ping
Apr 11 17:20:36 vm2 heartbeat: [16251]: info: Link vm1:eth0 up.
Apr 11 17:20:37 vm2 heartbeat: [16251]: info: Status update for node vm1: status active
Apr 11 17:20:37 vm2 heartbeat: [16251]: info: Comm_now_up(): updating status to active
Apr 11 17:20:37 vm2 heartbeat: [16251]: info: Local status now set to: 'active'
harc(default)[16261]: 2016/04/11_17:20:37 info: Running /etc/ha.d//rc.d/status status
Apr 11 17:20:48 vm2 heartbeat: [16251]: info: local resource transition completed.
Apr 11 17:20:48 vm2 heartbeat: [16251]: info: Initial resource acquisition complete (T_RESOURCES(us))
Apr 11 17:20:48 vm2 heartbeat: [16280]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys vm2] to acquire.
Apr 11 17:20:48 vm2 heartbeat: [16251]: info: remote resource transition completed.

检查drbd、磁盘挂载、mysql进程信息：
vm1上：
[root@vm1 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:2004 nr:1148 dw:3152 dr:38227 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm1 ~]# df -h
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda3       19G 11G  7.5G  58% /
tmpfs          246M    0  246M 0% /dev/shm
/dev/sda1    190M 53M  128M  30% /boot
/dev/drbd1    988M  178M  760M  19% /drbd

[root@vm1 ~]#  ps -ef | grep mysql
root    8337    1  0 17:20 ?       00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/drbd/mysql3307/my.cnf
mysql    8599  8337  0 17:20 ?       00:00:00 /usr/local/mysql/bin/mysqld --defaults-file=/drbd/mysql3307/my.cnf --basedir=/usr/local/mysql --datadir=/drbd/mysql3307 --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/drbd/mysql3307/vm1.err --pid-file=/drbd/mysql3307/vm1.pid --socket=/tmp/mysql3307.sock --port=3307
root    8633  1874  0 17:24 pts/3 00:00:00 grep mysql

[root@vm1 ~]# mysql -uroot -pmysql -h20.0.0.20 -P3307
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.27-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> Ctrl-C -- exit!
Aborted

vm2上：
[root@vm2 resource.d]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:1412 nr:6664 dw:8076 dr:24838 al:9 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[root@vm2  ~]#  df -h
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda3       19G 11G  7.5G  58% /
tmpfs          246M    0  246M 0% /dev/shm
/dev/sda1    190M 53M  128M  30% /boot
[root@vm2 ~]#  ps -ef | grep mysql
root    16304 11810  0 17:24 pts/2 00:00:00 grep mysql
[root@vm2 resource.d]#

[root@vm2 resource.d]# mysql -uroot -pmysql -h20.0.0.20 -P3307
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.27-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> Ctrl-C -- exit!
Aborted
[root@vm2 resource.d]#

关掉vm1,模拟故障：
[root@vm1 ~]# init 0

[root@vm2 ~]# tail -20f /var/log/ha-log
Apr 11 17:27:00 vm2 heartbeat: [16251]: info: Received shutdown notice from 'vm1'.
Apr 11 17:27:00 vm2 heartbeat: [16251]: info: Resources being acquired from vm1.
Apr 11 17:27:00 vm2 heartbeat: [16310]: info: acquire local HA resources (standby).
Apr 11 17:27:00 vm2 heartbeat: [16310]: info: local HA resource acquisition completed (standby).
Apr 11 17:27:00 vm2 heartbeat: [16251]: info: Standby resource acquisition done [all].
Apr 11 17:27:00 vm2 heartbeat: [16311]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys vm2] to acquire.
harc(default)[16336]: 2016/04/11_17:27:00 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[16353]:    2016/04/11_17:27:00 info: Taking over resource group IPaddr::10.0.0.20
ResourceManager(default)[16380]:       2016/04/11_17:27:00 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[16408]:    2016/04/11_17:27:00 INFO:  Resource is stopped
ResourceManager(default)[16380]:       2016/04/11_17:27:00 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 start
IPaddr(IPaddr_10.0.0.20)[16504]:       2016/04/11_17:27:00 INFO: Adding inet address 10.0.0.20/24 with broadcast address 10.0.0.255 to device eth0
IPaddr(IPaddr_10.0.0.20)[16504]:       2016/04/11_17:27:00 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.0.0.20)[16504]:       2016/04/11_17:27:00 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.0.20 eth0 10.0.0.20 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[16490]:    2016/04/11_17:27:00 INFO:  Success
ResourceManager(default)[16380]:       2016/04/11_17:27:00 info: Running /etc/ha.d/resource.d/drbddisk db start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[16632]: 2016/04/11_17:27:01 INFO:  Resource is stopped
ResourceManager(default)[16380]:       2016/04/11_17:27:01 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 start
Filesystem(Filesystem_/dev/drbd1)[16719]:    2016/04/11_17:27:01 INFO: Running start for /dev/drbd1 on /drbd
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[16711]: 2016/04/11_17:27:01 INFO:  Success
ResourceManager(default)[16380]:       2016/04/11_17:27:01 info: Running /etc/ha.d/resource.d/manage_mysql  start
mach_down(default)[16353]:    2016/04/11_17:27:01 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[16353]:    2016/04/11_17:27:01 info: mach_down takeover complete for node vm1.
Apr 11 17:27:01 vm2 heartbeat: [16251]: info: mach_down takeover complete.
Apr 11 17:27:07 vm2 heartbeat: [16251]: WARN: node vm1: is dead
Apr 11 17:27:07 vm2 heartbeat: [16251]: info: Dead node vm1 gave up resources.
Apr 11 17:27:07 vm2 heartbeat: [16251]: info: Link vm1:eth0 dead.

[root@vm2 ~]# ps -ef | grep mysql
root    16808    1  0 17:27 ?       00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/drbd/mysql3307/my.cnf
mysql 17085 16808  0 17:27 ?       00:00:00 /usr/local/mysql/bin/mysqld --defaults-file=/drbd/mysql3307/my.cnf --basedir=/usr/local/mysql --datadir=/drbd/mysql3307 --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/drbd/mysql3307/vm2.err --pid-file=/drbd/mysql3307/vm2.pid --socket=/tmp/mysql3307.sock --port=3307
root    17112 11810  0 17:28 pts/2 00:00:00 grep mysql

[root@vm2 ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by mockbuild@Build64R6, 2016-01-12 13:27:11

1: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
ns:1820 nr:6856 dw:8760 dr:36747 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:84
[root@vm2 ~]# df -h
Filesystem    Size  Used Avail Use% Mounted on
/dev/sda3       19G 11G  7.5G  58% /
tmpfs          246M    0  246M 0% /dev/shm
/dev/sda1    190M 53M  128M  30% /boot
/dev/drbd1    988M  178M  760M  19% /drbd

[root@vm2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:9b:2c:f6 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.12/24 brd 10.0.0.255 scope global eth0
inet 10.0.0.20/24 brd 10.0.0.255 scope global secondary eth0
inet6 fe80::a00:27ff:fe9b:2cf6/64 scope link
   valid_lft forever preferred_lft forever

[root@vm2 ~]#  mysql -uroot -pmysql -h20.0.0.20 -P3307
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.27-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

VIP、磁盘资源、mysql进程都已在vm2上。

完！

########################################

后记

资源文件配置解释：

[root@vm1 ~]# grep -v "^#" /etc/ha.d/haresources
vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql

vm1 主服务器的hostname,后面都是由heartbeat负责管理的资源。

IPaddr::10.0.0.20 VIP

drbddisk::db DRBD资源，db为drbd.conf中resource的名字

Filesystem::/dev/drbd1::/drbd::ext4 需要挂载的文件系统、挂载点、文件类型

manage_mysql 系统或自己定义的资源管理文件，需要能配合start\stop参数。一般位于/etc/init.d或/etc/ha.d/resource.d目录下，

这个manage_mysql是我自己写的。

执行/etc/init.d/heartbeat start命令后，主节点的/var/log/ha-log内容如下：

ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2052]:    2016/04/11_18:05:35 INFO:  Resource is stopped
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2061]:    2016/04/11_18:05:35 INFO:  Resource is stopped
Apr 11 18:05:35 vm1 heartbeat: [1980]: info: Local Resource acquisition completed.
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 start
IPaddr(IPaddr_10.0.0.20)[2218]: 2016/04/11_18:05:35 INFO: Adding inet address 10.0.0.20/24 with broadcast address 10.0.0.255 to device eth0
IPaddr(IPaddr_10.0.0.20)[2218]: 2016/04/11_18:05:35 INFO: Bringing device eth0 up
IPaddr(IPaddr_10.0.0.20)[2218]: 2016/04/11_18:05:35 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.0.20 eth0 10.0.0.20 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2204]:    2016/04/11_18:05:35 INFO:  Success
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/drbddisk db start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[2348]:    2016/04/11_18:05:35 INFO:  Resource is stopped
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 start
Filesystem(Filesystem_/dev/drbd1)[2434]:       2016/04/11_18:05:35 INFO: Running start for /dev/drbd1 on /drbd
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[2426]:    2016/04/11_18:05:35 INFO:  Success
ResourceManager(default)[2006]: 2016/04/11_18:05:35 info: Running /etc/ha.d/resource.d/manage_mysql  start
Apr 11 18:05:35 vm1 heartbeat: [1979]: info: all HA resource acquisition completed (standby).
Apr 11 18:05:35 vm1 heartbeat: [1590]: info: Standby resource acquisition done [all].
harc(default)[2523]: 2016/04/11_18:05:35 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[2560]:    2016/04/11_18:05:35 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[2560]:    2016/04/11_18:05:36 info: mach_down takeover complete for node vm2.
Apr 11 18:05:36 vm1 heartbeat: [1590]: info: mach_down takeover complete.
harc(default)[2626]: 2016/04/11_18:05:36 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[2626]: 2016/04/11_18:05:36 received ip-request-resp IPaddr::10.0.0.20 OK yes
ResourceManager(default)[2680]: 2016/04/11_18:05:36 info: Acquiring resource group: vm1 IPaddr::10.0.0.20 drbddisk::db Filesystem::/dev/drbd1::/drbd::ext4 manage_mysql
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[2741]:    2016/04/11_18:05:36 INFO:  Running OK
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[2929]:    2016/04/11_18:05:36 INFO:  Running OK
ResourceManager(default)[2680]: 2016/04/11_18:05:36 info: Running /etc/ha.d/resource.d/manage_mysql  start
Apr 11 18:05:41 vm1 heartbeat: [1590]: WARN: node vm2: is dead
Apr 11 18:05:41 vm1 heartbeat: [1590]: info: Dead node vm2 gave up resources.
Apr 11 18:05:41 vm1 heartbeat: [1590]: info: Link vm2:eth0 dead.

#######################

在主节点执行/etc/init.d/heartbeat stop命令后，/var/log/ha-log内容如下：

ResourceManager(default)[17275]:       2016/04/11_18:05:31 info: Running /etc/ha.d/resource.d/manage_mysql  stop
ResourceManager(default)[17275]:       2016/04/11_18:05:33 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /drbd ext4 stop
Filesystem(Filesystem_/dev/drbd1)[17343]:    2016/04/11_18:05:33 INFO: Running stop for /dev/drbd1 on /drbd
Filesystem(Filesystem_/dev/drbd1)[17343]:    2016/04/11_18:05:33 INFO: Trying to unmount /drbd
Filesystem(Filesystem_/dev/drbd1)[17343]:    2016/04/11_18:05:33 INFO: unmounted /drbd successfully
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[17335]: 2016/04/11_18:05:33 INFO:  Success
ResourceManager(default)[17275]:       2016/04/11_18:05:33 info: Running /etc/ha.d/resource.d/drbddisk db stop
ResourceManager(default)[17275]:       2016/04/11_18:05:34 info: Running /etc/ha.d/resource.d/IPaddr 10.0.0.20 stop
IPaddr(IPaddr_10.0.0.20)[17492]:       2016/04/11_18:05:34 INFO: IP status = ok, IP_CIP=
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_10.0.0.20)[17473]:    2016/04/11_18:05:34 INFO:  Success
Apr 11 18:05:34 vm2 heartbeat: [17262]: info: All HA resources relinquished.
Apr 11 18:05:35 vm2 heartbeat: [16251]: WARN: 1 lost packet(s) for [vm1] [64:66]
Apr 11 18:05:35 vm2 heartbeat: [16251]: info: No pkts missing from vm1!
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBFIFO process 16254 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBWRITE process 16255 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBREAD process 16256 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBWRITE process 16257 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: killing HBREAD process 16258 with signal 15
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16254 exited. 5 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16256 exited. 4 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16255 exited. 3 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16258 exited. 2 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: Core process 16257 exited. 1 remaining
Apr 11 18:05:36 vm2 heartbeat: [16251]: info: vm2 Heartbeat shutdown complete.