drbd heartbeat déconnexion du noeud primaire

par **Lassaad MATHLOUTHI** » 10 Oct 2008 10:57

Bonjour,

je débarque sur un problème il ya 4 jours, en fait je suis entrain de mettre en place un cluster
avec 2 noeuds un primaire et un secondaire.

voici l'architecture et les conf :

2 IBM servers with RAID 1
drbd version: 8.2.6 (API: 88/proto :86-88) and heartbeat installed on 2 servers
############################################
############################################
drbd.conf:

#
# drbd.conf
#
resource r1 {
protocol B;
#incon-degr-cmd "halt -f";

#incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
#handlers { pri-on-incon-degr "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f"; }

startup {
#degr-wfc-timeout 120; # 2 minutes.
}

disk {
#on-io-error detach;
}

net {
#sndbuf-size 512k;
#timeout 60; # 6 seconds (unit = 0.1 seconds)
#connect-int 10; # 10 seconds (unit = 1 second)
#ping-int 10; # 10 seconds (unit = 1 second)
#ping-timeout 50; # 500 ms (unit = 0.1 seconds)
#max-buffers 8000;
#max-epoch-size 8000;
}

syncer {
rate 2048;
#group 1;
#al-extents 257;
}

on serv11 {
device /dev/drbd0;
disk /dev/sda4;
address 192.168.1.246:7788;
meta-disk internal;
}

on serv12 {
device /dev/drbd0;
disk /dev/sda4;
address 192.168.1.247:7788;
meta-disk internal;
}
}
############################################
/etc/ha.d/ha.cf:

#bcast eth0 car le reseau contient 2 cluster donc on va utiliser le
unicast

ucast eth0 192.168.1.247
#baud 19200
#serial /dev/ttyS0
#bcast eth1

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0

keepalive 2
deadtime 10
warntime 6
initdead 60

udpport 694
node serv11
node serv12

auto_failback off
##############################################
/etc/ha.d/haressources
serv11 drbddisk::r1 Filesystem::/dev/drbd0::/data::ext3 IPaddr::192.168.1.250 MailTo::toto@toto.com::Cluster1-StatusUpdated fetchmail
#################################################

donc, lorsque je redémarre heartbeat sur le noeud primaire le noeud secondaire prend la main
et monte la partition /data et devient primaire -----> comportement correct

mais le sousci, losque je débranche le noeud primaire-----> blocage du cluster et voila le log sur
le noeud secondaire:

#####################################################

Oct 10 12:16:31 serv12 kernel: drbd0: PingAck did not arrive in time.
Oct 10 12:16:31 serv12 kernel: drbd0: peer (Primary -> Unknown) conn (
SyncTarget -> NetworkFailure) pdsk (UpToDate -> DUnknown)
Oct 10 12:16:31 serv12 kernel: drbd0: asender terminated
Oct 10 12:16:31 serv12 kernel: drbd0: Terminating thread asender
Oct 10 12:16:31 serv12 kernel: drbd0: short read expecting header on sock: r =- 512
Oct 10 12:16:31 serv12 kernel: drbd0: Writing meta data super block now.
Oct 10 12:16:31 serv12 kernel: drbd0: tl_clear ()
Oct 10 12:16:31 serv12 kernel: drbd0: Connection closed
Oct 10 12:16:31 serv12 kernel: drbd0: conn (NetworkFailure -> Unconnected)
Oct 10 12:16:31 serv12 kernel: drbd0: receiver terminated
Oct 10 12:16:31 aserv12 kernel: drbd0: receiver (re) started
Oct 10 12:16:31 serv12 kernel: drbd0: conn (Unconnected -> WFConnection
)
Oct 10 12:16:31 serv12 heartbeat [2810]: WARN: node serv11: is deadOct 10 12:16:31 serv12 heartbeat [2810]: WARN: No stonith device configured.
Oct 10 12:16:31 serv12 heartbeat [2810]: WARN: Shared disks are not protected.
Oct 10 12:16:31 aserv12 heartbeat [2810]: info: Resources being acquired
from serv11.
Oct 10 12:16:31 serv12 heartbeat [2810]: info: Link serv11: eth2 dead.
Oct 10 12:16:31 serv12 heartbeat [3039]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Oct 10 12:16:31 serv12 heartbeat: info: Running / etc / ha.d / rc.d / status status
Oct 10 12:16:31 serv12 heartbeat [3040]: info: No local resources [/ usr / lib / heartbeat / ResourceManagement listkeys serv12] to acquire.
Oct 10 12:16:31 serv12 heartbeat [2810]: debug: StartNextRemoteRscReq ():
1 child count
Oct 10 12:16:31 serv12 heartbeat: info: Taking over resource group drbddisk: r1
Oct 10 12:16:31 serv12 heartbeat: info: Acquiring resource group: serv11 drbddisk: Filesystem r1:: / dev/drbd0:: / data:: ext3 IPAddr: 192.168.1.250 MailTo: otot@ ttt: Cluster1-StatusUpdated fetchmail
Oct 10 12:16:31 serv12 heartbeat: info: Running / etc / ha.d / resource.d / drbddisk r1 start
Oct 10 12:16:31 serv12 kernel: drbd0: State change failed: Refusing to
Primary be without at least one disk UpToDate
Oct 10 12:16:31 serv12 kernel: drbd0: (state = cs: WFConnection st: Secondary / Unknown ds: Inconsistent / DUnknown --- r)
Oct 10 12:16:31 aserv12 kernel: drbd0: = (wanted cs: WFConnection st: Primary / Unknown ds: Inconsistent / DUnknown --- r)
Oct 10 12:16:32 serv12 kernel: drbd0: State change failed: Refusing to
Primary be without at least one disk UpToDate
Oct 10 12:16:32 arserv12 kernel: drbd0: (state = cs: WFConnection st: Secondary / Unknown ds: Inconsistent / DUnknown --- r)

Oct 10 12:16:50 serv12 heartbeat: debug: / etc / ha.d / resource.d / drbddisk
r1 start done. RC = 1
Oct 10 12:16:50 serv12 heartbeat: ERROR: Return code 1 from / etc / ha.d / resource.d / drbddisk
Oct 10 12:16:50 aserv12 heartbeat: CRIT: Giving up resources due to failure of drbddisk: r1
Oct 10 12:16:50 aserv12 heartbeat: info: Releasing resource group: serv11 drbddisk: Filesystem r1:: / dev/drbd0:: / data:: ext3 IPAddr: 192.168.1.250 MailTo: @ : Cluster1-StatusUpdated fetchmail
Oct 10 12:16:50 serv12 heartbeat: info: Running / etc / init.d / fetchmail
stop
Oct 10 12:16:50 serv12 heartbeat: debug: Starting / etc / init.d / fetchmail
stop
Oct 10 12:16:50 aserv12 heartbeat: debug: / etc / init.d / fetchmail stop done. RC = 0
Oct 10 12:16:50 serv12 heartbeat: info: Running / etc / ha.d / resource.d / MailTo Cluster1-stop StatusUpdated
Oct 10 12:16:50 serv12 heartbeat: debug: Starting / etc / ha.d / resource.d / MailTo Cluster1-stop StatusUpdated
Oct 10 12:16:50 serv12 heartbeat: debug: / etc / ha.d / resource.d / MailTo l Cluster1-stop StatusUpdated done. RC = 0
Oct 10 12:16:50 serv12 heartbeat: info: Running / etc / ha.d / resource.d / 192.168.1.250 stop IPAddr
Oct 10 12:16:50 serv12 heartbeat: debug: Starting / etc / ha.d / resource.d / 192.168.1.250 stop IPAddr
Oct 10 12:16:50 serv12 heartbeat: debug: / etc / ha.d / resource.d / IPAddr 192.168.1.250 stop done. RC = 0
Oct 10 12:16:50 serv12 heartbeat: info: Running / etc / ha.d / resource.d / Filesystem / dev/drbd0 / data ext3 stop
Oct 10 12:16:50 serv12 heartbeat: debug: Starting / etc / ha.d / resource.d / Filesystem / dev/drbd0 / data ext3 stop
Oct 10 12:16:50 serv12 heartbeat: WARNING: Filesystem / data not mounted?
Oct 10 12:16:50 serv12 heartbeat: debug: / etc / ha.d / resource.d / Filesystem / dev/drbd0 / data ext3 stop done. RC = 0
Oct 10 12:16:50 serv12 heartbeat: info: Running / etc / ha.d / resource.d / drbddisk stop r1
Oct 10 12:16:50 serv12 heartbeat: debug: Starting / etc / ha.d / resource.d / drbddisk stop r1
Oct 10 12:16:50 serv12 heartbeat: debug: / etc / ha.d / resource.d / drbddisk
r1 stop done. RC = 0
Oct 10 12:16:50 serv12 heartbeat: info: / usr / lib / heartbeat / mach_down: nice_failback: foreign resources acquired
Oct 10 12:16:50 serv12 heartbeat [2810]: info: mach_down complete takeover.
Oct 10 12:16:50 serv12 heartbeat: info: mach_down takeover complete for
serv11 node.
##############################################

Note: lorsque je remet le cable réseau du noeud primaire le cluster se débloque et monte sur le primaire.

Merci beaucoup en avance.
Lassaad.

par **Lassaad MATHLOUTHI** » 10 Oct 2008 13:53

pas de réponse

par **arapaho** » 10 Oct 2008 16:34

La configuration de la ressource drdb 'r1' ne prend pas en compte la situation dans laquelle tu mets ton cluster, c'est à dire une node secondaire qui doit prendre la main sur une ressource qui n'est a priori plus à jour (puisque la node primaire n'est plus joignable) et donc dans un état inconsistant.
Vu que drbd ne sait pas quoi faire dans ce cas, il ne prend pas la ressource en tant que primaire.

C'est à configurer dans drbd.conf.

Encore un point: si ce cluster ha ne comporte comme seul lien de keepalive qu'un lien ethernet, ce n'est absolument pas suffisant.

Dernier conseil: édite ton message et enlève ton mail qui apparait dans la conf de ressources HeartBeat. Surtout lorsque cet email amène directement au site de ta boîte qui parle de haute disponibilité. Si ton patron tombe dessus, il t'arriverait quelques bricoles

par **Muzo** » 10 Oct 2008 17:48

arapaho a écrit:Dernier conseil: édite ton message et enlève ton mail qui apparait dans la conf de ressources HeartBeat. Surtout lorsque cet email amène directement au site de ta boîte qui parle de haute disponibilité. Si ton patron tombe dessus, il t'arriverait quelques bricoles

C'est fait

Mais ca ne servira à rien, il a mis sont vrai nom en pseudo.

/Muzo

par **Lassaad MATHLOUTHI** » 10 Oct 2008 18:39

bonjour à tous,

voila vous avez remarqué que j'ai pris la conseille en consédiration mais pourtant je suis toujours sur le mm probleme, et voila le log maintenant sur le noeud secondaire aprés le débranchement du primaire :

#########################
Oct 10 20:04:32 serv12 heartbeat[4969]: WARN: node serv11: is dead
Oct 10 20:04:32 serv12 heartbeat[4969]: WARN: No STONITH device configured.
Oct 10 20:04:32 serv12 heartbeat[4969]: WARN: Shared disks are not protected.
Oct 10 20:04:32 serv12 heartbeat[4969]: info: Resources being acquired from serv11.
Oct 10 20:04:32 serv12 heartbeat[4969]: info: Link serv11:eth2 dead.
Oct 10 20:04:32 serv12 heartbeat[4995]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Oct 10 20:04:32 serv12 heartbeat: info: Running /etc/ha.d/rc.d/status status
Oct 10 20:04:32 serv12 heartbeat[4996]: info: No local resources [/usr/lib/heartbeat/ResourceManager listkeys serv12] to acquire.
Oct 10 20:04:32 serv12 heartbeat[4969]: debug: StartNextRemoteRscReq(): child count 1
Oct 10 20:04:32 serv12 heartbeat: info: Taking over resource group IPaddr::192.168.1.250
Oct 10 20:04:32 serv12 heartbeat: info: Acquiring resource group: serv11 IPaddr::192.168.1.250 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3
Oct 10 20:04:33 serv12 heartbeat: info: Running /etc/ha.d/resource.d/IPaddr 192.168.1.250 start
Oct 10 20:04:33 serv12 heartbeat: debug: Starting /etc/ha.d/resource.d/IPaddr 192.168.1.250 start
Oct 10 20:04:33 serv12 heartbeat: info: /sbin/ifconfig eth2:0 192.168.1.250 netmask 255.255.255.0^Ibroadcast 192.168.1.255
Oct 10 20:04:33 serv12 heartbeat: info: Sending Gratuitous Arp for 192.168.1.250 on eth2:0 [eth2]
Oct 10 20:04:33 serv12 heartbeat: /usr/lib/heartbeat/send_arp -i 1010 -r 5 -p /var/lib/heartbeat/rsctmp/send_arp/send_arp-192.168.1.250 eth2 192.168.1.250 auto 192.168.1.250 ffffffffffff
Oct 10 20:04:33 serv12 heartbeat: debug: /etc/ha.d/resource.d/IPaddr 192.168.1.250 start done. RC=0
Oct 10 20:04:33 serv12 heartbeat: info: Running /etc/ha.d/resource.d/drbddisk r0 start
Oct 10 20:04:33 serv12 heartbeat: debug: Starting /etc/ha.d/resource.d/drbddisk r0 start
Oct 10 20:04:33 serv12 kernel: drbd0: PingAck did not arrive in time.
Oct 10 20:04:33 serv12 kernel: drbd0: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Oct 10 20:04:33 serv12 kernel: drbd0: asender terminated
Oct 10 20:04:33 serv12 kernel: drbd0: Terminating asender thread
Oct 10 20:04:33 serv12 kernel: drbd0: Writing meta data super block now.
Oct 10 20:04:33 serv12 kernel: drbd0: short read receiving data: read 1560 expected 4096
Oct 10 20:04:33 serv12 kernel: drbd0: error receiving RSDataReply, l: 32792!
Oct 10 20:04:33 serv12 kernel: drbd0: tl_clear()
Oct 10 20:04:33 serv12 kernel: drbd0: Connection closed
Oct 10 20:04:33 serv12 kernel: drbd0: conn( NetworkFailure -> Unconnected )
Oct 10 20:04:33 serv12 kernel: drbd0: receiver terminated
Oct 10 20:04:33 serv12 kernel: drbd0: receiver (re)started
Oct 10 20:04:33 serv12 kernel: drbd0: State change failed: Refusing to be Primary without at least one UpToDate disk
Oct 10 20:04:33 serv12 kernel: drbd0: state = { cs:Unconnected st:Secondary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:33 serv12 kernel: drbd0: wanted = { cs:Unconnected st:Primary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:33 serv12 kernel: drbd0: conn( Unconnected -> WFConnection )
Oct 10 20:04:34 serv12 kernel: drbd0: State change failed: Refusing to be Primary without at least one UpToDate disk
Oct 10 20:04:34 serv12 kernel: drbd0: state = { cs:WFConnection st:Secondary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:34 serv12 kernel: drbd0: wanted = { cs:WFConnection st:Primary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:35 serv12 kernel: drbd0: State change failed: Refusing to be Primary without at least one UpToDate disk
Oct 10 20:04:35 serv12 kernel: drbd0: state = { cs:WFConnection st:Secondary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:35 serv12 kernel: drbd0: wanted = { cs:WFConnection st:Primary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:36 serv12 kernel: drbd0: State change failed: Refusing to be Primary without at least one UpToDate disk
Oct 10 20:04:36 serv12 kernel: drbd0: state = { cs:WFConnection st:Secondary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:36 serv12 kernel: drbd0: wanted = { cs:WFConnection st:Primary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:37 serv12 kernel: drbd0: State change failed: Refusing to be Primary without at least one UpToDate disk
Oct 10 20:04:37 serv12 kernel: drbd0: state = { cs:WFConnection st:Secondary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:37 serv12 kernel: drbd0: wanted = { cs:WFConnection st:Primary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:38 serv12 kernel: drbd0: State change failed: Refusing to be Primary without at least one UpToDate disk
Oct 10 20:04:38 serv12 kernel: drbd0: state = { cs:WFConnection st:Secondary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:38 serv12 kernel: drbd0: wanted = { cs:WFConnection st:Primary/Unknown ds:Inconsistent/DUnknown r--- }
Oct 10 20:04:38 serv12 heartbeat: debug: /etc/ha.d/resource.d/drbddisk r0 start done. RC=1
Oct 10 20:04:38 serv12 heartbeat: ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk
Oct 10 20:04:38 serv12 heartbeat: CRIT: Giving up resources due to failure of drbddisk::r0
Oct 10 20:04:38 serv12 heartbeat: info: Releasing resource group: serv11 IPaddr::192.168.1.250 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3
Oct 10 20:04:38 serv12 heartbeat: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /data ext3 stop
Oct 10 20:04:38 serv12 heartbeat: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /data ext3 stop
Oct 10 20:04:38 serv12 heartbeat: WARNING: Filesystem /data not mounted?
Oct 10 20:04:38 serv12 heartbeat: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /data ext3 stop done. RC=0
Oct 10 20:04:38 serv12 heartbeat: info: Running /etc/ha.d/resource.d/drbddisk r0 stop
Oct 10 20:04:38 serv12 heartbeat: debug: Starting /etc/ha.d/resource.d/drbddisk r0 stop
Oct 10 20:04:38 serv12 heartbeat: debug: /etc/ha.d/resource.d/drbddisk r0 stop done. RC=0
Oct 10 20:04:38 serv12 heartbeat: info: Running /etc/ha.d/resource.d/IPaddr 192.168.1.250 stop
Oct 10 20:04:38 serv12 heartbeat: debug: Starting /etc/ha.d/resource.d/IPaddr 192.168.1.250 stop
Oct 10 20:04:38 serv12 heartbeat: info: /sbin/route -n del -host 192.168.1.250
Oct 10 20:04:38 serv12 heartbeat: info: /sbin/ifconfig eth2:0 down
Oct 10 20:04:38 serv12 heartbeat: info: IP Address 192.168.1.250 released
Oct 10 20:04:38 serv12 heartbeat: debug: /etc/ha.d/resource.d/IPaddr 192.168.1.250 stop done. RC=0
Oct 10 20:04:38 serv12 heartbeat: info: /usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired
Oct 10 20:04:38 serv12 heartbeat[4969]: info: mach_down takeover complete.
Oct 10 20:04:38 serv12 heartbeat: info: mach_down takeover complete for node serv11.
#######################

j'attend vos conseils.

Merci.

Lassaad.

par **arapaho** » 10 Oct 2008 19:08

C'est toujours le même cas:

1 - le secondaire perd de vue la ressource 'r0'
2 - Le dispositif drbd0 entre dans un état inconsistant: Inconsistent/Unknown, autrement dit, je-ne-sais-pas-quoi-faire/je-ne-sais-pas-ce-que-fait-l'autre
3 - HeartBeat ne reçoit plus de réponse de la part du primaire
4 - HeartBeat prend en charge les ressources déclarées et lance le script de mise en oeuvre de 'r0'
5 - Le device drbd0 est toujours dans un état inconsistant. Le script ne fonctionne pas, la ressource 'r0' ne peut être prise en charge.

Le problème n'est donc pas réglé. DRBD ne sait toujours pas quoi faire lorsqu'il est dans un état inconsistant et qu'il ne peut pas communiquer avec le primaire.
Tout est dans la documentation de DRBD à propos de cette situation.

par **Lassaad MATHLOUTHI** » 10 Oct 2008 19:24

Merci beaucoup pour la réponse.

mais je trouve plus le truc dans la doc du drbd.

bien cordialement...

par **Lassaad MATHLOUTHI** » 11 Oct 2008 11:40

bonjour,

je peux plus encore identifier le problème , j'ai passé plus que 10 heurs dans le doc du drbd et j'arrive pas à résoudre le problème.

j'attend vos idées.

Merci en avance.

drbd heartbeat déconnexion du noeud primaire

drbd heartbeat déconnexion du noeud primaire

Qui est en ligne ?