Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Master is set on both sides #2176

Closed
abolinhas opened this issue Aug 6, 2022 · 6 comments
Closed

Master is set on both sides #2176

abolinhas opened this issue Aug 6, 2022 · 6 comments

Comments

@abolinhas
Copy link

abolinhas commented Aug 6, 2022 •
edited
Loading

Describe the bug
I'm trying to create a Master Slave setup, but the Master is set in both sides.
I'm seeing to many messages like this on keepalived.log
Aug 5 21:00:03 CENVLIFEPROXB Keepalived_vrrp[3983]: (VI_1) Received advert from 10.200.6.22 with lower priority 150, ours 200, forcing new election

Expected behavior
Master in one server and slave in other

Keepalived version
2.2.4
Output of keepalived -v
Keepalived v2.2.4 (08/21,2021)

Copyright(C) 2001-2021 Alexandre Cassen, acassen@gmail.com

Built with kernel headers for Linux 4.19.194
Running on Linux 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18)
Distro: Debian GNU/Linux 10 (buster)

configure options: --build=x86_64-linux-gnu --prefix=/usr --includedir=/include --mandir=/share/man --infodir=/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=/lib/x86_64-linux-gnu --libexecdir=/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --enable-snmp --enable-sha1 --enable-snmp-rfcv2 --enable-snmp-rfcv3 --enable-dbus --enable-dbus-create-instance --enable-json --enable-bfd --disable-systemd build_alias=x86_64-linux-gnu

Config options: LVS VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 DBUS DBUS_CREATE_INSTANCE INIT=SYSV

System options: VSYSLOG MEMFD_CREATE IPV4_DEVCONF LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA IPTABLES NET_LINUX_IF_H_COLLISION LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS VRRP_IPVLAN IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE VRF SO_MARK

Distro (please complete the following information):

  • Name: Debian
  • Version: 10.10
  • Architecture: 64

Configuration file:
Master

global_defs {
   max_auto_priority
}
vrrp_script chk_vi_1 {
   script "/etc/keepalived/keepalived-health-checks-1.sh"
   interval 2
   fall 1
   rise 1
   timeout 3
}

! vrrp_primary_node for cenvlifeproxb.aminerals.cl
vrrp_instance VI_1 {
   state MASTER
   interface eth0
   virtual_router_id 30
   priority 200
   advert_int 1
   virtual_ipaddress {
     10.200.6.26/24 dev eth0 label eth0:1
   }
   track_script {
     chk_vi_1
   }
   notify "/etc/keepalived/keepalived-state.sh"
}

Slave

global_defs {
   max_auto_priority
}
vrrp_script chk_vi_1 {
   script "/etc/keepalived/keepalived-health-checks-1.sh"
   interval 2
   fall 1
   rise 1
   timeout 3
}

! vrrp_primary_node for cenvlifeproxa.aminerals.cl
vrrp_instance VI_1 {
   state BACKUP
   interface eth0
   virtual_router_id 30
   priority 150
   advert_int 1
   virtual_ipaddress {
     10.200.6.26/24 dev eth0 label eth0:1
   }
   track_script {
     chk_vi_1
   }
   notify "/etc/keepalived/keepalived-state.sh"
}

Notify and track scripts
Notify script

#!/bin/sh
### BEGIN INIT INFO
# Provides:         keepalived-server
# Required-Start:    $local_fs $syslog
# Required-Stop:     $local_fs $syslog
# Should-Start:
# Should-Stop:
# Default-Start:     3 4 5
# Default-Stop:      0 1 6
# Short-Description: keepalived server
# chkconfig: - 80 75
# description: keepalived server
### END INIT INFO
/usr/bin/php /usr/share/artica-postfix/exec.keepalived-state.php --state $1 $2 $3

Track Script

#!/bin/bash
### BEGIN INIT INFO
# Provides:         keepalived-health-checks-1
### END INIT INFO

### BEGIN Proxy INFO
Proxy_id=$(/bin/pidof squid)
if [[ -z $Proxy_id ]]
then
    /usr/bin/php /usr/share/artica-postfix/exec.keepalived-state.php --state primary_node _1 'Proxy is down'
    exit 1
fi
### END Proxy INFO


### BEGIN Proxy INFO
Proxy_id=$(/bin/pidof squid)
if [[ -z $Proxy_id ]]
then
    /usr/bin/php /usr/share/artica-postfix/exec.keepalived-state.php --state primary_node _1 'Proxy is down'
    exit 1
fi
### END Proxy INFO


### BEGIN DISK SPACE INFO
DISK_USAGE_sda1=`/bin/df /dev/sda1 | tail -1 | awk '{print $5}' | sed 's/[^0-9]*//g'`
if (( $DISK_USAGE_sda1 > 96 ))
then
    /usr/bin/php /usr/share/artica-postfix/exec.keepalived-state.php --state primary_node _1 'sda1 low disk space'
    exit 1
fi
### END DISK SPACE INFO

exit 0

System Log entries

Aug  5 20:58:17 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
Aug  5 20:58:17 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
Aug  5 20:58:17 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
Aug  5 20:58:17 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
Aug  5 20:58:17 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
Aug  5 21:00:03 CENVLIFEPROXB Keepalived_vrrp[3983]: (VI_1) Received advert from 10.200.6.22 with lower priority 150, ours 200, forcing new election
Aug  5 21:00:03 CENVLIFEPROXB Keepalived_vrrp[3983]: (VI_1) Sending/queueing gratuitous ARPs on eth0 for 10.200.6.26
Aug  5 21:00:03 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
Aug  5 21:00:03 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
Aug  5 21:00:03 CENVLIFEPROXB Keepalived_vrrp[3983]: Sending gratuitous ARP on eth0 for 10.200.6.26
@pqarmitage
Copy link
Collaborator

Please provide the full keepalived syslogs on both systems from when keepalived starts.

@abolinhas
Copy link
Author

From Server 1 (Master)

Aug  5 22:09:41 proxy01 Keepalived[18638]: Stopping

Aug  5 22:09:41 proxy01 Keepalived_vrrp[18639]: (VI_1) sent 0 priority

Aug  5 22:09:41 proxy01 Keepalived_vrrp[18639]: (VI_1) removing VIPs.

Aug  5 22:09:42 proxy01 Keepalived_vrrp[18639]: Stopped - used (self/children) 0.007992/0.466628 user time, 0.114720/0.061698 system time

Aug  5 22:09:42 proxy01 Keepalived[18638]: CPU usage (self/children) user: 0.000730/0.474672 system: 0.000000/0.177170

Aug  5 22:09:42 proxy01 Keepalived[18638]: Stopped Keepalived v2.2.4)

Aug  5 22:09:47 proxy01 Keepalived[19072]: Starting Keepalived v2.2.4

Aug  5 22:09:47 proxy01 Keepalived[19072]: WARNING - keepalived was built for newer Linux 4.19.255, running on Linux 4.19.0-20-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17)

Aug  5 22:09:47 proxy01 Keepalived[19072]: Command line: '/sbin/keepalived' '-D' '-S' '0' '-p' '/var/run/keepalived/keepalived.pid'

Aug  5 22:09:47 proxy01 Keepalived[19072]: Opening file '/etc/keepalived/keepalived.conf'.

Aug  5 22:09:47 proxy01 Keepalived[19072]: Configuration file /etc/keepalived/keepalived.conf

Aug  5 22:09:47 proxy01 Keepalived[19073]: Starting VRRP child process, pid=19074

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: Registering Kernel netlink reflector

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: Registering Kernel netlink command channel

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: Script user 'keepalived_script' does not exist

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: Assigned address 10.200.6.23 for interface eth0 

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: Registering gratuitous ARP shared channel

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: (VI_1) removing VIPs.

Aug  5 22:09:47 proxy01 Keepalived[19073]: Startup complete

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: (VI_1) Entering BACKUP STATE (init)

Aug  5 22:09:47 proxy01 Keepalived_vrrp[19074]: VRRP sockpool: [ifindex(  4), family(IPv4), proto(112), fd(13,14) , unicast, address(10.200.6.23)]

Aug  5 22:09:48 proxy01 Keepalived_vrrp[19074]: (VI_1) received lower priority (102) advert from 10.200.6.22 - discarding

Aug  5 22:09:49 proxy01 Keepalived_vrrp[19074]: (VI_1) received lower priority (102) advert from 10.200.6.22 - discarding

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: (VI_1) received lower priority (102) advert from 10.200.6.22 - discarding

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: (VI_1) Receive advertisement timeout

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: (VI_1) Entering MASTER STATE

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: (VI_1) setting VIPs.

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: (VI_1) Sending/queueing gratuitous ARPs on eth0  for 10.200.6.26

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:50 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:51 proxy01 Keepalived_vrrp[19074]: (VI_1) Received advert from 10.200.6.22 with lower priority 102, ours 200, forcing new election

Aug  5 22:09:51 proxy01 Keepalived_vrrp[19074]: (VI_1) Sending/queueing gratuitous ARPs on eth0  for 10.200.6.26

Aug  5 22:09:51 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:51 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:51 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:51 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 22:09:51 proxy01 Keepalived_vrrp[19074]: Sending gratuitous ARP on eth0  for 10.200.6.26

From Server 2 (Slave)

Aug  5 21:57:53 artica-applianc Keepalived[10616]: Stopping

Aug  5 21:57:53 artica-applianc Keepalived_vrrp[10617]: (VI_2) sent 0 priority

Aug  5 21:57:53 artica-applianc Keepalived_vrrp[10617]: (VI_2) removing VIPs.

Aug  5 21:57:54 artica-applianc Keepalived_vrrp[10617]: Stopped - used (self/children) 0.144277/21.939434 user time, 2.430889/13.413394 system time

Aug  5 21:57:54 artica-applianc Keepalived[10616]: CPU usage (self/children) user: 0.000622/22.083757 system: 0.000000/15.845044

Aug  5 21:57:54 artica-applianc Keepalived[10616]: Stopped Keepalived v2.2.7 (01/16,2022)

Aug  5 21:57:59 artica-applianc Keepalived[28513]: Starting Keepalived v2.2.7 (01/16,2022)

Aug  5 21:57:59 artica-applianc Keepalived[28513]: WARNING - keepalived was built for newer Linux 4.19.98, running on Linux 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26)

Aug  5 21:57:59 artica-applianc Keepalived[28513]: Command line: '/sbin/keepalived' '-D' '-S' '0' '-p' '/var/run/keepalived/keepalived.pid'

Aug  5 21:57:59 artica-applianc Keepalived[28513]: Opening file '/etc/keepalived/keepalived.conf'.

Aug  5 21:57:59 artica-applianc Keepalived[28513]: Configuration file /etc/keepalived/keepalived.conf

Aug  5 21:57:59 artica-applianc Keepalived[28514]: NOTICE: setting config option max_auto_priority should result in better keepalived performance

Aug  5 21:57:59 artica-applianc Keepalived[28514]: Starting VRRP child process, pid=28515

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: Registering Kernel netlink reflector

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: Registering Kernel netlink command channel

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: Script user 'keepalived_script' does not exist

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: Assigned address 192.168.60.160 for interface eth0 

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: Registering gratuitous ARP shared channel

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: (VI_2) removing VIPs.

Aug  5 21:57:59 artica-applianc Keepalived[28514]: Startup complete

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: (VI_2) Entering BACKUP STATE (init)

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: VRRP sockpool: [ifindex(  4), family(IPv4), proto(112), fd(13,14) , unicast, address(192.168.60.160)]

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: VRRP_Script(chk_vi_2) succeeded

Aug  5 21:57:59 artica-applianc Keepalived_vrrp[28515]: (VI_2) Changing effective priority from 100 to 102

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: (VI_2) Receive advertisement timeout

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: (VI_2) Entering MASTER STATE

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: (VI_2) setting VIPs.

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: (VI_2) Sending/queueing gratuitous ARPs on eth0  for 10.200.6.26

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:03 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:08 artica-applianc Keepalived_vrrp[28515]: (VI_2) Sending/queueing gratuitous ARPs on eth0  for 10.200.6.26

Aug  5 21:58:08 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:08 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:08 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:08 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

Aug  5 21:58:08 artica-applianc Keepalived_vrrp[28515]: Sending gratuitous ARP on eth0  for 10.200.6.26

@pqarmitage
Copy link
Collaborator

Once you have resolved issue #2175, if you still are experiencing this issue please update this issue report and it can be reopened.

@abolinhas
Copy link
Author

Fixing #2175 don't solve this issue.
I already update to version 2.2.7 and the problem continues, please reopen.

@pqarmitage
Copy link
Collaborator

This problem is ALWAYS caused by some sort of network (including firewall) problem. The problem is that the lower priority keepalived VRRP instance is not receiving the adverts from the higher priority instance.

@pqarmitage pqarmitage reopened this Aug 6, 2022
@abolinhas
Copy link
Author

You are absolutely right, is a network issue.
The problem is that i have multiple nics on each server and by mistake i had configured two nics under same network segment.
So when VRRP broadcast adverts there is routing issue that cause the problem.
Changing the network segment fron one of the nics fix the issue.
Thanks a lot for you help, you can close it now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@abolinhas @pqarmitage