Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Both nginx VMs are master master but traffic is flowing thru one VM #2011

Closed
Adi-AA opened this issue Oct 14, 2021 · 8 comments
Closed

Both nginx VMs are master master but traffic is flowing thru one VM #2011

Adi-AA opened this issue Oct 14, 2021 · 8 comments

Comments

@Adi-AA
Copy link

Adi-AA commented Oct 14, 2021 •
edited by pqarmitage
Loading

Describe the bug
Right after this message in keepalived logs - A thread timer expired 1.034729 seconds ago, the Keepalived changed stage to MASTER
0001 should be MASTER and 0002 should be the BACKUP, the state change happened on the 0002 VM and it changed from BACKUP to MASTER many times as per logs. Now both nodes are MASTER
To Reproduce
I have the logs.
They are still MASTER MASTER however the the traffic is routing only through 0001 node and 0002 node is not routing traffic which is strange.

Expected behavior
We have a MASTER (0001) and BACKUP (0002) HA setup, now both VMs are MASTER MASTER

Keepalived version
Keepalived v2.2.2 (03/05,2021)
Output of keepalived -v
[root@estullbp0002 entlbusr]# keepalived -v
Keepalived v2.2.2 (03/05,2021)

Copyright(C) 2001-2021 Alexandre Cassen, acassen@gmail.com

Built with kernel headers for Linux 3.10.0
Running on Linux 3.10.0-1160.21.1.el7.x86_64 #1 SMP Mon Feb 22 18:03:13 EST 2021
Distro: Red Hat Enterprise Linux Server 7.9 (Maipo)

configure options: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-bfd --disable-lvs --disable-snmp --disable-sha1 --with-init=none build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic LDFLAGS=-Wl,-z,relro

Config options: VRRP VRRP_AUTH VRRP_VMAC BFD OLD_CHKSUM_COMPAT INIT=none

System options: VSYSLOG RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA IFA_FLAGS NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE SO_MARK
[root@estullbp0002 entlbusr]#

Distro (please complete the following information):

  • Name: [e.g. Fedora, Ubuntu]
  • Version: [e.g. 29]
  • Architecture: [e.g. x86_64]

Details of any containerisation or hosted service (e.g. AWS)
If keepalived is being run in a container or on a hosted service, provide full details

Configuration file:
[root@estullbp0002 entlbusr]# cat /var/run/nginx-ha-keepalived.state
STATE=MASTER
[root@estullbp0002 entlbusr]#

A full copy of the configuration file, obfuscated if necessary to protect passwords and IP addresses
[root@estullbp0002 entlbusr]# cat /etc/keepalived/keepalived.conf
global_defs {
vrrp_version 3
}

vrrp_script chk_manual_failover {
script "/usr/libexec/keepalived/nginx-ha-manual-failover"
interval 10
weight 50
}

vrrp_script chk_nginx_service {
script "/usr/libexec/keepalived/nginx-ha-check"
interval 3
weight 50
}

vrrp_instance VI_1 {
interface Prod
priority 100
virtual_router_id 51
advert_int 1
accept
garp_master_refresh 5
garp_master_refresh_repeat 1
unicast_src_ip 10.232.20.6
unicast_peer {
10.232.20.5
}
virtual_ipaddress {
10.232.20.7
10.232.20.8
10.232.20.9
10.232.20.10
10.232.20.11
10.232.20.12
10.232.20.13
10.232.20.14
10.232.20.15
10.232.20.16
10.232.20.17
10.232.20.18
10.232.20.19
10.232.20.20
10.232.20.21
10.232.20.22
10.232.20.23
10.232.20.24
10.232.20.25
10.232.20.26
10.232.20.27
10.232.20.28
10.232.20.29
10.232.20.30
10.232.20.31
10.232.20.32
10.232.20.33
10.232.20.34
10.232.20.35
10.232.20.36
10.232.20.37
10.232.20.38
10.232.20.39
10.232.20.40
10.232.20.41
10.232.20.42
10.232.20.43
10.232.20.44
10.232.20.45
10.232.20.46
10.232.20.47
10.232.20.48
10.232.20.49
10.232.20.50
10.232.20.51
10.232.20.52
10.232.20.53
10.232.20.54
10.232.20.55
10.232.20.56
10.232.20.57
10.232.20.58
10.232.20.59
10.232.20.60
10.232.20.61
10.232.20.62
10.232.20.63
10.232.20.64
10.232.20.65
10.232.20.66
10.232.20.67
10.232.20.68
10.232.20.69
10.232.20.70
10.232.20.71
10.232.20.72
10.232.20.73
10.232.20.74
10.232.20.75
10.232.20.76
10.232.20.77
10.232.20.78
10.232.20.79
10.232.20.80
10.232.20.81
10.232.20.82
10.232.20.83
10.232.20.84
10.232.20.85
10.232.20.86
10.232.20.87
10.232.20.88
10.232.20.89
10.232.20.90
10.232.20.91
10.232.20.92
10.232.20.93
10.232.20.94
10.232.20.95
10.232.20.96
10.232.20.97
10.232.20.98
10.232.20.99
10.232.20.100
10.232.20.101
10.232.20.102
10.232.20.103
10.232.20.104
10.232.20.105
10.232.20.106
10.232.20.107
10.232.20.108
10.232.20.109
10.232.20.110
10.232.20.111
10.232.20.112
10.232.20.113
10.232.20.114
10.232.20.115
10.232.20.116
10.232.20.117
10.232.20.118
10.232.20.119
10.232.20.120
10.232.20.121
10.232.20.120
10.232.20.123
10.232.20.124
10.232.20.125
10.232.20.126
10.232.20.127
10.232.20.128
10.232.20.129
10.232.20.130
10.232.20.131
10.232.20.132
10.232.20.133
10.232.20.134
10.232.20.135
10.232.20.136
10.232.20.137
10.232.20.138
10.232.20.139
10.232.20.140
10.232.20.141
10.232.20.142
10.232.20.143
10.232.20.144
10.232.20.145
10.232.20.146
10.232.20.147
10.232.20.148
10.232.20.149
10.232.20.150
10.232.20.151
10.232.20.152
10.232.20.153
10.232.20.154
10.232.20.155
10.232.20.156
10.232.20.157
10.232.20.158
10.232.20.159
10.232.20.160
10.232.20.161
10.232.20.162
10.232.20.163
10.232.20.164
10.232.20.165
10.232.20.166
10.232.20.167
10.232.20.168
10.232.20.169
10.232.20.170
10.232.20.171
10.232.20.172
10.232.20.173
10.232.20.174
10.232.20.175
10.232.20.176
10.232.20.177
10.232.20.178
10.232.20.179
10.232.20.180
10.232.20.181
10.232.20.182
10.232.20.183
10.232.20.184
10.232.20.185
10.232.20.186
10.232.20.187
10.232.20.188
10.232.20.189
10.232.20.190
10.232.20.191
10.232.20.192
10.232.20.193
10.232.20.194
10.232.20.195
10.232.20.196
10.232.20.197
10.232.20.198
10.232.20.199
10.232.20.200
10.232.20.201
10.232.20.202
10.232.20.203
10.232.20.204
10.232.20.205
10.232.20.206
10.232.20.207
10.232.20.208
10.232.20.209
10.232.20.210
10.232.20.211
10.232.20.212
10.232.20.213
10.232.20.214
10.232.20.215
10.232.20.216
10.232.20.217
10.232.20.218
10.232.20.219
10.232.20.200
10.232.20.201
10.232.20.202
10.232.20.203
10.232.20.204
10.232.20.205
10.232.20.206
10.232.20.207
10.232.20.208
10.232.20.209
10.232.20.230
10.232.20.231
10.232.20.232
10.232.20.233
10.232.20.234
10.232.20.235
10.232.20.236
10.232.20.237
10.232.20.238
10.232.20.239
10.232.20.240
10.232.20.241
10.232.20.242
10.232.20.243
10.232.20.244
10.232.20.245
10.232.20.246
10.232.20.247
10.232.20.248
10.232.20.249
10.232.20.250
10.232.20.251
10.232.20.252
10.232.20.253
10.232.20.254
10.232.20.255

    }
    track_script {
            chk_nginx_service
            chk_manual_failover
    }
    notify "/usr/libexec/keepalived/nginx-ha-notify"

}
vrrp_instance VI_2 {
interface vlan1119
priority 100
virtual_router_id 52
advert_int 1
accept
garp_master_refresh 5
garp_master_refresh_repeat 1
unicast_src_ip 10.83.10.2
unicast_peer {
10.83.10.1
}
virtual_ipaddress {
#10.83.8.0
#10.83.8.1
#10.83.8.2
#10.83.8.3
#10.83.8.4
#10.83.8.5
#10.83.8.6
#10.83.8.7
#10.83.8.8
#10.83.8.9
#10.83.8.10
#10.83.8.11
#10.83.8.12
#10.83.8.13
#10.83.8.14
#10.83.8.15
#10.83.8.16
#10.83.8.17
#10.83.8.18
#10.83.8.19
#10.83.8.20
#10.83.8.21
#10.83.8.22
#10.83.8.23
#10.83.8.24
#10.83.8.25
#10.83.8.26
#10.83.8.27
#10.83.8.28
#10.83.8.29
#10.83.8.30
#10.83.8.31
#10.83.8.32
#10.83.8.33
#10.83.8.34
#10.83.8.35
#10.83.8.36
#10.83.8.37
#10.83.8.38
#10.83.8.39
#10.83.8.40
#10.83.8.41
#10.83.8.42
#10.83.8.43
#10.83.8.44
#10.83.8.45
#10.83.8.46
#10.83.8.47
#10.83.8.48
10.83.8.49
#10.83.8.50
#10.83.8.51
#10.83.8.52
#10.83.8.53
#10.83.8.54
#10.83.8.55
#10.83.8.56
#10.83.8.57
#10.83.8.58
#10.83.8.59
#10.83.8.60
#10.83.8.61
#10.83.8.62
#10.83.8.63
#10.83.8.64
#10.83.8.65
10.83.8.66
#10.83.8.67
#10.83.8.68
#10.83.8.69
#10.83.8.70
#10.83.8.71
#10.83.8.72
#10.83.8.73
10.83.8.74
#10.83.8.75
#10.83.8.76
#10.83.8.77
#10.83.8.78
#10.83.8.79
#10.83.8.80
#10.83.8.81
10.83.8.82
#10.83.8.83
#10.83.8.84
#10.83.8.85
#10.83.8.86
#10.83.8.87
#10.83.8.88
#10.83.8.89
#10.83.8.90
#10.83.8.91
#10.83.8.92
#10.83.8.93
#10.83.8.94
#10.83.8.95
#10.83.8.96
#10.83.8.97
#10.83.8.98
#10.83.8.99
#10.83.8.100
#10.83.8.101
#10.83.8.102
#10.83.8.103
#10.83.8.104
#10.83.8.105
#10.83.8.106
#10.83.8.107
#10.83.8.108
#10.83.8.109
#10.83.8.110
#10.83.8.111
#10.83.8.112
#10.83.8.113
#10.83.8.114
#10.83.8.115
#10.83.8.116
#10.83.8.117
#10.83.8.118
#10.83.8.119
#10.83.8.120
#10.83.8.121
#10.83.8.122
#10.83.8.123
#10.83.8.124
10.83.8.125
#10.83.8.126
10.83.8.127
#10.83.8.128
#10.83.8.129
#10.83.8.130
10.83.8.131
#10.83.8.132
10.83.8.133
#10.83.8.134
10.83.8.135
10.83.8.136
#10.83.8.137
#10.83.8.138
#10.83.8.139
#10.83.8.140
#10.83.8.141
#10.83.8.142
#10.83.8.143
#10.83.8.144
#10.83.8.145
#10.83.8.146
#10.83.8.147
#10.83.8.148
#10.83.8.149
#10.83.8.150
#10.83.8.151
#10.83.8.152
#10.83.8.153
#10.83.8.154
#10.83.8.155
#10.83.8.156
#10.83.8.157
#10.83.8.158
#10.83.8.159
#10.83.8.160
#10.83.8.161
#10.83.8.162
#10.83.8.163
#10.83.8.164
#10.83.8.165
#10.83.8.166
10.83.8.167
#10.83.8.168
10.83.8.169
#10.83.8.170
#10.83.8.171
#10.83.8.172
#10.83.8.173
#10.83.8.174
#10.83.8.175
#10.83.8.176
#10.83.8.177
#10.83.8.178
#10.83.8.179
#10.83.8.180
#10.83.8.181
#10.83.8.182
#10.83.8.183
#10.83.8.184
#10.83.8.185
#10.83.8.186
#10.83.8.187
#10.83.8.188
#10.83.8.189
#10.83.8.190
#10.83.8.191
10.83.8.192
#10.83.8.193
#10.83.8.194
#10.83.8.195
#10.83.8.196
#10.83.8.197
#10.83.8.198
#10.83.8.199
#10.83.8.200
#10.83.8.201
#10.83.8.202
#10.83.8.203
10.83.8.204
10.83.8.205
#10.83.8.206
#10.83.8.207
#10.83.8.208
#10.83.8.209
#10.83.8.210
#10.83.8.211
#10.83.8.212
#10.83.8.213
#10.83.8.214
#10.83.8.215
#10.83.8.216
#10.83.8.217
#10.83.8.218
#10.83.8.219
#10.83.8.220
#10.83.8.221
#10.83.8.222
#10.83.8.223
#10.83.8.224
#10.83.8.225
#10.83.8.226
#10.83.8.227
10.83.8.228
#10.83.8.229
#10.83.8.230
10.83.8.231
#10.83.8.232
10.83.8.233
#10.83.8.234
#10.83.8.235
#10.83.8.236
10.83.8.237
#10.83.8.238
#10.83.8.239
#10.83.8.240
#10.83.8.241
#10.83.8.242
#10.83.8.243
#10.83.8.244
#10.83.8.245
#10.83.8.246
10.83.8.247
#10.83.8.248
10.83.8.249
#10.83.8.250
#10.83.8.251
#10.83.8.252
10.83.8.253
#10.83.8.254
#10.83.8.255
}
track_script {
chk_nginx_service
chk_manual_failover
}
notify "/usr/libexec/keepalived/nginx-ha-notify"
}
vrrp_instance VI_3 {
interface vlan100
priority 100
virtual_router_id 53
advert_int 1
accept
garp_master_refresh 5
garp_master_refresh_repeat 1
unicast_src_ip 10.76.76.81
unicast_peer {
10.76.76.80
}
virtual_ipaddress {
#10.76.76.58 - Stage not yet migrated
10.76.77.10
10.76.77.17
10.76.79.107
10.76.79.114
10.76.79.116
10.76.79.118
#10.76.79.120 - Notifications/Meals rollback
10.76.79.124
#10.76.79.138 - Rollback Wrls
10.76.79.167
10.76.79.168
10.76.79.27
#10.76.79.57 - Sharepoint Rollback
#10.76.79.58 - Sharepoint Rollback
#10.76.79.59 - Sharepoint Rollback
#10.76.79.60 - Sharepoint Rollback
#10.76.79.61 - Sharepoint Rollback
#10.76.79.62 - Sharepoint Rollback
#10.76.79.63 - Sharepoint Rollback
#10.76.79.64 - Sharepoint Rollback
#10.76.79.65 - Sharepoint Rollback
10.76.79.94
#10.76.79.95 - Sharepoint Rollback
#10.76.79.97 - Sharepoint Rollback
}
track_script {
chk_nginx_service
chk_manual_failover
}
notify "/usr/libexec/keepalived/nginx-ha-notify"
}[root@estullbp0002 entlbusr]#
Notify and track scripts
please see above in the config itself.
[root@estullbp0002 keepalived]# ll
total 12
-rwxr-xr-x 1 root root 477 Apr 12 2021 nginx-ha-check
-rwxr-xr-x 1 root root 74 Apr 12 2021 nginx-ha-manual-failover
-rwxr-xr-x 1 root root 638 Apr 12 2021 nginx-ha-notify
[root@estullbp0002 keepalived]# cat nginx-ha-check
#!/bin/sh

PATH=/bin:/sbin:/usr/bin:/usr/sbin

STATEFILE=/var/run/nginx-ha-keepalived.state

if [ -s "$STATEFILE" ]; then
. "$STATEFILE"
case "$STATE" in
"BACKUP"|"MASTER"|"FAULT")
service nginx status || service nginx-debug status
exit $?
;;
*|"")
logger -t nginx-ha-keepalived "Unknown state: '$STATE'"
exit 1
;;
esac
fi

service nginx status
exit $?
[root@estullbp0002 keepalived]# cat nginx-ha-manual-failover
#!/bin/sh

[ -f "/var/run/keepalived-manual-failover" ] && exit 1

exit 0
[root@estullbp0002 keepalived]# cat nginx-ha-notify
#!/bin/sh

PATH=/bin:/sbin:/usr/bin:/usr/sbin

umask 022

TYPE=$1
NAME=$2
STATE=$3

STATEFILE=/var/run/nginx-ha-keepalived.state

logger -t nginx-ha-keepalived "Transition to state '$STATE' on VRRP instance '$NAME'."

case $STATE in
"MASTER")
service nginx start ||:
echo "STATE=$STATE" > $STATEFILE
exit 0
;;
"BACKUP"|"FAULT")
echo "STATE=$STATE" > $STATEFILE
exit 0
;;
*) logger -t nginx-ha-keepalived "Unknown state: '$STATE'"
exit 1
;;
esac
[root@estullbp0002 keepalived]#

[root@estullbp0001 entlbusr]# cat /etc/keepalived/keepalived.conf
global_defs {
vrrp_version 3
}

vrrp_script chk_manual_failover {
script "/usr/libexec/keepalived/nginx-ha-manual-failover"
interval 10
weight 50
}

vrrp_script chk_nginx_service {
script "/usr/libexec/keepalived/nginx-ha-check"
interval 3
weight 50
}

vrrp_instance VI_1 {
interface Prod
priority 101
virtual_router_id 51
advert_int 1
accept
garp_master_refresh 5
garp_master_refresh_repeat 1
unicast_src_ip 10.232.20.5
unicast_peer {
10.232.20.6
}
virtual_ipaddress {
10.232.20.7
10.232.20.8
10.232.20.9
10.232.20.10
10.232.20.11
10.232.20.12
10.232.20.13
10.232.20.14
10.232.20.15
10.232.20.16
10.232.20.17
10.232.20.18
10.232.20.19
10.232.20.20
10.232.20.21
10.232.20.22
10.232.20.23
10.232.20.24
10.232.20.25
10.232.20.26
10.232.20.27
10.232.20.28
10.232.20.29
10.232.20.30
10.232.20.31
10.232.20.32
10.232.20.33
10.232.20.34
10.232.20.35
10.232.20.36
10.232.20.37
10.232.20.38
10.232.20.39
10.232.20.40
10.232.20.41
10.232.20.42
10.232.20.43
10.232.20.44
10.232.20.45
10.232.20.46
10.232.20.47
10.232.20.48
10.232.20.49
10.232.20.50
10.232.20.51
10.232.20.52
10.232.20.53
10.232.20.54
10.232.20.55
10.232.20.56
10.232.20.57
10.232.20.58
10.232.20.59
10.232.20.60
10.232.20.61
10.232.20.62
10.232.20.63
10.232.20.64
10.232.20.65
10.232.20.66
10.232.20.67
10.232.20.68
10.232.20.69
10.232.20.70
10.232.20.71
10.232.20.72
10.232.20.73
10.232.20.74
10.232.20.75
10.232.20.76
10.232.20.77
10.232.20.78
10.232.20.79
10.232.20.80
10.232.20.81
10.232.20.82
10.232.20.83
10.232.20.84
10.232.20.85
10.232.20.86
10.232.20.87
10.232.20.88
10.232.20.89
10.232.20.90
10.232.20.91
10.232.20.92
10.232.20.93
10.232.20.94
10.232.20.95
10.232.20.96
10.232.20.97
10.232.20.98
10.232.20.99
10.232.20.100
10.232.20.101
10.232.20.102
10.232.20.103
10.232.20.104
10.232.20.105
10.232.20.106
10.232.20.107
10.232.20.108
10.232.20.109
10.232.20.110
10.232.20.111
10.232.20.112
10.232.20.113
10.232.20.114
10.232.20.115
10.232.20.116
10.232.20.117
10.232.20.118
10.232.20.119
10.232.20.120
10.232.20.121
10.232.20.120
10.232.20.123
10.232.20.124
10.232.20.125
10.232.20.126
10.232.20.127
10.232.20.128
10.232.20.129
10.232.20.130
10.232.20.131
10.232.20.132
10.232.20.133
10.232.20.134
10.232.20.135
10.232.20.136
10.232.20.137
10.232.20.138
10.232.20.139
10.232.20.140
10.232.20.141
10.232.20.142
10.232.20.143
10.232.20.144
10.232.20.145
10.232.20.146
10.232.20.147
10.232.20.148
10.232.20.149
10.232.20.150
10.232.20.151
10.232.20.152
10.232.20.153
10.232.20.154
10.232.20.155
10.232.20.156
10.232.20.157
10.232.20.158
10.232.20.159
10.232.20.160
10.232.20.161
10.232.20.162
10.232.20.163
10.232.20.164
10.232.20.165
10.232.20.166
10.232.20.167
10.232.20.168
10.232.20.169
10.232.20.170
10.232.20.171
10.232.20.172
10.232.20.173
10.232.20.174
10.232.20.175
10.232.20.176
10.232.20.177
10.232.20.178
10.232.20.179
10.232.20.180
10.232.20.181
10.232.20.182
10.232.20.183
10.232.20.184
10.232.20.185
10.232.20.186
10.232.20.187
10.232.20.188
10.232.20.189
10.232.20.190
10.232.20.191
10.232.20.192
10.232.20.193
10.232.20.194
10.232.20.195
10.232.20.196
10.232.20.197
10.232.20.198
10.232.20.199
10.232.20.200
10.232.20.201
10.232.20.202
10.232.20.203
10.232.20.204
10.232.20.205
10.232.20.206
10.232.20.207
10.232.20.208
10.232.20.209
10.232.20.210
10.232.20.211
10.232.20.212
10.232.20.213
10.232.20.214
10.232.20.215
10.232.20.216
10.232.20.217
10.232.20.218
10.232.20.219
10.232.20.200
10.232.20.201
10.232.20.202
10.232.20.203
10.232.20.204
10.232.20.205
10.232.20.206
10.232.20.207
10.232.20.208
10.232.20.209
10.232.20.230
10.232.20.231
10.232.20.232
10.232.20.233
10.232.20.234
10.232.20.235
10.232.20.236
10.232.20.237
10.232.20.238
10.232.20.239
10.232.20.240
10.232.20.241
10.232.20.242
10.232.20.243
10.232.20.244
10.232.20.245
10.232.20.246
10.232.20.247
10.232.20.248
10.232.20.249
10.232.20.250
10.232.20.251
10.232.20.252
10.232.20.253
10.232.20.254
10.232.20.255

    }
    track_script {
            chk_nginx_service
            chk_manual_failover
    }
    notify "/usr/libexec/keepalived/nginx-ha-notify"

}
vrrp_instance VI_2 {
interface vlan1119
priority 101
virtual_router_id 52
advert_int 1
accept
garp_master_refresh 5
garp_master_refresh_repeat 1
unicast_src_ip 10.83.10.1
unicast_peer {
10.83.10.2
}
virtual_ipaddress {
#10.83.8.0
#10.83.8.1
#10.83.8.2
#10.83.8.3
#10.83.8.4
#10.83.8.5
#10.83.8.6
#10.83.8.7
#10.83.8.8
#10.83.8.9
#10.83.8.10
#10.83.8.11
#10.83.8.12
#10.83.8.13
#10.83.8.14
#10.83.8.15
#10.83.8.16
#10.83.8.17
#10.83.8.18
#10.83.8.19
#10.83.8.20
#10.83.8.21
#10.83.8.22
#10.83.8.23
#10.83.8.24
#10.83.8.25
#10.83.8.26
#10.83.8.27
#10.83.8.28
#10.83.8.29
#10.83.8.30
#10.83.8.31
#10.83.8.32
#10.83.8.33
#10.83.8.34
#10.83.8.35
#10.83.8.36
#10.83.8.37
#10.83.8.38
#10.83.8.39
#10.83.8.40
#10.83.8.41
#10.83.8.42
#10.83.8.43
#10.83.8.44
#10.83.8.45
#10.83.8.46
#10.83.8.47
#10.83.8.48
10.83.8.49
#10.83.8.50
#10.83.8.51
#10.83.8.52
#10.83.8.53
#10.83.8.54
#10.83.8.55
#10.83.8.56
#10.83.8.57
#10.83.8.58
#10.83.8.59
#10.83.8.60
#10.83.8.61
#10.83.8.62
#10.83.8.63
#10.83.8.64
#10.83.8.65
10.83.8.66
#10.83.8.67
#10.83.8.68
#10.83.8.69
#10.83.8.70
#10.83.8.71
#10.83.8.72
#10.83.8.73
10.83.8.74
#10.83.8.75
#10.83.8.76
#10.83.8.77
#10.83.8.78
#10.83.8.79
#10.83.8.80
#10.83.8.81
10.83.8.82
#10.83.8.83
#10.83.8.84
#10.83.8.85
#10.83.8.86
#10.83.8.87
#10.83.8.88
#10.83.8.89
#10.83.8.90
#10.83.8.91
#10.83.8.92
#10.83.8.93
#10.83.8.94
#10.83.8.95
#10.83.8.96
#10.83.8.97
#10.83.8.98
#10.83.8.99
#10.83.8.100
#10.83.8.101
#10.83.8.102
#10.83.8.103
#10.83.8.104
#10.83.8.105
#10.83.8.106
#10.83.8.107
#10.83.8.108
#10.83.8.109
#10.83.8.110
#10.83.8.111
#10.83.8.112
#10.83.8.113
#10.83.8.114
#10.83.8.115
#10.83.8.116
#10.83.8.117
#10.83.8.118
#10.83.8.119
#10.83.8.120
#10.83.8.121
#10.83.8.122
#10.83.8.123
#10.83.8.124
10.83.8.125
#10.83.8.126
10.83.8.127
#10.83.8.128
#10.83.8.129
#10.83.8.130
10.83.8.131
#10.83.8.132
10.83.8.133
#10.83.8.134
10.83.8.135
10.83.8.136
#10.83.8.137
#10.83.8.138
#10.83.8.139
#10.83.8.140
#10.83.8.141
#10.83.8.142
#10.83.8.143
#10.83.8.144
#10.83.8.145
#10.83.8.146
#10.83.8.147
#10.83.8.148
#10.83.8.149
#10.83.8.150
#10.83.8.151
#10.83.8.152
#10.83.8.153
#10.83.8.154
#10.83.8.155
#10.83.8.156
#10.83.8.157
#10.83.8.158
#10.83.8.159
#10.83.8.160
#10.83.8.161
#10.83.8.162
#10.83.8.163
#10.83.8.164
#10.83.8.165
#10.83.8.166
10.83.8.167
#10.83.8.168
10.83.8.169
#10.83.8.170
#10.83.8.171
#10.83.8.172
#10.83.8.173
#10.83.8.174
#10.83.8.175
#10.83.8.176
#10.83.8.177
#10.83.8.178
#10.83.8.179
#10.83.8.180
#10.83.8.181
#10.83.8.182
#10.83.8.183
#10.83.8.184
#10.83.8.185
#10.83.8.186
#10.83.8.187
#10.83.8.188
#10.83.8.189
#10.83.8.190
#10.83.8.191
10.83.8.192
#10.83.8.193
#10.83.8.194
#10.83.8.195
#10.83.8.196
#10.83.8.197
#10.83.8.198
#10.83.8.199
#10.83.8.200
#10.83.8.201
#10.83.8.202
#10.83.8.203
10.83.8.204
10.83.8.205
#10.83.8.206
#10.83.8.207
#10.83.8.208
#10.83.8.209
#10.83.8.210
#10.83.8.211
#10.83.8.212
#10.83.8.213
#10.83.8.214
#10.83.8.215
#10.83.8.216
#10.83.8.217
#10.83.8.218
#10.83.8.219
#10.83.8.220
#10.83.8.221
#10.83.8.222
#10.83.8.223
#10.83.8.224
#10.83.8.225
#10.83.8.226
#10.83.8.227
10.83.8.228
#10.83.8.229
#10.83.8.230
10.83.8.231
#10.83.8.232
10.83.8.233
#10.83.8.234
#10.83.8.235
#10.83.8.236
10.83.8.237
#10.83.8.238
#10.83.8.239
#10.83.8.240
#10.83.8.241
#10.83.8.242
#10.83.8.243
#10.83.8.244
#10.83.8.245
#10.83.8.246
10.83.8.247
#10.83.8.248
10.83.8.249
#10.83.8.250
#10.83.8.251
#10.83.8.252
10.83.8.253
#10.83.8.254
#10.83.8.255
}
track_script {
chk_nginx_service
chk_manual_failover
}
notify "/usr/libexec/keepalived/nginx-ha-notify"
}
vrrp_instance VI_3 {
interface vlan100
priority 101
virtual_router_id 53
advert_int 1
accept
garp_master_refresh 5
garp_master_refresh_repeat 1
unicast_src_ip 10.76.76.80
unicast_peer {
10.76.76.81
}
virtual_ipaddress {
#10.76.76.58 - Stage not yet migrated
10.76.77.10
10.76.77.17
10.76.79.107
10.76.79.114
10.76.79.116
10.76.79.118
#10.76.79.120 - Notifications/Meals rollback
10.76.79.124
#10.76.79.138 - Rollback Wrls
10.76.79.167
10.76.79.168
10.76.79.27
#10.76.79.57 - Sharepoint Rollback
#10.76.79.58 - Sharepoint Rollback
#10.76.79.59 - Sharepoint Rollback
#10.76.79.60 - Sharepoint Rollback
#10.76.79.61 - Sharepoint Rollback
#10.76.79.62 - Sharepoint Rollback
#10.76.79.63 - Sharepoint Rollback
#10.76.79.64 - Sharepoint Rollback
#10.76.79.65 - Sharepoint Rollback
10.76.79.94
#10.76.79.95 - Sharepoint Rollback
#10.76.79.97 - Sharepoint Rollback
}
track_script {
chk_nginx_service
chk_manual_failover
}
notify "/usr/libexec/keepalived/nginx-ha-notify"
}[root@estullbp0001 entlbusr]#

If any notify or track scripts are in use, please provide copies of them

System Log entries
##Attached##

Full keepalived system log entries from when keepalived started
##Attached all logs####
[root@estullbp0002 nginx]# zcat keepalived.log-20211006.gz | grep thread -i
Oct 9 17:45:17 estullbp0002 Keepalived_vrrp[42905]: A thread timer expired 5.205448 seconds ago
[root@estullbp0002 nginx]# zcat keepalived.log-20211006.gz | grep state -i
Oct 9 17:45:17 estullbp0002 Keepalived_vrrp[42905]: (VI_1) Entering MASTER STATE
[root@estullbp0002 nginx]#
Did keepalived coredump?
n/a
If so, can you please provide a stacktrace from the coredump, using gdb.
n/a
Additional context
Add any other context about the problem here.
ip a outputs from both nodes:

[root@estullbp0001 aa846592]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: vlan1120: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:32:6e brd ff:ff:ff:ff:ff:ff
    inet 10.232.17.1/22 brd 10.232.19.255 scope global noprefixroute vlan1120
       valid_lft forever preferred_lft forever
3: Prod: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:aa:d0 brd ff:ff:ff:ff:ff:ff
    inet 10.232.20.5/22 brd 10.232.23.255 scope global noprefixroute Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.7/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.8/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.9/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.10/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.11/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.12/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.13/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.14/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.15/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.16/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.17/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.18/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.19/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.20/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.21/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.22/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.23/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.24/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.25/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.26/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.27/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.28/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.29/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.30/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.31/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.32/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.33/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.34/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.35/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.36/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.37/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.38/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.39/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.40/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.41/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.42/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.43/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.44/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.45/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.46/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.47/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.48/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.49/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.50/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.51/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.52/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.53/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.54/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.55/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.56/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.57/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.58/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.59/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.60/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.61/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.62/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.63/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.64/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.65/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.66/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.67/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.68/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.69/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.70/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.71/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.72/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.73/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.74/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.75/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.76/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.77/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.78/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.79/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.80/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.81/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.82/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.83/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.84/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.85/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.86/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.87/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.88/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.89/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.90/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.91/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.92/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.93/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.94/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.95/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.96/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.97/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.98/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.99/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.100/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.101/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.102/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.103/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.104/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.105/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.106/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.107/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.108/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.109/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.110/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.111/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.112/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.113/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.114/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.115/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.116/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.117/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.118/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.119/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.120/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.121/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.123/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.124/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.125/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.126/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.127/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.128/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.129/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.130/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.131/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.132/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.133/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.134/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.135/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.136/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.137/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.138/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.139/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.140/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.141/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.142/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.143/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.144/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.145/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.146/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.147/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.148/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.149/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.150/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.151/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.152/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.153/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.154/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.155/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.156/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.157/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.158/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.159/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.160/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.161/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.162/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.163/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.164/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.165/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.166/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.167/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.168/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.169/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.170/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.171/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.172/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.173/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.174/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.175/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.176/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.177/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.178/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.179/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.180/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.181/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.182/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.183/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.184/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.185/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.186/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.187/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.188/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.189/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.190/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.191/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.192/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.193/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.194/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.195/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.196/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.197/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.198/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.199/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.200/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.201/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.202/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.203/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.204/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.205/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.206/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.207/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.208/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.209/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.210/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.211/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.212/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.213/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.214/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.215/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.216/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.217/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.218/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.219/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.230/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.231/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.232/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.233/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.234/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.235/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.236/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.237/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.238/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.239/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.240/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.241/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.242/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.243/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.244/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.245/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.246/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.247/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.248/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.249/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.250/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.251/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.252/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.253/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.254/32 scope global Prod
       valid_lft forever preferred_lft forever
    inet 10.232.20.255/32 scope global Prod
       valid_lft forever preferred_lft forever
4: vlan1119: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:3b:22 brd ff:ff:ff:ff:ff:ff
    inet 10.83.10.1/22 brd 10.83.11.255 scope global noprefixroute vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.49/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.66/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.74/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.82/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.125/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.127/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.131/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.133/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.135/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.136/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.167/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.169/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.192/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.204/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.205/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.228/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.231/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.233/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.237/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.247/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.249/32 scope global vlan1119
       valid_lft forever preferred_lft forever
    inet 10.83.8.253/32 scope global vlan1119
       valid_lft forever preferred_lft forever
5: Mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:cf:9a brd ff:ff:ff:ff:ff:ff
    inet 10.89.37.1/22 brd 10.89.39.255 scope global noprefixroute Mgmt
       valid_lft forever preferred_lft forever
6: vlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:ab:5d brd ff:ff:ff:ff:ff:ff
    inet 10.76.76.80/22 brd 10.76.79.255 scope global noprefixroute vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.77.10/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.77.17/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.107/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.114/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.116/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.118/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.124/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.167/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.168/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.27/32 scope global vlan100
       valid_lft forever preferred_lft forever
    inet 10.76.79.94/32 scope global vlan100
       valid_lft forever preferred_lft forever
7: vlan3005: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:7a:70 brd ff:ff:ff:ff:ff:ff
    inet 10.232.8.201/23 brd 10.232.9.255 scope global noprefixroute vlan3005
       valid_lft forever preferred_lft forever
[root@estullbp0001 aa846592]#

####Node 0002####

[root@estullbp0002 aa846592]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: vlan1120: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:b1:0f brd ff:ff:ff:ff:ff:ff
    inet 10.232.17.2/22 brd 10.232.19.255 scope global noprefixroute vlan1120
       valid_lft forever preferred_lft forever
3: Prod: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:dd:09 brd ff:ff:ff:ff:ff:ff
    inet 10.232.20.6/22 brd 10.232.23.255 scope global noprefixroute Prod
       valid_lft forever preferred_lft forever
4: vlan1119: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:87:46 brd ff:ff:ff:ff:ff:ff
    inet 10.83.10.2/22 brd 10.83.11.255 scope global noprefixroute vlan1119
       valid_lft forever preferred_lft forever
5: Mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:27:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.89.37.2/22 brd 10.89.39.255 scope global noprefixroute Mgmt
       valid_lft forever preferred_lft forever
6: vlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:53:bc brd ff:ff:ff:ff:ff:ff
    inet 10.76.76.81/22 brd 10.76.79.255 scope global noprefixroute vlan100
       valid_lft forever preferred_lft forever
7: vlan3005: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:91:e4 brd ff:ff:ff:ff:ff:ff
    inet 10.232.8.202/23 brd 10.232.9.255 scope global noprefixroute vlan3005
       valid_lft forever preferred_lft forever
[root@estullbp0002 aa846592]#

Node0002-messages-20211010.gz
Node0002-keepalived.log-20211006.gz
Node0002-keepalived.log-20211001.gz
@pqarmitage
Copy link
Collaborator

The answer lies in the log messages:

Sep 30 17:38:22 estullbp0002 Keepalived_vrrp[42905]: A thread timer expired 8.727992 seconds ago
Sep 30 17:38:22 estullbp0002 Keepalived_vrrp[42905]: (VI_1) Receive advertisement timeout
Sep 30 17:38:22 estullbp0002 Keepalived_vrrp[42905]: (VI_1) Entering MASTER STATE

and

Oct 9 17:45:17 estullbp0002 Keepalived_vrrp[42905]: A thread timer expired 5.205448 seconds ago
Oct  9 17:45:17 estullbp0002 Keepalived_vrrp[42905]: (VI_1) Receive advertisement timeout
Oct  9 17:45:17 estullbp0002 Keepalived_vrrp[42905]: (VI_1) Entering MASTER STATE

These show that the keepalived process is not being scheduled to run when it needs to. In the first case it was delayed from running by 8.7... seconds, and in the second case 5.2... seconds. With an advert_int of 1, and priority 100, then if the backup does not receive an advert for 3.6 seconds it will timeout and become master. When keepalived ran at 17:38:22, it was the first time it had been scheduled to run since at least 17:38:14, and so keepalived will not have received an advert for at least 8.7... seconds, so it times out and becomes master.

There are two possibilities here. Either keepalived is not being scheduled to run in the VM because there are too many higher priority processes running, or alternatively the VM is not being scheduled to run by the host system. Either way, keepalived cannot work reliably if it is not scheduled to run in a timely manner.

If the problem is that keepalived is not being scheduled to run in the VM, you can increase its priority by setting global_def vrrp_priority to a negative value, with -20 giving it the highest priority. Better still, set vrrp_rt_priority to a value between 1 and 99, with higher values being higher priorities.

On the other hand, if the problem is that the VM is not being scheduled to run by the host system, then that is something you will need to sort out.

@Adi-AA
Copy link
Author

Adi-AA commented Oct 15, 2021

Hi,
What do you mean by "keepalived is not being scheduled to run in the VM because there are too many higher priority processes running". If VM is not running then the traffic wont pass , but traffic is passing well. The status of keepalived is running. vrrp_rt_priority 50 (master node) and vrrp_rt_priority 49 (for backup node)? is this correct?

@pqarmitage
Copy link
Collaborator

@Adi-AA I did not say the VM was not running; what I said is that keepalived is not being scheduled to run. As the log extracts above show, keepalived on the first occasion did not run until over 8.7 seconds after it should have done, and this is not caused by keepalived but by other processes running, presumably with higher priority than keepalived.

The fact that the VM is passing traffic at least means that the VM is being scheduled to run by the host OS. However it does not imply that keepalived itself is being scheduled to run (it clearly isn't), since the traffic that is being passed is not processed by keepalived; keepalived sets up the network configuration and then the kernel handles the forwarding of the traffic.

It is probably sensible to set vrrp_rt_priority to the same value on both your VMs; you can set it to any value between 1 and 99, the higher the value the higher the priority.

@Adi-AA
Copy link
Author

Adi-AA commented Oct 15, 2021

Oh ok. so in this case should we checking what processes are scheduled at the kernel level?
And since we are setting vrrp_rt_priority to a higher valuse, does it mean this takes priority above most of the other processes at the kernel level?
Will keepalived restart fix it temporarily? or is there any command to check whether keepalived process is running or not?
Ok. so is the vrrp_rt_priority needs to be done in global_defs?
Is the below config correct?

#On VM0001
global_defs {
vrrp_version 3
vrrp_rt_priority 49
vrrp_garp_master_repeat 1
}

#On VM0002
global_defs {
vrrp_version 3
vrrp_rt_priority 49
vrrp_garp_master_repeat 1
}

@Adi-AA
Copy link
Author

Adi-AA commented Oct 15, 2021

@pqarmitage One more thing. By adjusting vrrp_rt_priority 49 or making vrrp_priority to -20 (in global_defs) disrupt any traffic?
Do we need to reload keepalived by doing this or just simply add and it will take effect?

@pqarmitage
Copy link
Collaborator

Setting vrrp_rt_priority should make the keepalived vrrp process have higher priority than most other processes, but it rather depends on what priorities the other processes have set.

If you want keepalived to adjust its own priority, then you would need to reload keepalived to make it reads the new configuration; this should have minimal impact on traffic. Alternatively, edit the keepalived configuration so that next time keepalived starts it gets the new configuration, but manually change the scheduling priority of keepalived using chrt -r -p 49 PID where PID is the pid of the vrrp process; so chrt -r -p 49 $(cat /run/vrrp.pid) should do it.

@Adi-AA
Copy link
Author

Adi-AA commented Oct 18, 2021

Sounds good.
What is good number to put for vrrp_rt_priority?
OR since -20 is highest priority, can we just do the following instead of vrrp_rt_priority?
Do you have a command to check what are the other processes priorities? and will the priority changes as the processes gets restarted?

#On VM0001
global_defs {
vrrp_version 3
vrrp_priority -20
vrrp_garp_master_repeat 1
}

#On VM0002
global_defs {
vrrp_version 3
vrrp_priority -20
vrrp_garp_master_repeat 1
}

@pqarmitage
Copy link
Collaborator

@Adi-AA There are two types of priority at play here. What you are referring to is the traditional nice priority, and indeed you can specify vrrp_priority -20. However there is also real-time scheduling, with priorities 1 to 99, and anything that is using real-time scheduling takes priority over any process that does not have realtime scheduling. Therefore, setting vrrp_rt_priority 50 will mean that the VRRP process has a higher priority the any process that is using the traditional nice scheduling. For more details, see the sched(7) man page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pqarmitage @Adi-AA