v1.3.5 segmentation fault

[osixia]

Hello,
not sure what's the error but i have some segmentation fault:

Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Unable to resolve default script username 'keepalived_script' - ignoring
Opening file '/etc/keepalived/keepalived.conf'.
Starting Healthcheck child process, pid=43
Starting VRRP child process, pid=44
Initializing ipvs
Opening file '/etc/keepalived/keepalived.conf'.
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Registering gratuitous ARP shared channel
Opening file '/etc/keepalived/keepalived.conf'.
------< Global definitions >------
 Router ID = osixia
VRRP_Instance(VI_1) removing protocol VIPs.
 Default interface = eth0
 LVS flush = false
 VRRP IPv4 mcast group = 224.0.0.18
 VRRP IPv6 mcast group = ff02::12
 Gratuitous ARP delay = 5
 Gratuitous ARP repeat = 5
 Gratuitous ARP refresh timer = 0
 Gratuitous ARP refresh repeat = 1
 Gratuitous ARP lower priority delay = 4294
 Gratuitous ARP lower priority repeat = -1
 Send advert after receive lower priority advert = true
 Send advert after receive higher priority advert = false
 Gratuitous ARP interval = 0
 Gratuitous NA interval = 0
 VRRP default protocol version = 2
 Iptables input chain = INPUT
 Using ipsets = true
 ipset IPv4 address set = keepalived
 ipset IPv6 address set = keepalived6
 ipset IPv6 address,iface set = keepalived_if6
 VRRP check unicast_src = false
 VRRP skip check advert addresses = false
 VRRP strict mode = false
 VRRP process priority = 0
 VRRP don't swap = false
 Checker process priority = 0
 Checker don't swap = false
 Network namespace = (default)
 Script security disabled
 Default script uid:gid 0:0
------< SSL definitions >------
 Using autogen SSL context
Keepalived_vrrp exited due to segmentation fault (SIGSEGV).
  Please report a bug at https://github.com/acassen/keepalived/issues
  and include this log from when keepalived started, what happened
  immediately before the crash, and your configuration file.
VRRP child process(44) died: Respawning
Starting VRRP child process, pid=46
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Registering gratuitous ARP shared channel
Opening file '/etc/keepalived/keepalived.conf'.
VRRP_Instance(VI_1) removing protocol VIPs.
Keepalived_vrrp exited due to segmentation fault (SIGSEGV).
  Please report a bug at https://github.com/acassen/keepalived/issues
  and include this log from when keepalived started, what happened
  immediately before the crash, and your configuration file.
[...]

build config:

Keepalived configuration
------------------------
Keepalived version       : 1.3.5
Compiler                 : gcc
Preprocessor flags       : -I/lib/modules/4.8.0-42-generic/build/include -I/usr/include/libnl3     
Compiler flags           : -Wall -Wunused -Wstrict-prototypes -Wextra -g -O2      
Linker flags             : 
Extra Lib                : -lcrypto -lssl -lnl-genl-3 -lnl-route-3 -lnl-3 -lip4tc -lip6tc -lxtables -ldl
Use IPVS Framework       : Yes
IPVS use libnl           : Yes
IPVS syncd attributes    : No
IPVS 64 bit stats        : No
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication  : Yes
With ip rules/routes     : Yes
SNMP vrrp support        : No
SNMP checker support     : No
SNMP RFCv2 support       : No
SNMP RFCv3 support       : No
DBUS support             : No
SHA1 support             : No
Use Debug flags          : No
Stacktrace support       : No
Memory alloc check       : No
libnl version            : 3
Use IPv4 devconf         : Yes
Use libiptc              : Yes
Use libipset             : Yes
init type                : undetected
Build genhash            : Yes
Build documentation      : No

Keepalived.conf :

vrrp_instance VI_1 {
  interface wlp1s0

  track_interface {
    wlp1s0
  }

  state BACKUP
  virtual_router_id 51
  priority 150
  nopreempt

  unicast_peer {
    192.168.1.10
    192.168.1.11
  }

  virtual_ipaddress {
    192.168.1.231
    192.168.1.232
  }

  authentication {
    auth_type PASS
    auth_pass d0cker
  }

  notify "/container/service/keepalived/assets/notify.sh"
}

does this come from a missing library with the build ?
Thanks

[pqarmitage]

At the moment I'm not sure what the problem is, and it doesn't segfault for me.

Can you provide the command line options you are running keepalived with. Also, what flavour of Linux are you running. I might be able to reproduce the problem if I can install the same distro as you are using into a VM).

Is a core file being generated when keepalived segfaults? If not, can you run keepalived with the -m option, (you may also need to specify -M depending on you system configuration).

Once you have a coredump can you generate a stack trace so we can see where the problem is occurring.

[pqarmitage]

I've noticed something slightly strange in the configure output. You appear to have Linux 4.8.0 headers, but the output shows

IPVS syncd attributes    : No
IPVS 64 bit stats        : No

and they have been included in Linux since versions 4.3 and 4.1 respectively.

[osixia]

I run keepalived in a docker container from an debian jessie image, here is the docker file that compile keepalived
https://github.com/osixia/docker-keepalived/blob/stable/image/Dockerfile

before 1.3.5 eveything was ok :S

the command run is:
/usr/local/sbin/keepalived -f /etc/keepalived/keepalived.conf --dont-fork --log-console --log-detail --dump-conf

i don't know how to get a coredump or generate a stack trace but i will try as soon as i can :)

if you have docker installed to reproduce the problem:

git clone https://github.com/osixia/docker-keepalived.git
cd docker-keepalived
make build
docker run --name keepalived --cap-add=NET_ADMIN --net=host --env KEEPALIVED_INTERFACE=eth0 -d osixia/keepalived:1.3.5 

to get the logs:

docker logs keepalived 2>&1 | less

to open a shell in the container:

docker exec -it keepalived bash

remove the container:

docker rm -f keepalived

I'll try to give you more details

[pqarmitage]

I did manage to build a docker image and I can reproduce this problem using docker, and the stack trace I get is:
#0 0x0000000000000000 in ?? ()
#1 0x000000000042b21c in remove_ipsets () at vrrp_ipset.c:244
#2 0x000000000041db49 in vrrp_complete_init () at vrrp.c:2458
#3 0x0000000000414652 in start_vrrp () at vrrp_daemon.c:264
#4 0x0000000000414a28 in start_vrrp_child () at vrrp_daemon.c:537
#5 0x0000000000414b06 in vrrp_respawn_thread (thread=) at vrrp_daemon.c:452
#6 0x000000000042fc4d in thread_call (thread=0x7ffde1b15d00) at scheduler.c:846
#7 launch_scheduler () at scheduler.c:871
#8 0x0000000000405999 in keepalived_main (argc=8, argv=0x7ffde1b16158) at main.c:976
#9 0x00007fbb8f0dfb45 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x000000000040471e in _start ()

This looks like a problem with linking to libipset at runtime. Can you try building keepalived with the --disable-dynamic-linking configure option and see if that makes a difference.

At the moment I can't build a docker image, since I keep getting error messages like

Err http://deb.debian.org jessie Release.gpg
  Could not resolve 'deb.debian.org'

even though the system on which docker is running can resolve them. Do you have any suggestions how to resolve this issue?

[pqarmitage]

It seems that the image build problems were due to the system running docker had lost its IPv4 address, but still had its IPv6 address.

[pqarmitage]

Using the --disable-dynamic-linking configure option resolves the segfault for me. I'll have to look later to find the underlying cause of the problem.

[osixia]

it solved the issue :)

thanks for your time and this great software !

[pqarmitage]

There is still a problem with the segfault when not configured with --disable-dynamic-linking, so reopening this issue.

[pqarmitage]

@osixia How do I modify the Dockerfile to use a keepalived.tar.gz from the host system rather than downloading it from keepalived.org?

[osixia]

The tar.gz can be added in the Dockerfile file directory or in any subdirectory of this directory.
After that we just need to tell docker to add the file to the image we are building by adding in the Dockerfile a COPY instruction.

doc: https://docs.docker.com/engine/reference/builder/#copy

For example in osixia/docker-keepalived we would do something like this:
-> add keepalived.tar.gz in the image directory:

+ image
     + environment
     + service
    - Dockerfile
    - keepalived.tar.gz
+ test
+ [...]

-> Modify the Dockerfile by adding before the first RUN instruction:
COPY keepalived.tar.gz /path/in/the/image

-> adjust the RUN instruction to use /path/in/the/image/keepalived.tar.gz

[pqarmitage]

Update: The underlying cause of this problem is that a kernel module cannot be loaded from within a docker container. After the attempt to load the module has failed keepalived then doesn't handle it properly. I have produced some patches that I am still testing to ensure that keepalived handles the problem as best it can.

It seems to me though that there is an underlying problem that keepalived cannot operate properly if the kernel modules aren't loaded, but from within the container it has no way of loading them. Is there a way for the container to be configured so that it loads the ip_tables, ip6_tables and xt_set modules before the code in the container is executed?

[pqarmitage]

Commits f7cd991 and dac727e resolve this for me. As referred to above, the problem was around keepalived when running in a container not being able to load a kernel module, and then not handling the error properly.

Is there a way to tell the container that the ip_tables, ip6_tables, xt_set and ip_vs modules need to be loaded, so that it loads them before keepalived is run?

[pqarmitage]

I note in the Dockerfile that iptables-dev, libipset-dev, libnl-3-dev, libnl-genl-3-dev, libnl-route-3-dev, libnfnetlink-dev and libssl-dev are all left installed in the container. Since these packages include the header files, that seems somewhat unnecessary.
If the packages iptables, libipset3, libnl-3-200, libnl-genl-3-200, libnl-route-3-200, libnfnetlink0 and libssl1.0.0 were specified to be installed as well, then the dev packages could be removed. My understanding is that these are the packages that contain the runtime libraries, and they are installed anyway as dependencies of the -dev packages.

[osixia]

Sorry for the late reply,

It seems that a container can load a module itself, but it must be run in privileged mode and mount /lib/modules host directory.
https://dummdida.tumblr.com/post/117157045170/modprobe-in-a-docker-container

That give to the container a lot of capabilities.
Here we prefer loading modules at the node boot and then run the container.

thanks, i will remove unnecessary packages in Dockerfile for the next release :)

[osixia]

Would you be interested to provide an official docker image of keepalived ?

I can help to add a Dockerfile in this project and setting up a docker hub account with an automated build so when you create a tag on this github repository the corresponding image will be build and made available on the docker hub.

[acassen]

sure it would be certainly valuable for lot of users. Please make it happen.

[pqarmitage]

@osixia your Dockerfile is now merged - many thanks for providing this.

Can you now set up the docker hub account with the automated build. Where will the docker image be created, and what is the right way to install and run it (I think this information should be added to the website).