Nomad hard links prevent cron from running

[jshaw86]

Nomad version

Output from nomad version
nomad 0.4.0

Operating system and Environment details

ubuntu 14.04

Issue

Nomad prevents cron from running after creating chroot.

Reproduction steps

1. create a chroot (via exec)
2. run a cron
3. observe that cron does not run

Nomad Client logs (if appropriate)

From syslog:

Jul 29 23:03:01 nomad-agent1 cron[1043]: (*system*) NUMBER OF HARD LINKS > 1 (/etc/crontab)
Jul 29 23:03:01 nomad-agent1 cron[1043]: (*system*ansible-pull) NUMBER OF HARD LINKS > 1 (/etc/cron.d/ansible-pull)

[dadgar]

Are you trying to set up a cron inside the task's chroot? There is no cron daemon running so that will not work

[jshaw86]

@dadgar no just in the "global" root or whatever you want to call it

[tantra35]

This can be due hardlinking of /etc/crontab and /etc/cron.d/* which nomad do when make chroot enviroment, for those files nomad must make copy but not hardlink, and issue will be resolved

[mac-abdon]

Here is the CVE: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-1856

[nferch]

Any updates or recent experiences with this one? It is possible to work around the issue by specifying chroot_env but it's rather cumbersome and error prone as you need to guess and include what bits of /etc need to be included. Perhaps there could be something like a chroot_env_exclude but that seems a bit kludgy as well.

[MikeN123]

Some additional information/verification. I think this is at least an issue on all Debian-based distros. One of the maintainers for the Debian cron package here mentions the check for hard links cannot be disabled: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=647193

So Nomad needs to exclude these files somehow. The user can do that by setting the chroot_env, but that's quite cumbersome and I also think a vanilla Nomad install probably does not want to break the host cron. Seems to make sense to have a list of files to exclude from chroot_env, and setting the exclude to /etc/crontab and /etc/cron.d by default.