systemd-based containers not initializing properly

[antifob]

Required information

• Distribution: Debian
• Distribution version: sid
• The output of "lxc info":

config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICBDCCAYmgAwIBAgIQTMvgJLVhsjmSl2vLoD3gbTAKBggqhkjOPQQDAzA0MRww
    GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGRlYmlh
    bjAeFw0xOTA2MDkwMDA4MzFaFw0yOTA2MDYwMDA4MzFaMDQxHDAaBgNVBAoTE2xp
    bnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZGViaWFuMHYwEAYHKoZI
    zj0CAQYFK4EEACIDYgAESnPzSGOEMnUrxVubs9wIAn5y/AV7q/cHjGE5zeHrNQnJ
    LO1/i+vng5c4n9ova6gUdTJ65pVph83MNSW+tsPgjb523iObrgI8tyDqjmq/wC2l
    8TO5JY1AZL9I0p1MQmM5o2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI
    KwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZkZWJpYW6HBAqLz0yH
    EP1CYZqI45aaUFQA//4SNFYwCgYIKoZIzj0EAwMDaQAwZgIxANp9T7Ld16NL7KFV
    abv5fpX5H1UCkjuoFT5w/JscfObUpsCFreUxHPgYRDJ4fs6O2QIxAKaRJZSMihXs
    BOrYGRgJoCzCQIvuTntSFcOiId1Uw5T8eTO9rhXW9VOCf/ni3WZWzA==
    -----END CERTIFICATE-----
  certificate_fingerprint: e2a6aba1a30c8d89f3acff9feb85c80108a862d54a8a7c6a8c9ebcd8e972d7ac
  driver: lxc
  driver_version: 3.0.3
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "false"
    seccomp_listener: "false"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 4.19.0-5-amd64
  lxc_features:
    mount_injection_file: "false"
    network_gateway_device_route: "false"
    network_ipvlan: "false"
    network_l2proxy: "false"
    seccomp_notify: "false"
  project: default
  server: lxd
  server_clustered: false
  server_name: debian
  server_pid: 28560
  server_version: "3.13"
  storage: dir
  storage_version: "1"

Additional host information

LXD is built from source. The init system of the host is runit (runit-init package). cgroupfs-mount and lxcfs are installed and active/running. apparmor is installed and activated.

I can provide a minimal bootable virtual disk image if it can help to debug the issue.

Issue description

systemd-based containers fail to start services. lxc launch ubuntu:18.04 c1, e.g., followed by lxc ls indicates the container failed to acquire a network address. Also, lxc exec c1 -- ps auxf indicates that only /sbin/init is running. ptrace-ing pid 1 shows that container entered its signals-waiting loop. Additionally, systemctl fails, stating that systemd is not pid 1 despite /sbin/init being provided by the systemd-sysv package.

Steps to reproduce

1. Build LXD from source

Follow the build instruction on https://lxd.readthedocs.io/

2. Install additional dependencies

• apparmor, apparmor-utils, apparmor-profiles-extra (update GRUB's configuration accordingly)
• cgroupfs-mount
• lxcfs

Make sure the last two services are activated/running.

3. Initialize LXD

lxd init

4. Start a container

lxc launch ubuntu:18.04 c1

Debugging steps taken so far

• I tried strace-ing /sbin/init in the container, but the program seems to have anti-debugging "features". It simply stops trying to do ptrace(PTRACE_ME). I replaced /sbin/init with a script that logs some info before running exec systemd --user.
• lxd --debug and lxc --debug indicates no relevant messages apart from old devpts not being unmounted properly (identical to [1]).
• images:alpine/3.9 (OpenRC) and ubuntu:trusty (upstart) are starting properly. ubuntu:{xenial,bionic,disco} all fail to start properly.
• On the same machine, but with Ubuntu bionic, the containers work properly.

Additional information

• In the failing containers, it is possible to get a network address by running dhclient directly and services can be started using the init scripts.

1: https://discuss.linuxcontainers.org/t/warning-in-the-container-log/4072

[stgraber]

Unlikely to be a LXD bug, far more likely to be systemd crashing on missing cgroup configuration or kernel feature. Closing the bug but we can still keep chatting here to track down the issue with your setup.

[stgraber]

What does lxc console --show-log c1 get you?
Also what's the process list in your container? lxc exec c1 -- ps fauxww

[stgraber]

When systemd itself is unhappy about something, you usually end up with a container which only has PID1 running and normally you get error output in the console log.

[antifob]

Awesome, thanks a lot. I didn't know there is a console log.

ps fauxww

# lxc exec c1 -- ps fauxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        30  0.0  1.3  37792  3368 ?        R+   15:15   0:00 ps fauxww
root         1  0.0  2.2  76416  5540 ?        Ss   15:14   0:00 systemd --system

console

# lxc console --show-log c1
Console log:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

Solution

By, [1] and [2], the solution is to create a systemd cgroup. Restarting the container after creating the cgroup makes everything work. 👍

mkdir /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd
lxc restart --force c1

1: https://wiki.debian.org/LXC#Incompatibility_with_systemd
2: debops/ansible-lxc#15 (comment)