Raspberry aarch64 install becomes unresponsive on first boot

[Shikachuu]

Describe the bug

While I install the stable branch on the SD card as described in the documentation u-boot seciton durring the first boot I can clearly follow the boot process.
However once it is finished booting, the system drops (?) the HDMI signal, both the green and the red led will continously light up and the system becomes unresponsive, no matter how long I let it run. (The longest was like 5 hours running like this.)

After power-cycling the system, it boots but will throw an error regarding user creation in my ignition config: useradd: cannot lock /etc/group; try again later.
(Probably realted to #1250 ?)

Reproduction steps

1. Generate a fedora coreos 37 sd-card using the provided ignition config.

sudo podman run --pull=always --privileged --rm \
    -v /dev:/dev -v /run/udev:/run/udev -v .:/data -w /data \
    quay.io/coreos/coreos-installer:release \
    install -a aarch64 -s stable -i fcos-raspi.ign /dev/sdb

FCOSEFIPARTITION=$(lsblk /dev/sdb -J -oLABEL,PATH  | jq -r '.blockdevices[] | select(.label == "EFI-SYSTEM")'.path)

mkdir /tmp/FCOSEFIpart

sudo mount $FCOSEFIPARTITION /tmp/FCOSEFIpart

sudo rsync -avh --ignore-existing /tmp/piboot/boot/efi/ /tmp/FCOSEFIpart/

sudo umount $FCOSEFIPARTITION

2. Boot the system.

Expected behavior

The system boots correctly.

Actual behavior

Unexplainable freez during first boot, then error with user creation (useradd: cannot lock /etc/group; try again later.)

System details

• Bare Metal (aarch64 raspberry pi 4)

Butane or Ignition config

butane:

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ9B6zMB/ycG9cMfWtwIT/9mWaNGv7nu+h5nrHJ4drkl shika@pop-os

systemd:
  units:
    # Enable autologin (See https://docs.fedoraproject.org/en-US/fedora-coreos/tutorial-autologin/)
    - name: serial-getty@ttyS0.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure`
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM

storage:
  directories:
  - path: /opt/containers
    overwrite: true

  links:
    # Modify the timezone symlink to my region (See https://docs.fedoraproject.org/en-US/fedora-coreos/time-zone/)
    - path: /etc/localtime
      target: ../usr/share/zoneinfo/Europe/Budapest

  files:
    # Set the system's hostname (See https://docs.fedoraproject.org/en-US/fedora-coreos/hostname/)
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: raspi

    # Change the keymap to someting I can blindly type on (See https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-setting-keymap/)
    - path: /etc/vconsole.conf
      mode: 0644
      contents:
        inline: KEYMAP=hu

    # Set the systemd pager (See https://docs.fedoraproject.org/en-US/fedora-coreos/tutorial-autologin/)
    - path: /etc/profile.d/systemd-pager.sh
      mode: 0644
      contents:
        inline: |
          # Tell systemd to not use a pager when printing information
          export SYSTEMD_PAGER=cat

    # Add an update strategy for zincati with the most absurd time periods (See https://coreos.github.io/zincati/usage/updates-strategy/#periodic-strategy)
    - path: /etc/zincati/config.d/55-updates-strategy.toml
      contents:
        inline: |
          [updates]
          strategy = "periodic"
          [updates.periodic]
          time_zone = "Europe/Budapest"
          [[updates.periodic.window]]
          days = [ "Wed", "Sun" ]
          start_time = "02:30"
          length_minutes = 60

    # Create a container type systemd unit, sadly you can't use `system.units` for this. (See https://github.com/coreos/fedora-coreos-tracker/issues/998)
    - path: /etc/containers/systemd/blocky.kube
      contents:
        inline: |
          [Unit]
          Description=Blocky k8s deployment
          Before=local-fs.target

          [Kube]
          Yaml=/opt/containers/blocky-deployment.yaml

          [Install]
          # Start by default on boot
          WantedBy=multi-user.target default.target

    # Add the deployment k8s file
    - path: /opt/containers/blocky-deployment.yaml
      mode: 0644
      contents:
        inline: |
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: blocky-config
          data:
            config.yml: |
              upstream:
                default:
                  - tcp-tls:dot.libredns.gr:853
                  - https://doh.libredns.gr/dns-query
                  - https://fi.doh.dns.snopyta.org/dns-query
                  - tcp-tls:fi.dot.dns.snopyta.org
              blocking:
                blackLists:
                  ads:
                    - https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
                    - https://dbl.oisd.nl
                clientGroupsBlock:
                  default:
                    - ads
              customDNS:
                rewrite:
                  home: local
                  lan: local
                mapping:
                  openwrt.local: 192.168.1.1
                  picloud.local: 192.168.1.10
                  dns.local: 192.168.1.20
              ports:
                dns: 53
                http: 4000
              log:
                level: error
                format: json
                privacy: true
              prometheus:
                enable: true
                path: /metrics
          ---
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: blocky-deployment
            labels:
              app: blocky
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: blocky
            template:
              metadata:
                labels:
                  app: blocky
              spec:
                containers:
                - name: blocky
                  image: docker.io/spx01/blocky:v0.21
                  imagePullPolicy: IfNotPresent
                  resources:
                    limits:
                      cpu: 500m
                      memory: 500m
                  volumeMounts:
                    - name: blocky-config
                      mountPath: /app/config.yml
                  ports:
                  - containerPort: 4000
                  - containerPort: 53
                volumes:
                  - name: blocky-config
                    configMap:
                      name: blocky-config

ignition:

{
  "ignition": {
    "version": "3.3.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ9B6zMB/ycG9cMfWtwIT/9mWaNGv7nu+h5nrHJ4drkl shika@pop-os"
        ]
      }
    ]
  },
  "storage": {
    "directories": [
      {
        "overwrite": true,
        "path": "/opt/containers"
      }
    ],
    "files": [
      {
        "path": "/etc/hostname",
        "contents": {
          "compression": "",
          "source": "data:,raspi"
        },
        "mode": 420
      },
      {
        "path": "/etc/vconsole.conf",
        "contents": {
          "compression": "",
          "source": "data:,KEYMAP%3Dhu"
        },
        "mode": 420
      },
      {
        "path": "/etc/profile.d/systemd-pager.sh",
        "contents": {
          "compression": "",
          "source": "data:,%23%20Tell%20systemd%20to%20not%20use%20a%20pager%20when%20printing%20information%0Aexport%20SYSTEMD_PAGER%3Dcat%0A"
        },
        "mode": 420
      },
      {
        "path": "/etc/zincati/config.d/55-updates-strategy.toml",
        "contents": {
          "compression": "gzip",
          "source": "data:;base64,H4sIAAAAAAAC/2SNsQoCMRBE+/2KZWs5DwUL4RrBL7CwCCEEdzkDXhKSDcf59ZLiKtuZeW9My+xVqoWqxavMG05IWUpIHF4Eez/skQUNi7hvitKX91ZSluOtsc9SlcD8EcMaIqfVWmC/VZzQID2F6YD0aJGwX/uirnu7cjxdzyPBR+Ksb7eE2FQ6dhnhFwAA//9REs38rgAAAA=="
        }
      },
      {
        "path": "/etc/containers/systemd/blocky.kube",
        "contents": {
          "compression": "gzip",
          "source": "data:;base64,H4sIAAAAAAAC/0TOsWrEMAzG8V1PYeicZO3iJXQpHUspR8ggO8phIkvBlge//cFx4faP3/df/iTZCl9UY0mnJRU/s8aju+Ozuo1O1p5JDGbatZBnjcjDXkfDcicDWH5aoBVumNlPetoUVQyTUKlTeErDWxk7ZgZYvqUaMq/w4X4Ni7nQ3UY7Njan4oKqwT+K0TZ3nxtbGlql8jq9plfDIwAA///j65V0xAAAAA=="
        }
      },
      {
        "path": "/opt/containers/blocky-deployment.yaml",
        "contents": {
          "compression": "gzip",
          "source": "data:;base64,H4sIAAAAAAAC/3yU3W7bOgzH7/0UfAHbSc/p0OmyKzAMaIsABXavyEysRRI1ikoXYA8/OI6/0myXJH+k/qJI6Wi/IydLQcFxXRxsaBR8obCz+xcdC4+iGy1aFQBBe1SwdWQOp9KckWKI9WZ18k7B7wIAIMckjNqrswXQ4E5nJ4MJUIKYWIpLqiGpnN0yNiFVe1YP9//NqFYkJlXXDbVzqm5CKn9m5NMNdmerDu/IFCieRFfE+5spg4hzilynnMHzlW3YD9q3TpvDs02SptvoZmbMpbB+r/ZW2rzNCdlQEAxSGfL1m+ARw2NXrG4pSaq9ToLcGzeLNVtXkU1NFdwlbpzFIF+ZckyPnc5JxYeOd5V001c2OQn5p9e3Icz4zlZwolvqntuR0W70OR2WLq9jnHUGgCKGd5bqDClYf76r1p8eqnW1HpFojaPc3EBWk/aQPsbv+ngknjrfhKRgnJeuTwr+X6160tGozOERnQJkJr64dsRei4IficLFFdketTkpEM7YH8bkUVrM44kY9NbhDAGIWloFtUdha1JRlmUx3ysdY6rH5XrC6OjkMcjft6uZmK7pW3SX43WMA1SkiKbzMkZnjU4Kuh4ndGiEuOe9FtM+zwosSwAI+uj08O5zQf17zzOvcwEGCeeJoiDaBuSRLxeXGotYr/eooCFzQK4s1Sn+Wq3rnlLHVXW3XrKb7NyGnO0e5tvulWTDmPreDLObKLPBxQo6660sPAAmZgX3q5VfeD164tNV4Egue3yhHORqs2/9g4tyXc6mHwkdYz39jdMKzEe4qzk2b0Msswm+GR3Hvde4KPRvcWb42JdtuZX0JwAA//8iR0tBGQYAAA=="
        },
        "mode": 420
      }
    ],
    "links": [
      {
        "path": "/etc/localtime",
        "target": "../usr/share/zoneinfo/Europe/Budapest"
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "dropins": [
          {
            "contents": "[Service]\n# Override Execstart in main unit\nExecStart=\n# Add new Execstart with `-` prefix to ignore failure`\nExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM\n",
            "name": "autologin-core.conf"
          }
        ],
        "name": "serial-getty@ttyS0.service"
      }
    ]
  }
}

[dustymabe]

hey @Shikachuu - sorry you are having trouble. I've recently run through the pi4 install docs (using both install methods) and was successful.

I guess it's hard to know what's going on here without some logging. Here are a few options that might help us understand more:

1. Reduce your Ignition/Butane down to just a simple config that specifies an SSH key. This will help us eliminate any problems with the software you are running.
2. Record your Raspberry Pi4 as it boots and share it. There might be some clues in there as to what is going on.

[Shikachuu]

Hey @dustymabe, thanks for your response!

Just tried it with the followin ignition config:

{
  "ignition": {
    "version": "3.3.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ9B6zMB/ycG9cMfWtwIT/9mWaNGv7nu+h5nrHJ4drkl shika@pop-os"
        ]
      }
    ]
  }
}

as you recommended.

Still the same issue.

Can you recommend some tools to properly record the boot process?

[dustymabe]

Still the same issue.

Weird.. Can you make sure your EEPROM is fully up to date?

Can you recommend some tools to properly record the boot process?

Super low tech: just record it with a cell phone camera and post the video somewhere.

[Shikachuu]

Still the same issue.

Weird.. Can you make sure your EEPROM is fully up to date?

Can you recommend some tools to properly record the boot process?

Super low tech: just record it with a cell phone camera and post the video somewhere.

Just to make sure, I did a bootloader recovery with version 2023-01-11-vl805-000138c0, still the same problem with both configuration.

[Shikachuu]

Still the same issue.

Weird.. Can you make sure your EEPROM is fully up to date?

Can you recommend some tools to properly record the boot process?

Super low tech: just record it with a cell phone camera and post the video somewhere.

Also created a screen recording of the first boot on a new install using the simple ssh key only configuration.

[dustymabe]

Really odd. Any chance that the system is actually up and you can SSH into it? I'm guessing not since you said the LEDs are blinking continuously.

I really don't know what's going on here. I'm guessing you don't have a second Raspberry Pi4 to test on to sanity check it's not some hardware problem?

[jlebon]

Just a wild guess, but might be worth also making sure that the power supply is good. If you have another one, could be worth trying again with that one. I've had weird random reboots like that on the Pi in the past due to inadequate power.

[Shikachuu]

Really odd. Any chance that the system is actually up and you can SSH into it? I'm guessing not since you said the LEDs are blinking continuously.

I really don't know what's going on here. I'm guessing you don't have a second Raspberry Pi4 to test on to sanity check it's not some hardware problem?

According to my DHCP, it doesn't even have an IP address.
I'll try this on an another Pi just in case tho'.

[Shikachuu]

Just a wild guess, but might be worth also making sure that the power supply is good. If you have another one, could be worth trying again with that one. I've had weird random reboots like that on the Pi in the past due to inadequate power.

I'm using the offical adapter by the Raspberry team, but tried with a random charger too, same issue.

[dustymabe]

@Shikachuu were you able to debug this further?

[nielsenb-jf]

I believe I am seeing the same issue with "next" (Fedora 39). Solid red LED. Green LED flashes 7 times, stops, flashes 7 times again, repeats forever. Nothing over HDMI, cannot connect via SSH.

Config is nothing but an SSH key for the core user.

4GB Pi 4b. EEPROM version is pieeprom-2023-05-11.bin which I believe to be the latest.

"Regular" aarch64 Fedora boots fine.

[dustymabe]

I'm going to close this issue out since there wasn't any new information to work off of.

@nielsenb-jf - can you open a new issue with as much detail as possible? Please be sure to include if you are doing a fresh install or your system is upgrading.

[nielsenb-jf]

I'm going to close this issue out since there wasn't any new information to work off of.

@nielsenb-jf - can you open a new issue with as much detail as possible? Please be sure to include if you are doing a fresh install or your system is upgrading.

Done, see #1585