Error Failed to run rpmdb --importdb: bwrap(rpmdb) VROOM-16788

[BeeGrech]

command:
# rpm-ostree -A --idempotent --allow-inactive -y install $PKG_NAME

logged result:

use_pty:FALSE rstrnt-package install util-linux-ng
Inactive requests:
  python3 (already provided by python3-3.9.18-1.el9.x86_64)
  chrony (already provided by chrony-4.3-1.el9.x86_64)
  util-linux (already provided by util-linux-2.37.4-15.el9.x86_64)
  tar (already provided by tar-2:1.34-6.el9.x86_64)
  git (already provided by git-2.39.3-1.el9.x86_64)
  audit (already provided by audit-3.1.2-2.el9.x86_64)
  beakerlib (already provided by beakerlib-1.29.3-1.el9.noarch)
  virt-what (already provided by virt-what-1.25-5.el9.x86_64)
  bc (already provided by bc-1.07.1-14.el9.x86_64)
  curl (already provided by curl-7.76.1-28.el9.x86_64)
  wget (already provided by wget-1.21.1-7.el9.x86_64)
  util-linux-ng (already provided by util-linux-2.37.4-15.el9.x86_64)
  yum-utils (already provided by yum-utils-4.3.0-12.el9.noarch)
Checking out tree 88023c6...done
Enabled rpm-md repositories: centos-automotive extras-common kernel-cki beaker-harness baseos appstream autosd
Importing rpm-md...done
rpm-md repo 'centos-automotive' (cached); generated: 2024-01-03T22:59:02Z solvables: 92
rpm-md repo 'extras-common' (cached); generated: 2023-10-24T11:42:58Z solvables: 72
rpm-md repo 'kernel-cki' (cached); generated: 2024-01-10T03:41:57Z solvables: 13
rpm-md repo 'beaker-harness' (cached); generated: 2023-05-17T19:37:41Z solvables: 83
rpm-md repo 'baseos' (cached); generated: 2024-01-09T11:58:18Z solvables: 4300
rpm-md repo 'appstream' (cached); generated: 2024-01-09T12:00:25Z solvables: 15949
rpm-md repo 'autosd' (cached); generated: 2024-01-10T02:18:39Z solvables: 6612
Resolving dependencies...done
Applying 4 overrides and 107 overlays
Processing packages...done
Running pre scripts...done
Running post scripts...done
Running posttrans scripts...done
Writing rpmdb...done
error: Writing rpmdb: Rewriting rpmdb for target native format: Failed to run rpmdb --importdb: bwrap(rpmdb): Child process killed by signal 22

rpm-ostree version (observed on both):

rpm-ostree-2024.1-2.el9
rpm-ostree-2023.12-1.el9

Additional information:

Occurring in test automation pipeline, where package install phase is in setup stage for the test runner. There were a number of other dependencies installed successfully prior to the error.

[cgwalters]

Signal 22 is SIGTTOU which is really weird to see here...my best guess here is that rpm is trying to do an interactive prompt in this scenario for some reason. An even wilder guess is this somehow relates to GPG keys or something?

Can you try to describe a more minimal reproducer? Does this occur every time? Have you ever reproduced this manually?

[BeeGrech]

I have not been able to reproduce manually.

In our pipeline each test sets up its own dependencies, at some point in that setup phase for a test case this is triggered. In some cases there was even normal behavior within a test case then this seams to get triggered.

once its triggered, its consistent and stays. No more packages can be requested. Also noticed the signal changes. I see 20, 24, 18

Maybe its a case where many calls for apply-live causes an issue?

I can run more manual testing to try and match this behavior

[BeeGrech]

  python3 (already provided by python3-3.9.18-1.el9.x86_64)
  chrony (already provided by chrony-4.3-1.el9.x86_64)
  util-linux (already provided by util-linux-2.37.4-15.el9.x86_64)
  tar (already provided by tar-2:1.34-6.el9.x86_64)
  git (already provided by git-2.39.3-1.el9.x86_64)
  audit (already provided by audit-3.1.2-2.el9.x86_64)
  beakerlib (already provided by beakerlib-1.29.3-1.el9.noarch)
  time (already provided by time-1.9-18.el9.x86_64)
  bc (already provided by bc-1.07.1-14.el9.x86_64)
  curl (already provided by curl-7.76.1-28.el9.x86_64)
  wget (already provided by wget-1.21.1-7.el9.x86_64)
  virt-what (already provided by virt-what-1.25-5.el9.x86_64)
Checking out tree 88023c6...done
Enabled rpm-md repositories: centos-automotive extras-common kernel-cki beaker-harness baseos appstream autosd
Importing rpm-md...done
rpm-md repo 'centos-automotive' (cached); generated: 2024-01-03T22:59:02Z solvables: 92
rpm-md repo 'extras-common' (cached); generated: 2023-10-24T11:42:58Z solvables: 72
rpm-md repo 'kernel-cki' (cached); generated: 2024-01-10T15:40:42Z solvables: 13
rpm-md repo 'beaker-harness' (cached); generated: 2023-05-17T19:37:41Z solvables: 83
rpm-md repo 'baseos' (cached); generated: 2024-01-09T11:58:18Z solvables: 4300
rpm-md repo 'appstream' (cached); generated: 2024-01-09T12:00:25Z solvables: 15949
rpm-md repo 'autosd' (cached); generated: 2024-01-10T02:18:39Z solvables: 6612
Resolving dependencies...done
Applying 4 overrides and 107 overlays
Processing packages...done
Running pre scripts...done
Running post scripts...done
Running posttrans scripts...done
Writing rpmdb...done
error: Writing rpmdb: Rewriting rpmdb for target native format: Failed to run rpmdb --importdb: bwrap(rpmdb): Child process killed by signal 24
Inactive requests:
  yum-utils (already provided by yum-utils-4.3.0-12.el9.noarch)
  python3 (already provided by python3-3.9.18-1.el9.x86_64)
  chrony (already provided by chrony-4.3-1.el9.x86_64)
  util-linux (already provided by util-linux-2.37.4-15.el9.x86_64)
  tar (already provided by tar-2:1.34-6.el9.x86_64)
  git (already provided by git-2.39.3-1.el9.x86_64)
  audit (already provided by audit-3.1.2-2.el9.x86_64)
  beakerlib (already provided by beakerlib-1.29.3-1.el9.noarch)
  time (already provided by time-1.9-18.el9.x86_64)
  bc (already provided by bc-1.07.1-14.el9.x86_64)
  curl (already provided by curl-7.76.1-28.el9.x86_64)
  wget (already provided by wget-1.21.1-7.el9.x86_64)
  virt-what (already provided by virt-what-1.25-5.el9.x86_64)
Checking out tree 88023c6...done
Enabled rpm-md repositories: centos-automotive extras-common kernel-cki beaker-harness baseos appstream autosd
Importing rpm-md...done
rpm-md repo 'centos-automotive' (cached); generated: 2024-01-03T22:59:02Z solvables: 92
rpm-md repo 'extras-common' (cached); generated: 2023-10-24T11:42:58Z solvables: 72
rpm-md repo 'kernel-cki' (cached); generated: 2024-01-10T15:40:42Z solvables: 13
rpm-md repo 'beaker-harness' (cached); generated: 2023-05-17T19:37:41Z solvables: 83
rpm-md repo 'baseos' (cached); generated: 2024-01-09T11:58:18Z solvables: 4300
rpm-md repo 'appstream' (cached); generated: 2024-01-09T12:00:25Z solvables: 15949
rpm-md repo 'autosd' (cached); generated: 2024-01-10T02:18:39Z solvables: 6612
Resolving dependencies...done
Applying 4 overrides and 107 overlays
Processing packages...done
Running pre scripts...done
Running post scripts...done
Running posttrans scripts...done
Writing rpmdb...done
error: Writing rpmdb: Rewriting rpmdb for target native format: Failed to run rpmdb --importdb: bwrap(rpmdb): Child process killed by signal 18
** ERROR: Command returned non-zero 256
** Completed Task : 7```

[jlebon]

The changing signal is definitely odd and smells a bit like UB/corruption of some kind. Worth checking the journal also.

[BeeGrech]

finally got a successful log capture - think i got it Full Disk:

Jan 12 19:27:39 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: Regenerating rpmdb for target
Jan 12 19:27:45 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: INSERT OR REPLACE INTO 'Packages' VALUES(?, ?): 13: database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'Packages': no such savepoint: Packages
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': no such savepoint: rwlock
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: INSERT OR REPLACE INTO 'Packages' VALUES(?, ?): 13: database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'Packages': no such savepoint: Packages
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': no such savepoint: rwlock
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: INSERT OR REPLACE INTO 'Packages' VALUES(?, ?): 13: database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'Packages': no such savepoint: Packages
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': no such savepoint: rwlock
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: INSERT OR REPLACE INTO 'Packages' VALUES(?, ?): 13: database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'Packages': no such savepoint: Packages
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': no such savepoint: rwlock
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: INSERT OR REPLACE INTO 'Packages' VALUES(?, ?): 13: database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'Packages': no such savepoint: Packages
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': no such savepoint: rwlock
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: INSERT OR REPLACE INTO 'Packages' VALUES(?, ?): 13: database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'Packages': no such savepoint: Packages
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': no such savepoint: rwlock
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: RELEASE 'rwlock': database or disk is full
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: PRAGMA synchronous = FULL: Safety level may not be changed inside a transaction
Jan 12 19:27:46 ip.us-east-1.aws.redhat.com rpm-ostree[360810]: error: sqlite failure: PRAGMA wal_checkpoint = TRUNCATE: database table is locked
Jan 12 19:27:47 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: Txn UpdateDeployment on /org/projectatomic/rpmostree1/centos failed: Writing rpmdb: Rewriting rpmdb for target native format: Failed to run rpmdb --importdb: bwrap(rpmdb): Child process killed by signal 12
Jan 12 19:27:47 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: Unlocked sysroot
Jan 12 19:27:47 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: Process [pid: 360270 uid: 0 unit: session-2.scope] disconnected from transaction progress
Jan 12 19:27:47 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: client(id:cli dbus:1.155 unit:session-2.scope uid:0) vanished; remaining=0
Jan 12 19:27:47 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: In idle state; will auto-exit in 63 seconds
Jan 12 19:27:48 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: client(id:cli dbus:1.156 unit:session-2.scope uid:0) added; new total=1
Jan 12 19:27:48 ip.us-east-1.aws.redhat.com rpm-ostree[360216]: Loaded sysroot

[jlebon]

Is the VM disk actually full or is something else going on with the sqlite database?

[BeeGrech]

Separately we are facing an issue with resizing the disk. We usually receive the failure message for less than 3% left on device
but i think this confirms

temd-journald[333200]: Failed to open system journal: No space left on device