atomic-desktop-gnome ships a corrupted sqlite rpmdb in about half of the builds, breaking dnf in derived images

[lidolee]

Summary

The sqlite rpm database shipped in quay.io/almalinuxorg/atomic-desktop-gnome is corrupted in roughly half of the published builds. Package headers survive, so rpm -qa looks healthy, but the indexes are damaged: any provide lookup fails and dnf install therefore aborts on any derived image, claiming an installed dependency is missing.

The same source tree produces both healthy and corrupt images, so this looks like a non-deterministic problem in the build pipeline rather than a change in this repository.

Impact

Deriving from these images is not possible. A minimal Containerfile such as

FROM quay.io/almalinuxorg/atomic-desktop-gnome:10
RUN dnf -y install subscription-manager

fails with

error: SELECT hnum, idx FROM 'Providename' WHERE key=?: 11: database disk image is malformed
Error: transaction check vs depsolve:
python3-gobject-base is needed by subscription-manager-1.30.12-1.el10.alma.1.x86_64

python3-gobject-base-3.46.0-7.el10.x86_64 is installed — rpm -q finds it. Only the index lookup fails.

Evidence

PRAGMA integrity_check on /usr/share/rpm/rpmdb.sqlite, one build per row:

Tag	Digest	rpmdb
10.2.20260902.0	sha256:6bd0dd952a9bdcd5edc1d5d36c743fbc4bace6b4190fed12ce0096210c31e587	corrupt
10.2.20260829.0	sha256:4b9de11213729b0fbf0827f7e14204884984ca690709804ed70b5189830421fc	corrupt
10.2.20260808.3	sha256:40b28642ede04b12aa63d406865c538c1e59c2e483d9c013c93ea96620d74b38	corrupt
10.2.20260808.2	sha256:7cd7029173aad616be25ce760011d1e2d8bedc8c7e05b039f88a55105012c948	ok
10.2.20260808.1	sha256:d6d4b067acd7362063c8576aa2b23fd276bd393deca5df00d3e3bd8d1482c689	corrupt
10.2.20260808.0	sha256:71c342533fed0d7a3f9a12220365d53034be4fcbb859fdaf2e93c057103619ce	ok

The four 20260808.* builds come from the same commit — the most recent commit before them is 70ff4f69 (2026-07-21) — and alternate between healthy and corrupt. That is what makes this look like a race or an environment-dependent step in the pipeline rather than a source change.

quay.io/almalinuxorg/almalinux-bootc:10 is not affected; its rpmdb passes integrity_check.

Excerpt of the failure on 10@sha256:6bd0dd95:

*** in database main ***
Tree 31 page 9757: btreeInitPage() returns error code 11
Tree 30 page 3603 cell 146: Rowid 458 out of order
Tree 26 page 10087: btreeInitPage() returns error code 11
...
wrong # of entries in index Sha1header_key_idx
wrong # of entries in index Dirnames_hnum_idx
wrong # of entries in index Dirnames_key_idx
wrong # of entries in index Basenames_hnum_idx

rpm -Va --nofiles --nodigest on the same image reports phantom unsatisfied dependencies, e.g.:

Unsatisfied dependencies for p11-kit-0.26.4-1.el10_2.x86_64:
	libtasn1.so.6()(64bit) is needed by (installed) p11-kit-0.26.4-1.el10_2.x86_64

The image also ships a zero-length rpmdb.sqlite-wal next to rpmdb.sqlite. Removing it does not change anything, so it is probably a symptom rather than the cause — but shipping WAL/SHM sidecars in an image layer is worth looking at in its own right.

Reproducer

podman run --rm quay.io/almalinuxorg/atomic-desktop-gnome:10@sha256:6bd0dd952a9bdcd5edc1d5d36c743fbc4bace6b4190fed12ce0096210c31e587 \
  bash -c 'rpm -qa | wc -l;
           rpm -q python3-gobject-base;
           rpm -q --whatprovides python3-gobject-base;
           python3 -c "import sqlite3; print(sqlite3.connect(\"/usr/share/rpm/rpmdb.sqlite\").execute(\"PRAGMA integrity_check\").fetchone()[0][:200])"'

Expected on an affected build: 1331 packages listed, rpm -q succeeds, --whatprovides fails, integrity_check is not ok.

Note that rpm -q --whatprovides rpm and rpm -q --whatprovides /usr/bin/rpm both still succeed on an affected image, so they are not usable as a probe — PRAGMA integrity_check is.

Workaround for downstreams

Rebuilding the indexes from the intact headers restores the database (integrity_check returns ok, the package count is unchanged, rpm -Va becomes clean, dnf install works). rpm --rebuilddb cannot swap the result in by itself here, so the swap has to be done manually:

RUN rm -f /usr/share/rpm/rpmdb.sqlite-shm /usr/share/rpm/rpmdb.sqlite-wal; \
    rpm --rebuilddb || true; \
    new="$(ls -d /usr/share/rpmrebuilddb.* | head -1)"; \
    rm -rf /usr/share/rpm && mv "$new" /usr/share/rpm

Environment

Reproduced with podman 5.8.2 on Fedora 44, x86_64, against the digests listed above.