prometheus-blackbox fails with Podman
Dies immediately with no logs.
Output with --log-level=debug:
root@host3:/home/vagrant# podman run --log-level debug --env-host --rm --cgroup-manager=cgroupfs --cgroup-parent /system.slice/docker-prometheus-blackbox --name prometheus-blackbox --network=host --expose=9115 --mount=type=tmpfs,destination=/tmp,tmpfs-mode=01777,tmpfs-size=64M --mount=type=tmpfs,destination=/run/lock,tmpfs-mode=01777,tmpfs-size=64M --mount=type=bind,source=/dev/log,destination=/dev/log --mount=type=bind,source=/etc/credentials/x509/prometheus,destination=/etc/credentials/x509/prometheus --mount=type=bind,source=/etc/prometheus,destination=/etc/prometheus --mount=type=bind,source=/etc/tracing,destination=/etc/tracing --user=1006:1006 --group-add=990 registry.git.autistici.org/ai3/docker/prometheus-blackbox:master --config.file /etc/prometheus/blackbox.yml
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is not being used
DEBU[0000] cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] using runtime "/usr/lib/cri-o-runc/sbin/runc"
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.git.autistici.org/ai3/docker/prometheus-blackbox:master"
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@338024506d41730ded658850ff9cc41338d6ea23969045e9bec58236c6c57c73"
DEBU[0000] exporting opaque data as blob "sha256:338024506d41730ded658850ff9cc41338d6ea23969045e9bec58236c6c57c73"
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@338024506d41730ded658850ff9cc41338d6ea23969045e9bec58236c6c57c73"
DEBU[0000] exporting opaque data as blob "sha256:338024506d41730ded658850ff9cc41338d6ea23969045e9bec58236c6c57c73"
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@338024506d41730ded658850ff9cc41338d6ea23969045e9bec58236c6c57c73"
DEBU[0000] Using host netmode
DEBU[0000] Adding mount /proc
DEBU[0000] Adding mount /dev
DEBU[0000] Adding mount /dev/pts
DEBU[0000] Adding mount /dev/mqueue
DEBU[0000] Adding mount /sys
DEBU[0000] Adding mount /sys/fs/cgroup
DEBU[0000] setting container name prometheus-blackbox
DEBU[0000] created OCI spec and options for new container
DEBU[0000] Allocated lock 11 for container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@338024506d41730ded658850ff9cc41338d6ea23969045e9bec58236c6c57c73"
DEBU[0000] exporting opaque data as blob "sha256:338024506d41730ded658850ff9cc41338d6ea23969045e9bec58236c6c57c73"
DEBU[0000] created container "04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864"
DEBU[0000] container "04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864" has work directory "/var/lib/containers/storage/overlay-containers/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/userdata"
DEBU[0000] container "04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864" has run directory "/var/run/containers/storage/overlay-containers/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/userdata"
DEBU[0000] New container created "04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864"
DEBU[0000] container "04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864" has CgroupParent "/system.slice/docker-prometheus-blackbox/libpod-04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864"
DEBU[0000] Not attaching to stdin
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/CKI34KHF277EKM2BGMYVSUO374:/var/lib/containers/storage/overlay/l/L3UPP3FQ5VB4KEQ2Q23USHXFGK:/var/lib/containers/storage/overlay/l/YCG3UETY4JJGQLNUCK5MJFGNQ4,upperdir=/var/lib/containers/storage/overlay/9c091e0991612ac2f796017de0db088bab22e548bdf8072e7cddbc323399a553/diff,workdir=/var/lib/containers/storage/overlay/9c091e0991612ac2f796017de0db088bab22e548bdf8072e7cddbc323399a553/work
DEBU[0000] mounted container "04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864" at "/var/lib/containers/storage/overlay/9c091e0991612ac2f796017de0db088bab22e548bdf8072e7cddbc323399a553/merged"
DEBU[0000] Created root filesystem for container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 at /var/lib/containers/storage/overlay/9c091e0991612ac2f796017de0db088bab22e548bdf8072e7cddbc323399a553/merged
DEBU[0000] adding entry to /etc/passwd for non existent default user
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0000] Setting CGroup path for container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 to /system.slice/docker-prometheus-blackbox/libpod-04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d
DEBU[0000] Created OCI spec for container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 at /var/lib/containers/storage/overlay-containers/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/userdata/config.json
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/libexec/podman/conmon args="[--api-version 1 -c 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 -u 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 -r /usr/lib/cri-o-runc/sbin/runc -b /var/lib/containers/storage/overlay-containers/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/userdata -p /var/run/containers/storage/overlay-containers/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864]"
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: write /sys/fs/cgroup/unified/system.slice/docker-prometheus-blackbox/conmon/tasks: open /sys/fs/cgroup/unified/system.slice/docker-prometheus-blackbox/conmon/tasks: permission denied
DEBU[0000] Received: 30313
INFO[0000] Got Conmon PID as 30303
DEBU[0000] Created container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 in OCI runtime
DEBU[0000] Attaching to container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864
DEBU[0000] connecting to socket /var/run/libpod/socket/04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864/attach
DEBU[0000] Starting container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 with command [/usr/bin/prometheus-blackbox-exporter --config.file /etc/prometheus/blackbox.yml]
DEBU[0000] Started container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864
DEBU[0000] Enabling signal proxying
DEBU[0000] Cleaning up container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] unmounted container "04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864"
DEBU[0000] Successfully cleaned up container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864
DEBU[0000] Container 04cef00f779338631dae54a496d81d0f3e0974b4a5007eef3855351aa815e864 storage is already unmounted, skipping...
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is not being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
Edited by ale