Skip to content

ci(e2e): add CentOS Stream 10 Docker/Podman/SELinux suite#2209

Open
ericcurtin wants to merge 1 commit into
NVIDIA:mainfrom
ericcurtin:ci-centos-selinux-vm-script
Open

ci(e2e): add CentOS Stream 10 Docker/Podman/SELinux suite#2209
ericcurtin wants to merge 1 commit into
NVIDIA:mainfrom
ericcurtin:ci-centos-selinux-vm-script

Conversation

@ericcurtin

@ericcurtin ericcurtin commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in E2E suite that boots a CentOS Stream 10 VM under QEMU/KVM
on the existing self-hosted linux-amd64-cpu8 runner, installs Docker
Engine alongside Podman, verifies SELinux is enforcing, and runs the
standard e2e:docker and e2e:podman suites against both engines on the
same SELinux-enforcing guest, per the discussion in #2092.

VM creation is a standalone, locally runnable script rather than being
embedded in the workflow YAML, so contributors can reproduce the exact
CI environment on their own machine when debugging a SELinux/bind-mount
issue.

Related Issue

Follow-up to the CI request in #2092 (comment), superseding the earlier attempt in #2093 (closed pending an org-provisioned RHEL 10 runner; this revision needs no new runner class).

Changes

  • New script tasks/scripts/vm/centos-selinux-vm.sh with create /
    status / ssh / destroy subcommands:
    • Downloads a CentOS Stream 10 GenericCloud qcow2 image (checksum
      verified when published), caches it, and boots a copy-on-write
      overlay disk per run so repeated local invocations don't re-download
      or mutate the cached base image.
    • Seeds the guest via cloud-init over a generated ISO and boots it
      under QEMU with hardware acceleration (KVM on Linux, HVF on Intel
      Mac, falling back to TCG with a warning otherwise).
    • Fixes a -nographic cannot be used with -daemonize QEMU
      incompatibility that was present in the prior attempt (-nographic
      redirects the serial/monitor console to the controlling tty, which
      QEMU rejects when daemonizing) — found by actually booting a VM with
      the script locally. Uses -display none -monitor none with an
      independent -serial file:... instead.
    • Wired up as mise run vm:centos-selinux:create / :status /
      :ssh / :destroy in tasks/vm.toml.
  • New reusable workflow .github/workflows/e2e-centos-selinux.yml:
    • Runs on the existing self-hosted linux-amd64-cpu8 runner (the same
      class already used by driver-vm-linux.yml with --privileged) —
      no new org-provisioned runner class is required.
    • Calls the new script to create the guest, then drives the rest of
      the suite over SSH: installs Docker via get.docker.com alongside
      Podman (RHEL-family distros ship Podman but not Docker) with the
      --ip-forward-no-drop dockerd override so both engines coexist,
      confirms SELinux is Enforcing, runs mise run e2e:docker then
      mise run e2e:podman, and greps the guest audit log for AVC denials
      recorded during the run, failing the job if any are found.
  • Wires the suite into branch-e2e.yml behind a new test:e2e-centos
    label, following the existing test:e2e-kubernetes pattern (optional,
    not part of the required CI gate, since it boots a VM inside the job
    rather than using standard runner infrastructure).
  • Updates e2e-label-help.yml, CI.md, CONTRIBUTING.md, and
    architecture/build.md to document the new label and workflow.
  • Fixes a branch-e2e.yml wiring bug: centos-selinux-e2e and
    centos-selinux-e2e-result unnecessarily depended on build-gateway,
    which the CentOS suite never uses. build-gateway only runs for
    run_core_e2e/run_kubernetes_ha_e2e, so with only test:e2e-centos
    applied it was skipped, GitHub's implicit success() check on needs
    skipped centos-selinux-e2e in turn, and the always()-gated result
    job then failed the check. Dropped build-gateway from both jobs.

Testing

  • mise run pre-commit passes — partially: license:check,
    markdown:lint, actionlint, and shellcheck -x were run directly
    (all pass) and cargo fmt --all -- --check passes. The full
    mise run lint/pre-commit chain currently fails locally on
    license:check due to an unrelated uv project-build environment
    issue (uv run tries to build the openshell sdist and fails),
    reproducible on main and unrelated to this change; worked around
    with uv run --no-project. No Rust or Python source changed, so
    rust:lint/clippy and the Python checks weren't otherwise affected.
  • Manually created, tracked (status), and destroyed a VM locally
    with tasks/scripts/vm/centos-selinux-vm.sh (using a dummy disk image
    in place of the multi-GB CentOS download) to verify the QEMU
    daemonize/pidfile/ssh-wait/teardown lifecycle end to end.
  • Not runnable end-to-end from this session: no /dev/kvm-capable
    runner available to boot the real CentOS Stream 10 guest and exercise
    e2e:docker/e2e:podman inside it.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (architecture/build.md)

@copy-pr-bot

copy-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@elezar

elezar commented Jul 10, 2026

Copy link
Copy Markdown
Member

/ok-to-test 66b1809

PR NVIDIA#2092 adds SELinux relabeling (:z/:Z) support to the Docker and
Podman driver bind mounts, but our existing E2E lanes run inside the
Ubuntu-based ghcr.io/nvidia/openshell/ci container image, which does
not enforce SELinux. That leaves the new relabeling behavior, and any
future SELinux-sensitive change, without a host that can actually deny
a mislabeled mount.

Per the discussion on PR NVIDIA#2092 and the follow-up in NVIDIA#2093, add a
Docker/Podman/SELinux compatibility suite backed by a CentOS Stream 10
VM booted under QEMU/KVM on the existing self-hosted linux-amd64-cpu8
runner (already used by driver-vm-linux.yml with --privileged). This
needs no new runner class: it downloads a CentOS Stream 10 GenericCloud
image, seeds it with cloud-init over a generated ISO, boots it under
hardware-accelerated QEMU, and drives the guest entirely over SSH.

Unlike the earlier attempt in NVIDIA#2093, VM creation lives in a standalone
script (tasks/scripts/vm/centos-selinux-vm.sh) with create/status/ssh/
destroy subcommands instead of being embedded directly in the workflow
YAML, wired up as mise run vm:centos-selinux:*. This lets a contributor
reproduce the exact CI environment on their own machine to debug a
SELinux/bind-mount issue without needing CI access. The script caches
the downloaded base image and boots a copy-on-write overlay disk so
repeated local runs don't re-download or mutate the cached image, and
auto-detects KVM/HVF/TCG acceleration so it also runs (slowly) on
non-KVM hosts such as a contributor's Mac. It also fixes a QEMU
'-nographic cannot be used with -daemonize' incompatibility present in
the prior attempt, found by actually booting a VM with the script.

Once the guest is up, the workflow installs Docker Engine and Podman
side by side, confirms SELinux is Enforcing, runs the standard
e2e:docker and e2e:podman suites, and fails the job if the audit log
shows AVC denials during the run. It is wired in as optional/
non-blocking via a new test:e2e-centos label, matching the existing
test:e2e-kubernetes pattern.

CI.md, CONTRIBUTING.md, and architecture/build.md are updated to
document the new label and workflow.

The centos-selinux-e2e and centos-selinux-e2e-result jobs in
branch-e2e.yml unnecessarily depended on build-gateway, which the
CentOS suite never uses (it only pulls the supervisor image).
build-gateway only runs for run_core_e2e/run_kubernetes_ha_e2e, so
when a PR carries only the test:e2e-centos label, build-gateway is
skipped and GitHub's implicit success() check on needs skips
centos-selinux-e2e in turn, tripping the always()-gated result job.
Drop build-gateway from both jobs' needs so the suite runs correctly
when only test:e2e-centos is applied.

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@ericcurtin ericcurtin force-pushed the ci-centos-selinux-vm-script branch from 66b1809 to 9a01e1d Compare July 10, 2026 14:59
@ericcurtin

Copy link
Copy Markdown
Contributor Author

@elezar fixed the build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants