ci(e2e): add CentOS Stream 10 Docker/Podman/SELinux suite#2209
Open
ericcurtin wants to merge 1 commit into
Open
ci(e2e): add CentOS Stream 10 Docker/Podman/SELinux suite#2209ericcurtin wants to merge 1 commit into
ericcurtin wants to merge 1 commit into
Conversation
Member
|
/ok-to-test 66b1809 |
PR NVIDIA#2092 adds SELinux relabeling (:z/:Z) support to the Docker and Podman driver bind mounts, but our existing E2E lanes run inside the Ubuntu-based ghcr.io/nvidia/openshell/ci container image, which does not enforce SELinux. That leaves the new relabeling behavior, and any future SELinux-sensitive change, without a host that can actually deny a mislabeled mount. Per the discussion on PR NVIDIA#2092 and the follow-up in NVIDIA#2093, add a Docker/Podman/SELinux compatibility suite backed by a CentOS Stream 10 VM booted under QEMU/KVM on the existing self-hosted linux-amd64-cpu8 runner (already used by driver-vm-linux.yml with --privileged). This needs no new runner class: it downloads a CentOS Stream 10 GenericCloud image, seeds it with cloud-init over a generated ISO, boots it under hardware-accelerated QEMU, and drives the guest entirely over SSH. Unlike the earlier attempt in NVIDIA#2093, VM creation lives in a standalone script (tasks/scripts/vm/centos-selinux-vm.sh) with create/status/ssh/ destroy subcommands instead of being embedded directly in the workflow YAML, wired up as mise run vm:centos-selinux:*. This lets a contributor reproduce the exact CI environment on their own machine to debug a SELinux/bind-mount issue without needing CI access. The script caches the downloaded base image and boots a copy-on-write overlay disk so repeated local runs don't re-download or mutate the cached image, and auto-detects KVM/HVF/TCG acceleration so it also runs (slowly) on non-KVM hosts such as a contributor's Mac. It also fixes a QEMU '-nographic cannot be used with -daemonize' incompatibility present in the prior attempt, found by actually booting a VM with the script. Once the guest is up, the workflow installs Docker Engine and Podman side by side, confirms SELinux is Enforcing, runs the standard e2e:docker and e2e:podman suites, and fails the job if the audit log shows AVC denials during the run. It is wired in as optional/ non-blocking via a new test:e2e-centos label, matching the existing test:e2e-kubernetes pattern. CI.md, CONTRIBUTING.md, and architecture/build.md are updated to document the new label and workflow. The centos-selinux-e2e and centos-selinux-e2e-result jobs in branch-e2e.yml unnecessarily depended on build-gateway, which the CentOS suite never uses (it only pulls the supervisor image). build-gateway only runs for run_core_e2e/run_kubernetes_ha_e2e, so when a PR carries only the test:e2e-centos label, build-gateway is skipped and GitHub's implicit success() check on needs skips centos-selinux-e2e in turn, tripping the always()-gated result job. Drop build-gateway from both jobs' needs so the suite runs correctly when only test:e2e-centos is applied. Signed-off-by: Eric Curtin <eric.curtin@docker.com>
66b1809 to
9a01e1d
Compare
Contributor
Author
|
@elezar fixed the build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in E2E suite that boots a CentOS Stream 10 VM under QEMU/KVM
on the existing self-hosted
linux-amd64-cpu8runner, installs DockerEngine alongside Podman, verifies SELinux is enforcing, and runs the
standard
e2e:dockerande2e:podmansuites against both engines on thesame SELinux-enforcing guest, per the discussion in #2092.
VM creation is a standalone, locally runnable script rather than being
embedded in the workflow YAML, so contributors can reproduce the exact
CI environment on their own machine when debugging a SELinux/bind-mount
issue.
Related Issue
Follow-up to the CI request in #2092 (comment), superseding the earlier attempt in #2093 (closed pending an org-provisioned RHEL 10 runner; this revision needs no new runner class).
Changes
tasks/scripts/vm/centos-selinux-vm.shwithcreate/status/ssh/destroysubcommands:verified when published), caches it, and boots a copy-on-write
overlay disk per run so repeated local invocations don't re-download
or mutate the cached base image.
under QEMU with hardware acceleration (KVM on Linux, HVF on Intel
Mac, falling back to TCG with a warning otherwise).
-nographic cannot be used with -daemonizeQEMUincompatibility that was present in the prior attempt (
-nographicredirects the serial/monitor console to the controlling tty, which
QEMU rejects when daemonizing) — found by actually booting a VM with
the script locally. Uses
-display none -monitor nonewith anindependent
-serial file:...instead.mise run vm:centos-selinux:create/:status/:ssh/:destroyintasks/vm.toml..github/workflows/e2e-centos-selinux.yml:linux-amd64-cpu8runner (the sameclass already used by
driver-vm-linux.ymlwith--privileged) —no new org-provisioned runner class is required.
the suite over SSH: installs Docker via
get.docker.comalongsidePodman (RHEL-family distros ship Podman but not Docker) with the
--ip-forward-no-dropdockerdoverride so both engines coexist,confirms SELinux is
Enforcing, runsmise run e2e:dockerthenmise run e2e:podman, and greps the guest audit log for AVC denialsrecorded during the run, failing the job if any are found.
branch-e2e.ymlbehind a newtest:e2e-centoslabel, following the existing
test:e2e-kubernetespattern (optional,not part of the required CI gate, since it boots a VM inside the job
rather than using standard runner infrastructure).
e2e-label-help.yml,CI.md,CONTRIBUTING.md, andarchitecture/build.mdto document the new label and workflow.branch-e2e.ymlwiring bug:centos-selinux-e2eandcentos-selinux-e2e-resultunnecessarily depended onbuild-gateway,which the CentOS suite never uses.
build-gatewayonly runs forrun_core_e2e/run_kubernetes_ha_e2e, so with onlytest:e2e-centosapplied it was skipped, GitHub's implicit
success()check onneedsskipped
centos-selinux-e2ein turn, and thealways()-gated resultjob then failed the check. Dropped
build-gatewayfrom both jobs.Testing
mise run pre-commitpasses — partially:license:check,markdown:lint,actionlint, andshellcheck -xwere run directly(all pass) and
cargo fmt --all -- --checkpasses. The fullmise run lint/pre-commitchain currently fails locally onlicense:checkdue to an unrelateduvproject-build environmentissue (
uv runtries to build theopenshellsdist and fails),reproducible on
mainand unrelated to this change; worked aroundwith
uv run --no-project. No Rust or Python source changed, sorust:lint/clippyand the Python checks weren't otherwise affected.status), and destroyed a VM locallywith
tasks/scripts/vm/centos-selinux-vm.sh(using a dummy disk imagein place of the multi-GB CentOS download) to verify the QEMU
daemonize/pidfile/ssh-wait/teardown lifecycle end to end.
/dev/kvm-capablerunner available to boot the real CentOS Stream 10 guest and exercise
e2e:docker/e2e:podmaninside it.Checklist
architecture/build.md)