Kubeflow Pipelines: Cross-tenant execution-cache poisoning / output substitution via a namespace-blind cache key in the cache webhook (multi-user)

CISA KEV lists vulnerabilities known to be exploited. FIRST EPSS estimates the likelihood of exploitation in the next 30 days. An unavailable score does not mean zero risk.

No exploitation or prediction data is available here for these CVEs. This does not mean they are safe or unlikely to be exploited.

Kubeflow Pipelines enables users to build and deploy portable, scalable machine learning workflows. Prior to 2.18.0, the legacy V1/raw-Argo execution cache mutating webhook and cache-server storage path hash selected template fields without the Kubernetes namespace, use the key alone for lookups and duplicate checks, and trust the mutable pipelines.kubeflow.org/execution_cache_key annotation when publishing outputs. In multi-user deployments where tenant namespaces share the cache service, a tenant with workload-submission privileges can read another namespace's cached outputs or seed matching outputs that are substituted into a later step. Standard KFP V2 component pods bypass this cache, matching requires the selected template fields and staleness policy, and arbitrary code execution or service availability impact is not established. This issue is fixed in version 2.18.0.

Source published

5 Nov 2026 22:25:58 UTC

Source modified

10 Nov 2026 04:44:31 UTC

Saved here

10 Nov 2026 06:02:11 UTC

Source terms

CVE Program Terms of Use

Original source record · Cite this exact observation · Complete original publisher JSON

Source: cve-program · CVE Program Terms of Use. The publisher's wording is preserved. ChangeIntel groups the information and labels changes. The JSON record includes additional source fields.

CVSS 3.1 · 5.4 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N

Kubeflow Pipelines enables users to build and deploy portable, scalable machine learning workflows. Prior to 2.18.0, the legacy V1/raw-Argo execution cache mutating webhook and cache-server storage path hash selected template fields without the Kubernetes namespace, use the key alone for lookups and duplicate checks, and trust the mutable pipelines.kubeflow.org/execution_cache_key annotation when publishing outputs. In multi-user deployments where tenant namespaces share the cache service, a tenant with workload-submission privileges can read another namespace's cached outputs or seed matching outputs that are substituted into a later step. Standard KFP V2 component pods bypass this cache, matching requires the selected template fields and staleness policy, and arbitrary code execution or service availability impact is not established. This issue is fixed in version 2.18.0.

Source published

Nov 5, 2026, 10:25:58 PM (timezone not stated)

Source modified

Nov 10, 2026, 4:44:31 AM (timezone not stated)

Saved here

11 Nov 2026 23:57:28 UTC

Source terms

ENISA EUVD · source terms and attribution apply

Original source record · Cite this exact observation · Complete original publisher JSON

Source: euvd · ENISA EUVD · source terms and attribution apply. The publisher's wording is preserved. ChangeIntel groups the information and labels changes. The JSON record includes additional source fields.

EUVD CVSS 3.1 · 5.4 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N

Kubeflow Pipelines enables users to build and deploy portable, scalable machine learning workflows. Prior to 2.18.0, the legacy V1/raw-Argo execution cache mutating webhook and cache-server storage path hash selected template fields without the Kubernetes namespace, use the key alone for lookups and duplicate checks, and trust the mutable pipelines.kubeflow.org/execution_cache_key annotation when publishing outputs. In multi-user deployments where tenant namespaces share the cache service, a tenant with workload-submission privileges can read another namespace's cached outputs or seed matching outputs that are substituted into a later step. Standard KFP V2 component pods bypass this cache, matching requires the selected template fields and staleness policy, and arbitrary code execution or service availability impact is not established. This issue is fixed in version 2.18.0.

Source published

5 Nov 2026 23:17:01 UTC

Source modified

10 Nov 2026 05:18:12 UTC

Saved here

11 Nov 2026 23:56:51 UTC

Source terms

NIST NVD usage terms; underlying source attribution retained

Original source record · Cite this exact observation · Complete original publisher JSON

Source: nvd · NIST NVD usage terms; underlying source attribution retained. The publisher's wording is preserved. ChangeIntel groups the information and labels changes. The JSON record includes additional source fields.

CVSS 3.1 · [email protected] · Secondary · 5.4 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N

## Summary The legacy Kubeflow Pipelines V1/raw-Argo execution cache does not separate cached results by Kubernetes namespace. In multi-user deployments where tenants share the cache service, a tenant can receive another tenant's cached outputs or seed outputs that are subsequently substituted into another tenant's step. This affects the Argo-template cache mutating webhook and its cache-server storage path. Standard KFP V2 component pods bypass this webhook cache; the separate V2 cache implementation is outside this advisory's scope. ## Affected deployments and impact The issue requires multiple tenant namespaces sharing the legacy cache service, cache-enabled V1/raw-Argo steps, and cache entries that satisfy the configured staleness policy. The standard multi-user configuration watches pods across namespaces. A tenant able to submit a step with the same cache-relevant template fields as another tenant's step can: - Read cached step outputs originally produced in another namespace. - Populate the shared cache before another tenant's matching step runs, causing that step to report the cached outputs instead of executing its intended containers. Matching refers to the selected template fields used by the cache key, not merely an identical container image. This is an application-level cache-key collision caused by missing tenant scope, not a cryptographic SHA-256 collision. The finding establishes cross-namespace confidentiality and integrity impact; it does not establish arbitrary code execution. ## CVSS rationale **Medium — CVSS 3.1: 5.4** `CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N` - **AV:N / PR:L / UI:N:** a tenant with ordinary workload-submission privileges can exercise the shared cache through network-accessible APIs. Reading an existing cached result or poisoning a subsequent automated execution requires no additional victim interaction. - **AC:L:** known or shared templates can produce the same cache key deterministically. Cache-enabled execution and matching template fields describe the affected configuration; they do not establish an unavoidable narrow race or secret prerequisite. - **S:U:** the established impact is disclosure or substitution of output data managed by the same cache service. Crossing tenant namespaces alone does not establish a change in security authority; execution under a separate victim authority has not been established here. - **C:L / I:L / A:N:** the confirmed impact is selective cached-output disclosure and substitution. Arbitrary code execution, complete workload compromise, and a service-availability impact are not established. ## Root cause The vulnerable implementation hashes selected Argo template fields without a namespace. Cache lookups and duplicate checks use the key alone, and stored rows have no namespace ownership field. On a hit, the admission handler restores cached outputs and substitutes a dummy container for the step's containers. The completion watcher also trusts the mutable `pipelines.kubeflow.org/execution_cache_key` annotation when publishing outputs. Adding a namespace to the hash alone is therefore insufficient: a tenant able to patch its own pod could supply another namespace's deterministic key. Namespace ownership must be enforced independently on cache writes and reads. Relevant implementation files are `backend/src/cache/server/mutation.go`, `backend/src/cache/server/watcher.go`, `backend/src/cache/storage/execution_cache_store.go`, and `backend/src/cache/model/execution_cache.go`. ## Fix status The source fix merged on September 10, 2026 in [PR #14359](https://github.com/kubeflow/pipelines/pull/14359), commit [620d0cc89ae0d042a484a003f8ffe6e9a55d347e](https://github.com/kubeflow/pipelines/commit/620d0cc89ae0d042a484a003f8ffe6e9a55d347e). The merged fix: - Derives namespaced keys from the API server's admission request namespace and selected template fields. - Recomputes completed pods' keys from their observed namespace and `ARGO_TEMPLATE`, rejecting mismatched or malformed identities rather than trusting the annotation. - Stores the observed namespace separately and scopes normal reads and duplicate detection by namespace and key. - Adds a namespace column and composite index, leaving historical rows unowned and excluded from normal lookups. - Rejects structurally malformed template objects without panicking. The first fixed release is **[2.18.0](https://github.com/kubeflow/pipelines/releases/tag/2.18.0)**. Releases through **2.17.3** remain affected; the affected-version upper bound is `<= 2.17.3` and the patched version is `2.18.0`. ## Upgrade requirements and legacy fallback The standard cache-server Deployment supports a normal rolling upgrade from 2.17.2. Apply the upgraded manifests without manually stopping old replicas or pausing submissions. Namespace isolation is fully enforced only after all old cache-server processes, including terminating pods, have exited and every new replica uses `enforce`. Custom installations sharing the cache database must upgrade every writer. Mixed-version operation does not yet provide the full isolation guarantee. Expect reduced cache reuse and increased step execution/resource use as tasks next run; there is no startup cache rebuild. Do not infer or backfill namespace ownership for historical rows. Pre-upgrade pods carrying old-format keys do not seed the new namespaced cache. Completing the upgrade does not undo previously disclosed data or reused poisoned outputs. Do not roll back to the vulnerable implementation to recover cache hits. [PR #14427](https://github.com/kubeflow/pipelines/pull/14427) merged into `release-2.18` on October 5, 2026 and standardizes the compatibility setting as **`KFP_SECURITY_LEGACY_CACHE_MODE=enforce|audit` on the cache-server Deployment**. This supersedes the original fix's `ALLOW_LEGACY_CACHE_FALLBACK` interface. Unset, empty, or `enforce` keeps full namespace isolation; invalid values prevent startup. Explicit `audit` permits a genuine namespaced cache miss to reuse a historical template-only entry whose stored namespace is empty. That entry may contain another tenant's outputs, including poisoned outputs: audit accepts residual cross-tenant confidentiality and integrity risk, and is not full remediation. Scoped entries take precedence, known foreign namespaces remain excluded, and a database error does not enable legacy fallback. Legacy hits are not promoted or backfilled into namespaced rows. The server emits a startup warning and scoped `security_audit` events for legacy reuse without logging cached outputs or credentials. Keep replicas configured consistently, persist the mode in deployment configuration, and return to `enforce` after migration. Native V2 caching is outside this control and this advisory. See the [released cache-server guide](https://github.com/kubeflow/pipelines/blob/2.18.0/backend/src/cache/README.md#namespace-isolation-and-upgrades) for rollout completion checks and audit-to-enforce migration. This fix does not establish an integrity boundary between users who can modify workloads within the same namespace. ## Mitigation until the fixed release is deployed Disable legacy cache reuse for affected workloads, or ensure mutually untrusted namespaces do not share the legacy cache service and storage. Restricting the webhook's namespace selector or watcher scope alone is insufficient if mutually untrusted tenants still share cached entries. ## Validation Source review identified the shared key, unscoped storage, and mutable-annotation write path. The public fix includes regressions exercising admission and completion handling, forged foreign-namespace keys, normal same-namespace reuse, malformed templates, legacy-row exclusion, and optional fallback behavior. During development, the cache test suite and lint passed; populated and repeatable schema migration and scoped storage behavior were verified on SQLite, MySQL 8.4, and PostgreSQL 16. Negative controls removing the watcher identity check or storage namespace filter failed their corresponding regressions. These tests use fake Kubernetes clients; they are not a claim of a live multi-tenant Kubernetes exploit reproduction. ### Private patch disposition The earlier [private PR #1](https://github.com/kubeflow/pipelines-ghsa-2xwh-426q-42c5/pull/1) is closed as superseded by [public PR #14359](https://github.com/kubeflow/pipelines/pull/14359) and the released follow-ups described above. The private branch and review history are retained; this closure does not dismiss the report or change reporter credit. The advisory remains a draft pending the publication review.

Source published

9 Nov 2026 22:43:33 UTC

Source modified

9 Nov 2026 23:00:03 UTC

Saved here

9 Nov 2026 23:04:27 UTC

Source terms

GitHub Advisory Database · CC-BY-4.0

Original source record · Cite this exact observation · Complete original publisher JSON

Source: osv · GitHub Advisory Database · CC-BY-4.0. The publisher's wording is preserved. ChangeIntel groups the information and labels changes. The JSON record includes additional source fields.

CVSS_V3 · CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:N

Publisher rating · MODERATE