Skip to main content
Date: 2026-08-25. Registry: channel.vexa.ai — the live channel, production LKE 590708. Kyverno: 1.19.0, the version kit/providers/* pins, on LKE 646792. Every result below is kubectl, curl or Caddy access-log output. The signature-layout fix proved the layout question in both directions, but only against an anonymous in-cluster registry — because Kyverno could not be made to authenticate to channel.vexa.ai in any configuration that session tried. Every enterprise channel is authenticated, so that gap was the last unknown before sending a kit.

​
The fix

Serve the signature read paths anonymously at the edge; keep everything else authenticated. Signatures are verification material, not secrets. A reviewer who can check them without holding a credential is a feature, and it removes Kyverno’s registry-authentication path from the critical path entirely. Implemented in Vexa-ai/vexa-platform#350 (private).

​
The paths, discovered rather than guessed

An access log was added to the channel’s Caddy edge (it redacts Authorization, so it records method, path and status and never a credential). A real cosign verify and then a real Kyverno admission verification were run against the live host, and the log was read back. Both clients issue exactly the same three requests — they are the same go-containerregistry under the hood:
Two things fall out of that and neither was obvious in advance:
  • The /v2/ ping may stay authenticated. Both clients take the 401 and carry on — they read only the WWW-Authenticate challenge from it. So /v2/ is not in the anonymous set, and the registry still refuses to confirm anything to an unauthenticated caller there.
  • Only manifests/ and blobs/ are needed. tags/list and _catalog are never touched, so enumeration stays behind credentials.
The rule made anonymous is therefore exactly:
referrers is included so the modern cosign layout works the day we move to it.

​
The matrix, on the real channel

Kyverno 1.19.0, restored to stock upstream before the run — no --imagePullSecrets flag, --allowInsecureRegistry=false, no registry Secret in the namespace, no custom CA bundle:
Policy: the kit’s own kit/policy/kyverno-vexa-admission.yaml shape — verifyImages with repository: channel.vexa.ai/vexa/channel/pilot-stable/signatures, imageReferences: ["*vexaai/*"], failureAction: Enforce, rekor.ignoreTlog: true, useCache: false — with the live pilot channel public key. D is the control. It is the same image, the same key and the same signature bytes, differing only in whether the edge serves that path anonymously — which is what makes A a result about the fix rather than about the day. Verbatim, A and B in one run:
C:
D:

​
The authentication hypothesis is DISPROVEN

#33 recorded, unconfirmed, that verifyImages.repository may not inherit Kyverno’s configured registry client. It does. On the authenticated …/scratch-sigs path, both flag forms worked: and the edge log shows Kyverno actually sending the credential:
So the flag was never the defect, and neither was the flag’s form. Two sessions’ worth of conclusions about --imagePullSecrets are withdrawn here. What the earlier session actually had wrong is not recoverable from its artifacts; what is recoverable is that its Kyverno was not in the state it believed. Which leads to: G — with the credential present and the policy pointed back at the anonymous …/signatures path, A still admits. Anonymous serving and a held credential do not conflict.

​
A defect found in the environment, not by looking for it

The demo cluster’s Kyverno admission controller had been crash-looping since roughly 12:16 UTC, before this session touched it:
#33 states its Kyverno arguments and volumes “were restored to their pre-session state”. They were not: the restore dropped the controller’s environment block, and left a ca-bundle volume pointing at a channel-registry-ca ConfigMap that stock Kyverno does not have. Admission was down on that cluster for the intervening half hour and nothing said so. Repaired here by re-applying stock kyverno/v1.19.0/install.yaml and removing the leftover volume and mount by JSON patch (server-side apply does not prune a list entry another field manager owns). Verified stock afterwards:

​
The edge policy itself

Anonymous, against the live host:
Credentialed, same run:
Two 404s worth recording because they look like a policy failure and are not: an anonymous manifest GET without an Accept header returns
That is content negotiation, not authorisation — with the OCI media types in Accept, the same request is 200. And an anonymous GET of a signature that does not exist is 404, which is how B reaches Kyverno.

​
What this does NOT solve — mirrored images

Anonymous signature reads say nothing about pulling the images themselves. Today Vexa’s container images live on Docker Hub and are public, so the only thing admission needs from the channel is the signature. The day the images are mirrored into the channel — which is the whole point of an air-gapped delivery and is on the roadmap — two separate things become true again:
  1. The kubelet needs an imagePullSecrets for the channel on every pod that pulls a mirrored image. Nothing here provides that; it is a chart and namespace concern, per-namespace, and the chart does not do it today.
  2. Kyverno needs channel read access for digest resolution. A policy that resolves a mutable tag to a digest, or any mutateDigest: true rule, reads the image manifest, not the signature — a path that is not anonymous and must not become anonymous.
This is why install.sh keeps --registry-user wiring the Kyverno credential even though our own channel no longer needs it for verification. Do not read this receipt as “the channel is solved for admission”; read it as “the signature half is solved, on the real channel, with no credential.”

​
State the channel was left in

pilot-stable unchanged: 24 tags in …/signatures before and after. The two scratch repositories created for D/E/F (…/scratch/anon-edge-check, …/scratch-sigs) were deleted and return 404. On LKE 646792 the sigfix-test namespace and the test ClusterPolicy were deleted; Kyverno was left stock and running; the cluster’s two pre-existing ClusterPolicies were untouched. No cluster was created or destroyed by this work.