Receipt — clean-pull of the production estate onto an empty cluster
Date 2026-08-25 · Cluster throwaway LKE647376 (us-sea, 3× g6-standard-4, tag throwaway), destroyed at the end of this run · Channel channel.vexa.ai/vexa/channel/vexa-internal · Entry 0.12.22-estate-20260825, seq 1, digest sha256:42fcfa3a488afb29fb8724451d7edef90613e39f37777e8d59228d58e7e51f40
Production was never touched. Every read against vexa-production and monitoring was get/describe/helm get. No object was created, patched or deleted in either namespace. vexa-staging was not touched at all.
What this proves, in one sentence
The production estate — all 115 objects — resolved and applied onto an empty cluster from the channel’s signed artifacts alone, with images pulled fromchannel.vexa.ai and Kyverno verifying every signature, after exactly two operator acts: kit/install.sh, and seeding the cluster’s Secrets.
And what it does not prove: that the estate then WORKS. 16 of 24 workloads reached Running. The rest failed, and every failure is written up below. This receipt is the finding list, not a green tick — the clean pull was worth running precisely because it produced them.
The validation contract
Everything below was validated at a declared fidelity, not against a vague “test environment”. The contract isstation/validation-contract-internal.yaml, id vexa-internal-estate-2026-08-25, sha256 f66b2a938661d95c8558979b06efc7bf52dc32457be4d70b1dbbe0c4cde90f1f, and that hash is inside the signed entry (estate.validation_contract), so it cannot drift from the release it describes.
The rule it states, and which the publisher now enforces by refusing an entry that breaks it: real by default; each double carries a justification; an unjustifiable double is a contract violation.
Four real, two doubles, three absent, zero dummy endpoints. Claims the absences make unavailable are enumerated in the contract’s
unproven_claims and are not asserted anywhere in this receipt.
The findings — in the order the cluster produced them
Each of these was invisible until an estate was pulled onto a cluster that had nothing. That is the argument for running this proof at all.1. The mirror was decoration. Nothing pulled from it.
Neither thevexa-platform chart nor the vendored vexa subchart has any registry-prefix mechanism — no global.imageRegistry, no image.registry key, nothing. Every image is pinned as a bare Docker Hub reference (vexaai/v012-gateway@sha256:…, caddy@sha256:…).
So the channel can mirror every image the estate needs — 13.01 GB of it — and the estate still pulls from docker.io.
For us that is invisible: Docker Hub is reachable, the pods come up, everything looks correct. For a customer whose firewall allows exactly one host — the entire premise of mirroring — the install simply fails, at their site, on their maintenance window.
Fixed by chart/vexa-platform/values-channel-images.yaml (Vexa-ai/vexa-platform#353): 33 references rewritten. Verified by rendering the packaged channel chart:
2. Nothing could pull anyway: the kit never provisioned a kubelet pull secret
Argo’s repo-server fetches charts with a repository Secret in theargocd namespace. That is what the kit created. But images are pulled by the kubelet, in the workload namespace, from spec.imagePullSecrets — and no such Secret existed.
Every pod:
Succeeded, 115/115 Synced, phase: Succeeded. From Argo’s side it was successful — it applied every object correctly. The failure is one layer below the layer that reports success.
The kit created a namespace registry credential only when --verifier-image was passed, and named it for the PreSync verifier. Nothing existed for the kubelet, because until the channel carried images nothing needed one.
Fixed in kit/install.sh: the credential is now created unconditionally with --registry-user.
3. Four workloads have no imagePullSecrets at all, and no values key to give them one
caddy, capacity-resize, collector-watchdog, system-host-labeler render with none. Correct while their images were public; fatal against an authenticated channel, and unreachable by any values overlay because the chart has no key.
Fixed in the kit by attaching the pull secret to the namespace’s ServiceAccounts — the kubelet applies it to every pod using that account, so it covers workloads whose chart forgot one without the chart having to change first. The chart should still be fixed.
4. templates/cronjob-transcription-check.yaml hardcodes dockerhub-secret
Two sites, no values path. The two txcheck CronJobs carry a credential name the channel does not provision. Filed as Vexa-ai/vexa-platform#359 (private), and shipped as a declared hole inside the signed entry (estate.known_holes[0]) so a subscriber reads it before hitting it.
5. The published estate carries OUR PRODUCTION DATABASE HOSTNAME
This is the one that matters most.vexa.database.host is a chart value, baked into the published chart, reading:
pg_hba does not list the throwaway cluster’s egress IP. That is a firewall saving us, not a design. The estate had no per-cluster default and no required-value gate.
database.host must become a required customer value that the chart refuses to render without — the subchart already has the required helper for exactly this when postgres.enabled=false; the parent supplies a default that silently satisfies it.
6. The channel does not deliver the CRDs its own content depends on
The first sync failed:ServiceMonitor, and the CRD that defines it comes from kube-prometheus-stack — which the channel does not carry (declared hole kube-prometheus-stack-not-mirrored). An undeclared ordering dependency: the estate cannot install until a chart from a different source is installed first, which also breaks the one-host claim for monitoring.
7. targetRevision: "*" silently matches nothing when the chart version is a prerelease
The chart was first published as 0.1.0-estate.20260825. Argo:
Healthy while resolving nothing. Semver * does not match prereleases. Republished as 0.1.20260825 and it resolved immediately. A channel whose publisher happens to use a prerelease version string produces a subscription that reports healthy and delivers nothing.
8. Subscriber overrides applied to the Application are reverted
Patchingspec.source.helm.valuesObject directly is undone by the ApplicationSet controller on the next reconcile. Correct behaviour, and worth stating: the only supported place for per-cluster values is the kit’s --customer-values at install time.
9. The billing worker refuses a test-mode Stripe key
stripe_test_custody.py is a real guard doing its job. It also means the estate as published is hardwired to live billing mode, so the highest-fidelity Stripe validation available (a real test-mode sandbox) cannot be exercised without a values change the published estate does not offer.
10. Kyverno verifies only *vexaai/*
The admission policy’s imageReferences is ["*vexaai/*"]. The mirrored third-party images — caddy, redis, postgres, bitnami/kubectl — pass admission unverified, even though we mirrored them and could attest that they are what we copied. Not wrong (we cannot attest to upstream builds) but it is a narrower guarantee than “the channel verifies what it serves”.
What did work, stated precisely
Images mirrored
43 images, 43 succeeded, 0 failed. 16.24 GB (17,433,428,939 bytes across all platforms of every index). Every destination digest was verified equal to its source digest withcrane digest, twice — inline during the run and again in an independent pass reading the result file back against the live registry.
40 landed on the first pass; 3 failed on Docker Hub’s rate limit and succeeded on retry:
redirect: disable: true property from Vexa-ai/vexa-platform#352 doing its job — a 307 to linodeobjects.com would silently reintroduce a second egress host:
channel.vexa.ai/vexa/channel/vexa-internal/signatures; the repo lists 43 signature tags.
One measurement worth recording: registry.k8s.io/pause measures 551 MB. That is not an error — the index carries seven children, two of them windows/amd64, each with a ~265 MB uncompressed Windows base layer. They were checked for being foreign/non-distributable (URL-referenced, never transferred) and they are not: plain application/vnd.docker.image.rootfs.diff.tar, no urls field, and the channel genuinely stores and serves the 265 MB blob. So the 16.24 GB is real transferred data, and mirroring multi-arch indexes wholesale is most of the cost — the estate needs linux/amd64.
Both the production estate’s image set and the pilot-stable OSS set are mirrored, the latter against the standing promise.
Disaster recovery, as far as this goes
production = channel + secret store + database backup is partly demonstrated. The channel half is real: the object graph and the images came from it. The secret half is a list of names, not a restore. The data half was not attempted. A DR claim needs a run that restores a real backup into a real managed instance, which this was not.