Skip to main content
Date: 2026-08-24. Authorised by: the founder, for this rehearsal, on the day. Channel: channel.vexa.ai/vexa/channel/pilot-stable — the live registry, not a test rig. Cluster: LKE vexa-rehearsal (id 646952), us-sea, k8s v1.36.3, 2× g6-standard-4, tag throwaway, created and destroyed the same day. Container work ran on bbb, never on the laptop. The question this rehearsal was built to answer is not “does the machinery work”. It is “if we send the kit to the pilot subscriber on Monday, what happens?” So it was run the way they will run it: from an empty directory, with a pull-only credential, against the real channel, with nothing borrowed from this repository. The short answer is that the customer would have got stuck four separate times before a single Vexa pod started, and none of the four would have looked like our fault to them. Every one is now fixed and proven. Three things remain open and are named at the bottom.

​
What was proven, in order

​
1 · The entry on the customer channel was not one we could send

Before anything else, the entry pilot-stable:current was pulled and read. It carried: Decision taken, and why. The brief allowed for minting an entry at the chart’s lineage. There is no honest way to do that: v0.12.23 is the newest tag that exists, and it is the only one with a real candidate map and a real delivery receipt. So the entry was rebuilt at v0.12.23 from genuine inputs — nothing fabricated, no --break-glass — and the chart was repackaged by the publisher from that tag and republished, so the entry names a chart that exists and was built from the release it claims.
vexa_channel.py verify then passed all ten offline checks including the entry signature and the SLSA provenance against the archive bytes. The 0.12.30-vs-0.12.23 “lineage mismatch” turned out to be a symptom of a design defect, not a mistake by whoever published it. cmd_chart wrote the release version into both of Chart.yaml’s version lines. Helm keeps two because they are two facts — version is the chart revision a subscriber’s targetRevision: "*" ranks, appVersion is the release it deploys. Collapsed, a release could never ship a second chart revision, and chart revisions shared a number space with releases. --chart-version separates them.

​
2 · VERIFY.md documented a command that has never worked

The published VERIFY.md told the customer to run:
which returns, on every entry this publisher has ever produced:
The channel signs offline against a pinned key and uploads nothing to Rekor, deliberately — that is what makes it air-gappable. So the flag that works is --insecure-ignore-tlog=true, and --new-bundle-format must be absent because the bundles are legacy. Both write_verify_md and the publisher’s own cmd_verify carried the wrong flags, so the publisher’s verify refused genuine entries. Recorded working commands, verbatim, against the live channel:
cosign here is v3.1.3, not 2.4.3. It has --use-signing-config, and in 3.x it requires it: --tlog-upload=false is refused on its own with "--tlog-upload=false is not supported with --signing-config". Signing needs both --tlog-upload=false --use-signing-config=false. Both flags are already deprecation-warned; a cosign bump will need this revisited. VERIFY.md now also tells the reader to check the key they pin against the identity the entry names — the defect in §1 existed precisely because nothing did.

​
3 · The kit had never been published to a real registry

kit/release.sh aborted at the first push:
${PLAIN[@]} and ${COSIGN_INSECURE[@]} are empty unless --insecure or --plain-http is passed, and an empty array under set -u is “unbound variable” on bash 3.2. So the scripts worked only on the plain-HTTP test rig and broke on exactly the path a real TLS registry takes. install.sh had already learned this and carries the ${arr[@]+"${arr[@]}"} idiom in a comment; release.sh, bootstrap.sh and self-update.sh had not. Underneath it sat a second one: all four scripts and cosign_env() wrote {"auths": {}} to defeat a hanging Docker credential helper. That is correct against a public registry and fatal against an authenticated one — the entry pushed and its signature failed UNAUTHORIZED. The hazard is the helper, not the credentials. They now carry the on-disk auths across, drop credsStore, and accept an explicit credential from VEXA_CHANNEL_{REGISTRY,USER,PASS} — environment, never argv. After the fix, published to the live channel: kit v0.1.0 … v0.1.5, each signed, each moving latest.

​
4 · A customer with nothing but a credential gets a verified kit

A clean directory holding only bootstrap.sh and channel.pub, run with HOME and TMPDIR isolated and only the pull-only subscriber credential:
The negative, same run, wrong pinned key:
Verify-before-unpack holds in both directions, and nothing came from this repository. Write-scoping proven at the registry. With the subscriber credential:

​
5 · Install: authenticated channel, and a Kyverno flag that lies quietly

install.sh had no way to give Argo CD or Kyverno a registry credential. The Argo repository Secrets carried url/name/type and nothing else, so against channel.vexa.ai the repo-server 401s and the subscription never syncs. Added --registry-user (password from VEXA_CHANNEL_PASS). Kyverno needed the same, and cost the most time for the smallest reason. The controller flag takes a bare secret name resolved in Kyverno’s own namespace. Passing kyverno/channel-registry-creds is accepted silently and the fetch stays anonymous, which surfaces as:
— indistinguishable from an unsigned image, while the signature was in fact present and readable by that very credential (verified independently: HTTP 200 for the publisher, 200 for the subscriber, 401 anonymous). Preflight on the clean cluster: PASS, P1–P9.

​
6 · The channel flows, digest-pinned, into a fresh cluster

vexa-enterprise-staging resolved * to chart 0.12.31, pulled it from the authenticated OCI registry, and reached Synced / Healthy with 13 pods: admin-api ×2, agent-api, gateway, meeting-api ×2, runtime, terminal ×2, plus postgres, redis, minio and its init Job. vexa-enterprise-prod tracked UNPINNED and synced nothing — the customer-held gate, working.

​
7 · The station loop was broken at the join

kit/validate produced station.tar.gz, redaction verified — 4 values removed, 0 found anywhere in the archive that was written. The publisher then refused it:
validate names its receipt smoke-receipt-20260824-165142.md, because an operator runs smoke more than once. And station.json carried neither the station name the ingest checks --station against, nor the files[] digest list its S3 check reads — it wrote a contents list of bare names instead. Every genuine bundle was refused, and had always been. The ingest had never seen one. Fixed on both sides, keeping the load-bearing half intact: validate emits station (new --station flag) and files[] with digests; ingest accepts the dated receipt name while still refusing an ambiguous match. Then:

​
8 · The station gate refused the chart, and was right to

This is the most important product finding of the day. The delivery system’s entire claim to an enterprise buyer is an exact, attested image-digest set. That claim covered only Vexa’s own images. The four data-plane dependencies floated, two of them on :latest — and the Kyverno digest-pinning policy never caught them because it matches *vexaai/*. Three were pinnable from kit/profiles/vexa/node-baseline.yaml and now are, at digests resolved on the day. The fourth, minio/mc:latest, was hardcoded in the chart template with no value to override, and the same Job declared no resources at all — the exact LimitRange-squeeze class the preflight’s P2 exists to catch. That needed a change in the OSS chart: Vexa-ai/vexa#1321. Proven that the fix closes the finding: a chart packaged from the v0.12.23 tree with #1321 applied (in a throwaway clone, local tag, never pushed) gates clean —
Gate-report stamping works and did not need new machinery: --extra-evidence other=<name>=<path> carries it into the signed entry. One fix was required — every attached file was stamped application/json, so a markdown gate report would have made the entry lie about its own evidence. Media type now follows the extension, and entry seq 3 carries the real report:
The report stamped into the published entry is the refusal, because that is what the real chart earns today. That is the honest state, not a tidy one.

​
9 · Pull-only self-upgrade: 26 seconds, zero kubectl

The newly pinned dependency arrived with it: postgres:17-alpine@sha256:18cfe3ef… live in the StatefulSet. Nothing on the vendor side touched the cluster; the cluster pulled.

​
10 · Kit self-update

Run five times over the day, each time as the customer, each time verifying the signature against .kit-source.pub before unpacking:
It is worth saying plainly that this loop is why the rehearsal finished: every fix found in the customer’s hands was published to the channel and pulled back down through the same path a customer uses. The delivery mechanism was used to deliver its own repairs.

​
11 · The PreSync verify gate, switched on for the first time

It had been off in every install to date. Turning it on found four defects in a row, each of which fails the sync for a reason that has nothing to do with the evidence: Defect 4 is the one worth losing sleep over. It is the gate’s loudest possible alarm — this release may be forged — fired on a good release, by our own wrong flag. With verifier v0.1.1 (built on bbb, linux/amd64, pushed and signed):
And the negative, with one line added to the station’s contract in its own cluster (require_evidence_kinds += soak):
Pod phase Failed. That a failed verifier stops the sync was observed separately and involuntarily, during defect 4: the Application stayed OutOfSync and did not advance until the verifier passed.

​
12 · mail_cursor — the question does not arise

A fresh install creates six tables and none of them is mail_cursor:
So ALTER TABLE mail_cursor ADD COLUMN token TEXT (per Vexa-ai/vexa#1318) is not needed on a fresh install of this chart, and cannot be applied: the flows schema is not provisioned at all, because the flows tier is not part of the delivered set. Confirmed independently by smoke S4 (below). The ALTER is a question for an existing flows deployment, not for the pilot subscriber’s first install.

​
What FAILED or was partial

​
S3 — the bot joined nothing, and the cause is our own tenant

The M365 rig produced a real Teams meeting via Graph with lobbyBypassSettings: {scope: everyone} accepted by the API. The bot pod spawned from the correct digest-pinned image (vexaai/vexa-bot:v0.12.23@sha256:2bd879c6…, 1.6 GB, pulled in 33s) and failed:
completion_reason: join_failure, bot_outcome: never_admitted, segments_captured: 0. This is a rig-tenant policy, not a per-meeting setting and not a product defect. lobbyBypassSettings governs the lobby; it does not grant anonymous join, which is a Teams meeting policy on the organiser (AllowAnonymousUsersToJoinMeeting). Changing it is a tenant setting change and was not made. S3 is therefore PARTIAL: dispatch, image, pod and error reporting all proven; capture not proven, and no audio question was ever reached. Making S3 genuinely hands-free needs that policy flipped on the rig by a human, once. Worth noting for its own sake: the product recorded a full, correct ExitReason and provenance block. This is precisely the per-reason detail that an internal taxonomy note (private) says the aggregate snapshot throws away.

​
S3, first attempt — a default install cannot run a meeting at all

Before the join was even attempted:
customer-values.example.yaml mentions transcriptionServiceToken only as a commented-out line reading “from your subscription pack”, and never mentions transcriptionServiceUrl at all. Nothing in preflight, install, or the docs tells an operator this is mandatory before anything can work. A customer following the documented path installs successfully, reaches Synced/Healthy, and then cannot run a single meeting. Supplied by hand here from the dev backend to get past it. This is a customer-facing gap and it is still open — see below.

​
S4 — the flows tier is not delivered, and used to crash the run

Before the fix, --flows ended the entire run in a Python traceback: no receipt, no verdict, and an operator with no way to tell “not delivered” from “broken”. The station bundle produced by that run was missing its smoke receipt entirely.

​
The agent/model tier

Not exercised. The flows tier is not in the delivered set, so the question of whether a model credential exists for it never arose. Recorded as a known gap, not tested.

​
Kyverno signature admission — BLOCKED, and this is the headline

The customer-side admission gate — the thing that is supposed to verify our attestations independently of us — cannot read the signatures we publish. The mechanism, isolated: Confirmed both ways on the live channel. --registry-referrers-mode=legacy does not change it; --new-bundle-format=false does not change it. The signature is present and correct — COSIGN_REPOSITORY=… cosign verify passes — and Kyverno cannot see it. The consequence for a customer is the worst available shape: a correctly signed release is denied at admission with a message that says it is unsigned. For the rehearsal to continue past this point, the vexa-verify-channel-signature ClusterPolicy was deleted, and everything after §6 was proven with image-signature admission OFF. That is recorded here rather than smoothed, and nothing downstream should be read as “admission-verified”. The fix was not attempted because the obvious route — signing the images in place on docker.io/vexaai/* — writes to a public namespace and is a publishing act that needs its own decision.

​
Fixes this rehearsal forced

In this repository, on branch rehearsal-receipt: In Vexa-ai/vexa: #1321 — the minio-init Job was unpinnable (minio/mc:latest hardcoded in the template) and declared no resources. make test and make lint green.

​
What the customer must do differently

  1. Set the transcription backend before installing. transcriptionServiceUrl and transcriptionServiceToken. Without both, the install succeeds and no meeting can ever run. This belongs in install.mdx and in customer-values.example.yaml as a required field, not a comment.
  2. Pass --registry-user and export VEXA_CHANNEL_PASS. Every enterprise channel is authenticated; without it Argo never syncs.
  3. Expect enterprise-prod to sit at Unknown with improper constraint: UNPINNED until they move their own pin. That is the gate, not a fault, but the error text reads like one.
  4. Do not enable image-signature admission yet. It will deny correctly signed images. Digest-pinning admission is fine and should stay on.

​
What this does NOT prove

  • Nothing about transcription quality, or about capture at all. No audio ever reached the pipeline. S3 proved dispatch, image identity, pod lifecycle and error reporting — and stopped there.
  • Nothing about image-signature admission. It was deleted to proceed. Every result from §6 onward was obtained with it off.
  • Nothing about the flows/agent tier, which is not in the delivered set.
  • Nothing about OpenShift. One provider profile (lke) was exercised.
  • Nothing about the chart the customer would actually receive passing its own gate. The gate passes only against a chart carrying the unreleased #1321. On the real v0.12.23 tree it refuses, correctly, and the entry now on the channel carries that refusal as evidence.
  • Nothing about scale, upgrade-under-load, rollback, or break-glass. One install, one upgrade, one cluster, no failure injection.
  • Nothing about the dependency images’ provenance. They are pinned now; pinned is reproducible, not attested.
  • The 26-second upgrade is one measurement on a two-node cluster with a warm image cache — a demonstration that the transport works unattended, not a latency guarantee.
  • Nothing about the LKE volume lifecycle beyond one teardown.
  • publication.approved_by names the founder for an approval given to run this rehearsal, recorded in approval_receipt pointing at this file. It is not an approval of v0.12.23 for the pilot subscriber’s production.

​
13 · The state the channel was left in

Publishing five chart revisions over the day left targetRevision: "*" resolving to 0.12.35 while the entry’s chart.digest named 0.12.32 — the subscriber would have run a chart its own entry does not describe. A rehearsal that leaves that behind has broken the thing it was checking, so the channel was made coherent before the session ended: The hand-packaged 0.12.30 — the chart matching no release that started this whole investigation — was deleted, along with the four superseded rehearsal revisions. Final check with the commands VERIFY.md now prints:

​
Cluster

Destroyed. LKE 646952 deleted at the end of the session. No failure required keeping it. And its volumes, which the cluster deletion does not take with it. LKE’s default StorageClass is linode-block-storage-**retain**; the four PVCs this install created survived the cluster and had to be detached and deleted by hand, and the API reported them still attached to an already-deleted node for about four minutes — a detach/delete retry loop is required, not a single call. The lke provider profile already warns about this class (measured: 69 stranded PVs, $888/yr) and it is right to. 32 stranded volumes remain on the account from earlier sessions — not from this one, and not cleaned here because deleting other people’s data is not this agent’s call. A customer running the kit on LKE inherits the same behaviour and should be told: deleting the cluster does not delete the data, or stop the bill.

​
The single biggest thing between here and sending the kit

The signature-admission incompatibility. Everything else found today is fixed and proven. That one is not, and it is the only defect that breaks the central promise: your admission layer independently verifies our attestations before a byte runs. Today it cannot — and it fails in the direction that makes our own good releases look forged. A bank’s security review will ask to see that gate work. Until it does, the kit ships with its most quotable control switched off.