Greentic · Phase E

Keeping secrets out of the cluster

How Greentic changed the way digital workers get their passwords and API keys — so the secret values never get copied into Kubernetes.

✓ Complete Status snapshot · 27 June 2026 · all 5 steps shipped — proven end-to-end in a live cluster

The problem in one minute

A digital worker often needs secrets — a Telegram bot token, a Slack key, an LLM API key — to do its job.

The old way: when we deployed a worker to a Kubernetes cluster, we took the operator's local secrets file and copied the actual secret values into the cluster (as a Kubernetes "Secret" object). It worked, but it meant the real passwords now sat inside the cluster, where anyone with enough access could read them.

Think of it like this: instead of photocopying every key in the building and leaving the copies lying around the office, we now give each worker an ID badge that lets them open the safe only when they actually need something.

The new approach: an ID badge + a vault

In the new model — now live — no secret values are shipped into the cluster. Instead:

The worker pod gets its own identity — a Kubernetes ServiceAccount (the "ID badge").
A real, governed secret store — HashiCorp Vault — holds the actual secrets.
At runtime the worker shows its badge to Vault, Vault checks it, and hands back the secret it's allowed to see.
We only ship non-secret setup into the cluster: the badge and the Vault address. The passwords stay in Vault.

Old way: real secret values land in the cluster. New way: only the badge and the Vault address — the secrets stay in Vault.

Progress — the 5 steps

Built as a "vertical slice": one backend (HashiCorp Vault) wired all the way through, then proven live in a real cluster. More backends can follow the same seam.

E.1

Teach the vault client to log in with the badge Shipped

Added Kubernetes login to the Vault client: it takes the pod's ServiceAccount token and swaps it for a short-lived Vault token. No long-lived passwords baked in.

repo greentic-secrets · PR #94 · merged
E.2

Teach the runtime to fetch secrets from Vault Shipped

The worker runtime can now read secrets from Vault at run-time (read-only). Added a "vault" backend option and scoped each worker to its own tenant/team so it can't read other teams' secrets.

repo greentic-start · PR #303 · merged
E.3
Teach the deployer to set up the badge & stop copying secrets Shipped

When you deploy a Vault-backed environment, the deployer renders the worker with its ID badge (ServiceAccount), the Vault address and role, opens a network path to Vault — and skips creating the secret object entirely. The runtime then picks the Vault backend from a single pod env var at boot. The traffic router gets no secret access (it never needs one).
- Deployer renders the badge + Vault settings, no gtc-dev-secrets object (E.3a).
- Runtime selects the backend from GREENTIC_SECRETS_BACKEND on the worker pod (E.3b).
- Verified by rendering tests: badge present, Vault settings present, no secret object created.
repo greentic-deployer · PR #392 (E.3a) · repo greentic-start · PR #305 (E.3b) · both merged
E.4

Set up Vault itself Shipped

An idempotent setup script that prepares Vault: turn on Kubernetes login, create a read-only policy scoped to one environment + tenant, and bind the gtc-worker badge to it (plus KV-v2 + transit). The permissions side of the handshake.

demo provisioning · my_demos/k8s-vault-demo/vault-bootstrap.sh + vault.yaml
E.5
Prove it end-to-end in a real cluster Shipped · proven

A live demo in a local Kubernetes (kind) cluster with Vault running inside it. The real webchat-bot bundle deploys, the worker pulls its secret straight from Vault under its badge, and an inbound webhook returns HTTP 200 — while listing the cluster's secrets reveals nothing sensitive (no gtc-dev-secrets).
- Seeding writes secrets transit-encrypted into Vault — the plaintext value never leaves the operator host (op secrets put, built on a raw-write primitive in the core).
- Each messaging endpoint's webhook secret reference is scoped to the environment owner, so a single-tenant Vault env can store and resolve it.
- Audit confirms it: the worker pod's auth/kubernetes/login + KV read + transit-decrypt all show up in the Vault audit log.
repo greentic-secrets · PR #97 (raw write) · repo greentic-deployer · PR #394 (seed Vault) + PR #395 (ref scoping) · all merged · demo my_demos/k8s-vault-demo/deploy.sh

What the live run uncovered

The first live run looked like a runtime bug: the worker made zero calls to Vault and the inbound webhook returned 401. It turned out none of the code was at fault — every layer was proven correct in isolation.

Root cause: a demo networking gap. kind's network plugin enforces Kubernetes NetworkPolicies, and the deployer's deny-by-default posture also covered the in-cluster Vault pod. Worker egress was open, but nothing opened Vault's ingress to the workers — so the worker's login request was dropped at the network layer before it ever reached Vault.

Reproduced the read path working off-cluster (with both a root token and a freshly-minted worker badge token) — the secrets code was fine.
Ran the same binary in-cluster as a pod with the worker's badge — it reproduced the exact connect failure.
A plain test pod showed DNS resolved but TCP to Vault timed out, while the kubelet probe (which bypasses NetworkPolicies) passed — isolating the policy as the sole cause.

Fix: a one-line allow-vault-ingress-from-workers NetworkPolicy in the demo's vault.yaml. With it applied, the webhook returns HTTP 200 and the Vault audit shows the full login → read → decrypt chain. (A standalone in-cluster secret store like Vault sits outside the deployer's policy model — whoever deploys it must open its ingress.)

Why this matters

✓ Safer by default

Secret values never sit in the cluster or its database. A leak of cluster access no longer means a leak of every password.

✓ Real governance

Vault adds rotation, leases and an audit trail — who fetched what, when. The dev-store had none of that.

✓ No magic wiring

It reuses the worker's existing secret:// lookups — we just point them at a real backend instead of a local file.

✓ Least privilege

Only the worker that uses secrets gets the badge. The traffic router stays untouched.

For the curious — the technical bits

Mechanism	Workload identity + runtime resolution (not External Secrets Operator, not CSI driver).
First backend	HashiCorp Vault with Kubernetes auth — chosen because it runs fully in a local kind cluster end-to-end.
The badge	A Kubernetes ServiceAccount (`gtc-worker`); its projected token is exchanged at `auth/kubernetes/login` for a Vault token. A pod under the `default` SA is refused.
Deployer → runtime contract	The worker pod env carries `GREENTIC_SECRETS_BACKEND=vault` plus `VAULT_ADDR` / `VAULT_K8S_ROLE` (and KV mount/prefix when non-default). Non-secret values only.
Selection	Driven by the environment's bound Secrets pack: `greentic.secrets.dev-store` vs `greentic.secrets.vault`. Unknown kinds fail closed.
Seeding	Admin-side `op secrets put` writes a transit-wrapped envelope (AES-256-GCM data key wrapped by Vault transit) — a raw `vault kv put` would leak plaintext, so it isn't used.
Read-only at runtime	The worker resolves under a read-only Vault role; writes/deletes are rejected.
KV path	`{mount}/data/{prefix}/<env>/<tenant>/<team>/<category>/<name>`; the webhook ref's tenant is the environment owner.