Skip to content

Operator manual

Production-oriented guide for platform teams installing and operating Kollect for tenant workloads. If you are evaluating locally, start with Quick start or Kind local lab first.

Assumptions

This guide assumes Helm 3, kubectl, a working Kubernetes cluster, and cert-manager for the default webhook-enabled chart. New to Kollect CRDs, sink roles, or watch scope? Read Understand the basics and Platform decisions before changing production values.

Pre-beta API

v1alpha1 fields and defaults may change until the first release candidate. Check ROADMAP before production rollout.

In this manual

Topic Page
Install (Helm, OCI, CRDs, tenant mode) Install (below)
Version upgrades (CRD + operator two-step) Upgrading Kollect
Helm values (production knobs) Helm values reference
Prometheus metrics and alerts Operator metrics
Sink and webhook secrets Secrets (below)
Informer scope and tenancy Watch scope (below)
Replicas and leader election High availability (below)
Read-only UI (early adopter preview) Read-only UI · ADR-0409 · UI local dev (mock)

Install

Kollect ships as a Helm chart (charts/kollect) with CRDs in crds/ and the controller Deployment in templates/. Chart structure and install modes are documented in ADR-0704: Helm chart and CRD lifecycle.

From the repository

helm install kollect ./charts/kollect -n kollect-system --create-namespace

From GHCR (OCI)

Published releases push the chart to GHCR (ADR-0705):

helm install kollect oci://ghcr.io/platformrelay/kollect -n kollect-system --create-namespace

Omitting --version installs the latest published chart. In production, pin a specific version (e.g. --version 0.5.0 — see the releases page).

Pin image.tag to the release image when not using the chart default — see Helm values reference.

Lab clusters where GHCR is denied

A live OCI pull is the preferred path. When package ACLs return 403 on a disconnected lab (for example a Talos driving-range), use the fallback in Lab image bootstrap when GHCR is unavailable (403).

CRDs (first install)

Helm installs CRDs from crds/ on first install only. Apply the release CRD bundle explicitly when installing from raw manifests or when you need a known schema version:

kubectl apply -f dist/install-crds.yaml

Two install artifacts

Day-2 upgrades treat CRD schema and operator Deployment as separate steps — see Upgrading Kollect. Full operator manifest: dist/install.yaml.

For tenant isolation, enable namespaced RBAC and restrict the informer cache (ADR-0203, ADR-0201):

tenantMode: true
watchNamespaces:
  - team-a
mode: single
featureGates:
  inventoryHttp:
    enabled: false

helm install kollect ./charts/kollect -n kollect-system --create-namespace -f values-team.yaml

Namespaced KollectProfile, family sinks, KollectTarget, and KollectInventory live in the team namespace. Portal read path uses Postgres or Kafka sink export — not direct operator HTTP.

Full value reference: Helm values.

Secrets

Sink credentials

Never put passwords or tokens on family sink CRs (KollectSnapshotSink, KollectDatabaseSink, KollectEventSink). Reference Kubernetes Secrets instead:

Sink family Backends Secret keys Field
KollectDatabaseSink Postgres, MongoDB dsn, url, connectionString, or DATABASE_URL spec.postgres.databaseRef / MongoDB ref
KollectSnapshotSink Git, GitLab, S3, GCS deploy key or token spec.secretRef
KollectEventSink Kafka, NATS broker credentials spec.secretRef

Maturity tiers (Core / Beta / Planned): Roadmap — Supported & planned sinks.

Walkthrough: Postgres state store.

Credentials in Secrets only

Inline credentials on CRs are rejected by policy and leak in kubectl get -o yaml. Store DSNs and tokens in Secrets; grant the operator ServiceAccount read access via RBAC.

TLS trust for sinks uses caBundle or caSecretRef on the sink spec — same resolution as export and connection probes (ADR-0403).

Git snapshot sinks (spec.git.engine)

spec.git.engine Runtime needs Notes
go-git (default) None beyond the manager binary Pure Go transport; works on minimal images
cli git and openssh-client in PATH Native clone/commit/push; SSH uses GIT_SSH_COMMAND with openssh-client

The published operator image (ghcr.io/platformrelay/kollect) ships Debian bookworm-slim with git, openssh-client, and ca-certificates on UID/GID 65532. Default go-git export is unchanged; the image change enables engine: cli and full git ls-remote connection probes. Custom images built from an older distroless base must install git (and openssh-client for SSH) when using engine: cli.

Webhook serving certificate

Validating webhooks require a TLS serving cert mounted on every manager replica (ADR-0105):

Webhook TLS

  • Default: cert-manager Certificate in webhook-certmanager.yaml. cert-manager is required for the default chart values.
  • Operator-provided certificates: set webhooks.certManager.create: false only after arranging the named serving Secret and CA trust for the ValidatingWebhookConfiguration. This setting suppresses cert-manager resources; it does not generate a certificate.
  • Development only: webhooks.enabled: false avoids the TLS dependency by disabling validating admission. Do not use this as the production workaround.

Example: Cert-manager webhooks.

Production connection tests

Production sink manifests should use spec.connectionTest: false (chart default) and trigger probes with the kollect.dev/test-connection: "true" annotation when needed. CI and samples may set connectionTest: true (ADR-0403).

Watch scope

Kollect collection scope is controlled at three layers:

Helm: watchNamespaces and tenantMode

Setting Effect
watchNamespaces: [] Informer cache watches all namespaces (cluster-wide install)
watchNamespaces: [team-a, team-b] Cache restricted to listed namespaces
tenantMode: true Namespaced Role/RoleBinding instead of ClusterRole

Use tenantMode: true + watchNamespaces for per-team operator installs (ADR-0203). Team path profile: Team-owned operator (minimal RBAC). Example: Multi-tenant watch scope.

Helm keys: Helm values reference.

KollectScope allow-lists

Optional KollectScope CRs enforce GVK, namespace, and sink allow-lists. Violations set Degraded on affected pipelines (ADR-0203).

Watch labels and annotations

Teams can opt individual namespaces or resources in or out without changing Helm values (ADR-0205):

Key Values Effect
kollect.dev/watch (label) enabled / disabled Opt in or out a namespace or resource
kollect.dev/namespace-watch (annotation) enabled / disabled Opt in or out all resources in a namespace

KollectTarget.spec.watchMode defaults to All. Set watchMode: OptIn to collect only enabled namespaces/resources.

High availability

Controller HA relies on controller-runtime leader election: scale replicaCount and let one elected leader own reconciliation while standby replicas wait.

Summary for operators:

Concern Default chart Production guidance
Controller replicas replicaCount: 1 replicaCount: 2+, leaderElection.enabled: true
Duplicate exports Prevented by leader election Never set replicaCount > 1 with leaderElection.enabled: false
Webhooks Served on every ready replica Apiserver targets webhook Service; not gated by leader election
Memory at scale resourcesProfile default Use resourcesProfile: large for 100k-row design target (ADR-0603)

Single-cluster mode only

The operator runs mode: single. Multi-cluster fleets run one single-mode operator per cluster exporting to a shared sink, partitioned by spec.cluster — there is no hub/spoke runtime tier (ADR-0501).

Lab image bootstrap when GHCR is unavailable (403)

Fallback only — not a production path

A live OCI pull from GHCR is the preferred install path (see Install). Use this bootstrap only when package ACLs deny access — for example a disconnected Talos driving-range where ghcr.io/platformrelay/kollect:<tag> returns 403 to anonymous and authenticated pulls, and a personal ghcr.io push is also denied (no write:packages). It stands up a plain-HTTP lab registry and a temporary Talos registry mirror; both are insecure by design and must be torn down after the run.

The bootstrap has three parts: install the chart from the GitHub Release asset, rebuild the manager image from the release git tag into a lab registry, and point a temporary Talos registry mirror at that registry so nodes resolve ghcr.io pulls locally.

Set these placeholders for your lab (do not hard-code a single lab's addresses):

export LAB_HOST_IP=192.168.100.1                 # lab host IP reachable by Talos nodes
export TALOS_NODES=192.168.100.10,192.168.100.11,192.168.100.12  # all control-plane + workers
export VERSION=0.10.0                             # target release, matches the chart/image tag

1. Chart from the GitHub Release asset

Install the chart from the release .tgz instead of the OCI oci://ghcr.io/platformrelay/kollect pull:

curl -sSLO https://github.com/platformrelay/kollect/releases/download/v${VERSION}/kollect-${VERSION}.tgz
helm install kollect ./kollect-${VERSION}.tgz -n kollect-system --create-namespace -f values.yaml \
  --set image.tag=${VERSION}

The chart keeps its default ghcr.io/platformrelay/kollect repository — step 3 makes nodes resolve that host from the lab registry, so no image.repository override is needed. You must, however, set image.tag=${VERSION}: the chart's default tag is latest, but step 2 pushes only :${VERSION} to the lab registry (the mirror rewrites the host, not the tag), so a default-tag install would try to pull a :latest image that was never pushed and fail with ImagePullBackOff.

2. Rebuild the manager image into a dual-bound lab registry

Run a host registry bound to both loopback (for an insecure local push — Docker treats 127.0.0.1 as insecure automatically) and the node-facing lab IP (so Talos nodes can pull):

docker run -d --restart=always --name kollect-dr-registry \
  -p 127.0.0.1:5000:5000 -p ${LAB_HOST_IP}:5000:5000 registry:2

Rebuild the manager image from the release git tag and push it under the platformrelay/kollect repository path (so the mirror in step 3 resolves the same path GHCR would):

git fetch --tags
git checkout v${VERSION}
docker build -t 127.0.0.1:5000/platformrelay/kollect:${VERSION} .
docker push 127.0.0.1:5000/platformrelay/kollect:${VERSION}

Rebuilt digest differs from the release

This image is rebuilt from source, so its digest will not match the published release digest and cosign verification against the release will not apply. That is acceptable for a lab; never ship a self-rebuilt image to production.

3. Temporary Talos registry mirror (with mandatory revert)

Point the ghcr.io mirror at the lab registry on every node. Applying the whole /machine/registries block (absent by default) keeps the revert a clean wholesale remove:

cat > /tmp/registries.patch.yaml <<EOF
- op: add
  path: /machine/registries
  value:
    mirrors:
      ghcr.io:
        endpoints:
          - http://${LAB_HOST_IP}:5000
EOF
talosctl -n ${TALOS_NODES} patch mc --patch @/tmp/registries.patch.yaml

If nodes already define machine.registries

The add above assumes the block is absent (the driving-range default). If your nodes already carry a machine.registries block, merge the ghcr.io mirror into it instead — and on revert restore the prior block rather than removing the whole path.

Cleanup / revert is mandatory. After the run, remove the mirror and confirm no node still references the lab host:

# revert: remove the entire block added above
talosctl -n ${TALOS_NODES} patch mc --patch '[{"op":"remove","path":"/machine/registries"}]'

# verify machine.registries no longer references the lab host (expect no output)
talosctl -n ${TALOS_NODES} get mc v1alpha1 -o yaml | grep -F "${LAB_HOST_IP}:5000"

# tear down the lab registry container
docker rm -f kollect-dr-registry

Then uninstall the release and delete the lab namespace as usual (helm uninstall kollect -n kollect-system). Once the org GHCR ACL is fixed, revert to the live OCI pull — this bootstrap should not outlive the access outage that forced it.

See also