Skip to content

Troubleshooting

Central hub for diagnosing Kollect collection, export, and sink failures — what operators see and what to do about it. For error-class semantics and reconcile behavior, see ADR-0602: Error taxonomy. Step-by-step install and upgrade live in the Operator manual; per-scenario walkthroughs in Examples.

First checks

When export stalls, run kubectl describe on the sink and inventory — ConnectionVerified, SinkReachable, and Synced conditions usually pinpoint credential, namespace, or selector issues before diving into controller logs.

No hub/spoke runtime

There is no KollectHub CRD — hub/spoke ingest was removed in v0.3. Multi-cluster uses N single-mode operators, each writing a cluster-partitioned record to a shared sink with spec.cluster set. There are no hub-specific conditions, metrics, or failure modes in current releases (ADR-0501, Multi-cluster fleet).

How errors surface

Kollect reports failures through four channels. Start with conditions on the CR that owns the pipeline; use metrics and logs when status is stale or ambiguous. Kollect follows Kubernetes condition conventions (Ready, Synced, Degraded) with sink-specific types on the family sinks.

Conditions

Condition Typical objects Meaning
Ready KollectTarget, KollectInventory, KollectClusterInventory Pipeline healthy enough to collect or export
Synced Inventory / cluster inventory / target Last export or collection cycle outcome (aggregate across sinks)
PartiallySynced Inventory (Ready or Synced reason) Some sinks exported in the pass; others failed
Degraded Target, inventory, sink family CRs Terminal misconfig or export gate — spec change required
ExportShardWarning KollectInventory Namespace aggregate ≥ ~1,800 rows — split before cap
SinkReachable Inventory / target Sink ref resolves and backend reachable
ConnectionVerified KollectSnapshotSink, KollectDatabaseSink, KollectEventSink Last connectivity probe succeeded (credentials, TLS, network)

Per-sink detail lives in status.sinkExports[] — each entry has its own Synced condition and lastExportTime. Condition semantics in brief: Conditions and status.

A sink can show ConnectionVerified=True while inventory shows SinkReachable=False if the name or namespace in a *SinkRefs entry is wrong — fix the reference, not just credentials.

Conditions by kind

Kind Conditions When to inspect
Family sinks (KollectSnapshotSink, KollectDatabaseSink, KollectEventSink) ConnectionVerified, TLSInsecure, Degraded Before export; credential or endpoint problems
KollectTarget Ready, Synced, Degraded, SinkReachable Collection stalled or scope denied
KollectInventory Ready, Synced, Degraded, SinkReachable Export not running or payload errors
KollectConnectionTest ConnectionVerified, Ready Audited composite probes

Static kinds (KollectProfile, KollectScope) do not set Ready — admission webhooks and events surface validation errors instead.

Full per-kind tables: KollectInventory, KollectTarget, KollectClusterTarget, KollectSnapshotSink.

Common Degraded reasons

Reason Typical object Cause Fix
SinkNotFound Inventory, Target Typo or wrong namespace in *SinkRefs Match exact sink name in same namespace
SinkUnreachable Inventory, Target ConnectionVerified=False on sink Fix Secret, DSN, network; re-probe sink
ScopeSinkDenied Inventory Sink not in KollectScope allow-list Add sink to scope allow-list refs
ScopeGVKDenied Target, ClusterTarget GVK blocked by scope Update allowedGVKs on KollectScope (Target) or KollectClusterScope (ClusterTarget)
ScopeNamespaceDenied Target, ClusterTarget Workload namespace blocked; on ClusterTarget, profileRef.namespace outside allowedStaticRefNamespaces Add to allowedNamespaces, or permit the profile namespace
ProfileNotFound Target Missing KollectProfile Apply profile in same namespace as target
PayloadTooLarge Inventory Exceeds maxExportBytes Split targets or trim attributes
ExportTerminal Inventory Non-retryable sink error Fix sink config; check operator logs
CleanupTerminal Inventory / ClusterInventory (deleting) Sink cleanup failed non-retryably while the object was deleting (e.g. revoked git token); the cleanup finalizer is retained so deletion waits for the operator Fix the sink credential/config so cleanup can reach the backend, or set kollect.dev/force-cleanup: "true" to drop the finalizer and accept the exported objects stay behind; alert rule: increase(kollect_cleanup_terminal_total[1h]) > 0
Suspended Target, Inventory spec.suspend: true Set suspend: false
Progressing Inventory Transient network or 429 Usually self-heals; inspect metrics

Events

Warning events carry stable reason enums (not free-form types). Common reasons: ScopeGVKDenied, PayloadTooLarge, ExportFailed, Progressing, ConnectionTestFailed, ReconcilePanic. Deletion-time reasons: CleanupRetainedByPolicy (Normal: the snapshot sink's deletionPolicy is Retain, the default — its exported objects were deliberately left in place, ADR-0421), CleanupSharedIdentity (a Delete retraction was skipped because a KollectClusterInventory and a KollectInventory in namespace cluster with the same name share the export identity), CleanupSinkGone (a bound sink CR was already gone when the inventory finalized — deletion proceeds, but objects that sink had exported may remain in its backend), CleanupRetained (the backend cannot retract previously exported data — event streams — or a full retraction cannot be proven: git layout-tree files that can interleave with other inventories' trees, an export auto-upgraded to the per-resource tree (visible from the recorded lastExportPaths in the inventory's sinkExports status), or a pathTemplate with {generation} leaving past-generation objects on an object store; deletion proceeds and the event names the retained identity — a default git/gitlab sink whose recorded export was the plain document and whose candidates were all retracted stays silent), CleanupForced (kollect.dev/force-cleanup dropped the finalizer without backend contact), CleanupTerminal (see the Degraded reasons table above — this one does block deletion).

Metrics

Metric Labels Use
kollect_reconcile_errors_total kind, error_class Reconcile failures: transient, terminal, forbidden
kollect_sink_errors_total reason Export failures — separate from reconcile errors
kollect_sink_connection_test_total type, result Probe outcomes per sink family
kollect_export_duration_seconds sink_type Slow exports (Git clone, Postgres bulk, etc.)
kollect_workqueue_depth controller Reconcile backlog / conflict storms

Sink error reason values include: transient, terminal, forbidden, payload_too_large, spill_required, unknown.

Full catalog: Operator metrics.

Inventory status.sinkExports[]

Each bound sink (snapshotSinkRefs, databaseSinkRefs, eventSinkRefs) gets a status slice entry:

Per-sink Synced Operator read
True, reason Exported Last attempt succeeded
True, reason Debounced Up to date, not a failure — payload unchanged since the last export, write skipped; the message names the interval (re-exported every 15m0s, or exports only on change (interval 0s))
False, reason ExportFailed Export attempt failed — read message

lastExportTime and lastChecksum keep the last successful export to that sink across debounced and failed passes. The read API reports each bound sink from its own <family>/<name> entry: debounced for reason Debounced, ok for any other True, degraded for False, with that entry's lastExportTime (empty when the sink never exported) (metrics note).

Diagnostic commands

# Pipeline status (short names — family sinks: ksnap/kdb/kevt)
kubectl get kprof,ksnap,kdb,kevt,ktgt,kinv -n <namespace>
kubectl describe kollectsnapshotsink <name> -n <namespace>
kubectl describe kollectdatabasesink <name> -n <namespace>
kubectl describe kollecttarget <name> -n <namespace>
kubectl describe kollectinventory <name> -n <namespace>
kubectl get events -n <namespace> --field-selector involvedObject.name=<name>

# Wait for sink probe (pick the family kind you installed)
kubectl wait --for=condition=ConnectionVerified kollectdatabasesink/<name> \
  -n <namespace> --timeout=60s

# Re-probe without editing spec
kubectl annotate kollectdatabasesink <name> -n <namespace> \
  kollect.dev/test-connection=true --overwrite

# Operator logs
kubectl -n kollect-system logs deployment/kollect-controller-manager -f --tail=200

Production sinks should keep spec.connectionTest: false and use the annotation for ad-hoc probes (ADR-0403). More shortcuts: Command reference.

Error classes (ADR-0602)

Class Meaning Reconcile Self-heal?
Transient Network blip, 429, API conflict, sink timeout, circuit breaker open Requeue with backoff; Synced=False, reason Progressing Yes — when root cause clears
Terminal Bad config, invalid extraction path, auth permanently wrong, payload over cap No requeue; Degraded=True + Warning event No — fix spec/credentials, then observe new generation
Forbidden SAR/RBAC denied for list/watch on a namespace/GVK Degrade scope; partial collection; metric error_class=forbidden Partial — grant RBAC or narrow selectors

Details, examples, and circuit-breaker rules: ADR-0602.

Catalog by symptom

Each row: what you see → likely cause → how to confirm → fix → escalate when stuck.

Scope and tenancy

Symptom Likely causes Identify Handle Escalate
Target Degraded, reason ScopeGVKDenied KollectScope allow-list excludes profile GVK Event reason; scope CR in same namespace Add GVK to scope or remove scope binding Platform team owns scope policy
Inventory Degraded, reason ScopeSinkDenied Scope disallows snapshot/database/event family ref kubectl describe kollectinventory; scope spec Use allowed sink family/type or widen scope Same
Target Degraded, reason ScopeNamespaceDenied Target/intent namespaces outside scope Event + target spec namespaceSelector Fix selectors or scope allowedNamespaces Same
Target Degraded, reason Forbidden SAR denied for list in workload namespace kollect_reconcile_errors_total{error_class="forbidden"}; target message cites namespace/GVK Grant operator Role/ClusterRole list/watch on GVR; or narrow target to permitted namespaces RBAC audit / cluster admin
Namespace empty in inventory (or status.itemCount is 0) although a target exists namespaceSelector mismatch, watch opt-out, spec.suspend: true, or scope deny Target Ready; status.itemCount=0; labels kollect.dev/watch Align selector with workloads; check watch labels and Annotations and labels If SAR OK but still empty — profile/GVK issue

Collection

Symptom Likely causes Identify Handle Escalate
Target Degraded, ProfileNotFound Wrong profileRef or cross-namespace ref kubectl get kollectprofile -n <target-ns> Create profile in same namespace as target —
Target Degraded, InformerRegistrationFailed Unknown/uninstalled GVK, CRD missing Target message; apiserver discovery Install CRD/API; fix KollectProfile.spec.targetGVK Vendor CRD not on cluster
Target Degraded, AccessCheckFailed SAR API error (not denial) during list pre-check Logs: access check failed; transient error metric Fix apiserver connectivity; check operator pod network Sustained apiserver errors
Target Degraded, Forbidden (collection) List denied for namespace error_class=forbidden; engine marks forbidden scope Fix RBAC or reduce target scope —
Partial/empty attributes CEL/JSONPath eval error on object Logs: extract attributes (no secret values logged) Fix attribute paths in profile; test with kubectl explain sample Webhook should catch most invalid paths at admission
Growing kollect_watch_map_list_errors_total List failure registering watch map handler PromQL increase; controller logs on inventory/target Fix RBAC for mapped GVR; check apiserver load API server degradation

Export — payload size

Symptom Likely causes Identify Handle Escalate
Inventory Degraded, PayloadTooLarge Monolithic export > ~1.5 MiB (maxExportBytes) Condition message with byte counts; kollect_sink_errors_total{reason="payload_too_large"} Shard: multiple KollectInventory per namespace (<~2k rows each) Architecture review for 10k+ row namespaces
Inventory Degraded, SpillRequired Payload exceeds the 1 MiB inline cap and no object-store sink is configured (no spill write path — the export is not buffered anywhere) Reason SpillRequired; spill_required metric Add KollectSnapshotSink type s3 or gcs to inventory refs —
ExportShardWarning=True ≥ ~1,800 rows in one namespace aggregate Condition + increase(kollect_export_shard_warn_total[1h]) Split inventories before hard cap See Performance and scalability
kollect_export_spill_warn_total increasing Payload ≥ 1 MiB warn threshold Metric + log export payload exceeds spill warn threshold Shard or tune spec.maxExportBytes (within global cap) —

Export — sink backends

Family CRDs: KollectSnapshotSink (git/gitlab/s3/gcs), KollectDatabaseSink (postgres), KollectEventSink (kafka/nats). Inventory refs must use the matching family field (snapshotSinkRefs, etc.) in the same namespace.

Symptom Likely causes Identify Handle Escalate
Git push failures (NFF) Remote ahead of operator; concurrent writers Logs git push; terminal or transient on sink errors pushPolicy: Commit retries merge+push; ensure single writer per branch/path Protected branch / hook rejects — platform Git admin
Git auth terminal Bad token, expired credential, 401/403 Sink ConnectionVerified=False; ConnectionTestFailed event Rotate secretRef; re-annotate kollect.dev/test-connection=true. The next export uses the new Secret, but a terminal error does not requeue and no controller watches Secrets, so the annotation is what starts that export IdP / Git provider outage
Export fails with secret "<name>" not found in namespace "<namespace>" A Secret the sink references (credentials, database or CA) was deleted or renamed Inventory Degraded, reason Progressing (retryable) when every sink fails, else that sink's entry in status.sinkExports[]; the message names the Secret, never a value Recreate the Secret. The next retry exports with no change to the sink or inventory: within the export interval, or within 30 s for exportMinInterval: 0s —
Export fails with secret "<name>" in namespace "<namespace>" has no data The referenced Secret exists but every value is empty (also for a caSecretRef Secret) As above; kollect-pipeline prints the same text for an empty Secret manifest Put the keys back. Recovers like the row above. S3/GCS do not fall back to ambient credentials for such a Secret —
Rotated Secret not in use yet Nothing has exported since the change: a Secret change alone does not trigger an export lastExportTime in status.sinkExports[] predates the Secret change Wait for the export interval; for exportMinInterval: 0s, set kollect.dev/requestedAt on the inventory to export now —
Postgres connection failures DSN, network policy, TLS, pool timeout Database sink conditions; transient sink errors; connection test metric Verify Secret keys, egress NetworkPolicy, server reachable DBA for server-side limits
Postgres rows stale while status looks healthy Upsert-only drift, or an export error on that ref Compare lastExportTime on the database entry in status.sinkExports[] Follow Postgres state store — Troubleshooting DBA for server-side limits
S3/GCS 403 IAM, wrong bucket, signature Export logs; terminal/forbidden Fix credentials and bucket policy Cloud IAM review
sink circuit breaker open in logs 5 consecutive counted backend write failures to one sink (breaker key <sink namespace>/<sink name>) within one 60 s counting interval (the counts reset every 60 s while the breaker is closed). Counted: transient, forbidden and unclassified errors, and sink-wide terminal errors such as a Git auth or protected-branch rejection. A failure about one inventory (an ownership conflict on its own paths) does not count; it still fails that inventory's export (status, Warning event, logs) Sink errors, then sink circuit breaker open for "<namespace>/<name>" (transient) for ~30 s; exports to that sink do not reach the backend while it is open Fix the backend. After 30 s one trial export is admitted: success closes the breaker, a counted failure re-opens it for another 30 s, an inventory-scoped result (ownership conflict) leaves it half-open so the next export is the trial Backend SLA breach
SinkReachable=False, SinkNotFound Wrong sink name or cross-namespace ref in *SinkRefs Inventory message; kubectl get kollect*sink -n <inv-ns> Fix ref name; create sink in inventory namespace —
SinkReachable=False, SinkUnreachable Backend down despite CR present; bad DSN or TLS failure Sink ConnectionVerified; probe annotation Fix network/credentials first —
Inventory / namespace stuck Terminating Sink cleanup failed terminally during deletion; finalizer retained Events reason CleanupTerminal on the inventory; increase(kollect_cleanup_terminal_total[1h]) > 0; Degraded reason CleanupTerminal Fix the sink backend and let the next reconcile finish cleanup — the terminal wedge re-checks every 5 minutes, so a Secret-only credential fix (no watch fires on Secret edits) clears within minutes; or accept the exported objects and set kollect.dev/force-cleanup: "true" on the deleting object to drop the finalizer —
Deletion left files in git/bucket after inventory removal The snapshot sink's deletionPolicy is Retain — the default; a Normal CleanupRetainedByPolicy event names the sink (ADR-0421). With Delete: the sink CR was deleted before the inventory (CleanupSinkGone), a full retraction cannot be proven (CleanupRetained: event streams, per-resource layout trees, {generation} templates, gitlab merge_request mode whose deletion MR was never merged, a pathTemplate/format change after the last export — visible from the recorded lastExportPaths in sinkExports status — or an export from before those paths were recorded), the export identity is shared with a same-named inventory in the other scope (CleanupSharedIdentity), or force-cleanup was used Events on the deleting inventory name the sink and the retained identity Set deletionPolicy: Delete on the sink if deletions should retract; otherwise retract manually in the backend (delete inventory/<ns>/<name>.* and .part-* siblings, parquet name=<name>/ and name=<name>.part-*/ partitions, plus any path the old sink config rendered) —
Synced=False with nothing else obvious A prior export attempt failed Manager logs plus the Degraded condition on the inventory Fix the reported sink error; the next cycle re-exports —

Export — debounce (not a failure)

Symptom Likely causes Identify Handle Escalate
Synced=True, reason Exported, but no new write in the backend Per-sink exportMinInterval; unchanged payload checksum status.sinkExports[].conditions True, reason Debounced; no kollect_sink_errors_total spike Expected — the sink is up to date and re-exports when the interval elapses or the payload changes; tighten the interval only if SLA requires fresher writes Mistaking debounce for outage
Some sinks Exported, others Debounced, aggregate Synced=True Dual cadence (e.g. Postgres 30s + Git 1h) Per-sink reasons in status.sinkExports[]; kollect_export_debounced_total up Normal — the aggregate message counts both as up to date (2/2 sink(s) up to date) —
Stale data in Postgres but Synced=True on Git ref Different intervals per ref Compare lastExportTime per sinkExports entry Set ref-level exportMinInterval intentionally —

Connection test

Symptom Likely causes Identify Handle Escalate
Sink ConnectionVerified=False, ConnectionTestFailed TLS verify fail, missing secretRef, wrong Secret key, wrong endpoint kubectl describe sink; kollect_sink_connection_test_total{result="failure"} Fix secretRef, CA bundle, URL; one-shot: annotation kollect.dev/test-connection=true Corporate TLS inspection
Sink fails with spec.tls: … (terminal, no retry) A CA that cannot take effect: a caSecretRef Secret with data but no CA key (any type; an entirely empty one is the retryable has no data row above), both caBundle and caSecretRef (any type), PEM without a certificate, an S3/GCS http endpoint, a Postgres sslmode/sslrootcert conflict, a MongoDB tlsCAFile, TLS-off or tlsInsecure URI The error names the setting; sink status and connection test Fix the spec or Secret per snapshot / database TLS trust; after fixing a Secret, change any annotation on the sink (kubectl annotate <kind> <name> ca-fixed-at="$(date +%s)" --overwrite), since no controller watches it —
Postgres sink with a CA fails on the hostname after upgrade Unset sslmode with a CA now means verify-full; the DSN host is not in the certificate SAN Verification error naming spec.tls Use a DSN host in the SAN, or set sslmode=verify-ca explicitly —
KollectConnectionTest stuck false One-shot CR probe failed kubectl describe kollectconnectiontest Same as sink probe; check spec.sinkRef family field —
TLSInsecure=True on sink Explicit insecure TLS (non-default; only possible with --allow-insecure-sinks) Condition on sink Prefer proper CA; document exception per security policy Security review
Sink rejected or export fails with tls.insecureSkipVerify is not permitted Sink sets spec.tls.insecureSkipVerify but the manager runs without --allow-insecure-sinks (K-14, default since the upgrade) Admission error, sink status / manager log Supply the server CA (spec.tls.caBundle / caSecretRef) and remove the field; or, for development only, set Helm allowInsecureSinks: true Security review

Reconcile and workqueue

Symptom Likely causes Identify Handle Escalate
Slow inventory updates, no Degraded Optimistic-lock conflicts, high churn kollect_workqueue_depth sustained; conflict requeues in logs Raise maxConcurrentReconciles; increase exportMinInterval; shard inventories etcd/apiserver slow
Event ReconcilePanic Unexpected panic (should not crash pod) Event reason; log reconcile panic recovered Upgrade to fixed release; file bug with stack from logs Repeat panics on same controller
kollect_collect_dispatch_backpressure_total rising Dispatch queue saturated — informer events blocking on enqueue Metric + dispatch queue depth Increase collect.dispatchWorkers / dispatchQueueSize CPU throttle on controller
Status update lag Many inventories, frequent export Reconcile duration p95; etcd metrics Debounce, sharding, fewer sinks per inventory Load test runbook

Webhook vs runtime validation

Symptom Likely causes Identify Handle Escalate
kubectl apply rejected CEL validation on CRD, validating webhook Admission error message (no object created) Fix spec before create —
CR accepted but Degraded at runtime GVK/CRD absent on cluster, scope enforced only at reconcile, SAR not checked at admission Compare admission vs kubectl describe conditions Install CRDs; fix runtime-only constraints Gap between webhook and runtime — upstream issue
Scope ceiling on cluster targets KollectClusterScope webhook deny Forbidden on apply Adjust cluster target GVKs to allowed set —
A working CR stopped reconciling right after an upgrade Pre-beta CRD schema change without conversion kubectl explain the new CRD; compare against the applied spec Re-apply install-crds.yaml, then fix renamed or removed fields — see Upgrading —

Multi-sink partial success

Symptom Likely causes Identify Handle Escalate
Ready=True, PartiallySynced; Postgres OK, Git failed Independent per-sink export; some sinks exported in the pass, some failed status.sinkExports[] — mixed Exported / ExportFailed Fix failing sink only; successful sinks stay current —
Synced=False, PartiallySynced; some failed One backend terminal while others exported in the same pass Failed count in condition message (1/2 sinks exported; 1 failed) Terminal sink needs spec/cred fix; others self-heal —
Aggregate Synced=False with no sink exported, one per-sink entry True and one False One sink was skipped as up to date (payload unchanged), the other failed status.sinkExports[]: one entry True, the other ExportFailed Fix the failing sink; an up-to-date sink never turns a failure into Synced=True —
Aggregate Synced=False, all per-sink failed Shared payload inline-cap gate before export Inventory-level Degraded + SpillRequired/PayloadTooLarge reasons Fix size/sharding first —

Multi-cluster fleet

Symptom Likely causes Identify Handle Escalate
Fleet rows in the shared sink are missing a cluster Wrong or empty spec.cluster on that cluster's inventory Compare the cluster partition in the sink backend against spec.cluster Set a unique spec.cluster per cluster — see Multi-cluster fleet —

Resources (brief)

Symptom Likely causes Identify Handle Escalate
OOMKilled controller Large collect store, cluster-wide informers Pod status; kollect_collect_items_total; RSS resourcesProfile: large; namespace-scope targets; shard inventories Load test runbook
CPU throttle High dispatch/reconcile load pprof; kollect_reconcile_duration_seconds p95 Raise limits; tune workers; reduce churn —
etcd / API slow Status write rate, large fleets Apiserver metrics; many inventories Longer export intervals; fewer status transitions Platform cluster health

PromQL cheat sheet

Tie queries to symptoms (adjust namespace/job labels for your scrape config):

# Sustained reconcile failures by class
sum(rate(kollect_reconcile_errors_total[5m])) by (error_class)

# Inventory export errors only
sum(rate(kollect_reconcile_errors_total{kind="KollectInventory"}[5m])) by (error_class)

# Export failure reasons (auth, size, transient, …)
sum(increase(kollect_sink_errors_total[15m])) by (reason)

# Slow exports — Git vs Postgres vs event
histogram_quantile(0.95, sum(rate(kollect_export_duration_seconds_bucket[5m])) by (le, sink_type))

# Debounce (expected) vs failure — debounce should NOT correlate with sink_errors
sum(rate(kollect_export_debounced_total[5m])) by (controller)

# Workqueue backlog — conflict storms or under-provisioned workers
max_over_time(kollect_workqueue_depth[10m])

# Collect store growth — OOM/sharding signal
kollect_collect_items_total

# Approaching export shard cap
increase(kollect_export_shard_warn_total[1h])

Default alert rules: metrics.md — Default alerts.

Log patterns

Structured controller logs (logr). Grep operator pod logs (namespace typically kollect-system):

Key / message fragment Indicates
error_class transient / terminal / forbidden on wrapped errors
reason Inline-cap gate, export failure, scope denial (stable enum)
inventory, target Which CR pipeline
sink Backend key during export
access check failed SAR API error → target AccessCheckFailed
extract attributes CEL/JSONPath failure on a resource
export failed Sink export path
export payload exceeds spill warn threshold Approaching 1 MiB — shard soon
debounced Export skipped by interval/coalesce
sink circuit breaker open Repeated sink failures (transient, forbidden or sink-wide terminal); see Export — sink backends
reconcile panic recovered Panic converted to requeue (EC-P2-01)
git push / git auth failed Snapshot sink transport

Never expect secrets, tokens, or full payloads in logs.

kubectl logs -n kollect-system deploy/kollect-controller-manager --tail=500 \
  | rg 'export failed|PayloadTooLarge|debounced|access check failed|circuit breaker'

Common questions

Why do CRD schema changes not apply on helm upgrade?

Helm installs CRDs from crds/ on first install but does not upgrade them on helm upgrade. Kollect documents an explicit two-step path: kubectl apply -f dist/install-crds.yaml, then helm upgrade (ADR-0704, Operator manual — Upgrade).

Should I delete CRDs to fix a schema mismatch?

No. Deleting a CRD garbage-collects all custom resources — including the ones you are trying to save. Apply the release CRD bundle before upgrading the controller and follow Upgrading; never delete CRDs in production.

# kollect-doc: ignore Helm values, not a kollect CR
tenantMode: true
watchNamespaces:
  - team-a

See Operator manual — Watch scope and Multi-tenant watch scope.

Why does my inventory show SinkNotFound or SinkReachable=False?

KollectInventory binds sinks via typed lists — spec.snapshotSinkRefs, spec.databaseSinkRefs, and/or spec.eventSinkRefs — naming family sink objects in the same namespace as the Inventory. Cross-namespace sink refs are not supported for namespaced inventory (ADR-0201, ADR-0414). List the sinks in the inventory namespace with the diagnostic commands above.

The same rule applies to KollectTarget.spec.profileRef → KollectProfile in the target namespace, and KollectConnectionTest.spec.sinkRef → a family sink in the test namespace.

Same-namespace sink refs

Create family sinks in the same namespace as KollectInventory before expecting export. Cluster-wide rollup uses KollectClusterInventory with spec.sinkNamespace instead.

I moved the sink to another namespace — why did export stop?

Update the matching *SinkRefs list on the Inventory to names in the new namespace, or recreate the Inventory in the sink namespace. The operator does not follow cross-namespace sink references for namespaced inventory; namespaced inventories resolve family sinks in their own namespace only.

How do I re-test sink connectivity without editing the CR?

Annotate the family sink with kollect.dev/test-connection=true for a one-shot probe, then wait for ConnectionVerified — the exact invocation is in Diagnostic commands above (ADR-0403). Production manifests should keep spec.connectionTest: false and use the annotation for ad-hoc tests.

Does exportMinInterval delay exports after a change?

No. The interval debounces re-export of an identical payload only; it coalesces changes since the last successful export. A material change (payload checksum or metadata.generation bump) exports immediately per sink, regardless of the configured interval. Set exportMinInterval: 0s for material-change only semantics (typical for Kafka/NATS event sinks): instant export on change, identical payloads not re-sent (a controller restart, or more than 48 h degraded without an export attempt, re-sends the current payload once). A sink Secret change is not a material change, so it does not export by itself; the new Secret is used from the next export. A failed export under 0s is retried every 30 s until it succeeds, unless the error is terminal. 0 and sub-second durations are valid (cap: 24h), but wake-ups floor at 1s, so anything below 1s behaves like 0s. Effective-interval precedence and full detail: Export pipeline and debouncing and ADR-0413.

How do I confirm a multipart YAML export set is complete (not torn or stale)?

A Git/GitLab snapshot that exceeds maxExportBytes is sharded into several parts written one after another. The human-readable YAML data files carry no envelope metadata, so completeness is confirmed from the per-set manifest sidecar at inventory/<namespace>/<name>.manifest.json (ADR-0405). A single-part export writes no sidecar and is complete on its own.

Read the sidecar and apply two checks:

# 1) Fetch the manifest for the export set.
cat inventory/default/partitioned-inventory.manifest.json
# {
#   "kind": "KollectExportSetManifest",
#   "generation": 7,
#   "partTotal": 3,
#   "parts": [1, 2, 3],
#   "paths": ["default/team-a/Deployment/api.yaml", "…/web.yaml", "…/worker.yaml"]
# }

# 2) COMPLETE-check: every path in "paths" must exist on disk. Any missing path = a torn set
#    (a part failed to persist). This prints the paths that are MISSING (empty output = complete):
jq -r '.paths[]' inventory/default/partitioned-inventory.manifest.json \
  | while read -r p; do [ -f "$p" ] || echo "MISSING: $p"; done
  • Torn set — a path in paths is missing on disk: an earlier part failed before the set finished. The manifest is only written on the successful final part, so no manifest at all beside part files is itself a torn set (the run never completed).
  • Stale set — the manifest's generation does not match the generation you expect (from the inventory's status / the commit metadata). A torn re-export can leave a prior-generation manifest in place; compare generation before trusting the set.

generation is uniform across a healthy set, partTotal is the expected part count, and parts lists the per-part identifiers a complete set must hold. Writer/verifier split: the controller only writes the manifest (it does not read it back), so torn-set detection is the consumer's responsibility — run the checks above before trusting a set. layout.VerifySet is a helper you can vendor into a Go consumer to apply exactly this rule; the operator equivalent is the jq snippet above.

Is Kollect safe for production today?

Pre-1.0 API

Pre-1.0 API: Kollect is a public alpha with a v1alpha1 API. Minor releases may change the API, defaults and export layout; every breaking change is in the upgrade guide and the release notes. Only the latest minor is supported. See Stability and support.

v1alpha1 has no conversion webhook: a schema change takes effect when you apply the new CRDs, and may need CR updates (ADR-0206).

Evaluate against your risk tolerance. Use pinned chart and image versions; read Unreleased notes in CHANGELOG.md before upgrading.

Why did my CR stop working after an upgrade?

Pre-beta CRD fields can change without conversion. After upgrading CRDs (install-crds.yaml), validate sample manifests and kubectl explain for renamed or removed fields. Breaking changes use feat!: or BREAKING CHANGE: in commit messages (CONTRIBUTING.md).

Is the export JSON format stable?

Sink payloads and Read API responses are moving toward a versioned envelope — today many exports emit a bare JSON array (ADR-0405). Plan downstream consumers for possible wrapper fields before v1.0.

How do I inventory many clusters?

Default path: run one Kollect operator per cluster and export to a shared sink (Postgres, Kafka, NATS, Git) with spec.cluster set on inventory. The sink backend merges rows by cluster id — no central hub tier (ADR-0501, ADR-0401).

Walkthrough: Multi-cluster fleet; concept: Multi-cluster fleet.

My operator uses too much memory — what can I tune?

Restrict watchNamespaces, use tenantMode, narrow KollectTarget selectors, and increase exportMinInterval on inventories. See Performance and scalability and ADR-0603.

Why is a namespace skipped even though a target exists?

Check kollect.dev/namespace-watch: disabled on the namespace, kollect.dev/watch: disabled on resources, watchMode: OptIn without enabled labels, or KollectScope deny rules (ADR-0205, Annotations and labels).

Example troubleshooting guides

Scenario Guide
First inventory pipeline on kind Your first inventory
Postgres DSN and delete reconciliation Postgres state store
Helm / Argo Application attributes Helm release inventory
Sink connectivity probes Connection test
Multi-tenant watch scope Multi-tenant watch namespaces
Fleet rows, shared sinks, and spec.cluster partitioning Multi-cluster fleet

When to escalate

Pre-beta API

v1alpha1 fields may change without conversion webhook. Check ROADMAP before production use.

  1. Collect kubectl describe output for sink, target, and inventory.
  2. Capture operator logs (sanitize Secrets before sharing).
  3. Note the Helm tenantMode and watchNamespaces values, and the chart version.
  4. Open a GitHub issue with repro steps and condition JSON from status.conditions.

See also