Stack API Debt Audit

Stack API Debt Audit

This page is the canonical backlog for external stack API drift. A stack is not considered reviewed until it has an official source, the local code path, and a decision: keep, replace with SDK/native API, or delete.

Rules

• Use official vendor docs or upstream repository docs first.
• Do not add project-specific migration settings as a shortcut.
• Prefer existing SDKs, Temporal durable execution, Studio lifecycle policy, or connector runtime APIs over direct bespoke HTTP loops.
• When replacing custom integration code, prefer OSS that has at least 500 GitHub stars and activity within the last 6 months. Record the GitHub evidence date in this page before switching.
• Mark generated or static catalogs stale when they pin old API dates or old endpoint shapes.
• Keep queued rows visible so stacks are not silently dropped from the audit.

2026-06-25 Findings

Stack	Official API signal	Local debt decision	Status
GitHub REST	GitHub docs say REST calls should send X-GitHub-Api-Version; latest docs show 2026-03-10, while missing headers default to 2022-11-28 and unsupported versions return 410 Gone.	Removed 2022-11-28 from raw GitHub HTTP paths and wizard probe; centralized default version in github_rest_api_version. PyGithub path remains tracked because its constructor does not expose the version header directly.	fixed
Microsoft Graph	Graph docs recommend delta query to avoid constant polling, @odata.nextLink handling for collection paging, and Retry-After handling for throttling.	Added @odata.nextLink paging to the /domains collection lookup. Keep retry/delta as open follow-up; evaluate microsoftgraph/msgraph-sdk-python or Kiota before adding more raw HTTP code.	fixed-first-pass
Daytona	Daytona docs expose SDK/API sandbox lifecycle, filesystem, process execution, snapshots, regions, and runner APIs.	Audit custom sandbox cleanup/listing code against Daytona APIs before adding more local reconciliation loops.	candidate
Dokploy	Dokploy docs expose Swagger/API references for applications, deployments, projects, and servers.	Audit direct deployment calls and static-site delivery assumptions before adding new GitOps/Dokploy adapters.	candidate
Kubernetes runtime access	Kubernetes docs recommend using language client libraries instead of hand-written REST calls. The official Python client released v36.0.2 on 2026-06-01.	cliproxy deployment health, OpenBao pod discovery/status exec, the fractalops-worker readiness gate and Temporal schedule exec, workspace-asset upload, and Studio session-state reads now use the official kubernetes Python client. The API read/write paths use the Kubernetes service proxy instead of selecting an API pod and shelling into it. Keep kubectl only as arbitrary operator-command transport, not as named probe/read-model plumbing. lightkube remains a candidate for typed resource work, but its latest release documents support through Kubernetes 1.32 while the live cluster is v1.34.	fixed-first-pass

OSS Replacement Candidates

These candidates meet the replacement screen on 2026-06-25. They are not automatic dependencies; first prove they remove local code or duplicated behavior.

Stack	Candidate	Evidence	Replacement target
Microsoft Graph	microsoftgraph/msgraph-sdk-python	609 stars, pushed 2026-06-19	Replace raw request/paging/retry code when Graph operations grow beyond the current small executor.
Microsoft Graph / generic OpenAPI	microsoft/kiota	3778 stars, pushed 2026-06-25	Generate typed clients for OpenAPI-described APIs instead of hand-writing request builders for broad connector surfaces.
Kubernetes runtime access	kubernetes-client/python	Official Kubernetes Python client, v36.0.2 released 2026-06-01	Adopted for cliproxy deployment health, OpenBao pod discovery/status exec, the fractalops-worker readiness gate and Temporal schedule exec, verify_agentsquad_execution_plane Studio reads, and workspace-asset upload through the Kubernetes service proxy. Do not build a custom kubectl-string parser.
Headlamp	kubernetes-sigs/headlamp chart 0.43.0	Latest checked release on 2026-07-01. The latest app release is 0.43.0, the Helm release is headlamp-helm-0.43.0, and the plugin tool is headlamp-plugin 0.14.0. Upstream moved under Kubernetes SIG UI (kubernetes-sigs/headlamp) while container images still use ghcr.io/headlamp-k8s/headlamp. Current in-cluster docs expose Cluster Inventory with ClusterProfile; chart plugins are provider binary image volumes, not UI plugins. UI plugin delivery is separate: pluginsManager.enabled, pluginsManager.configContent, and config.watchPlugins=true install/watch Artifact Hub UI plugins. UI plugin APIs include Headlamp.setCluster, registerRoute, registerSidebarEntry, registerResourceTableColumnsProcessor, and project resource APIs such as registerProjectApiResource. The 0.43 line also adds proxy auth, workload Diagnostics, YAML Dry Run, Job logs, Advanced Search persistence, and Project CRD registration; prefer those before local dashboard widgets or bespoke polling endpoints.	Adopted in platform/k8s/apps/headlamp. Keep unsafeUseServiceAccountToken=false so OIDC group RBAC stays intact; automation uses the separate read-only observer token synced to OpenBao. Explicit probes and read-only root filesystem are enabled. Pin image.tag: v0.43.0 until the chart default image is verified not to lag the app release; issue #6065 reports the 0.43 chart still pointing at an older image for unsafeUseServiceAccountToken. Headlamp is the default operator UI for Kubernetes observation; raw kubectl remains bootstrap/break-glass only. Deployment status and pod label named reads now accept headlamp_proxy and call the Headlamp /clusters/main/... backend proxy before falling back to the kubernetes_operator_api transport. Keep Cluster Inventory disabled until ClusterProfile ownership replaces local observation code. Keep pluginsManager disabled only while no UI plugin removes more local dashboard/proxy code than it adds; new custom Kubernetes UI must first be evaluated as a Headlamp plugin.
Gateway API	Kubernetes Gateway API v1.6.0 Standard channel plus Pomerium Gateway support	Official Gateway API latest supported bundle is v1.6.0; GA v1 resources now include GatewayClass, Gateway, ListenerSet, HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute, BackendTLSPolicy, and ReferenceGrant. Use the Standard channel by default; Experimental has no backwards compatibility guarantee. Pomerium Gateway API support is still experimental/partial: HTTPS listeners only, HTTPRoute only, no HTTPHeaderMatch, unsupported HTTPHeaderFilter add, and each route needs an explicit PolicyFilter.	Candidate. Headlamp now carries a disabled httpRoute parity candidate matching the legacy Pomerium Ingress host/path. The migration invariant is not route syntax; it is Keycloak identity/group claims plus SpiceDB CheckPermission authorization. Do not remove legacy Ingress until GatewayClass/Gateway/PolicyFilter exist in GitOps and Keycloak login, Pomerium claims, SpiceDB-backed route/resource authorization, Headlamp OIDC, diagnostics, logs, events, and WebSocket watches pass through the Gateway route. Use ingress2gateway only as an offline conversion aid; it does not preserve provider annotations automatically. Do not adopt ListenerSet, TLS/TCP/UDP routes, or backend TLS policy for the Pomerium edge until controller conformance supports them.
Secret lifecycle / authorization lifecycle	OpenBao Kubernetes secrets engine + External Secrets Operator + SpiceDB	OpenBao can generate Kubernetes service-account tokens and optionally service accounts, roles, and role bindings with leases. ESO syncs external secret-store values into Kubernetes Secrets. SpiceDB models relationships and permissions and evaluates CheckPermission; it does not mint or rotate secrets.	Keep them split. OpenBao owns key/token generation, rotation, TTL, and lease cleanup. ESO owns Kubernetes Secret projection. Kubernetes ServiceAccount owns pod API identity. SpiceDB decides whether a principal may receive or use a secret template/service-account grant. Any Gateway/agent canary must call SpiceDB before OpenBao/ESO/controller materialization, but SpiceDB must not be modeled as the issuer.

2026-07-01 Kubernetes Platform API Findings

These rows are the platform stack baseline for the current Headlamp-first cleanup. They record official API shape before code changes, so follow-up PRs can remove local shims without re-litigating source truth.

Stack	Official API signal	Local debt decision	Status
Headlamp	Upstream 0.43.0 adds alpha ClusterProfile discovery through the Cluster Inventory API, proxy auth, workload Diagnostics, YAML Dry Run, Job logs, Advanced Search persistence, and Project CRD registration. UI extensions belong in plugins using registerRoute, registerSidebarEntry, registerResourceTableColumnsProcessor, and registerProjectApiResource. Chart plugins mount provider binaries for Cluster Inventory; pluginsManager installs UI plugins from Artifact Hub and config.watchPlugins=true enables automatic plugin updates in the main container.	Treat Headlamp as the primary Kubernetes observation UI. Before adding custom dashboard/read-model UI, prove Headlamp plugin APIs or the 0.43 built-ins cannot own the surface. Keep service-account-token auth disabled; keep observer automation read-only. Environment-specific host/profile/hash values belong in the lxc-pve-lab CUE-generated values/headlamp.yaml; the base Headlamp chart values must not own the live domain. Keep the local image.tag: v0.43.0 override while chart/app image skew remains an open upstream risk.	fixed-first-pass
CloudNativePG	CloudNativePG 1.29.0 is the current official release line. It adds Image Catalog and extension artifact management; 1.27.4 is final and no longer supported. Operator upgrades are a two-step process: controller/CRDs first, then instance manager in PostgreSQL pods.	Keep PostgreSQL state in CNPG Cluster/Pooler resources with Longhorn-owned PVC lifecycle. Do not hand-create stable Postgres PVCs or per-app PostgreSQL StatefulSets. App DB/role/secret projection belongs in CUE-generated environment values plus OpenBao/ESO.	candidate
Valkey cache plane	Official Valkey current release is 9.1.0 (2026-05-19); latest supported 8.x is 8.1.8 (2026-06-02). The official valkey-operator repository exposes ValkeyCluster and latest release v0.2.0; the official Helm chart is active but still maturing cluster/HA features. Local cache-plane currently pins valkey/valkey:9.0.0, so the cache plane has explicit image drift to review.	Redis-compatible cache/session workloads use the shared cache-plane Valkey operator aliases (novu-valkey, glitchtip-valkey, etc.). Do not add app-owned Redis PVCs or bespoke Redis charts for cache-only use. Review the 9.1.0 image bump against operator v0.2.0 in a preview Job before changing the shared cache plane.	candidate
SeaweedFS object/file plane	SeaweedFS exposes Kubernetes-native Helm/CSI surfaces. The official Helm chart owns Master, Volume, Filer, and S3 API components; the operator remains a later option only if it removes local lifecycle code.	Object/blob/file assets move to the shared SeaweedFS/S3-compatible plane where durable DB is not required. seaweedfs is a data-plane Argo app pinned to chart 4.37.0; it uses native PVC templates, an environment-owned durable StorageClass, chart anti-affinity and replication, OpenBao-projected S3 authentication, and the upstream private-bucket hook. See the SeaweedFS 4.37 trap sheet. CNPG remains Supabase SQL authority and is not SeaweedFS filer metadata. Nexus remains artifact/package/blob-store owner; do not treat CNPG PVC reduction as permission to store object data in Postgres.	active
OpenTelemetry / Grafana Alloy	OpenTelemetry Collector releases are on short cycles; current official collector line is v0.155.0. Grafana recommends Alloy as its OpenTelemetry Collector distribution for production observability and can run/convert OTel Collector configs via --config.format=otelcol or alloy convert --source-format=otelcol.	Centralize app, AgentSquad, build, and cache metrics through the OTel plane. Do not add new one-off metrics pollers unless they emit OTel and have a deletion path. Evaluate Alloy only if it removes local collector config/shims.	candidate
Grafana Mimir	Grafana Mimir ingests Prometheus or OpenTelemetry metrics and provides multi-tenant long-term metric storage. The mimir-distributed chart is the Kubernetes install/upgrade surface; docs currently show chart release notes under latest v6.1.x. Production guidance says chart-managed MinIO is only for getting started, and object storage should be external S3-compatible/GCS/Azure/Swift with separate blocks, ruler, and alertmanager buckets. When multitenancy is enabled, authenticated HTTP requests must carry X-Scope-OrgID with the tenant ID.	Mimir is the metrics backend, not another bespoke event DB. Grafana dashboards should read OTel/Mimir signals before local Studio widgets grow custom polling APIs. platform/k8s/apps/mimir/values.yaml now uses SeaweedFS/S3-compatible object storage through mimir-object-storage, keeps separate blocks/ruler/alertmanager buckets, enables Mimir multitenancy, renders mimir-object-storage-migration-contract, and disables Mimir ingester/store-gateway/compactor app-local PVCs. The central OpenTelemetry Collector writes platform metrics to the fractalops tenant through otlphttp/mimir with X-Scope-OrgID: fractalops; project-scoped read/write wrappers must derive the tenant from gateway identity and must not trust client-provided tenant parameters. Because removing volumeClaimTemplates is a StatefulSet immutable-field change, only those three StatefulSets carry resource-scoped Force=true,Replace=true Argo sync options. Object readback proof gates old filesystem PVC retirement.	active
Gateway API	Gateway API v1.6.0 is the latest supported bundle for the v1 API. Standard channel should be the default; Experimental channel has no backwards compatibility guarantee. GA v1 resources include GatewayClass, Gateway, ListenerSet, HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute, BackendTLSPolicy, and ReferenceGrant.	Prefer Gateway API Standard channel for new edge routing only after Pomerium PolicyFilter, Keycloak claims, SpiceDB checks, Headlamp OIDC, logs/events/watch streams, and legacy Ingress parity are proven. Pomerium current Gateway support still constrains us to HTTPS Gateway plus HTTPRoute plus PolicyFilter, so keep disabled parity config until controller conformance covers the broader v1.6.0 resource set.	candidate

2026-07-01 Storage/Observability Debt Queue

These are concrete repo targets from the Kubernetes stack scan. Do not delete a PVC only because it exists; first classify the data shape, then migrate to the right authority and add a contract proving the old authority is gone.

Target	Current local authority	Desired authority	Status
armory / ContextForge	shared-CNPG contextforge database/role through OpenBao/ESO DATABASE_URL; the gateway is stateless.	Migration run 30326100497 captured a ready 8 GiB SQLite VolumeSnapshot, loaded CNPG, proved exact row readback, and verified gateway health. legacyPvcRetained=false removes the unmounted application PVC while the migration snapshot preserves rollback bytes.	fixed
agent-memory-archive	CNPG agent_memory_record; pod-local emptyDir is only a non-durable JSONL fallback path	CNPG remains metadata/state authority; object attachments move to SeaweedFS/S3 only if needed. The chart renders no PVC.	active
cliproxy	cliproxy-auths PVC stored OAuth/account auth and logs together	CNPG cliproxy database via official PGSTORE_DSN; pod local path is only mirror/cache	fixed-first-pass
penpot database	Bundled Bitnami PostgreSQL subchart/PVC	Shared CNPG penpot DB/role via postgres-application-database; Penpot reads penpot-postgresql Secret	fixed-first-pass
windmill database	Bundled demo PostgreSQL deployment	Shared CNPG windmill DB/role via postgres-application-database; Windmill reads windmill-postgresql:url	fixed-first-pass
playwrightgrid database	Bundled Bitnami PostgreSQL subchart/PVC with hardcoded public password	Shared CNPG browse DB/role via postgres-application-database; chart consumes existingSecretEnv	fixed-first-pass
daytona database	Shared CNPG through upstream externalDatabase; region hooks connect with namespace-local Secret	Bundled Bitnami PostgreSQL and pod-exec coupling removed after verified 139-snapshot/3-region migration	fixed
datahub database	Prerequisites MySQL StatefulSet now durable on Longhorn, but still app-local SQL	Shared CNPG datahub DB/role contract exists as datahub-postgresql; chart flip waits for global.sql.datasource Postgres JDBC proof and system-update readback	staged
datahub search/Kafka prerequisites	Elasticsearch/MySQL/Kafka state previously rendered pod-local emptyDir volumes	Immediate wipe guard: use fractalops-longhorn-durable-rwo claim templates. Longer-term: move SQL to CNPG and evaluate shared Kafka/search before deleting remaining prerequisite state	fixed-first-pass
Supabase Storage	Previous file-backed volume is retained only as the read-only source for a one-time orphan handoff	Authenticated SeaweedFS S3 object plane; the single upstream supabase release renders no Storage PVC	source-cleaned; live handoff pending
penpot assets	penpot-data-assets Longhorn claim	SeaweedFS/S3-compatible object plane, while cache stays Valkey	candidate
grafana	RWO Longhorn PVC for embedded Grafana state	Evaluate CNPG-backed Grafana DB before treating dashboard state as durable PVC authority	candidate
mimir	Filesystem-backed blocks/ruler/alertmanager state on Longhorn PVCs	SeaweedFS/S3-compatible object plane through the Mimir chart structuredConfig S3 backend, with ExternalSecret-projected credentials, separate blocks/ruler/alertmanager buckets, and disabled ingester/store-gateway/compactor app-local PVCs	active
BuildKit cache PVCs	Longhorn cache PVCs allowed by runtime-storage policy	Nexus/GHCR registry cache for durable cache; scratch PVCs disposable only	candidate
daytona-runner-dind-data	Longhorn PVC explicitly marked scratch cache	Keep as bounded scratch exception; durable image/snapshot authority is Nexus/GHCR/OCI	contracted-exception

Observability debt from the same scan:

• App-side config exposes OTEL_LOGS_EXPORTER, and the collector has a logs pipeline. Fixed first pass: configure_otel now wires OTel logs through the OTLP HTTP /v1/logs endpoint and a stdlib root LoggingHandler; remaining work is Loguru-to-stdlib correlation and cardinality guardrails before Mimir.
• Fixed first pass: AgentSquad process state/blocker dashboard panels now have one canonical Studio OTel emitter for fops_agent_process_state_total and fops_agent_process_blocker_total. Remaining work is live Mimir label cardinality review and board proof from a real squad run.
• Fixed second pass: studio_otel_metrics.summarize_studio_run_sessions is now the canonical Studio run/session count helper for both OTel emission and the fleet board projection. Remaining overlap is session_metrics, project_agents_observability, and JSON feed cost/process surfaces; migrate them toward OTel/Mimir-backed read models before adding more Portal widgets.
• Fixed second pass: backend OTel exporters and build-plane event telemetry now share fractalops.foundation.endpoints.otlp_signal_endpoint, so /v1/traces, /v1/metrics, and /v1/logs endpoint normalization cannot drift between API pods and CI/build-plane scripts.
• Fixed third pass: project_agents_observability compact and metrics outputs now share summarize_project_agents_observability_feed, so CLI JSON feed event/log/agent counts cannot drift while the remaining feed surfaces migrate toward OTel/Mimir-backed read models.
• Fixed Armory observability pass: observability is now a known Armory MCP server, so the observability-triage bundle can attach GlitchTip, project Mimir metrics, and HUD together without unknown-server drift. The compact AgentSquad CLI feed is allowlisted and has a regression proof that drops raw per-agent payload weight by more than 40%.
• Fixed Daytona toolbox pass: DaytonaWorkspaceService now uses the current daytona API toolbox route (/toolbox/<sandbox>/toolbox/...) directly for process execution, file upload, and file download. The retired single-form probe path is gone, so each command/upload skips one guaranteed 404 hop while keeping cold-start route warming retries.
• Fixed fourth pass: runtime build pre-cleanup now observes buildkit pods through operation-assets request --role kubernetes_observation and the Headlamp /clusters/main/... proxy, while the only write path remains the bounded fractalops_control_plane pod delete mutation. This removes kubectl from the buildx observation path without weakening the stuck-pod self-heal.
• Fixed fifth pass: runtime rollout observation now reads Argo CD Application, Deployment, Pod, and Event JSON through the Headlamp-backed kubernetes_observation asset. The remaining kubectl path in that script is the Argo CD hard-refresh annotation, which is an explicit control-plane mutation.
• Fixed sixth pass: runtime storage and network plane audits now share platform/k8s/lib/headlamp_observation.sh and fetch Kubernetes JSON through operation-assets request --role kubernetes_observation. The audits still degrade missing resources to empty lists, but they no longer own raw kubectl get ... -o json read models.
• Fixed seventh pass: Daytona snapshot and DinD disk runbooks no longer route operator evidence collection through raw kubectl guidance. Snapshot inspect reads runner pods through the shared Headlamp observation helper, and the DinD disk handoff names Headlamp REST paths for PVC, Longhorn replica/volume, and runner pod evidence.
• Workspace repo copy metrics still cross the sandbox boundary through FOPS_WORKSPACE_REPO_COPY stdout markers. Keep the marker only as a temporary transport until the execution workspace can emit OTel directly or attach structured evidence through the agent process adapter.
• Fixed second pass: agent-memory-archive now treats CNPG agent_memory_record as the durable authority and renders no PersistentVolumeClaim. The mounted archive path is emptyDir only, so FRACTALOPS_DATABASE_URL is required in Kubernetes and local JSONL fallback is non-durable by design. The chart keeps agent-memory-archive-storage-migration-contract as the visible cutover contract.

Whole-Repo Static Scan

2026-06-25 scan coverage:

• rg --files: 2969 files total, 2098 Python files, 434 YAML files, 116 Markdown files.
• Vulture at 100% confidence over backend/src tests ops: one finding, fixed by renaming the required redirect-handler argument to _fp.
• Tach external dependency gate: found openinference used from foundation.observability without a package contract; fixed by declaring openinference-instrumentation-langchain in pyproject.toml/uv.lock.
• Tach dependency/interface gate: keep as a required gate for every cleanup batch.

Top reinvention hotspots from the whole-repo scan:

Pattern	Scope	Highest-hit files	Next action
Direct HTTP clients	72 files / 225 hits	ops/lxc/_api.py, Daytona preview/toolbox paths, github_app_service.py, Nexus provisioning	Prefer connector SDKs or generated clients where an official SDK/OpenAPI client exists.
Raw subprocess orchestration	82 files / 444 hits	repository_workspace_materializer.py, daytona_runner.py, studio_agent_session_cli_runner.py	Collapse shell hops behind one process boundary or Temporal activity where durable execution is required.
Polling loops	Historical whole-repo hotspot	daytona_workspace.py, GHCR retention, LXC drift check, remaining bespoke CLIs	Replace indefinite polling with bounded waits, Temporal timers, SDK waiters, or explicit operator commands. Live Studio CLI, Studio session runner, AgentSquad turn wait, and execution-plane verification now use ops/cli/control_flow/polling.py.
Raw JSON/YAML serialization	298 files / 761 hits	repository materialization, Daytona bootstrap, CLI run/session paths	Move recurring payload shapes to typed contracts instead of ad hoc dict serialization.
Retired/compat/tooling wording	250 files / 1482 hits	Daytona bootstrap/runner, preview gateway, Studio tests	Keep compatibility only where a current public/runtime contract proves it; otherwise delete or rename to ubiquitous product language.

Workspace Tree Cleanup

2026-06-25 cleanup was performed from repo root with tree -a -L 2 plus git status --ignored, git check-ignore, git ls-files, du, and git worktree list --porcelain.

Deleted ignored/generated workspace artifacts:

• .playwright-cli
• build
• docs/dist
• output
• var/backups
• .state/tmp
• .state/tmp.rendered.env
• all __pycache__ directories
• ignored Helm dependency tarball directories under platform/k8s/apps/*/charts
• ignored retired Verdaccio chart cache at platform/k8s/apps/verdaccio

Deleted stale Claude worktrees:

• Removed 30 .claude/worktrees/* entries only after proving each was a Git worktree, clean, and unlocked.
• Preserved 3 locked worktrees and 1 dirty worktree.
• .claude/worktrees dropped from about 6.3GB to about 772MB.

Do not delete these by name alone:

• supabase/migrations/*baseline*: immutable schema adoption history, not stale source.
• yamonco/fractalops-contract:schemas/*/fixtures/legacy-v1.json: compatibility contract fixtures referenced by contract docs/tests.
• .agentsquad/threads and .serena/memories: agent/context memory surfaces.

Full Coverage Queue

Connector catalog stacks:

Stack	Review status
GitHub	fixed-first-pass
Microsoft Graph / Entra	fixed-first-pass
Keycloak	queued
Daytona	candidate
Argo CD	queued
Dokploy	candidate
DataHub	queued
Playwright Grid	queued
SearXNG Grid	queued
Agent Memory Archive	queued
Armory Gateway	queued
GlitchTip	queued
Phoenix	queued
CLIProxy	fixed-first-pass
Agent Studio	queued
Penpot	queued
Proxmox VE	queued
Nexus	queued
Supabase	queued
LangBoard	queued
Windmill	queued

Kubernetes app stacks:

Stack	Review status
agent-memory-archive	queued
armory	queued
clickhouse	queued
cliproxy	fixed-first-pass
datahub	queued
datahub-mcp	queued
datahub-operator	queued
daytona-bootstrap	candidate
devpod-developer-workspaces	fixed-first-pass
edge-bridge	queued
external-secrets-config	queued
fractalops-api	queued
fractalops-build	queued
fractalops-github-runner	queued
fractalops-portal	queued
fractalops-postgresql	queued
fractalops-preview-a	queued
fractalops-preview-b	queued
fractalops-preview-c	queued
fractalops-test	queued
fractalops-worker	queued
glitchtip	queued
grafana	queued
headlamp	fixed-first-pass
jupyter-mcp	queued
kafka	queued
mcp-edge-bridge	queued
mimir	queued
nexus	queued
novu	queued
opentelemetry-collector	queued
penpot-persistence	queued
phoenix	queued
playwrightgrid-runtime-secrets	queued
playwrightgrid-terminal-pod-janitor	queued
pomerium	queued
runtime-storage	queued
runtime-storage-policy	queued
searxng	queued
searxnggrid	queued
spicedb	queued
supabase	queued
supabase-realtime	queued
supabase-storage	queued
windmill	queued

Official Sources Used In First Pass

• GitHub REST API versions: https://docs.github.com/en/rest/about-the-rest-api/api-versions
• Microsoft Graph delta query: https://learn.microsoft.com/en-us/graph/delta-query-overview
• Microsoft Graph throttling: https://learn.microsoft.com/en-us/graph/throttling
• Daytona API reference: https://www.daytona.io/docs/en/tools/api/
• Daytona sandboxes: https://www.daytona.io/docs/en/sandboxes/
• Dokploy API docs: https://docs.dokploy.com/docs/api
• Dokploy application API reference: https://docs.dokploy.com/docs/api/reference-application
• CUE export: https://cuelang.org/docs/reference/command/cue-help-export/
• CUE with YAML: https://cuelang.org/docs/concept/how-cue-works-with-yaml/
• ORAS OCI artifacts: https://oras.land/docs/concepts/artifact/
• Dev Container Features: https://devcontainers.github.io/implementors/features/
• Kubernetes client libraries: https://kubernetes.io/docs/reference/using-api/client-libraries/
• Kubernetes Python client releases: https://github.com/kubernetes-client/python/releases
• LightKube releases: https://github.com/gtsystem/lightkube/releases
• Headlamp releases: https://github.com/kubernetes-sigs/headlamp/releases
• Headlamp in-cluster deployment: https://headlamp.dev/docs/latest/installation/in-cluster/
• Headlamp authentication: https://headlamp.dev/docs/latest/installation/
• Headlamp plugin architecture: https://headlamp.dev/docs/latest/development/architecture/
• Headlamp plugin development: https://headlamp.dev/docs/latest/development/plugins/
• Headlamp plugin functionality: https://headlamp.dev/docs/latest/development/plugins/functionality/
• Headlamp plugin manager: https://headlamp.dev/docs/latest/installation/in-cluster/#plugin-management
• CloudNativePG releases: https://cloudnative-pg.io/releases/
• CloudNativePG installation and upgrades: https://cloudnative-pg.github.io/docs/1.29/installation_upgrade/
• Valkey releases: https://valkey.io/
• Valkey release archive: https://valkey.io/download/releases/
• Valkey operator: https://github.com/valkey-io/valkey-operator
• SeaweedFS operator: https://github.com/seaweedfs/seaweedfs-operator
• SeaweedFS CSI driver: https://github.com/seaweedfs/seaweedfs-csi-driver
• OpenTelemetry Collector releases: https://github.com/open-telemetry/opentelemetry-collector-releases/releases
• Grafana Alloy OpenTelemetry Collector migration: https://grafana.com/docs/alloy/latest/set-up/migrate/from-otelcol/
• Grafana Mimir Helm chart: https://grafana.com/docs/helm-charts/mimir-distributed/latest/
• Grafana Mimir production Helm guidance: https://grafana.com/docs/helm-charts/mimir-distributed/latest/run-production-environment-with-helm/
• Grafana Mimir overview: https://grafana.com/docs/mimir/latest/
• Grafana Mimir object storage backend: https://grafana.com/docs/mimir/latest/configure/configure-object-storage-backend/
• Gateway API Ingress migration: https://gateway-api.sigs.k8s.io/guides/getting-started/migrating-from-ingress/
• Kubernetes Gateway API: https://kubernetes.io/docs/concepts/services-networking/gateway/
• Gateway API status: https://github.com/kubernetes-sigs/gateway-api
• Gateway API versioning: https://gateway-api.sigs.k8s.io/docs/concepts/versioning/
• Gateway API releases: https://github.com/kubernetes-sigs/gateway-api/releases
• Pomerium Gateway API: https://www.pomerium.com/docs/deploy/k8s/gateway-api
• Ingress2gateway: https://github.com/kubernetes-sigs/ingress2gateway
• OpenBao Kubernetes secrets engine: https://openbao.org/docs/secrets/kubernetes/
• OpenBao Kubernetes secrets engine API: https://openbao.org/api-docs/secret/kubernetes/
• External Secrets Operator: https://external-secrets.io/
• SpiceDB schema: https://authzed.com/docs/spicedb/concepts/schema
• SpiceDB caveats: https://authzed.com/docs/spicedb/concepts/caveats

Workspace Hygiene

The resource reaper CLI now owns local cleanup as an explicit dry-run/apply step:

uv run fractalops operations reap-runtime-resources --local-workspace-hygiene-only
uv run fractalops operations reap-runtime-resources --local-workspace-hygiene-only --apply

The hygiene pass deletes only ignored generated artifacts and clean unlocked .claude/worktrees/* entries. It skips tracked paths, dirty worktrees, locked worktrees, runtime memory, Serena memory, local secrets, dependency installs, virtualenvs, and tool caches.

2026-06-25 cleanup evidence:

• Removed ignored artifacts: .pytest_cache, .hypothesis, backend/src/fractalops.egg-info, ops/workspace-tools/.hypothesis, ops/workspace-tools/dist, platform/plugins/keycloak/identity-bridge/target, tests/contracts/system/build.
• Removed Python bytecode caches under ops/cli and backend/src/fractalops.
• Preserved worktrees: .claude/worktrees/agent-abd7872b86404587e, .claude/worktrees/agent-ad31b05f161e7c325, .claude/worktrees/agent-ade91875a51e66a3d (locked_worktree), .claude/worktrees/claude+hop-cleanup (dirty_worktree).

SSOT And OSS Replacement Notes

• Runtime SSOT: platform/k8s/environments/lxc-pve-lab/runtime.cue owns generated Helm values and the runtime contract; regenerate with platform/k8s/cue-generate.sh.
• Kubernetes access: fractalops infra kubectl now routes through the unified runtime asset service and the fractalops_control_plane Kubernetes asset. kubernetes_operator_api assets fail loud when the operator API path is unavailable; Proxmox/VM relay is only used by assets that explicitly declare relay access mode.
• Live Studio API CLIs no longer expose qm-guest/qm-local, direct control-plane API fallback, or shell transport choices as normal transports. verify_agentsquad_execution_plane is typed Kubernetes-client only for runtime-plane reads, and live_studio_api.py uses the Kubernetes service proxy for Studio API calls.
• Removed reinvention candidate: hand-built SSH kubectl command planning in infra kubectl.
• AgentSquad loop ownership: durable polling, retry exhaustion, and blocker intervention belong in Temporal schedules/workflows/activities. Studio owns run/session/mailbox decisions and Agent Execution owns provider/process policy. Standalone CLI watchers are retired. Local CLI waits that cannot yet be Temporal-owned must use the named feature slice ops/cli/control_flow/polling.py; root ops/cli/*.py helper modules are blocked by the quality audit.
• Workspace tools OSS boundary: keep @fractalops/workspace-tools limited to AgentSquad session semantics. Generic tool installation should move to Dev Container Features; signed tarball/binary distribution should remain OCI/ORAS/cosign/age instead of private bootstrap logic.
• Kubernetes OSS replacement progress: KubernetesRuntimeAssetController.read_deployment_status, read_first_pod_name_by_label, and exec_pod_command use the maintained kubernetes Python client for named read/probe paths (cliproxy health, OpenBao status, and fractalops-worker Temporal schedule checks). verify_agentsquad_execution_plane uses the typed Kubernetes client for Argo Application status, Deployment image reads, and Studio session-state reads through the Kubernetes service proxy. live_studio_api.py now prefers the direct control-plane API when --portal-url/--api-token or FRACTALOPS_INTERNAL_API_BASE_URL is supplied, and keeps the service proxy as fallback for in-cluster/break-glass use. The old API-pod shell selector and named pod-shell probes are removed. Keep kubectl only as arbitrary-command transport, not as topology/selection owner.
• Proxmox/LXC boundary: Proxmox shell commands are not Kubernetes fallback. The remaining pvesh usage is isolated to ops/lxc/ensure_github_runner.py through _proxmox_cluster_resources() for CT node inventory; backend/application code must not call raw pvesh, qm guest exec, or pct exec. The quality audit treats those as high-severity infra boundary violations outside explicit runtime asset adapters.
• AgentSquad documentation sync: docs contracts and quality exceptions now use the canonical wall_control_contract, guardrail retry prompt, execution surface, execution workspace, and agent process adapter terms. The execution feature-slice map now names the Daytona boundary as workspace instead of a generic runtime slice.
• Break-glass Kubernetes relay boundary: the old relay role/env/helper naming is removed from normal topology and CLI defaults. The Kubernetes operation path defaults to fractalops_control_plane; the VM SSH relay is named only as break_glass_kubernetes_relay and remains reachable through explicit break_glass_relay_asset_id wiring.
• Secret delivery: the retired imperative bootstrap is removed. Argo CD owns External Secrets configuration, while ESO reconciles pull and push delivery.
• GitOps machine token bootstrap: bootstrap_gitops_machine_token.sh now fails loud when the configured OpenBao API is unreachable instead of writing the GHCR secret through a CT-local pct exec path.
• PostgreSQL and Supabase bootstrap: bootstrap_fractalops_postgresql.sh and bootstrap_supabase.sh require an explicit OpenBao token or token file and never read /opt/fractalops/dev-live/.state/openbao.token from a live CT. The Supabase bootstrap seeds referenced runtime material and publishes the one manual supabase Application; it does not install a second release or synchronize Argo CD. Their ensure helpers do not accept or forward --pct-vmid.
• OpenBao secret-store contract cleanup: OpenBaoConfig, openbao_secret_store_from_runtime, openbao_sync_scope.py, connector SSOT reconcile, ensure_spicedb_runtime_secret.py, and ensure_supabase_runtime_secrets.py no longer expose PCT/VMID transport parameters. Environment rendering no longer emits FRACTALOPS_OPENBAO_PCT_VMID; CT access remains only as explicit break-glass/install input such as OPENBAO_CTID, not as a secret delivery fallback.
• App runtime: the retired FractalOps and Pomerium imperative bootstraps are removed. CUE-generated values and Argo CD Applications own steady state.
• Quality gate: ops/quality/karpathy_simplify_audit.py now blocks the removed platform bootstrap fallback tokens (LIVE_ENV_CT_ID, /opt/fractalops/shared/.env, POMERIUM_SOURCE_CT_ID, POMERIUM_CONFIG_PATH, and OpenBao PCT VMID env names) in the bootstrap files that previously carried those paths.
• Project preview edge: _OperationAssetConfigMutator now delegates cloudflared CT commands to LxcRuntimeAssetController with a RuntimeAssetDescriptor. The preview edge service no longer constructs its own SSH + pct exec command string.
• LXC release env activation: _ct_runtime.sh, run_control_plane_task.sh, remediate_from_topology.sh, activate_dev_runtime.sh, and drift_check.py no longer discover shared/.env or root .env through fallback chains. LXC tasks use the explicit FRACTALOPS_ENV_FILE_TARGET when provided, otherwise the active release .env; missing env is a hard operator error.
• Dokploy operation compose plane: operation_compose_exec.py no longer accepts Proxmox PVE_SSH_HOST/PVE_SSH_USER or unscoped DOKPLOY_PROJECT_PREFIX as operation aliases. Compose operations must use the operation-plane names (FRACTALOPS_OPERATION_EXEC_* or FRACTALOPS_DOKPLOY_OPERATION_*) so Proxmox break-glass credentials and retired deployment env names do not leak into application deployment controls.
• Coding-agent model-plane auth: coding_agent_auth_env.py now resolves only the cliproxy client key needed by the AgentProcess adapter. It no longer resolves direct provider tokens, GitHub/Dokploy control-plane tokens, or Proxmox/break-glass SSH secrets as part of agent auth env construction.
• Argo CD identity: the retired imperative identity bootstrap is removed; the self-managed Application and named GitOps secret contract own reconciliation.
• GitOps machine token bootstrap: bootstrap_gitops_machine_token.sh no longer accepts the GitHub token as a positional shell argument. Operators must pass the named BOOTSTRAP_GITHUB_TOKEN env var so the bootstrap contract is explicit and shell-history leakage is not encouraged.