ADR-0001 — Daytona sandbox mutable-image registry runs on Harbor

Status: Accepted (2026-07-06)

Context

Daytona CREATE_SANDBOX builds a per-sandbox derived image daytona-<contenthash> from a base workspace snapshot, PUSHES the derived image to its INTERNAL registry, and the runner PULLS it back. For ~5h across one session this failed with Error response from daemon: manifest unknown.

Root cause (live-verified): DockerRegistryService.getRegistryPushAccess mints push access via a Harbor-style token/robot service (/service/token). ghcr.io returns 405 for that flow, so daytona never obtains a push token → the derived image never lands → runner pull = manifest unknown. The stored ghcr PAT is valid (token_match=true); the failure is upstream in token minting, not the credential.

Empirically proven this session (do not re-test):

Flipping only the transient docker_registry row to Harbor while leaving internal=ghcr → still manifest unknown. So daytona’s derived push uses the INTERNAL registry, not transient.
daytona resolves the base snapshot’s registry via the DEFAULT internal row too. Setting internal=Harbor while the base snapshot image is still ghcr.io/... produces No registry found for snapshot. So base-pull and derived-push are COUPLED on the internal registry.

Decision

The daytona mutable sandbox-image (internal/transient) registry is a chart-managed Harbor in namespace daytona. Daytona validates and authorizes the registry through the in-cluster TLS endpoint (https://harbor.daytona.svc.cluster.local), so the chart-managed daytona-harbor-nginx CA is mounted into both daytona-api and the runner Docker daemon trust path (/etc/docker/certs.d/harbor.daytona.svc.cluster.local). Build/push image refs use the same canonical TLS host (harbor.daytona.svc.cluster.local/...). Harbor supports the token/robot service ghcr lacks.
Harbor’s dependencies are centralized: blobs on SeaweedFS S3, DB on shared CNPG, cache on a non-cluster Redis (daytona-redis-master, standalone — the shared cluster-mode fractalops-cache Valkey only exposes db0 and cannot host Harbor’s multi-DB layout). Admin secret via OpenBao/ESO daytona-harbor-admin.
The base daytona-workspace image must be published INTO Harbor (harbor.daytona.svc.cluster.local/library/fractalops-api:...) by runtime-release (build_daytona_workspace), and the daytona snapshot.imageName set to that Harbor path. A manual skopeo/one-off copy is forbidden — it drifts back to ghcr on the next build (violates the settings-reproducibility law).
The docker_registry DB rows (runtime DB state, NOT GitOps — Argo never touches them) for internal + transient point at the same Harbor host and use the OpenBao/ESO-managed Harbor admin secret (daytona-harbor-admin/HARBOR_ADMIN_PASSWORD). Repository ownership follows ADR-0013.

Consequences

Harbor is a hard runtime dependency for studio sandboxes; its 6 bring-up traps are catalogued in the daytona-registry keystone memory (RFC1123 Job name, nginx TLS secret, redis cluster-mode, admin-secret hook-GC prune, generated-secret checksum drift, jobservice/core startup race).
The docker_registry row values, the daytona-api INTERNAL/TRANSIENT_REGISTRY_URL env, the Harbor admin password env refs, and the snapshot imageName must all name ONE consistent Harbor authority — reconcile them together, never one in isolation. The internal and transient DB rows must use harbor.daytona.svc.cluster.local (no scheme, no port). If they drift to the old harbor.daytona.svc.cluster.local:80 authority, Harbor redirects Docker’s blob-upload POSTs and the client resumes them as GETs, ending in blob upload unknown.
A daytona-api registry-config change requires a pod roll to take effect.
Under the native Kubernetes provider (ADR-0007) there is no runner DaemonSet: daytona-api pulls base snapshots and pushes derived sandbox images to Harbor directly, authenticated by the daytona-internal-registry dockerconfig ExternalSecret (rendered by the daytona-bootstrap chart from the Harbor admin credential). Harbor’s TLS CA reaches daytona-api via the daytona-harbor-nginx volume mount in daytona-values.yaml, not a hook-injected runner CA mount.

Registry topology & consolidation (Nexus vs Harbor)

Operator question: can we keep only ONE of Nexus/Harbor? Analysis across our actual components (Daytona, DevPod, BuildKit):

BuildKit — registry-agnostic; pushes to any OCI registry. Not a constraint.
DevPod — pulls devcontainer OCI images from any registry. Not a constraint.
Daytona — the ONLY constraint. v0.184 getRegistryPushAccess expects a Harbor-style token/robot registry API; ghcr and Nexus’s docker registry (standard Docker v2 bearer-token) do not implement it.
Artifact types — docker/OCI: both can host. npm/pypi/maven: Nexus ONLY — Harbor is a container+Helm registry and cannot host language packages.

Therefore:

Harbor-only is NOT viable — it cannot replace Nexus’s npm/pypi/maven hosting.
Nexus-only IS viable IFF daytona pushes via basic-auth to Nexus (skipping the Harbor token flow). Nexus is the superset (docker + language artifacts + proxy-cache); Harbor exists solely to satisfy daytona’s token-API quirk.

Role split (operator law, until/unless consolidated): Nexus = DEV artifact hub (npm/pypi/maven + docker dev images); Harbor = PROD/operations image hub (daytona runtime sandbox images). Do not mix roles.

Open decision (record, don’t silently keep Harbor): verify whether daytona v0.184 can basic-auth push to Nexus. If yes → consolidate to Nexus-only and retire Harbor + its 6-trap bring-up (leanest). If no → the two-registry split stands, justified purely by daytona’s hard requirement. Record the result in the Assembly-owned ADR corpus before changing topology.

Alternatives rejected

ghcr basic-auth push (patch daytona to skip the token service): requires forking/patching daytona; rejected — no-fork constraint. NOTE: the same basic-auth mechanism against Nexus is the consolidation path above — revisit it, since it would retire Harbor.
Object-storage / MinIO push path: ObjectStorageService.getPushAccess is for build-context (context.tar) upload, NOT snapshot image propagation, which stays Docker-registry based. SeaweedFS also fails daytona’s MinIO-STS assumption. Rejected as not a registry replacement.
Keep ghcr, bump credentials/scope/rows: necessary-but-insufficient — the 405 is architectural (ghcr has no Harbor token service), so no ghcr-side fix ever lands the push.
Harbor bundled internal Redis: rejected in favour of reusing the existing standalone daytona-redis-master (centralization law).