Skip to content
FractalOps

SeaweedFS 4.37

SeaweedFS 4.37

Section titled “SeaweedFS 4.37”

FractalOps uses the upstream seaweedfs Helm chart 4.37.0 (application 4.37) from tag commit c06a2dca879cdbe742246d812431fbe2de01357b. The published chart archive SHA-256 is 7016e13d5be393312a5b678c95479d40ece712463ece6d73704db3b762292b0a. Investigation date: 2026-08-04.

Supported Path

Section titled “Supported Path”
  • Master, Volume, and Filer data use the chart’s native persistentVolumeClaim values. The lxc-pve-lab profile selects the environment-owned fractalops-longhorn-durable-rwo-v2 StorageClass. Another Kubernetes profile replaces only this standard value; SeaweedFS contains no host, disk tag, or node-label identity.
  • Three Master replicas and two Volume replicas retain the chart’s required hostname anti-affinity. Placement 001 keeps two SeaweedFS copies on distinct volume servers. Filer remains one chart-native LevelDB2 instance on its PVC; CNPG owns Supabase SQL data, not SeaweedFS filer metadata.
  • The durable target requests 2Gi per Master, 600Gi per Volume, and 10Gi for Filer. The retained migration source remains 20Gi per Volume and 5Gi for Filer; do not size the target from those source values. Expansion remains available through the selected StorageClass.
  • S3 authentication uses s3.enableAuth, s3.existingConfigSecret, and the required seaweedfs_s3_config key. External Secrets projects existing runtime and Supabase Storage credentials from OpenBao. Static actions are bucket-scoped; no administrator action or key value exists in Git.
  • s3.createBuckets and the upstream post-install/post-upgrade hook create all private buckets, including fractalops-supabase-storage, idempotently.
  • The upstream hook runs only during Helm install or upgrade. It does not repair bucket-directory metadata lost afterward. The assembly-owned reconciler calls the same pinned s3.bucket.list and s3.bucket.create commands every five minutes, creates only declared missing buckets, and verifies the complete set.
  • Kubernetes retains StatefulSet PVCs when a StatefulSet is deleted or scaled unless a retention policy overrides that default. PV reclamation remains a property of the selected cluster StorageClass.
  • Source Filer entry-tree protection uses the pinned native fs.meta.save and fs.meta.load commands. A generation-scoped manual Application exports a checked artifact, snapshots its PVC, restores that snapshot, loads it into an isolated loopback-only Master/Volume/Filer stack, re-exports the restored tree, and requires the original and restored counts to agree. A one-way native filer.sync.verify then proves every restored entry still agrees with the live source while intentionally ignoring entries added to that source after export.

Compatibility and Migration Traps

Section titled “Compatibility and Migration Traps”
  • enableAuth: false is allow-all development behavior. Enabling authentication before every S3 consumer uses an identity in the static config causes immediate access failures. Verify the existing Velero/Mimir and Supabase Storage OpenBao references. Replace Harbor/Daytona’s literal development credential with a projected, bucket-scoped identity before cutover.
  • Switching existing StatefulSets from hostPath to volumeClaimTemplates is not an in-place data migration. Quiesce writes, copy and verify every bucket in a parallel PVC-backed cluster, then replace the immutable StatefulSets. Retire old host directories only after authenticated readback and restore proof.
  • The legacy source Application resolves values from immutable Assembly revision b7342da69afd9c747d0d4776ef0637b715047d57, which matches its live hostPath topology. Moving that pin or manually syncing a newer source revision can attempt an immutable StatefulSet storage conversion. Only the parallel target may consume current PVC-backed values.
  • fs.meta.save protects the Filer Entry tree, including paths, attributes, and chunk references. It intentionally excludes the SystemLog tree and does not preserve arbitrary Filer KV values or copy object payload bytes. The restore proof is therefore a prerequisite, not a replacement for authenticated byte readback and the final frozen-source archive.
  • weed shell reports command errors on stderr while the shell process can still exit successfully. The export/load Jobs reject error: output, require count and completion lines, validate a non-empty gzip artifact and SHA-256, and then require the strict verifier’s totalErrors: 0 summary.
  • A healthy S3 Deployment does not prove bucket metadata exists. On 2026-08-03, five declared bucket directories were absent while both S3 pods were Ready; Mimir then restarted because fractalops-mimir-blocks did not exist. Verify the exact bucket list and keep the native-command reconciler healthy.
  • Ready S3 and Volume pods also do not prove write capacity. On 2026-08-04, both 150Gi Volume filesystems reached 95%, Master reported Free=0, Barman WAL archive returned exit status 4, and synchronous PostgreSQL writes stalled. The GitOps resize hook expanded both PVCs to 200Gi; recovery required Free>0, a non-empty fractalops-runtime-backups writable set, zero new S3 No writable volumes errors, and CNPG replication catch-up. Never delete WAL or Filer paths to create emergency headroom.
  • -max=0 is not a startup-only slot cache. SeaweedFS 4.37 recalculates the limit during volume heartbeats from free bytes after subtracting capacity already reserved inside existing volumes. On 2026-08-20, filesystems still showed 4-8% free while one replica advertised zero new slots, so restarting Volume pods changed nothing. Expand capacity through the CUE-owned PVC size; do not restart pods, delete objects, or restore storage as a capacity repair.
  • SeaweedFS 4.37 fs.meta.load starts file CreateEntry calls asynchronously but returns at archive EOF without waiting for the final call. The proof keeps the same pinned weed shell process alive with the native sleep command, then re-exports the isolated tree and requires its directory/file counts to equal the original export. A reported load count and is loaded line alone are not restore evidence. Re-export bytes are not a gate because upstream protobuf serialization is not canonical.
  • Filer metadata changes are also written to the native SystemLog object stream. The isolated proof therefore includes one loopback-only Volume server with one bounded 128MiB volume; disabling or ignoring SystemLog errors would make the restore proof false-green. The Job retains all three daemon logs and rejects metadata log write failed or No writable volumes before success.
  • The live source keeps accepting writes during this pre-seed proof. Verification therefore runs from restored artifact (A) to live source (B) with upstream -isActivePassive: every artifact entry must exist and match, while newer source-only entries are ignored. This does not replace the symmetric zero-diff verification after writer freeze; that remains a separate cutover gate.
  • Proof resources use a generation suffix (-001) and Prune=false. Never overwrite or rerun a completed generation; advance the suffix so the artifact, snapshot, restore volume, Jobs, and logs remain attributable together. Argo CD also requires one immutable Application per generation: changing only nameSuffix on an existing Application makes retained Jobs participate in the next sync and hits immutable PodTemplate fields. Pin the old Application to its source revision and create the next numbered Application instead.
  • The native filer.backup command persists its backup. checkpoint in the source Filer KV and resumes it during the tail phase. -initialSnapshot walks the live tree and overwrites that checkpoint on every process start. Sync wave 1 therefore runs one Job with backoffLimit: 0, a six-hour deadline, and -initialSnapshot. It succeeds only after the pinned upstream message initialSnapshot done; subscribing from proves the walk watermark was saved, then stops that process. The base tail Deployment stays at zero replicas and never contains -initialSnapshot. A separate commit pins the manual Application to this immutable composition and patches only tail replicas to one. Sync wave 2 can therefore resume the saved checkpoint only after the seed Job succeeds; no Argo patch or operator edit can mix seed and tail arguments.
  • The copy rail denies ingress. Egress permits only DNS, source Filer HTTP/gRPC on TCP 8888/18888, target Filer gRPC on TCP 18888, and target Volume HTTP on TCP 8080 for chunk writes. No Kubernetes API, public network, S3 endpoint, or master access is granted.
  • -doDeleteFiles is intentionally absent. A source deletion must not erase the parallel target before writer freeze, strict verification, and cutover approval.
  • The copier also sets -ignore404Error=false; a missing source chunk is a visible failed copy, never an implicitly accepted hole. Its replication TOML is a Kustomize-generated, name-hashed ConfigMap so a configuration change changes the Pod template reference before the copier can resume.
  • The separate manual verification Application first adds -modifiedTimeAgo=5m for lag-tolerant checks. After every writer is frozen, remove that patch and run the strict base Job. Each run advances its generation suffix and retains the ordinary Job; no successful proof is deleted as a Hook. SeaweedFS filer.sync.verify proves source-to-target presence, size, and metadata ETag; target fs.verify proves referenced needles exist. Neither reads all payload bytes. Cutover therefore remains blocked until a pinned standard download verifier, such as rclone check --download, records byte-level evidence.
  • Pinned upstream filer.backup intentionally ignores metadata events whose event directory is below /buckets/<bucket>/.uploads/; these are S3 multipart staging entries, not completed objects. Retained verification generation 001 consequently exited 2 with directories=30717, files=0, skippedRecent=0, missing=3, and every other difference counter at zero. The recovered four-line NDJSON and one-line terminal error are retained as the generation 002 contract fixture, including all three observed staging paths.
  • Generation 002 keeps upstream filer.sync.verify as the metadata verifier and validates its pinned NDJSON with the official digest-pinned jq 1.7.1 image. It accepts only exact MISSING directory records strictly below /buckets/<one-segment>/.uploads/<descendant>. The summary’s missing and totalErrors, the accepted record count, and the pinned terminal error count must agree; sizeMismatch, etagMismatch, and onlyInB must remain zero. Exit status must be exactly 2, and stderr must contain only the pinned found N differences terminal line. A near path, file, extra field, extra log, malformed JSON, or any count disagreement fails closed. The native target fs.verify container still runs only after that bounded metadata proof passes.
  • Live generation 002 then observed 25 misses after a failed Harbor upload: eight shaped upload directories and seventeen shaped multipart .part files, with sizeMismatch=0, etagMismatch=0, and onlyInB=0. Generation 003 accepts files only when the path contains the pinned 40-hex object hash, 32-hex upload id, numeric part index, UUID, and .part suffix; it also requires a positive integer size and a 32-hex ETag. Directory and part counts are emitted separately. Any ordinary object path, malformed upload path, zero-size part, unexpected field, warning, or verifier status still fails closed.
  • Generation 003 proved the shaped records but rejected the native terminal line because the live 4.37 binary omitted the historical ] after source line filer_sync_verify.go:94. Generation 004 accepts exactly those two observed source-location spellings while retaining full-line anchoring and the exact difference count. Any other stderr content remains a hard failure.
  • Byte proof uses the separately pinned Rclone 1.74.2 composition. Its manual generation compares every completed object in the five legacy buckets with rclone check --download and proves the target-only Supabase Storage bucket is empty. This online proof still requires a fresh generation after writer freeze before endpoint cutover.
  • Supabase is not one of those source SeaweedFS writers: its retiring Storage API uses a file PVC. The target-only fractalops-supabase-storage bucket may adopt the durable endpoint independently; Velero, Mimir, and Daytona remain on the source endpoint until their frozen-source proof completes.
  • The target Argo Application is manual. Disable auto-sync on the live child Application before merging these values; otherwise its existing main source can begin the immutable cutover before the StorageClass and credentials exist.
  • The chart emits a retained random seaweedfs-db-secret even while LevelDB2 is selected. It is not S3 authentication or CNPG wiring. Do not consume it.
  • The standalone S3 Deployment does not checksum an existing config Secret. Secret rotation therefore requires an explicit controlled S3 rollout and authenticated read/write proof; refreshing External Secrets alone is not proof.
  • Create fractalops-longhorn-durable-rwo-v2 before this environment’s PVCs. Its required contract is two CSI replicas, Retain, expansion, and no diskSelector. Another cluster must provide an equivalent environment-owned class and verify capacity, failure domains, snapshots/backups, and restore.

Verification

Section titled “Verification”
Terminal window
CUE_BIN=.tools/cue/cue platform/k8s/cue-generate.sh lxc-pve-lab
uv run pytest -q platform/k8s/object_store_plane_contract_test.py
uv run pytest -q \
  platform/k8s/composition/seaweedfs/bucket_reconcile_contract_test.py
uv run pytest -q \
  platform/k8s/composition/seaweedfs-durable-copy/seaweedfs_durable_copy_contract_test.py
uv run pytest -q \
  platform/k8s/composition/seaweedfs-source-metadata-restore-proof/seaweedfs_source_metadata_restore_proof_contract_test.py
helm template seaweedfs seaweedfs \
  --repo https://seaweedfs.github.io/seaweedfs/helm \
  --version 4.37.0 --namespace seaweedfs \
  -f platform/k8s/environments/lxc-pve-lab/values/seaweedfs.yaml

Authoritative references: pinned chart source, pinned values, pinned filer.backup source, pinned fs.meta.save source, pinned fs.meta.load source, pinned filer.sync.verify source, pinned fs.verify source, published chart repository, External Secrets templates, Kubernetes StatefulSets, and Kubernetes StorageClasses.