AgentSquad Operations

AgentSquad 런/세션 운영 (AgentSquad Operations)

이 런북은 라이브 Studio/API 표면에서 AgentSquad 런을 구동하고 관찰하는 SSOT입니다. 라이브 API를 직접 보고 작성했기 때문에 경로를 추측하지 않고 따라갈 수 있습니다.

운영 접근 (Operations access)

API base는 topology control_plane.fractalops.edge_bridge.ports.api 또는 배포 env의 FRACTALOPS_API_GATEWAY_URL에서 옵니다. workflow/script에 lab 숫자 주소를 다시 하드코딩하지 않습니다.
Kubernetes 관찰은 Headlamp를 우선합니다. 외부 UI URL은 connector inventory/Gateway route가 투영한 FRACTALOPS_HEADLAMP_URL이고, 자동 관찰은 Kubernetes Service DNS인 http://headlamp.kube-system.svc.cluster.local에서 투영한 FRACTALOPS_HEADLAMP_EXECUTOR_URL을 씁니다. Headlamp 자동 관찰 경로에는 node IP, hostPort, edge bridge endpoint를 SSOT로 두지 않습니다. GitOps는 kube-system/fractalops-headlamp-observer service account와 token Secret을 만들고 ESO PushSecret가 token을 OpenBao fractalops/default/apps/headlamp로 화해합니다. 이 token은 read-only view와 metrics.k8s.io read 권한만 갖습니다. 삭제/재시작 같은 쓰기 개입은 사람 OIDC admin group으로 Headlamp에서 수행하세요.
Kubernetes read path는 Headlamp 화면 또는 headlamp_proxy operation asset의 http_request가 기본입니다. 자동 관찰은 Headlamp /clusters/main/... proxy path를 명시적으로 호출하고, observer bearer token은 env 또는 OpenBao apps/headlamp:FRACTALOPS_HEADLAMP_OBSERVER_TOKEN에서 해석합니다. operation-assets check --role kubernetes_observation은 proxy URL만 보지 않고 credential source까지 보여야 합니다. kubectl 형태의 CLI는 fractalops_control_plane 같은 kubernetes_operator_api transport나 bootstrap/break-glass에만 남깁니다.
인증: bearer 토큰이 아니라 엣지 Pomerium claim 헤더입니다.
- x-pomerium-claim-email: <you>@yamon.io
- x-pomerium-claim-groups: /org/platform/roles/super_admin
Health: GET /healthz → {"status":"ok"}.
GET /openapi.json 이 권위 있는 엔드포인트 목록 — grep 하고, 경로를 추측하지 마세요.
Local Studio CLI observation should use the control-plane API when credentials are available: python ops/cli/live_studio_api.py --portal-url "$FRACTALOPS_API_GATEWAY_URL" --api-token "$FRACTALOPS_API_TOKEN" studio-fleet. This keeps local operators off host-specific /etc/rancher/k3s/k3s.yaml. If no direct API credential is supplied, the CLI falls back to the Kubernetes service proxy for in-cluster/break-glass use.

# health
curl -s "${FRACTALOPS_API_GATEWAY_URL:?set from topology/env}/healthz"
# 권위 있는 경로 목록 (스튜디오 관련만)
curl -s "${FRACTALOPS_API_GATEWAY_URL:?set from topology/env}/openapi.json" \
  | jq -r '.paths | keys[] | select(contains("studio"))'

Headlamp-first Kubernetes observation

Headlamp is the default Kubernetes operator console. Use it before adding shell transport, because it already gives the cluster read model, watch stream, events, logs, and manifest preview in one authenticated UI.

Architecture view: Kubernetes Observation Edge.
Open the Headlamp URL projected from connector inventory (FRACTALOPS_HEADLAMP_URL) through the Pomerium session.
Start from namespace fractalops for API, worker, Temporal, and Studio run symptoms. Use workload labels before opening individual pods.
For stalled rollouts, open the Deployment or ReplicaSet and inspect Pod/Workload detail의 Diagnostics, conditions, related events, and container restarts from the same page.
For run execution stalls, open Pods and use Logs from the target container; use the Events table to separate scheduling, image pull, probe, PVC, and RBAC failures before touching the workload.
For Temporal or batch repair paths, use Job detail instead of the generic workload page so completions, parallelism, active/succeeded/failed counts, and related pod logs are visible together.
For manifest edits, use YAML editor의 Dry Run first. Apply only from a human OIDC admin session when the dry-run output matches the intended diff.
Advanced Search can be used for repeated app-label or namespace queries; saved searches are browser-local operator conveniences, not repo config.
Do not confuse Cluster Inventory provider plugins with UI plugins. The chart plugins entries mount provider binary image volumes for Cluster Inventory; UI plugin install/watch is the separate pluginsManager + config.watchPlugins=true surface. Custom operator screens should use the Headlamp UI plugin APIs such as registerRoute, registerSidebarEntry, registerResourceTableColumnsProcessor, and registerProjectApiResource before adding another local dashboard widget or bespoke Kubernetes read model.
Treat Headlamp 0.43 built-ins as the default implementation surface for Kubernetes UI features: proxy auth, workload Diagnostics, Job logs, YAML Dry Run, Advanced Search, and Projects CRD registration all come before a Portal widget or a custom polling endpoint.

Automation uses the same Headlamp backend proxy contract instead of raw Kubernetes client calls. Automation authentication is observer bearer only: never forward a browser/Pomerium cookie from env or caller headers into the Headlamp proxy. The cookie belongs to a human OIDC session; the automation identity is the OpenBao-synced FRACTALOPS_HEADLAMP_OBSERVER_TOKEN.

uv run fractalops operation-assets request --role kubernetes_observation \
  --method GET \
  --path '/api/v1/namespaces/fractalops/pods?labelSelector=app%3Dfractalops-api'

uv run fractalops operation-assets request --role kubernetes_observation \
  --method GET \
  --path /apis/postgresql.cnpg.io/v1/namespaces/fractalops/clusters/fractalops-postgresql

For CI/runbook probes that must fail loud instead of returning an empty object, set FRACTALOPS_HEADLAMP_OBSERVE_STRICT=true. This prevents Headlamp transport, credential, or proxy failures from being misread as “no pods/no nodes”.

The kubernetes_observation asset maps to Headlamp’s /clusters/main/... proxy and carries only read intent. Its URL source is FRACTALOPS_HEADLAMP_EXECUTOR_URL, derived from the connector executor_internal_url Kubernetes Service DNS. It is not the public Pomerium URL and not a pinned edge node/hostPort. Write operations such as annotate, apply, delete, rollout restart, and pod exec are not part of that transport.

Runtime build pre-cleanup follows the same split. It reads buildkit pod state through operation-assets request --role kubernetes_observation and only uses the fractalops_control_plane Kubernetes asset for the bounded delete pod mutation when a pod is Pending, Terminating, failed, or Running/NotReady. Runtime rollout observation follows the same read/write boundary: Argo CD app, Deployment, Pod, and Event JSON reads use the Headlamp-backed observation asset, while the Argo CD hard-refresh annotation remains the explicit Kubernetes mutation through fractalops_control_plane.

RSS ranking requires the metrics API through the same observation boundary. If nodes.metrics.k8s.io or pods.metrics.k8s.io is forbidden, the operator can still trust Node Ready/MemoryPressure/DiskPressure conditions and workload placement, but it cannot claim top pod memory usage from that run. Grant the observer read-only metrics access before making memory-top decisions from automation.

Do not enable Headlamp’s unsafe in-cluster service-account-token auth as a shortcut for automation. It authenticates users as the pod service account behind a proxy and would erase the group-based RBAC split. Automation gets the separate read-only observer token from OpenBao; humans use OIDC groups.

Gateway API migration is allowed only after it preserves the current legacy Ingress contract and the authorization chain. Keycloak remains the identity and group source; SpiceDB remains the application-resource permission SSOT and must continue to serve the real CheckPermission decision path. Headlamp keeps the Pomerium Ingress active today. The disabled httpRoute values in platform/k8s/apps/headlamp/values.yaml are a parity candidate, not a live route. Before enabling it, the cluster must have a Pomerium GatewayClass, an HTTPS Gateway listener for k3s.yamon.io, and a route-level Pomerium PolicyFilter; Pomerium denies Gateway routes without an explicit policy. Validate Keycloak login, Pomerium claim headers, SpiceDB-backed group/resource authorization, Headlamp workload diagnostics, logs, events, OIDC callback, and WebSocket-backed watches before removing the Ingress.

Feature flags for Gateway API must not bypass the auth chain. Human traffic cohorts are canonical group paths materialized through Keycloak or principal-assignments and checked through SpiceDB. Armory loadouts are useful for Studio validation canaries because they select MCP, skill, policy, and test-subject bundles for agents; they are not an edge traffic router and must not be treated as the source of truth for per-user Gateway rollout.

Secret lifecycle stays separate from authorization lifecycle. OpenBao owns secret/token generation, rotation, lease, and expiry. ESO syncs external secret values into Kubernetes Secrets. Kubernetes ServiceAccounts are the pod identity for Kubernetes API access. SpiceDB can decide whether a user, agent, workspace, or project may receive a secret template, service-account grant, or deployment evidence surface, but it must not be modeled as the secret issuer or Kubernetes RBAC creator. Deployment rollout evidence uses K8s least-privilege events/pods/deployments reads; user/agent visibility is a SpiceDB deployment_evidence#view decision.

런 라이프사이클 한눈에 보기

flowchart LR
  launch["POST /runs\n(template_id + project_slug)"] --> ready["status: ready\nagent_roster=[planner,curator]"]
  ready --> exec["POST /runs/{id}/execute\nstatus: scheduled (ASYNC)"]
  exec --> poll["GET /runs/{id}/session-state\n(poll)"]
  poll --> warm["waiting_first_tool\n(agent process warming)"]
  warm --> work["first tool call -> working"]
  work --> done["session_completed"]
  work -. 핸드오프 .-> joined["roster에 에이전트 합류"]
  warm -. 첫 도구 미호출 .-> blocked["blocked\nagent_first_tool_required"]

스튜디오 런 API (admin)

프리픽스는 /v1/admin/studio 입니다 (/v1/studio 아님 — 그건 404). 테넌트는 X-FractalOps-Tenant-Id 헤더로 전달합니다.

동작	엔드포인트
런 시작	`POST /v1/admin/studio/runs`
런 조회	`GET /v1/admin/studio/runs/{run_id}`
실행(에이전트 tick)	`POST /v1/admin/studio/runs/{run_id}/execute`
세션 상태(관찰)	`GET /v1/admin/studio/runs/{run_id}/session-state`
리포트(tick별)	`GET /v1/admin/studio/runs/{run_id}/reports` (LIST 반환)
핸드오프 추적	`GET /v1/admin/studio/runs/{run_id}/handoff-trail`
정리/제어/리플레이	`POST .../runs/{run_id}/{cleanup,control,replay}`
템플릿	`GET /v1/admin/studio/templates`

템플릿

agentsquad, agentsquad, agentsquad (프로젝트 딜리버리 / 마이그레이션), research-team.

런 시작 바디 (`StudioRunIn`)

{
  "template_id": "agentsquad",
  "project_slug": "sanmopia-modernization",
  "agent_process_kind": "project_delivery",
  "work_scope": "<the task>",
  "next_goal": "<first concrete step>"
}

런 시작은 ~6초 안에 {run_id, status: "ready", agent_roster: [...]}를 반환합니다. 로스터는 intake 페어(planner + curator)로 시작하고, 핸드오프로 에이전트가 더 합류합니다.
execute는 status: "scheduled"로 빠르게 반환됩니다 — tick은 ASYNC로 돕니다. session-state를 폴링해 진행을 보세요.
GitHub Actions agentsquad-ensure는 기본값으로 dispatch-only입니다(progress_witness_timeout_seconds=0). Actions runner가 관측 루프를 점유하면 여러 프로젝트가 병렬로 시작하지 못하므로, 장기 감시는 Studio/Temporal/Portal feed가 맡습니다. 일회성 라이브 증명이 필요할 때만 witness timeout을 명시적으로 올립니다.
observability-feed는 coordination_friction을 노출합니다. 이 블록은 최종 응답 텍스트를 파싱하지 않고 session-state의 구조화 필드만 집계해 first_tool_waiting_agents, handoff_blocked_agents, process_switch_handoff_required_agents, mailbox_pending_count, known_duration_ms, known_turn_count를 보여줍니다.
GitHub Actions runner pool은 역할별 label을 요구합니다. Image build, runtime-release, and image-producing AgentSquad delivery jobs use build; control/diagnostic/merge-court/AgentSquad ensure jobs use general; static preview delivery uses delivery. Heavy builds must not target the broad [self-hosted, linux, internal, k8s] label set without a pool label, or they can starve control-plane proof and project start workflows.
Runner lifecycle trap (myoung34/github-runner:2.335.1-ubuntu-noble, checked 2026-07-26): EPHEMERAL is for autoscaling runners whose container is replaced after every job. The fixed one-replica Deployments therefore use the image’s persistent default; enabling EPHEMERAL makes Kubernetes restart the same container with growing backoff and starves queued jobs. Adopt GitHub ARC runner scale sets before re-enabling ephemeral mode.
Rollout order matters: deploy the chart labels first while general replicas stay at 0, then migrate workflows to explicit general labels, then scale the general pool up. Existing broad-label workflows must keep scheduling on the build pool until that migration lands.

AgentSquad assembly migration contract

AgentSquad source migration은 project assets의 repositoryBindings와 sourceRefRepos가 계약입니다. assembly repository는 큐와 증거를 소유합니다. 구현 레포가 아닙니다.

Delivery repos: repositoryBindings must include one assembly binding and the writable delivery lanes (backend, contract, frontend, or project-specific lane keys).
Assembly repo: feature issues, evidence pack, source map, handoff state, preview config.
Writable implementation lanes: non-assembly delivery repos only. Assembly-only PR은 orchestration artifact로 보고 구현 완료 증거로 세지 않습니다.
Mandatory source refs: project sourceRefRepos. Source refs and repository bindings are separate assets; do not mix them.
Session-state projection policy: Studio read models and Temporal monitors must accept both API camelCase (projectRepositoryBindings, sourceRefRepos) and persisted snake_case (project_repository_bindings, source_ref_repos) metadata. Retry/control paths may merge snake_case payloads; casing drift must not be interpreted as missing repository ownership or missing source refs. Keep accepted aliases in the Studio projection feature slice, not in standalone watcher scripts.
Agent process assignment default: AgentSquad migration sessions use stable project-agent identity with replaceable process assignment. Coordination and delivery roles default to codex with the CliProxy alias fractalops-codex; tester roles default to antigravity with the alias fractalops-antigravity. If claude-code-cli is selected for a role and no model override is provided, it resolves to fractalops-claude-code. CliProxy owns alias-to-upstream model routing; FractalOps must not scatter upstream model ids across project config. This is not the agent identity; project agent identity stays stable across provider/model replacement.
Agent process assignment switch: use fractalops projects agents assign-processes --assignment target:provider:model:effort:speed. target may be an agent id (planner), a role key (role_tester), or a tester group alias (testers). The command resolves existing sessions, writes agent_process_assignment / agent_process_switch_* metadata, preserves the same worktree, and requires a takeover handoff before the replacement process starts. Do not hand-roll shell loops around switch_agent_process.
Suit proof: planner/curator spawn tester handoffs for every reusable actor suit declared in agentSquadSuitTesterRoles; each project declares its own actor list.
Queue policy: pendingRunExecution is processed FIFO by requested_at; newer follow-up kicks must not starve older scheduled runs.
Watch timeout policy: operator projection controller calls are bounded by --controller-timeout-seconds (default 25s), below the 30s monitor polling cadence. Named Studio/API reads use the Kubernetes service proxy or typed client paths; they must not select an API pod and shell into it. A slow live lookup must become a structured blocker such as project_assets_lookup_failed, with source_refs.lookup_error carrying the transport/API reason, not a hung watcher process.
Live API probe shell policy: live_studio_api.py transport probes run through non-login shells. They must not source stale .bashrc workspace env files or depend on per-agent worktree activation just to read Studio/project-assets state.
Watch source policy: the watcher derives project context only from explicit --project-assets-json or Studio session-state. It must not hide sparse session-state by calling project-assets as a secondary lookup.
First-tool policy: agent_first_tool_required restarts only the stale agent process in the same assigned workspace, then interviews/probes there (pwd && git status --short) before any fresh workspace is allowed. Controller actions (dispatch_slot_prompt, agent_mailbox, acquire_execution_slot, HUD reports) are not delivery first tools; they must not become first_tool_not_required_tool evidence.
Workspace policy: fresh workspace is only for source materialization failure, corrupt checkout, or missing runtime-memory checkpoint after terminal cleanup. Do not use Claude/session reset as a synonym for fresh workspace.
Self-heal pending policy: executionWorkspaceSelfHealStatus=pending_reprovision is a workspace lifecycle signal, even when another wall such as project_delivery_completion_requires_pr is currently displayed. If the same session also shows a stale runner/toolbox signal (daytona_toolbox_observe_failed, sandbox_missing, sandbox reaped, No such container, or detached execution runner process is not alive), the dispatcher must clear stale workspace identity metadata and requeue a fresh launch instead of leaving the session permanently blocked.
Daytona start recovery policy: a provider workspace whose /start returns Docker/runc container creation corruption is not a resumable stopped sandbox. Treat it like a missing provider binding and recreate once through the shared workspace registry. After the project workspace rebinds, any session row whose execution_workspace_id still points at the old provider id must be revived as a stale execution-slot binding in the active shared workspace instead of reasserting waiting_first_tool.
Daytona credential policy: daytona_api_credentials_not_configured is not proven by an empty Studio session row alone. The studio profile resolves its OpenBao-backed rotation credential through DaytonaWorkspaceService(credential_profile="studio").credential_metadata(); the admin profile uses only the explicit FRACTALOPS_DAYTONA_ADMIN_API_TOKEN secret and must not alias the default Daytona token. If the primary API key status endpoint returns 401/403, classify it as daytona_api_credentials_invalid, not as missing configuration; rotate the secret instead of creating fresh workspaces. A 401/403 from the advisory runner scheduler endpoint is daytona_runner_status_forbidden and must not block sandbox launch by itself.
Wall retry policy: if a wall has force_fresh_workspace=false, its prompt must not ask for a fresh execution workspace; it may only ask for a fresh agent process, same-workspace probe, or fresh delivery branch when the branch/PR is contaminated.
Reaper policy: off Daytona workspaces are stop/delete candidates only after terminal status and runtime memory checkpoint evidence (fractalops_runtime_memory_archived=true). Live runs, idle-warm workspaces, and stuck-materializing runs stay observable and converge through the heartbeat sweep instead of ad hoc deletion.
Local workspace hygiene policy: ignored generated artifacts and clean unlocked .claude/worktrees/* entries are removed through fractalops operations reap-runtime-resources --local-workspace-hygiene-only --apply. The command skips tracked content, locked worktrees, dirty worktrees, .agentsquad/threads, .serena/memories, .state/secrets, dependency installs, and virtualenv/tool caches. Run without --apply first to get the deletion report.
Monitor projection policy: display the currently active blocker, not stale launch-contract metadata. If a fresh runner is started/running and the session is waiting_first_tool or daytona_toolbox_observe_failed, old no_browser_before_completion / first_tool_not_required_tool reasons must be hidden or cleared before the YOLO monitor decides the next action.

Reusable migration runtime profile:

python ops/cli/live_studio_api.py update-project-studio-config \
  --project-slug <project-slug> \
  --suits planner,curator,contract,backend,frontend,tester,committer,closer \
  --agent-profile-preset project-migration \
  --coding-runtime-provider codex \
  --testing-runtime-provider antigravity \
  --agent-profile-effort medium \
  --agent-profile-speed normal \
  --tester-agent-ids <optional-extra-suit-tester-agent-ids> \
  --replace-agent-profiles

The preset is project-agnostic and provider-neutral: it only separates coding/planning roles from tester and *-tester suit roles. Runtime selection is supplied by --coding-runtime-provider, --coding-runtime-model, --testing-runtime-provider, --testing-runtime-model, --agent-profile-model, --agent-profile-effort, and --agent-profile-speed. --agent-profile-model is an optional shared override; leave it empty when coding and tester roles must use provider-specific CliProxy aliases. Unknown future providers must either expose the derived fractalops-<provider> alias in CliProxy or pass an explicit model alias, instead of adding a new project-specific preset.

CliProxy is the upstream model SSOT. The current GitOps alias mapping lives in platform/k8s/apps/cliproxy/values.yaml under provider.oauthModelAliases; the chart renders it to CliProxy oauth-model-alias. Changing upstream model ids is a CliProxy config change, not a Studio project/profile migration. FractalOps code should carry only client-facing aliases from the agent-process alias policy (fractalops-codex, fractalops-claude-code, fractalops-antigravity). Live verification uses CliProxy’s own /v1/models catalog through the cliproxy-identity-probe action and fails if any required alias is missing. Do not add upstream model ids to project assets, Studio profiles, run metadata, or delivery repos. Use --tester-agent-ids for project-specific actor-suit tester ids. Use explicit assign-processes --assignment target:provider:model:effort:speed entries to override individual roles without forking the preset. For example, Sanmopia can run planner:claude-code-cli:fractalops-claude-code:medium:normal and role_tester:antigravity-cli:fractalops-antigravity:medium:normal while keeping the same planner/tester nicknames and worktrees.

Studio session-state is the operator projection source. If the payload is sparse, the projection must expose the missing repository/source-ref context instead of silently repairing it with a second project-assets lookup.

Orchestration ownership split:

Temporal owns durable polling, timers, FIFO pendingRunExecution, retry cadence, and wakeup activities.
Studio owns run/session/mailbox truth, route frontier, next action, and control verbs.
Agent Execution owns provider policy and the process adapter boundary.
Standalone watcher CLIs are retired. Operator projection belongs to Studio read models; durable cadence and retries belong to Temporal.
Local CLI waits must be bounded through ops/cli/control_flow/polling.py. live_studio_api.py, run_studio_session.py, verify_agentsquad_execution_plane.py, and AgentSquad turn wait paths use that shared primitive; do not add new root-level ops/cli/*.py wait loops.
Pending run execution calls Studio execute_run once. Do not insert a second queue, remote planner, or polling gateway between Temporal and Studio.

The Studio projection classifies agent_first_tool_required, missing source refs, no-diff completions, hollow PRs, assembly implementation claims, repeated blockers, and 10-minute stalls into next actions. It emits control_plan[] entries that map live session_id values to existing Studio control verbs (retry_prompt or comment_agent). It must not own a polling timer, retry cadence, workspace lifecycle decision, or autonomous recovery loop. Packaged wall-policy entries are projected directly: force_restart_process=true becomes force_restart_execution_process=true, and force_fresh_workspace=true stays explicit. session_runner_orphaned restarts only the agent process because its wall policy does not allow a fresh workspace; toolbox/container/workspace blockers may enter the workspace recovery choke point only when wall policy or recovery classification allows it. Terminal/off workspace deletion still requires runtime-memory checkpoint first.

Studio stores one active pendingRunControl plus a FIFO pendingRunControlQueue. Multiple session controls emitted from one watch pass must be queued and promoted one at a time after the active control clears; they must not overwrite each other or only the last agent will be retried.

Control application uses existing Studio control-session verbs. It does not poll, sleep, or decide a new cadence; use it for operator/YOLO kicks when the projection is already acceptable and let Studio/Temporal consume the queued controls.

Agent-process autonomy and observation are adapter-owned. Do not treat “YOLO” as one global CLI flag:

Codex uses autonomyProfile=codex_workspace_no_prompts with launch args --sandbox workspace-write --ask-for-approval never; full access requires the explicit codex_danger_full_access override.
Claude uses autonomyProfile=claude_bypass_permissions with --permission-mode bypassPermissions.
Antigravity uses autonomyProfile=antigravity_allow_rules; the installed Antigravity environment must provide the matching permission-rule preset through its shared settings.

The board must render and control sessions from autonomyProfile, observationEventMap, and boardControlVerbs, not by re-parsing provider-specific stream text. Armory state is separate: mcpServerIds, armorySkillIds, and armoryMcpCapabilityNegotiations describe what was mounted; agent_process_mcp_status describes what the process stream actually reported. Empty MCP status means unknown, not failed.

Studio control state must come from explicit tools, hooks, or runner status-channel payloads: HUD MCP report/handoff calls, hook payloads, observed real tool calls, and STUDIO_SESSION_PAYLOAD: runner lines. Agent final prose, raw stdout JSON, and repaired final JSON are diagnostic text only. If an agent exits without HUD MCP report/handoff or runner status payload, classify it as agent_report_transport_required and resume the same session; do not parse the final answer to infer completion, blocker, PR delivery, or handoff.

Workspace-tools consumes this contract. Codex, Claude, and Antigravity runner args come from launchArgs/FRACTALOPS_AGENT_PROCESS_LAUNCH_ARGS; they must not hardcode a separate permission mode. Antigravity tester sessions launch the official agy CLI adapter with print mode and the configured autonomy args. If agy is not installed or authenticated, the session blocks with a specific Antigravity blocker instead of silently falling back to Claude.

Client runtime packages are shared per workspace/runtime version. Do not install one Claude, Codex, or Antigravity client per named agent. Session isolation lives under FRACTALOPS_AGENT_PROCESS_STATE_ROOT; provider package bits live under the shared runtime package root and the shared workspace-tools prefix. If eight Claude agents run in one shared workspace, they may have eight state roots, but they must not have eight client package installs.

Repeated first-tool walls are finite. After three same-blocker/controller interventions, the watch control plan switches from retry_prompt to comment_agent interview in the same workspace. This preserves agent memory and diffs while forcing the agent to report the concrete missing tool/workspace reason instead of cycling on another wake. Session read models must expose agent_effort; if the run/profile does not provide one, AgentSquad sessions use the configured default (medium) so live Studio views show the same effort that the runner sends to the model.

Current live verification contract:

Runtime images must match the GitOps digest before declaring a control-plane fix live. Verify the fractalops-api and fractalops-worker Argo Applications, Deployments, and pod imageID values.
Runtime rollout observe is bounded: the workflow job has timeout-minutes: 12, and each rollout observe phase uses FRACTALOPS_ROLLOUT_TIMEOUT=180s. A slow Argo/image/deployment convergence must surface as CI evidence instead of occupying the self-hosted runner for tens of minutes.
Project-specific run IDs are incident probes, not reusable runtime contracts. For the 2026-06-24 Sanmopia incident, run e02d23fc-e6d0-4537-993f-a21a35d0c939 is the active probe for controller pseudo-tool leakage and stale workspace self-heal behavior. DB checks must report bad_first_tool_not_required_tool=0 and slot_acquire_bad=0 before treating the first-tool monitor path as healthy.
Remaining blockers after that check (session_runner_orphaned, daytona_toolbox_observe_failed, project_delivery_completion_requires_pr, role-tool walls such as closer_datahub_tool_required) are separate runtime/workspace/delivery blockers. Do not fold them back into first-tool policy.

For offline proof or incident replay, archive the captured Studio session-state, workspace diff summary, PR status, and wall-policy projection as evidence artifacts on the project assembly issue.

에이전트 신원 모델 (로스터에 보이는 대로)

각 로스터 에이전트는 다음을 가집니다.

user_principal_name / subject_key = <role>@yamon.io (예: planner@yamon.io), identity_binding_mode: named_agent_microsoft, secret_lease_ref: named-agent/<role> — msgraph-바인딩된 named-agent UPN.
agent_nickname = 자동 생성 핸들(예: dangerous-vector). 프로젝트 squad + role별 결정적이며 provider/model 교체나 run 재시작으로 바뀌지 않습니다. 명시 nickname이 있으면 그 값을 사용합니다.
ContextForge gateway subject는 agentsquad::profile::<tenant>::<project>::<profile-key>입니다. provider나 run이 바뀌어도 같은 ProjectAgentProfile이면 유지됩니다. Gateway가 검증한 X-Forwarded-User-* 신원으로 upstream MCP가 프로젝트와 profile을 바인딩합니다. Agent가 보낸 신원 헤더는 권한 근거가 아닙니다.
skill_ids / mcp_server_ids는 로스터 스냅샷에서 비어 있습니다. loadout은 로스터 뷰가 아니라 세션 시작 시 해결됩니다.

런 관찰 (Observing a run)

session-state를 폴링하세요. 각 세션은 live_status, current_step, blocker_code, session_id, claude_session_id, tool_call_count를 보고합니다.

라이프사이클: ready → waiting_first_tool (agent process warming) → 첫 도구 호출 → working → session_completed. 한 에이전트라도 blocked면 런은 blocked입니다.

흔한 블로커 (와 그 의미)

agent_first_tool_required — 에이전트의 Claude 세션은 시작됐지만(claude_session_id가 설정됨) 요구된 첫 도구 호출을 하지 않아 런치 컨트랙트가 막은 것. intake 단계에서 재발합니다. 확인: (1) 에이전트의 MCP 도구가 실제로 붙었는지(요구되는 첫 도구는 보통 MCP/HUD 도구 — MCP 설정이나 gateway auth가 깨지면 에이전트가 그 도구를 아예 호출할 수 없음), (2) 디스패치 프롬프트가 도구 호출을 끌어냈는지.
전체 블로커 집합은 GET get_hud_snapshot → execution_report_contract.BLOCKERS에서 보세요.

라이브 검증됨 (2026-08)

ContextForge 1.0.5 gateway는 profile subject-scoped token을 검증합니다. Agent traffic은 admin token으로 fallback하지 않습니다. Admin 신원은 catalog health, user provisioning, rollout registration control path에만 제한됩니다.
ContextForge는 shared CNPG를 사용하며 application SQLite PVC는 없습니다. Armory rollout은 strict Argo PostSync Job으로 필수 MCP 7개를 한 번 reconcile하고, runtime refresh와 health recovery는 ContextForge가 소유합니다.
배포된 fractalops-api 이미지는 머지마다 앞으로 자동 핀(예: gha-2820-<sha>)되어, 머지된 백엔드 변경이 한 번의 핀 범프 안에 라이브가 됩니다.
스킬 poll-pull(/v1/skills)은 관찰된 런에서 intake 에이전트가 호출하지 않았습니다(/v1/skills 호출 0) — 풀 전에 first-tool에서 막히면 workspace provisioning package/image pin이 api 이미지보다 뒤처졌는지 확인하세요.

GitHub 쓰기는 CodexGate App을 거칩니다

스쿼드 PR/이슈는 raw 조직 PAT가 아니라 CodexGate GitHub App(fractalops-codexgate[bot])이 작성합니다. CodexGateGitHubAppService._installation_token이 실제 ghs_… 인스톨레이션 토큰(app id 3249317, 인스톨레이션 120794317 on yamonco, all-repo)을 발급해 git auth와 REST/GraphQL 호출에 씁니다.

flowchart LR
  agent["스쿼드 에이전트"] --> svc["CodexGateGitHubAppService\n._installation_token"]
  secret["k8s secret\nfractalops-codexgate-github-app"] --> svc
  svc --> token["ghs_... installation token\napp 3249317 / inst 120794317"]
  token --> gh["api.github.com\n(PR/issue/git)"]
  bot["fractalops-codexgate[bot]"] -. author .-> gh

자격증명은 k8s secret fractalops-codexgate-github-app에서 옵니다. API/worker/Studio worker가 envFrom으로 소비하며 agent workspace에는 private key를 전달하지 않습니다.
옛 FRACTALOPS_GITHUB_APP_TOKEN override env(조직 PAT)는 모든 곳에서 제거됐습니다. 그게 _installation_token 안에서 App 자격증명을 이기곤 했습니다.
이슈/PR lifecycle은 CodexGate API가 App installation token으로 수행합니다. 로컬/에이전트 작업에서 raw gh issue close/gh pr create를 직접 쓰지 말고 fops github-app ... --via-api 또는 admin API /v1/admin/codexgate/github-app/*를 사용하세요.
workspace gh proxy는 gh pr create/list/view/edit/checks/merge, gh issue list/view/close, issue comment read를 fops github-app ... --via-api로 변환합니다. App private key는 workspace에 두지 않고, GH_TOKEN/GITHUB_TOKEN도 에이전트 세션에 주입하지 않습니다.
api.github.com으로의 control-plane egress는 동작 확인됨.

전체 액터 모델은 Agent PR Submission Pipeline을 보세요.

에러 추적 (GlitchTip)

tester와 compactor 역할은 observability-triage 번들로 GlitchTip MCP를 받습니다(서버 glitchtip + fractalops-hud; 정책 error-triage, performance-regression-check, issue-resolution-proof). /mcp의 공식 GlitchTip MCP는 17개 도구를 노출하고 ContextForge에 gateway yamon-glitchtip으로 등록됩니다.
스캐폴드된 모든 프로젝트는 starter .env에 자동 발급된 프로젝트별 DSN을 받습니다(프론트엔드 PUBLIC_SENTRY_DSN, 백엔드 SENTRY_DSN) — fail-open, 스캐폴딩을 막지 않습니다.
Ingest는 공개(exposure-scope: public)입니다: 프리뷰 앱이 DSN으로 이벤트를 인증 없이 POST(DSN 공개 키가 프로젝트별 auth). UI/관리 API//mcp는 여전히 auth가 필요합니다.

평면 상세는 Build Plane Observability를 보세요.

수동 시크릿 (OpenBao 후속 대기)

다음 런타임 시크릿은 오늘 수동으로 provisioning됩니다. OpenBao + ExternalSecret 마이그레이션은 후속 과제입니다.

fractalops-codexgate-github-app — CodexGate App 자격증명(조직 PAT용 fractalops-github-runtime secret과 같은 수동 패턴).
glitchtip-mcp-bearer (armory 네임스페이스) — armory에 등록된 yamon-glitchtip gateway가 Authorization: Bearer …로 쓰는 GlitchTip 조직 API 토큰.

Daytona 웹/SSH 접근 재현성

Daytona 접근은 세 계층에 걸쳐 있습니다.

flowchart TB
  subgraph gitops["Kubernetes GitOps"]
    tcp["Traefik daytona-ssh TCP entryPoint"]
    route["IngressRouteTCP daytona-ssh-gateway-edge"]
    apienv["Daytona API env\nSSH_GATEWAY_URL=daytona-ssh.yamon.io:2222"]
  end
  subgraph reconcile["daytona-bootstrap region-routing reconcile"]
    region["region.sshGatewayUrl\n= daytona-ssh.yamon.io:2222"]
  end
  subgraph edge["관리형 엣지"]
    dnat["PVE/router 선언\nTCP 443/2222 -> Traefik"]
  end
  gitops --> region
  region --> dnat

Kubernetes GitOps가 Traefik daytona-ssh TCP entryPoint, daytona-ssh-gateway-edge IngressRouteTCP, Daytona API env SSH_GATEWAY_URL=daytona-ssh.yamon.io:2222, 기본 워크스페이스 이미지를 소유.
daytona-bootstrap region-routing reconcile이 Daytona DB region drift를 소유: region.sshGatewayUrl은 반드시 daytona-ssh.yamon.io:2222 — 아니면 대시보드가 쓸 수 없는 ssh -p 2222 ...@daytona.yamon.io 명령을 생성합니다.
PVE/router 포워딩은 Kubernetes 애플리케이션 코드가 소유하지 않습니다. 인프라 프로비저너에서 TCP 443/2222/30000을 Traefik endpoint로 선언하고 변경 증거를 남기세요.

상단 라우터에는 여전히 명시적 포트 포워드가 필요합니다.

TCP 443   -> 192.168.219.10:443
TCP 2222  -> 192.168.219.10:2222
TCP 30000 -> 192.168.219.10:30000

proxy.monstore.io, *.proxy.monstore.io, daytona-ssh.yamon.io, *.ssh.daytona.yamon.io의 Cloudflare DNS 레코드는 DNS-only여야 합니다. SSH 호스트를 Cloudflare로 프록시하지 마세요.

VS Code 접근

VS Code Remote SSH와 Daytona SSH Access를 쓰세요. 현재 Daytona API는 /api/sandbox/{sandboxId}/ssh-access에서 단명 SSH 토큰을 만들고, FractalOps는 그 토큰을 Portal Local VS Code 런치 플로에 다음 형태로 노출합니다.

ssh -p 2222 <token>@daytona-ssh.yamon.io

이 명령을 VS Code Remote SSH에 붙여넣거나, Portal 런치 진행 패널에서 복사하세요. 현재 Daytona 릴리스에서 지원되는 경로입니다.

previous-generation daytonaio.daytona VS Code 확장은 이 런타임의 지원 워크스페이스 클라이언트가 아닙니다. 지원 표면은 Daytona 현재 API(/api/sandbox/*, /api/users/me, 조직/runner 엔드포인트), DevPod Remote SSH, OpenVSCode, Portal preview routing입니다. 옛 루트 레벨 엔드포인트(/cluster, /team, /workspace)와 id.daytona.yamon.io 호환 브리지는 제거됐으며, 해당 확장이 현대 샌드박스를 나열/열 수 있다는 증거로 쓰면 안 됩니다.

트러블슈팅: 런은 뜨는데 샌드박스가 안 뜰 때 (dispatch/materialization)

세션이 ready에 머물거나 current_tool=acquire_execution_slot을 무한 반복하고 실제 armory 툴 사용이 0인 경우. 아래 순서대로 진단하세요 — 한 겹만 고쳐서는 안 되는 다층 문제이므로 각 겹을 스킵하지 마세요.

1. OpenBao 봉인 먼저 (503 → create_run 자체가 막힘)

create_run이 named-agent 시크릿을 읽다 503 Vault is sealed로 죽으면 런 생성이 안 됩니다. 가장 먼저 확인:

ssh pve0 "pct exec 125 -- sh -c 'export BAO_ADDR=http://127.0.0.1:8200; /usr/local/bin/bao status | grep -i sealed'"
# Sealed true 이면 Azure Arc, Key Vault RBAC, KMS plugin 로그를 확인하고 재시작:
ssh pve0 "pct exec 125 -- systemctl restart openbao"
ssh pve0 "pct exec 125 -- journalctl -u openbao -n 100 --no-pager"

수동 Shamir key 제출과 openbao-autounseal.timer 복구는 금지합니다. CT125는 Azure Key Vault auto-unseal이 정본이며 정상 재시작 뒤 Sealed false가 되어야 합니다.

2. `ready` 세션은 자동 tick이 없을 뿐 — 수동 디스패치 드라이버가 있음

temporal-disabled 랩에서 ready 세션은 코드 갭이 아니라 자동 reconcile tick이 없어서 멈춰 있습니다. 수동 드라이버:

API: POST /v1/admin/studio/runs/{run_id}/execute?execution_mode=synchronous → _queue_pending_run_execution + execute_pending_run_execution(reclaim_running=True).
CLI: fractalops projects agents execute --run-id <id> --project-slug <p> --via-api --full (--full이 synchronous로 전환).

3. Daytona `manifest unknown` = 레지스트리 인증 다층 문제 (이미지 부재 아님)

CREATE_SANDBOX ... Error response from daemon: manifest unknown은 보통 “이미지 없음”이 아니라 Daytona가 ghcr 인증을 못 하는 것입니다. runner에서 직접 docker pull을 먼저 찔러 보세요 — 여기서 unauthorized가 나오면 이미지가 아니라 인증 문제입니다.

R=$(kubectl get pods -n daytona -o name | grep -E 'runner-[a-z0-9]+$' | head -1)
kubectl exec -n daytona $R -- docker pull ghcr.io/yamonco/fractalops-api:daytona-workspace-<pin>

진단 순서 (겹):

golden 이미지 핀 stale — GitOps DEFAULT_SNAPSHOT이 재발행 안 된 옛 daytona-workspace-gha-<N>을 가리킴 (런타임 롤은 daytona-workspace 이미지를 재빌드하지 않음). 수정: runtime-release build_daytona_workspace=true + 핀 bump.
Argo daytona app OutOfSync — 핀이 origin/main엔 있으나 라이브 미적용. kubectl get applications -n argocd daytona.
runner DinD ghcr PULL 크레덴셜 — 위 docker pull이 unauthorized. runner 도커 데몬에 ghcr pull cred 필요(secret ghcr-pull). PUSH와 별개.
Daytona 스냅샷 오브젝트 stale — Daytona는 raw 태그가 아니라 이름있는 스냅샷 오브젝트(DEFAULT_SNAPSHOT_NAME, 예: fractalops-agent-runtime-k3s)로 샌드박스를 만듭니다. GET {DAYTONA_API_BASE}/api/snapshots(Bearer FRACTALOPS_DAYTONA_API_TOKEN, header X-Daytona-Organization-ID)로 조회 — 그 오브젝트가 state=inactive + 옛 이미지에 핀되어 있을 수 있음. write:snapshots로 DELETE + {name, imageName:<현재 gha>} POST하면 재활성.
daytona-api PUSH 패스워드 비어있음 — TRANSIENT_REGISTRY_PASSWORD / INTERNAL_REGISTRY_PASSWORD가 ${#VAR}=0. 차트 value:"" + extraEnv secretKeyRef 중복이 Argo SSA로 collapse.
PUSH 토큰 스코프 — env가 채워져 있어도(len 40) 토큰이 read-only면 DockerRegistryService ... Unauthorized. write:packages+delete:packages PAT 필요. kubectl logs -n daytona deploy/daytona-api | grep DockerRegistryService로 확인 — 토큰이 있는데 401이면 스코프 문제.

zero-output first-tool 블록 진단 (2026-06 관찰)

증상: 에이전트가 agent_first_tool_required로 blocked, 안정적 claude_session_id(세션 resume / #1528은 동작), 그런데 reports는 {"status":"blocked"}이고 observability-feed에 이벤트 0 / runtime_logs 0 — Claude 세션은 resume되지만 아무것도 emit하지 않음.

어디서 도는가: 에이전트 Claude 세션은 Daytona 샌드박스(execution_plane: daytona-sandbox)에서 실행됩니다. API/worker 로그가 아니라 Daytona 워크스페이스 로그가 provider 출력의 진실입니다.

실행-평면 health를 먼저 확인하세요 (보통 control plane이 아니라 이게 원인입니다). Headlamp에서 다음 순서로 봅니다.

kube-system: Headlamp 자체가 Healthy인지 확인.
fractalops: fractalops-api, fractalops-worker Deployment의 ready/available/replicas 확인.
Nodes: control/worker node Ready, pressure condition 확인.
longhorn-system: Longhorn volume CRD의 faulted/unknown volume 확인.
daytona: API/runner/core pods Running 확인.

자동화가 필요하면 Headlamp/OpenBao observer token을 읽어 typed Kubernetes client로 named probe를 만들고, Studio read-model에 결과를 저장하세요. 새 kubectl 문자열 파서는 만들지 않습니다.

Longhorn 스토리지: 관찰된 런에서 11개 faulted/unknown 볼륨(detached/orphaned)이 있었습니다. degraded 스토리지 평면은 새 Daytona 워크스페이스가 건강한 PVC를 붙이는 것을 막아 → claude가 못 돌고 → 이벤트 0.
스킬 풀은 런치를 멈출 수 없습니다(5초 타임아웃으로 bound).

2026-06 당시 원인이 아닌 것으로 검증된 것: ContextForge 1.0.3 gateway의 당시 auth 경로와 세션 resume(#1528). 해당 사건의 zero-output 블록은 실행-평면(Daytona/스토리지) 문제였습니다. 이 기록은 현재 auth 설계 근거가 아닙니다. 현재 agent 경로에 admin token을 복구하지 마세요.