Kubernetes sandbox configuration

The kubernetes managed sandbox provider starts a dedicated runner Pod for each managed session. Configure it under sandbox.kubernetes in the server configuration. The agent_sandbox provider (self-reclaiming agent-sandbox Sandbox resources) reads the same sandbox.kubernetes settings and adds an optional sandbox.agent_sandbox block for native warm pools.

See Shared Server for the deployment steps.

Pod-start wait budget

Before a managed session goes online, Omnigent waits for the runner Pod to schedule, pull its image, clone its repository (or several, in parallel, for a multi-repo session), and reach Running. Increase the wait budget when cold image pulls regularly take longer than the default:

sandbox:
  provider: kubernetes
  kubernetes:
    pod_ready_timeout_s: 300

The budget covers Pod discovery, replacement Pod discovery after an eviction or node drain, Kubernetes API read errors, and time spent Pending. When it expires, the launch error reports the configured deadline and last observed Pod state.

The value must be a positive YAML integer. Values such as 0, "90", and true fail server startup.

To shorten the clone step itself, set the provider-independent sandbox.git_clone block (shallow depth, single-branch, blobless filter). The Kubernetes provider honors it for every repository it clones, including warm-pool Pods.

Sandboxed container runtime (runtime_class)

Set runtime_class to schedule every runner Pod onto a sandboxed container runtime your cluster exposes through a RuntimeClass object — for example Kata Containers micro-VMs or gVisor. Omnigent writes the value verbatim to the Pod's spec.runtimeClassName:

sandbox:
  provider: kubernetes
  kubernetes:
    runtime_class: kata

Leaving runtime_class unset keeps the Pod on the cluster's default runtime — today's behaviour, byte-for-byte. The name must be a DNS-1123 subdomain and must match a RuntimeClass you pre-created in the cluster; a malformed value (for example Not_A_DNS_Name) fails server startup rather than the first launch. Omnigent only references the RuntimeClass — provisioning it and the underlying node runtime is an out-of-band operator step.

Scheduling onto a tainted NodePool (tolerations)

Set tolerations to let runner Pods schedule onto a tainted NodePool you've dedicated to sandboxes. Each entry is a Kubernetes Toleration, written verbatim to the Pod's spec.tolerations:

sandbox:
  provider: kubernetes
  kubernetes:
    tolerations:
      - key: sei.io/node-role
        operator: Equal
        value: omnigent-sandbox
        effect: NoSchedule
    node_selector:
      sei.io/node-role: omnigent-sandbox

A toleration only permits the Pod to schedule onto matching tainted nodes — it does not attract it there. Pair tolerations with node_selector to also pin runner Pods onto that pool; without a selector the Pod may still land on any untainted node. Leaving tolerations unset adds no spec.tolerations key, keeping the manifest byte-for-byte compatible with prior deployments.

Each entry accepts these fields:

• key — the taint key to tolerate. Must be a valid Kubernetes label key. Omit it only with operator: Exists (Kubernetes' "tolerate everything" form).
• operator — Equal (default) or Exists. Exists matches any value, so it may not be combined with a value.
• value — the taint value, matched when operator: Equal. Must be a valid Kubernetes label value.
• effect — one of NoSchedule, PreferNoSchedule, or NoExecute. Omit to tolerate the key under every effect.
• tolerationSeconds — an integer, valid only with effect: NoExecute; how long the Pod stays bound to a node after a matching taint is added.

Entries are validated at server startup, so a malformed or self-contradictory toleration (an Exists operator with a value, an empty key with an operator other than Exists, or a tolerationSeconds without effect: NoExecute) fails fast rather than at the first managed launch.

Setting the budget from the environment

You can also set the wait budget with the OMNIGENT_K8S_POD_READY_TIMEOUT_S environment variable, which is handy when the same bundle runs across deployments with different image-pull times:

export OMNIGENT_K8S_POD_READY_TIMEOUT_S=300

Precedence: the explicit sandbox.kubernetes.pod_ready_timeout_s config key always wins when set; the env var is only consulted when that key is unset, and it overrides the built-in 90-second default. A non-numeric env value (for example not-a-number) fails fast with a clear error; a float-looking value such as 120.0 is accepted and truncated to a whole number of seconds.

Persistent storage for runner Pods

Runner Pods are ephemeral: the workspace lives on an emptyDir and is destroyed with the Pod. To expose durable data (datasets, model caches, shared output directories), mount pre-created PersistentVolumeClaims into every runner Pod with sandbox.kubernetes.pvc_mounts:

sandbox:
  provider: kubernetes
  kubernetes:
    pvc_mounts:
      - claim_name: omnigent-datasets
        mount_path: /mnt/datasets
        read_only: true       # default true; set false only for shared scratch

Each entry is a {claim_name, mount_path, read_only?} mapping. read_only defaults to true. Omnigent only references the claim — you pre-create the PV/PVC in the runner namespace (omnigent-sandboxes) out of band, so the server RBAC stays unchanged. Mounts land on the host container only; the init container that clones the repo never sees them.

Keep these caveats in mind:

• Mount paths are validated at server startup. A mount_path may not overlap /home/omnigent, Secret projections, or the image's OS directories (e.g. /home, /var, /run, /tmp, /etc, /opt); a malformed or reserved path fails startup rather than the first launch. Paths must be absolute and normalized, and may not duplicate or nest within each other.
• Writable claims are shared across concurrent runners. Use a ReadWriteMany-capable backend (NFS/SMB/CephFS) for anything writable, and prefer read_only: true everywhere else — a writable shared mount lets one session read and modify what another wrote, and its contents outlive the Pod.
• ReadWriteOnce claims pin all runners to one node — combine with node_selector deliberately, or later Pods sit Pending.
• A mount visible in the Pod is not automatically visible to a harness's own OS-level sandbox (Omnibox path grants are separate).

See deploy/kubernetes/README.md and deploy/kubernetes/overlays/sandbox-runners/README.md in the repo for the full guide.

Rotation-friendly credentials (secret_mounts)

Under sandbox.kubernetes, an optional secret_mounts list projects Kubernetes Secrets you pre-created in the runner namespace as read-only files on each runner Pod. Each entry is a { secret_name, mount_path } mapping:

sandbox:
  provider: kubernetes
  server_url: https://your-server.example.com
  kubernetes:
    secret_mounts:
      - secret_name: git-token          # a Secret in the runner namespace
        mount_path: /mnt/secrets/git     # absolute, non-reserved in-Pod path
      - secret_name: npm-token
        mount_path: /mnt/secrets/npm

Unlike credentials injected via envFrom (read once at container start), a Secret projected as a volume is refreshed in place by the kubelet, so a long-lived runner picks up a rotated credential without a restart. The refresh is eventually consistent (kubelet sync, up to ~1 min), so the in-sandbox consumer must re-read the file on each use — a value cached at start defeats the rotation.

Each volume mounts read-only on the host container only (never the clone-time init container), so a Secret volume has no read_only knob. Entries are validated at server startup: secret_name must be a DNS-1123 subdomain and mount_path an absolute, normalized path that isn't reserved and doesn't duplicate, nest with, or overlap another secret_mounts or pvc_mounts path — an operator typo fails startup, not the first launch. A missing Secret keeps the Pod from starting.

Native warm pools for agent_sandbox (warm_pool)

With the agent_sandbox provider, a warm pool prepares spare Pods before a session asks for one. Image pulls, scheduling, container startup, and initial PVC provisioning happen ahead of the request; Omnigent then claims a compatible Sandbox, delivers the managed-host identity to the waiting bootstrap, and runs the normal repository preparation and host startup. GitHub and Databricks credentials still come from the existing owner-bound broker and credential store, and the suspend/resume lifecycle with a durable HOME PVC is unchanged.

Opt in by naming the operator-owned pool:

sandbox:
  provider: agent_sandbox
  server_url: http://omnigent.omnigent.svc.cluster.local:8000
  agent_sandbox:
    warm_pool: omnigent-default-v1
  kubernetes:
    namespace: omnigent-sandboxes
    service_account: omnigent-runner
    image: your-registry/omnigent-host:your-version

Validation happens at server startup: warm_pool is the only key allowed under sandbox.agent_sandbox, the value must be a non-empty DNS-1123 subdomain (surrounding whitespace is trimmed), and setting sandbox.agent_sandbox with any other provider fails with requires provider: agent_sandbox. Omitting the block, or setting warm_pool: null, keeps direct Sandbox creation. The key also works inside a providers: list entry in a multi-provider deployment.

Prerequisites

• The upstream agent-sandbox v1.0.0 controller with extensions — the SandboxTemplate, SandboxWarmPool, and SandboxClaim CRDs (extensions.agents.x-k8s.io/v1beta1):

  kubectl --context YOUR_CONTEXT apply --server-side -f \
    https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v1.0.0/sandbox-with-extensions.yaml

• A host image built from the same Omnigent revision as the server. Warm Pods run the omnigent.host.warm_bootstrap module, which must be present in the image.

• The additive warm-pool RBAC overlay (see below).

Generate the template and pool

Omnigent reads the pool but never creates, resizes, or updates it — the SandboxTemplate and SandboxWarmPool are operator-owned. Render them from the exact server config you deploy, with the same workspace-volume environment settings the server uses, so the template's profile matches what the server expects:

export OMNIGENT_AGENT_SANDBOX_WORKSPACE_SIZE=5Gi
python -m omnigent.onboarding.sandboxes.agent_sandbox_warm_pool \
  --config server-config.yaml --name omnigent-default-v1 --replicas 2 --shared \
  > warm-pool.yaml
kubectl --context YOUR_CONTEXT apply -f warm-pool.yaml
kubectl --context YOUR_CONTEXT apply -k deploy/kubernetes/overlays/sandbox-runners/warm-pool

Generator options:

• --config PATH (required) — the server config; it must enable sandbox.provider: agent_sandbox.
• --name NAME (required) — used for both the SandboxTemplate and the SandboxWarmPool; set sandbox.agent_sandbox.warm_pool to the same value.
• --replicas N — spare Pods to keep ready (default 1, must be non-negative).
• --shared or --agent-name NAME — mutually exclusive; they choose which sessions the pool may serve (see the table below).

The output is a SandboxTemplate with networkPolicyManagement: Unmanaged and a SandboxWarmPool using the Recreate update strategy, which only replaces unused inventory. Use a new versioned pool name (e.g. …-v2) whenever you change the mode or any profile setting so old spare Pods never serve the previous profile.

--shared is recommended for deployments using the owner-bound GitHub and Databricks brokers, so built-in and uploaded agents share the same spare inventory. A shared template omits the omnigent.ai/agent label and carries the annotation omnigent.ai/warm-pool-shared: "true"; the marker is part of the profile fingerprint, so an existing pool that merely lacks the agent label is not shared. Per-agent admission-time credential injection does not apply to an unlabeled shared pool — use --agent-name if you rely on it. Shared mode changes pool compatibility only; credentials stay scoped to the host's owner.

RBAC overlay

deploy/kubernetes/overlays/sandbox-runners/warm-pool adds a namespaced Role/RoleBinding for the omnigent-server ServiceAccount in omnigent-sandboxes:

• sandboxclaims: create, get, patch, delete
• sandboxtemplates, sandboxwarmpools: get
• pods/exec: create, get — used to deliver the session identity to the bootstrap after allocation

Kubernetes RBAC cannot restrict exec to a particular command, so enabling warm mode is an explicit permission expansion for the server within the runner namespace. Runner ServiceAccounts receive none of these grants, and direct mode does not need the overlay.

Behaviour and caveats

• Allocated profiles stay fixed. An allocated warm Sandbox keeps its original image, mounts, resources, scheduling, security settings, and agent classifier. Changing those server settings can block a later wake of that session; the mismatch fails with a profile error and preserves the Sandbox and its PVC. Shared allocations tolerate agent or harness changes across suspension when the infrastructure profile still matches; dedicated and legacy unclassified allocations require an exact classifier match.
• Changing warm_pool affects new allocations only. Existing claims and workspaces are not migrated, and disabling pooling does not convert existing warm allocations into direct ones. Use direct provisioning for sessions that need infrastructure changes across wake.
• Capacity planning counts both containers. The bootstrap and host run as regular containers and each reserves the configured resources, so 250m CPU / 384Mi memory in the config reserves 500m / 768Mi per pooled Pod (before injected sidecars). Size nodes for spare and allocated Pods.
• Networking is yours. Generated templates set networkPolicyManagement: Unmanaged and use cluster DNS, matching the direct provider. Provide production NetworkPolicy objects yourself, including DNS and egress to the Omnigent callback URL and any private Git/Databricks endpoints.
• Do not customise individual claims. Adding environment variables or PVC overrides to a SandboxClaim makes upstream create a cold Sandbox instead. Session-specific data arrives after allocation; each template-created HOME PVC belongs to its Sandbox and survives suspension. Allocated Sandboxes are never returned to the spare pool.
• An empty pool still works. Upstream may create a cold member from the same template, so a successful allocation alone does not prove a warm hit — compare Pod UIDs recorded before the request when measuring latency.
• Static harness Secrets injected via envFrom are read when a spare Pod starts, so rotate them by replacing unused Pods before relying on the new value.

Verify a deployment

An opt-in live end-to-end test exercises allocation, host/runner registration, workspace persistence, pool replenishment, and suspend/wake against a real cluster. It needs a test server that accepts API requests without an authentication header (such as a single-user development server), a shared pool with at least one ready spare in a namespace isolated from other launchers, and a host image that supports the Claude SDK harness. Without OMNIGENT_WARM_POOL_E2E=1 the test skips; no model or connected-account credentials are required.

OMNIGENT_WARM_POOL_E2E=1 \
  OMNIGENT_WARM_POOL_SERVER_URL=http://127.0.0.1:8000 \
  OMNIGENT_WARM_POOL_KUBECONFIG=/absolute/path/to/test-kubeconfig \
  OMNIGENT_WARM_POOL_CONTEXT=YOUR_TEST_CONTEXT \
  OMNIGENT_WARM_POOL_NAMESPACE=omnigent-sandboxes \
  OMNIGENT_WARM_POOL_NAME=omnigent-default-v1 \
  .venv/bin/python -m pytest tests/e2e/test_agent_sandbox_warm_pool.py -v

Validate credential behaviour separately in an authenticated deployment: connect test GitHub and Databricks accounts, create a pooled session with a private repository, and check git fetch and gh api user --jq .login; confirm the workspace survives suspend and wake and that both services still authenticate afterwards. Keep static provider credential Secrets out of the pool during this check so they cannot mask a failed broker lookup.

See deploy/kubernetes/overlays/sandbox-runners/warm-pool/README.md in the repo for the full runbook.