Sandbox on Kubernetes

The Kubernetes runtime runs every sandbox as its own pod in a namespace of your cluster, optionally inside a lightweight VM (Kata Containers, for example AKS pod sandboxing or OpenShift sandboxed containers). It needs no privileged Docker-in-Docker sidecar and no seccomp or SELinux exceptions: a sandbox pod is an ordinary restricted pod that the cluster isolates from the node.

This page covers what is specific to that runtime and to running many people's sandboxes on it: persistent workspaces, sleeping and waking, private sandboxes, limits, network blocking, scheduling priority and the Helm chart. For what a sandbox is and how to use one, start with Sandbox.

Enterprise module

The Sandbox module, and so everything on this page, is part of the enterprise edition. See Licensing.

Choose the runtime

Set SANDBOX_RUNTIME_MODE=kubernetes (k8s is accepted too) and name the namespace the pods are created in with SANDBOX_K8S_NAMESPACE, usually a namespace of their own. The default mode is docker; jail selects a separate sandbox-host (see Sandbox host token).

Variable	Default	Purpose
SANDBOX_K8S_NAMESPACE	Console's own namespace	Namespace of the sandbox pods. Inside the cluster it falls back to the namespace of Console's service account; outside the cluster it is required.
SANDBOX_K8S_RUNTIME_CLASS	none	runtimeClassName of the pods: kata-vm-isolation on AKS pod sandboxing, kata or kata-qemu on OpenShift and upstream Kata, empty on a cluster without Kata (the same code path, without the VM). Check with kubectl get runtimeclass.
SANDBOX_K8S_NODE_SELECTOR	none	key=value,key2=value2 node selector.
SANDBOX_K8S_TOLERATIONS	none	JSON list of tolerations, for a tainted Kata node pool.
SANDBOX_K8S_POD_ANNOTATIONS	none	JSON map of pod annotations, for example Kata hypervisor settings.
SANDBOX_K8S_IMAGE_PULL_SECRET	none	Name of a docker-registry secret in the sandbox namespace.
SANDBOX_K8S_IMAGE_PULL_POLICY	cluster default	Set IfNotPresent for local or air-gapped registries, where the cluster default for :latest images (Always) fails.
SANDBOX_K8S_SERVICE_ACCOUNT	none	Service account of the sandbox pods. They never mount a token.
SANDBOX_K8S_RUN_AS_NON_ROOT	off	true sets runAsNonRoot on the container; the template image must then run as a non-root user.
SANDBOX_K8S_CAPABILITIES_ADD	none	Comma-separated Linux capabilities added to the container. Leave it unset: any added capability takes the pod out of a restricted profile such as OpenShift's restricted-v2.
SANDBOX_K8S_DEFAULT_MEM_MB / SANDBOX_K8S_DEFAULT_CPU_CORES	1024 / 0.5	Size of a sandbox that sets none.
SANDBOX_K8S_READY_TIMEOUT_MS	120000	How long a pod may take to become ready; a Kata VM boots slower than a plain container.
SANDBOX_K8S_UNSCHEDULABLE_GRACE_MS	30000	How long a pod may stay Unschedulable (for example while the autoscaler adds a node) before the create fails.
SANDBOX_K8S_CONTEXT / SANDBOX_K8S_KUBECONFIG	in-cluster	Only when Console runs outside the cluster (local development). The context is required then: Console never falls back to your current kubectl context.

Console reaches the sandbox through the Kubernetes API (pods/exec), so commands, files, git and terminals need no network path into the pod.

What a pod looks like

• One pod per sandbox, named cognipeer-sbx-<id>, in the sandbox namespace.
• Labelled cognipeer.io/tenant, cognipeer.io/tenant-db, cognipeer.io/project and cognipeer.io/owner, so a NetworkPolicy, a ResourceQuota or a cost report can select pods by workspace, project or owner.
• No service account token, allowPrivilegeEscalation: false, all capabilities dropped, the RuntimeDefault seccomp profile. A sandbox has no access to the cluster API.
• The container filesystem is writable, so pip and npm work. With all capabilities dropped apt-get needs a user-writable cache: apt-get -o APT::Sandbox::User=root -o Dir::Cache::archives=/tmp/aptc -o Dir::State::lists=/tmp/aptl install -y <package>, after creating those folders. Packages whose install scripts change ownership of files still fail.
• Memory and CPU limits are always set (a template or sandbox size, else the defaults above), so a pod cannot reserve a Kata VM's default size. Disk becomes the ephemeral-storage limit.
• On this runtime snapshots and fork are not available, and no warm pool is kept. Port preview works for any port the process listens on.

Request floor

The scheduler places a pod by what it requests, not by what it may use. A sandbox's size is its limit; its request is a small fixed floor: at most 500m CPU and 1024 Mi memory, or the limit when that is smaller. A 4-core sandbox therefore does not need four free cores on one node to start, and it may burst up to its limit. SANDBOX_K8S_REQUEST_CPU_M and SANDBOX_K8S_REQUEST_MEM_MB change the floor.

Persistent workspace

A Kubernetes pod cannot be paused, only deleted. So a persistent sandbox gets a PersistentVolumeClaim named <pod name>-ws (ReadWriteOnce), mounted at /workspace:

• Stop or sleep: the pod is deleted, the claim stays. Start or wake: a new pod mounts the same claim. Delete: the pod and the claim are deleted. Attaching a volume recreates the pod but keeps the claim.
• Only /workspace is kept. Packages installed into the image's own folders (apt, a global pip) go with the pod; put what must last into the template.
• The claim is at least SANDBOX_K8S_PERSIST_MIN_SIZE_MB (default 5120), or the sandbox's disk setting when that is larger.

Variable	Default	Purpose
SANDBOX_K8S_PERSIST	on	off for a cluster without a usable StorageClass or without claim rights. Persistent sandboxes are then kept running only, and stopping one loses its files.
SANDBOX_K8S_PERSIST_STORAGE_CLASS	cluster default	StorageClass of the claims (AKS: managed-csi). It must work with the Kata runtime.
SANDBOX_K8S_PERSIST_MIN_SIZE_MB	5120	Smallest claim.
SANDBOX_K8S_PERSIST_MOUNT_PATH	/workspace	Mount path.

Console's role needs persistentvolumeclaims rights in the sandbox namespace for this; without them a create fails with a message that says to apply the chart's sandbox RBAC or set SANDBOX_K8S_PERSIST=off. With a volume class that binds on first use (Azure Disk), the claim binds when the pod is scheduled, and that wait is not reported as missing capacity.

Sleep and wake

Idle persistent sandboxes are stopped, not deleted, and start again when they are used. This applies on this runtime and on the Docker runtime, wherever a stop keeps the files.

• Window. A sandbox's own Stop after idle (minutes), else the workspace's Persistent sandbox sleep under Operate → Sandbox → Settings, else the platform default SANDBOX_PERSISTENT_IDLE_STOP_SECONDS (10 minutes). A window set in the dashboard or per sandbox is between 60 seconds and SANDBOX_IDLE_STOP_MAX_SECONDS (4 hours); a value outside that range is refused with 400. There is no "always on": only the platform's own SANDBOX_PERSISTENT_IDLE_STOP_SECONDS=0 disables sleeping.
• Wake. An exec, code run, file call, git call, terminal or preview request to a stopped persistent sandbox starts it first. If a limit would be exceeded, the start is refused and the reason is returned in the result.
• Holds. A command that may run longer than 120 seconds holds the sandbox for its timeout plus 30 seconds (at most SANDBOX_MAX_HOLD_SECONDS, 1 hour) and lets go when it finishes. An agent's turn holds its sandbox for AGENT_SANDBOX_TURN_HOLD_SECONDS (900 seconds). A sandbox that is still using CPU (SANDBOX_IDLE_BUSY_CPU_PERCENT, 10 %) is left running for up to SANDBOX_IDLE_BUSY_GRACE_SECONDS (4 hours) past its window.
• Ephemeral sandboxes are deleted at the workspace's Idle timeout, and at SANDBOX_MAX_DURATION_SECONDS of age when that is set; no hold postpones the age cap.
• Evictions. The reaper compares the sandbox list with the pods in the namespace each round. An evicted pod, or one whose node disappeared, is cleaned up: a persistent sandbox is marked stopped and starts again on next use, any other becomes failed with the reason (for example Evicted: The node was low on resource: memory).
• Billing. Each stop records only the run it ended, from when it started.

Private sandboxes

A sandbox created with Private ticked (visibility: "private") is for its owner only: commands, files, terminal and preview. Visibility can be changed later, by the owner only (PATCH /api/sandbox/instances/:id/options with visibility set to project or private).

• What others see. Everyone with Sandbox access still sees the sandbox in the list, so usage is visible, but as a locked summary: name, state, owner, resources, start and last activity. No environment names, image, ports or errors. CPU and memory figures stay visible. Opening its details, starting, stopping or deleting it, or reading its content answers 403 with code: "sandbox_private" (and canRequestAccess: false for a member who is not a sandbox admin).
• Sandbox admins of the project see it, and can stop and delete it. To open it they ask for access with a reason (8 to 500 characters): POST /api/sandbox/instances/:id/access with { "reason": "…" }. The grant lasts one hour (SANDBOX_ACCESS_GRANT_SECONDS) and is written to the audit log as sandbox.private.access_granted, with the reason, the owner and the expiry. On more than one Console replica set CACHE_PROVIDER=redis, because the grant is kept in the cache.
• Snapshots and forks carry the privacy of their source: a private sandbox's snapshot is listed only to its owner and to sandbox admins, restoring needs ownership or an access grant, and the result is private to whoever restored it.
• API tokens. On the client API only the owner's token reaches a private sandbox or its snapshots; there is no grant path.
• Agents. A sandbox an agent opens for a user's conversation is private to that user, and an agent cannot attach to another user's private sandbox.

Limits

Active sandboxes (running, starting or being created) are limited per workspace, per project and per user. A create, start or wake that would exceed a limit is refused with 429 and code: "sandbox_quota_reached", plus the scope (tenant, project or user) and a message that names the limit.

Limit	Set in	Default
Per workspace	SANDBOX_MAX_CONCURRENT (platform only)	no limit
Per project	Active sandboxes per project under Operate → Sandbox → Settings, else SANDBOX_MAX_ACTIVE_PER_PROJECT	no limit
Per user	Active sandboxes per user in the same place, else SANDBOX_MAX_ACTIVE_PER_USER	no limit

0 means no limit; blank in the dashboard uses the platform value.

Network blocking

A sandbox created with Block all network access (blockNetwork: true) has no network at all. On Kubernetes that is done with a NetworkPolicy:

• The pod carries the label cognipeer.io/network=blocked. On the first need Console creates one deny-all policy named cognipeer-sandbox-network-blocked (SANDBOX_K8S_BLOCK_NETWORK_POLICY renames it) in the sandbox namespace that selects those pods and allows no ingress and no egress, not even DNS. It restores the policy if someone changes it. Commands, files and terminals keep working, because they go through the API server.
• The pod is not created until the policy is verified. A request is also refused when another policy in the namespace selects the pod and allows traffic, since policies add up.
• It needs a CNI that enforces NetworkPolicy (on AKS Azure network policy, Calico or Cilium) and the networkpolicies rights in the sandbox namespace. On a cluster that does not enforce policy set SANDBOX_K8S_BLOCK_NETWORK=off: a request for a sandbox without network is then refused with an error instead of silently giving a pod with full egress.
• A blocked sandbox has no preview.

The chart's own namespace-wide sandbox policies leave pods labelled blocked out, so they cannot open traffic for them.

Scheduling priority

When the cluster is full, a sandbox a person is waiting on is placed before one an agent, an MCP server or a flow started. Create two PriorityClasses and name them:

Variable	Pods
SANDBOX_K8S_PRIORITY_CLASS_INTERACTIVE	every other sandbox, for example one created from the dashboard
SANDBOX_K8S_PRIORITY_CLASS_BACKGROUND	sandboxes started by agents, MCP servers, flows and the platform itself

Unset means no class. The chart can create both classes (sandbox.priorityClasses.create, below); they use preemptionPolicy: Never, so a queued sandbox goes first in line but nothing that is running is evicted for it.

Sandbox host token

This applies to the jail runtime, where Console calls a separate sandbox-host service (SANDBOX_HOST_URL).

Variable	Purpose
SANDBOX_HOST_TOKEN	Bearer token Console sends with every call; the value of the sandbox-host's own token setting.
SANDBOX_HOST_REQUIRE_TOKEN	true refuses calls when no token is set. Default false.
SANDBOX_HOST_ALLOW_NO_TOKEN	true for a local sandbox-host that was itself started without a token; silences the warning.

Without a token, calls go out unauthenticated and Console logs a warning once per process. Set the token, and set SANDBOX_HOST_REQUIRE_TOKEN=true in production.

Licence check of the client sandbox API

The client sandbox API (/api/client/v1/sandbox/*, called with an API token) checks that the token's workspace has an active Enterprise licence after the token is authenticated.

• By default the check warns first: an unlicensed workspace is still served, and a warning naming the workspace and route is logged at most once every 10 minutes. This lets you check a rollout before enforcing.
• With SANDBOX_API_LICENSE_ENFORCE=1 (exactly 1) an unlicensed workspace, or a failed licence lookup, gets 402 Payment Required with module: "sandbox" and requiresEnterprise: true.

The dashboard API (/api/sandbox/*) always needs the licence.

Helm chart

The Enterprise Kubernetes chart helm-chart-kata sets all of the above for you. It has no Docker-in-Docker sidecar. The Console service account needs automount: true there.

Value	Default	What it does
sandbox.namespace	cognipeer-sandbox	Namespace of the RBAC objects. Must equal env.SANDBOX_K8S_NAMESPACE.
sandbox.rbac.create	true	Creates the namespace-scoped Role and RoleBinding for the Console service account. It is a Role, never a ClusterRole.
sandbox.rbac.allowExec / allowLog / allowMetrics	true	pods/exec, pods/log and pods/status, and metrics.k8s.io (for CPU and memory figures; optional). Without exec a sandbox cannot run anything.
sandbox.rbac.allowNetworkPolicies	true	networkpolicies get, list, create and update, for network blocking. Set it to false with env.SANDBOX_K8S_BLOCK_NETWORK: "off" on a cluster that does not enforce policy.
sandbox.rbac.roleName / roleBindingName	<release>-sandbox-manager	Names, for a naming standard.
sandbox.networkPolicy.enabled	false	Namespace-wide policy for sandbox pods: no ingress, egress only to DNS plus what you list in sandbox.networkPolicy.egress. Turn it on and list your package registry or proxy if sandboxes should install packages. Pods labelled cognipeer.io/network=blocked are excluded.
sandbox.priorityClasses.create	false	Creates the two PriorityClasses (cognipeer-sandbox-interactive, value 1000; cognipeer-sandbox-background, value 100). The installer needs rights for scheduling.k8s.io/priorityclasses. Then set the two SANDBOX_K8S_PRIORITY_CLASS_* values under env.
sandbox.envDelivery	legacy	file hands launch values to sandboxes through an in-memory file; see How a sandbox receives its environment.

The Role always includes pods (create, get, list, watch, delete, deletecollection). It adds persistentvolumeclaims (create, get, list, delete) unless env.SANDBOX_K8S_PERSIST is off. Install the chart, and deploy the RBAC, before the Console version that uses persistent workspaces, or persistent sandbox creation fails with the message described above.

The other SANDBOX_* values of the chart sit under env (SANDBOX_K8S_RUNTIME_CLASS, SANDBOX_K8S_TOLERATIONS, SANDBOX_PERSISTENT_IDLE_STOP_SECONDS, SANDBOX_MAX_ACTIVE_PER_PROJECT, SANDBOX_MAX_ACTIVE_PER_USER, and the rest of the tables above). For the chart in your installation, or help choosing values, contact developer@cognipeer.com.

Docker-in-Docker installations

The Docker-in-Docker chart (helm-chart-dind) keeps the engine's data on emptyDir volumes of the Console pod. A persistent sandbox there survives sleeping, but not the pod being replaced (a rollout, a node drain): its container and files go with the pod. Use the Kubernetes runtime when persistent workspaces must outlive the Console pod.

Where to go next

• Sandbox — templates, preview, volumes, environment variables and the agent sandbox.
• Deployment — Kubernetes notes for the Console itself.
• Licensing — enabling enterprise modules.