Configure a production deployment

The standard layout deploys two namespace-isolated stacks into the same cluster.

NAMESPACE=agent-farm
ENVIRONMENT=production

NAMESPACE=agent-farm-staging
ENVIRONMENT=staging

Only the Kubernetes namespaces retain the agent-farm name after the Agent Barn rebrand, preserving existing workloads and PVC continuity.

Namespace isolation is not a separate cluster or provider security boundary. Use distinct application and LiteLLM database passwords, signing and encryption keys, bootstrap administrators, kubeconfigs, hosts, and sender subdomains.

The current workflow shares the registry, OpenRouter, Google OAuth, Cloudflare account and token, Slack alert webhook, Firecrawl password, and Firecrawl key unless operators change it. Shared inputs also share quota, revocation, and failure impact.

Verify that the staging API receives its release namespace as K8S_NAMESPACE; otherwise it can fall back to production when creating Agent workloads.

kubectl get deployment agentbarn-api \
  --namespace agent-farm-staging \
  -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="K8S_NAMESPACE")].value}{"\n"}'

Expected: agent-farm-staging.

The GitHub workflow runs on pushes to main and staging, and on manual dispatch. Manual dispatch is accepted only from those branches. Production deploys from main into agent-farm; staging deploys from staging into agent-farm-staging.

Understand change detection and concurrency

• Change detection compares with the latest successful deployment on the same branch.
• A failed deployment does not advance that baseline.
• Manual dispatch or an unavailable baseline builds all four Agent Barn images.
• Branch-group concurrency does not cancel an in-progress deployment when a newer run starts.

Understand image behavior

The workflow stamps the git SHA into pod templates, forcing API and UI rollouts even though their branch tag moves. Use explicit, mutually compatible tags for a manual release bundle.

Keep authoritative production values in a protected secret manager. Store signing and encryption keys, database passwords, provider tokens, registry credentials, API-facing Kubernetes access, Grafana credentials, and alert webhooks outside populated repository files.

Registry, OpenRouter, Cloudflare, Google OAuth, Slack webhook, and Grafana credentials are usually independently rotatable. Test emergency rotation without changing the stable recovery set.

Bootstrap administrator

PLATFORM_ADMIN_CREDENTIALS creates an administrator only when none exists. Updating the Secret later does not reset an existing account password. Create at least two independently controlled, named Platform Administrators and avoid routine use of the bootstrap account.

Use different identities for deployment and API runtime orchestration.

The API identity needs namespaced access to create Agent Deployments, Services, Secrets, ConfigMaps, and PVCs, and to inspect Pods, health, logs, exec, and port-forward operations. It should not access staging, unrelated namespaces, cluster Secrets, or nodes.

Verify the LiteLLM bootstrap identity

The LiteLLM key bootstrap Job runs as LITELLM_KEY_SERVICE_ACCOUNT. Verify that the configured account exists.

kubectl get serviceaccount \
  --namespace agent-farm \
  agent-farm-user

Audit effective permissions

kubectl auth can-i create deployments --namespace agent-farm
kubectl auth can-i create services --namespace agent-farm
kubectl auth can-i create secrets --namespace agent-farm
kubectl auth can-i create persistentvolumeclaims --namespace agent-farm
kubectl auth can-i get pods --namespace agent-farm
kubectl auth can-i get pods/log --namespace agent-farm

Repeat the checks with impersonation or the intended kubeconfig, then confirm that the same identity cannot read another namespace.

The default deployment requests at least 19 GiB before any Agents: 5 GiB for application PostgreSQL, 2 GiB each for LiteLLM and Firecrawl PostgreSQL, and 10 GiB for Prometheus. Every Agent PVC requests another 1 GiB.

The default k3s local-path StorageClass is node-local. Choose storage that meets the production durability target. Changing STORAGE_CLASS on an existing PostgreSQL StatefulSet is not an in-place migration because its volume claim template is immutable.

Back up the complete recovery set

Agent Barn does not install a database or Agent PVC backup controller. Integrate an external backup system, define frequency, retention, off-cluster storage, encryption, access, RPO/RTO, restore order, verification, and failure ownership.

Redis has no persistent volume in the current chart. Durable Domain Event intent remains in PostgreSQL, and reconciliation can republish eligible Event Deliveries when Redis returns.

UI_HOST=agentbarn.example.com
API_HOST=api.agentbarn.example.com
GRAFANA_HOST=grafana.agentbarn.example.com
WEB_APP_URL=https://agentbarn.example.com

Confirm DNS resolution, ports 80 and 443, certificate issuance and renewal monitoring, API-to-Service routing, and restricted Grafana authentication.

Introduce network isolation in staging

The charts do not create NetworkPolicies. If the cluster enforces them, allow only required UI-to-API, service-to-database, worker-to-Redis, LiteLLM-to-provider, Firecrawl, Agent, and Prometheus flows. Test egress carefully: a bad policy can leave Agents running but unable to reach model or messaging providers.

OpenRouter and LiteLLM

Use a production-dedicated OpenRouter key when isolation is required. Assign its budget, credit limit, rate limits, billing alerts, owner, rotation path, and emergency replacement procedure. The OpenRouterCreditsLow alert is useful only when the provider key has a credit limit; an unlimited key reports infinite remaining credit.

Keep LITELLM_MASTER_KEY stable and separate from the OpenRouter provider key.

Transactional email

SENDER_EMAIL=[email protected]

Transactional email requires the Cloudflare account ID, API token, and sender address, plus a verified sending domain. Verify invitations, password recovery, lifecycle notifications, reputation, and quota. Use a different sender subdomain for staging.

Production and staging share the Cloudflare account, token, and account quota in the current workflow, so rotation and quota exhaustion can affect both.

Google Workspace

https://agentbarn.example.com/api/v1/integrations/google/callback

Register the exact production callback. If an OAuth client is shared with staging, register that callback too; separate clients provide stronger failure and consent-screen isolation.

Firecrawl

Treat the platform Firecrawl key as a production credential. Its Services remain internal, but Firecrawl makes outbound requests, so enforce appropriate egress, destination, abuse-prevention, and capacity policies.

API, UI, worker, LiteLLM, Redis, all PostgreSQL databases, each Firecrawl component, Prometheus, Grafana, and Alertmanager run as single replicas by default.

Each Agent adds a Deployment, pod, Service, 1 GiB PVC, CPU and memory demand, model usage, platform API traffic, and monitoring traffic. Agent resource profiles are not configurable through .env.deploy.

Load-test representative Hermes and OpenClaw workloads in staging, including concurrent Agents, peak tools, large model responses, browser or Firecrawl work, Template and Skill loading, and restart behavior.

If the availability target requires multiple replicas, replication, or failover, treat that as custom architecture and validation work—not an environment-variable toggle.

SLACK_ALERTS_WEBHOOK_URL=REPLACE_WITH_PRODUCTION_WEBHOOK
GRAFANA_ADMIN_PASSWORD=REPLACE_WITH_STRONG_PASSWORD
GRAFANA_HOST=grafana.agentbarn.example.com

For every alert, document severity, on-call owner, response target, diagnostic link, escalation, recovery action, and user-notification threshold. Trigger a controlled staging alert before go-live.

Understand monitoring scope

Prometheus requests a 10 GiB PVC and retains 15 days by default. The namespace-scoped configuration does not collect cluster-wide node, kubelet, or cAdvisor metrics.

Add cluster monitoring for nodes, filesystems, volumes, container CPU and memory, the control plane, ingress, and cert-manager. Grafana dashboards are operational views, not audit logs or business records.

Write and schedule a recovery exercise using this minimum sequence:

1. Provision an isolated recovery namespace or cluster.
2. Restore stable keys and the three PostgreSQL databases.
3. Restore required Agent PVCs and compatible deployment artifacts.
4. Deploy without changing the restored keys and verify the expected schema.
5. Verify authentication, LiteLLM virtual keys, and a model request.
6. Verify Shared Credentials and Agent credentials can decrypt.
7. Start a test Agent and verify messages, Tool Calls, and costs.
8. Verify Event Delivery reconciliation, monitoring, and alerts.

Preserve version compatibility

Record the git commit, API/UI and runtime image tags, chart versions, Alembic revision, PostgreSQL and LiteLLM versions, backup time, and secret-manager references with every recovery set. Do not restore into an arbitrary application version.

Plan migration rollback

Database migrations run as a Helm pre-install/pre-upgrade hook. Helm rollback changes manifests and images; it does not reverse Alembic migrations. Before schema changes, review compatibility, back up the application database, test upgrade and recovery in staging, and decide whether rollback uses an older compatible image or a database restore.

In staging, verify API and UI health, administrator login, Organization isolation, invitations and email, OAuth, Agent creation, Hermes and OpenClaw startup, messaging platforms, LiteLLM requests, Activity, Tool Calls, logs, costs, Shared Credentials, Event Delivery processing, dashboards, alerts, migrations, and restart recovery.

Use dedicated staging chat applications, credentials, and data. Do not test with production workspaces or customer conversations.

Immediately before production

kubectl config current-context
kubectl get pods --namespace agent-farm
kubectl get pvc --namespace agent-farm
kubectl get certificate --namespace agent-farm

Confirm the production backup completed, is readable, and is associated with the matching stable-key set.

Verify releases, workloads, storage, ingress, certificates, and the public API.

helm list --namespace agent-farm

kubectl get deployments,statefulsets,pods,jobs,cronjobs \
  --namespace agent-farm

kubectl get pvc,ingress,certificate \
  --namespace agent-farm

curl --fail https://api.agentbarn.example.com/api/v1/health

Then sign in with a named Platform Administrator, select the production Organization, invite a controlled test user, verify email enrollment, and start a non-sensitive test Agent. Send a message and verify the response, Activity, Tool Calls, cost attribution, Agent metrics, and alert state. Retire the test Agent according to policy and record the deployed commit and verification result.