Self-hosting

On this page

Overview
Understand the upgrade boundaries
Choose an upgrade path
Before you begin
1. Record the current state
2. Review the release
3. Back up state
4. Test in staging
5. Preview the upgrade
6. Run the upgrade
7. Watch upgrade hooks
8. Verify the upgrade
9. Upgrade running Agents
Rollback and recovery
Troubleshooting
Next steps

Upgrade outcome

A staged release with a deliberate recovery path

Complete this guide to move the platform and selected Agents forward without losing track of data, keys, images, or partial deployment state.

Staging proves the complete release and every affected runtime/platform pair.
Backups, stable secrets, image references, and Alembic state are recorded.
Hooks and rollouts are watched before product and provider verification.
Agent runtime adoption proceeds through canaries and controlled batches.

Overview

An Agent Barn upgrade can change the API, UI, worker, database schema, supporting services, monitoring, Agent builders, and runtime images.

ReviewBack upStageVerifyApproveDeployVerifyRestart selected Agents

GitHub branch deployments and versioned release bundles use the same Helmfile release graph. Neither path automatically rebuilds existing Agent workloads, so platform deployment and runtime rollout remain separate operations.

Understand the upgrade boundaries

Upgrade area	What can change	Primary risk
API	Routes, services, workers, Agent builders, integrations	Contract, migration, or background-processing failure
UI	Pages, schemas, queries, onboarding flows	UI and API incompatibility
Application database	Alembic schema and data migrations	Irreversible data or compatibility change
Hermes runtime	Base image and runtime behavior	Existing Hermes Agents retain the old image until restarted
OpenClaw runtime	Base image and runtime behavior	Existing OpenClaw Agents retain the old image until restarted
LiteLLM	Proxy image, configuration, master key, virtual keys	Brief interruption or invalid Agent and API keys
Firecrawl	API, browser service, RabbitMQ, database integration	Integration or upstream-image incompatibility
PostgreSQL and Redis	Stateful service images and configuration	Data compatibility and availability
Helm charts	Deployments, Services, Secrets, Jobs, ingress, storage	Partial or incompatible Kubernetes rollout
Monitoring	Rules, dashboards, Grafana, Alertmanager	Lost visibility or alert delivery
Environment configuration	Credentials, URLs, models, storage, OAuth, email	Cross-environment or secret mismatch

Changes that are not routine upgrades

Changing PostgreSQL passwords on initialized volumes.
Rotating SECRET_SIGNING_KEY, AGENT_TOKEN_ENCRYPTION_KEY, or LITELLM_MASTER_KEY.
Changing a StatefulSet’s existing StorageClass.
Renaming agent-farm or agent-farm-staging.
Moving persistent data to another cluster or storage provider.

Choose an upgrade path

Source-operated environments

GitHub branch deployment

Use the agent-barn repository when environments deploy directly from source.

Branch	Environment	Namespace	API/UI tags	Runtime tags
`staging`	Staging	`agent-farm-staging`	`latest-staging`	`<version>-staging`
`main`	Production	`agent-farm`	`latest`	`<version>`

Pushes and manual dispatch run only from staging or main; any other branch is rejected. Per-branch concurrency serializes runs and does not cancel an in-progress deployment.

Change detection compares with the latest successful deployment on that branch. A failure does not advance the baseline. Manual dispatch, a missing baseline, or an unavailable or non-ancestor baseline rebuilds API, UI, Hermes, and OpenClaw.

Packaged self-hosting

Release-bundle deployment

A bundle supplies helm/, k8s/, helmfile.yaml.gotmpl, deploy.sh, and a generated .env.deploy.

Component	Version source
API	`Release tag`
UI	`Release tag`
Hermes base	`hermes-base/VERSION`
OpenClaw base	`openclaw-base/VERSION`

Transfer environment-specific values into the new bundle’s file. Do not replace it wholesale with an older copy because requirements can change between releases.

Before you begin

Assign a release owner, database and backup owner, verification owner, incident decision-maker, Agent restart owner, and communication owner.

Confirm staging, production monitoring, DNS, certificates, storage, registries, and providers are healthy.
Confirm there is no active incident or overlapping automated or manual deployment.
Verify operators can reach Kubernetes, Helm history, application logs, and backups.
Confirm every required image exists before approval.

Record the current state

Capture enough evidence to distinguish an upgrade regression from pre-existing state and to reproduce the current deployment.

helm list --namespace agent-farm

helm history agentbarn-api --namespace agent-farm
helm history agentbarn-ui --namespace agent-farm
helm history litellm --namespace agent-farm
helm history monitoring --namespace agent-farm

kubectl get deployments,statefulsets \
  --namespace agent-farm \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image'

kubectl get cronjobs \
  --namespace agent-farm \
  -o custom-columns='NAME:.metadata.name,IMAGES:.spec.jobTemplate.spec.template.spec.containers[*].image'

kubectl exec \
  --namespace agent-farm \
  deployment/agentbarn-api \
  -- sh -c 'cd /app/api && alembic current'

kubectl get pods,deployments,statefulsets,jobs,cronjobs,persistentvolumeclaims \
  --namespace agent-farm

Record the source commit or bundle version, image references, Helm revisions, Alembic revision, backup identifiers, Agent count by runtime and platform, restart candidates, and known warnings.

Review the release

Application contracts

API and UI schema changes
Authorization and tenancy
Alembic migrations
Workers and Event Deliveries
Agent builders, Templates, Skills, and credentials
Provider and monitoring changes

Deployment contracts

Charts, dependencies, images, and strategies
Resources, Services, ingress, and certificates
Secrets, ConfigMaps, Jobs, hooks, and CronJobs
PVC size and StorageClass
Helmfile dependencies and Kubernetes permissions

Understand version identifiers

Identifier	Meaning
Git commit or PR	Product release identifier for branch deployments
API/UI image tag	Explicit deployment input; moving tag in branch environments
Hermes/OpenClaw VERSION	Runtime image version
Helm chart version	Independent packaging version for that chart
GitHub release tag	Pinned API/UI version in a release bundle
Alembic revision	Application database schema state

API and UI images are explicit Helmfile inputs, not values derived from chart appVersion. Chart versions change with chart packaging and do not form a shared product version.

Back up state

Back up Agent Barn, LiteLLM, and Firecrawl PostgreSQL databases; Agent PVCs when workspace continuity matters; environment configuration; stable keys; database credentials; and external provider configuration needed for recovery.

SECRET_SIGNING_KEY
AGENT_TOKEN_ENCRYPTION_KEY
LITELLM_MASTER_KEY
POSTGRES_APP_PASSWORD
POSTGRES_LITELLM_PASSWORD
POSTGRES_FIRECRAWL_PASSWORD

A database backup without its matching encryption key can leave Agent credentials unreadable. A LiteLLM database backup without its master key can leave virtual keys unusable.

Confirm the backup is complete and encrypted.
Store it outside the affected namespace.
Record its identifier in the release record.
Verify restoration through a tested procedure.

Test in staging

Deploy the complete release to agent-farm-staging and wait for the staging workflow to finish.

Namespace:      agent-farm-staging
Environment:    staging
API/UI tag:     latest-staging
Runtime suffix: -staging

Platform

API, UI, login, and Organization isolation
Invitations, email, and Google OAuth
Database migration and restart recovery
Workers and Event Deliveries

Agents

Create, start, stop, and restart
Hermes on every used platform
OpenClaw on every used platform
Messages, Tool Calls, costs, and credentials

Providers

LiteLLM and OpenRouter requests
Firecrawl and Shared Credentials
Grafana and alert delivery
Canary for every affected runtime/platform pair

Account for shared providers

Namespace separation does not guarantee provider isolation. Registry, OpenRouter, Google OAuth, Cloudflare, and Slack alert delivery may be shared. Avoid tests that exhaust quotas, revoke shared credentials, or send unwanted production-facing messages.

Preview the upgrade

For a manual deployment, load the protected environment and inspect the Helmfile diff.

set -a
source .env.deploy
set +a

export POD_KUBECONFIG_B64="$(base64 "$KUBECONFIG" | tr -d '\n')"

helmfile -f helmfile.yaml.gotmpl diff --suppress-secrets

unset POD_KUBECONFIG_B64

Review StatefulSet replacement, PVC, StorageClass, database Secret, and stable-key changes.
Review images, resources, ingress, ServiceAccounts, hooks, deletions, and namespace scope.

The GitHub path proceeds directly to helmfile sync --wait. Pull-request review, source diff, and staging provide its preview boundary.

Run the upgrade

GitHub deployment

After approval, merge into main. The commit or PR is the product release identifier.

Namespace:   agent-farm
Environment: production
API/UI tag:  latest

Change detection
Parallel component tests and image publishing
Environment resolution
Helmfile sync after required builds succeed or skip correctly

Protect main with checks and review; the repository does not add a separate GitHub Environment approval gate.

Release-bundle deployment

Fill the new .env.deploy with existing environment values.
Preserve pinned image repositories and tags.
Add newly required variables.
Verify kubeconfig and NAMESPACE=agent-farm.

ENV_FILE=.env.deploy bash deploy.sh

SLACK_ALERTS_WEBHOOK_URL=
GRAFANA_ADMIN_PASSWORD=
GRAFANA_HOST=

Understand Helmfile ordering

Application, LiteLLM, and Firecrawl PostgreSQL
Redis
LiteLLM and Firecrawl
Agent Barn API and hooks
Agent Barn UI
Monitoring

helmfile -f helmfile.yaml.gotmpl sync --wait

Default timeout: 600 seconds

Expect brief service interruption

API, UI, worker, and LiteLLM are single-replica, non-surge rollouts in the constrained namespace. LiteLLM specifically uses:

maxSurge: 0
maxUnavailable: 1

Watch upgrade hooks

The API chart completes ordered pre-install and pre-upgrade work before its rollout.

1Secrets and hook resourcesweight −10 2LiteLLM virtual keyweight −5 3Alembic migrationweight −1 4API rolloutDeployment

Secret and configuration hooks

The API Secret, registry pull Secret, and key-generation script run at weight -10. Existing pods reread changed envFrom values only after replacement.

LiteLLM virtual-key hook

At weight -5, agentbarn-api-litellm-key waits for LiteLLM, deletes the agentbarn-api alias, generates a replacement key, and updates litellm-api-key. A later failure can leave old API and worker pods holding the invalid previous key.

kubectl get job agentbarn-api-litellm-key \
  --namespace agent-farm

kubectl logs job/agentbarn-api-litellm-key \
  --namespace agent-farm

Database migration hook

At weight -1, agentbarn-api-migrate runs the target API image and executes cd /app/api && alembic upgrade head.

kubectl get job agentbarn-api-migrate \
  --namespace agent-farm

kubectl describe job agentbarn-api-migrate \
  --namespace agent-farm

kubectl logs job/agentbarn-api-migrate \
  --namespace agent-farm

kubectl get pods --namespace agent-farm --watch

kubectl rollout status deployment/agentbarn-api \
  --namespace agent-farm --timeout=10m

kubectl rollout status deployment/agentbarn-api-worker \
  --namespace agent-farm --timeout=10m

kubectl rollout status deployment/agentbarn-ui \
  --namespace agent-farm --timeout=10m

helm list --namespace agent-farm
helm status agentbarn-api --namespace agent-farm
helm status agentbarn-ui --namespace agent-farm

When supplied, the deployment commit annotation rolls API, UI, and worker pods even though branch image tags are mutable.

Verify the upgrade

Do not declare success from a green workflow or Helm result alone.

kubectl get pods,deployments,statefulsets,jobs,cronjobs,persistentvolumeclaims \
  --namespace agent-farm

curl --fail https://api.agentbarn.example.com/api/v1/health

kubectl exec \
  --namespace agent-farm \
  deployment/agentbarn-api \
  -- sh -c 'cd /app/api && alembic current'

Infrastructure

Pods ready; no unexplained crashes
PVCs retained
Hooks completed
Ingress, certificates, and monitoring ready

Product and data

UI, authentication, and Platform access
Organizations and owned resources present
Alembic matches release head
Workers, Redis, and Event Deliveries healthy

Providers and signals

LiteLLM canary model request
OpenRouter, Firecrawl, email, and OAuth
Messaging platforms
Targets, dashboards, and Slack alerts

Record final source or tag, images, Helm revisions, Alembic revision, backup identifiers, results, Agent restart status, and follow-up work.

Upgrade running Agents

A platform upgrade changes the runtime references used for future builds; it does not replace existing Agent Deployments. Each running Agent retains its image, ConfigMap, Secret, ingest identity, and Deployment.

Stop and start an Agent through Agent Barn to rebuild its resources from the selected runtime and platform, pinned Template or Override, Skills, credentials, current builders and runtime image, and a fresh ingest identity.

Use a canary rollout

Select one non-critical Agent for an affected runtime and platform.
Record its health, image, and configuration.
Stop and start it.
Verify connection, messages, Tool Calls, Skills, integrations, logs, and monitoring.
Observe it, then continue in small batches.
Pause if error rates, spend, restarts, or provider failures increase.

Runtime	Platforms to verify
Hermes	Slack, Telegram, and Discord where used
OpenClaw	Slack, Teams, Telegram, and Discord where used

Rollback and recovery

Failure state	Preferred response
Build or test failed before deployment	Fix the release; production is unchanged
Helmfile failed before API hooks	Inspect every dependency release that may already have changed
LiteLLM key rotated but API rollout failed	Restart or redeploy a compatible API and worker with the new key
Migration failed and rolled back cleanly	Correct the migration and redeploy
Migration succeeded but API failed	Deploy an image compatible with the migrated schema
API/UI regression with compatible schema	Revert source or redeploy the previous pinned bundle
Runtime regression	Restore the previous runtime reference, then restart only affected Agents
Irreversible data change	Stop writes and restore the tested recovery set
Secret mismatch	Restore the exact stable value belonging to the current data
Storage migration failure	Follow the storage recovery plan; Helm rollback does not move data
Monitoring failure	Keep application recovery separate and restore external visibility

Branch-deployment rollback

Production API and UI use moving latest tags. A Helm rollback can restore an old manifest revision while still pulling the current image. Prefer a reviewed corrective or revert commit that rebuilds, republishes, rolls out with a new commit annotation, and passes verification.

Release-bundle rollback

Use the previous compatible bundle’s charts and pinned images, retain stable secrets unless restoring matching data and keys, confirm schema compatibility, review the diff, deploy, and verify every release and data path.

Database rollback

Helm rollback does not execute alembic downgrade. Use a tested, data-safe downgrade only when compatible with the restored application; otherwise use a forward fix or restore the database recovery set.

Agent runtime rollback

Restoring the old platform runtime reference does not change Agents already restarted. Identify affected Agents, verify one canary on the restored reference, then proceed in batches.

Inspect partial Helmfile state

helm list --namespace agent-farm

helm status RELEASE_NAME --namespace agent-farm
helm history RELEASE_NAME --namespace agent-farm

Releases have independent histories. Do not assume they share a Helm revision or upgrade status.

Troubleshooting

Symptom	Likely cause	Resolution
Deployment from a feature branch fails immediately	Only staging and main may deploy	Promote through a supported branch
An unchanged component was rebuilt	Manual dispatch or no valid successful baseline was available	Confirm the workflow warning and complete full verification
A changed component was not rebuilt	Change detection missed its source path	Stop deployment and review the detected-component outputs
Helmfile reports a required value missing	The release introduced a configuration requirement	Compare the new environment with the bundle and documentation
PostgreSQL authentication fails	A password changed while the initialized volume retained the old value	Restore the old value or run a coordinated credential migration
Existing credentials cannot decrypt	AGENT_TOKEN_ENCRYPTION_KEY changed	Restore the key matching the application database
Sessions fail after upgrade	SECRET_SIGNING_KEY changed	Restore the old key or complete a planned session migration
Agent model calls fail after an API hook failure	The LiteLLM key changed while old pods retain the previous value	Redeploy compatible API and worker pods so they load the Secret
Migration Job fails	The target revision cannot apply to the database	Inspect the retained Job, logs, current revision, and backup
Helm rollback does not restore behavior	Database state or moving image tags stayed on the new release	Deploy a compatible explicit image or follow the recovery plan
LiteLLM briefly becomes unavailable	Its non-surge single-replica rollout replaced the old pod first	Wait for readiness and schedule future work in a maintenance window
Upgrade times out after ten minutes	A hook or workload exceeded the 600-second timeout	Inspect Jobs, pods, events, and release status before retrying
Only some platform releases upgraded	Helmfile failed after earlier dependencies completed	Inventory every release and recover from actual state
Running Agents still use the old runtime	Platform deployment does not rebuild Agent workloads	Stop and start Agents through a canary and batch rollout
Restarted Agents behave differently	The new image or generated configuration was applied	Compare canary logs, settings, pinned versions, and integrations
Agent PVC data is missing	The upgrade recreated or changed storage	Stop further restarts and follow the tested PVC recovery plan
UI and API disagree on a contract	Incompatible images were deployed	Deploy the matching API/UI pair
Grafana loses visibility	Monitoring or target discovery failed	Use Kubernetes and external monitoring while restoring the release
New image cannot be pulled	Tag, credentials, repository, or pull Secret is wrong	Verify the pinned image and registry hook before retrying
Staging creates Agent resources in production	K8S_NAMESPACE or API kubeconfig is wrong	Stop staging and correct namespace-scoped configuration

Upgrade checklist

Next steps

Keep the troubleshooting guide available while upgrading the platform and Agent fleet.

Continue the self-hosting sequence Troubleshoot self-hosting → Diagnose deployment, database, runtime, provider, and monitoring failures.

A staged release with a deliberate recovery path

Overview

Understand the upgrade boundaries

Changes that are not routine upgrades

Choose an upgrade path

GitHub branch deployment

Release-bundle deployment

Before you begin

Application contracts

Deployment contracts

Understand version identifiers

Platform

Agents

Providers

Account for shared providers

GitHub deployment

Release-bundle deployment

Understand Helmfile ordering

Expect brief service interruption

Secret and configuration hooks

LiteLLM virtual-key hook

Database migration hook

Infrastructure

Product and data

Providers and signals

Use a canary rollout

Rollback and recovery

Branch-deployment rollback

Release-bundle rollback

Database rollback

Agent runtime rollback

Inspect partial Helmfile state

Troubleshooting

Upgrade checklist

Before staging

Before production

After production

Next steps