Monitor the platform

GitHub Actions deployment

The workflow derives the environment label from the target branch. Use distinct Grafana passwords and hostnames; the current workflow shares the Slack webhook and OpenRouter key.

Manual deployment

ENVIRONMENT=production
NAMESPACE=agent-farm

GRAFANA_HOST=grafana.agentbarn.example.com
GRAFANA_ADMIN_PASSWORD=REPLACE_WITH_STRONG_PASSWORD
SLACK_ALERTS_WEBHOOK_URL=https://hooks.slack.com/services/REPLACE_WITH_WEBHOOK

STORAGE_CLASS=REPLACE_WITH_DURABLE_STORAGE_CLASS
OPENROUTER_API_KEY=sk-or-REPLACE_WITH_OPENROUTER_KEY

Do not commit a populated deployment environment file.

Configure DNS and TLS

Point GRAFANA_HOST at cluster ingress. Grafana uses Traefik, the grafana-tls Secret, and the fixed letsencrypt-http01 ClusterIssuer. INGRESS_CLUSTER_ISSUER does not change the Grafana issuer.

Configure OpenRouter credit monitoring

The main API polls OpenRouter GET /key with the inference key and records limit_remaining. An unlimited key reports positive infinity, so OpenRouterCreditsLow cannot fire until the key has a limit.

The monitoring release is part of the regular Helmfile deployment and depends on the Agent Barn API release.

ENV_FILE=.env.deploy bash deploy.sh

ENV_FILE=.env.deploy.staging bash deploy.sh

ENVIRONMENT=staging
NAMESPACE=agent-farm-staging
GRAFANA_HOST=grafana-staging.agentbarn.example.com

helm list --namespace agent-farm

kubectl get pods,services,persistentvolumeclaims,ingress \
  --namespace agent-farm

kubectl get certificate,challenge \
  --namespace agent-farm

kubectl get pods \
  --namespace agent-farm \
  --watch

Expected components include Prometheus, Alertmanager, Grafana, and kube-state-metrics. Confirm the Prometheus claim and Grafana certificate are ready.

https://GRAFANA_HOST

Username: admin
Password: the value of GRAFANA_ADMIN_PASSWORD

1. Confirm the provisioned Prometheus data source is healthy.
2. Open every built-in dashboard and confirm current environment data appears.
3. Restrict the Grafana hostname through organizational network or identity controls.

Dashboards are provisioned from the grafana-dashboards ConfigMap. Treat JSON under helm/monitoring/dashboards/ as the source of truth rather than UI-only edits.

Prometheus is cluster-internal. Open a temporary authorized port-forward:

kubectl port-forward \
  --namespace agent-farm \
  service/monitoring-prometheus-server \
  9090:80

Open http://localhost:9090/targets and verify:

up{job="agentbarn-api"}
agentbarn_database_up
up{job="litellm"}
up{job="agent"}
agentbarn_agents_in_error
agentbarn_openrouter_credits_scrape_ok

up == 1 means successful scraping. up == 0 means a discovered target cannot be scraped. An absent target indicates discovery, label, Service, or endpoint failure.

Understand the two API endpoints

The database, Agent ERROR, and OpenRouter gauges exist only on the main API process. Tool Call outcome metrics are emitted by ingest.

Agent Barn provisions four dashboards from a ConfigMap; no Grafana sidecar discovers them.

Agent series receive stable app, agent_name, org_id, and org_name labels from Service labels, preserving identity across pod replacements.

A high Tool Call error ratio does not necessarily mean the API is failing. Check the affected tool, Agent credentials, provider access, and recent configuration changes.

Metrics remain for 15 days. Use Agent Barn cost surfaces and provider billing records for durable cost investigation.

1. Confirm whether the alert is from production or staging.
2. Identify the affected service, Organization, or Agent.
3. Confirm the signal in Grafana or Prometheus.
4. Check the current Kubernetes workload state.
5. Review recent deployments, migrations, and configuration changes.
6. Inspect service or Agent logs.
7. Mitigate user impact.
8. Verify recovery from the original signal.
9. Record the incident and any missing monitoring.

kubectl get pods,deployments,statefulsets,jobs,cronjobs \
  --namespace agent-farm

kubectl get events \
  --namespace agent-farm \
  --sort-by=.lastTimestamp

kubectl describe pod POD_NAME \
  --namespace agent-farm

kubectl logs POD_NAME \
  --namespace agent-farm \
  --all-containers \
  --tail=200

Inspect Alertmanager

kubectl port-forward \
  --namespace agent-farm \
  service/monitoring-alertmanager \
  9093:9093

Open http://localhost:9093 to inspect grouping, delivery, and active silences. A silence suppresses notifications; it does not repair the failure.

Correlate product evidence

Platform Administrators can inspect Event Deliveries at /dashboard/platform/event-deliveries. The built-in rules do not alert on Redis, workers, reconciliation, or Event Delivery lifecycle state.

Prometheus discovers Agent metrics on port 8081 through Services with agentbarn.io/component=agent and an endpoint named healthz.

Agents created before monitoring

Existing Agents may lack the health script, endpoint, or labels. Stop and start each affected Agent once so the API rebuilds its Kubernetes resources.

Services with the old component label

When metrics already exist and only the old pre-rebrand label is missing, patch the Service without restarting:

kubectl label services \
  --namespace agent-farm \
  --selector agentfarm.io/component=agent \
  agentbarn.io/component=agent \
  --overwrite

kubectl get services \
  --namespace agent-farm \
  --selector agentbarn.io/component=agent \
  --show-labels

Prometheus derives app, agent-name, org-id, and org-name from the Service. Restart an Agent when its target lacks current identity labels.

helm dependency build helm/monitoring
make check-monitoring

make check-monitoring renders the chart, extracts alert rules, parses dashboard PromQL, runs promtool checks, and executes alert threshold and annotation unit tests. It needs Helm, Docker, and built chart dependencies.

Chart.lock pins dependencies; downloaded archives under helm/monitoring/charts/ are not committed. Monitoring CI runs for helm/monitoring/**, the workflow, and relevant Make target changes.

Verify alert delivery in staging

• Produce a safe, known staging alert and observe it pending, then firing.
• Confirm Alertmanager receives it and Slack shows the environment-tagged notification.
• Remove the condition and confirm both the rule and Slack notification resolve.
• Verify the response procedure identifies the correct first checks.

Operational checklist

• Initial setup

  • Production and staging have distinct Grafana hosts.
  • Grafana administrator passwords are stored securely.
  • Grafana DNS and TLS are ready.
  • The Slack webhook is configured and protected.
  • Prometheus uses an appropriate StorageClass.
  • OpenRouter key limits support useful credit alerting.
  • External monitoring covers cluster and monitoring-stack availability.

• After deployment

  • Prometheus, Grafana, Alertmanager, and kube-state-metrics are ready.
  • The Prometheus volume is bound.
  • Grafana ingress and certificate are ready.
  • Main API and ingest targets are up.
  • LiteLLM is up.
  • Every expected running Agent is discoverable.
  • All four dashboards display data.
  • Staging alert delivery and resolution were tested.
  • Operators know where to review Agent logs, Tool Calls, costs, and Event Deliveries.

• Ongoing operations

  • Alerts have named owners and response procedures.
  • Prometheus storage growth is reviewed.
  • Grafana access is periodically audited.
  • Missing or noisy alerts are corrected in source and tested.
  • External backup, certificate, capacity, and synthetic checks remain healthy.
  • Monitoring is reviewed after every significant platform or deployment change.