---
title: Test a change
canonical: "https://agentbarn.dev/guides/develop/testing"
pubDate: "2026-08-29T00:00:00.000Z"
updatedDate: "2026-09-14T04:31:13.000Z"
author: Agent Barn
description: "Select API, UI, runtime, and Email Worker coverage while distinguishing required checks from automatic CI path selection."
tags: [Develop and extend, How-to, All contributors, runtime workflow selection, messaging hooks, testing, CI, Email Worker, coverage]
categories: [Guides, Develop and extend]
---

Test the behavior your change owns at the lowest layer that can prove its contract reliably. Add broader coverage when the contract crosses HTTP, PostgreSQL, authentication, browser behavior, runtime images, Kubernetes, or deployment configuration.

**Outcome**

By the end of this guide, you will be able to:

-   Map a change to the appropriate test layers
-   Write repository-consistent API and Playwright tests
-   Run focused checks
-   Widen verification according to risk
-   Report failures precisely

## Overview

The principle

### Test the contract, not the implementation detail

A useful test proves externally meaningful behavior:

```
Given a meaningful starting state
When the system performs an operation
Then the observable contract holds
```

Avoid tests that merely mirror the implementation’s internal control flow. They tend to pass when the same mistake exists in both the code and the test.

## Plan the test surface

Plan verification before or alongside implementation.

1.  **Describe the changed contract** Write one sentence describing what becomes true. If the sentence only describes a private method call, move one level outward until it describes behavior that matters.
    
2.  **Identify the contract owner** Determine which layer owns the behavior, from a parser or builder through to a Helm chart or monitoring rule.
    
3.  **Identify important failure modes** Consider invalid input, authentication and permission gaps, cross-Organization identifiers, conflicts, partial persistence, stale cache, and version drift.
    
4.  **Select the smallest reliable test** Use a unit test for focused logic, an integration test when the contract depends on HTTP composition or PostgreSQL, and Playwright when a browser-owned workflow is at risk.
    
5.  **Add a regression test** For a defect, write a test that demonstrates the original failure before the fix lands.
    
6.  **Plan the final verification set** Focused tests provide fast feedback. Required area checks prove the change still composes with the rest of the system. Record the commands you intend to run before handoff.
    

### Change-impact worksheet

Work through each column before writing the first test.

| Changed behavior | Contract owner | Failure risk | Test layer |
| --- | --- | --- | --- |
| One sentence describing what becomes true | parser, builder, service, repository, HTTP route, PostgreSQL constraint, migration, and so on | invalid input, missing authentication, insufficient permission, cross-Organization identifiers, missing Agent Access, not-found resources, and so on | The lowest layer that reliably proves the contract |

Example contract sentences:

```
A member with Viewer access can read an Agent but cannot update it.

Publishing a Skill draft creates a new immutable version and removes the draft.

A Platform Plugin rejects an unmentioned shared-channel message before creating a durable Delivery.

The shared Communications runtime adapter claims a Connection-scoped Delivery and submits one idempotent reply.

Runtime telemetry posts Tool Call state matching the current Ingest models.
```

The full contract-owner list runs: parser, builder, service, repository, HTTP route, PostgreSQL constraint, migration, UI query or mutation, browser interaction, Platform Plugin or shared Communications runtime adapter, runtime image, Kubernetes client, Helm chart, monitoring rule.

The failure modes worth considering include invalid input, missing authentication, insufficient permission, cross-Organization identifiers, missing Agent Access, not-found resources, lifecycle conflicts, uniqueness conflicts, partial persistence, external dependency failures, stale cache, duplicate handler execution, browser navigation and retry, runtime-version drift, deployment rendering errors.

Use an integration test when the contract depends on FastAPI routing, dependency injection, authentication, authorization, tenant scoping, SQLModel or SQLAlchemy queries, PostgreSQL constraints, transactions, Alembic migrations, serialized request or response contracts.

### Add a regression test

For a defect, write a test that demonstrates the original failure. A strong regression test:

-   Fails before the fix
-   Passes after the fix
-   Uses the smallest reliable surface
-   States the broken invariant in its name
-   Does not depend on unrelated implementation details

**A passing test written after the fix is not automatically a regression test**

Confirm that the test would fail against the defective behavior. This matters most when the test uses mocks, because an incorrect mock can encode the fixed behavior even while production code remains broken.

## Choose a test layer

Agent Barn uses several verification layers.

### Static checks

Lint, formatting, and type checking.

### Focused unit tests

Payload validation, parsers, builders, formatters, policy decisions, response classification, retry calculations, event-handler behavior, and runtime plugin logic.

### FastAPI and PostgreSQL integration tests

Status and response shape, user and Organization resolution, permission enforcement, repository visibility, constraints, transactions, migrations, outbox persistence, and DI wiring.

### Playwright browser tests

Navigation, forms, loading and empty states, permission-sensitive controls, confirmations, visible cache invalidation, error and retry, Organization switching, and streaming UI.

### Pinned runtime-image contracts

Proof that a runtime still exposes the hook shape and session behavior a plugin expects.

### Kubernetes and deployment checks

Real Kubernetes client behavior, chart rendering, alert rules, and dashboard queries.

```
Static checks
    │
    ▼
Focused unit tests
    │
    ▼
FastAPI + PostgreSQL integration tests
    │
    ▼
Playwright browser tests
    │
    ▼
Pinned runtime-image contracts
    │
    ▼
Kubernetes and deployment checks
```

The layers run from static checks, through focused unit tests, FastAPI and PostgreSQL integration tests, and Playwright browser tests, to pinned runtime-image contracts and Kubernetes and deployment checks.

Use the layers required by the changed contract, rather than running every environment-dependent suite indiscriminately.

### Change-to-coverage matrix

| Change | Minimum verification |
| --- | --- |
| Pure parser, formatter, validator, or builder | Focused unit test |
| Service business rule | Service and unit coverage, plus integration behavior |
| API route or authentication contract | API integration test |
| Repository query | PostgreSQL-backed test |
| Agent or subordinate-resource visibility | Agent RBAC and cross-Organization integration coverage |
| Database schema or constraint | Migration plus PostgreSQL integration coverage |
| Transactional workflow | Commit and rollback integration tests |
| Domain Event producer | Payload tests plus atomic outbox integration coverage |
| Infrastructure adapter | Focused unit test, and integration where protocol or wiring matters |
| UI Zod schema or query hook | Typecheck, lint, and focused browser coverage |
| User interaction or navigation | Playwright |
| Platform Plugin behavior | Plugin unit tests plus Communications gateway tests |
| Communication Connection CRUD or authorization | Postgres-backed API integration tests |
| Inbound events and Deliveries | Communications integration tests covering admission, idempotency, ordering, retries, and durable state |
| Shared Communications runtime adapter | Adapter unit tests plus Hermes and OpenClaw builder/start-script contract tests |
| Runtime telemetry | Tool Call and Tool Result payload tests against the real Ingest request models |
| Bundled Skill contribution | Bundle-structure, metadata, Template-reference, seeding, pinning, and runtime-mount tests |
| Kubernetes client behavior | The separate Kubernetes integration target |
| Monitoring alert or dashboard | Monitoring chart checks |
| Agent-facing documentation only | Link, path, and formatting validation. Application tests optional |

### What each layer is for

### Unit tests

-   Payload validation
-   Parsers
-   Builders
-   Formatters
-   Policy decisions with controlled collaborators
-   Infrastructure-client response classification
-   Retry calculations
-   Event-handler behavior
-   Platform Plugin and shared Communications runtime adapter logic

A unit test should make the relevant input, result, and edge case obvious.

### Integration tests

-   API status and response shape
-   Current-user and Organization resolution
-   Permission enforcement
-   Repository visibility
-   PostgreSQL constraints
-   Transactions
-   Migration results
-   Outbox persistence
-   Request composition
-   Dependency-injection wiring

### Browser tests

-   Page navigation
-   Form submission
-   Loading and empty states
-   Permission-sensitive controls
-   Confirmation dialogs
-   Cache invalidation visible to the user
-   Error and retry states
-   Organization switching
-   Platform View separation
-   URL-backed filters
-   Streaming UI behavior

**Do not substitute SQLite for a PostgreSQL contract**

Do not replace PostgreSQL with SQLite for a contract that depends on PostgreSQL enums, JSONB, constraints, locking, migration ordering, or SQL behavior.

**A mocked browser test is not a backend proof**

A mocked browser test proves the web app’s behavior against the mocked wire contract. It does not independently prove that FastAPI implements that contract.

## Write API tests

API tests use pytest and live under:

```
api/tests/
├── unit/
├── integration/
├── core/
├── steps/
├── helpers/
├── mocks/
└── fixtures/
```

### Focused unit test

Keep focused test setup close to the contract, using the repository’s Given/When/Then helpers and PyHamcrest assertions:

```
from hamcrest import assert_that, equal_to

from api.tests.core.givenpy import given, then, when

def test_normalizes_provider_identifier():
    with given():
        raw_value = "  GitHub  "

        with when("the provider identifier is normalized"):
            result = normalize_provider(raw_value)

        with then("the canonical lower-case identifier is returned"):
            assert_that(result, equal_to("github"))
```

Use parameterization when several inputs express the same behavior:

```
@pytest.mark.parametrize(
    ("raw_value", "expected"),
    [
        ("GitHub", "github"),
        (" github ", "github"),
        ("GITHUB", "github"),
    ],
)
def test_normalizes_provider_identifier(raw_value, expected):
    assert_that(normalize_provider(raw_value), equal_to("github"))
```

Split separate behaviors into separate tests, rather than accumulating unrelated assertion clusters. Each test should prove one focused behavior where practical.

### API integration test

Integration tests use the real FastAPI application, the real route and dependencies, a migrated PostgreSQL Testcontainer, reusable test setup steps, additive Injector overrides, `given`, `when`, and `then`, and PyHamcrest assertions.

```
def test_member_cannot_update_an_inaccessible_agent():
    with given(
        [
            prepare_injector(
                modules=[
                    MockK8sModule(),
                    MockLiteLLMModule(),
                ]
            ),
            prepare_api_server(),
            create_test_client(),
            database_repo_is_ready(),
            database_is_clean(),
            there_is_a_user(),
            there_is_an_access_token_for_user(),
            there_is_an_agent(),
        ]
    ) as context:
        with when("the member updates an inaccessible Agent"):
            response = context.client.patch(
                context.agent_url,
                headers={
                    "Authorization": f"Bearer {context.access_token}",
                },
                json={"name": "Changed"},
            )

        with then("the Agent is concealed"):
            assert_that(
                response.status_code,
                equal_to(status.HTTP_404_NOT_FOUND),
            )
```

Use the actual neighboring helpers and fixtures for the domain. The example illustrates ownership rather than defining a new universal fixture contract.

### Reusable setup

Put repeated setup under the existing support structure. A test step can:

-   Create a User
-   Create an Organization Membership
-   Issue an access token
-   Create an Agent
-   Seed a Template or Skill
-   Clean the database
-   Install an Injector override
-   Manage temporary environment values
-   Expose IDs and response data through the context

Avoid copying large object graphs into every test.

### Additive Injector overrides

```
prepare_injector(
    modules=[
        MockK8sModule(),
        MockLiteLLMModule(),
    ]
)
```

The helper composes the default application modules first and then adds test modules. This preserves real application wiring while replacing only the intended infrastructure seam.

**Warning**

Do not construct a simplified test application that omits authentication, providers, or router registration when those are part of the contract.

### Test the HTTP contract

For an API behavior change, cover the applicable cases:

| Case | Status |
| --- | --- |
| Success response and body | `200` |
| Create | `201` |
| Delete or bodyless action | `204` |
| Invalid body or parameters | `422` |
| Business precondition failure | `400` |
| Missing authentication | `401` |
| Visible but disallowed operation | `403` |
| Missing or tenant-hidden resource | `404` |
| Lifecycle or uniqueness conflict | `409` |

Also classify internal or external failure where relevant. Assert response fields that describe the behavior; do not snapshot a large response merely because it is easy.

## Test authentication and tenancy

Authentication and authorization are part of the behavior, not optional negative cases.

| Case | Expected result |
| --- | --- |
| No bearer token | 401 |
| Invalid or expired token | 401 |
| User lacks Organization Membership | Usually 403 at Organization resolution |
| Organization management action without permission | 403 |
| Resource belongs to another Organization | 404 for tenant-concealed resources |
| Agent is outside the caller’s visibility | 404 |
| Agent is visible but the caller lacks the action Permission | 403 |
| Platform route called by a non-admin | 403 |
| Platform Administrator calls an Organization route without Membership | Rejected like any other non-member |

### Agent and subordinate-resource visibility

For every new Agent or subordinate-resource endpoint, test the permission-backed Agent model, not only Organization isolation. Subordinate resources include Communication Connections, conversations, Tool Calls, activity, logs, costs, Secrets, Skills, Agent configuration.

A cross-Organization Agent resource normally returns `404`. A visible resource the caller lacks the action Permission for returns `403`. Platform Administrators do not receive implicit Organization membership.

### List isolation

Object-level tests are not enough for collection routes. Prove that:

-   Inaccessible records are absent
-   The total count excludes inaccessible records
-   Pagination occurs after visibility is applied
-   Search cannot reveal inaccessible records
-   Cross-Organization IDs in filters do not broaden access
-   General Agent Access is applied only when eligible
-   Explicit Agent Access grants and revocations take effect on the next request

**Warning**

A service test that filters results after a repository returns them does not prove safe pagination. Exercise the repository-backed HTTP behavior.

### Communication Connection isolation

-   Unauthenticated requests return `401`; inaccessible Agents and other Organizations are hidden with `404`; visible Agents without update permission return `403`.
-   List, count, search, and pagination exclude inaccessible Connections.
-   Multiple Connections for the same Platform are supported, and credentials remain isolated to their owning Connection.
-   Credential uniqueness and release behavior are covered, while stale Connection revisions return `409`.
-   Conversations and durable thread state are scoped by `connection_id`.
-   Webhook and driver authentication remain separate from ordinary Product API authentication.

UI permission controls are useful behavior coverage, but Postgres-backed API integration tests are the authorization and tenant-isolation proof.

### Mutation authority

Test that the backend resolves effective Permissions fresh for every mutation. Do not treat any of these as authorization proof:

-   A role sent by the browser
-   An allowedActions response cached by the UI
-   A client-computed boolean
-   A role name embedded in a request body
-   A stale query-cache entry

### Test the UI boundary too

When the web app changes, permitted users should see the appropriate control, disallowed users should see the documented hidden or disabled state, a backend `403` should render an access-denied result, and a `404` should not reveal another tenant’s resource.

The API integration test remains the security proof. The Playwright test proves the user experience.

## Test database changes

The API test session starts a PostgreSQL Testcontainer and upgrades a fresh database to Alembic head.

**Fresh-schema coverage is not backfill coverage**

Upgrading a fresh database proves that it can reach the current schema. It does not automatically prove that a data backfill correctly transforms every legacy state.

For a schema change:

1.  Update the database model
2.  Create an Alembic migration
3.  Review the generated operations
4.  Apply it locally
5.  Verify one Alembic head
6.  Test PostgreSQL constraints and behavior
7.  Add a focused migration test for non-trivial data transformation
8.  Exercise downgrade and re-upgrade when rollback is supported

Run:

```
make makemigrations
make migrate
make check-migrations
```

When appropriate:

```
make rollback
make migrate
```

### Migration coverage

Test:

-   New columns and defaults
-   Nullability
-   Foreign keys
-   Uniqueness
-   Check constraints
-   Enum values
-   Index-dependent query behavior where important
-   ON DELETE behavior
-   Existing-row backfills
-   Legacy malformed or edge-case data
-   Upgrade ordering
-   Downgrade behavior where supported

A data-migration unit test can load the migration module and exercise a pure transformation helper. The integration test must still prove the PostgreSQL result when database behavior is part of the contract.

### Test atomicity

For a multi-write workflow, include failure cases. Prove that:

-   Every required row commits on success
-   No partial business state remains when validation fails
-   No partial state remains when a later insert fails
-   Uniqueness races become the intended conflict
-   Business state and Domain Event rows commit or roll back together
-   External failure behavior matches the service contract

Do not use a sequence of successful repository assertions as proof that the writes share one transaction. Cause a later operation to fail, and inspect the resulting database state.

**Docker is required**

The API test configuration starts PostgreSQL through Testcontainers. Ensure Docker is running before invoking focused or complete API pytest commands.

## Test Communications and Runtime boundaries

Platform Plugins belong to the Communications service. Hermes and OpenClaw share a Communications runtime adapter; Runtime Ingest receives Tool Call telemetry only.

```
Runtime adapter
        │
        ▼
Tool Call or Tool Result payload
        │
        ▼
Real Ingest model validation
```

### Test Platform Plugins

Cover registry keys, ordering, duplicate detection, strict settings and credentials, safe fingerprints without raw secrets, external validation, normalization, admission policies and typed dispositions, outbound idempotency, optional capabilities, best-effort enrichment, and redacted operator-safe failures.

```
api/tests/unit/test_communications_plugins.py
api/tests/unit/test_communications_gateway.py
api/tests/unit/test_communications_processor.py
api/tests/unit/test_communications_supervisor.py
api/tests/unit/test_communications_operations.py
api/tests/unit/test_communication_error_details.py
api/tests/unit/test_communications_metrics.py
```

### Test the Communications pipeline

Prove that denied events create no Delivery; accepted inbound events are idempotent; enrichment failure preserves valid events; journal failure does not drop provider traffic; leases permit one active replica per Connection; supervisor failures are isolated; reconnect backoff is bounded; Conversation ordering is retained; retries do not duplicate replies; failures remain distinct; and diagnostics never expose credentials.

```
api/tests/integration/test_communication_connections.py
api/tests/integration/test_communication_deliveries.py
api/tests/integration/test_conversations.py
```

### Test the shared runtime adapter

Hermes and OpenClaw use `api/domains/agents/scripts/communications-runtime-adapter.py`. Cover bounded idle backoff that resets after work, both builder mounts and start scripts, required environment, Delivery claim, reply and completion on the same Delivery path, Connection/location/thread-derived session identity, Delivery ID idempotency, required session headers, and explicit successful or failed completion.

```
api/tests/unit/test_communications_runtime_adapter.py
api/tests/unit/test_hermes_builders.py
api/tests/unit/test_openclaw_builders.py
api/tests/integration/test_agents.py
```

**Use a pinned-image test only when needed**

A pinned Runtime-image test is needed when behavior depends on the actual Runtime API or packaged image, not for ordinary Platform Plugin logic.

### Test Ingest telemetry

`IngestBatchRequest` contains only `tool_calls` and `tool_results`. Conversation messages are owned by Communications; Ingest receives Tool Call telemetry only.

```
payload = flush_and_capture(telemetry_adapter)

batch = IngestBatchRequest.model_validate(payload)

assert batch.tool_calls or batch.tool_results
```

### Test bundled Skills

Test exactly one root `SKILL.md`, an isolated `aai-<integration>` directory, root-relative paths, non-empty files, provider and credential-free metadata, deterministic sorted manifests, collision detection that permits matching relative paths in separate roots, Template references to isolated paths, missing-only bootstrap seeding, exact Skill Version pins, and Runtime mount roots.

```
api/tests/unit/test_aai_cli_skills.py
api/tests/unit/test_template_skill_paths.py
api/tests/integration/test_skills.py
api/tests/integration/test_templates.py
api/tests/integration/test_agents.py
```

## Write Playwright tests

UI behavior tests use Playwright, with this ownership split:

```
ui/tests/e2e/
  Behavior descriptions and assertions

ui/tests/pages/
  Selectors and user interactions

ui/tests/pages/data-support/
  Reusable API interception

ui/tests/fixtures/
  Static response data
```

### Spec

A spec describes the user-visible behavior, and holds the assertions:

```
test("publishes a Skill draft", async ({ page }) => {
  const dataSupport = new DataSupport(page);
  const skillPage = new SkillDetailPage(page);

  await dataSupport.skills.interceptSkillDetail();
  await dataSupport.skills.interceptSkillDraft();
  await dataSupport.skills.interceptPublishDraft();

  await skillPage.goto();
  await skillPage.publishDraft();

  await expect(
    page.getByText("Skill published"),
  ).toBeVisible();
});
```

### Page object

A page object owns selectors and actions:

```
export class SkillDetailPage {
  constructor(private page: Page) {}

  async goto() {
    await this.page.goto("/dashboard/example/settings/skills/example");
  }

  async publishDraft() {
    await this.publishButton().click();
    await this.confirmPublishButton().click();
  }

  publishButton() {
    return this.page.getByRole("button", {
      name: "Publish",
    });
  }

  confirmPublishButton() {
    return this.page.getByRole("button", {
      name: "Publish draft",
    });
  }
}
```

Do not put `expect(...)` assertions in the page object.

### Data support

A data-support helper owns network interception:

```
await page.route(
  "**/api/v1/organizations/*/skills/*/draft/publish",
  async (route) => {
    if (route.request().method() !== "POST") {
      await route.fallback();
      return;
    }

    await route.fulfill({
      status: 201,
      contentType: "application/json",
      body: JSON.stringify(publishedSkillFixture),
    });
  },
);
```

Match method and path precisely. A broad pattern such as `**/skills*` can accidentally intercept collection, detail, version, draft, and mutation requests with the same response.

### Wire-format mocks

Mock responses represent backend wire format:

```
{
  "skill_id": "00000000-0000-0000-0000-000000000000",
  "created_at": "2026-08-29T10:00:00Z"
}
```

The shared API response interceptor converts them to the camelCase fields validated by the frontend Zod schema. When inspecting request bodies at the network boundary, expect the API client to have converted frontend camelCase to backend snake\_case.

### Selector priority

1.  Accessible role and name
2.  Label or stable visible text
3.  An existing test ID, when semantic selection is insufficient

```
page.getByRole("button", { name: "Publish draft" });
page.getByLabel("Skill name");
page.getByText("No Skills yet");
```

Avoid selectors tied to Tailwind classes, DOM nesting, Generated IDs, Icon-only structure, Positional nth() calls where semantic identity exists. Reliable selectors also encourage accessible application UI.

### Run Playwright interactively

From `ui/`:

```
pnpm test:watch
pnpm test:ui
pnpm test:debug
```

Playwright runs locally on port `3003` with a separate `.next-e2e` directory. It does not reuse a manually running Next.js server. On failure, the configured test run retains a trace and captures a screenshot.

## Test Kubernetes and monitoring

Environment-dependent tests have separate targets.

### Kubernetes integration

The normal API suite excludes `api/tests/integration/test_kubernetes_client.py`. Run it explicitly:

```
make test-api-k8s
```

The test uses a real Kubernetes client and assumes the target namespace already exists. The default namespace is the deliberately retained stable identifier `agent-farm`. You can provide:

```
K8S_NAMESPACE
K8S_KUBECONFIG_PATH
```

The suite creates resources with a unique `test-run-id` label, and cleans up only resources carrying that label.

Coverage includes Deployments, Services, PVCs, Secrets, ConfigMaps, Idempotent create behavior, Safe deletion of missing resources. Run this target when changing Kubernetes client operations, Agent resource builders where real API behavior matters, Namespace handling, Create, update, and delete semantics, Cleanup behavior.

### Monitoring checks

Changes under `helm/monitoring/` use a dedicated check. Build the chart dependencies first:

```
helm dependency build helm/monitoring
```

Then run:

```
make check-monitoring
```

The monitoring check renders the helm chart, extracts prometheus alert rules, runs promtool rule checks, runs promtool alert unit tests, parse-checks dashboard panel queries.

Run it after changing alert rules, dashboard promql, monitoring chart templates, monitoring values, dashboard provisioning.

**Warning**

Do not broaden Kubernetes cleanup to an entire namespace. The suite deletes only the resources carrying its own run label.

## Run focused checks

Start with the closest test file or behavior.

### API selection

A focused unit test file:

```
cd api
uv run python -m pytest \
  tests/unit/test_domain_events.py \
  -v
```

A focused integration test file:

```
cd api
uv run python -m pytest \
  tests/integration/test_agents.py \
  -v
```

One test:

```
cd api
uv run python -m pytest \
  tests/integration/test_agents.py::test_create_agent \
  -v
```

Match a test name:

```
cd api
uv run python -m pytest \
  tests \
  -k "cross_org" \
  -v
```

### UI selection

A focused spec:

```
cd ui
pnpm exec playwright test \
  tests/e2e/settings-skills-panel.spec.ts
```

Match a test name:

```
cd ui
pnpm exec playwright test \
  tests/e2e/settings-skills-panel.spec.ts \
  --grep "publishes"
```

Debug one test:

```
cd ui
pnpm exec playwright test \
  tests/e2e/settings-skills-panel.spec.ts \
  --grep "publishes" \
  --debug
```

The root API `conftest.py` starts the PostgreSQL Testcontainer for API pytest runs, including focused selections beneath `api/tests/`.

After the focused test passes, run static checks for the touched area before widening to its complete suite.

## Run required verification

Choose verification according to the files and contracts changed.

| Touched area | Commands | Notes |
| --- | --- | --- |
| API change | `make check-api, make check-migrations, make test-api` | check-api runs Ruff lint, Ruff formatting, and Python type checking. test-api excludes the real Kubernetes integration file. |
| UI change | `make lint-ui, make check-ui, make test-ui` | ESLint, TypeScript type checking, and Playwright. |
| Next routing, route handlers, rewrites, or production build behavior | `pnpm build` | Conditional. Run from ui/ in addition to the UI commands. |
| An API contract consumed by the UI | `Both the API and UI command sets` | Update API DTOs, Zod schemas, mocks, hooks, and browser expectations together. |
| Kubernetes behavior | `make test-api-k8s` | Conditional. Requires a reachable cluster and the target namespace. |
| Monitoring | `helm dependency build helm/monitoring, make check-monitoring` | Conditional. Run after changing alert rules, dashboards, chart templates, or values. |
| Runtime image or telemetry plugin | `The relevant pinned base-image contract check` | Conditional. A fake cannot prove the runtime still exposes the expected hook shape. |
| Documentation only | `Link, path, and formatting validation` | Application test suites are optional unless the documentation accompanies changed behavior. |

### API change

```
make check-api
make check-migrations
make test-api
```

### UI change

```
make lint-ui
make check-ui
make test-ui
```

If the change touches Next.js routing, route handlers, rewrites, server/client boundaries, or production build behavior, also run:

```
cd ui
pnpm build
```

### An API contract consumed by the UI

Run both sets, and update API DTOs, Zod schemas, mocks, hooks, and browser expectations together:

```
make check-api
make check-migrations
make test-api
make lint-ui
make check-ui
make test-ui
```

### Kubernetes and monitoring

Run the Kubernetes target in an environment with a reachable cluster and the target namespace:

```
make test-api-k8s
```

For monitoring:

```
helm dependency build helm/monitoring
make check-monitoring
```

### Coverage investigation

```
make coverage
```

Use coverage to find unexercised behavior, not as a substitute for reviewing whether the important contract is tested. The current coverage target runs the API test tree; ensure the environment can satisfy any included environment-dependent tests before treating it as a routine focused command.

### Documentation-only change

Validate:

-   Internal links
-   Route paths
-   Code-block formatting
-   Headings and table-of-contents anchors
-   Terminology
-   Commands against the current repository

Application test suites are optional for a documentation-only change, unless the documentation accompanies changed behavior.

## Interpret failures

1.  **Reproduce** Re-run the smallest command that reproduces the failure, and read the first relevant failure rather than only the final summary.
    
2.  **Isolate** Determine which contract failed, and which layer owns it.
    
3.  **Determine introduced versus pre-existing** Check whether the failure is caused by the change, or reproduces without it.
    
4.  **Fix or report** Fix the implementation or the test at the owning layer, or report an unrelated failure precisely.
    
5.  **Rerun** Re-run the focused test, then the required area checks.
    

### Implementation failure

The test correctly identifies broken product behavior. Fix the implementation and keep the test.

### Test failure

The product contract is correct, but the test has stale setup, an incorrect mock, or the wrong expectation. Update the test without weakening the behavior it should prove.

### Contract changed deliberately

Update authoritative documentation, API DTOs, the migration, service and repository behavior, UI Zod schemas, fixtures, mocks, and Playwright expectations together.

### Flaky failure

A flaky test is still a failure. Prefer waiting for an observable condition over increasing sleeps or global timeouts.

### Flaky failures

A flaky test is still a failure. Investigate:

-   Shared mutable state
-   Broad network interception
-   Missing cleanup
-   Unordered results
-   Time-based assertions
-   Race conditions
-   Stale query caches
-   Fixed sleeps
-   Dependency startup
-   Development-only compilation delay
-   Tests that rely on execution order

CI retries can reduce noise, but they do not make a nondeterministic contract reliable.

### Pre-existing or unrelated failure

**Do not reshape unrelated code to make the suite green**

Report the exact command, the failing test or check, a concise failure message, evidence that it is unrelated or pre-existing, and which requested verification remains incomplete.

A report should include exact command, failing test or check, concise failure message, evidence that it is unrelated or pre-existing, which requested verification remains incomplete. For example:

```
make test-ui did not complete.

Failure:
ui/tests/e2e/users-page.spec.ts
"loads the next page"

The failure reproduces without the changed feature and occurs in the existing
users-page interception setup. Focused tests for the changed Skill workflow
passed, but the full UI suite remains unverified.
```

Do not report “all tests pass” when only a focused subset ran.

## Understand CI coverage

CI selects workflows according to changed paths.

### Inbound Email Worker

Triggered by workers/email-inbound/ changes

-   Install pinned dependencies
-   Bundle staging and production Wrangler environments with dry-run deployment commands

Bundling does not prove routing, matching secrets, or end-to-end mail delivery is configured.

### API workflow

Triggered by API or Makefile changes

-   API lint and formatting checks
-   Python type checking
-   Alembic head check
-   API tests
-   A separate Kubernetes integration job with k3d
-   API image build

The CI Kubernetes job creates the agent-farm namespace before running the real client tests.

### UI workflow

Triggered by UI or Makefile changes

-   ESLint
-   TypeScript type checking
-   Playwright
-   UI image build

CI Playwright serves a production Next.js build rather than compiling routes on first access.

### Runtime workflows

Triggered by Runtime-specific path matchers; see Runtime workflow selection below

-   Image build
-   Smoke or runtime-contract checks

A plugin-only change can therefore require both the API suite and a runtime image build.

### Monitoring workflow

Triggered by Monitoring chart, its workflow, or Makefile changes

-   Helm dependency setup
-   Monitoring rule tests
-   Dashboard-query validation

CI is a backstop, not a replacement for focused local verification.

**CI is a backstop**

Run the smallest complete local verification set before relying on CI for environment-specific breadth.

## Completion checklist

Before handing off a change, confirm:

-   The changed behavior has a focused test
-   A regression fix has a test that fails against the original defect
-   Success and important failures are covered
-   Authentication and permission behavior are covered
-   Cross-Organization access is covered where applicable
-   Agent visibility and action Permissions are covered where applicable
-   List counts and pagination preserve visibility
-   Database changes include a reviewed migration
-   Multi-write workflows include rollback coverage
-   External dependencies use controlled test seams
-   Runtime plugins are checked at the posted-payload boundary
-   UI mocks match backend wire format
-   Playwright assertions remain in specs
-   Kubernetes tests use isolated labeled resources
-   Documentation reflects changed contracts
-   The required lint, type, migration, and test commands passed
-   Any command not run is identified with a reason
-   Any unrelated failure is reported precisely
-   No unrelated code was changed to satisfy verification

Report verification with exact commands:

```
Passed:
- make check-api
- make check-migrations
- make test-api
- make lint-ui
- make check-ui
- make test-ui

Not run:
- make test-api-k8s: the change does not affect Kubernetes behavior.
```

## Troubleshooting

| Symptom | Likely cause | What to check |
| --- | --- | --- |
| API tests cannot start | Docker or Testcontainers is unavailable | Start Docker and verify that the PostgreSQL image can be pulled |
| A focused unit test still starts PostgreSQL | The root API test fixture is session-wide | This is expected for tests under the current API test tree |
| Tests use the wrong database schema | Alembic heads diverged, or the migration is incomplete | Run make check-migrations and inspect migration order |
| A test passes with SQLite but fails in the suite | The contract depends on PostgreSQL behavior | Use the repository’s PostgreSQL-backed test environment |
| Real email or provider calls are attempted | The intended infrastructure seam was not overridden | Use additive Injector modules, or the established provider mock |
| An Injector override has no effect | The test built dependencies outside the application Injector | Use prepare\_injector(modules=\[...\]) and injected application dependencies |
| An API test returns an unexpected 403 | The test user lacks Membership or the required Permission | Check Organization setup, Agent Access, and current-user context |
| An API test returns 404 for an existing Agent | Repository visibility intentionally conceals it | Check Organization scope and Agent access setup |
| Playwright receives a Zod validation error | The intercepted response does not match the backend wire format | Return the correct snake\_case fields and types |
| A Playwright request reaches the wrong mock | The route wildcard is too broad, or setup order overlaps | Match method and endpoint precisely, and centralize interception |
| A Playwright selector becomes ambiguous | The accessible name is shared by multiple controls | Scope by region, or use a more specific accessible name |
| Local Playwright reports a Next lock conflict | Another test process is using the test output directory | Stop the stale test process. Normal dev and .next-e2e should remain separate |
| Playwright passes locally but fails in the CI build | The behavior depends on development compilation, or a server/client mismatch | Run pnpm build and the focused spec against production behavior |
| The OpenClaw plugin test reports missing Node | Node is not installed, or not on PATH | Install the required Node runtime. Do not skip the contract |
| Kubernetes integration cannot connect | The kubeconfig or cluster is unavailable | Check K8S\_KUBECONFIG\_PATH, the current context, and cluster reachability |
| Kubernetes tests report namespace not found | The target namespace has not been created | Create or select the configured K8S\_NAMESPACE |
| Kubernetes cleanup leaves resources behind | Test resources lack the unique run label | Preserve the suite’s test-run-id labels and cleanup fixture |
| Monitoring checks cannot render | Helm dependencies are missing | Run helm dependency build helm/monitoring |
| A full suite fails outside the changed area | A pre-existing failure may be present | Reproduce it, report the exact command and failure, and avoid unrelated fixes |
| CI did not run an expected component | Its paths were not matched by change detection | Inspect .github/workflows/ci.yml and the relevant reusable workflow |

## Next steps

[Next **Add an integration** Apply this test strategy to an external provider.](/guides/develop/add-integration) [**Develop against the API** API layering, authentication, transactions, and migrations.](/guides/develop/api) [**Develop the web app** Zod contracts, query keys, cache isolation, and Playwright ownership.](/guides/develop/web-app) [**Work with domain events** Outbox atomicity, handler idempotency, and delivery testing.](/guides/develop/domain-events) [**Testing and verification** The compact repository-wide testing reference.](/guides/testing-and-verification) [**Backend code standards** The implementation rules every API test should enforce.](/guides/backend-code-standards)

## Runtime workflow selection

CI selects runtime image workflows using explicit path expressions. OpenClaw startup scripts are included: a change under `api/domains/agents/scripts/openclaw/`, including `init-openclaw.js`, selects the OpenClaw image workflow. Shared messaging scripts select both runtime workflows.

| Changed path | Image workflow selected |
| --- | --- |
| `hermes-base/` | Hermes |
| `api/domains/agents/builders/hermes.py` | Hermes |
| `api/domains/agents/scripts/hermes/` | Hermes |
| `api/runtime_tests/` | Hermes |
| Under `api/tests/fixtures/`: `hermes_pvc_permissions_driver.py`, `hermes_session_store_driver.py`, or `hermes_message_completion_driver.py` | Hermes |
| `openclaw-base/` | OpenClaw |
| `api/domains/agents/scripts/openclaw/` | OpenClaw |
| `api/tests/fixtures/openclaw_message_hooks_driver.mjs` | OpenClaw |
| `api/domains/agents/scripts/messaging/` | Both |
| `api/domains/agents/builders/openclaw.py` alone | Neither image workflow |
| `api/domains/agents/scripts/communications-runtime-adapter.py` alone | Neither image workflow |

The last two rows are remaining image-workflow selection gaps. API paths still select the normal API workflow; “neither” does not mean no CI runs. Automatic selection also does not determine which runtime contracts a change needs. Follow the change's actual boundary and the repository's verification guidance.

The image contracts include runtime-facing behavior beyond telemetry: Hermes scheduler/boot-related integration and messaging capture, and OpenClaw completion/tool hooks have dedicated source and fixtures. A successful generic API suite is not proof that the pinned runtime image accepts the generated configuration or invokes its hooks.

## Inbound Email Worker

Worker source lives under `workers/email-inbound/`. Worker-path changes select the reusable Worker workflow. Its check job installs pinned dependencies and bundles the staging and production Wrangler environments with dry-run deployment commands. This describes CI behavior; bundling is not proof that routing, matching secrets, or end-to-end mail delivery is configured.