Develop and extend

On this page

Overview
Plan the test surface
Choose a test layer
Write API tests
Test authentication and tenancy
Test database changes
Test runtime plugins
Write Playwright tests
Test Kubernetes and monitoring
Run focused checks
Run required verification
Interpret failures
Understand CI coverage
Completion checklist
Troubleshooting
Next steps

Test the behavior your change owns at the lowest layer that can prove its contract reliably. Add broader coverage when the contract crosses HTTP, PostgreSQL, authentication, browser behavior, runtime images, Kubernetes, or deployment configuration.

Overview

The principle

Test the contract, not the implementation detail

A useful test proves externally meaningful behavior:

Given a meaningful starting state
When the system performs an operation
Then the observable contract holds

Avoid tests that merely mirror the implementation’s internal control flow. They tend to pass when the same mistake exists in both the code and the test.

Plan the test surface

Plan verification before or alongside implementation.

Describe the changed contract Write one sentence describing what becomes true. If the sentence only describes a private method call, move one level outward until it describes behavior that matters.
Identify the contract owner Determine which layer owns the behavior — from a parser or builder through to a Helm chart or monitoring rule.
Identify important failure modes Consider invalid input, authentication and permission gaps, cross-Organization identifiers, conflicts, partial persistence, stale cache, and version drift.
Select the smallest reliable test Use a unit test for focused logic, an integration test when the contract depends on HTTP composition or PostgreSQL, and Playwright when a browser-owned workflow is at risk.
Add a regression test For a defect, write a test that demonstrates the original failure before the fix lands.
Plan the final verification set Focused tests provide fast feedback. Required area checks prove the change still composes with the rest of the system. Record the commands you intend to run before handoff.

Change-impact worksheet

Work through each column before writing the first test.

Changed behavior	Contract owner	Failure risk	Test layer
One sentence describing what becomes true	parser, builder, service, repository, HTTP route, PostgreSQL constraint, migration, and so on	invalid input, missing authentication, insufficient permission, cross-Organization identifiers, missing Agent Access, not-found resources, and so on	The lowest layer that reliably proves the contract

Example contract sentences:

A member with Viewer access can read an Agent but cannot update it.

Publishing a Skill draft creates a new immutable version and removes the draft.

An Agent telemetry plugin posts an outbound message with the correct conversation identity.

The full contract-owner list runs: parser, builder, service, repository, HTTP route, PostgreSQL constraint, migration, UI query or mutation, browser interaction, runtime plugin, runtime image, Kubernetes client, Helm chart, monitoring rule.

The failure modes worth considering include invalid input, missing authentication, insufficient permission, cross-Organization identifiers, missing Agent Access, not-found resources, lifecycle conflicts, uniqueness conflicts, partial persistence, external dependency failures, stale cache, duplicate handler execution, browser navigation and retry, runtime-version drift, deployment rendering errors.

Use an integration test when the contract depends on FastAPI routing, dependency injection, authentication, authorization, tenant scoping, SQLModel or SQLAlchemy queries, PostgreSQL constraints, transactions, Alembic migrations, serialized request or response contracts.

Add a regression test

For a defect, write a test that demonstrates the original failure. A strong regression test:

Fails before the fix
Passes after the fix
Uses the smallest reliable surface
States the broken invariant in its name
Does not depend on unrelated implementation details

Choose a test layer

Agent Barn uses several verification layers.

Static checks

Lint, formatting, and type checking.

Focused unit tests

Payload validation, parsers, builders, formatters, policy decisions, response classification, retry calculations, event-handler behavior, and runtime plugin logic.

FastAPI and PostgreSQL integration tests

Status and response shape, user and Organization resolution, permission enforcement, repository visibility, constraints, transactions, migrations, outbox persistence, and DI wiring.

Playwright browser tests

Navigation, forms, loading and empty states, permission-sensitive controls, confirmations, visible cache invalidation, error and retry, Organization switching, and streaming UI.

Pinned runtime-image contracts

Proof that a runtime still exposes the hook shape and session behavior a plugin expects.

Kubernetes and deployment checks

Real Kubernetes client behavior, chart rendering, alert rules, and dashboard queries.

Static checks
    │
    ▼
Focused unit tests
    │
    ▼
FastAPI + PostgreSQL integration tests
    │
    ▼
Playwright browser tests
    │
    ▼
Pinned runtime-image contracts
    │
    ▼
Kubernetes and deployment checks

The layers run from static checks, through focused unit tests, FastAPI and PostgreSQL integration tests, and Playwright browser tests, to pinned runtime-image contracts and Kubernetes and deployment checks.

Use the layers required by the changed contract, rather than running every environment-dependent suite indiscriminately.

Change-to-coverage matrix

Change	Minimum verification
Pure parser, formatter, validator, or builder	Focused unit test
Service business rule	Service and unit coverage, plus integration behavior
API route or authentication contract	API integration test
Repository query	PostgreSQL-backed test
Agent or subordinate-resource visibility	Agent RBAC and cross-Organization integration coverage
Database schema or constraint	Migration plus PostgreSQL integration coverage
Transactional workflow	Commit and rollback integration tests
Domain Event producer	Payload tests plus atomic outbox integration coverage
Infrastructure adapter	Focused unit test, and integration where protocol or wiring matters
UI Zod schema or query hook	Typecheck, lint, and focused browser coverage
User interaction or navigation	Playwright
Runtime telemetry plugin	Posted-payload unit contract, plus a pinned-image contract
Kubernetes client behavior	The separate Kubernetes integration target
Monitoring alert or dashboard	Monitoring chart checks
Agent-facing documentation only	Link, path, and formatting validation. Application tests optional

What each layer is for

Unit tests

Payload validation
Parsers
Builders
Formatters
Policy decisions with controlled collaborators
Infrastructure-client response classification
Retry calculations
Event-handler behavior
Runtime plugin logic

A unit test should make the relevant input, result, and edge case obvious.

Integration tests

API status and response shape
Current-user and Organization resolution
Permission enforcement
Repository visibility
PostgreSQL constraints
Transactions
Migration results
Outbox persistence
Request composition
Dependency-injection wiring

Browser tests

Page navigation
Form submission
Loading and empty states
Permission-sensitive controls
Confirmation dialogs
Cache invalidation visible to the user
Error and retry states
Organization switching
Platform View separation
URL-backed filters
Streaming UI behavior

Write API tests

API tests use pytest and live under:

api/tests/
├── unit/
├── integration/
├── core/
├── steps/
├── helpers/
├── mocks/
└── fixtures/

Focused unit test

Keep focused test setup close to the contract, using the repository’s Given/When/Then helpers and PyHamcrest assertions:

from hamcrest import assert_that, equal_to

from api.tests.core.givenpy import given, then, when


def test_normalizes_provider_identifier():
    with given():
        raw_value = "  GitHub  "

        with when("the provider identifier is normalized"):
            result = normalize_provider(raw_value)

        with then("the canonical lower-case identifier is returned"):
            assert_that(result, equal_to("github"))

Use parameterization when several inputs express the same behavior:

@pytest.mark.parametrize(
    ("raw_value", "expected"),
    [
        ("GitHub", "github"),
        (" github ", "github"),
        ("GITHUB", "github"),
    ],
)
def test_normalizes_provider_identifier(raw_value, expected):
    assert_that(normalize_provider(raw_value), equal_to("github"))

Split separate behaviors into separate tests, rather than accumulating unrelated assertion clusters. Each test should prove one focused behavior where practical.

API integration test

Integration tests use the real FastAPI application, the real route and dependencies, a migrated PostgreSQL Testcontainer, reusable test setup steps, additive Injector overrides, given, when, and then, and PyHamcrest assertions.

def test_member_cannot_update_an_inaccessible_agent():
    with given(
        [
            prepare_injector(
                modules=[
                    MockK8sModule(),
                    MockLiteLLMModule(),
                ]
            ),
            prepare_api_server(),
            create_test_client(),
            database_repo_is_ready(),
            database_is_clean(),
            there_is_a_user(),
            there_is_an_access_token_for_user(),
            there_is_an_agent(),
        ]
    ) as context:
        with when("the member updates an inaccessible Agent"):
            response = context.client.patch(
                context.agent_url,
                headers={
                    "Authorization": f"Bearer {context.access_token}",
                },
                json={"name": "Changed"},
            )

        with then("the Agent is concealed"):
            assert_that(
                response.status_code,
                equal_to(status.HTTP_404_NOT_FOUND),
            )

Use the actual neighboring helpers and fixtures for the domain. The example illustrates ownership rather than defining a new universal fixture contract.

Reusable setup

Put repeated setup under the existing support structure. A test step can:

Create a User
Create an Organization Membership
Issue an access token
Create an Agent
Seed a Template or Skill
Clean the database
Install an Injector override
Manage temporary environment values
Expose IDs and response data through the context

Avoid copying large object graphs into every test.

Additive Injector overrides

prepare_injector(
    modules=[
        MockK8sModule(),
        MockLiteLLMModule(),
    ]
)

The helper composes the default application modules first and then adds test modules. This preserves real application wiring while replacing only the intended infrastructure seam.

Test the HTTP contract

For an API behavior change, cover the applicable cases:

Case	Status
Success response and body	`200`
Create	`201`
Delete or bodyless action	`204`
Invalid body or parameters	`422`
Business precondition failure	`400`
Missing authentication	`401`
Visible but disallowed operation	`403`
Missing or tenant-hidden resource	`404`
Lifecycle or uniqueness conflict	`409`

Also classify internal or external failure where relevant. Assert response fields that describe the behavior — do not snapshot a large response merely because it is easy.

Test authentication and tenancy

Authentication and authorization are part of the behavior, not optional negative cases.

Case	Expected result
No bearer token	401
Invalid or expired token	401
User lacks Organization Membership	Usually 403 at Organization resolution
Organization management action without permission	403
Resource belongs to another Organization	404 for tenant-concealed resources
Agent is outside the caller’s visibility	404
Agent is visible but the caller lacks the action Permission	403
Platform route called by a non-admin	403
Platform Administrator calls an Organization route without Membership	Rejected like any other non-member

Agent and subordinate-resource visibility

For every new Agent or subordinate-resource endpoint, test the permission-backed Agent model — not only Organization isolation. Subordinate resources include conversations, Tool Calls, activity, logs, costs, Secrets, Skills, Agent configuration.

A cross-Organization Agent resource normally returns 404. A visible resource the caller lacks the action Permission for returns 403. Platform Administrators do not receive implicit Organization membership.

List isolation

Object-level tests are not enough for collection routes. Prove that:

Inaccessible records are absent
The total count excludes inaccessible records
Pagination occurs after visibility is applied
Search cannot reveal inaccessible records
Cross-Organization IDs in filters do not broaden access
General Agent Access is applied only when eligible
Explicit Agent Access grants and revocations take effect on the next request

Mutation authority

Test that the backend resolves effective Permissions fresh for every mutation. Do not treat any of these as authorization proof:

A role sent by the browser
An allowedActions response cached by the UI
A client-computed boolean
A role name embedded in a request body
A stale query-cache entry

Test the UI boundary too

When the web app changes, permitted users should see the appropriate control, disallowed users should see the documented hidden or disabled state, a backend 403 should render an access-denied result, and a 404 should not reveal another tenant’s resource.

The API integration test remains the security proof. The Playwright test proves the user experience.

Test database changes

The API test session starts a PostgreSQL Testcontainer and upgrades a fresh database to Alembic head.

For a schema change:

Update the database model
Create an Alembic migration
Review the generated operations
Apply it locally
Verify one Alembic head
Test PostgreSQL constraints and behavior
Add a focused migration test for non-trivial data transformation
Exercise downgrade and re-upgrade when rollback is supported

Run:

make makemigrations
make migrate
make check-migrations

When appropriate:

make rollback
make migrate

Migration coverage

Test:

New columns and defaults
Nullability
Foreign keys
Uniqueness
Check constraints
Enum values
Index-dependent query behavior where important
ON DELETE behavior
Existing-row backfills
Legacy malformed or edge-case data
Upgrade ordering
Downgrade behavior where supported

A data-migration unit test can load the migration module and exercise a pure transformation helper. The integration test must still prove the PostgreSQL result when database behavior is part of the contract.

Test atomicity

For a multi-write workflow, include failure cases. Prove that:

Every required row commits on success
No partial business state remains when validation fails
No partial state remains when a later insert fails
Uniqueness races become the intended conflict
Business state and Domain Event rows commit or roll back together
External failure behavior matches the service contract

Do not use a sequence of successful repository assertions as proof that the writes share one transaction. Cause a later operation to fail, and inspect the resulting database state.

Test runtime plugins

Hermes and OpenClaw telemetry plugins ship inside Agent images. They are not ordinary modules installed into the API environment.

Source-path harness
        │
        ▼
Posted payload
        │
        ▼
Real Ingest model validation

Hermes

Hermes test support:

Loads the plugin from its source path
Registers hooks against a controlled context
Invokes runtime-like hook events
Flushes the plugin
Captures the posted JSON
Validates the result against the real Ingest models

Assert the payload the plugin sends, not only its internal buffer. A useful contract assertion is:

payload = flush_and_capture(plugin)

batch = IngestBatchRequest.model_validate(payload)

assert_that(
    batch.messages[0].direction,
    equal_to(MessageDirection.OUTBOUND),
)

OpenClaw

The OpenClaw plugin is JavaScript. Its tests:

Requires Node
Runs a Node subprocess
Starts a temporary real HTTP listener
Points the plugin at that listener
Drives runtime-like hook events
Collects every posted batch
Validates the resulting payload

A missing Node executable must fail. Do not skip the test, because the JavaScript contract was not exercised.

Write Playwright tests

UI behavior tests use Playwright, with this ownership split:

ui/tests/e2e/
  Behavior descriptions and assertions

ui/tests/pages/
  Selectors and user interactions

ui/tests/pages/data-support/
  Reusable API interception

ui/tests/fixtures/
  Static response data

Spec

A spec describes the user-visible behavior, and holds the assertions:

test("publishes a Skill draft", async ({ page }) => {
  const dataSupport = new DataSupport(page);
  const skillPage = new SkillDetailPage(page);

  await dataSupport.skills.interceptSkillDetail();
  await dataSupport.skills.interceptSkillDraft();
  await dataSupport.skills.interceptPublishDraft();

  await skillPage.goto();
  await skillPage.publishDraft();

  await expect(
    page.getByText("Skill published"),
  ).toBeVisible();
});

Page object

A page object owns selectors and actions:

export class SkillDetailPage {
  constructor(private page: Page) {}

  async goto() {
    await this.page.goto("/dashboard/example/settings/skills/example");
  }

  async publishDraft() {
    await this.publishButton().click();
    await this.confirmPublishButton().click();
  }

  publishButton() {
    return this.page.getByRole("button", {
      name: "Publish",
    });
  }

  confirmPublishButton() {
    return this.page.getByRole("button", {
      name: "Publish draft",
    });
  }
}

Do not put expect(...) assertions in the page object.

Data support

A data-support helper owns network interception:

await page.route(
  "**/api/v1/organizations/*/skills/*/draft/publish",
  async (route) => {
    if (route.request().method() !== "POST") {
      await route.fallback();
      return;
    }

    await route.fulfill({
      status: 201,
      contentType: "application/json",
      body: JSON.stringify(publishedSkillFixture),
    });
  },
);

Match method and path precisely. A broad pattern such as **/skills* can accidentally intercept collection, detail, version, draft, and mutation requests with the same response.

Wire-format mocks

Mock responses represent backend wire format:

{
  "skill_id": "00000000-0000-0000-0000-000000000000",
  "created_at": "2026-08-29T10:00:00Z"
}

The shared API response interceptor converts them to the camelCase fields validated by the frontend Zod schema. When inspecting request bodies at the network boundary, expect the API client to have converted frontend camelCase to backend snake_case.

Selector priority

Accessible role and name
Label or stable visible text
An existing test ID, when semantic selection is insufficient

page.getByRole("button", { name: "Publish draft" });
page.getByLabel("Skill name");
page.getByText("No Skills yet");

Avoid selectors tied to Tailwind classes, DOM nesting, Generated IDs, Icon-only structure, Positional nth() calls where semantic identity exists. Reliable selectors also encourage accessible application UI.

Run Playwright interactively

From ui/:

pnpm test:watch
pnpm test:ui
pnpm test:debug

Playwright runs locally on port 3003 with a separate .next-e2e directory. It does not reuse a manually running Next.js server. On failure, the configured test run retains a trace and captures a screenshot.

Test Kubernetes and monitoring

Environment-dependent tests have separate targets.

Kubernetes integration

The normal API suite excludes api/tests/integration/test_kubernetes_client.py. Run it explicitly:

make test-api-k8s

The test uses a real Kubernetes client and assumes the target namespace already exists. The default namespace is the deliberately retained stable identifier agent-farm. You can provide:

K8S_NAMESPACE
K8S_KUBECONFIG_PATH

The suite creates resources with a unique test-run-id label, and cleans up only resources carrying that label.

Coverage includes Deployments, Services, PVCs, Secrets, ConfigMaps, Idempotent create behavior, Safe deletion of missing resources. Run this target when changing Kubernetes client operations, Agent resource builders where real API behavior matters, Namespace handling, Create, update, and delete semantics, Cleanup behavior.

Monitoring checks

Changes under helm/monitoring/ use a dedicated check. Build the chart dependencies first:

helm dependency build helm/monitoring

Then run:

make check-monitoring

The monitoring check renders the helm chart, extracts prometheus alert rules, runs promtool rule checks, runs promtool alert unit tests, parse-checks dashboard panel queries.

Run it after changing alert rules, dashboard promql, monitoring chart templates, monitoring values, dashboard provisioning.

Run focused checks

Start with the closest test file or behavior.

API selection

A focused unit test file:

cd api
uv run python -m pytest \
  tests/unit/test_domain_events.py \
  -v

A focused integration test file:

cd api
uv run python -m pytest \
  tests/integration/test_agents.py \
  -v

One test:

cd api
uv run python -m pytest \
  tests/integration/test_agents.py::test_create_agent \
  -v

Match a test name:

cd api
uv run python -m pytest \
  tests \
  -k "cross_org" \
  -v

UI selection

A focused spec:

cd ui
pnpm exec playwright test \
  tests/e2e/settings-skills-panel.spec.ts

Match a test name:

cd ui
pnpm exec playwright test \
  tests/e2e/settings-skills-panel.spec.ts \
  --grep "publishes"

Debug one test:

cd ui
pnpm exec playwright test \
  tests/e2e/settings-skills-panel.spec.ts \
  --grep "publishes" \
  --debug

The root API conftest.py starts the PostgreSQL Testcontainer for API pytest runs, including focused selections beneath api/tests/.

After the focused test passes, run static checks for the touched area before widening to its complete suite.

Run required verification

Choose verification according to the files and contracts changed.

Touched area	Commands	Notes
API change	`make check-api, make check-migrations, make test-api`	check-api runs Ruff lint, Ruff formatting, and Python type checking. test-api excludes the real Kubernetes integration file.
UI change	`make lint-ui, make check-ui, make test-ui`	ESLint, TypeScript type checking, and Playwright.
Next routing, route handlers, rewrites, or production build behavior	`pnpm build`	Conditional. Run from ui/ in addition to the UI commands.
An API contract consumed by the UI	`Both the API and UI command sets`	Update API DTOs, Zod schemas, mocks, hooks, and browser expectations together.
Kubernetes behavior	`make test-api-k8s`	Conditional. Requires a reachable cluster and the target namespace.
Monitoring	`helm dependency build helm/monitoring, make check-monitoring`	Conditional. Run after changing alert rules, dashboards, chart templates, or values.
Runtime image or telemetry plugin	`The relevant pinned base-image contract check`	Conditional. A fake cannot prove the runtime still exposes the expected hook shape.
Documentation only	`Link, path, and formatting validation`	Application test suites are optional unless the documentation accompanies changed behavior.

API change

make check-api
make check-migrations
make test-api

UI change

make lint-ui
make check-ui
make test-ui

If the change touches Next.js routing, route handlers, rewrites, server/client boundaries, or production build behavior, also run:

cd ui
pnpm build

An API contract consumed by the UI

Run both sets, and update API DTOs, Zod schemas, mocks, hooks, and browser expectations together:

make check-api
make check-migrations
make test-api
make lint-ui
make check-ui
make test-ui

Kubernetes and monitoring

Run the Kubernetes target in an environment with a reachable cluster and the target namespace:

make test-api-k8s

For monitoring:

helm dependency build helm/monitoring
make check-monitoring

Coverage investigation

make coverage

Use coverage to find unexercised behavior, not as a substitute for reviewing whether the important contract is tested. The current coverage target runs the API test tree — ensure the environment can satisfy any included environment-dependent tests before treating it as a routine focused command.

Documentation-only change

Validate:

Internal links
Route paths
Code-block formatting
Headings and table-of-contents anchors
Terminology
Commands against the current repository

Application test suites are optional for a documentation-only change, unless the documentation accompanies changed behavior.

Interpret failures

Reproduce Re-run the smallest command that reproduces the failure, and read the first relevant failure rather than only the final summary.
Isolate Determine which contract failed, and which layer owns it.
Determine introduced versus pre-existing Check whether the failure is caused by the change, or reproduces without it.
Fix or report Fix the implementation or the test at the owning layer, or report an unrelated failure precisely.
Rerun Re-run the focused test, then the required area checks.

Implementation failure

The test correctly identifies broken product behavior. Fix the implementation and keep the test.

Test failure

The product contract is correct, but the test has stale setup, an incorrect mock, or the wrong expectation. Update the test without weakening the behavior it should prove.

Contract changed deliberately

Update authoritative documentation, API DTOs, the migration, service and repository behavior, UI Zod schemas, fixtures, mocks, and Playwright expectations together.

Flaky failure

A flaky test is still a failure. Prefer waiting for an observable condition over increasing sleeps or global timeouts.

Flaky failures

A flaky test is still a failure. Investigate:

Shared mutable state
Broad network interception
Missing cleanup
Unordered results
Time-based assertions
Race conditions
Stale query caches
Fixed sleeps
Dependency startup
Development-only compilation delay
Tests that rely on execution order

CI retries can reduce noise, but they do not make a nondeterministic contract reliable.

Pre-existing or unrelated failure

A report should include exact command, failing test or check, concise failure message, evidence that it is unrelated or pre-existing, which requested verification remains incomplete. For example:

make test-ui did not complete.

Failure:
ui/tests/e2e/users-page.spec.ts
"loads the next page"

The failure reproduces without the changed feature and occurs in the existing
users-page interception setup. Focused tests for the changed Skill workflow
passed, but the full UI suite remains unverified.

Do not report “all tests pass” when only a focused subset ran.

Understand CI coverage

CI selects workflows according to changed paths.

API workflow

Triggered by API or Makefile changes

API lint and formatting checks
Python type checking
Alembic head check
API tests
A separate Kubernetes integration job with k3d
API image build

The CI Kubernetes job creates the agent-farm namespace before running the real client tests.

UI workflow

Triggered by UI or Makefile changes

ESLint
TypeScript type checking
Playwright
UI image build

CI Playwright serves a production Next.js build rather than compiling routes on first access.

Runtime workflows

Triggered by Runtime image or telemetry plugin path changes

Image build
Smoke or runtime-contract checks

A plugin-only change can therefore require both the API suite and a runtime image build.

Monitoring workflow

Triggered by Monitoring chart, its workflow, or Makefile changes

Helm dependency setup
Monitoring rule tests
Dashboard-query validation

CI is a backstop, not a replacement for focused local verification.

Completion checklist

Before handing off a change, confirm:

The changed behavior has a focused test
A regression fix has a test that fails against the original defect
Success and important failures are covered
Authentication and permission behavior are covered
Cross-Organization access is covered where applicable
Agent visibility and action Permissions are covered where applicable
List counts and pagination preserve visibility
Database changes include a reviewed migration
Multi-write workflows include rollback coverage
External dependencies use controlled test seams
Runtime plugins are checked at the posted-payload boundary
UI mocks match backend wire format
Playwright assertions remain in specs
Kubernetes tests use isolated labeled resources
Documentation reflects changed contracts
The required lint, type, migration, and test commands passed
Any command not run is identified with a reason
Any unrelated failure is reported precisely
No unrelated code was changed to satisfy verification

Report verification with exact commands:

Passed:
- make check-api
- make check-migrations
- make test-api
- make lint-ui
- make check-ui
- make test-ui

Not run:
- make test-api-k8s — the change does not affect Kubernetes behavior.

Troubleshooting

Symptom	Likely cause	What to check
API tests cannot start	Docker or Testcontainers is unavailable	Start Docker and verify that the PostgreSQL image can be pulled
A focused unit test still starts PostgreSQL	The root API test fixture is session-wide	This is expected for tests under the current API test tree
Tests use the wrong database schema	Alembic heads diverged, or the migration is incomplete	Run make check-migrations and inspect migration order
A test passes with SQLite but fails in the suite	The contract depends on PostgreSQL behavior	Use the repository’s PostgreSQL-backed test environment
Real email or provider calls are attempted	The intended infrastructure seam was not overridden	Use additive Injector modules, or the established provider mock
An Injector override has no effect	The test built dependencies outside the application Injector	Use prepare_injector(modules=[...]) and injected application dependencies
An API test returns an unexpected 403	The test user lacks Membership or the required Permission	Check Organization setup, Agent Access, and current-user context
An API test returns 404 for an existing Agent	Repository visibility intentionally conceals it	Check Organization scope and Agent access setup
Playwright receives a Zod validation error	The intercepted response does not match the backend wire format	Return the correct snake_case fields and types
A Playwright request reaches the wrong mock	The route wildcard is too broad, or setup order overlaps	Match method and endpoint precisely, and centralize interception
A Playwright selector becomes ambiguous	The accessible name is shared by multiple controls	Scope by region, or use a more specific accessible name
Local Playwright reports a Next lock conflict	Another test process is using the test output directory	Stop the stale test process. Normal dev and .next-e2e should remain separate
Playwright passes locally but fails in the CI build	The behavior depends on development compilation, or a server/client mismatch	Run pnpm build and the focused spec against production behavior
The OpenClaw plugin test reports missing Node	Node is not installed, or not on PATH	Install the required Node runtime. Do not skip the contract
Kubernetes integration cannot connect	The kubeconfig or cluster is unavailable	Check K8S_KUBECONFIG_PATH, the current context, and cluster reachability
Kubernetes tests report namespace not found	The target namespace has not been created	Create or select the configured K8S_NAMESPACE
Kubernetes cleanup leaves resources behind	Test resources lack the unique run label	Preserve the suite’s test-run-id labels and cleanup fixture
Monitoring checks cannot render	Helm dependencies are missing	Run helm dependency build helm/monitoring
A full suite fails outside the changed area	A pre-existing failure may be present	Reproduce it, report the exact command and failure, and avoid unrelated fixes
CI did not run an expected component	Its paths were not matched by change detection	Inspect .github/workflows/ci.yml and the relevant reusable workflow

Next steps

Next Add an integration Apply this test strategy to an external provider. Develop against the API API layering, authentication, transactions, and migrations. Develop the web app Zod contracts, query keys, cache isolation, and Playwright ownership. Work with domain events Outbox atomicity, handler idempotency, and delivery testing. Testing and verification The compact repository-wide testing reference. Backend code standards The implementation rules every API test should enforce.