Skip to content

feat(testing): property-based tests for validation, numeric utils, and Cedar determinism (CA-02) #253

@krokoko

Description

@krokoko

This is a finding from https://github.com/krokoko/cairn (action item CA-02).

Component

API or orchestration / Agent (Python runtime)

Describe the feature

Add property-based tests (fast-check for TypeScript, Hypothesis for Python) for the exactly-verifiable surfaces of the codebase:

  • Validation / bounds (cdk/src/handlers/shared/validation.ts, numeric.ts) — generators driven from contracts/constants.json so the oracle cannot rot when bounds change. Assert: in-range values accepted, out-of-range rejected, normalization is idempotent (normalize(normalize(x)) == normalize(x)), parse↔serialize round-trips, monotonic clamping.
  • Numeric / timestamp utils — round-trip and monotonicity properties.
  • Cedar decision determinism — decision invariant under attribute reordering; deny overrides permit; unknown action → default deny (complements the existing contracts/cedar-parity/ differential golden suite).

Mirror the validation properties in Python with Hypothesis where the agent re-validates input.

Use case

The validation/bounds and numeric suites today assert hardcoded expected values duplicated from contracts/constants.json. This is an oracle-rot risk: when a bound changes in constants.json, the hardcoded expectations silently drift and the suite proves nothing. Property-based testing replaces brittle example-based oracles with invariants derived from the single source of truth, and is the cheapest high-leverage oracle for the deterministic, "exactly-verifiable" surfaces. This also closes AI009 (happy-path-only coverage) on these surfaces.

Proposed solution

  1. Add fast-check as a dev dependency to cdk (and cli where format.ts warrants it).
  2. For each bounded field, load the bound from contracts/constants.json and generate values inside/outside the range, asserting accept/reject.
  3. Assert normalization idempotence and parse↔serialize round-trips.
  4. Add Hypothesis to agent/ and mirror the validation properties where the agent re-validates.
  5. Add a permutation property for Cedar decision determinism alongside the existing parity fixtures.

Acceptance criteria

  • fast-check (TS) and Hypothesis (Py) are wired into the existing test tasks (mise //cdk:test, mise //agent:test).
  • Validation/bounds generators read ranges from contracts/constants.jsonno hardcoded bound literals in the property tests.
  • Properties cover: in-range accept, out-of-range reject, idempotent normalize, round-trip serialize.
  • Cedar decision-determinism property added (attribute reordering invariant; unknown action → deny).
  • Tests run in the standard unit lane and pass in CI's required build check.

Other information

Source reports: readiness-roadmap.md, verification-report.md (Missing Oracles / oracle-rot watch), verification-strategy.md (§"Validation / bounds (highest impact-per-effort)"), ai-smells-gates-report.md (AI009). Effort: M. Depends on CA-01 (coverage floor) landing first. Per ADR-003 this issue needs the approved label before work begins.

Metadata

Metadata

Assignees

No one assigned

    Labels

    validation-loopTasks related to improve the validation loop for ABCA's codebase

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions