# CatDB benchmark and falsification plan

## Purpose and current status

The benchmark program tests separate propositions about semantic compilation,
native execution, integration maintenance, migration review, provenance,
incremental maintenance, workspaces, federation, routing, and distribution.
Evidence from one proposition does not transfer automatically to another.

This document is a confirmatory protocol plan, not a results report. No
benchmark harness, populated dataset, measured run, human study, or numerical
CatDB result exists. Benchmark execution is deferred beyond the current
research-program completion boundary. All artifact IDs, baselines, workloads,
thresholds, and tests below remain future-work contracts rather than pending
completion tasks.

A fast local SQL plan would not establish distributed convergence. A smaller
mapping count would not establish lower runtime cost. A plan that avoids
loading rows into the semantic kernel would not by itself establish
billion-row scalability.

## Evidence-taxonomy boundary

The evidence labels in `research/claim-register.md` classify what the immutable
context sources say. They are not outcomes of this benchmark program:

- `PROPOSED` and `INFERRED` remain hypotheses until external, formal, or
  experimental evidence passes its gate;
- `REPORTED_IMPLEMENTED` describes a report about another system and requires
  current primary-source or executable verification before it can support a
  prior-art statement;
- `UNRESOLVED` remains unresolved;
- `CONTRADICTED` records a source conflict and cannot be repaired by a CatDB
  benchmark.

Program labels are assigned separately. Checked primary sources may justify
`supported by prior literature`; reviewed proofs may justify `proved`; exact
Lean kernel checks may justify `machine-checked`; and completed reproducible
experiments may justify `experimentally supported`. A fixture expectation,
implementation, or benchmark design alone does not promote a claim.

## Planned artifact registry

Every run uses a unique `run_id`. Files are append-only after their SHA-256 is
entered in `benchmarks/runs/<run_id>/SHA256SUMS`; a rerun creates a new
directory rather than overwriting evidence.

| Artifact ID | Planned path | Required content |
|---|---|---|
| `ART-RUN-01` | `benchmarks/runs/<run_id>/run-manifest.json` | run ID, status, `ART-SRC-01` reference, every input/output artifact ID and checksum |
| `ART-SRC-01` | `benchmarks/runs/<run_id>/source-state.json` | clean commit/tree IDs, recursive submodule commits, dependency-lock checksums, isolated checkout/container identity |
| `ART-PRE-01` | `benchmarks/protocols/confirmatory-v1.yaml` | preregistered hypotheses, workloads, metrics, margins, power analysis, stopping/exclusion rules |
| `ART-BASE-01` | `benchmarks/baselines/manifest-v1.yaml` | baseline ID, source citation, exact revision/image digest, license, configuration, executable/reconstructed status |
| `ART-DATA-01` | `benchmarks/datasets/manifest-v1.json` | generator revision, parameters, seed, row counts, file/content checksums |
| `ART-ENV-01` | `benchmarks/runs/<run_id>/environment.json` | complete machine, software, database, compiler, network, and clock identity |
| `ART-ORDER-01` | `benchmarks/runs/<run_id>/schedule.json` | immutable randomized block order and randomization seed |
| `ART-PG-01` | `benchmarks/runs/<run_id>/postgres-differential.jsonl` | portable-subset case, backend version, normalized result/outcome/provenance |
| `ART-RAW-01` | `benchmarks/runs/<run_id>/observations.parquet` | one typed row per raw observation, including failures and timeouts |
| `ART-PROV-01` | `benchmarks/runs/<run_id>/provenance-audit.jsonl` | expected and observed version/snapshot/record/plan lineage |
| `ART-MIG-01` | `benchmarks/runs/<study_id>/migration-study/` | consent-safe assignments, normalized scoring projections, escrowed condition map, reviewer condition guesses, tool-aware audit, analysis |
| `ART-WS-01` | `benchmarks/runs/<run_id>/workspace-results.jsonl` | checkout/diff/merge/equivalence/isolation/atomicity outcomes |
| `ART-DIST-01` | `benchmarks/runs/<run_id>/distributed-histories/` | generated histories, delivery schedules, canonical states, minimized failures |
| `ART-STAT-01` | `benchmarks/runs/<run_id>/analysis/` | pinned analysis code, per-workload intervals, plots, exclusions, and decision table |

The top-level run manifest records the `ART-SRC-01` reference/checksum, every
other artifact ID and checksum, and the checksum of its own canonical content
excluding only its checksum field. Any missing/mismatched input checksum makes
the run non-confirmatory.

## Claim-to-falsification matrix

Formal invariant IDs refer to `research/formalization-roadmap.md`. `SYS-*`
invariant IDs refer to the systems ledger in
`research/implementation-roadmap.md`. `N/A` is not used: every row names the
specific formal or executable obligation whose failure blocks the claim.
Every row requires the base bundle `ART-RUN-01`, `ART-SRC-01`,
`ART-PRE-01`, `ART-DATA-01`, `ART-ENV-01`, `ART-ORDER-01`,
`ART-RAW-01`, and `ART-STAT-01`; a comparative row also requires
`ART-BASE-01`. Cells repeat some base IDs for readability and name the
claim-specific artifacts.

| Claim ID and context state | Program label before experiment | Paper IDs | Invariant IDs | Dataset IDs | Named artifact IDs; base bundle also required | Confirmatory test | Result that rejects or narrows the claim |
|---|---|---|---|---|---|---|---|
| `CR-028` `PROPOSED`: kernel reasons over types/plans rather than rows | engineering hypothesis | P2, P5, P12, P14, P15, S | `QRY-01`, `QRY-02`, `SYS-QRY-01` | D1, D3 | `ART-PRE-01`, `ART-DATA-01`, `ART-ENV-01`, `ART-RAW-01`, `ART-STAT-01` | hold schema/query shape fixed, vary physical row count, measure compiler memory/time separately from backend execution, and inspect the generated native plan | the compiler materializes source rows, performs per-row semantic interpretation, or compiler resource use grows with row count beyond preregistered metadata/statistics inputs |
| `CR-036` `UNRESOLVED`: billion-row scalability | engineering hypothesis | P2, P5, P12, P14, P15, S | `SYS-QRY-01`, `SYS-PG-01`, `SYS-CON-01` | D3 tier `D3-1B`, D4 | `ART-PRE-01`, `ART-BASE-01`, `ART-DATA-01`, `ART-ENV-01`, `ART-ORDER-01`, `ART-PG-01`, `ART-RAW-01`, `ART-PROV-01`, `ART-STAT-01` | execute the full correctness and performance protocol on at least `1,000,000,000` physical input rows for each backend/workload claimed | any correctness failure or materially worse compiler/exchange/plan behavior; if `D3-1B` is not actually run, CR-036 remains unresolved and no billion-row evidence may be claimed |
| `CR-030` `UNRESOLVED`: near-linear integration maintenance | engineering hypothesis | P4, P7, P13, S | `MAP-04`, `MAP-05`, `LOSS-01` | D2, D5 | `ART-PRE-01`, `ART-BASE-01`, `ART-DATA-01`, `ART-RAW-01`, `ART-STAT-01` | compare pairwise, GAV, LAV, Information Manifold-style, and CatDB structures under controlled source/domain/consumer evolution | repair work remains pairwise, the domain/source-description/query-rewrite burden erases the expected benefit, or opaque mapping exceptions absorb the work |
| `CR-040` `PROPOSED` and `CR-046` `UNRESOLVED`: typed migration impact/meaning | engineering hypothesis | P3, P4, P9, P17, S | `LOSS-01`, `DIFF-01`, `DIFF-02` | D5, D5-H | `ART-PRE-01`, `ART-DATA-01`, `ART-MIG-01`, `ART-STAT-01` | seeded-fault tool evaluation plus the preregistered migration human study | CatDB misses faults found by SQL/ORM baselines, impact sets omit dependencies, false positives are unusable, or reviewers cannot recover the stated meaning |
| `CR-063` `UNRESOLVED`: incremental recomputation with transactional cursors | engineering hypothesis | P6, P10, P11, P14, S | `SYS-MAT-01` | D5, D6 | `ART-PRE-01`, `ART-BASE-01`, `ART-DATA-01`, `ART-RAW-01`, `ART-PROV-01`, `ART-STAT-01` | compare full recomputation, invalidation, DBSP/differential maintenance, and CatDB under data/schema/mapping/provenance-policy changes and crash failpoints | result/provenance diverges, a cursor loses/duplicates changes, or supported-fragment incremental work is noncompetitive without another measured semantic benefit |
| `CR-060` `UNRESOLVED`: semantic metadata improves physical planning | engineering hypothesis | P5, P12, P14, S | `SYS-QRY-01`, `SYS-CON-01` | D4, D6 | `ART-PRE-01`, `ART-BASE-01`, `ART-DATA-01`, `ART-ENV-01`, `ART-RAW-01`, `ART-STAT-01` | ablate semantic features against Calcite/Trino-style and R*-style communication-aware baselines with identical capabilities/statistics | semantic metadata does not improve routing/cost error or end-to-end cost after its planning and maintenance overhead |
| `CR-062` `PROPOSED`: versioned operational provenance | engineering hypothesis | P6, P9, P13, P14, S | `SYS-PROV-01` | D3, D4, D5, D7 | `ART-PRE-01`, `ART-DATA-01`, `ART-PROV-01`, `ART-RAW-01`, `ART-STAT-01` | audit every sampled result against expected commit/schema/mapping/source snapshot/plan/record lineage and measure storage/runtime overhead | any result cannot be traced to the required versions and records, lineage composition is wrong, or preregistered overhead makes admitted workloads impractical |
| `CR-069` `PROPOSED`: overlay-based semantic workspaces | engineering hypothesis | P9, P10, P11, P13, P17, S | `SYS-CAT-01`, `SYS-WS-01`, `SYS-WS-02`, `SYS-WS-03` | D7 | `ART-PRE-01`, `ART-BASE-01`, `ART-DATA-01`, `ART-WS-01`, `ART-PROV-01`, `ART-STAT-01` | compare shared version operations with Decibel/ORPHEUSDB baselines, then test CatDB-only cross-object overlay, isolation, conflict, recovery, and atomic merge cases | overlay and materialized results differ, one workspace leaks into another, merge publication is partial/order-dependent, or the co-versioning delta has no demonstrated value over prior systems |
| `CR-045` `PROPOSED`: reproducible version histories and merge | open conjecture | P9, P10, P11, S | `SYS-WS-03`, `SYS-DIST-01` | D6, D7 | `ART-PRE-01`, `ART-DATA-01`, `ART-DIST-01`, `ART-WS-01`, `ART-STAT-01` | replay duplicate/reordered/partitioned histories and compare canonical states/conflicts | any operation classified as convergent diverges, recovery loses the log, or a noncommutative transition silently merges |

## Baseline registry and fairness contract

No baseline enters a confirmatory run until `ART-BASE-01` records its exact
source, revision or container digest, build, configuration, capability limits,
and validation cases. Published numbers from different hardware are context,
not co-measured evidence.

| Baseline ID | Required baseline | Workload use | Operationalization rule |
|---|---|---|---|
| `BL-NATIVE-SQLITE-01` | hand-written native SQLite | local execution, transactions, provenance-off lower bound | same DDL, indexes, result decoding, process boundary, and cache state |
| `BL-NATIVE-PG-01` | hand-written native PostgreSQL | portable execution and parity | same portable-subset manifest, isolation, data, indexes, client path, and cache state |
| `BL-PAIRWISE-01` | direct pairwise mappings/pipelines | integration maintenance | same transformations and test oracle; count all exception and identity work |
| `BL-GAV-01` | Global-as-View integration | integration maintenance | independent encoding reviewed against B59 and the B03–B08/B15 data-integration sources; count global-view revisions |
| `BL-LAV-01` | Local-as-View/source-description integration | integration maintenance | independent source descriptions and query rewriter reviewed against B59 and the B03–B08/B15 sources; count source descriptions and query rewrites |
| `BL-IM-01` | Information Manifold-style mediated-domain planning | integration/federation | source descriptions, source selection, and mediated query rewriting follow B59's reviewed contract; this is labeled a reconstruction unless executable original code is available |
| `BL-CALCITE-01` | Apache Calcite-style adapter/rule/cost planning | federation/planning | pin the B22 upstream Calcite lineage and adapters where runnable; record unsupported features |
| `BL-TRINO-01` | Trino-style connector pushdown/federation | federation/planning | pin the B23 Trino lineage and connector versions; compare only mutually supported pushdowns and failure modes |
| `BL-DBSP-01` | DBSP incremental maintenance | incremental materialization | pin the B27 compiler/runtime lineage and supported query fragment; compare exact data-delta cases |
| `BL-DIFF-01` | differential-dataflow maintenance | incremental materialization | pin the B25–B26 implementation lineage and logical-time arrangement; separate unsupported schema/mapping changes |
| `BL-DECIBEL-01` | Decibel relational versioning | workspace storage/query/diff/merge | use B60 released code/artifacts if reproducible; otherwise a reviewed reconstruction is reported separately and cannot inherit published performance |
| `BL-ORPHEUS-01` | ORPHEUSDB relational versioning | workspace storage/query/diff/merge | apply the same executable-versus-reconstruction rule to B61 |
| `BL-RSTAR-01` | R*-style communication-aware distributed planning | routing/join placement | independently implement B62's cited message/I/O/CPU cost terms and relation shipping, fetch-matches, semijoin, and Bloom alternatives; validate on hand-solvable plans |
| `BL-SQL-MIG-01` | ordered SQL migration scripts | migration study | expert-authored, reviewed, same target and recovery requirements |
| `BL-ORM-MIG-01` | one mainstream ORM migration workflow | migration study | version pinned before recruitment; same schemas, faults, documentation, and time budget |

For unavailable historic systems, a reconstruction is a baseline model, not a
claim about the original implementation. A reconstructed baseline must pass an
independent source-to-behavior review. If it does not, the corresponding
comparison is removed from confirmatory claims and reported as blocked.

## Dataset families

Every generated instance has a stable dataset ID, generator revision, seed,
parameter tuple, exact logical/physical row counts, and SHA-256 in
`ART-DATA-01`.

### D1: Schema topology

Parameters:

- objects: 10, 100, 1,000, 10,000;
- morphisms per object: 2, 5, 10;
- equations per object: 0, 1, 5;
- maximum normalized path length: 2, 4, 8, 16;
- valid and single-fault ill-typed paths.

Purpose: schema compilation, path normalization, equation indexing, error
reporting, and CR-028 compiler scaling.

### D2: Mapping composition

Parameters:

- source and target object counts;
- mapping density;
- chain length: 1, 2, 4, 8, 16;
- partial assignments;
- lossy and ambiguous transformations;
- equation-preservation obligations.

Purpose: validation, composition, loss classification, cache behavior, and
integration-structure construction.

### D3: Relational execution

Workloads:

- primary-key lookup;
- selective scan;
- projection;
- one-to-many inner join;
- three-way inner join;
- aggregation only after its own versioned semantic profile;
- path materialization;
- provenance disabled, record lineage, and transformation lineage.

Core scales are \(10^4\), \(10^6\), and \(10^8\) physical input rows with
uniform/skewed keys and narrow/wide rows. `D3-1B` is a separate optional
confirmatory tier with at least \(10^9\) physical input rows. Extrapolation,
compressed logical counts, or repeated views do not satisfy `D3-1B`.
CR-036 remains unresolved unless `D3-1B` is actually executed on every backend
and workload named by the claim.

### D4: Heterogeneous federation

Sources:

- SQLite;
- PostgreSQL;
- Markdown;
- CSV;
- JSON;
- one deterministic HTTP API fixture.

Variables include local/remote join, selectivity, latency, source failure,
dialect/capability mismatch, reconciliation rate, data movement, and
provenance level.

### D5: Evolution, migration, and maintenance

Changes:

- additive field;
- renamed field with preserved identity;
- object split and merge;
- cardinality change;
- strengthened constraint;
- value-domain conversion;
- mapping revision;
- source deletion;
- reconciliation reversal;
- provenance-policy revision.

`D5-H` contains isomorphic, human-reviewable migration tasks with seeded
faults and an independently adjudicated ground truth.

Information-loss scoring follows the shared fixture contract. Concrete plus
inconclusive findings are `lossy`; only inconclusive findings are `unknown`;
zero findings are `lossless_under_checked_criteria`, never proof of
invertibility or safe business migration.

### D6: Incremental and distributed histories

Parameters:

- operation/update stream length and batch size;
- data, schema, mapping, source-capability, and provenance-policy changes;
- replicas: 2, 3, 5, 9;
- partitions/healing;
- duplicated, delayed, and reordered delivery;
- version skew;
- stale materializations;
- coordinated and uncoordinated administrative changes.

Purpose: incremental equivalence, cursor recovery, replay idempotence,
convergence, conflicts, causal ordering, checkpoints, and routing.

### D7: Versioned semantic workspaces

Parameters:

- version graph width/depth;
- base data size and changed fraction;
- schema, mapping, data, provenance, identity, and authority-policy edits;
- checkout, diff, historical query, branch, rebase, and merge;
- independent, commutative, conflicting, and reconciliation-required edits;
- crash failpoint before and after commit/ref publication.

Shared relational operations support fair Decibel/ORPHEUSDB comparison.
CatDB-only cross-object semantic cases are reported separately and cannot be
used to imply those systems fail requirements they never claim.

## Confirmatory reproducibility protocol

### Preregistration and immutable inputs

Before inspecting confirmatory outcomes:

1. require a clean committed source tree and finalize `ART-SRC-01`,
   `ART-PRE-01`, `ART-BASE-01`, and `ART-DATA-01`;
2. record their SHA-256 values in the run manifest;
3. pin the exact source commit/tree, recursive submodule commits, dependency
   lockfiles, compiler profiles, container/VM image digests, database images,
   SQL/fixture manifests, and analysis environment;
4. generate `ART-ORDER-01` from a committed randomization seed;
5. run correctness gates before performance gates.

Pilot data may tune harnesses and estimate variance but is stored under a
different run ID and cannot enter confirmatory analysis.

### Confirmatory source-state gate

A confirmatory run executes only from a detached isolated checkout, Git
worktree, or container build context created from the exact commit recorded in
`ART-SRC-01`. The source root, every initialized submodule, and the committed
benchmark inputs must satisfy:

```bash
test -z "$(git status --porcelain=v1 --untracked-files=all)"
git diff --quiet
git diff --cached --quiet
test -z "$(git submodule status --recursive | sed -n '/^[+-U]/p')"
git submodule foreach --recursive 'test -z "$(git status --porcelain=v1 --untracked-files=all)" && git diff --quiet && git diff --cached --quiet'
```

`ART-SRC-01` records `git rev-parse HEAD`, `git rev-parse HEAD^{tree}`,
`git submodule status --recursive`, the SHA-256 of every present dependency
lock (`Cargo.lock`, `cabal.project.freeze`, `lake-manifest.json`,
`package-lock.json`, and the analysis lock), and the OCI/container build
context and image digests when used. It also records the isolated source-root
identity and the command that created it. Missing required locks, submodule
drift, any tracked/untracked change, or execution from the mutable development
checkout makes the run non-confirmatory.

A dirty run may be retained under `benchmarks/exploratory/` with a complete
patch/untracked manifest, but it is watermarked `exploratory` and can never
promote a claim. Promotion requires a new preregistered run from the committed
clean source state.

### Environment identity

`ART-ENV-01` records:

- `ART-SRC-01` checksum and isolated checkout/container identity;
- OS image, kernel, architecture, CPU model/stepping, microcode, core/NUMA
  topology, memory, storage model/filesystem, power/performance governor, and
  relevant firmware;
- Rust, GHC, Cabal, Lean, Lake, Node, SQLite, PostgreSQL, JVM, Calcite, Trino,
  DBSP, differential-dataflow, ORM, and analysis-tool versions as applicable;
- compiler flags, features, allocator, database settings/extensions,
  isolation, collation, locale, time zone, and connection-pool settings;
- network topology, link rate, traffic shaping, and clock synchronization for
  remote/distributed runs;
- background-load check and thermal/power telemetry.

A textual machine nickname is insufficient. Environment capture is verified
before each block; drift creates a new environment stratum or invalidates the
block.

### Experimental units, machines, seeds, and repetitions

Repeated timed invocations in one process are not independent units.

| Group | Independent experimental unit | Nested repetitions |
|---|---|---|
| A semantic compilation | fresh process + independently generated dataset seed on a named physical machine | fixed warmups and at least 30 measured invocations |
| B execution/PostgreSQL | fresh database instance/volume + dataset seed on a named physical machine | fixed query repetitions within cache stratum |
| C integration maintenance | independently generated evolution scenario reviewed before treatment | repeated changes within scenario |
| D migration study | participant; task observations are nested within participant and crossed with task family | actions within the task are not independent |
| E incremental | fresh base state + update-history seed | update batches within history |
| F distributed | fresh cluster deployment + history/fault-schedule seed | deliveries within history |
| G workspace | fresh version graph + edit-history seed | operations within graph |
| H routing/federation | fresh source placement/data seed + network profile | query repetitions within placement |

The preregistered power analysis chooses unit count. Any performance claim
intended to generalize beyond one machine requires at least three independently
administered physical machines and at least five independent units per
machine/treatment; otherwise it is explicitly machine-specific exploratory
evidence. Dataset, history, and randomization seeds are independent and
recorded. Nested repetitions improve measurement precision but do not inflate
the independent sample size.

### Cache-state control

Cache state is a treatment, not an informal note:

- `cold-database` recreates a database container/VM and data volume from the
  checksummed image; if host page cache cannot be reset and verified, the run
  is labeled `fresh-database/host-cache-uncontrolled`, not cold;
- `warm-database` executes the fixed priming sequence and verifies expected
  buffer residency before measurement;
- `cold-plan` starts a fresh CatDB process with an empty plan cache;
- `warm-plan` compiles once, verifies the cache key, then measures lookup and
  execution;
- baseline and CatDB treatments use the same declared cache stratum.

The exact reset/priming commands and observed cache counters are stored in the
raw record. Cold and warm observations are never pooled.

### Randomization and masking

Treatments run in randomized blocks within each independent unit. The schedule
is generated before measurement, checksummed as `ART-ORDER-01`, and records
warm/cold strata. Where carryover is possible, a reset occurs between
treatments. Human-study primary scoring uses the condition-neutral projection
defined below and hides the condition map and participant identity until
scores freeze. Masking is not assumed perfect: reviewers record a condition
guess and confidence after each primary score because projection structure may
still make a condition inferable. Performance benchmark analysis reads
treatment labels only after raw validation and exclusions are frozen.

### Measurements and units

Raw observations include explicit units:

- planning, first-row, total, CPU, and refresh latency in nanoseconds;
- bytes scanned/transferred/stored and peak resident bytes;
- rows scanned/returned/recomputed;
- remote calls, retries, cancellations, and failures;
- native plan/operator statistics;
- result and provenance checksum;
- cache state, treatment order, machine, unit, seed, repetition, and timeout.

One-time semantic compilation is separated from cached execution. Provenance
levels are separate treatments.

### Confidence, noninferiority, and multiple workloads

Confirmatory estimates use 95% confidence. For each preregistered local
execution workload \(w\), define the paired independent-unit log ratio

\[
\theta_w = \operatorname{median}\left(
  \log(T_{\mathrm{CatDB},w}/T_{\mathrm{native},w})
\right).
\]

The initial native-like noninferiority margin is
\(\log(1.10)\) for median cached execution latency. A workload passes only when:

1. result equivalence and required provenance checks pass;
2. the Holm-adjusted one-sided 95% cluster-bootstrap upper confidence bound
   for \(\theta_w\) is below \(\log(1.10)\);
3. timeout/failure incidence does not exceed the preregistered margin.

Intervals are reported per workload, backend, scale, cache stratum, and
provenance level. Passing workloads are not averaged with failing workloads;
“across simple workloads” is not a substitute for per-workload intervals.
Any p95 noninferiority claim needs its own preregistered estimand and margin.
Skewed latency summaries include median, p95, dispersion, and cluster
bootstrap intervals.

Superiority analyses, planner variants, or secondary endpoints use the
preregistered multiplicity correction. Unplanned slicing is exploratory.
Correctness failures are categorical failures, not benchmark variance.

### Stopping rules

Confirmatory runs collect the fixed unit count from the power analysis. They
do not stop early for favorable performance. A case stops only for:

- a correctness/provenance mismatch;
- a preregistered timeout or resource-safety ceiling;
- hardware/environment failure that could damage the test system;
- a consent/safety issue in the human study.

A correctness or timeout stop remains an outcome. If infrastructure prevents
the planned sample size, the result is `inconclusive`, not a pass. Additional
units after outcome inspection form a new preregistered run.

### Exclusion rules

Before labels are unblinded, exclude only:

- input or environment checksum mismatch;
- failed clock/telemetry validation;
- operating-system or harness failure external to the system under test;
- a documented reset failure that invalidates both members of a paired block;
- for human recruitment, missing consent or failure of preregistered
  eligibility/training checks before randomization.

Performance outliers, slow trials, system-under-test crashes, query errors,
timeouts, and wrong results are never discarded as noise. A paired
infrastructure exclusion removes both treatments. Every exclusion and reason
is published. After randomization, incomplete tasks, tool failures, timeouts,
and loss to follow-up remain in the primary intention-to-treat analysis under
the preregistered missing-outcome rule; data withdrawn from use are handled
only as required by consent/ethics and reported. No randomized participant is
replaced to improve the analyzed sample.

## PostgreSQL versioned portable-subset differential protocol

`compat/postgres-portable-subset-v1.yaml` is the semantic contract and its
checksum is included in `ART-PRE-01` and `ART-PG-01`. It pins:

- manifest semantic version;
- PostgreSQL major versions and image digests;
- SQLite library version;
- supported scalar types and canonical encodings;
- DDL, constraints, indexes, selection/projection/inner-join operators;
- transaction, isolation, error, collation, locale, and time-zone rules;
- introspection normalization;
- cancellation outcome and session-reuse requirements;
- plan-cache key inputs and invalidation triggers;
- unsupported capability/error codes;
- required provenance fields and normalization.

Each case starts from clean checksummed storage and runs the following paired
protocol:

1. lower the same canonical schema to SQLite and PostgreSQL;
2. introspect and compare normalized tables, columns, types, nullability,
   keys, constraints, and indexes;
3. execute commit, rollback, savepoint, failed-statement, and declared
   isolation schedules and compare normalized durable outcomes;
4. cancel a bounded long statement, assert the allowed cancellation class,
   prove no forbidden partial effect, and reuse the session;
5. warm a plan, independently change DDL, schema version, mapping version,
   source snapshot, and connector capabilities, and assert stale keys miss or
   invalidate;
6. execute query fixtures and compare canonical result bags;
7. compare commit/schema/mapping/query/plan/source/record provenance;
8. invoke every deliberately unsupported feature and require the manifest's
   exact stable diagnostic.

The output is per PostgreSQL version. Physical plans, backend wording, and
internal catalog OIDs are not compared. Failure on one supported major cannot
be hidden by aggregate parity. Adding a backend version or changing any
portable semantic rule creates a new manifest version.

## Benchmark groups

### A: Semantic compilation

Measurements:

- parse, validate, normalize, and serialize separately;
- path typing, mapping validation/composition, loss/diff analysis;
- logical/physical planning and cache lookup;
- peak allocation and resident memory;
- valid and single-fault invalid inputs.

Initial engineering targets:

- a 100-object schema compiles within 100 ms on the named reference machine;
- cached plan lookup is at least one order of magnitude faster than uncached
  physical planning for the same version tuple.

These are preregistered decision thresholds, not current claims. CR-028 also
requires compiler resource use to be tested across D3 row counts while
schema/query shape is held fixed.

### B: Execution performance and provenance

Baselines: `BL-NATIVE-SQLITE-01`, `BL-NATIVE-PG-01`, CatDB-generated SQL,
full/partial materialization, and cached/uncached CatDB execution.

Correctness precedes timing. The protocol captures native plans, controls
transaction/cache states, separates compilation from execution, and treats
provenance levels separately. The 10% median margin is evaluated using the
per-workload noninferiority interval above; a plan difference is reported and
analyzed, not silently attributed to framework overhead.

### C: Integration complexity

Conditions:

- `BL-PAIRWISE-01`;
- `BL-GAV-01`;
- `BL-LAV-01`;
- `BL-IM-01`;
- CatDB typed source-to-domain/domain-to-consumer mappings.

Use \(N,M \in \{2,4,8,16\}\), one good shared-domain condition, and one
poor-fit condition. Apply controlled source, domain, and consumer changes.

Measurements include mapping/pipeline/source-description/query-rewrite count,
executable and declarative size, source-specific branches, reviewable semantic
decisions, touched modules, failing tests, repair time, identity/reconciliation
work, exceptions, domain-model maintenance, and runtime data movement. The
\(N+M\) hypothesis passes only if total repair effort, not merely definition
count, improves under its stated reuse assumptions.

### D: Migration correctness and human study

Tool conditions: `BL-SQL-MIG-01`, `BL-ORM-MIG-01`, and CatDB typed mappings
with the same physical target and recovery requirement.

Seeded faults include silent truncation, invalid default, lost identity,
broken foreign-key meaning, ambiguous split, incomplete merge, violating rows
under a stronger constraint, and a compatibility view that changes
cardinality.

#### Human-study design

1. Obtain the required institutional ethics determination, informed consent,
   and data-retention plan before recruitment.
2. Recruit practicing database/application engineers who pass a
   preregistered SQL/schema-migration competency screen. Record experience and
   prior tool familiarity for stratification.
3. Use a pilot only to validate task duration and estimate variance. A
   preregistered power analysis, not convenience, sets the confirmatory sample
   size in randomized eligible participants; the planning floor is 36
   randomized participants unless the power analysis requires more. Inflate
   recruitment separately for expected attrition, without redefining the
   powered target as completers.
4. Use a randomized, counterbalanced within-participant design. Each
   participant uses all three conditions on isomorphic D5-H tasks; a Latin
   square balances tool order and task family. Participants never see the same
   underlying fault instance twice.
5. Give standardized, timed training and reference documentation for every
   tool, followed by an unscored practice task. Record but do not exclude
   unfamiliarity after the eligibility gate.
6. Primary endpoint: seeded semantic faults correctly identified before
   execution. Secondary endpoints: false positives, time to an executable
   plan, correct impact set, successful rollback/recovery, final-plan
   correctness, confidence calibration, and reviewer-rated explanation
   quality.
7. Before scoring, a condition-neutral adapter projects every SQL, ORM, or
   CatDB submission into the same closed schema: pseudonymous task ID,
   normalized target-schema delta, selected fault-category IDs,
   affected-object IDs, loss/ambiguity decisions, proposed execution outcome,
   and normalized recovery-operation IDs. It removes source syntax, filenames,
   tool labels, native diagnostics, formatting, and free-text explanation.
   Projection conformance fixtures prove that equivalent answers from all
   three tools produce the same scoring value.
8. Two reviewers score only these normalized projections against the
   independently adjudicated ground truth. After each score they guess the
   condition and record confidence. A third projection-blinded reviewer
   resolves scoring disagreement. Report inter-rater agreement, condition-
   guess accuracy, and a sensitivity analysis for likely unmasking.
9. A separate tool-aware audit verifies projection fidelity and scores
   secondary usability, explanation quality, and tool-specific failure modes.
   Tool-aware judgments cannot change the frozen primary score.
10. Analyze detection with a mixed-effects logistic model and time with a
   preregistered censored/mixed model, with participant and task family as
   random effects and order/familiarity as fixed covariates. The primary
   intention-to-treat analysis retains every randomized eligible participant
   whose data may ethically be retained; incomplete/timeout tasks use the
   preregistered conservative outcome rule. A per-protocol completer analysis
   is secondary. Report 95% intervals for both without substituting the
   completer result for the primary estimand.

The study cannot show that mappings are universally safer. It tests the
selected faults, tools, participants, training, and time budgets.

### E: Incremental materialization

Treatments:

- full recomputation;
- invalidation plus lazy recomputation;
- `BL-DBSP-01`;
- `BL-DIFF-01`;
- CatDB incremental update;
- reused CatDB materialized path.

Measure update latency, recomputed rows, retained/invalidated state, freshness,
storage amplification, result/provenance equivalence, and recovery after data,
schema, mapping, capability, and provenance-policy changes. Unsupported
transformations must invalidate/recompute. Transactional cursor failpoints are
correctness gates before performance measurement.

### F: Distributed behavior

Generate deterministic D6 histories, execute multiple delivery orders and
partitions, compare canonical replica states/conflict sets, and preserve every
failing/minimized history in `ART-DIST-01`.

Properties:

- replay idempotence;
- convergence of only the declared commutative subset;
- explicit conflict or coordination for noncommutative transitions;
- no forbidden causal dependency inversion;
- visible stale version/freshness;
- operation-log survival through partition recovery.

Divergence in the declared subset is a correctness failure.

### G: Versioned workspaces

Compare CatDB, `BL-DECIBEL-01`, and `BL-ORPHEUS-01` on mutually supported
storage, checkout, diff, merge, and historical-query operations. Report
version-storage amplification, checkout/diff/query latency, and merge
throughput.

Separately test CatDB's claimed delta: co-versioned schema, mapping, data,
provenance, identity/reconciliation, and authority-policy overlays. Required
correctness gates are `SYS-WS-01` through `SYS-WS-03`, including query
equivalence, isolation, conflict classification, and atomic merge under every
declared failpoint. Do not use CatDB-only semantics to claim performance
superiority over systems outside that scope.

### H: Federation and communication-aware routing

Baselines: `BL-CALCITE-01`, `BL-TRINO-01`, and `BL-RSTAR-01`.

Treatments:

- physical statistics/capabilities only;
- semantic metadata added one feature at a time;
- full CatDB planner.

R*-style alternatives include local/remote execution, whole-relation
shipping, fetch-matches, semijoin, and Bloom-assisted transfer. Calcite/Trino
comparisons cover mutually supported adapters/pushdowns, partial failure, and
planning cost. Measure result equivalence, planning time, first/total latency,
transferred rows/bytes, calls, CPU/I/O, reconciliation/provenance/coordination
cost, and predicted-versus-observed error.

A “by semantic metadata” claim requires the named metadata to be visibly
enabled in the ablation and to improve a preregistered endpoint after its own
maintenance/planning overhead.

## Planned execution gates

The future harness must expose these commands:

```bash
test -n "${CATDB_RUN_ID:-}"
catdb_isolated_root="../catdb-confirmatory-${CATDB_RUN_ID}"
cargo run -p catdb-bench -- prepare-source --run-id "$CATDB_RUN_ID" --require-clean --isolation detached-worktree-or-container --destination "$catdb_isolated_root"
cd "$catdb_isolated_root"
cargo run -p catdb-bench -- capture-source --run-id "$CATDB_RUN_ID" --require-clean --require-isolated
cargo run -p catdb-bench -- verify-inputs benchmarks/protocols/confirmatory-v1.yaml
cargo run -p catdb-bench -- capture-environment --run-id "$CATDB_RUN_ID"
cargo run -p catdb-bench -- generate-schedule --run-id "$CATDB_RUN_ID" --protocol benchmarks/protocols/confirmatory-v1.yaml
cargo run -p catdb-bench -- correctness --run-id "$CATDB_RUN_ID"
cargo run -p catdb-bench -- execute --run-id "$CATDB_RUN_ID"
Rscript benchmarks/analysis/confirmatory-v1.R --run-id "$CATDB_RUN_ID"
cargo run -p catdb-bench -- verify-artifacts --run-id "$CATDB_RUN_ID"
```

Entry: all referenced implementation-slice correctness gates pass and every
input checksum resolves. Exit: commands return zero; raw data include failures
and timeouts; `ART-STAT-01` reports every preregistered workload, interval,
exclusion, and decision; and an independent reproducibility review can rerun
the analysis from immutable artifacts.

## Execution order

1. fixture and profile correctness;
2. cross-language semantic agreement;
3. PostgreSQL portable-subset correctness;
4. catalog and connector conformance;
5. semantic compilation;
6. SQLite/PostgreSQL result and provenance equivalence;
7. local execution noninferiority;
8. integration and migration studies;
9. incremental correctness, crash recovery, then performance;
10. workspace correctness, prior-art comparison, then performance;
11. federation/routing baselines and ablations;
12. distributed histories;
13. optional `D3-1B` scale tier.

No later performance result can waive an earlier correctness failure. No
artifact produced by this plan is evidence until the corresponding run has
actually completed and passed review.