Execution, provenance and failure semantics

The workflow envelope

_workflow provides one monotonic deadline, tree-memory limit and output quota for an operation. Public wrappers perform bounded request validation; the heavy work runs in a child inside a Windows Job configured for tree memory and kill-on-close. A gated bootstrap lets the parent assign the process before allowing the requested workload to proceed. The higher-degree workflow starts serial nested basis workers inside the same monitored tree.

The workflow child handles preparation, state staging for resume, sampling, strict store audit, snapshot replay, diagnostics and summaries. The parent monitors job committed memory, elapsed time, output bytes and process exits. It terminates/reaps descendants and reads only a bounded final descriptor. Request metadata is capped at 1 MiB and depth/structure are bounded before encoding/parsing; the returned descriptor is capped at 64 KiB.

Failure precedence is part of correctness. A recorded child failure is not overwritten by an empty result; a nonzero exit invalidates a forged success; resource or orphan-process failure overrides an intermediate completed result. Final accounting includes receipt bytes. A completed run must have no active descendants and the correct request binding. CLI exit conversion preserves nonzero failures.

Resource estimates and boundaries

ResourceBudget is per operation. Backend size/scratch/live-RAM preflights and decoder geometry budgets independently limit supported requests. The planner’s expected-output estimate is heuristic; actual monitored usage is authoritative. A late failure can leave over-quota artifacts, which must remain classified as failure evidence rather than a completed calculation.

The hard executor supports Windows and workers=1. Its child environment sets OMP/OpenBLAS/MKL/NumExpr thread variables before NumPy import. Library import does not change the caller’s environment. A BLAS limit is not a Python-kernel parallelization feature. A future concurrent executor needs aggregate limits over all participating children, not independent per-child promises.

Strict canonical manifests

_controller parses JSON with duplicate-key refusal. It rejects unknown or missing fields, booleans used as numerical parameters, fractional integers, nonfinite values and incompatible lattice/degree/source declarations. Binary g is bound to x_harmonics[1] and h to z_harmonics[1]; neither copy can be edited independently. Supports, basis groups, sampling counts, storage cadence and metadata hashes are reconstructed and validated together.

Plan IDs are deterministic content hashes, not timestamps or UUIDs. Runtime staging filenames and unique audit output paths can contain UUIDs without changing a deterministic model/plan identity. Canonical declaration hashing does not promise that every mathematically equivalent source encoding, coordinate translation or redundant support spelling shares one hash.

Runtime freezing

implementation_freeze hashes every Python module beneath the installed hcphases import package and records the distribution version. It scans the module tree rather than relying on which modules happen to have been imported. Relative module identifiers make the freeze portable between equivalent editable and wheel installs. The manifest separately records NumPy and model representation information.

Checks occur at launch and declared retained/checkpoint/completion boundaries; the vertex worker also checks periodically during updates. No automatic compatibility migration is performed. Changes to a module, including comments, can invalidate an existing freeze. Preserve original wheels/source and environment identities before releasing a changed implementation. Historical source lookup in a host repository must remain separate from current-code identity checks.

Streams and counters

The higher-degree worker derives stable SeedSequence spawn keys from the root, basis ID, chain ID, purpose ID and internal lane. Initialization and update purposes are separate. It never uses Python hash(), completion order or a mutable global RNG to select a chain’s stream.

The vertex runner first derives a child seed from [root,chain,0], then uses PCG64 inside that trajectory. Its hot start and updates use the same chain’s generator in a fixed order. This is a different stream scheme from the higher-degree worker and is not interchangeable with it.

The higher-degree counted wrapper checks optional internal draw limits before the actual random(), integers() or random_raw() call. Count raw-word branches used by rare Metropolis decisions. The vertex sampler reports requested scalar variates via its own counters. These counts are accounting units, not proofs of independent entropy or pseudorandom independence. Project-specific root/lane allocation belongs outside the library.

Persistence transactions

The general RunStore writes a payload to a unique staging path, reads/hashes it, promotes it without overwriting an existing payload, verifies the final digest, then records the committed inventory entry. NPZ writes use a file handle to avoid NumPy silently adding a filename suffix. Commit paths are containment checked. Append-only JSONL records are appended once, flushed and synchronized; an ambiguous append is not retried because that can duplicate measurements. Inventory sizes/digests are updated and finalization seals the complete schedule.

Atomic metadata writes use bounded Windows-sharing retries. When a staging file disappears, retry the entire deterministic write/replace sequence, rebuilding the payload; merely retrying rename on the missing source cannot recover it. Inventory insertion follows successful digest verification. A transaction is not a claim that every metadata path is protected against every possible hardware or external filesystem failure; interrupted attempts are audited and classified rather than imputed.

Vertex histories use per-chain NPZ plus final JSON checkpoints and a separate strict inventory. Measurements, selected snapshot indices/bitsets, metadata schemas, finite values and exact schedules are validated. It is not the general RunStore layout. Both paths require an authoritative workflow receipt and request digest for completed input loading.

Replay and continuation

General replay enumerates the exact expected group/chain/retention/checkpoint inventory and recomputes physical measurements on saved configurations under the original implementation. It preserves joint a,b,d and decoder metadata. It rejects absent/extra/inconsistent artifacts and zero replayable snapshots. Scalar-only records cannot certify unsaved configurations.

General continuation requires checkpoints-index.json binding a complete set of latest chain states. It validates source receipts, digests, cache/state/RNG metadata and the retention phase, then stages state under the new operation’s budget. It inherits the root and refuses reseeding. The warmup is not repeated; absolute sweep numbering and retention/checkpoint cadence continue exactly. The decisive regression is equality of an uninterrupted N+K trajectory’s saved suffix and an N-then-resume-K suffix, including state, caches, generator, counters and measurement positions. Compare persisted artifacts, not only two in-memory objects.

Vertex low-level restore reconstructs the model, bitset, PCG64 state and counters. It does not provide a run-level continuation index or bounded public resume workflow. Adding one would require new inventory, phase, freeze and total-budget design rather than relabeling a final checkpoint as a completed continued run.