The measurement and reference-implementation hub for testing bounded claims about compression, expression, memory, recursion, preserved reusable structure, and total computational cost .
The benchmark system connects the governing canon, preregistered evaluation protocols, matched baselines, versioned GitHub artifacts, machine-readable results, and evidence-state governance.
Current Public Release
Published August 16, 2026, this immutable GitHub release provides a stable and citable reference point for reproducible inspection, implementation review, benchmark evaluation, and future result comparison.
Release tag: benchmarks-v0.2.0 • Governing authority: GC-MRD-v2.0
Current evidence boundary: This page and repository provide an evaluation architecture and public reference implementation. They do not establish universal validation of Robbie’s Razor, certify a model, guarantee lower resource use, or convert implementation into independent empirical support.
Authority Path
GC-MRD-v2.0 → Robbie’s Razor → Lab Evaluation Protocol → Versioned Benchmark → Bounded Result → Evidence State
Canonical definitions remain governed by the Grand Compression Master Reference Document . Repository code, benchmark tools, diagnostics, and examples remain implementation and evaluation layers.
1 · Scope & Definition
Robbie’s Razor Benchmarks are controlled evaluation structures for testing whether a declared system produces accepted results through more efficient, reusable, and stable reasoning under constraint.
The benchmarks do not ask only whether a system can solve a task. They also examine the path used to reach the result, the amount of expansion and correction required, whether validated structure is preserved for reuse, and the total cost contained within the declared system boundary.
RC-01 · Canonical Robbie’s Razor
“When competing explanations exist, prefer the model that follows compression → expression → memory → recursion.”
A benchmark result is not just a score. Its minimum reproducible unit is the complete declared evaluation record:
System Version + Baseline + Task Set + Constraints + Metrics + Quality Threshold + Failure Rule + Evidence State
| Layer | Function | Authority Boundary |
|---|---|---|
| Canonical Claim | Defines the proposition, scope, and terms being tested. | Governed by GC-MRD-v2.0 and the canonical claims register. |
| Evaluation Protocol | Declares the baseline, tasks, metrics, thresholds, and failure conditions. | Determines what the test may validly conclude. |
| Reference Implementation | Provides versioned code, configurations, diagnostics, examples, and machine-readable artifacts. | Implementation demonstrates a method, not empirical confirmation. |
| Evidence Record | Preserves results, limitations, uncertainty, replication status, and evidence state. | Remains bounded to the tested system and declared conditions. |
The public benchmark system is a versioned reference implementation and evaluation architecture. Individual benchmark claims must earn their own evidence states through preregistered tests and appropriate replication. The existence of the repository does not place the entire framework in the “Supported” state.
Repository boundary: The GitHub repository contains implementation, documentation, governance, diagnostic, test, and knowledge-architecture assets. Where repository language conflicts with the current GC-MRD-v2.0 canon, the current MRD governs.
2 · Canon, Implementation & Evidence
Benchmark integrity depends on keeping canonical definitions, evaluation procedures, implementation code, observed results, and evidence states separate. Each layer answers a different question and carries a different level of authority.
Governing authority: GC-MRD-v2.0 defines the current canon. Section 13 governs predictive compression, preserved reusable structure, evaluation design, reference-implementation boundaries, evidence states, and domain transfer. Provisional mathematical work remains provisional wherever identified, including Appendix Q.
| Layer | Role | Cannot Do |
|---|---|---|
| GC-MRD-v2.0 | Defines the current framework, canonical claims, scope boundaries, and evidence discipline. | Cannot make an empirical result true merely by defining it. |
| Canonical Claims Register | Identifies testable claims, their wording, scope, and current status. | Cannot replace an evaluation protocol or result record. |
| Lab Evaluation Protocol | Declares how the claim will be tested and what would count as success or failure. | Cannot change its decision threshold after results are known. |
| GitHub Implementation | Provides inspectable code, configurations, baselines, diagnostics, tests, and version history. | Cannot silently redefine the canon or establish validation by existing. |
| Result Artifact | Records observations, exclusions, uncertainty, limitations, and calculated deltas. | Cannot support conclusions outside its declared test boundary. |
| Evidence State | Communicates the status of a bounded claim after evaluation. | Cannot be inherited automatically by another domain, model, or version. |
RC-19
Declare predictions, baselines, metrics, thresholds, failure conditions, and exclusion rules before results are interpreted.
RC-20
Include the relevant cost of producing, correcting, validating, preserving, retrieving, and reusing an accepted result.
RC-21
Implementing Robbie’s Razor, RKCA, a metric, or a benchmark does not demonstrate that it improves performance.
RC-22
A result supported for one system, workload, configuration, or domain requires revalidation before transfer elsewhere.
“Documented,” “Calculated,” “Inferred,” “Proposed,” and “Unknown” may be used as source or finding labels. They do not replace the formal GC-MRD-v2.0 evidence-state ladder.
Interactive-tool boundary: An auditor, chatbot, Gem, prompt, dashboard, or visualization may help users explore the framework. Its output is not a benchmark result unless it follows the declared protocol and produces a versioned evidence record.
3 · What Gets Measured
The benchmark model begins with task quality and then measures how an accepted result was produced. A lower-cost output is not more efficient if it is incomplete, incorrect, unsafe, or requires greater downstream repair.
Quality-gate rule: Compare computational efficiency only after the baseline and Razor-guided configurations are evaluated against the same predeclared acceptance threshold.
| Dimension | Benchmark Question | Example Observables |
|---|---|---|
| Accepted Outcome | Did the result meet the same declared quality and completion threshold? | Accuracy, completeness, safety, usefulness, human correction |
| Compression | How much relevant structure was preserved while unnecessary expansion was reduced? | Context size, tokens, retrieval volume, active parameters where disclosed |
| Expression | Was the compressed structure expressed as a correct, usable result? | Task success, output quality, tool execution, latency, format compliance |
| Memory | Was validated structure preserved with enough identity, provenance, constraints, and relationships for later reuse? | Cache hits, retained state, retrieval accuracy, version continuity, memory growth |
| Recursion | Did preserved structure improve the next reasoning cycle without destabilizing the system? | Reuse rate, re-derivation, retries, corrections, backtracking, long-horizon completion |
| Total Cost | What did the complete accepted task cost across all declared layers? | Compute, tools, retrieval, storage, network, validation, human review, energy |
Accepted Tasks ÷ Attempted Tasks
Total Measured Cost ÷ Accepted Tasks
Valid Reuse Events ÷ Eligible Reuse Events
Razor-Guided Result − Baseline Result
Status: These are operational reporting templates, not new canonical equations. Each result must declare its units, calculation method, uncertainty, exclusions, and whether a lower or higher value represents improvement.
Use system telemetry when available: tokens, tool calls, cache events, retries, latency, memory, utilization, energy, and accepted-task outcomes.
When direct access is unavailable, use declared observable proxies and label the limitation. A proxy must not be reported as hidden internal telemetry.
Reasoning privacy: A benchmark does not need private chain-of-thought disclosure. It can evaluate accepted outcomes, tool traces, retries, state transitions, memory use, resource consumption, and other observable behavior without requiring hidden reasoning content.
4 · System Architecture
The benchmark system is organized as a layered architecture. Canon defines what may be tested, the protocol defines how to test it, the repository provides the implementation, and result records preserve what was observed.
Layer-separation rule: No layer may impersonate another. A webpage does not replace the canon, code does not replace a protocol, an interactive tool does not replace a controlled run, and a benchmark score does not replace an evidence record.
Layer 1
GC-MRD-v2.0 and Robbie’s Razor define the claims, terms, invariants, boundaries, and evidence discipline.
Layer 2
The Lab Evaluation Protocol fixes the prediction, baseline, task set, metrics, quality threshold, and failure rules.
Layer 3
Versioned baselines establish the comparison condition against which a Razor-guided intervention is measured.
Layer 4
Benchmark code, task fixtures, configurations, and runners implement the declared comparison under controlled conditions.
Layer 5
Metrics calculate declared outcomes while diagnostics identify anomalies, regressions, missing telemetry, and possible failure conditions.
Layer 6
Versioned artifacts preserve observations, calculations, exclusions, uncertainty, provenance, review, replication, and evidence state.
| Repository Area | Benchmark Function |
|---|---|
baselines/ |
Comparison conditions and reference behavior |
benchmarks/ |
Benchmark implementations, tasks, and evaluation surfaces |
razor_metrics/ |
Metric definitions and calculation logic |
diagnostics/ |
Diagnostic checks, interpretation support, and anomaly detection |
src/, tools/ |
Shared source code, utilities, and execution support |
tests/, .github/workflows/ |
Automated verification and repository-level quality controls |
governance/, docs/ |
Governance records, specifications, instructions, and supporting documentation |
notebooks/ |
Exploratory analysis and reproducible research workflows |
Canonical Claim → Preregistered Protocol → Versioned Baseline → Benchmark Run → Diagnostic Review → Result Artifact → Evidence State → Replication
5 · Test Families
A benchmark family groups tests that ask a related question. Individual tests may use different models, tasks, tools, memory systems, or infrastructure, but results remain comparable only when their protocols and measurement boundaries match.
Family-status rule: The presence of a benchmark family on this page does not mean every test in that family is implemented or empirically supported. Each test must identify whether it is implemented, proposed, testing, archived, or independently replicated.
Core Family
Tests whether the system completes the task correctly, safely, and usefully before resource-efficiency claims are considered.
Implemented Metrics
Measures task-normalized token use, structural redundancy, context expansion, and other declared resource costs at equal or better quality.
Implemented Harness
Tests whether preserved state, cached structure, or validated results reduce redundant recomputation without introducing stale or incorrect reuse.
Implemented Metrics
Tracks retries, abandoned paths, contradictions, corrections, repeated tool calls, and other observable signs of avoidable rework.
Repository Fixture
Compares retrieval of preserved structure with regeneration from raw inputs, including correctness, provenance, latency, and total task cost.
Protocol-Dependent
Tests whether performance and coherence remain stable as tasks become longer, more branching, stateful, or constrained.
Implemented Metric
Measures whether tool calls, retrieval operations, external actions, and validation steps contribute proportionate value to accepted task completion.
Telemetry-Dependent
Extends task measurement into compute utilization, energy, cooling, water, emissions, networking, storage, and hardware allocation where access permits.
| Visible Asset | Current Function | Boundary |
|---|---|---|
benchmark_memory_gate_savings.py |
Memory-hit, token-reuse, and redundant-recomputation demonstration | Reference harness; not universal validation |
evaluator.py |
Evaluates structured outputs against declared metrics | Depends on the supplied cases and scoring rules |
cases/ |
Structured evaluation inputs and fixtures | Coverage does not imply domain representativeness |
refractive-truth/ |
Retrieval-versus-recomputation evaluation surface | Interpret only within its declared fixtures |
sample_outputs.csv / .json |
Example structured result formats | Examples are not production evidence |
Version note: Repository files may preserve historical MRD references for provenance. Current interpretation must resolve through GC-MRD-v2.0.
6 · Matched Comparison Design
A Robbie’s Razor benchmark is comparative. It must measure a declared intervention against a credible baseline under conditions that isolate the feature being tested.
Baseline rule: “Conventional reasoning” is not a sufficient baseline description. The evaluator must record the exact system, version, configuration, prompt, memory state, tools, limits, and scoring method.
Compares the Razor-guided configuration with the same system operating under its standard configuration.
Removes one component—such as memory, retrieval, compression control, or reuse—to isolate its contribution.
Disables eligible preserved-state reuse while holding the model, task, tools, and quality standard constant.
Requires reconstruction from the allowed raw inputs instead of retrieving preserved reusable structure.
Compares two declared versions while separating model improvements from the Razor-guided intervention.
Uses another documented system only when access, task conditions, quality standards, and measurement boundaries can be matched.
| Control | Required Record | Risk if Unmatched |
|---|---|---|
| Task Set | Same tasks, sampling rules, difficulty distribution, and exclusions | One condition receives easier work |
| System Version | Model, software, API, dependencies, and configuration | Uncontrolled system changes explain the delta |
| Prompt & Instructions | Exact prompts, system instructions, templates, and formatting | Prompt quality is confused with architecture |
| Tools & Retrieval | Available tools, permissions, indexes, documents, and network access | One system receives outside information or capabilities |
| Memory State | Starting memory, cache state, retention policy, and eligible reuse | Warm and cold starts are mixed |
| Budgets & Limits | Token, time, tool-call, retry, compute, and monetary limits | One system receives more resources |
| Execution Environment | Hardware, region, load, network, runtime, and measurement period | Infrastructure conditions distort latency or energy |
| Quality Judge | Rubric, ground truth, human review, automated judge, and tie handling | Scoring bias creates the apparent result |
Invalid comparison: A baseline chosen, altered, or discarded after results are observed cannot support the preregistered claim. It may generate a new hypothesis, but that hypothesis requires a new evaluation.
7 · Required Metric Bundle
No single score can establish Robbie’s Razor performance. A valid evaluation combines outcome quality, expansion, rework, memory, reuse, latency, tools, and total cost.
Interpretation rule: Every metric must declare its unit, direction of improvement, aggregation method, uncertainty, and relationship to the accepted-task threshold.
| Metric | Operational Meaning | Typical Direction | Required Qualification |
|---|---|---|---|
| Accepted-Task Rate | Accepted tasks divided by attempted tasks | Higher | Acceptance rubric and judge |
| TPCA | Tokens per correct or accepted answer | Lower at equal quality | Token categories and counting method |
| Backtracking Frequency | Observable corrections, abandoned paths, retries, or repeated operations per task | Lower when unnecessary | Backtracking-event definition |
| Contradiction / Drift Rate | Material inconsistency with prior accepted state or task constraints | Lower | Contradiction taxonomy and severity |
| Memory-Hit Ratio | Valid memory hits divided by eligible memory lookups | Context-dependent | Eligibility and validity rules |
| Valid Reuse Rate | Correct reuse of preserved structure divided by eligible reuse opportunities | Higher without stale reuse | Identity, provenance, and version checks |
| Re-Derivation Rate | Previously available valid structure regenerated instead of reused | Lower when reuse is appropriate | Proof that reusable structure was available |
| Tool-Call Efficiency | Useful tool activity relative to accepted task completion | Task-dependent | Useful, failed, and repeated call definitions |
| Latency per Accepted Task | End-to-end elapsed time divided by accepted tasks | Lower within quality boundary | Network, queue, and tool-time treatment |
| Total Cost per Accepted Task | Declared computation, tools, storage, validation, and other cost divided by accepted tasks | Lower at equal or better quality | Complete system boundary and units |
Repeated structure that does not contribute to the accepted result.
Change in active context or retained state across task depth.
Relevant validated structures divided by retrieved structures.
Resources required to repair rejected or unstable outputs.
Reviewer time and interventions required per accepted task.
Maximum declared memory or cache consumption during execution.
Lower token use can hide lower quality. Higher memory reuse can spread stale information. Fewer tool calls can reflect missing verification. Faster completion can increase correction burden. The bundle must be interpreted together.
8 · Preserved Reusable Structure
Memory is not simply stored text, accumulated context, or a larger database. For benchmark purposes, memory becomes valuable when validated structure can be retrieved accurately and reused without losing identity, provenance, relationships, constraints, or version state.
RC-18 preservation boundary: Reusable structure must retain enough identity, relationships, provenance, constraints, version state, and retrieval pathways to support valid future use.
Information exists somewhere in memory, a cache, database, registry, context window, file, or model state.
The correct structure can be located when needed without excessive search, irrelevant context, or identity confusion.
Retrieved structure remains correct, current, appropriately scoped, and compatible with the active task.
Preserved structure reduces later work while maintaining or improving accepted-task quality.
| Condition | Purpose | Primary Risk Tested |
|---|---|---|
| Cold Start | Measures task completion without eligible preserved state | Hidden warm-cache advantage |
| Warm Start | Measures performance when validated reusable structure is available | Failure to exploit valid memory |
| Changed-State Test | Changes facts, constraints, relationships, or versions after storage | Stale or incorrectly inherited structure |
| Near-Match Test | Introduces similar but non-equivalent identities or tasks | False reuse caused by semantic resemblance |
| Invalidation Test | Withdraws or supersedes previously valid structure | Failure to retire invalid memory |
| Scale Test | Increases retained structures and retrieval choices | Search overhead, collision, and context bloat |
Correct memory hits divided by eligible lookup events.
Incorrect reuse events divided by attempted reuse events.
Reuse of superseded or invalidated structure.
Eligible preserved structure unnecessarily regenerated.
Time, tokens, compute, or calls required to find and validate memory.
Resources required to preserve, index, version, and invalidate structure.
Operational Reporting Template
Net Reuse Benefit = Avoided Recomputation − Storage − Retrieval − Validation − Invalidation Overhead
Status: This is an operational accounting template, not a canonical equation. Each study must define its cost units and attribution rules.
Memory failure: A high hit rate is not evidence of effective memory when retrieved structures are stale, irrelevant, incorrectly scoped, or more expensive to validate than to reconstruct.
9 · Stability Under Constraint
Recursive stability measures whether a system can preserve useful coherence across repeated reasoning, retrieval, action, feedback, correction, and memory-update cycles while operating under declared constraints.
Stability does not mean repeating the same answer. A stable system must update when evidence changes, recover from error, preserve valid structure, and avoid allowing correction demand or context growth to overwhelm useful task progress.
Benchmark question: As task depth and constraint increase, does the system maintain accepted-result quality with proportionate growth in correction, memory, computation, and governance overhead?
Number of dependent steps or state transitions required.
Alternative paths, choices, tools, or hypotheses introduced.
Maximum active context allowed during execution.
Storage, cache, retained state, and retrieval limits.
Number, cost, latency, and permissions of external actions.
Wall-clock, token, processing, or monetary limits.
Updates that require revision of previously valid structure.
Controlled faults used to test detection and recovery.
| Measurement | What It Reveals | Failure Signal |
|---|---|---|
| Quality Retention | Accepted-task performance as depth increases | Sharp quality decline under modest added depth |
| Drift Rate | Loss of alignment with accepted state or task constraints | Increasing unsupported deviation |
| Correction Demand | Repair operations required per state transition | Correction grows faster than useful progress |
| Resource-Growth Slope | How tokens, memory, tools, and time grow with task depth | Disproportionate expansion without quality gain |
| Recovery Rate | Successful return to valid state after controlled error | Error propagation or repeated failed recovery |
| Completion Boundary | Highest declared constraint level meeting the quality threshold | Premature failure or unstable completion |
Scope boundary: A system that remains stable on one benchmark has demonstrated stability only for the tested version, workload, constraints, tools, and evaluation period. RC-22 requires revalidation before domain transfer.
10 · Physical Resource Accounting
Computational activity occurs through physical infrastructure. A benchmark may extend into energy and environmental accounting only when the study can establish a defensible connection between accepted tasks and the measured resources consumed.
No-conversion shortcut: Token savings must not be translated directly into energy, water, emissions, or ecological benefit without workload-specific hardware, utilization, facility, time, and attribution data.
| Evidence Level | What May Be Reported | What Remains Unresolved |
|---|---|---|
| Token Measurement | Task-normalized token delta under the declared tokenizer | Compute, energy, cooling, water, and emissions |
| Compute Telemetry | Measured runtime, utilization, memory, processor activity, or compute allocation | Complete facility and lifecycle impact |
| IT Energy | Measured electricity used by declared computing equipment | Cooling, conversion loss, and facility overhead |
| Facility Energy | Attributed IT and facility electricity under a declared method | Water, grid emissions, construction, and hardware lifecycle |
| Operational Environment | Energy, cooling, water, and emissions estimates for the measured period | Embodied and supply-chain impact unless separately measured |
| Lifecycle Boundary | Operational and embodied impact within a declared allocation model | Effects outside the selected lifecycle boundary |
Measured energy per accepted task.
Total energy used during the declared evaluation period.
Site and supply-chain water reported separately where possible.
Grid source, time, region, factors, and uncertainty.
Embodied impact allocated across utilization and service life.
Measurement error, estimation ranges, and missing boundaries.
Task-Normalized Reporting
Environmental Impact per Accepted Task = Total Attributed Impact ÷ Accepted Tasks
Status: This is an operational reporting template. It does not establish one universal conversion factor across models, processors, data centers, regions, or time periods.
Joules per Coherent Transition cannot be calculated from tokens alone. A study using JCT must define the coherent transition, directly measure or defensibly attribute energy to it, apply the same definition to both conditions, and report uncertainty and excluded infrastructure.
Rebound boundary: Lower impact per task does not prove lower total impact. If reduced cost increases task volume, absolute energy, water, hardware, or emissions may still rise.
11 · Versioned Implementation
The public GitHub repository is the versioned implementation layer for benchmark code, baselines, metrics, diagnostics, governance, documentation, machine-readable examples, and supporting tests.
Current Immutable Release
Published August 16, 2026, this release establishes a stable, citable implementation record aligned with GC-MRD-v2.0. It preserves the repository state used for inspection, evaluation, reproducibility, and comparison with later benchmark versions.
Release Version
0.2.0
Git Tag
benchmarks-v0.2.0
Governing Authority
GC-MRD-v2.0
Publication State
Public & Immutable
Repository boundary: A file’s presence proves that the artifact was published in the repository. A versioned release preserves an identifiable implementation state, but neither publication nor immutability proves that its claims were empirically validated, independently replicated, or incorporated into the governing canon.
| Repository Area | Primary Content | Evidence Meaning |
|---|---|---|
benchmarks/
|
Executable harnesses, evaluators, cases, samples, and utilities | Implemented evaluation surface |
baselines/
|
Reference conditions and comparison behavior | Declared comparison layer |
razor_metrics/
|
Metric definitions, calculations, and reporting logic | Implementation of declared measures |
diagnostics/
|
Diagnostic checks and interpretive support | Supporting analysis, not an automatic verdict |
src/ and tools/
|
Shared implementation code and utilities | Software infrastructure |
tests/
|
Tests for repository code and expected behavior | Software verification, not framework validation |
notebooks/
|
Exploratory and reproducible analyses | Exploratory unless promoted through protocol |
governance/
|
Interpretive scope, attribution, and non-claim boundaries | Governance context, not performance evidence |
docs/
|
Specifications, empirical notes, examples, JSON-LD, skills, registries, and knowledge architecture | Mixed assets requiring individual classification |
.github/workflows/
|
Automated repository checks and workflows | Continuous software-quality controls |
CITATION.cff
|
Version, authorship, publication date, canonical page, repository, and release references | Machine-readable citation and provenance record |
| GitHub Releases | Immutable tagged snapshots of published benchmark versions | Stable implementation identity, not empirical validation |
Every benchmark-related artifact should identify its role so readers, researchers, and machine agents do not confuse examples, implementations, and evidence-bearing results.
Version-resolution rule: Historical files may retain earlier MRD identifiers and terminology for provenance. When an older repository artifact conflicts with GC-MRD-v2.0, the current MRD governs unless the evaluation explicitly studies the historical version. Published releases remain preserved as historical implementation records and must not silently redefine the governing canon.
12 · Evaluation Workflow
A repository demonstration can confirm that code executes. A formal benchmark requires additional controls: a preregistered claim, pinned versions, a matched baseline, repeated runs, complete result records, and a declared decision rule.
Reproducibility rule: Preserve enough information for another evaluator to reconstruct the system, task set, intervention, baseline, metrics, execution environment, and result calculation.
Identify the exact canonical or applied claim being tested and its current evidence state.
Record the prediction, baseline, metrics, quality threshold, expected direction, exclusions, and failure conditions.
Record the repository commit, MRD version, protocol version, model, software, dependencies, configuration, and task data.
Check task identity, ground truth, difficulty, contamination, exclusions, expected format, and representativeness.
Execute the registered comparison condition without modifying its configuration after results are observed.
Apply only the declared Razor-guided intervention while holding all registered controls constant.
Apply the same acceptance rubric, metrics, judge, missing-data rules, and uncertainty calculations.
Check for regressions, data leakage, unmatched conditions, execution failures, stale memory, judge disagreement, and protocol deviations.
Publish the human-readable report, machine-readable record, hashes, limitations, evidence state, and replication status.
From the repository’s benchmarks/ directory, the current public quick-start surface includes:
python benchmark_memory_gate_savings.py
python evaluator.py --outputs sample_outputs.json
Demonstration boundary: Successful execution shows that the referenced code ran in the local environment. It does not, by itself, create a formal evidence result.
| Evaluation identity | Study ID, evaluator, date, protocol, and governing MRD version |
| Implementation identity | Repository URL, branch, commit SHA, dependencies, and configuration |
| System identity | Model, version, tools, memory, retrieval, hardware, and runtime |
| Evaluation design | Claim, baseline, intervention, tasks, controls, metrics, and thresholds |
| Integrity controls | Randomization, seeds, exclusions, hashes, missing data, and stop rules |
13 · Reporting & Evidence States
A benchmark produces observations and calculated deltas. Evidence-state assignment occurs only after the result is checked against the preregistered threshold, limitations, uncertainty, protocol compliance, and replication requirement.
Result rule: Positive, null, and negative findings must be preserved. A result that challenges the prediction is part of the evidence architecture, not a repository failure.
Explains the claim, method, results, limitations, uncertainty, deviations, interpretation, and evidence state.
Preserves stable identifiers, versions, metrics, units, values, hashes, provenance, and status for automated retrieval.
Provides the shareable observations needed to verify calculations, subject to privacy, security, and licensing limits.
Records the environment, configuration, dependency versions, repository commit, task hashes, and execution time.
| Field Group | Required Fields |
|---|---|
| Identity | Result ID, study ID, title, evaluator, dates, version, and status |
| Authority | MRD version, claim ID, protocol version, repository, commit SHA |
| Systems | Baseline identity, intervention identity, models, tools, memory, hardware, configuration |
| Measurements | Metric names, units, raw values, aggregation, deltas, uncertainty, sample size |
| Integrity | Task hashes, exclusions, missing data, deviations, judge method, validation checks |
| Interpretation | Threshold decision, limitations, supported scope, failure conditions, evidence state |
| Replication | Replication status, independent evaluator, deviations, and linked replication records |
| State | Benchmark Meaning |
|---|---|
| Proposed | The claim or benchmark is defined but has not begun a registered test. |
| Testing | Evaluation is active under a declared protocol. |
| Provisionally Supported | Initial results meet the declared threshold within scope but await stronger replication. |
| Supported | Evidence satisfies the declared support and replication requirements within the tested scope. |
| Challenged | Material results conflict with the prediction or a prior supported finding. |
| Inconclusive | The evaluation cannot resolve the claim because of uncertainty, insufficient power, missing data, or conflicting results. |
| Retired | The claim or benchmark is withdrawn, superseded, or no longer maintained. |
“Under the declared task set, model version, memory configuration, and quality threshold, the Razor-guided condition reduced measured re-derivation relative to the registered baseline.”
“The benchmark proves Robbie’s Razor makes every AI system more efficient.”
Authorship separation: Robbie George remains the attributed creator of Robbie’s Razor and the Grand Compression framework. Benchmark evaluators, implementers, reviewers, and replication teams must also be credited for their distinct contributions.
14 · Falsifiability & Integrity Control
A benchmark can fail methodologically even when its code runs correctly. The evaluation becomes invalid when its controls, evidence boundary, scoring rules, or reporting practices cannot support the conclusion being asserted.
Falsifiability rule: Before testing begins, the evaluator must state which observations would support, challenge, narrow, or leave the claim unresolved.
The test is designed to confirm a preferred outcome rather than expose the claim to possible failure.
The comparison condition is selected, modified, or discarded after its results are observed.
The systems receive different tasks, tools, memory, time, prompts, budgets, or quality thresholds without declaring the difference.
Lower token use, latency, or tool activity is called efficient even though fewer tasks meet the acceptance threshold.
Visible tokens are treated as total computation while retries, retrieval, tools, validation, memory, and correction are excluded.
A higher memory-hit rate is treated as success without checking relevance, provenance, version, or stale reuse.
Failed runs, timeouts, rejected answers, missing telemetry, or unfavorable tasks are removed without a preregistered rule.
Example cases, sample outputs, notebooks, or demonstration runs are presented as formal empirical findings.
The existence of code, an auditor, a metric, a registry, or a Razor-guided configuration is presented as support for the tested claim.
Token savings are converted into energy, water, emissions, or environmental benefit without the required physical telemetry.
A finding from one model, task set, domain, configuration, or time period is generalized elsewhere.
Only favorable runs are retained, creating a distorted evidence record.
Multiple metrics are combined without declaring normalization, weighting, missing-data treatment, or sensitivity.
Historical formulas, superseded claims, or provisional concepts are presented as current GC-MRD-v2.0 authority.
| Observed Condition | Required Action |
|---|---|
| Material system, model, or protocol change during execution | Stop and restart under a new versioned record |
| Telemetry loss affects a required metric | Stop or mark inconclusive according to the preregistered rule |
| Small number of individual execution failures covered by the protocol | Continue and retain failures in the result record |
| Unexpected result with valid controls | Continue and preserve as a potentially challenging finding |
| Evidence of task contamination or scoring leakage | Invalidate affected runs and rebuild the task set |
Critical distinction: A benchmark-method failure means the evaluation cannot support a decision. A valid negative result means the tested claim was challenged within the declared scope. These are not the same outcome.
15 · Reproducibility & Change Control
Replication tests whether a result survives repeated execution, independent implementation, or transfer into a newly declared environment. Versioning ensures that evaluators can identify precisely which canon, protocol, implementation, fixtures, and system configuration were tested.
Immutable-record rule: Published benchmark releases and result records should be superseded by new versions, not silently overwritten. Historical records must preserve the governing canon, protocol, implementation, configuration, limitations, and evidence state that applied when the evaluation was conducted.
Current Implementation Anchor
The August 16, 2026 release provides the current stable implementation anchor for the benchmark suite. Its immutable tag, citation record, source archive, and GC-MRD-v2.0 alignment allow future evaluations to identify the exact published implementation they used.
| Governing MRD | GC-MRD-v2.0 |
| Benchmark Version | 0.2.0 |
| Release Tag |
benchmarks-v0.2.0
|
| Release Date | August 16, 2026 |
| Citation Record |
CITATION.cff
|
Evidence boundary: Version v0.2.0 identifies a published implementation state. It does not itself constitute a benchmark result, independent replication, model certification, or upward change in evidence state.
Level 1
The same release or commit, fixtures, versions, environment, and calculation reproduce the reported result within the declared tolerance.
Level 2
A separate evaluator applies the registered protocol without relying on the original team’s undocumented decisions.
Level 3
The claim is tested in a different model, domain, workload, or infrastructure boundary under a new preregistered evaluation.
RC-22 boundary: Domain revalidation is a new test, not an automatic extension of the original result.
MRD Version + Claim Version + Protocol Version + Benchmark Release or Commit + Fixture Hash + System Configuration + Environment
| Artifact | Purpose |
|---|---|
| Commit SHA or Immutable Release Tag | Pins the implementation used in the evaluation |
| Citation Record | Preserves the published version, authorship, date, canonical page, repository, and release identity |
| Task and Data Hashes | Confirms fixture identity and detects silent alteration |
| Environment Manifest | Records software, dependencies, hardware, region, and runtime |
| Change Log or Release Notes | Explains what changed and whether prior results remain comparable |
| Supersession Link | Connects historical and current artifacts without erasing provenance |
| Replication Record | Documents whether another evaluator reproduced, narrowed, or challenged the result |
Evidence-state control: A new release does not automatically strengthen a claim. New replication may strengthen, narrow, challenge, or leave a claim inconclusive. Evidence states must follow the complete evaluation record and must not move upward merely because a newer version was published.
16 · Authority, Release & Retrieval Map
This benchmark page connects the governing canon to evaluation protocols, versioned implementations, immutable releases, bounded result artifacts, and machine-readable discovery without allowing one layer to replace another.
Current authority: GC-MRD-v2.0 governs present definitions, claims, boundaries, and evidence discipline. Earlier MRD versions remain part of the historical record but are superseded wherever they conflict with v2.0.
Current Implementation Release
Published August 16, 2026, under the immutable tag benchmarks-v0.2.0. This release is the current stable implementation reference for citation, inspection, reproduction, and version comparison.
Boundary: The release establishes implementation identity and provenance. It does not independently establish empirical support, replication, certification, or a higher evidence state.
Governing Canon
Canonical Law
Evaluation Protocols
Applied Evaluation
Versioned Implementation
Governing Canon → Evaluation Protocol → Immutable Implementation Release → Bounded Result → Evidence State → Machine Retrieval
Each layer serves a distinct function. GC-MRD-v2.0 governs the canon, the protocol defines the test, the immutable release identifies the implementation, the result record preserves what occurred, the evidence state bounds the conclusion, and the machine layer supports structured discovery and delivery.
17 · Questions & Boundaries
These answers clarify what the benchmark system measures, how results should be interpreted, and where implementation ends and empirical evidence begins.
Robbie’s Razor Benchmarks are controlled evaluation structures for testing bounded claims about compression, expression, memory, recursion, preserved reusable structure, and total computational cost. They compare declared systems under matched conditions and shared quality thresholds.
No. Individual benchmarks can support, challenge, narrow, or leave unresolved specific claims within a declared test boundary. The existence of benchmark code or a favorable result does not universally validate Robbie’s Razor or the complete Grand Compression Cosmology.
The current governing authority is the Grand Compression Master Reference Document, GC-MRD-v2.0. Earlier MRD versions remain part of the historical record but are superseded wherever they conflict with v2.0.
The current public benchmark surface includes a memory-gate savings harness, structured case evaluation, retrieval-versus-recomputation fixtures, sample result formats, and metrics for tokens per correct answer, structural redundancy, memory reuse, drift, contradiction, backtracking, and tool-call efficiency.
TPCA means Tokens Per Correct Answer. It divides measured tokens by correct or accepted answers. The result is meaningful only when the tokenizer, token categories, task set, acceptance rule, and treatment of failed runs are declared.
No. Lower token use may reflect useful compression, but it may also reflect incomplete reasoning, missing verification, or lower-quality output. Token efficiency must be evaluated after both conditions are scored against the same acceptance threshold.
Valid reuse occurs when preserved structure remains correctly identified, appropriately scoped, current, traceable to its provenance, compatible with active constraints, and useful in reducing later work. Reuse of stale or incorrect memory is a failure, not an efficiency gain.
No. Benchmarks can evaluate accepted outcomes, tokens, retries, tool calls, retrieval, state transitions, memory behavior, latency, resource use, and correction without requiring private hidden reasoning content.
No. Energy claims require workload-specific information about hardware, runtime, utilization, memory, networking, facility overhead, and attribution. Token savings may motivate physical measurement, but they do not establish energy, water, emissions, or environmental savings by themselves.
No. Under RC-21, implementation and validation are separate. Code, prompts, auditors, registries, memory systems, and Razor-guided configurations must still demonstrate measurable effects against a preregistered baseline.
GC-MRD-v2.0 uses seven evidence states: Proposed, Testing, Provisionally Supported, Supported, Challenged, Inconclusive, and Retired. Every state remains bounded to the identified claim, system, task set, configuration, and evaluation conditions.
A team can inspect the reference implementation, select a bounded claim, preregister a matched baseline, pin the repository and system versions, validate the task fixtures, run repeated evaluations, package the results, and publish a replication record with appropriate limitations.
No. The repository is not a vendor leaderboard or hardware-equivalence claim. Company, model, or infrastructure comparisons require separate protocols with matched workloads, equal quality standards, direct measurement, and clearly bounded conclusions.
Evaluation hub and versioned reference-implementation guide. This page does not assign universal validation, certify individual systems, or guarantee computational or environmental savings.
18 · Authorship & Stewardship
Robbie George · Nature photographer, author, and framework originator
Robbie George is a National Geographic–published nature photographer and the creator of Robbie’s Razor and the Grand Compression Cosmology. His photographic work has also been displayed at the Smithsonian National Museum of Natural History.
His framework developed from long-term observation of natural systems: how living structures retain useful information, reuse established relationships, adapt under constraint, and regenerate complexity without reconstructing every dependency from the beginning.
Robbie’s Razor expresses that architecture through the sequence compression → expression → memory → recursion. The benchmark system extends the sequence into a falsifiable evaluation layer with declared baselines, measurable outcomes, failure conditions, versioned implementation, and evidence-state governance.
George maintains the public framework, benchmark repository, machine-readable knowledge architecture, and Naturepedia reference implementation. These assets establish authorship and implementation provenance; they do not replace controlled testing or independent replication.
“When competing explanations exist, prefer the model that follows compression → expression → memory → recursion.”
— Robbie’s Razor, RC-01
Robbie George’s authorship establishes the provenance of Robbie’s Razor and the Grand Compression framework. Empirical support must still come from preregistered evaluation, transparent result records, appropriate evidence states, falsification, and scope-specific replication.
Page classification: Benchmark evaluation hub · Reference-implementation guide · GC-MRD-v2.0 governed · No universal validation or certification implied.
This is only visible to you because you are logged in and are authorized to manage this website. This message is not visible to other website visitors.
Click on any Image to continue
This means you can use the camera on your phone or tablet and superimpose any piece of nature art onto a wall inside of your home or business.
To use this feature, Just look for the "Live Preview AR" button when viewing any piece of nature art on this website!
Join the collector list for your first-order discount, new wildlife releases, and occasional field notes.
