Section 19: Compiler Testing and Acceptance Strategy¶

TECHSPEC-19 — Section 19 · Version 0.1 · Status: Initial Draft

Status: draft. This section is an Initial Draft (Section 57 is a Working Register). It describes intended or proposed behavior, not a finalized, implemented language rule. Do not treat examples here as normative.

1. Compiler Testing Philosophy¶

The compiler is responsible for enforcing the language's reliability guarantees. Its correctness must therefore be tested independently of the programs it compiles.

A compiler must not be considered trustworthy merely because it successfully compiles its own examples.

Testing must examine both sides of its behavior:

• Valid programs must be accepted and translated correctly.
• Invalid programs must be rejected for the correct reasons.

2. Compiler Test Categories¶

The compiler test suite must include:

• LEXER: Verify token recognition and invalid character handling.
• PARSER: Verify syntax recognition and rejection.
• TYPE_CHECKER: Verify valid and invalid type operations.
• SEMANTIC: Verify declarations, scopes, and language rules.
• CONTRACT: Verify preconditions, postconditions, and invariants.
• EFFECT: Verify effect declarations and propagation.
• CODEGEN: Verify generated code preserves source-language semantics.
• RUNTIME: Verify runtime checks and failure behavior.
• DIAGNOSTIC: Verify error identification and source locations.
• VERIFICATION: Verify that reports accurately represent available evidence.

3. Positive and Negative Tests¶

Every implemented language feature must have positive and negative tests.

Positive tests must demonstrate that valid programs are accepted.

Negative tests must demonstrate that invalid programs are rejected.

Examples:

A feature must not be considered complete until both its valid and invalid cases are tested.

4. Code Generation Testing¶

Generated Python must preserve the behavior defined by the source language.

Rules: - Source-language semantics take precedence over Python behavior. - Generated code must not bypass compiler-enforced restrictions. - Runtime checks must remain present when required. - Source-to-generated-code relationships must be traceable. - Generated programs must be tested against expected source-language behavior. - Differences between source semantics and target behavior must be reported.

5. Regression Testing¶

Every confirmed compiler defect must produce a regression test.

Rules: - A corrected defect must have a test that reproduces the original failure. - Regression tests must remain in the permanent test suite. - Changes to compiler components must trigger relevant regression tests. - Tests must not be silently removed or weakened. - Changes to expected behavior must be documented.

6. Fuzz and Property-Based Testing¶

The compiler should be tested with automatically generated source programs and inputs.

Testing should include: - Random valid programs within supported language features. - Random malformed source code. - Boundary values. - Deeply nested expressions and declarations. - Unusual combinations of supported features. - Large inputs intended to expose resource problems.

Rules: - Compiler crashes must be recorded as failures. - Invalid input must produce controlled diagnostics rather than uncontrolled termination. - Random seeds and failing inputs must be preserved. - Generated test cases must be reproducible where practical.

7. Mutation Testing¶

Mutation testing should evaluate whether the compiler's test suite detects deliberately introduced defects.

Examples: - Disable a type check. - Ignore a failed contract. - Allow an undeclared effect. - Remove a required runtime check. - Generate an incorrect return value. - Misclassify an unverified property as proven.

Rules: - Mutations must be documented. - The test suite should detect each relevant mutation. - Undetected mutations must be reported. - Mutation testing results must not be presented as proof that the compiler is correct.

8. Independent Verification¶

Critical compiler behavior should receive independent review or verification where practical.

Possible methods include: - Reviewing compiler logic against the formal language specification. - Testing components independently. - Comparing behavior against independently calculated expected results. - Using separate implementations of critical checks. - Formal verification of small, critical compiler components where feasible.

Independent checks must not simply repeat the same assumptions or implementation logic as the component being evaluated.

9. Compiler Acceptance Criteria¶

A compiler release must:

• Pass all mandatory compiler tests.
• Pass all required regression tests.
• Produce correct results for supported positive test cases.
• Reject required negative test cases.
• Preserve source-language behavior through code generation.
• Produce accurate diagnostics.
• Produce honest verification reports.
• Record known defects and limitations.
• Identify the exact compiler version tested.

Known defects affecting mandatory language guarantees must block release.

10. Compiler Test Integrity¶

Compiler tests must be protected from changes that conceal defects.

Rules: - Test changes must be recorded. - Removed tests must have documented justification. - Weakened assertions must be identified. - Expected results must not be changed solely to make a failing implementation pass. - Test suite modifications must trigger relevant test execution. - The compiler must not certify its own correctness solely through its own output.

11. Design Goal¶

The language must establish confidence in its compiler through repeatable testing, regression protection, independent checks, and transparent limitations.

The compiler's verification claims must be held to the same standard of evidence that the language requires of AI-generated programs.

Source and related sections¶

Source file: TECHSPEC-19.md (preserved in source-original/LANG/). See the specification index, the glossary, and the cross-reference index.