Section 11: Testing and Test Integrity¶

TECHSPEC-11 — Section 11 · Version 0.1 · Status: Initial Draft

Status: draft. This section is an Initial Draft (Section 57 is a Working Register). It describes intended or proposed behavior, not a finalized, implemented language rule. Do not treat examples here as normative.

1. Testing Philosophy¶

Testing must be built into the language and development workflow.

Tests must verify actual program behavior against defined expectations.

The language must distinguish between: - Tests that pass. - Tests that fail. - Tests that were not executed. - Properties that have been formally proven.

Passing tests must never be presented as proof of universal correctness.

2. Built-in Test Types¶

The initial testing system supports:

UNIT: Test individual functions and components.
INTEGRATION: Test interactions between modules and external systems.
BOUNDARY: Test minimum, maximum, empty, and invalid inputs.
PROPERTY: Test properties across generated inputs.
REGRESSION: Ensure previously corrected failures do not return.

Additional testing methods may be introduced as the language develops.

3. Test Declarations¶

Tests must be explicitly declared.

Example:

TEST "withdraw reduces account balance"

    LET account = Account(balance: 100.00)

    LET result = withdraw(account, 40.00)

    ASSERT result IS SUCCESS
    ASSERT result.balance == 60.00

END TEST

Rules: - Every test must identify the behavior it checks. - Assertions must have explicit expected conditions. - Test failures must identify the violated assertion. - Tests must be repeatable where their dependencies permit it. - Tests must not silently skip failed assertions.

4. Contract Testing¶

Function contracts must be tested when they cannot be fully established through static analysis or formal verification.

Tests should cover: - Preconditions. - Postconditions. - Object invariants. - Declared failure conditions. - State changes. - Declared side effects.

A passing contract test does not prove that the contract holds for every possible input.

5. Test Integrity¶

Tests must be protected against manipulation by the code-generating agent.

Rules: - An agent must not silently remove tests. - An agent must not weaken assertions to make an implementation pass. - An agent must not modify expected results without authorization. - Changes to tests must be recorded. - Changes to requirements must be distinguished from changes to implementation. - Test modifications must trigger re-execution of affected tests.

6. Test Independence¶

Where practical, expected results should be established independently of the implementation being tested.

Rules: - Tests must not simply reproduce the implementation's logic. - Test fixtures and expected values must be identifiable. - Shared assumptions between implementation and test must be reported where relevant. - Critical behavior should have independent test coverage.

7. Test Execution Records¶

Every test run must produce a record containing:

Test identifier.
Source version or hash.
Test framework version.
Execution environment.
Input data or random seed, where applicable.
Execution result.
Failure details.
Skipped or unexecuted tests.
Timestamp.

Test records must identify exactly which version of the program was tested.

8. Compiler and Runtime Responsibilities¶

The testing system must: - Execute declared tests. - Evaluate assertions. - Report failures with source locations. - Detect missing test execution. - Preserve test results. - Identify changes to tests. - Distinguish test evidence from formal verification evidence.

9. Design Goal¶

AI agents must not be able to manufacture confidence by weakening tests, hiding failures, or reporting incomplete test runs as successful.

Testing must provide honest, reproducible evidence of program behavior.

Source file: TECHSPEC-11.md (preserved in source-original/LANG/). See the specification index, the glossary, and the cross-reference index.