Skip to content

Section 22: Strings, Text Encoding, and Unicode Semantics

TECHSPEC-23 — Section 22 · Version 0.1 · Status: Initial Draft

Numbering note: this is file TECHSPEC-23.md but it contains Section 22 of the specification. File numbers and section numbers diverge from TECHSPEC-23 onward; neither has been renumbered. See the specification index and audit AUDIT-001 in reports/documentation-audit.md for details.

Status: draft. This section is an Initial Draft (Section 57 is a Working Register). It describes intended or proposed behavior, not a finalized, implemented language rule. Do not treat examples here as normative.

1. Text Philosophy

Text must have consistent meaning across supported platforms and compilation targets.

The language must define how strings are stored, compared, measured, modified, encoded, and transmitted.

It must not silently corrupt text or confuse character counts with byte counts.

2. String Type

String represents an immutable sequence of Unicode text.

Rules: - Strings must contain valid Unicode scalar values. - Strings must use UTF-8 as their standard external encoding. - Strings must not contain invalid Unicode sequences. - String values are immutable. - String concatenation must produce a new String. - Empty strings are valid. - String length must have a clearly defined measurement unit.

3. Character Type

Character represents exactly one Unicode scalar value.

Rules: - A Character cannot contain multiple Unicode scalar values. - Combining marks are separate scalar values unless represented as part of a string. - A visible symbol may consist of multiple scalar values. - Character validation must reject invalid scalar values. - Character operations must not assume that one scalar value always represents one displayed symbol.

4. String Length and Indexing

The language must distinguish between:

  • UTF-8 byte length.
  • Unicode scalar value count.
  • Grapheme cluster count.

These measurements are not interchangeable.

Rules: - String length must specify which measurement it returns. - Byte length must be explicitly requested. - Scalar indexing must not be confused with grapheme indexing. - Out-of-range access must produce an explicit failure. - String slicing must define its indexing unit. - The language must not silently interpret byte offsets as character positions.

5. Unicode Normalization

Visually identical text may have different underlying Unicode representations.

The language must support explicit Unicode normalization.

Initial supported forms: - NFC. - NFD. - NFKC. - NFKD.

Rules: - Strings must not be silently normalized during ordinary assignment. - Normalization must be an explicit operation. - Equality must compare the actual stored scalar sequence unless a normalized comparison is explicitly requested. - Security-sensitive comparisons must account for Unicode normalization and visually confusable characters where relevant. - Normalization must not be treated as a complete defense against Unicode-based security attacks.

6. String Operations

The initial language supports:

  • Concatenation.
  • Equality and comparison.
  • Searching.
  • Replacement.
  • Splitting.
  • Trimming.
  • Case conversion.
  • Substring extraction.
  • Explicit encoding and decoding.

Rules: - Operations must define their indexing behavior. - Invalid operations must return explicit failures where applicable. - Case conversion must follow documented Unicode rules. - Locale-dependent behavior must be explicit. - String operations must not silently discard invalid data.

7. Encoding and Decoding

UTF-8 is the standard external text encoding.

Other encodings may be supported through explicit conversion operations.

Rules: - Encoding and decoding must identify the source and destination encoding. - Invalid byte sequences must produce explicit failures unless a documented recovery policy is selected. - Replacement of invalid sequences must be explicit. - External text must not be assumed to be valid UTF-8 without validation. - Encoding failures must remain visible to the caller.

8. String Interpolation

String interpolation must be explicit.

Example:

text id="7x8k2v" LET name: String = "Adam" LET message: String = "Hello, {name}"

Rules: - Interpolated expressions must be type-checked. - Conversion to text must follow defined formatting rules. - Interpolation must not execute arbitrary code. - Sensitive values must not be inserted into logs or output without authorization. - Formatting behavior must be deterministic unless locale-specific formatting is explicitly requested.

9. Security Considerations

Text is a common source of security vulnerabilities.

The language and standard library must provide tools to address: - Invalid encodings. - Unicode normalization differences. - Confusable characters. - Injection attacks. - Unsafe string concatenation in queries. - Unexpected control characters. - Sensitive information exposure.

Rules: - External text must be treated as untrusted. - Validation must be appropriate to the intended use. - SQL and other structured queries must support parameterized operations. - HTML and other output contexts must use context-appropriate escaping. - Escaping must not be treated as universal input validation.

10. Compiler and Runtime Responsibilities

The compiler and runtime must: - Enforce String and Character rules. - Preserve Unicode values. - Provide defined length and indexing operations. - Validate encoding and decoding. - Enforce explicit normalization. - Generate consistent string behavior across compilation targets. - Prevent target-language defaults from silently changing source-language semantics.

11. Design Goal

AI agents must be able to manipulate text without confusing bytes, characters, and visible symbols.

String operations must have explicit behavior, predictable encoding, and clear failure handling across supported platforms.

Source file: TECHSPEC-23.md (preserved in source-original/LANG/). See the specification index, the glossary, and the cross-reference index.