Insight: The Adaptive Schema

1. Main Objectives

There are two opposing main objectives governing the workflow:

• Preserve data fidelity over schema flexibility: Attempt to reconcile unexpected structure or structure with unexpected syntax with the original schema.
• Conservatively introduce probabilistic inference: Attempt to incorporate data with unexpected semantics into the original schema.

2. The Constitutional Prompt

• Describe the original, invariant business intent as the immutable truth
• Contain the original input/output schema definitions
  • Define syntactic rules
  • Define mandatory/optional structure
  • Define types, ranges, memberships of data
• Use several manually selected pairs of optimal input/output pairs as benchmark
• Described invariants are immutable to avoid a semantic drift
• The size of the benchmark is monotonically increasing, i.e., existing benchmarks are immutable but new ones can be added

3. The Input Delta Heuristic

• Validates against the original input schema definition
• Accumulate validation errors and provide them to the input delta prompt

4. The Input Delta Prompt

• Compare the current input with the benchmark while taking validation errors into account
• Classify the given delta according to the following categories:
  • Continuity (no delta)
    • Consequence: no adaptation necessary
  • Evolution (benign delta)
    • Resolvable syntactic deviation
    • Resolvable semantic deviation
      • Allowed categories:
        • Terminological equivalence (synonyms)
        • Taxonomic/hierarchical alignment (generalization/specialization)
          • Constraint: this is only allowed if the specific input implies the general category with near-100% probability (conservative inference)
        • Compositional transformation (split/merge)
          • Merge: concatenate forename surname into full name
          • Split: split 1.99€ into amount and currency
          • Justification: no data is invented but it is merely restructured based on semantic understanding of the components
        • Contextual derivation (logical implication)
          • Example: schema expects PaymentStatus, input is missing this field but contains a CancellationDate
          • Inference: PaymentStatus = Cancelled
          • Conservative limit: this must be strictly distinguished from missing data (which falls under corruption) by insisting on guaranteed implication
        • Semantic normalization
          • Example: schema expects ISO country code but input provides Germany
          • Conservative limit: only apply if there are no ambiguities
      • Explicitly excluded categories:
        • Probabilistic filling (hallucination)
          • Guessing a missing value based on likely patterns is forbidden
          • E.g., assuming country is Germany because the phone number looks German
        • Ambiguous resolution
          • E.g., 10/02/2025 may be either 10. February or 2. October
        • Intent modification
          • If the input is an "offer" and the schema expects an "invoice", this is a Domain Shift because the business intent differs
    • Consequence: adapt transformation logic/specification accordingly
  • Corruption (invalid delta)
    • Unresolvable syntactic deviation
    • Missing data
    • Ambiguous data
    • Contradictory data
    • Consequence: halt and request human review
  • Domain shift (invalid delta)
    • Deviating business entity
    • Consequence: halt and request human review

5. The Legislative Prompt

• if delta = 0, reuse the existing implementation
• Else if delta > 0, implement adapted specification
  • Apply implementation on current input (outputA)
  • Apply implementation on the entire suite of benchmark inputs (outputB suite)

6. The Output Delta Heuristic

• Validate outputA against the original output schema definition
• Validate outputB suite against the original output schema definition
• Accumulate validation errors and provide them to the output delta prompt

7. The Output Delta Prompt

• Evaluate outputA against the constitutional intent
• Compare outputB suite with the benchmark outputs while taking validation errors into account (Ensures strict backward compatibility)
• Verify whether outputA or outputB violates the constitution
  • On violation, halt and request human review

8. The Confidence Scoring Prompt

• Quantify the system's confidence that the output is semantically valid
  • Take all deltas and validation errors into account
  • Make a weighted average of the self-assessment and the deterministic results of heuristics
• The confidence decision gate
  • If confidence >= threshold: accept adaptation and append the new input/outputA pair to the benchmark
  • If confidence < threshold: halt and request human review