# Ship expert heuristics as prompt content — NOT a raw data dump. # On SCOTUS, pasting the full codebook alone gained 0 points. Replacing # it with a short PRIORITIZED rule list gained +20 points. # Two properties matter: (a) priority order, and (b) meta-rules that # govern conflicts between numbered rules. DOMAIN_RULES = “”“ === META-RULES (govern how numbered rules compose) === A. SUBSTANCE beats procedural vehicle. A case “about” stays even when the legal machinery (preemption, RICO, First Amendment overlay) would normally push it elsewhere. B. Numbered rules CANNOT be overridden by Meta-Rule A. C. is a LAST RESORT — use only when no numbered rule fires. Persistent use of the catch-all is a signal to author a new numbered rule. === PRIORITY RULES (applied in order; first match wins) === RULE 1 — : items about belong to , regardless of surface features that would normally push them elsewhere. RULE 2 — : ALL items of type belong to , including sub-types that look like they fit . Tiebreaker: . RULE 3 — : when is present, the classification is , not the that the model commonly confuses it with. # … ~10 numbered rules, each grounded in a specific failure cluster # from a baseline run. The long tail usually collapses into this many. “”“ # Adversarial few-shots: pick examples that are NEAREST to the known # failure modes, not the easiest or most representative. This is where # most of the remaining gain comes from, post-rules. prompt = build_prompt( user_input, domain_rules=DOMAIN_RULES, few_shot_examples=pick_adversarial_examples(user_input), ) return await run_llm(prompt, temperature=0.0)

Anish

The Validation Gap in Self-Improving AI