Rewrite the instruction from the function's mistakes. A teacher model reads
the function's answers on a few rows with feedback in words ("wrong: the
right answer is billing") and writes a better instruction; the best of the
instructions it writes is kept, chosen on rows it is never shown (half the
rows, or selection). Candidates that are best on at least one choosing
row stay in the pool, so ideas that fix different mistakes both survive,
and every fourth step two of them are combined. The teacher sees what it
tried that failed; a proposal that copies an input is dropped; ties go to
the shorter instruction; no row runs twice for one instruction.
Returns { fn, trials }: an improved copy (with the written instruction
when nothing beat it) and the search. Scores on the choosing rows flatter the
one chosen: measure it with evaluate on rows it never saw.
Rewrite the instruction from the function's mistakes. A teacher model reads the function's answers on a few rows with feedback in words ("wrong: the right answer is billing") and writes a better instruction; the best of the instructions it writes is kept, chosen on rows it is never shown (half the rows, or
selection). Candidates that are best on at least one choosing row stay in the pool, so ideas that fix different mistakes both survive, and every fourth step two of them are combined. The teacher sees what it tried that failed; a proposal that copies an input is dropped; ties go to the shorter instruction; no row runs twice for one instruction.Returns
{ fn, trials }: an improved copy (with the written instruction when nothing beat it) and the search. Scores on the choosing rows flatter the one chosen: measure it withevaluateon rows it never saw.