random_search() tries no examples, labeled rows only, bootstrapped runs
(see bootstrap_few_shot()), then candidates bootstrapped sets made from
shuffled rows, each scored on valset (default: the same rows: the score
then flatters). instruction_search() has a model write instructions (each
after a tip: be concise, spell out the steps, name the common mistakes,
...), pairs them with sets of worked examples, tries trials pairs on
minibatches of valset, and keeps the best of the finalists on all of
it. ai_trials() gives the search. Measure the result on rows it never
saw, with evaluate().
Usage
random_search(
fn,
data,
valset = NULL,
candidates = 8L,
metric = NULL,
max_bootstrapped = 4L,
max_labeled = 16L,
teacher = NULL,
seed = 0L,
stop_at = NULL
)
instruction_search(
fn,
data,
candidates = 6L,
trials = 12L,
minibatch = 20L,
valset = NULL,
prompt_lm = NULL,
metric = NULL,
max_bootstrapped = 4L,
max_labeled = 4L,
seed = 0L,
finalists = 3L,
teacher = NULL
)Arguments
- fn
An AI function.
- data
Rows with known answers.
- valset
Rows to score on (default:
data).- candidates
How many sets (or instructions) to make.
- metric
(row, prediction)returning a score; defaultexact_match().- max_bootstrapped, max_labeled
As in
bootstrap_few_shot().- teacher
The model that writes bootstrapped examples.
- seed
The random seed.
- stop_at
Stop once a candidate scores this much.
- trials
How many (instruction, examples) pairs to try.
- minibatch
How many rows each trial is scored on.
- prompt_lm
The model that writes instructions (default: the function's).
- finalists
How many of the best pairs are scored on all of
valset.
Value
An AI function, with its search in ai_trials().