2. Answers you can compute with
Sixty bird-survey notes, written however each volunteer liked. By the end you will have turned them into a table of species, counts and behaviours you can sum, plot and check, and you will know what to do when a note doesn't say.
Can you skip this one? If you can answer these, jump to tutorial 3. The answers are at the bottom.
- A note says "a few mallards". What should a
countcolumn hold, and what type do you give the answer so the model is allowed to say it? - How do you get three answers from one call, as three columns of a
DataFrame? - What happens to a row whose call fails, and how do you find it?
You will: give answers Julia types (Int, Union{Int,Missing}, an @enum, a struct, a Vector), get several answers from one call, and see what a type promises when the model has nothing to say.
Setting up
using FunctAI, DataFrames, CairoMakie, Statistics
log_folder = mktempdir()
FunctAI.configure!(lm = "gpt-6-luna", log_calls = log_folder);The notes
Volunteers walk four sites and write down what they see, in their own words. FunctAI.field_notes() holds sixty of their notes, and what the survey's coordinator recorded from each:
field_notes = DataFrame(FunctAI.field_notes())
select(field_notes, :id, :site, :note)60×3 DataFrame
Row │ id site note
│ Int64 String String
─────┼───────────────────────────────────────────────────────────
1 │ 1 Marsh boardwalk Great blue heron standing in the…
2 │ 2 North field A pair of robins pulling worms o…
3 │ 3 Creek trail Heard a chickadee calling 'chick…
4 │ 4 Old orchard Downy woodpecker drumming on a d…
5 │ 5 Marsh boardwalk About 40 Canada geese flying ove…
6 │ 6 North field Red-tailed hawk perched on the f…
7 │ 7 Creek trail 3 blue jays squabbling at the fe…
8 │ 8 Old orchard Male cardinal singing from the t…
⋮ │ ⋮ ⋮ ⋮
54 │ 54 North field Chickadees, a few, flying from t…
55 │ 55 Creek trail Robin feeding worms to 3 chicks …
56 │ 56 Old orchard Downy woodpecker at the suet fee…
57 │ 57 Marsh boardwalk Goose family: 2 adults, 5 goslin…
58 │ 58 North field Blue jays (4) eating acorns unde…
59 │ 59 Creek trail One great blue heron flying down…
60 │ 60 Old orchard Song sparrow and its mate singin…
45 rows omittedThe coordinator follows a protocol (it's in ?FunctAI.field_notes):
- Species is a name from the checklist of twelve; nicknames count ("robin", "red-tail", "downy"), and a bird not on the list is
other. - Count is every bird seen or heard. One bird named on its own ("a blue jay") is 1, "a pair" is 2, "about 40" is 40, but a note with no number ("a few", "a flock") has no count: never guess.
- Behaviour is one of five: feeding, nesting, flying, resting or calling (singing and drumming are calling; sitting on a nest is nesting).
Put the coordinator's answers aside as the key, and keep only what the volunteers wrote:
key = select(field_notes, :id, :species, :count, :behaviour)
notes = select(field_notes, :id, :site, :date, :note);A number
Start with the count. The obvious function answers an integer:
@ai function how_many(note::String)::Int
"How many birds does the note report?"
end
how_many("About 40 Canada geese flying over in a V, heading north.")40::Int is a promise: whatever the model writes, you get an Int back, or an error. Not the string "about 40", not "40 geese". You can add it, average it, plot it.
Now the whole column, next to the key:
counted = transform(notes, :note => ByRow(how_many) => :count)
counted = leftjoin(counted, select(key, :id, :count => :true_count), on = :id, order = :left)
filter(:true_count => ismissing, counted)[:, [:note, :true_count, :count]]7×3 DataFrame
Row │ note true_count count
│ String Int64? Int64
─────┼──────────────────────────────────────────────────────
1 │ Mallards, a few of them, dabblin… missing 3
2 │ Crows, a whole noisy flock, goin… missing 0
3 │ Red-winged blackbirds everywhere… missing 0
4 │ Several song sparrows hopping in… missing 3
5 │ Lots of crows mobbing a hawk and… missing 0
6 │ Several Canada geese flying low … missing 3
7 │ Chickadees, a few, flying from t… missing 3Here is the trap. The protocol says a note with no number has no count. But we asked for an Int, and an Int is what we got: the model had to invent one. "A few" became a number. Every one of those numbers will end up in a sum, looking exactly like a real count.
The type made the model answer. It should have let it not answer. In Julia, a value that may be absent from data is missing, so the answer's type is Union{Int,Missing}: an integer, or nothing to report. And the counting rule belongs next to the answer, as words the model reads about it: name the output, and describe it with ai"…":
@ai function how_many(note::String)
"How many birds does the note report?"
count::Union{Int,Missing} = ai"every bird seen or heard, young included; one bird named on its own ('a blue jay') is 1; 'a pair' is 2; an approximate number ('about 40', 'maybe 6') is that number; no number in the note ('a few', 'several', 'a flock') means no count: never guess"
end
counted.count = how_many.(counted.note)
filter(:true_count => ismissing, counted)[:, [:note, :true_count, :count]]7×3 DataFrame
Row │ note true_count count
│ String Int64? Int64
─────┼──────────────────────────────────────────────────────
1 │ Mallards, a few of them, dabblin… missing 0
2 │ Crows, a whole noisy flock, goin… missing 0
3 │ Red-winged blackbirds everywhere… missing 0
4 │ Several song sparrows hopping in… missing 0
5 │ Lots of crows mobbing a hawk and… missing 0
6 │ Several Canada geese flying low … missing 0
7 │ Chickadees, a few, flying from t… missing 0Look at the counts: the notes without a number still got one (mostly a zero, where the model had to write something). The words say "never guess", so why? Do what you did in tutorial 1, and read what the model reads:
FunctAI.prompt(how_many, "Mallards, a few of them.")model: gpt-6-luna
system
Function: how_many
How many birds does the note report?
Output guidance:
- count: every bird seen or heard, young included; one bird named on its own ('a blue jay') is 1; 'a pair' is 2; an approximate number ('about 40', 'maybe 6') is that number; no number in the note ('a few', 'several', 'a flock') means no count: never guess
Reply in exactly this form:
<count>
(integer)
</count>
user
<note>
Mallards, a few of them.
</note>Your rule is there, under "Output guidance". But the form the model must fill in says (integer), and nothing on the page says how to write "nothing". Faced with a form that wants a number, the model wrote one.
How the model is asked, and how its reply is read, is the function's layout. The default one, which you've been reading, writes the question as plain text and works with any model. The :json layout also sends the answer's exact type as a JSON schema (here: "an integer, or null"), and OpenAI, Anthropic and Gemini hold the model to that schema. For pulling typed fields out of text, especially fields that may be missing, it is the safer choice:
how_many_json = configure(how_many; adapter = :json)
counted.count = how_many_json.(counted.note)
filter(:true_count => ismissing, counted)[:, [:note, :true_count, :count]]7×3 DataFrame
Row │ note true_count count
│ String Int64? Int64?
─────┼────────────────────────────────────────────────────────
1 │ Mallards, a few of them, dabblin… missing missing
2 │ Crows, a whole noisy flock, goin… missing missing
3 │ Red-winged blackbirds everywhere… missing missing
4 │ Several song sparrows hopping in… missing missing
5 │ Lots of crows mobbing a hawk and… missing missing
6 │ Several Canada geese flying low … missing missing
7 │ Chickadees, a few, flying from t… missing missingA missing count is missing, and missing is contagious: sum(counted.count) is missing until you say sum(skipmissing(counted.count)), so a note with no number can't quietly become a zero in a total.
How close are the counts overall? isequal treats two missings as equal, which is exactly the comparison we want here:
mean(isequal.(counted.count, counted.true_count))1.0A choice
Behaviour is one of five words, so its type is an @enum:
@enum Behaviour feeding nesting flying resting calling
@ai function doing(note::String)::Behaviour
"What is the bird doing, by the survey's protocol?"
end
doing.(["Robin singing at dawn from the roof antenna.",
"Canada goose sitting on eggs on the island, mate standing guard."])2-element Vector{Behaviour}:
calling::Behaviour = 4
nesting::Behaviour = 1Singing is calling, sitting on eggs is nesting: the protocol's words, which the model can only choose among. A reply outside the five is not accepted: FunctAI asks again, and never hands you a sixth value.
Three answers from one call
You could write one function per column and call the model three times per note. It's cheaper to ask once, for all three. Declare each answer in the body, with its type and its words; the last one declared is the main answer, and calling returns them all:
species_list = ["American robin", "black-capped chickadee", "blue jay", "northern cardinal",
"mallard", "Canada goose", "great blue heron", "red-tailed hawk",
"downy woodpecker", "song sparrow", "American crow", "barn swallow", "other"]
@ai adapter = :json function survey(note::String)
"Record the note as the bird survey's protocol says."
species::OneOf(species_list) = ai"the checklist name; nicknames count; a bird not on the checklist is 'other'"
count::Union{Int,Missing} = ai"every bird seen or heard, young included; one bird named on its own ('a blue jay') is 1; 'a pair' is 2; an approximate number ('about 40', 'maybe 6') is that number; no number in the note ('a few', 'several', 'a flock') means no count: never guess"
behaviour::Behaviour = ai"singing, calling and drumming are calling; building, sitting on a nest or feeding young are nesting; perched, swimming, roosting or standing still are resting"
end
survey("Pair of downies (male + female) excavating a hole in the old pear tree.")(species = "downy woodpecker", count = 2, behaviour = nesting)A few things are new here:
OneOf(species_list)is a choice among values you have in a variable, without declaring an@enum(whose names can't have spaces). The answer is one of theStrings in the list.adapter = :jsonbeforefunctionis a setting of this function alone; any setting goes there (lm,temperature, …).- The answer is a
NamedTuple: destructure it,(; species, count) = survey(note), or turn it into columns.
DataFrames turns a NamedTuple per row into columns with AsTable:
recorded = transform(notes, :note => ByRow(survey) => AsTable)
select(recorded, :note, :species, :count, :behaviour)60×4 DataFrame
Row │ note species count behaviour
│ String String Int64? Behaviour
─────┼───────────────────────────────────────────────────────────────────────────────
1 │ Great blue heron standing in the… great blue heron 1 feeding
2 │ A pair of robins pulling worms o… American robin 2 feeding
3 │ Heard a chickadee calling 'chick… black-capped chickadee 1 calling
4 │ Downy woodpecker drumming on a d… downy woodpecker 1 calling
5 │ About 40 Canada geese flying ove… Canada goose 40 flying
6 │ Red-tailed hawk perched on the f… red-tailed hawk 1 resting
7 │ 3 blue jays squabbling at the fe… blue jay 3 feeding
8 │ Male cardinal singing from the t… northern cardinal 1 calling
⋮ │ ⋮ ⋮ ⋮ ⋮
54 │ Chickadees, a few, flying from t… black-capped chickadee missing flying
55 │ Robin feeding worms to 3 chicks … American robin 4 nesting
56 │ Downy woodpecker at the suet fee… downy woodpecker 1 feeding
57 │ Goose family: 2 adults, 5 goslin… Canada goose 7 resting
58 │ Blue jays (4) eating acorns unde… blue jay 4 feeding
59 │ One great blue heron flying down… great blue heron 1 flying
60 │ Song sparrow and its mate singin… song sparrow 2 calling
45 rows omittedSixty calls, three typed columns. Now it's data, so check it against the key, one column at a time:
checked = leftjoin(recorded, key, on = :id, renamecols = "" => "_key", order = :left)
(species = mean(checked.species .== checked.species_key),
count = mean(isequal.(checked.count, checked.count_key)),
behaviour = mean(string.(checked.behaviour) .== checked.behaviour_key))(species = 1.0, count = 0.9666666666666667, behaviour = 0.9666666666666667)And look at what it got wrong, because that's where you learn whether to trust it:
wrong(col) = checked[.!isequal.(string.(checked[!, col]), string.(checked[!, "$(col)_key"])),
["$(col)_key", col, "note"]]
wrong("species")0×3 DataFrame
Row │ species_key species note
│ String? String String
─────┴──────────────────────────────wrong("behaviour")2×3 DataFrame
Row │ behaviour_key behaviour note
│ String? Behaviour String
─────┼─────────────────────────────────────────────────────────────
1 │ feeding flying Osprey hovering then diving into…
2 │ nesting feeding Song sparrow carrying a caterpil…wrong("count")2×3 DataFrame
Row │ count_key count note
│ Int64? Int64? String
─────┼───────────────────────────────────────────────────────
1 │ 1 missing Chickadee pecking at birch catki…
2 │ 1 missing Heron stalking frogs at the edge…Read them before deciding anything. Some misses are arguable, the kind of disagreement two volunteers might have. Others are a rule applied less carefully when the model was asked for three things at once than when how_many only had to count. That's the trade-off to know about: one call instead of three is cheaper, and often just as good, but not always. Measure each column, and when one slips, give it back its own function or sharpen its words.
Now it's just data
The point of all this is what comes next, which is ordinary Julia:
birds = combine(groupby(dropmissing(recorded, :count), [:site, :behaviour]), :count => sum => :birds)
sites = sort(unique(birds.site))
kinds = instances(Behaviour)
colors = Makie.wong_colors()[1:length(kinds)]
fig = Figure(size = (720, 300))
ax = Axis(fig[1, 1], xlabel = "birds counted (notes with a number)", yticks = (1:length(sites), sites))
barplot!(ax, [findfirst(==(s), sites) for s in birds.site], birds.birds;
stack = Int.(birds.behaviour) .+ 1, color = colors[Int.(birds.behaviour) .+ 1], direction = :x)
Legend(fig[1, 2], [PolyElement(color = c) for c in colors], collect(string.(kinds)))
fig
Records and lists
An answer can be a whole record. Declare it as a struct, the way you'd declare any data in Julia; each field is typed, and may be missing:
struct Ages
adults::Union{Int,Missing}
young::Union{Int,Missing}
end
@ai function ages(note::String)::Ages
"Does the note report young birds, and how many of each age?"
end
young_notes = filter(:id => in((25, 42, 55, 57)), notes)
young_notes.ages = ages.(young_notes.note)
select(young_notes, :note, :ages => ByRow(a -> (; a.adults, a.young)) => AsTable)4×3 DataFrame
Row │ note adults young
│ String Int64? Int64
─────┼───────────────────────────────────────────────────
1 │ Mallard hen with 9 ducklings swi… 1 9
2 │ Robins, three adults and two spe… 3 2
3 │ Robin feeding worms to 3 chicks … missing 3
4 │ Goose family: 2 adults, 5 goslin… 2 5Each answer is an Ages, a value of your own type: ages(note).young works, and so does anything you've written for Ages. The last line spreads its fields into two columns.
An answer can also be a list. A Vector{String} is a list of quotes, each a string:
@ai function evidence(note::String)::Vector{String}
"Quote the words in the note that show what the bird is doing."
end
evidence("Robin carrying mud and grass into the hedge. Nest in progress!")1-element Vector{String}:
"carrying mud and grass into the hedge"Types compose the way they do everywhere in Julia: a Vector of structs, a struct with a Vector field, a Dict{String,Int} of counts by species. Each becomes the JSON schema the model is held to, and comes back as the type you wrote.
When a note is missing, or a call fails
A missing input is not sent to the model at all. It's missing, for free:
doing.(["Blue jay flew over, one.", missing])2-element Vector{Union{Missing, Behaviour}}:
flying::Behaviour = 2
missingAnd calls do fail: a provider has a bad minute, a reply can't be read even after asking again. FunctAI doesn't stop a whole column for one bad row: that row is missing, one warning says how many failed, and FunctAI.problems() lists them, with their errors, so you can fix the cause and run just those rows again. When every call of a column fails, nothing was answered, so there is nothing to keep: it throws the first error instead, which is almost always a setting to fix (a missing key, a model name). To see it, here's a function that can't succeed: it allows the model 16 tokens, far too few for it to think and answer.
starved = configure(survey; max_tokens = 16, retries = 0)
try
starved.(notes.note[1:3])
catch err
showerror(stdout, err)
end[parse-truncated] the provider cut the reply at its length limit; reader 'json_object' cannot tell which outputs ended before it (json_object: reply contains no JSON object (not a JSON object)); raise max_tokens or ask for less (the model spent 16 of its tokens thinking first; raise max_tokens)The error says why: the provider cut the reply off at its length limit. Remove the limit, and the same rows answer.
What it cost
prices = DataFrame(model = ["gpt-6-luna"], input = [0.10], output = [0.50]) # dollars per million tokens, 2026-09-27
bill = leftjoin(DataFrame(calls(folder = log_folder)), prices, on = :model)
(calls = nrow(bill), failed = count(!ismissing, bill.error),
dollars = sum(skipmissing(bill.input_tokens .* bill.input .+ (bill.total_tokens .- bill.input_tokens) .* bill.output)) / 1e6)(calls = 253, failed = 3, dollars = 0.0094867)Your turn
- Write
site_kind, a function whose answer is an@enumofwetland,field,woodland, from the note alone. Compare it with thesitecolumn: can the notes tell you where they were written? - Give
surveya note you write yourself with no number and a nickname ("Some cardinals in the hedge"). Iscountmissing? Is the species right? - Add a fourth answer to
survey,certain::Bool = ai"whether the volunteer sounds sure of the identification", and count how many notes the model thinks are uncertain.
What you learned
- The answer's type is a promise the answer keeps: an
Intis anInt, an@enumorOneOf(...)is one of your values, a struct is your struct, aVector{String}is a list of strings. - A type that can't say "nothing" forces an answer.
Union{Int,Missing}lets the model leave it empty, asmissing, and the:jsonlayout holds the model to the exact type. name::Type = ai"words"declares an answer with words about it; several make one call return aNamedTuple, andByRow(f) => AsTablemakes them columns.missingin,missingout, with no call. A failed row ismissing, with a warning andFunctAI.problems(); a column where every row fails throws.
Answers to the check at the top. (1) missing: the protocol says never guess; Union{Int,Missing} lets the model leave it empty, and adapter = :json holds it to that type. (2) Declare each answer in the body (species::OneOf(list) = ai"…", …); calling returns a NamedTuple, and transform(df, :note => ByRow(f) => AsTable) makes the columns. (3) It becomes missing, with one warning for all the failed rows; FunctAI.problems() lists them with their errors.
Next: 3. Is it right? turns "it looks good" into a number, an interval and a fair comparison.