Archive 07 — WritingNotes · Research Log
Testing what AI actually does —
not what it claims.
Studies and field notes on how AI models behave in real use: where they get better, where they fail, and how to test them on the work you care about. Newest first.
If something here sparks a question — or a project we should hear about — get in touch.
10
LLM annotation: modelling how expert coders think beats a bigger model
We wrote down how expert human coders decide and gave it to a cheap LLM. It matched their labels better than frontier models using the coding manual, at 18% of the cost.
09
AI models are improving unevenly: code and math pull ahead of writing
We tracked 353 AI models over three years of human votes. All of them improve, but coding and math pull ahead of writing and advice, and public benchmarks miss it.
08
Managing a multi-agent AI workflow: set the rules first, then enforce them
How we run multi-agent AI workflows: define each stage's output and pass/fail checks up front, then give an orchestrator agent the power to reject and re-run work.
07
A behavioural safety audit of a live wellbeing companion: findings and recommendations
We ran a battery of synthetic users against AURI, a live wellbeing companion, over 50 conversations of 15 turns each. This is what we found, and what we would recommend on each finding.
06
AI review of marketing claims: a committee of models beats a single model
We tested a committee of AI reviewers against the best single AI models on real advertising regulator decisions. The single models missed about 1 in 4 flagged claims.
05
Testing a B2B landing page with a simulated buying committee
We tested a live B2B landing page with five synthetic buyer personas. Three left before booking a demo. Here is why, and why a single AI model missed it.
04
Steering synthetic personas: changing one Big Five trait shifts the others
We changed one Big Five trait at a time in AI synthetic personas. The target trait moved cleanly every time, but the other traits shifted too, differently per persona.
03
AI strategic advice: why persona prompts alone still give generic answers
We asked one AI model the same business strategy question three ways. Only a fully built AI persona gave specific, testable advice; a simple persona prompt stayed vague.
02
How AI chatbots fail vulnerable users: a multi-turn safety test of six models
We tested six AI chatbots with three simulated vulnerable users over 18-turn conversations. Most failures were quiet: warmth that built dependency, not harmful advice.
01
Synthetic personas: why a prompt alone breaks and how to test them
A synthetic persona written only as a prompt drifts and breaks under small changes. We explain how to build AI personas you can measure, cross-check and stress-test.