Whybook / Benchmarks
Benchmarks
Every method that we tested for the small jobs of AI in Whybook, with its score on the same examples: rules built into Whybook, open models on the analyst's machine, and remote models. §8 of the front page sums them up and names the model that we would pick for each job.
§1 Table labels
Small language models for table labels
The task. A table too large for the bench shows as a tile with a description of one to three words and a headline, such as "3 significant". A model writes these labels. The remote model receives up to 3,000 characters of each table to do it. This benchmark measures which small model on the analyst's machine can write them instead, so that the table stays on the machine.
In the table, False counts headlines that their table contradicts, and Copies counts answers copied from the prompt's examples, which count as wrong.
The cases
Each model gets 20 tables and 10 data frames: 8 tables and 5 frames from the pain diary demo, 4 tables and 2 frames from the home energy demo, 8 tables from public notebooks (a Cox model, a confusion table, a correlation matrix, counts by state, a wide survey table and more), and 3 frames from the data sets of statsmodels. The table leaves out the titles that the models gave the frames: Whybook writes a sentence about each frame instead. Each model runs on 4 CPU threads with greedy decoding, through llama.cpp, and its answers are forced into JSON with word limits.
Speed on the target laptop
The table gives the estimated time of a label on the target: the laptop that universities buy as standard in 2026, such as a MacBook Air with the M5, or a Windows laptop with 16 to 32 GB and an Intel Core Ultra, AMD Ryzen AI or Snapdragon X2 chip, with a GPU build of llama.cpp.
Remote models on the same tables
Once a remote model is connected, Whybook asks it for the labels by default. Each remote model got Whybook's own request through OpenRouter, with zero data retention: up to 8 tables of a notebook in one call, each cut to 3,000 characters, with the code of its cell, and one call for the description and the headline. A frontier LLM graded the answers that no earlier grade covered, without knowing which model wrote them, by the rules of the local models' grading. Measured on 10 October 2026. Whybook's prompt to the remote model gives labels of the pain diary demo as examples, and that demo holds 8 of the 20 tables. Copies counts the tables where an answer repeats an example: each such answer is true of its table and counts as right, where the local models' copies of their prompt's examples were false and count as wrong.
§2 Question order
Which question to offer first
Each table is about one gesture, and checks whether Whybook's first option has the type of the analyst's question: descriptive, data quality, association, model or causal, read from the functions that the answer's cell calls. The first table is about dragging a variable onto another variable, or onto itself: for each question in the notebooks, the test works out which variables the analyst would drag to ask it. The second is about the gesture of the demo on the front page, a column dragged onto a cell. A notebook does not show which cell a question was about, so for each question that test makes up to three likely examples: a column that the question uses, dragged onto a cell above it.
With DASWOW's labels of exploring or modelling in place of the types read from code, the first option of the logistic regression was right for 79% of questions, and that of the rules for 71%. Asked one request at a time through OpenRouter, as Whybook asks, Claude Opus 5.5 put the analyst's type first for 48% of the 300 questions, and GPT-6 Astra, Grok 4.7 and MiMo-V2.6-Pro for 45% to 47%: all below the default's 57%. Two rankers with no model come close to the regression: the type that analysts ask most for the gesture, 55%, and the type of the question asked last, 55%. A model asked as Whybook asks is told neither rate. Told both in each request, Jev 1.13 rose from 49% to 56% of the 300 questions, and MiMo-V2.6-Pro from 48% to 56% of the first 100: level with the regression, not above it. Added to the regression's score, no model's answer raised it by more than chance. Agents of Claude Opus 5.5 that read the requests 100 at a time reached 61%, within chance of the regression, and for 58 of the 300 questions a later request of the same notebook among their 100 showed what the author asked.
What the score measures
The score compares the first offered question with the question that the notebook's author asked next. That stands in for a good question, and it is not one: it rewards a method that predicts what analysts do. A question that the author did not ask next can still be the better question, and Wonder mode is there to offer such questions.
The score counts the type of the first question, not the question itself. An example offers 5 questions in 2.7 types on average, so the score asks which of two or three types comes first. For 100 examples, a second reader named the offered question that the author's next cell answers. One was offered for 37 of them. On those 37, the first question of the logistic regression was that question for 70%, and that of Claude Opus 5.5 for 73%, and the better methods had it among their first three for 95% to 100%. For the other 63, none of the offered questions was what the author did next, so no order could put it first.
A remote model can be asked in two ways. By default, Whybook asks what the analyst asks next: that prompt fits this test, and the rows of the tables above use it. The setting "Question order: what the remote model is asked" asks instead what the analyst should ask, weighing the mode: in Wonder mode, the mode of every example here, the model favours what nobody asked yet. Asked that way, the same models put the author's type first less often:
Which of the two prompts helps an analyst more is a question for a study with analysts.
In a notebook of BuzzFeed News, the analyst drags the column WORKSTATT onto itself, before any other question. Whybook offers four questions.
- Descriptive Summarise WORKSTATT: distribution and missingness first, asked what the analyst asks next
- Model check Transform WORKSTATT: log, standardise or bin
- Model check Does WORKSTATT group repeated observations of the same unit?
- Data quality Are some levels of WORKSTATT too rare to analyse on their own? first, asked what the analyst should ask
value_counts(): a descriptive question, so the test counts the other prompt's first question as wrong. Yet the same counts show which levels are rare. Claude Opus 5.5 and MiMo-V2.6-Pro put the same questions first, one request at a time.§3 User questions
The type and the place of a user question
The analyst can type a question in place of picking one. Whybook gives it a type, and puts its answer in a place. The test takes the 276 questions that the authors of 650 public notebooks wrote in markdown, each asked about the code cell above it, and a reader who saw no method labelled them. The keywords alone gave the right type for 67%. Models built for choices did better, each asked about every question through OpenRouter: Jev 1.13 of TypeSafe 89%, Decider V1.1 27B of Perplexity 88% and Clef of Cloudflare 88%, in under a second a question, and Solar Decide of Upstage 84%, in 16 s. On the analyst's machine, the general models, each asked about every question as Whybook asks them, gave 64% (Qwen3 4B) to 75% (Ministral 3 3B), and Gemma 4 E2B 70%; asked only about the questions that no keyword matches, Gemma 4 E2B gave 66%. MiniCPM5 2B gave 22.5%: it labelled 206 questions as data quality, and Whybook does not offer it for user questions.
For the place, a new cell is the default. The Ask button changes to Edit or Branch only when the first words of the question ask for one, and a question that starts with "Show" gets a link that offers a preview. The place that a decision model predicts is offered under the Ask button whenever it differs from the button's: a branch as a box to tick, and a new cell, an edit or a preview as a link. Public notebooks hold almost no edit, branch or preview as they were saved, so the test of the place takes 360 questions written for a given place: three writers, GPT-6 Luna, Gemini 3.8 Flash and agents of a frontier LLM, each wrote one question for each place and each of 30 code cells from public notebooks. Blind readers named the writer's place first for 334 of the 360. The score is balanced accuracy, the mean over the four places of the share of each place's questions put there, so a new cell every time gets 25%. The rules on the first words move 91 questions, 73 of them right, for 42.5% with the preview counted as taken. With every offer taken, the place is the model's own prediction: the decision models get 57.5% to 86.9%, Decider V1.1 the most, and Claude Haiku 5.5 and Gemini 3.8 Flash, asked the same way, 88.6% and 87.8%. On the 276 questions from public notebooks, only 9% to 23% of the models' places other than a new cell were right.
§4 Placement
Where the answer to a gesture goes
In the 650 notebooks, most questions about a variable come right after the code that made or used it. When the analyst drags one variable onto another, the rules put the new cell after the last cell that uses both. When the analyst drags something, no model chooses the place of the answer.
§5 How we tested
The notebooks, the questions and the grading
- 650 public notebooks from five collections: the notebooks of papers in PubMed Central, three newsrooms, the Jupyter gallery, the examples of four statistics libraries, and five textbooks. For 4,790 of the questions their authors asked, the tests work out what an analyst would drag, and onto what, to ask it in Whybook.
- 272 notebooks of DASWOW, a random sample of data science notebooks on GitHub, whose every code cell data scientists labelled as exploring, building a model, or neither (Ramasamy and others, Empirical Software Engineering, 2023). Their labels check the types that the tests read from code, and they are a second collection to test on.
- 300 examples for each gesture. Every row of the two order tables is scored on the same 300 examples, so the rows compare question by question. A notebook does not show which cell a question was about, so for a column dragged onto a cell the test makes up to three likely examples for each question: a column that the question uses, dragged onto a cell above it.
- Held out. The logistic regression and the most common type are scored only on repositories that they were not fitted on: the notebooks go into 10 folds by repository, and each fold is ranked by a model fitted on the other nine. Fitted on the five collections and tested on DASWOW, which it never saw, the regression puts the analyst's type first for 54%.
- 276 user questions from the markdown of the 650 public notebooks, each with the code cell above it. A reader who saw no method gave each a type and a place. An earlier test of 120 questions had one author for the questions, their labels and the rules, and its numbers are in the research notes.
- Grading of the table labels: a frontier LLM graded every answer without knowing which model wrote it.
- Remote models got Whybook's own request, one at a time, through OpenRouter with zero data retention, or through the Claude Code login. The rows marked agents came from agents of Claude Opus 5.5 that read 100 requests each, which Whybook does not do. A later request among an agent's 100 could show what the author asked at an earlier one, and asked one request at a time, the same model did worse.
- Speed on the laptop that universities buy as standard in 2026, such as a MacBook Air with the M5, with a GPU build of llama.cpp: estimated as the time on the test machine divided by 20. Remote models are timed from the test machine.