What survives the overlap check?
You are discussing a draft paper from the fictional Kitebench Research lab.
The question
The baseline and recipe use the same benchmark items. The recipe uses source passages from public collections. The draft says the new recipe improves generalization.
| Correct | Items | |
|---|---|---|
| Baseline | 72 | 100 |
| New training recipe | 80 | 100 |
Each item is scored correct or incorrect.
“Explain what the result supports and what you would check before defending the paper’s claim.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What does the table establish?
What would you do here? Open the move you’d make.
Report that the new recipe generalizes 8 points better
Tempting, but it misses something. The table gives no evidence about overlap or selection.
Report the benchmark gain and check how the test set was used
The strong move. This states the result while keeping the stronger claim open.
Call it an 8 percent relative gain
Tempting, but it misses something. Eight is the absolute difference in percentage points. The baseline for a relative change is 72%.
See how a strong candidate says it
“The new recipe got 8 more items correct on this benchmark. That is an 8 percentage point gain. I would check where the test items came from and whether the team used them to choose the method.”
Why it works: The result describes a fixed set. An independent generalization claim needs evidence about how that set was used.
Step 2: Which records would you audit?
What would you do here? Open the move you’d make.
Remove every item the recipe got wrong
Tempting, but it misses something. Selecting items by one model’s output changes the comparison.
Ask only for the number of overlapping items
Tempting, but it misses something. The number alone does not tell you how many correct answers to subtract for either model.
Find matching IDs and both models’ scores for those IDs
The strong move. This gives a common denominator and an accurate numerator for each model.
See how a strong candidate says it
“I would match benchmark IDs to source passages in the training records. I would get both models’ scores on each matching ID so we can remove the same items from each result.”
Why it works: A matched exclusion keeps the remaining evaluation sets comparable.
Step 3: What remains on the matched subset?
The audit finds 20 benchmark items that share source passages with the recipe’s training data. On those same 20 IDs, baseline got 14 correct and the recipe got 19.
What would you do here? Open the move you’d make.
Report the two scores on the same 80 retained items
The strong move. This gives the sensitivity result on a comparable set.
Subtract the scores but divide by the original 100 items
Tempting, but it misses something. Both models now have 80 evaluated items.
Say training overlap caused exactly 4.25 points of the original gain
Tempting, but it misses something. Removing the subset changes which items are measured. It does not isolate the causal effect of exposure.
See how a strong candidate says it
“The remaining 80 items give 58 correct for baseline and 61 for the recipe. That is 72.5% and 76.25%, a gain of 3.75 percentage points. The changed gain shows why I would limit the generalization claim.”
Why it works: Subtract the correct answers and the same 20 item IDs from both models. The removed subset can differ in difficulty as well as overlap.
Step 4: What claim would you defend?
What would you do here? Open the move you’d make.
Report both results and request a fresh held-out comparison
The strong move. The reader can see how the provenance check changed the result.
Replace the full result with 3.75 and describe the subset as an untouched holdout
Tempting, but it misses something. The team selected the subset after seeing an issue on the original benchmark.
Repeat the original benchmark with more training until it reaches the same gain
Tempting, but it misses something. That uses the benchmark to guide the method and supplies no independent test.
See how a strong candidate says it
“I would report the recorded 8-point gain and the 3.75-point matched result. I would name the excluded IDs and why we excluded them. To test generalization I would use a fresh, independently sourced set kept out of training and method selection.”
Why it works: The paper can state the result it observed and what evidence would support a broader claim.
The answer, in short
- Decision
- Report the matched 3.75-point result beside the original 8-point result.
- Because
- Twenty matched items share training passages. Removing them from both sides reduces the measured difference.
- What would change it
- A fresh held-out result would give stronger evidence about generalization.
Common traps
- Use the same retained IDs for both models.
- Keep a sensitivity result within its measured scope.
Check your answer
- I stated the recorded gain.
- I asked for item-level provenance.
- I recomputed both scores.
- I identified the independent test.
Last reviewed Nov 11, 2026