Why did the replication shrink the gain?
Kitebench Research reports a gain from a new method. An independent rerun finds a smaller gain.
The question
Both teams save raw outputs for the same 100 prompts. The experiment specifies the exact normalized final ID as the primary scoring requirement.
| Baseline correct | Method correct | |
|---|---|---|
| Original report | 82 | 88 |
| Independent rerun | 79 | 80 |
Each score is out of 100 prompts.
“Trace what caused the reported score difference and explain what the paper should report.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What changed in the result?
What would you do here? Open the move you’d make.
Say the model improvement was overstated by exactly five points
Tempting, but it misses something. The two reported gains use conditions that still need comparison.
Check generation and scoring separately
The strong move. This identifies where the difference entered the result.
Use the average of the gains as the corrected effect
Tempting, but it misses something. An average does not explain the change in measurement.
See how a strong candidate says it
“The reported gain fell from 6 points to 1. I would compare the generation settings and scoring rules before I conclude that the model behaves differently.”
Why it works: The final score depends on how the outputs were produced and how they were scored.
Step 2: How would you narrow the cause?
The teams confirm the same prompt set, base checkpoint, model revision and decoding settings.
What would you do here? Open the move you’d make.
Generate all outputs again before inspecting the scorers
Tempting, but it misses something. Another generation adds variation and is unnecessary for comparing rules on fixed outputs.
Run both scorers on each saved output set
The strong move. This isolates the measurement change on the existing records.
Assume the outputs match because the settings match
Tempting, but it misses something. Matched settings alone do not establish identical output text.
See how a strong candidate says it
“I would take the saved outputs from each run and score them with both scorer versions. That lets me change the scoring rule while keeping the outputs fixed.”
Why it works: A crossed comparison tests whether the score follows the output set or the scorer.
Step 3: What does the crossed scoring show?
The original scorer accepted an expected ID anywhere in the answer. The exact-ID scorer checks only the normalized final ID. On both saved output sets the original scorer gives baseline 82 and method 88. The exact-ID scorer gives baseline 79 and method 80 on both.
What would you do here? Open the move you’d make.
Attribute the reported gap to the scoring rule in these records
The strong move. Both fixed output sets produce the same pair of gains under the two rules.
Use the original score because it gives the bigger gain
Tempting, but it misses something. The study specified the exact final ID as its primary requirement.
Claim every generated answer was identical across runs
Tempting, but it misses something. Matching aggregate counts can hide different individual outputs.
See how a strong candidate says it
“Both output sets show a 6-point gain with the original scorer and a 1-point gain with the exact-ID scorer. The observed difference follows the scoring rule in these records. I would use the exact-ID score for the primary claim.”
Why it works: Applying each rule to fixed outputs reproduces the observed score gap. The case specifies the exact normalized final ID.
Step 4: What belongs in the paper record?
What would you do here? Open the move you’d make.
Report only the older result because it came first
Tempting, but it misses something. It uses a different rule from the one specified for the experiment.
Keep both results with their rules and retain the evidence
The strong move. A reader can see what changed and repeat the scoring.
Discard the older output files after updating the paper
Tempting, but it misses something. The saved outputs are the record used to isolate the scoring difference.
See how a strong candidate says it
“I would report the 1-point gain under the specified rule, show the 6-point sensitivity result, and keep the raw outputs, scorer versions and settings. That gives the next reader enough to reproduce the comparison.”
Why it works: The primary result follows the planned rule. The sensitivity result explains the earlier discrepancy.
The answer, in short
- Decision
- Use the exact-ID scorer for the primary result and report the original scorer as a sensitivity check.
- Because
- The same output set yields a 1-point or 6-point gain depending on the scoring rule.
- What would change it
- A different result under the same scorer on matched outputs would reopen the generation question.
Common traps
- Change one comparison condition at a time.
- Distinguish equal aggregate counts from identical outputs.
Check your answer
- I quantified the gap.
- I chose a fixed-output comparison.
- I traced the scoring effect.
- I documented both rules.
Last reviewed Nov 11, 2026