Which change earned the result?
Kitebench Research introduced a new training mix and objective in the same pilot.
The question
All runs start from the same checkpoint and use the same pilot seed, training budget and scoring rule. Each uses the same 200 validation items.
| Correct answers | Validation items | |
|---|---|---|
| Old data and old objective | 128 | 200 |
| New data and new objective | 148 | 200 |
A two-by-two comparison also needs one run for each change by itself.
“Explain what the available runs say about each change and what you would run next.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What does the bundle establish?
What would you do here? Open the move you’d make.
Credit the new objective with all 10 points
Tempting, but it misses something. The data changed in the same run.
Use each one-change run to complete the comparison
The strong move. This tests both factors with the other factor held fixed.
Repeat the bundle and assign half the gain to each change
Tempting, but it misses something. A second bundled result still leaves the factor effects mixed.
See how a strong candidate says it
“The bundle has 20 more correct answers, or 10 percentage points. I would run the new data with the old objective and the old data with the new objective.”
Why it works: The observed bundle changes two factors. Each one-change run helps separate their effects.
Step 2: What does the data-only run show?
The new-data run with the old objective gets 140 correct out of 200.
What would you do here? Open the move you’d make.
State the two measured contrasts and request the missing cell
The strong move. The effect of one change can depend on the setting of the other.
Call the objective’s four-point gain independent of the data
Tempting, but it misses something. Four points is the contrast when both runs use new data.
Subtract six points from the bundle and assume that proves the interaction
Tempting, but it misses something. The objective-only result is needed to estimate the additive expectation.
See how a strong candidate says it
“With the old objective, new data adds 12 correct answers, or 6 points. With the new data, the new objective adds another 8 correct, or 4 points. I still need the objective-only result to compare the objective in both settings.”
Why it works: The data-only run gives two contrasts. The missing cell would show the effect of the objective with the old data.
Step 3: What does the last cell add?
The run with old data and the new objective gets 132 correct out of 200.
What would you do here? Open the move you’d make.
Report a 2-point interaction in the pilot
The strong move. The actual bundle is four correct answers above the additive expectation on 200 items.
Report a 4-point interaction because the objective adds four points with new data
Tempting, but it misses something. The objective already adds two points with old data. The interaction is the difference.
Say the result proves the same interaction on other seeds
Tempting, but it misses something. Every cell uses one pilot seed. Other runs can show a different interaction.
See how a strong candidate says it
“The new objective adds 4 correct answers with the old data, or 2 points. Adding the two separate gains predicts 144 correct. The bundle got 148, so its result is 4 correct answers above that prediction, a 2-point interaction in this pilot.”
Why it works: The interaction is the difference between the observed bundle and the sum of the one-change gains on this accuracy scale.
Step 4: What would you tell the research lead?
What would you do here? Open the move you’d make.
Repeat only the bundle because it had the highest score
Tempting, but it misses something. That cannot show whether the interaction holds.
Repeat the four cells with matched seeds
The strong move. The design can distinguish the data, objective and interaction across runs.
Change the validation set and budget in the next objective-only run
Tempting, but it misses something. Those changes make its result hard to compare with the other cells.
See how a strong candidate says it
“I would report the two effects at the baseline settings and the 2-point interaction on this seed. I would repeat all four cells on further matched seeds with the same budget and scoring.”
Why it works: Matched repetitions test whether the conditional effects hold while preserving the comparison.
The answer, in short
- Decision
- Report the conditional effects and the 2-point interaction on this pilot seed.
- Because
- The bundle got 148 correct. The two individual gains predict 144 on the same 200 items.
- What would change it
- Further matched runs would show whether the interaction persists.
Common traps
- Hold the comparison conditions fixed.
- Distinguish a conditional effect from the interaction.
Check your answer
- I separated the two changes.
- I checked the data-only contrast.
- I used all four cells.
- I chose a matched confirmation.
Last reviewed Nov 11, 2026