Which experiment can the remaining compute answer?
Kitebench Research wants to know whether a new data mix helps at the current training size.
The question
The plan requires 4 distinct paired seeds before making a multi-seed claim. Each pair has one baseline and one variant at the same seed and size. An AI draft suggests going straight to a larger model.
| GPU-hours | |
|---|---|
| Planned total | 72 |
| One current-size run | 8 |
| One larger run | 32 |
A comparison at either size needs a baseline and a variant at that size.
“Choose the next experiment and state which research claim it could support.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What would you run first?
What would you do here? Open the move you’d make.
Run the larger variant and compare it with the current-size baseline
Tempting, but it misses something. A change could come from the data, the size or their interaction.
Start with a matched current-size pair
The strong move. This checks the effect of the data mix at the size in the question.
Run two variants with no baseline
Tempting, but it misses something. There is no same-seed reference for the data change.
See how a strong candidate says it
“I would run one matched baseline and variant at the current size. The pair costs 16 GPU-hours and gives an initial comparison at that size.”
Why it works: A larger variant without a larger baseline changes both the data and the model size.
Step 2: What does the pilot justify?
The pilot used 16 GPU-hours. It scored 63% for baseline and 66% for the variant.
What would you do here? Open the move you’d make.
Make a multi-seed claim from the pilot
Tempting, but it misses something. The result covers one paired seed.
Use the pilot to justify three additional pairs
The strong move. That reaches the planned count within the original budget.
Count each run as an independent paired result
Tempting, but it misses something. One baseline and one variant together make one pair.
See how a strong candidate says it
“The pilot shows a 3-point gain on one paired seed. I would repeat it on three more pairs. At the planned budget, 56 GPU-hours remain after the pilot, and three pairs cost 48.”
Why it works: One pair provides a starting measurement. The planned claim needs independent seed pairs.
Step 3: What fits the revised budget?
The scheduler confirms that only 32 GPU-hours remain after the completed pilot.
What would you do here? Open the move you’d make.
Use the 32 hours for two more current-size pairs
The strong move. This answers the current-size question on more seeds and keeps the fourth pair pending.
Run one large variant and count it as the fourth pair
Tempting, but it misses something. It is a single run at a different size.
Run four variants and compare them all with the first baseline
Tempting, but it misses something. The study needs a baseline matched to each seed.
See how a strong candidate says it
“We can run 2 more paired seeds with 32 GPU-hours, for 3 paired seeds total. A matched larger comparison costs 64. The fourth current-size pair still needs 16 hours before we make the planned claim.”
Why it works: Two current-size runs cost 16 GPU-hours per pair. The revised budget leaves no spare time.
Step 4: What do you tell the research lead?
What would you do here? Open the move you’d make.
Use three pairs and state which claim is still pending
The strong move. The lead gets the available evidence and the cost of the next decision.
Reduce the required seed count because the compute changed
Tempting, but it misses something. The study planned its claim threshold before seeing the result.
Infer how the data mix behaves at the larger size
Tempting, but it misses something. The study has no matched larger-size result.
See how a strong candidate says it
“I would report the result after three distinct matched pairs and keep the fourth pending. I would ask for 16 more GPU-hours for that pair and leave the larger-scale question open.”
Why it works: The revised plan gathers relevant evidence and keeps the original claim rule intact.
The answer, in short
- Decision
- Run two additional current-size pairs and keep the fourth pair pending.
- Because
- The 32 remaining GPU-hours pay for two 16-hour pairs. Together with the pilot this gives three paired seeds.
- What would change it
- Another 16 GPU-hours would support the fourth pair in the planned comparison.
Common traps
- Count matched seed pairs, not individual runs.
- Keep the claim rule fixed when resources change.
Check your answer
- I matched the first comparison.
- I scoped the pilot result.
- I recalculated the next runs.
- I stated which claim remained open.
Last reviewed Nov 11, 2026