Thinking Machines Lab interviews
Thinking Machines Lab interviewsResearch posts
Research judgment
“Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models.”
Horace He in collaboration with others at Thinking MachinesConnectionism research blog post
Reproducibility is a bedrock
Practise checking what a benchmark gain still supports after an overlap check.
Example“The difference in logprobs between the trainer and the sampler probably isn’t a real bug, right? We reject this defeatism.”
Horace He in collaboration with others at Thinking MachinesConnectionism research blog post
Reject the defeatism
Practise checking what a benchmark gain still supports after an overlap check.
Example“These decisions require knowledge and judgment that people acquire through continuous contact with the work, increasingly done alongside AI.”
Thinking MachinesMission essay
Judgment comes from contact with the work
Practise checking what a benchmark gain still supports after an overlap check.
ExampleEngineering for research
“Research productivity is paramount and heavily depends on the reliability, efficiency, and ease of use of infrastructure.”
Thinking Machines LabCompany home page, "Solid foundations matter"
Infrastructure drives research productivity
Practise tracing a replication gap to the scoring rule, with the outputs held fixed.
ExampleLast reviewed