insideout Get Pro

13 · An agent that runs experiments

Pro

Agents, lesson 13

An agent that runs experiments

An agent edits code, runs a fixed-budget experiment and keeps what helps.

You know the agent loop and its limits, especially "Keeping an agent's window clean" and "Limits on what an agent may do". "Fixed-budget experiments" in Training and Tuning covers the training side of this loop; this lesson is about the agent and the setup around it.

Part 1. A night of experiments

In March 2026 Andrej Karpathy released autoresearch, a small open-source setup in which a coding agent does training research on its own. The agent edits one training script, trains for a fixed five minutes, reads one number, and keeps the change or throws it away. Then it does it again, all night, while a person sleeps.

Three files matter. prepare.py downloads the data, trains the tokenizer and holds the scoring function, and the agent may not change it. train.py holds the model, the optimiser and the training loop, and the agent may change anything in it. program.md holds the agent's instructions, and only the person edits it. Karpathy's framing is that you stop editing Python and start programming the research team, in Markdown.

Example 1.1

One experiment, as the agent runs it

  1. edit train.py→learning rate 0.04 → 0.05
    →git commit

    One change, committed, so every experiment has an id the agent can return to.

  2. uv run train.py > run.log 2>&1
    train for five minutes→score

    All output goes to a file, not into the agent's context. Training stops after five minutes of wall-clock time, whatever the change.

  3. grep "^val_bpb:" run.log→0.9932

    The agent reads two lines, the score and the peak memory, and nothing else. If they're missing, the run crashed, and it reads the last 50 lines to see why.

  4. 0.9932<bar 0.9945→keep: the branch moves on
    otherwisegit reset back

    Lower is better. A better score keeps the commit and becomes the new bar; an equal or worse one is thrown away.

  5. results.tsv+b2c3d4e 0.993200 44.2 keep raise learning rate

    Every run gets a row: kept, discarded or crashed. About 12 runs an hour, about 100 by morning.

Your turn

Why does the agent send training output to run.log and read back only two lines?

A night is one long run of the agent loop, and the rules from "Keeping an agent's window clean" apply: trim at the source, keep the full output in a file, read a slice only when something breaks. autoresearch's instructions say exactly this, in capitals.

Hill climbing

Trying one change at a time and keeping it only if the score beats the best so far. Each kept change becomes the starting point for the next.

Part 2. What makes the log worth reading

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. What makes the log worth reading
  2. The winner is partly luck
  3. Making a night's result hold up
See Pro plansI have Pro: sign in
Next lessonThe harness around a coding agentContext files, skills, hooks and permissions around the model. Start lesson 14

Agents overviewSpotted a mistake? Tell us.