04 · Fixed-budget experiments

You have a small GPT and two ideas for its training script. One makes the model bigger. The other takes faster but slightly rougher steps, say by doing the arithmetic in a smaller number format. You train the baseline and both ideas for 1,000 steps each and score them on held-back text in bits per byte (lower is better).

The bigger model scores 1.132, against the baseline's 1.152. The faster one scores 1.159. But a step of the bigger model takes 0.45 seconds, a baseline step 0.30, and a faster step 0.21. (These numbers are made up, but they have the usual shape.)

# Bits per byte after n steps (made-up curves of the usual shape)
def bpb(steps, floor, scale):
    return floor + scale * steps ** -0.3

runs = {  # name: (seconds per step, floor, scale)
    "baseline": (0.30, 0.90, 2.00),
    "bigger model": (0.45, 0.88, 2.00),
    "faster steps": (0.21, 0.90, 2.06),
}
print("                 1,000 steps   300 seconds")
for name, (sec, floor, scale) in runs.items():
    steps = int(300 / sec)
    print(f"{name:14}   {bpb(1000, floor, scale):.3f}         {bpb(steps, floor, scale):.3f} ({steps:,} steps)")
#                  1,000 steps   300 seconds
# baseline         1.152         1.152 (1,000 steps)
# bigger model     1.132         1.164 (666 steps)
# faster steps     1.159         1.133 (1,428 steps)

The order turns right round. At equal steps the bigger model wins and the faster one comes last. At equal time it's the other way about.

This happened for real in nanochat, Andrej Karpathy's small project for training a chat model end to end. In February 2026 its record run switched most of its matrix maths to FP8, an 8-bit number format, along with a slightly bigger model. Karpathy's notes say each step got a little worse, but steps came so much faster that the run came out ahead: the time to reach its target fell from 3.04 hours to 2.91. Compared step for step, FP8 would have looked like a loss.