2.1 Durable execution

We start Part 2 with the wound from 1.2: a naive agent keeps its progress in memory, so a crash wipes it — and on restart the agent re-does work it already did, sending the same email twice. That’s the failure. Now we build the fix.

The pain, one more time

Real agent work is long-running and touches the outside world: it sends messages, charges cards, files tickets. And processes die all the time — deploys, out-of-memory, a rate-limit that throws, a machine that reboots. If “what have I done so far?” lives only in RAM, every death is a catastrophe: lost progress and duplicated actions.

A plain while loop has no answer for this. It treats a task that takes minutes and spends real money as if it were a function call that either returns or doesn’t.

What the harness adds: write it down before you move on

The fix is almost embarrassingly simple to state:

Before doing the next step, durably write down that the current one happened.

Two pieces make that real:

An event log — an append-only record of every step, stored somewhere that survives a crash (a database, not memory).
Checkpointing — after each step completes, its result is written to the log before the loop continues.

Then recovery is just: on restart, replay the log. The harness reads what’s already done and resumes from the first unfinished step — so a completed sendEmail is never run again. That “never run again” property is called idempotency, and it’s the whole point.

See it survive

Here’s the exact task and the exact crash from the 1.2 demo — but now every completed step is checkpointed to a durable log. Do the same thing you did before: step until the email sends, kill the process, then restart.

Durable checkpoint log persisted · survives a crash

▶

Task received

Draft the summary

Send the email

side effect

Mark complete

Process running

at step 1

position comes from the durable log

The outside world real

0 emails sent

exactly once, even across a crash

Process started. Durable log is empty.

Try the same thing you did in 1.2: Next step until the email sends, then 💥 Kill process, then ↻ Restart and finish. Watch the email count.

Watch the difference. The checkpoint log at the top survives the crash. On restart, the harness reads it, sees the email was already sent, and resumes past it. The outside world ends at exactly one email — where the naive agent sent two.

The naive loop, made durable

Here’s the loop from 1.1, with the two new pieces. If you don’t read code, read the comments — each maps to something in the demo.

async function runAgent(taskId: string, task: string) {
  // On restart, rebuild progress by replaying the durable log.
  const log = await eventLog.load(taskId);        // survives crashes
  let messages = replay(log, task);

  while (true) {
    const reply = await model(messages);
    if (!reply.toolCall) return reply.text;

    // If this step is already in the log, DON'T run it again — just reuse the result.
    const done = log.find((e) => e.step === reply.toolCall.id);
    const result = done
      ? done.result
      : await tools[reply.toolCall.name](reply.toolCall.args);

    // Checkpoint: write the result to the durable log BEFORE continuing.
    if (!done) await eventLog.append(taskId, { step: reply.toolCall.id, result });

    messages.push({ role: 'tool', content: result });
  }
}

The shape barely changed. What changed is where the truth lives: the durable log, not the process’s memory. Kill this loop at any point and re-run it — it fast-forwards through everything already in the log and picks up exactly where it stopped.

In production

You would not build the event log and replay engine yourself. This is exactly what durable execution frameworks provide:

Temporal, DBOS, Inngest — you write what looks like ordinary code; they checkpoint every step and transparently resume after a crash.

You now understand what they’re doing under the hood, because you just watched the tiny version of it survive a crash.