
Cloud training discipline
Pod lifecycle decoupling, exit-path audits, watchdog cadences. So you don’t lose a 14-hour run.
Where training runs vanish
Lost training runs are the single most expensive class of incident in agentic ML work I’ve hit. A 14-hour run that vanishes is 14 hours of compute paid for, 14 hours of wall-clock waiting, and an unknown amount of operator confidence destroyed. The same incident class repeats in different forms (laptop slept, network dropped, terminal closed, atexit fired) and each time the symptom is the same. The pod is gone, the artifacts are gone, the run has to start over.
Structural cause is almost always the same. Pod lifecycle is coupled to a process on the operator’s machine. Anything that kills that process kills the pod. The cloud platform isn’t at fault. The launcher is.
The discipline below decouples pod lifecycle from launcher process. After it lands, none of the laptop-side failures can kill a run. The pod runs to completion or to a paid-for time threshold. The operator knows within 15 minutes if anything’s wrong. Artifacts get pulled by independent infrastructure.
Decoupling pod lifecycle from launcher
Single biggest move. Three properties to enforce, in order.
| Property | Why it matters | Implementation |
|---|---|---|
| Auto-destroy threshold on pod | Worst-case bound on lifetime if everything else fails | Set at pod creation, e.g., auto_destroy_after=24h. Generous but finite. |
| Launcher exits after creation | Removes the launcher process as a failure point | Print pod ID + dashboard URL; exit cleanly. No wait_for_completion(). |
| Cleanup conditional on positive completion | Pod destroys only when training succeeds, not on signals | Training script writes a “done” signal; pod self-destroys on the signal. No atexit. |
Auto-destroy threshold is the safety net. If the positive-completion signal never arrives (training crashed silently, network partition, anything), the pod runs to threshold and then dies. You pay for the full threshold but the pod is bounded. Not running for a week racking up charges.
Three loss patterns
Where 14-hour runs go to die. Hover any card to see the diagnosis.
Launcher death = pod death
atexit cleanup as the lifecycle
Silent watchdog failure
Fix in all three is the same. Pod has its own lifecycle, cleanup is conditional on success, watchdog runs independently. Without all three, one of the failure modes leaks through and the run vanishes.
Exit-path audit
Read the launcher with this prompt. It surfaces every code path that could destroy the pod at the wrong time.
Audit my training launcher for exit-path safety.
Read the launcher script and trace every code path
that could destroy or deallocate the pod. For each:
1. What triggers it (clean exit, signal, exception,
timeout, atexit, finally)?
2. Is the trigger correlated with "training script
finished successfully"?
3. If not, this is a wrong-time destroy path.
Recommend the fix.
Also check:
- Is there an explicit auto-destroy threshold on
the pod itself?
- Is the watchdog on independent infrastructure?
- After launcher exits cleanly, can the pod still
be reached / monitored / artifacts pulled?
Output: a list of every wrong-time destroy path with
the specific lines and the smallest fix per line.Walks every destroy path and grades whether the trigger correlates with training success. Wrong-time destroys surface immediately.
When I ran this for the first time I found 2 wrong-time destroy paths. Most common one is an atexit handler. Second-most-common is a try / finally with destroy in the finally. Both fire at the wrong times. Both are easy to remove once you see them.
Annotated transcript
A real conversation. I lost a 14-hour run, the agent triaged structurally, identified 3 specific changes in the launcher, and surfaced the postmortem requirement at the close. Click any pinned annotation for commentary on why each turn matters.
Three operator turns, three agent turns. Result: 3 launcher changes scheduled + watchdog standup + a postmortem to make the fix permanent.
- operator
I lost a 14-hour training run last night. Laptop slept, pod died, run is gone. What's the discipline that prevents this?
- agent
Three structural moves, in order:
1. Pod lifecycle independent of launcher. The launcher creates the pod with an explicit auto-destroy threshold and exits cleanly. The pod runs to completion or threshold, regardless of what the launcher does after.
2. Cleanup conditional on positive completion. Pod destroys itself only when the training script signals "done — artifacts written." Not on launcher exit, not on atexit, not on signal. Positive completion only.
3. Watchdog on independent infra. A second process — phone cron, cloud function, anything not on your laptop — pings pod health every 5-15 minutes. Alerts if pod is unhealthy or unreachable.The first move is non-negotiable. The other two harden against specific failure classes.
- operator
What does my launcher look like today and what do I need to change?
- agent
Read me the launcher and I'll diagnose. Three things I'm specifically looking for:
1. Does the launcher hold the pod handle after creation, or exit cleanly?
2. Does cleanup use atexit / signal handlers / try-finally with destroy in the finally?
3. Is there a watchdog at all, and where does it run?With the launcher in front of me I can show you the exact 3-5 lines that need to change.
- operator
[pastes launcher script, ~80 lines, uses atexit for pod destroy, holds the pod handle, has no watchdog]
- agent
Three changes:
1. Remove the atexit handler. It's the immediate cause of last night's loss — laptop sleep triggered atexit, atexit destroyed the pod. Replace with: pod has an explicit
auto_destroy_after=24hflag set at creation. No atexit.2. Launcher exits after pod creation. After
pod = create_pod(...)and confirming pod is healthy, the launcher prints the pod ID + dashboard URL and exits cleanly. It doesn'twait_for_completion()— that's the watchdog's job, not the launcher's.3. Add a watchdog on independent infra. Simplest version: a cron on a $5/mo VPS or a phone-side scheduled task that pings pod health every 10 minutes. Alerts (text / push) if unhealthy. Fancier later — start simple.
With these three changes, last night's failure mode is gone. The pod runs to completion or to the 24h threshold — paid-for time, not lost work — and the watchdog tells you within 10 minutes if anything's wrong.
One postmortem: write this up as an incident. Action items: the three changes above. Without the postmortem, the same class of failure recurs in 6 weeks because the structural fix never landed.
Training launcher checklist
Pre-flight checklist for any launcher before it runs in anger. I walk it once per launcher and re-walk after any structural change.
PRE-FLIGHT CHECKLIST FOR TRAINING LAUNCHER
[ ] Pod creation includes an explicit auto-destroy
threshold (timedelta, generous but finite)
[ ] Auto-destroy grace period is set (e.g., 10 min
buffer for artifact upload)
[ ] Launcher prints pod ID + dashboard URL after
creation
[ ] Launcher exits cleanly after pod is healthy and
confirmed running — does NOT call
wait_for_completion() or hold the pod handle
[ ] No atexit handlers tied to pod destruction
[ ] No try/finally with destroy in finally (cleanup
is positive-completion only)
[ ] Training script writes an explicit "done" signal
(file write, API call, queue message) on success
[ ] Pod self-destroys on done signal OR runs to
auto-destroy threshold — whichever is sooner
[ ] Watchdog runs on infrastructure independent of
the launcher's machine
[ ] Watchdog pings pod health every 5-15 minutes
[ ] Watchdog alerts (text / push / email) on
unhealthy or unreachable pod
[ ] Watchdog state persists across pings (so multi-
failure detection isn't lost)
[ ] Artifacts are pulled by the watchdog or pod-side
code — never by the launcher
[ ] If this launcher has lost a run before, there is
a postmortem with action items mapped to specific
items in this checklist
Last item is what makes the discipline compound. A postmortem whose action items map to specific checklist items lands as structural change instead of operator memory. Re-walk the checklist before the next run.
Prompt kit
Three prompts. Audit exit paths, set auto-destroy thresholds, add watchdogs. I keep mine in the project’s CLAUDE.md.
Audit my training launcher for exit-path safety.
Read the launcher script and trace every code path
that could destroy or deallocate the pod. For each:
1. What triggers it (clean exit, signal, exception,
timeout, atexit, finally)?
2. Is the trigger correlated with "training script
finished successfully"?
3. If not, this is a wrong-time destroy path.
Recommend the fix.
Also check:
- Is there an explicit auto-destroy threshold on
the pod itself?
- Is the watchdog on independent infrastructure?
- After launcher exits cleanly, can the pod still
be reached / monitored / artifacts pulled?
Output: a list of every wrong-time destroy path with
the specific lines and the smallest fix per line.Update the pod creation call to include an explicit
auto-destroy threshold.
Replace:
pod = create_pod(image=..., gpu=...)
With:
pod = create_pod(
image=...,
gpu=...,
auto_destroy_after=timedelta(hours=24), # adjust
# per run
auto_destroy_grace_period=timedelta(minutes=10),
)
The threshold is the worst-case bound on pod lifetime.
Set it to: training_duration_estimate * 1.5, plus
buffer for artifact upload. Don't set it to "infinite"
or omit it — that's how forgotten pods bill for days.Add a watchdog that runs on infrastructure independent
of my laptop / launcher.
Simplest version (pseudocode for any cron-capable
machine):
every 10 minutes:
ping pod health endpoint
if pod is unhealthy or unreachable:
send notification (text / push / email)
log incident with timestamp
else:
log healthy heartbeat
Specs:
- Runs on a machine that doesn't share fate with my
laptop — VPS, cloud function, phone scheduled task
- Alert latency: <15 minutes between failure and
notification
- State: persists across cron runs (so we know if
this is the first failed ping or the 6th)
- Cleanup: when pod completes successfully, watchdog
exits cleanly and stops pingingApply this — harden one launcher
~90-minute exercise.Audit, refactor, watchdog, postmortem. So you don’t lose the next run.
Your first launcher hardening pass
Each step takes 15-30 minutes. Progress saves automatically.
- 01Read your training launcher with the exit-path audit prompt. Identify every wrong-time destroy path.Most operators have at least one — usually atexit or a try/finally with destroy in finally. Find them all before fixing.
- 02Add an explicit auto-destroy threshold on the pod itself. Set it generously but finite.training_duration_estimate * 1.5 plus artifact upload buffer. Don't omit; don't set to infinite.
- 03Refactor the launcher to exit cleanly after pod creation. Print pod ID + dashboard URL; do not wait_for_completion.If your launcher needs to do post-training work, that work moves to the pod itself or to the watchdog — not the launcher.
- 04Stand up a watchdog on independent infrastructure. Simplest version first; harden later.$5/mo VPS, phone cron, cloud function — any of these work. The point is independence from the launcher's fate.
- 05Write a postmortem on the most recent training-run loss. Action items go in the launcher template, not in your head.Without the postmortem the structural fix doesn't compound across future runs. The discipline is to write it up.