Checkpoints & Resumption
Bioinformatics jobs can run for hours or even days. Re-running an entire pipeline because one late step failed — or because a spot instance was reclaimed — would be unacceptable. BioNodulo solves this with checkpointing and resumable runs.
How checkpointing works
- Every node’s output is content-addressed and cached when the node completes.
- The workflow orchestrator records the run’s progress durably as each node finishes (deterministic, replayable state).
- If a run is interrupted — a failure, a cancellation, or an evicted spot instance — the recorded checkpoints survive.
Resuming a run
When you resume (or re-run) a workflow:
- The engine compares each node’s inputs, parameters, and code version against the cached results.
- Nodes whose inputs are unchanged are marked Skipped and their cached outputs are reused instantly.
- Only the failed node and everything downstream of it actually re-execute.
This means you don’t re-pay credits for work already completed.
Cache invalidation
A cached node result is invalidated (and the node re-runs) when any of these change:
- An input dataset or upstream output.
- A parameter value.
- The node’s code/version (e.g. you upgraded the node from the registry).
Spot-instance resilience
Cloud execution pods often run on discounted spot instances. If a spot instance is reclaimed mid-run, the orchestrator reschedules the in-flight node on a fresh pod and resumes from the last checkpoint — transparently, without losing completed work.
Practical tips
- Develop incrementally. Run, inspect, tweak a downstream node, and re-run — upstream nodes are reused from cache, so iteration is fast and cheap.
- Long jobs are safe. A 48-hour assembly that fails at hour 47 resumes near the end, not from scratch.