Skip to content

Termination and notice

A Spot Instance is by definition an ephemeral instance: it can be revoked at any time. The revocation (called reclaim) is not immediate, though — the instance first receives a notice, and only when it expires is it terminated.

The 2-minute notice

When a spot is selected for reclaim:

  1. The instance status changes to terminating and a notice email is sent to the server owner.
  2. A 120-second countdown starts: the termination moment is set to terminate_at = notified_at + 2 minutes.
  3. When the countdown expires the instance is terminated and the status becomes terminated.

These two minutes let the workload shut down gracefully — save a checkpoint, drain connections, requeue the work in progress.

Besides the email, the notice can be detected programmatically: querying GET /api/v2/servers/<servername>/spot-status the status changes to terminating and the notified_at / terminate_at fields hold the notice moment and the exact termination time respectively. This endpoint can be polled to react to the reclaim automatically.

Reclaim is irreversible

Once the instance has entered terminating, it never comes back: there is no transition from terminating back to ok. For this reason Spot Instances should only be used for workloads designed to be interruptible (periodic checkpoints, work queues with retries, redundant workers).

Spot Instance states

State Meaning
ok Active and reclaimable. This is the initial state after creation.
terminating Notice sent: termination scheduled within 2 minutes. Not reversible.
terminated Terminated, either by completed reclaim or by user deletion.
promoted Promoted to on-demand: no longer subject to reclaim.

The status record survives the termination of the server: the status remains queryable via GET .../spot-status even after the instance has been terminated.

How to avoid termination

While the status is ok, the instance can be promoted to on-demand: the resource is locked and the instance permanently leaves the reclaimable pool. Mind the timing, though — once the notice has started (terminating) promotion is no longer possible.

Checking the status