Checkpointing

about 4 minutesStudents, Staffchecked 13 November 2024

Checkpointing a job means that you frequently save the job state so that you can resume computation from the last checkpoint in case of a crash. On this page we provide some useful information for making your own code checkpoint-able.

Why checkpointing is important

Imagine a job is already running for several hours when an event occurs which leads to the abortion of the job. Such events can be:

Exceeding the time limit
Exceeding allocated memory
Job gets preempted by another job (gpu partition only!)
Node failure

General recipe for checkpointing your own code

Introducing checkpointing logic in your code consists of 3 steps

Look for a state file containing a previously saved state.
If a state file exists, then restore the state. Else, start from scratch.
Periodically save the state.

Using UNIX signals

You can save the state of your job at specific points in time, after certain iterations, or at whatever event you choose to trigger a state saving. You can also trap specific UNIX signals and act as soon as the signal occurs. The following table lists common signals that you might want to trap in your program:

Signal Name	Signal Number	Description	Default Disposition
SIGTERM	15	SIGTERM initiates the termination of a process	Term - Terminate the process
SIGCONT	18	SIGCONT continues a stopped process	Cont - Continue the process if stopped
SIGUSR1	10	User-defined signals. SIGUSR1/SIGUSR2 are never sent by the kernel	Term - Terminate the process
SIGUSR2	12	User-defined signals. SIGUSR1/SIGUSR2 are never sent by the kernel	Term - Terminate the process

SLURM sends SIGCONT followed by SIGTERM just before a job is canceled. Trapping the signal (e.g. SIGTERM) gives you 60 seconds for housekeeping tasks, e.g. save current state. At the latest after that your job is canceled with SIGKILL. This is true for jobs canceled by the owner using scancel and jobs canceled by SLURM, e.g. because of exceeding time limit.

Register a signal handler for a UNIX signal

The following examples show how to register a signal handler in different languages, but omit the logic for creating a checkpoint and restart a job from an existing checkpoint. We will provide a working example further down below on this page.

Bash

1
#!/bin/bash
2

3
function signal_handler {
4
  # Save program state and exit
5
  (...)
6
  exit
7
}
8

9
trap signal_handler TERM
10
(...)

C/C++

1
#include <signal.h>  // C
2
#include <csignal>   // C++
3

4
void signal_handler(int signal) {
5
  // Save program state and exit
6
  (...)
7
 exit(0);
8
}
9

10
// Register signal handler for SIGTERM
11
signal(SIGTERM, signal_handler);  // signal_handler: function to handle signal
12
(...)

Python

1
#! /usr/bin/env python
2
import signal
3
import sys
4

5
def signal_handler(sig, frame):
6
  # Save program state and exit
7
  (...)
8
  sys.exit(0)
9

10
signal.signal(signal.SIGTERM, signal_handler)
11
(...)

Signaling checkpoint creation without canceling the job

SLURM distinguishes between the job script, its child processes and job steps. Job steps are launched using srun.

All applications which should reveive more signals than the default SIGERM at the end of the job, does need to be started using srun. Then signals (here SIGUSR1) can be send using:

1
scancel --signal=USR1 <jobID>

If you need/want to handle signals with the batch script, add --batch (signals only to the batch script) or --full (signal to all steps and the batch script), e.g.

1
scancel --full --signal=USR1 <jobid>

Therewith, you can use a UNIX signal to trigger the creation of a checkpoint of a running job. For example, consider a job that traps SIGUSR1 and saves intermediate results as soon as the signal occurs. You can then create a checkpoint by signaling SIGUSR1 to the job.