Table of contents
Checkpointing
LTP-PIC saves checkpoints which act as restart points for simulations which do not complete within the time frame of their allocated resources. Checkpoints are saved in to the Checkpoint directory which is created at the beginning of each runtime. A checkpoint consists of a file called checkpoint_data.dat which contains information about the checkpoint (currently this is just the simulation step at the time of checkpointing) as well as individual files containing the state of each species on each MPI process. That is, each MPI task prints the entire list of particles it owns (across all OpenMP threads or GPUs) for each species.
The naming convention for these files is,
p<mpi_rank>_s<species_id>.dat
Such that p004_s2.dat contains a list of all particles of species 2 which were owned by MPI task 4 at the time of checkpointing.
The interval over which checkpoints are saved can be set by the checkpoint_interval in the input file.
Restarts
A restart is an initialization of a simulation from the checkpoint data in the Checkpoint directory, as opposed to a new simulation. You still need to provide the same input file (input.dat) as was given for the simulation which produced that checkpoint in the first place.
To perform a restart the -restart flag must follow ./pic in the execution line.
For example, if a simulation was initially run via a slurm batch script with the execution line:
srun ./pic
A restart can be performed by modifying this to read:
srun ./pic -restart
WARNING: If you attempt to restart the simulation from a checkpoint without the -restart flag the default behaviour is to delete the Checkpoint directory and all of the output data in Output, and then start a fresh simulation. That is, you will lose all information of your previous simulation.
Note: There is a known error in that if the simulation terminates while a checkpoint is being written it will likely be unable to restart from that point. While checkpointing is infrequent, and therefore this is likely a rare occurrence, this is a future bug to be fixed.