In this section we assume that you have already compiled LTP-PIC with a suitable installation of the Hypre linear algebra package and prepared a well formatted input file.
- For instructions on compiling LTP-PIC see here.
- For instructions on installing Hypre see here.
- For instructions on preparing an input file see here.
Running LTP-PIC from the command line
These instructions are for running LTP-PIC from the command line on a generic Linux based system with an installation of OpenMPI.
First, ensure you are located in the directory with the LTP-PIC pic executable.
To execute the code with N processors run:
mpiexec -n N ./pic <input_file_path>
The <input_file_path> must point to a valid LTP-PIC input file with the .dat suffix. This file does not need to be located within the same directory as the executible, however the output from the code will be saved in the same directory as the executible.
Reminder: The number of processors called for the run must be equal to that requested in the input file. In 2D, this can be computed from the paramters num_process_x * num_process_y. In 3D, this can be computed from the paramters num_process_x * num_process_y * num_process_z.
Running LTP-PIC from a batch script
Located within the directory BatchScripts/ are numerous slurm based scripts for executing the code on various clusters. The general naming convention for these are b<cluster_name>. For details on how to modify the appropriate batch script refer to both the documentation for your cluster and the Slurm documentation.
The relevant scripts must be copied to the top directory before submission to the slurm job scheduler.
The only additional line which may need to be modified by the user that is not detailed within the above documentation is the execution line for the code. Generally this has the format:
srun ./pic <input_file_path>
The user will need to specify the correct <input_file_path> for the given run.
Restarting LTP-PIC from a checkpoint
During a checkpoint, LTP-PIC saves files to the Checkpoint/ directory containing the phase space information for all particles within the simulation. One file is saved for each MPI task called during the simulation run. Therefore when restarting the simulation you must call the same number of processors as the previous run. Only checkpoint data from the latest output is saved. Each new checkpoint overwrites the previous data.
To restart the simulation from a checkpoint simply add the -restart flag to the end of the execution line. For example, when executing from the linux command line, simply run:
mpiexec -n N ./pic <input_file_path> -restart
Or if from a slurm batch script, modify the relevant line in the script to read:
srun ./pic <input_file_path> -restart
Warning: If you accidentally re-submit the run without the -restart flag it will overwrite the Output and Checkpoint directories, effectively deleting all of the data which has been saved up to this point. The simulation will then start from scratch.
To the developers knowledge, the only parameters in the input file which can be safely adjusted for a restarted run are:
number_steps
print_interval
checkpoint_interval
For a full description of these parameters see here.
Note: If the job quits while a checkpoint is being written this could result in a failed checkpoint output and mean that you are unable to restart the simulation. Since checkpointing is infrequent, this occurance should be rare, however the developers are working to fix this potential bug.