Before compiling LTP-PIC, be sure that you are able to link to an appropriate installation of the linear algebra package Hypre

For instructions on how to install Hypre on your local Linux system or on some specific supercomputers (for CPU or GPU) see here

To link to the installation of Hypre be sure to set the appropriate environment variable via:

export HYPRE=<hypre-dir>/src/hypre

To save time for future compilations you can also add this line to your .bashrc or .bash_profile files in your home directory. This will create the environment variable every time you launch the terminal, or log into the cluster (i.e. start the bash environment).

To obtain a copy of LTP-PIC you can clone the GitHub repository to your desired directory:

git clone https://github.com/luohancfd/ltp_pic.git <dir-name>

Next navigate into the top level directory. If you did not set <dir-name> then the default name is ltp_pic.

By default, you should be located on the master branch. To check that this is the case run:

git branch
> * master

Compiler flag options

Compilation options for LTP-PIC are set in the Makefile within the top level directory.

The compiler executable is set as CC = mpic++ which is the OpenMPI C++ compiler. You may need to edit this line accordingly if you are using an MPI flavour other than OpenMPI.

Otherwise the only other line which should be edited is the COPTS line. Various lines have been included for the different systems which LTP-PIC has been used on. The default COPTS (that is uncommented, or without a # in front) is set up for execution on a local Linux system. You can comment out this line and uncomment another line for your given system, or set up a completely new line.

There are also two core options which can be set by the user which effect the physics and numerical methods used in the PIC code. The first is the dimension of the simulation. Accordingly the options are:

• For 2D simulations, set: -DTWO_DIM
• For 3D simulations, set: -DTHREE_DIM

You can also set the order of interpolation between the particles and the grid:

• For 0th order (nearest grid point) interpolation, set: -DINTERP_ZERO
• For 1st order (linear) interpolation, set: -DINTERP_ONE
• For Energy Conserving interpolation, set: -DENERGY_CONSERVING. NOTE: This is currently only available on the dev branch. Navigate here by using git checkout dev.

You must set one, and only one, of each of these settings, otherwise LTP-PIC will not work.

Compiling LTP-PIC on a Linux System

These instructions are for compiling on a generic Linux based system with an installation of OpenMPI.

The default COPTS in the Makefile line is prepared for running on a Linux system and should read:

COPTS = -g -w -O3 -march=native -fopenmp -DTWO_DIM -DINTERP_ONE -Drestrict=__restrict__

Adjust your compiler flags for the dimension of the simulation and the order of interpolation accordingly.

To compile the code simply run:

make clean
make

This generates the LTP-PIC executable called pic.

Compiling LTP-PIC on Stellar

These instructions are for compiling on the Stellar computer system at Princeton University.

Begin by loading the compiler and OpenMPI modules:

module load intel/2021.1.2
module load openmpi/intel-2021.1/4.1.2

It is necessary to modfy the COPTS line in the Makefile. Comment out the default line and uncomment one of the lines prepared for Stellar. For example, if you want to run 3D simulations using CPUs, ensure the following line is uncommented:

COPTS = -g -O3 -xhost -Wno-unknown-pragmas -restrict -qopenmp -DTHREE_DIM -DINTERP_ONE

Adjust your compiler flags for the dimension of the simulation and the order of interpolation accordingly.

To compile the code simply run:

make clean
make

Compiling LTP-PIC on Traverse with CPU+GPU

These instructions are for compiling on the Traverse computer system at Princeton University.

Begin by loading the compiler, OpenMPI and Cuda Toolkit modules:

module load nvhpc/22.5
module load openmpi/nvhpc-22.5/4.1.3/64
module load cudatoolkit/11.7

It is necessary to modfy the COPTS line in the Makefile. Comment out the default line and uncomment one of the lines prepared for Traverse. For example, if you want to run 3D simulations using GPUs, ensure the following line is uncommented:

COPTS     = -g -w -fast -mp -acc=gpu -cudalib=cusparse,cublas,curand -DTHREE_DIM -DINTERP_ONE

If you do not want to use GPU's simple remove the -acc=gpu line, i.e. for the same models COPTS should read:

COPTS     = -g -w -fast -mp -cudalib=cusparse,cublas,curand -DTHREE_DIM -DINTERP_ONE

Adjust your compiler flags for the dimension of the simulation and the order of interpolation accordingly.

To compile the code simply run:

make clean
make

Compiling LTP-PIC on Perlmutter

These instructions are for compiling on the DOE's Perlmutter computer system at the National Energy Research Scientific Computing center. Perlmutter has two partitions, one with heterogeneous CPU+GPU nodes, and one with CPU only nodes.

Compiling LTP-PIC for CPU+GPU partition Perlmutter

Begin by swapping the compiler modules loading the compiler and OpenMPI modules:

module swap PrgEnv-gnu/8.3.3 PrgEnv-nvidia/8.3.3

It is necessary to modify the default Makefile in two ways. First we need to make sure that the compiler targets the correct wrappers for the C++ Cray compiler. The CC line should read:

CC = CC

It is also necessary to modfy the COPTS line. Comment out the default line and uncomment one of the lines prepared for Perlmutter. For example, if you want to run 3D simulations, ensure the following line is uncommented:

COPTS = -g -w -fast -mp -acc=gpu -target-accel=nvidia80 -cudalib=cusparse,cublas,curand -DTHREE_DIM -DINTERP_ONE

You can of course adjust your compiler flags for the dimension of the simulation and the order of interpolation accordingly.

To compile the code then simply run:

make clean
make

Compiling LTP-PIC for CPU partition on Perlmutter

Begin by swapping the compiler modules loading the compiler and OpenMPI modules:

module swap PrgEnv-gnu/8.3.3 PrgEnv-cray/8.3.3

It is necessary to modify the default Makefile in two ways. First we need to make sure that the compiler targets the correct wrappers for the C++ Cray compiler. The CC line should read:

CC = CC

It is also necessary to modfy the COPTS line. Comment out the default line and uncomment one of the lines prepared for Perlmutter. For example, if you want to run 3D simulations, ensure the following line is uncommented:

COPTS = -g -w -fast -fopenmp -DTHREE_DIM -DINTERP_ONE -Drestrict=__restrict__

You can of course adjust your compiler flags for the dimension of the simulation and the order of interpolation accordingly.

To compile the code then simply run:

make clean
make

Compiling LTP-PIC for CPUs on Antya

These instructions are for compiling on the Antya computer system at IPR.

Begin by loading the compiler module (the correct MPI package is automatically loaded):

module load intel-2019

It is necessary to modfy the MPI compiler name in the Makefile for the Intel compiler. Modify the line CC=mpic++ to read:

CC=mpiicpc

It is necessary to modfy the COPTS line in the Makefile. Comment out the default line and uncomment one of the lines prepared for Stellar. For example, if you want to run 2D simulations, ensure the following line is uncommented:

COPTS = -g -O3 -xhost -Wno-unknown-pragmas -restrict -qopenmp -DTWO_DIM -DINTERP_ONE

Adjust your compiler flags for the dimension of the simulation and the order of interpolation accordingly.

To compile the code simply run:

make clean
make

A Warning about OpenMP

LTP-PIC uses a mixed programming model, whereby thread-level parallelism on multicore CPUs is targetted via OpenMP and parallelism on GPUs is targeted via OpenACC. Originally the code was written to be optimized for multicore CPUs with OpenMP, however re-writing for GPU performance has led to the tradeoff that many algorithms are now less efficient on multicore-CPUs.

LTP-PIC can be compiled with and run using OpenMP, however we cannot guarantee that this will give you the best performance for your system. For now we recommend relying on purely MPI for CPU parallelism and MPI+OpenACC for multi-GPU parallelism. Best praractice is to set the environment variable export OMP_NUM_THREADS=1, to ensure OpenMP is not being called.

If you really want to use OpenMP, we recommend carefully profiling your code to ensure it will receive a performance gain.

Additional compiler flag options

To run the Landmark 2a Benchmark (i.e. Examples/input_landmark.dat) ensure the following compiler flag is set:

-DBENCH_LANDMARK

To run the Penning discharge Benchmark (i.e. Examples/input_pen_bench.dat) ensure the following compiler flag is set:

-DPEN_BENCH