Before compiling LT-PIC, be sure that you are able to link to an appropriate installation of the linear algebra package Hypre

For instructions on how to install Hypre on some specific supercomputers (for CPU or GPU) see here

To link to the installation of Hypre be sure to set the appropriate environment variable via:

export HYPRE=<hypre-dir>/src/hypre

Compiling on Traverse

These instructions are for compiling on the Traverse computer system at Princeton University. Instead of installing Hypre you can also link to currently installed versions of the package.

To link to a GPU version of Hypre use:

export HYPRE=/scratch/gpfs/apowis/hypre_sources/hypre-gpu-nvhpc/src/hypre

To link to a CPU version of Hypre use:

export HYPRE=/scratch/gpfs/apowis/hypre_sources/hypre-cpu-nvhpc/src/hypre

Compiling on Traverse for CPU + GPU

1. Clone the repository to your scratch space on Traverse.
2. Load the relevant modules: module load nvhpc/20.7, module load openmpi/cuda-11.0/nvhpc-20.7/4.0.4/64.
3. Then make the code using make.

Note: With these latest compilers the environment variable CUDA_HOME must be unset, otherwise compilation has been observed to fail.

Compiling on Traverse for CPU only

Before running make be sure to modify the Makefile as follows, comment out the line:

#COPTS     = -g -w -fast -mp -acc -ta=tesla -Mcuda -Minfo=all -DDSFMT_MEXP=19937 -DINTERP_ONE

Uncomment the line

COPTS     = -g -w -fast -mp -Minfo=all -DDSFMT_MEXP=19937 -DINTERP_ONE

1. Clone the repository to your scratch space on Traverse.
2. Be sure you are on the branch dev or equivalent.
3. Load the relevant modules: module load nvhpc/20.7, module load openmpi/nvhpc-20.7/4.0.4/64.
4. Then make the code using make.

Compiling on Perseus

These instructions are for compiling on the Perseus computer system at Princeton University. Instead of installing Hypre you can also link to currently installed versions of the package:

export HYPRE=/scratch/gpfs/apowis/hypre_sources/hypre/src/hypre

Before running make be sure to modify the Makefile as follows, comment out the line:

#COPTS = -g -w -fast -mp -acc -ta=tesla -Mcuda -Mcudalib=curand,cusparse -DDSFMT_MEXP=19937 -DINTERP_ONE #-Minfo=all -DBENCH_LANDMARK

Uncomment the line

COPTS = -g -w -fast -mp -DDSFMT_MEXP=19937 -DINTERP_ONE #-Minfo=all -DBENCH_LANDMARK

1. Clone the repository to your scratch space on Perseus.
2. Be sure you are on the branch dev or equivalent.
3. Load the relevant modules: module load intel/19.0/64/19.0.1.144, module load openmpi/intel-17.0/1.10.2/64.
4. Then make the code using make.

Compiling on Cori

These instructions are for compiling on the Cori computer system at the National Energy Research Scientific Computing Center.

You can simply load Hypre through,

module load cray-tpsl

Before running make be sure to modify the Makefile as follows,

Modify the line

CC        = mpic++

To read,

CC        = cc

Modify the line

HYPRE_DIR = $(HYPRE)

To read,

HYPRE_DIR =

Comment out the line:

#COPTS = -g -w -fast -mp -acc -ta=tesla -Mcuda -Mcudalib=curand,cusparse -DDSFMT_MEXP=19937 -DINTERP_ONE #-Minfo=all -DBENCH_LANDMARK

Uncomment the line

COPTS     = -g -w -O3 -march=native -fopenmp -DDSFMT_MEXP=19937 -DINTERP_ONE

1. Clone the repository to your scratch space on Cori.
2. Be sure you are on the branch dev or equivalent.
3. Then make the code using make.

NOTE:

1. These instructions have only been tested on the Intel Haswell nodes, it is likely that different instructions will will be required for Knights Landing.
2. We observed poor weak scaling performance of the installed Hypre on the Knights Landing nodes, therefore Haswell is recommended.
3. These compiler flags have not been optimize for the cray compilers.
4. The Intel compiler appears to be the most optimal (intel/19.0.3.199) versus the Cray and GNU compilers. This compiler is loaded by default.