Table of contents
This page contains details on how to build the Hypre package on various clusters for both CPU and GPU implementations
Building Hypre for Traverse
The Traverse cluster at Princeton University has 46 IBM POWER9 nodes with four NVIDIA V100 GPUs per node. Hypre can be built to run on the NVIDIA GPUs, however depending on the problem size, LT-PIC may run faster when relying on the CPU build of Hypre.
LT-PIC relies on the OpenACC programming standard to accelerate the code on GPUs. The PGI compilers are best suited for this standard, and therefore where possible we build Hypre using PGI. On the GPU, Hypre automatically reverts to nvcc for compilation, therefore when compiling for the GPU we build using gcc.
Building Hypre for CPU on Traverse
Updated 1/22/2021 for latest NVIDIA compilers.
Load the following modules,
module load nvhpc/20.7
module load openmpi/cuda-11.0/nvhpc-20.7/4.0.4/64
Hypre has not yet been updated to be aware of the new naming of the NVIDIA compilers, therefore we set the following environment variables to trick it in to thinking they are PGI compilers (which the NVIDIA compilers are based on).
export CC=pgcc
export CXX=pgc++
export FC=pgf77
Download the latest version of Hypre:
git clone https://github.com/hypre-space/hypre.git
Navigate to hypre/src and configure the install with the following settings:
./configure --with-MPI-include="/usr/local/openmpi/cuda-11.0/4.0.4/nvhpc207/ppc64le/include" --with-MPI-libs="mpi" --with-MPI-lib-dirs="/usr/lib64 /usr/local/nvhpc/lib64 /usr/local/nvhpc/lib64/openmpi /usr/local/openmpi/cuda-11.0/4.0.4/nvhpc207/ppc64le/lib64"
Hypre must point to the correct version of OpenMPI to compile correctly. The directories provided here are correct for the above OpenMPI module. You can also locate the appropriate include and library directories via mpicc --showme:incdirs and mpicc --showme:libdirs respectively.
Install this configuration via:
make clean
make
Building Hypre for GPU on Traverse
Updated 1/22/2021 for latest NVIDIA compilers.
Load the following modules,
module load nvhpc/20.7
module load openmpi/cuda-11.0/nvhpc-20.7/4.0.4/64
Hypre has not yet been updated to be aware of the new naming of the NVIDIA compilers, therefore we set the following environment variables to trick it in to thinking they are PGI compilers (which the NVIDIA compilers are based on).
export CC=pgcc
export CXX=pgc++
export FC=pgf77
You must also explicitly set CUDA_HOME to point to the correct CUDA version:
export CUDA_HOME=/usr/local/cuda-11.0
Download the latest version of Hypre:
git clone https://github.com/hypre-space/hypre.git
Navigate to hypre/src and configure the install with the following settings:
./configure --with-cuda HYPRE_CUDA_SM=70 -enable-gpu-aware-mpi --with-MPI-include="/usr/local/openmpi/cuda-11.0/4.0.4/nvhpc207/ppc64le/include" --with-MPI-libs="mpi" --with-MPI-lib-dirs="/usr/lib64 /usr/local/nvhpc/lib64 /usr/local/nvhpc/lib64/openmpi /usr/local/openmpi/cuda-11.0/4.0.4/nvhpc207/ppc64le/lib64"
Hypre must point to the correct version of OpenMPI to compile correctly. The directories provided here are correct for the above OpenMPI module. You can also locate the appropriate include and library directories via mpicc --showme:incdirs and mpicc --showme:libdirs respectively.
Install this configuration via:
make clean
make
Building Hypre for Perseus
The Perseus cluster at Princeton University is a 320 node Dell Beowulf cluster, each node is a 28-core Intel Broadwell chip. Hypre can be built only for the CPU, and we generally rely on the Intel suite of compilers to best target the available architecture.
To begin, load the following modules:
module load intel/19.0/64/19.0.1.144
module load openmpi/intel-17.0/1.10.2/64
Download the latest version of Hypre:
git clone https://github.com/hypre-space/hypre.git
Navigate to hypre/src and configure the install with the following settings:
./configure
Install this configuration via:
make clean
make
Building Hypre for Cori
Building Hypre for Haswell nodes on Cori
./configure CC=cc CXX=CC FC=ftn CFLAGS="-qopenmp" CXXFLAGS="-qopenmp" FCFLAGS="-qopenmp" --with-openmp
Building Hypre for KNL nodes on Cori
I ran an interactive session on the KNL nodes to compile, otherwise Hypre throws an error. Then configure with:
./configure CC=cc CXX=CC FC=ftn CFLAGS="-qopenmp" CXXFLAGS="-qopenmp" FCFLAGS="-qopenmp" --with-openmp
Building Hypre for Ascent
On ASCENT (ascent.olcf.ornl.gov) I used the following modules:
module list
Currently Loaded Modules:
1) lsf-tools/2.0 4) cuda/10.1.243 7) gcc/6.4.0
2) DefApps 5) forge/19.1.2 8) spectrum-mpi/10.3.1.2-20200121
3) git/2.20.1 6) python/3.6.6-anaconda3-5.3.0
I copied the version of HYPRE that Alex had on Traverse, which is version 2.19.0 plus a few patches that were sent by Ruipeng Li from LLNL.
cd hypre-master/src
CC=mpicc CXX=mpicxx CUDA_PATH=${CUDAPATH} ./configure HYPRE_CUDA_SM=70 --with-MPI --with-cuda --enable-unified-memory --disable-fortran --enable-debug
make
PROBLEM WHEN SWITCHING FROM GCC TO PGI
Since HYPRE requires "nvcc" to build, along with a C++11 compliant version of g++, I loaded the default gcc module on Ascent, which is gcc 6.4.0:
module load gcc
gcc --version
gcc (GCC) 6.4.0
on Ascent: build HYPRE without managed memory
cd /gpfs/wolf/gen137/proj-shared/ltp-pic/software/hypre-master/src
CC=mpicc CXX=mpicxx CUDA_PATH=${CUDAPATH} ./configure HYPRE_CUDA_SM=70 --with-MPI --with-cuda --disable-fortran --prefix=/gpfs/wolf/gen137/proj-shared/ltp-pic/software/GCC-6_GPU_MANAGED
make
make install
To compile ltp_pic I need to use PGI so I swapped the modules
module swap gcc pgi
mpic++ -show
pgc++ -I/autofs/nccsopen-svm1_sw/ascent/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/pgi-19.9/spectrum-mpi-10.3.1.2-20200121-zy6l7p3w4q2ns7i2zutzng7ex3drw656/include -lpthread -L/autofs/nccsopen-svm1_sw/ascent/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/pgi-19.9/spectrum-mpi-10.3.1.2-20200121-zy6l7p3w4q2ns7i2zutzng7ex3drw656/lib -lmpiprofilesupport -lmpi_ibm
However, by swapping the gcc-6.4.0 module for pgi-19.9, the version of gcc reverts to the OS version, which is 4.8.5
gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-37)
That won't work since we need the same version of gcc/g++ as what we used to build HYPRE, and on top of it gcc 4.8.5 is not compliant with C++11. So Mat Colgrove showed us how to tell PGI to use a different version of gcc. One needs to use the "makelocalrc" command and set the PGI_LOCALRC environment variable:
cd /gpfs/wolf/gen137/proj-shared/ltp-pic/ethier/ltp_pic_gpu
makelocalrc -x -d . -gcc /sw/ascent/gcc/6.4.0/bin/gcc -g++ /sw/ascent/gcc/6.4.0/bin/g++ -g77 /sw/ascent/gcc/6.4.0/bin/gfortran
export PGI_LOCALRC=/gpfs/wolf/gen137/proj-shared/ltp-pic/ethier/ltp_pic_gpu/localrc
This last line tells the PGI compiler where to find the localrc file that contains the information about which version of gcc to use and where to find it.
The code now compiled fine but failed to run due to the following error:
./pic: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./pic)
This is related to the dynamically linked libraries so we still need to specify where to find the gcc 6.4.0 libraries at run time. This is accomplished by setting LD_LIBRARY_PATH in the job script before jsrun:
module show gcc
----------------------------------------------------------------------------
/sw/ascent/modulefiles/20180914/site/linux-rhel7-ppc64le/Core/gcc/6.4.0.lua:
----------------------------------------------------------------------------
family("compiler")
help([[GCC Compiler]])
whatis("Description: ")
prepend_path("MODULEPATH","/sw/ascent/modulefiles/20180914/site/linux-rhel7-ppc64le/gcc/6.4.0")
prepend_path("PATH","/sw/ascent/gcc/6.4.0/bin")
prepend_path("MANPATH","/sw/ascent/gcc/6.4.0/share/man")
prepend_path("LD_LIBRARY_PATH","/sw/ascent/gcc/6.4.0/lib64")
setenv("OLCF_GCC_ROOT","/sw/ascent/gcc/6.4.0")
Using module show gcc show how to set LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/sw/ascent/gcc/6.4.0/lib64:${LD_LIBRARY_PATH}
That solved the problem.