This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

To work with these tools, you must install Hypre in the format you plan to run the following tests to run all the following codes. Please refer to the page Building HYPRE. These codes are built to run both the CPU and GPU versions. The only difference between the two code versions is a compiler flag at the top (0 - CPU, 1 - GPU). All the tests performed below are relevant to Traverse.

Makefile configurations

Before running any code in this suite of tools, the user must modify the included makefile and makefile.config files to match their system configuration. This is divided into two sections: CPU and GPU makefiles. To find the makefiles to modify, navigate to any of the tools, and open the folder GPU or CPU to find the version you would like to change. All the makefiles for the CPU codes are the same, and all the GPU codes have identical makefiles.

CPU Makefile

This makefile is simple to configure for a user's system. All that must be done is to link to your version of HYPRE. The CPU version uses the environment variable HYPRE to find this installation, so to check this, run the command printenv HYPRE. If this is not the location of your hypre installation, run the command:

export HYPRE=<hypre location>/src/hypre

Replace <hypre location> with the address of your installation of the CPU version of HYPRE. You could add this statement to your .bashrc to make compiling these codes easier. This location can also be hardcoded in the makefiles by changing the line

HYPRE_DIR = $(HYPRE)

to

HYPRE_DIR = <hypre location>/src/hypre

This makefile can be copied into all codes which rely on the CPU version of HYPRE

GPU Makefile

For modifying the GPU makefile, there is also the relevant file makefile.config. Both must be modified for the codes to compile properly.

First, within makefile, the user must modify the line HYPRE_DIR = /scratch/gpfs/apowis/hypre_sources/hypre-gpu/src/hypre to match their location of the GPU installation of hypre. It should look like HYPRE_DIR = <hypre location>/src/hypre

Then, within the makefile.config, there are many changes to be made. The variables prefix, HYPRE_SRC_TOP_DIR, HYPRE_BUIlD_DIR, HYPRE_DISTRIBUTED_LS_DIR, HYPRE_EUCLID_FILES, HYPRE_PARASAILS_FILES, and HYPRE_PILUT_FILES must all be changed to match the user's installation of HYPRE in the same way it is modified in the makefile.

Furthermore, the version of CUDA used during the installation of HYPRE must also be defined in the variables LDFLAGS, LIBS, INCLUDES, HYPRE_CUDA_INCL, and HYPRE_CUDA_LIBS.

The GPU makefiles can be very tricky to get working.

Scalability

This tool is used to measure the scalability of the field solver in LTP-PIC. The field solver is recreated in an independent code to simulate behavior within LTP-PIC. Each code solves Poisson's equation with various right-hand sides. Many different ways of investigating the field solver are presented within the tool.

• Analytical - Solves the equation ▽²f(x,y) = sin(kπx)sin(kπy) in 2D or ▽²f(x,y,z) = sin(kπx)sin(kπy)sin(kπz) in 3D. The values of k are scanned from k=1 up to the nyquist limit. The numerical solution is compared to the analytical, which the error is found to be dominated by the central difference approximation and decreases with grid size. This code is used to build knowledge of how HYPRE works and familiarize the user with how these scaling tests work. The analytical RHS should not be used to predict the performance of the field solver within LTP-PIC.
• Random Right Hand Side - Solves the equation ▽²f(x,y) = rand(x,y) in 2D or ▽²f(x,y,z) = rand(x,y,z) in 3D. This represents the worst-case scenario for a PIC code, where charges are randomly distributed across the domain. In these codes, the random numbers range from -1 to 1.
• Simulation - Solves the equation ▽²E(x,y) = ρ(x,y)/ε₀, using the values of ρ(x,y)/ε₀ from a prior run in LTP-PIC. Successive itterations of the solver use the values of the next timestep, thus this is the optimal measure of performance of the solvers within LTP-PIC.

As a test, I will walk you through the way to run the CPU 2D analytical code within the scalability tool.

1. Create your personal makefile, configured it to your machine, and install CPU HYPRE.
2. Build the program using the command make.
3. Get help by running ./ex3 -help
4. Run a configuration of the program using the command mpirun -np 1 ex3 -n 30 -iter 1 -solver 1 -k 1.
5. If using Slurm, open btrav and modify the line which begins with srun to read srun -n 1 ./ex3 -n 30 -iter 1 -solver 1 -k 1.
6. If using Slurm, submit the job using sbatch btrav.

Running all the other programs work the same way. First, build the file, then execute it using the flags to specify run parameters. To change the default parameters, navigate to ex3.c and find the variables nvec, kvec, and n_iter. It is important to note that the flag -n must be the first flag for it to work. The flags represent the following:

-n = number of cells on 1 axis -k = the value of k used in sin(kπx) -solver = specifies one solver to use, ranging from 0 to 9. The matching numbers to the solver can be found in solver.c or by using ./ex3 help -iter = the number of iterations that the solver uses for timing statistics -debug = enters debug mode, printing out many checkpoints along the way.

After running a program, it generates an associated *.txt file for that run. This file is used for plotting using programs written in python.

Modifying LTP-PIC to record the RHS vector

If the user is interested in using the LTP-PIC simulation feature, they must record the RHS vectors from within LTP-PIC. This can be done by inserting the following code into field.c in the function MapChargeToHypre().

if (1)
{
   if (timer->step <= 50)
   {
      int i, n;
      char buff[0x100];
      FILE *fptr;
      n = nx;

      sprintf(buff, "timesteps/RHSn%its%i.txt", n, timer->step);
      fptr = fopen(buff, "w");
      for (i = 0; i < domain->ngrid; i++)
         fprintf(fptr, "%e\n", values[i]);
      fclose(fptr);
   }
}

Including Additional Solvers

These codes are structured in a way such that only the solver.c function must be modified to add additional solvers. There are many sections of if blocks that specify solver_id. All you have to do is include another if statement in each if block with the solver of your choice. Note, each solver_id is successive and starts from 0. You may need to re-order the solvers with respect to that list. You must also change the range of solverStart and solverEnd in ex3.c too.

Solver Performance with Grid Size

To generate the *.txt files needed for the strong scaling plots, the codes from the CPU and GPU versions of 2D and 3D random RHS must be run using the default values of n defined already. One has to navigate to that folder, open it in the terminal, make the program, then run it using srun -n 1 ex3 -iter 10 or mpirun -np 1 ex3 -iter 10. This generates an associated *.txt file from the run. This name must be changed by removing the number before CPU or GPU to read RunStats_CPU or GPU. This must be done for all 4 codes CPU/GPU and 2D/3D. Then, copy the text files to the folder Scaling Tests Random 2D&3D. Run the python code StrongScaling.py and it will generate all the figures and save them in the folder.

Examples of the strong scaling of the field solver on Traverse are shown below.

Weak Scaling

To generate the *.txt files needed for the weak scaling plots, we change the number of processors used while maintaining -n 1000 in 2D and -n 100 in 3D so that each process sees 10⁶ cells. These codes are run for CPU/GPU and 2D/3D for comparison.

First, navigate to the code you would like to run. Then configure your system to use the requested number of processors. This can be set using slurm by configuring btrav to read srun -n ### ./ex3 -n 100 -iter 10, where ### is the node size. You must request equal or greater resources from the job manager from the #SBATCH parameters. The codes must rely on a square grid, so only squares or cubes of numbers are allowed.

Run this code for all values of ### which you would like. It is recommended that ### is the same for all 2D codes and is also the same across the 3D runs.

Once the *.txt files have been gathered, put them in the folder Scaling Tests Random 2D&3D and run the program WeakScaling.py. This will generate and save the figures requested.

Examples of the weak scaling of the field solver on Traverse are shown below using a grid size of 10⁶ cells per process.

Optimization

These codes are intended for a user interested in optimizing their field solver for the particular hardware configuration they are using. These optimizations are specific to each hardware configuration (number of nodes, CPU/GPU HYPRE, number of cells, 2D/3D). There are included optimizations for each solver, which were done on GPU in 3D for n=100. Still, a user looking to minimize computation time should run this test and use the optimal parameters for their configuration.

These optimization codes work similarly to the scaling ones, except they scan the different parameters used by the solvers and preconditioners. Building these codes works identically to the codes used for scaling tests. Two types of codes can be used for optimizations.

• If you have RHS vectors from LTP-PIC, the simulation codes output the most representative optimization for the problem it is looking at. This is useful if you have a specific problem in mind and can generate 10 or so timesteps as a preliminary test.
• If you do not have the RHS vectors from LTP-PIC, the random RHS is the worst-case scenario for the field solver to be optimized. An advantage of this code is that it does not rely on running LTP-PIC before optimizing.

To change what parameters are scanned, change the vectors at the beginning of the code. Once the program is run, there will be a *.txt file that recorded all the run data. This file can be copied to the folder Parameter Scans and analyzed with the code ParameterScan.py. Note, the file's name will have to be changed in the fifth line data = np.genfromtxt(). This code can output some useful plots and will print out the optimal parameters to be used.

Although the included code prints the optimal solver to use with its parameters, shown below is a plot of the optimization configuration for PFMG in 3D on GPU for 10⁶ grid points.

List of Directories

• Scalability

1. 2D_analytical - Uses an analytical solution to Poisson's equation in 2D to check understanding.
2. 2D_randRHS - Uses random numbers for the right-hand side of Poisson's equation to simulate the performance of 2D LTP-PIC.
3. 2D_simulation - Uses real data from 2D LTP-PIC for the right-hand side of Poisson's equation.
4. 3D_analytical - Uses an analytical solution to Poisson's equation in 2D to check understanding.
5. 3D_randRHS - Uses random numbers for the right-hand side of Poisson's equation to simulate the performance of 3D LTP-PIC.
6. 3D_simulation - Uses real data from 3D LTP-PIC for the right-hand side of Poisson's equation.
7. Scaling Tests Random 2D&3D - Where to copy all the outputs of the computations. Contains the files StrongScaling.py and WeakScaling.py.
8. 1DataSetPlots.py

*Optimization

1. 2D_randRHS - Uses random numbers for the right-hand side of Poisson's equation to simulate the performance of 2D LTP-PIC.
2. 2D_simulation - Uses real data from 2D LTP-PIC for the right-hand side of Poisson's equation.
3. 3D_randRHS - Uses random numbers for the right-hand side of Poisson's equation to simulate the performance of 3D LTP-PIC.
4. 3D_simulation - Uses real data from 3D LTP-PIC for the right-hand side of Poisson's equation.
5. Parameter Scans - Where to copy all the outputs of the computations. Contains the file ParameterScan.py which is to be run to process all the data. This will output the optimal solver to use in your situation.