Use MPI function of PHREEQCRM

[Yongqiang]

Hi PHREEQC experts,

I am coupling the phreeqcrm with a flow solver. The flow solver is also coded with MPI capability. I wonder if it is possible to only use the mpi function of PHREEQCRM while serially run the flow solver.

Thanks,
Michael

[dlparkhurst]

I do not see why you cannot use MPI in PhreeqcRM whether your solver is parallel or serial.

You would have to compile PhreeqcRM with MPI. You would also need to execute the program with mpiexec and obtain an MPI communicator to create the PhreeqcRM instance [PhreeqcRM(nxyz, communicator)].

[Yongqiang]

Thanks, David.
I am a bit confused that the mpi function of flow solver or the PHREEQCRM will be invoked when the command "mpirun -np 6 newreactivetranscode" is executed.

Regards

[dlparkhurst]

mpirun and mpiexec are synonyms; each invokes an MPI program. newreactivetranscode must use MPI. Assuming you use CMake and Visual Studio, for PhreeqcRM, you must use the MPI option in CMake when you make your VS sln file.

Does that answer your question, or is it something else?

[Yongqiang]

Thanks, David.
It partly solved my confusion. If both the PHREEQCRM and the flow solver have mpi capability, does mpirun work on the flow solver or the PHREEQCRM?
I am trying to control the mpi to be only effective to PHREEQCRM while keeping the flow solver running serially.

Thanks

[dlparkhurst]

Well, a communicator is required to create an MPI PhreeqcRM instance. That communicator may be the one that is created when mpirun is executed, which is MPI_COMM_WORLD. However, MPI has ways to make a communicator that is a subset of of MPI_COMM_WORLD, or to create a completely new communicator (I think).  So, you could create PhreeqcRM with a communicator that has any number of processes, and it may or may not be related to MPI_COMM_WORLD. In my usage, I have always used MPI_COMM_WORLD in creating a PhreeqcRM instance.

As for your MPI transport code, I don't know how it was written. PhreeqcRM has #ifdefs that can be set for compilation as OpenMP, MPI, or neither, so it could be compiled serially. I don't know whether your transport code was written this way. If you can define the communicator that is used for the transport code, then you could create a communicator that has only one process, which would make the transport code effectively serial. My guess is that it uses the MPI_COMM_WORLD that is created with mpirun, and it may be difficult to change that, if MPI_COMM_WORLD is used throughout the transport code.

Again, a guess, but the easiest way to do what you suggest may be to use mpirun with -n 1, and then generate a new communicator for PhreeqcRM that has more processes. You would have to have an argument for the number of processes, and figure out how to use MPI to generate a new communicator with that number of processes. If you can figure it out, I think it would only take a few lines of code.

But, why not run both transport and PhreeqcRM with MPI and just use the MPI_COMM_WORLD communicator for both. I don't see the problem if the transport and geochemistry calculations are truly sequential.

I parallelized both transport and geochemistry in PHAST, where each element transport equation was assigned to a process. I think I may have used a subset of MPI_COMM_WORLD if the number of processes exceeded the number of elements. If you have enough processes, then all element transport calculations are solved simultaneously. The drawback is that you cannot make use of more processes than you have elements to transport (on the transport side of the calculations). If it is the matrix solver that is parallelized, then the transport calculation for each element is parallelized, but the element-transport equations may be solved sequentially. (I guess it is possible to use a parallel solver with multiple simultaneous transport calculations if you are more clever than I.) My understanding is that matrix solvers are tricky to get large speedups by parallelization, but the parallel solution of each element transport should be nearly ideal with a roughly n-fold speedup in the transport calculations, where n is the number of element transport calculations that are needed (usually in the range of 10 to 15 elements).