Fluent_21_2_0 addition from preCICE Workshop 2022

README.md

@@ -2,30 +2,164 @@
 *Developed by Bernhard Gatzhammer and valid for FLUENT 19.5 and preCICE v2.2 on Ubuntu 20.04*
 *Updated and maintained done by Ishaan Desai*
 
+The Fluent preCICE adapter operates using Fluent's User-Defined Function (UDF) feature. UDFs are
+functions written in the C programming language that are dynamically loaded with the Fluent
+solver and can be used to enhance and generalize its standard features. For example, UDFs can
+be used to:
+
+* Customize boundary conditions, material property definitions, or source functions
+* Customize different numerical/physics models being employed: multiphase mixture models,
+  discrete phase models, radiation models, chemical reaction models, diffusivity models, etc.
+* Execute code at different stages of a solution: at solution initialization, at every iteration,
+  upon reaching convergence, etc.
+* Many other things, as long as one can code it in C
+
+## 0. Fluent UDF Requirements
+
+Details about UDFs can be found in the UDF Manual provided by ANSYS Fluent. A summary of the
+major requirments follows.
+
+### 0.1 File requirements
+
+UDFs are identified by a .c file extension (for example fsi_udf.c). UDFs must be defined using 
+DEFINE macros supplied by Fluent. These macros are pre-defined functions that access the Fluent
+solver and perform other tasks. The .c file containing the UDFs (fsi_udf.c) must contain
+an include statement for a udf header file (#include "udf.h"). The udf header file is provided
+with the Fluent application and will be found upon UDF execution. It contains the DEFINE macros,
+among other things.
+
+Beyond this, the use of UDFs can be very general. For example, this preCICE adapter uses a very
+simple UDF file (fsi_udf.c) to handle fluid structure interaction coupling. The file contains only
+4 DEFINE macros, each containing a single function that is sourced from the fsi header file
+(fsi.h) and are written in fsi.c.
+
+### 0.2 Build requirements
+
+The user-written source code for the UDFs can either be compiled or interpreted in ANSYS Fluent
+through the Fluent GUI. We are using external functions from the preCICE source code, so we have
+to compile our UDFs. There appears to be no way to inform Fluent of the existance of external 
+libraries when compiling through the GUI. We need some preCICE functions in our UDFs, so we are
+going to have to compile these UDFs outside of the Fluent GUI. When one uses the GUI to compile
+Fluent UDFs, though, it dynamically writes files based on user inputs: a Makefile, a "user.udf" 
+text file, and a udf_names.c file. Fluent also requires a certain directory structure wherein
+ specific files must be placed. If this directory structure isn't strictly followed the UDFs
+ (even if compiled correctly) will not be loaded into the simulation at run time.
+
+### 0.3 Directory Structure
+
+The end result of compiling the UDF(s) for use in a Fluent simulation is a UDF shared library
+called "libudf.so". This library and the compiled code and source code used to build it must
+exist in a specific direcotry structure. The same direcotry that contains
+the *.cas file should contain a directory in it called "libudf". Within this directory, a
+sub-directory called "lnamd64" must exist. The name "lnamd64" actually depends on your system
+architecture, but because we are building this on a Linux system, we'll stick with "lnamd64".
+ If the Fluent run is to be executed in serial, the compiled library (libudf.so) should exist
+ within a sub-directory of "lnamd64". The name of the sub-directory is dependent on whether
+ the simulation is 2D or 3D and whether it is run in single precision or double precision.
+ The following list contains the names for the different simulations that can be run:
+
+- 2d: Two dimensional, single precision simulation
+- 3d: Three dimensional, single precision simulation
+- 2ddp: Two dimensional, double precision simulation
+- 3ddp: Three dimensional, double precision simulation
+
+If the simulation is to be run in parallel, two sub-directories of "lnamd64" need to exist, one
+with a "_host" suffix, and one with a "_node" suffix. A copy of the library libudf.so needs to
+exist in each sub-directory
+
+To summarize, the following directory structure needs to exist for a 2D, single precision
+FSI simulation run in serial:
+
+fluent.cas  
+libudf  
+--lnamd64  
+----2d  
+------fsi_udf.c: user-written UDF file containing Fleunt's DEFINE Macros  
+------fsi.c: user-written UDF file containing custom C-funtions that employ preCICE code  
+------fsi.h: user-written header file for fsi.c  
+------user.udf: text file used to define file names to be compiled (fsi_udf.c, fsi.c, and fsi.h); sourced by makefile to user-written header file for fsi.c; user may have to edit this file  
+------udf_names.c: auto-generated source code file produced by Fluent GUI; DO NOT EDIT THIS FILE  
+------makefile: instructions to create proper compile commands; may have to be edited by user to include proper directories  
+------fsi_udf.o: compiled object file from fsi_udf.c  
+------fsi.o: compiled object file from fsi.c  
+------udf_names.o: compiled object file from udf_names.c  
+------libudf.so: shared library file that Fluent uses  
+
+And the following directory structure needs to exist for a 2D, double precision simulation
+run in parallel:
+
+
+fluent.cas  
+libudf  
+--lnamd64  
+----2ddp_host  
+------fsi_udf.c  
+------fsi.c  
+------fsi.h  
+------user.udf  
+------udf_names.c  
+------makefile  
+------fsi_udf.o  
+------fsi.o  
+------udf_names.o  
+------libudf.so  
+----2ddp_node  
+------fsi_udf.c  
+------fsi.c  
+------fsi.h  
+------user.udf  
+------udf_names.c  
+------makefile  
+------fsi_udf.o  
+------fsi.o  
+------udf_names.o  
+------libudf.so  
+
+Please note that the "_node" directory and "_host" directory are copies of one another. Also, the
+makefile, source files, and intermediary object files (*.o) are not necessarily required to
+be in this directory structure. They are kept here just for convenience. The libudf.so file is
+the only thing required to be read by Fluent.
+
 ## 1. How to build the Fluent-preCICE adapter:
-  * Adapt `lnamd64/2ddp_host/user.udf` line 1 "CSOURCES=...": There are several main udf files
-  * The variable SOURCES needs one main udf file and the correspoding .c files:
-    + fsi_udf.c: This is the main ANSYS readable file used by Fluent to call functions during each iteration
-                 for FSI simulations. This ANSYS UDF file needs fsi.c and fsi.h for execution
-  * Adapt `lnamd64/2d_host/user.udf` line 3 `FLUENT_INC = ...` to the fluent installation
-    folder which will be of the type `./ansys_inc/v195/fluent`
-  * Repeat the above steps for `lnamd64/2d_node` folder in the exact same manner
-  * Adapt the path of the python library in `src/makefile` line 19.  
-    **NOTE**: Python shared library is available within the ANSYS installation. Example path is: `/ansys_inc/v195/commonfiles/CPython/2_7_15/linx64/Release/python/lib/libpython2.7.so`   
-  * Update the ANSYS Release version in `src/makefile` line 26
-    Example of line 26: `RELEASE=19.5.0`
-  * Type: `make "FLUENT_ARCH=lnamd64"` to start the build
-  * Use `make clean` to clean build process
+
+Given the above 2D, double precicions, parallel run directory structure:
+
+  * Adapt `lnamd64/2ddp_host/user.udf`
+    + change "CSOURCES=..." to include a space-separated list of *.c source files to be compiled;
+      for the FSI case we're building this should be fsi_udf.c and fsi.c
+    + change "HSOURCES=..." to include a space-separated list of *.h source heeader files to be
+      compiled; for the FSI case we're building this should be fsi.h
+    + change "FLUENT_INC= " to point to the Fluent install directory. One location may be
+      `/opt/Software/ansys/v202/fluent`. Locations should be of the type `./ansys_inc/v195/fluent`
+  * Adapt `lnamd64/2ddp_host/makefile`
+    +  change `USER_OBJECTS` variable (line 20) to be a space separated list of the absolute
+       path to libprecice.so and the python library shipped with Fluent
+    + the libprecice.so file can be found in the preCICE install location; for example,
+      `install/precice/2.3.0/lib64/libprecice.so`
+    + the python library can be found in the Fluent installation files; for example,
+      `/opt/Software/ansys/v202/commonfiles/CPython/3_7/linx64/Release/python/lib/libpython3.so`
+    + change `RELEASE` variable to be the ANSYS release version; for example,
+      `RELEASE=20.2.0`
+  * build libudf.so: type 'make "FLUENT_ARCH=lnamd64"'
+  * clean the build using "make clean"
+  * copy ALL of the contents of `lnamd64/2ddp_host/` to `lnamd64/2ddp_node/`
 
 ## 2. Running FLUENT
 
 ### 2.1 Installation using ANSYS GUI
 
 **General remarks**
 
-* Ubuntu 20.04 is not officially supported by ANSYS and hence only the FLUENT package works on this distribution. All other packages (ANSYS Workbench, etc.) do not work and hence the case setup needs to be done on a different compatible operating system. Current compatible distributions for ANSYS version 2019 R3 are: Ubuntu 16.04, CentOS 7.x, Linux Mint 18.x, Debian 9 (tested with 2019 R3, unknown for 2020 R2)
-* Generally it is recommended to only install the required packages, since the installation process might break (tested with 2019 R3)
-* Make sure to test your installation by starting fluent (see below). If fluent crashes, see the troubleshooting hints.
+* Ubuntu 20.04 is not officially supported by ANSYS and hence only the FLUENT
+  package works on this distribution. All other packages (ANSYS Workbench, etc.)
+  do not work and hence the case setup needs to be done on a different
+  compatible operating system. Current compatible distributions for ANSYS
+  version 2019 R3 are: Ubuntu 16.04, CentOS 7.x, Linux Mint 18.x, Debian 9
+  (tested with 2019 R3, unknown for 2020 R2)
+* Generally it is recommended to only install the required packages, since the
+  installation process might break (tested with 2019 R3)
+* Make sure to test your installation by starting fluent (see below). If fluent
+  crashes, see the troubleshooting hints.
 
 **ANSYS version 2020 R2 on Ubuntu 20.04**
 
@@ -39,24 +173,71 @@
 
 **ANSYS Version 2019 R3 on Ubuntu 16.04**
 
-* All packages of ANSYS Version 2019 R3 work on Ubuntu 16.04 and this [forum post](https://www.cfd-online.com/Forums/ansys/199190-ansys-18-2-ubuntu-16-04-installation-guide.html) describes the installation process.
+* All packages of ANSYS Version 2019 R3 work on Ubuntu 16.04 and this
+  [forum post](https://www.cfd-online.com/Forums/ansys/199190-ansys-18-2-ubuntu-16-04-installation-guide.html)
+  describes the installation process.
 
 **Troubleshooting**
 
-* If you try to start fluent via `fluent 2ddp` and the program exits with the error `Bad substitution`, the following [forum post}(https://www.cfd-online.com/Forums/fluent/149668-fluent-16-0-0-ubuntu-12-04-a.html) provides a solution. Short: `sudo dpkg-reconfigure dash`, answer **No** to the questions "Use dash as the default system shell (/bin/sh)?".
-* If the error: `undefined symbol: FT_Done_MM_Var` is encountered on starting FLUENT, the following [forum post](https://www.cfd-online.com/Forums/fluent/227651-fluent-ubuntu-20-04-a.html) has the solution
+* If you try to start fluent via `fluent 2ddp` and the program exits with the
+  error `Bad substitution`, the following
+  [forum post}(https://www.cfd-online.com/Forums/fluent/149668-fluent-16-0-0-ubuntu-12-04-a.html) provides a solution. Short: `sudo dpkg-reconfigure dash`, answer **No** to the questions "Use dash as the default system shell (/bin/sh)?".
+* If the error: `undefined symbol: FT_Done_MM_Var` is encountered on starting
+  FLUENT, the following
+  [forum post](https://www.cfd-online.com/Forums/fluent/227651-fluent-ubuntu-20-04-a.html)
+  has the solution
 
 ### 2.2 How to start Fluent
 
+In accordance with the [preCICE documentation](https://precice.org/installation-source-dependencies.html)
+preCICE, PETSc (if used), and all solvers should run the same MPI implementation and version. So, we need
+to make sure that the version of preCICE we are running and Fluent use the same MPI. Unfortunately, we
+don't really know what implementation (Intel, OpenMPI, etc.) or version of MPI Fluent uses. However,
+FLUENT can be made to run with any version and implementation of an MPI, within reason. Fluent does this
+using environment variables. To run with a version of OpenMPI set the environment variable `OPENMPI_ROOT`,
+and to run with a version of IntelMPI set the environment variable `INTELMPI_ROOT`. Upon execution, Fluent
+will append `/bin/mpirun` to this environment variable, so we should set whichever one we use to the
+corresponding path. For example, setting:
+
+```
+export OPENMPI_ROOT=/opt/Software/openmpi/3.1.4/mpirun
+```
+or
+```
+export INTELMPI_ROOT=/opt/Software/intel_parallel_studio/2018u4/impi/2018.4.274/intel64
+```
+
+will force Fleunt to use a system MPI rather than the MPI it's packaged with. A note to the effect
+will be reflected in Fluent's output when the simulation is run.
+
+**NOTE**
+One attempt was made to run Fluent with openmpi/4.0.0, and this failed. This is presumably
+because Fluent is calling some older MPI commands that are since deprecated
+
+**ANOTHER NOTE**
+Do not attempt to define both the `OPENMPI_ROOT` and `INTELMPI_ROOT` environment variables. This
+will just confuse things.
+
+With this environment variable in place, we can now run fluent with or without the GUI, making sure
+to define the `-mpi=` argument such that it aligns with our environment variable definition.
+
 **with GUI**
 
-* All ANSYS packages are installed in a folder `ansys_inc/` at the location defined by the user during installation. The FLUENT executable is located at `/ansys_inc/vXXX/fluent/bin`
+* All ANSYS packages are installed in a folder `ansys_inc/` at the location
+  defined by the user during installation. The FLUENT executable is located at
+  `/ansys_inc/vXXX/fluent/bin`
 
 **without GUI**
 
 * serial:   `fluent 2ddp -g < steer-fluent.txt`
 * parallel: `fluent 2ddp -g -t4 -mpi=openmpi < steer-fluent.txt` (-t4 sets 4 processes for computations, steer-fluent.txt is a driver file for Fluent and is only written for convenience)
 
+### 2.3 Line-by-Line Debugging UDFs in Fluent
+
+UDFs can be difficult to debug because they are interpreted within Fluent. Sometimes this leads
+to misleading statements and tracebacks upon crashing. Some instructions on how to use gdb to
+step through UDF code line-by-line are found [here](https://www.cfd-online.com/Wiki/Udf_debug)
+
 --------------------------------------------------------------------------------
 
 ## 3. Preparing a Fluent .cas file for UDF function usage:
@@ -102,7 +283,7 @@ Go through the menus on the left and perform the following steps:
       Phase-2-water set the volume fraction UDF
   * Dynamic Mesh:
     + Set the structure boundary to follow the UDF
-    + Set the domain to be deforming/remeshing as needed
+    + Set the domain to be deforming/re-meshing as needed
   * Reference values: Only needed for writing output (since Fluent usese
     dimensionless values)
   * Solution methods:

examples/Cavity2D/CSMdummy.py

@@ -15,7 +15,7 @@
 mesh_name = "beam"
 
 
-def computeDisplacements(force_vals, displ_vals):
+def computeDisplacements(force_vals, displ_vals, coords_x):
     print("Entering computeDisplacements")
 
     # Considering beam of square cross-section with thickness of 0.01 m
@@ -27,25 +27,34 @@ def computeDisplacements(force_vals, displ_vals):
     # Length of beam = 1 m as stated in FLUENT
     ll = 1
 
-    for vertex in range(0, vertexSize*dim, 2):
-        displ_vals[vertex] = force_vals[vertex]*vertex*(ll-vertex)*(ll*ll + vertex*(ll-x))/(24*ME*MI*ll)
+    displ_vals[:, 1] = (1 / (24 * ME * MI * ll)) * \
+        np.multiply(np.multiply(np.multiply(force_vals[:, 1],
+                                            coords_x),
+                                (ll - coords_x)),
+                    (ll * ll + coords_x * (ll - coords_x)))
+
+    # cap the displacement at 0.002
+    displ_vals = np.array([np.array([val[0], -0.002]) if val[1] < -0.002
+                           else np.array([val[0], 0.002]) if val[1] > 0.002
+                           else np.array([val[0], val[1]])
+                           for val in displ_vals])
+
     print("Updating displacement variable")
 
+    return displ_vals
+
 
 solver_process_index = 0
 solver_process_size = 1
 
 interface = precice.Interface(participant_name, configuration_file_name, solver_process_index, solver_process_size)
-    
+
 dim = interface.get_dimensions()
 mesh_id = interface.get_mesh_id(mesh_name)
 
 vertexSize = 100
-coords_x = []
-coords_y = []
-for i in range(vertexSize):
-    coords_x.append(i)
-    coords_y.append(0.0)
+coords_x = np.linspace(0, 1, num=vertexSize)
+coords_y = np.linspace(0, 0, num=vertexSize)
 
 coords = np.stack([coords_x, coords_y], axis=1)
 
@@ -61,28 +70,29 @@ def computeDisplacements(force_vals, displ_vals):
 dt = 1.0
 
 precice_dt = interface.initialize()
-    
+
 while interface.is_coupling_ongoing():
-   
+
     if interface.is_action_required(precice.action_write_iteration_checkpoint()):
         print("CSMdummy: Writing iteration checkpoint")
-        interface.fulfilled_action(precice.action_write_iteration_checkpoint())
-    
+        interface.mark_action_fulfilled(precice.action_write_iteration_checkpoint())
+
     forces = interface.read_block_vector_data(forceIDs, vertexIDs)
+    print("Forces read in:\n{}".format(forces))
 
-#    computeDisplacements(forces,displacements)
+    displacements = computeDisplacements(forces, displacements, coords_x)
+    print("Computed Displacements:\n{}".format(displacements))
 
     dt = min(precice_dt, dt)
 
     interface.write_block_vector_data(displIDs, vertexIDs, displacements)
 
     precice_dt = interface.advance(dt)
-    
+
     if interface.is_action_required(precice.action_read_iteration_checkpoint()):
         print("CSMdummy: Reading iteration checkpoint")
-        interface.fulfilled_action(precice.action_read_iteration_checkpoint())
+        interface.mark_action_fulfilled(precice.action_read_iteration_checkpoint())
     else:
         print("CSMdummy: advancing in time")
-
-interface.finalize()
 
+interface.finalize()