.. _regression_test: Regression tests ================ The regression test executes a series of test cases which intend to fully describe OpenFAST and its module's capabilities. Jump to one of the following sections for instructions on running the regression tests: - :ref:`python_driver` - :ref:`ctest_driver` - :ref:`regression_test_example` - :ref:`regression_test_windows` Each locally computed result is compared to a static set of baseline results. The results are computed using the following OS, compiler, and hardware combination: ================== ============== ============================ Operating System Compiler Hardware ================== ============== ============================ Ubuntu 22.04 GNU 12.3.0 GitHub actions ================== ============== ============================ Note that if you run the regression tests locally, there may small numerical differences due to your compiler and hardware combination that may cause a few tests to fail. If you use the ``bokeh`` package while running the tests, you will see a resulting *html* file with plots showing the differences for any tests that fail (these differences should be small). The compiler versions, specific math libraries, and more info on hardware used to generate the baseline solutions are documented in the `r-test repository documentation `__. Currently, the regression test supports only double precision builds. The regression test system can be executed with CMake (through its included test driver, CTest) or manually with a custom Python driver. Both systems provide similar functionality with respect to testing, but CTest integration provides access to multithreading, automation, and test reporting via CDash. Both modes of execution require some configuration as described in the following sections. In both modes of execution a directory is created in the build directory called ``reg_tests`` where all of the input files for the test cases are copied (but not overwritten) and all of the locally generated outputs are stored. Ultimately, both CTest and the manual execution program call a series of Python scripts and libraries in ``reg_tests`` and ``reg_tests/lib``. One such script is ``lib/pass_fail.py`` which reads the output files and computes a norm on each channel reported. If the maximum norm is greater than the given tolerance, that particular test is reported as failed. The failure criteria is outlined below. .. code-block:: python difference = abs(testData - baselineData) for i in nChannels: if channelRange < 1: norm[i] = MaxNorm( difference[:,i] ) else: norm[i] = MaxNorm( difference[:,i] ) / channelRange if max(norm) < tolerance: pass = True else: pass = False Testing Environment ------------------- We recommend using ``conda`` to create a local environment to install the required packages for *OpenFAST* testing. You can use the following process as a rough guide for setting up the necessary environment on Linux/MacOS based systems. 1. create a new ``conda`` environment for *OpenFAST* testing and install python: - ``conda install python`` 2. from the ``/build`` directory, setup the environment with the following commands: - ``pip install ../requirements.txt`` Installs basic dependencies for testing. - ``pip install -e ../glue-codes/python/.`` Installs the ``pyOpenFAST`` package from *OpenFAST* repository - ``pip install -e ../openfast_io/.`` Installs the ``openfast_io`` package from *OpenFAST* repository Dependencies ------------ The following packages are required for regression testing (see also the ``requirements.txt`` file in the root directory for the python modules): - CMake and CTest - Python >=3.7 - numpy - vtk - bokeh>=2.4,!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3 (Optional) In addition to the above packages, two packages from the *OpenFAST* repository are required. We recommend installing these in a ``conda`` enviroment as described in the above section so that it will not interfere with your system Python installation. Using the ``pip install -e`` command will install using the local directory instead placing them within the system Python directories. - ``pyOpenFAST`` is a package for interfacing Python to the c-bindings libraries of *OpenFAST* modules. This is used in some testing at the module level. - ``openfast_io`` is a package for reading and writing *OpenFAST* input files. This is used in some of the testing. .. _python_driver: Executing with Python driver ---------------------------- The regression test can be executed manually with the included driver at ``openfast/reg_tests/manualRegressionTest.py``. This program reads a case list file at ``openfast/reg_tests/r-test/glue-codes/openfast/CaseList.md``. Cases can be removed or ignored by starting that line with a ``#``. The driver program includes multiple optional flags which can be obtained by executing with the help option: :: >>>$ python manualRegressionTest.py -h usage: manualRegressionTest.py [-h] [-p [Plotting-Flag]] [-n [No-Execution]] [-v [Verbose-Flag]] [-case [Case-Name]] [-module [Module-Name]] Executable-Name Relative-Tolerance Absolute-Tolerance Executes OpenFAST or driver and a regression test for a single test case. positional arguments: Executable-Name path to the executable Relative-Tolerance Relative tolerance to allow the solution to deviate; expressed as order of magnitudes less than baseline. Absolute-Tolerance Absolute tolerance to allow small values to pass; expressed as order of magnitudes less than baseline. optional arguments: -h, --help show this help message and exit -p [Plotting-Flag], -plot [Plotting-Flag] bool to include plots in failed cases -n [No-Execution], -no-exec [No-Execution] bool to prevent execution of the test cases -v [Verbose-Flag], -verbose [Verbose-Flag] bool to include verbose system output -case [Case-Name] single case name to execute -module [Module-Name], -mod [Module-Name] name of module to execute .. note:: For the NREL 5MW turbine test cases, an external ServoDyn controller must be compiled and included in the appropriate directory or all NREL 5MW cases will fail without starting. More information is available in the documentation for the `r-test repository `__, but be aware that these three DISCON controllers must exist .. code-block:: bash openfast/build/reg_tests/glue-codes/openfast/5MW_Baseline/ServoData/DISCON.dll openfast/build/reg_tests/glue-codes/openfast/5MW_Baseline/ServoData/DISCON_ITIBarge.dll openfast/build/reg_tests/glue-codes/openfast/5MW_Baseline/ServoData/DISCON_OC3Hywind.dll .. _ctest_driver: Executing with CTest -------------------- CTest is included with CMake and is primarily a set of preconfigured targets and commands. To use the CTest driver for the regression test, execute CMake as described in :ref:`installation`, but with this additional flag: ``-DBUILD_TESTING=ON``. The regression test specific CMake variables are :: BUILD_TESTING CTEST_OPENFAST_EXECUTABLE CTEST_[MODULE]_EXECUTABLE where [MODULE] is the module name CTEST_PLOT_ERRORS CTEST_REGRESSION_TOL Some additional resources that are required for the full regression test suite are included in the CMake project. Specifically, external ServoDyn controllers must be compiled for a given system and placed in a particular location. Thus, be sure to execute the build command with the ``install`` target: .. code-block:: bash # Configure CMake with testing enabled and accept the default # values for all other test-specific CMake variables cmake .. -DBUILD_TESTING=ON # Build and install make install .. note:: REMINDER: For the NREL 5MW turbine test cases, an external ServoDyn controller must be compiled and included in the appropriate directory or all NREL 5MW cases will fail without starting. More information is available in the documentation for the `r-test repository `__, but be aware that these three DISCON controllers must exist .. code-block:: bash openfast/build/reg_tests/glue-codes/openfast/5MW_Baseline/ServoData/DISCON.dll openfast/build/reg_tests/glue-codes/openfast/5MW_Baseline/ServoData/DISCON_ITIBarge.dll openfast/build/reg_tests/glue-codes/openfast/5MW_Baseline/ServoData/DISCON_OC3Hywind.dll After CMake configuration and compiling, the automated regression test can be executed by running either of the commands ``make test`` or ``ctest`` from the ``build`` directory. If the entire OpenFAST package is to be built, CMake will configure CTest to find the new binary at ``openfast/build/glue-codes/openfast/openfast``. However, if the intention is to build only the test suite, the OpenFAST binary should be specified in the CMake configuration under the ``CTEST_OPENFAST_EXECUTABLE`` flag. There is also a corresponding ``CTEST_[MODULE]_NAME`` flag for each module included in the regression test. When driven by CTest, the regression test can be executed by running various forms of the command ``ctest`` from the build directory. The basic commands are: .. code-block:: bash # Run the entire regression test ctest # Disable actual execution of tests; # this is helpful in formulating a particular ctest command ctest -N # Run the entire regression test with verbose output ctest -V # Run tests by name where TestName is a regular expression (regex) ctest -R [TestName] # Run all tests with N tests executing in parallel ctest -j [N] Each regression test case contains a series of labels associating all of the modules used. The labeling can be seen in the test instantiation in ``reg_tests/CTestList.cmake`` or with the command: .. code-block:: bash # Print all available test labels ctest --print-labels The test cases corresponding to a particular label can be executed with this command: .. code-block:: bash # Filter the test cases corresponding to a particular label ctest -L [Label] Flags can be compounded making useful variations such as .. code-block:: bash # Run all cases that use SubDyn with verbose output ctest -V -L subdyn # Run all cases that use SubDyn in 16 concurrent processes ctest -j 16 -L subdyn # Run the case with name "5MW_DLL_Potential_WTurb" with verbose output ctest -V -R 5MW_DLL_Potential_WTurb # List all tests with the "beamdyn" label ctest -N -L beamdyn # List the labels included in all tests matching the regex "bd" ctest -N -R bd --print-labels The automated regression test writes new files only into the build directory. Specifically, all locally generated solutions are located in the corresponding glue-code or module within ``openfast/build/reg_tests``. The baseline solutions contained in ``openfast/reg_tests/r-test`` are strictly read and are not modified by the automated process. .. _regression_test_example: Regression test examples ------------------------ The following examples illustrate methods of running the regression tests on Unix-based systems. However, similar procedures can be used on Windows with CMake and CTest. An alternate method of running the regression tests on Windows is given in :ref:`reg_test_windows`. Compile OpenFAST and execute with CTest ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following example assumes the user is starting completely from scratch. The commands below download the source code, configure the OpenFAST project with CMake, compile all executables, and execute the full regression test suite. .. code-block:: bash # Download the source code from GitHub # Note: The default branch is 'main' git clone --recursive https://github.com/openfast/openfast.git cd openfast # If necessary, switch to another target branch and update r-test git checkout dev git submodule update # Create the build and install directories and move into build mkdir build install && cd build # Configure CMake for testing # - BUILD_TESTING - turn ON # - CTEST_OPENFAST_EXECUTABLE - accept the default # - CTEST_[MODULE]_EXECUTABLE - accept the default cmake .. -DBUILD_TESTING=ON # Compile and install make install # Execute the full test suite with 4 concurrent processes ctest -j4 Configure with CMake and a given executable ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example assumes the user has a fully functional OpenFAST executable available along with any necessary libraries, but does not have the source code repository downloaded. This might be the case when executables are distributed within an organization or downloaded from an `OpenFAST Release `__. Here, nothing will be compiled, but the test suite will be configured with CMake for use with the CTest command. .. code-block:: bash # Download the source code from GitHub # Note: The default branch is 'main' git clone --recursive https://github.com/openfast/openfast.git cd openfast # If necessary, switch to another target branch and update r-test git checkout dev git submodule update # Create the build directory and move into it mkdir build && cd build # Configure CMake with openfast/reg_tests/CMakeLists.txt for testing # - BUILD_TESTING - turn ON # - CTEST_OPENFAST_EXECUTABLE - provide a path # - CTEST_[MODULE]_EXECUTABLE - provide a path cmake ../reg_tests \ -DBUILD_TESTING=ON \ -DCTEST_OPENFAST_EXECUTABLE=/home/user/Desktop/openfast_executable \ -DCTEST_BEAMDYN_EXECUTABLE=/home/user/Desktop/beamdyn_driver # Install required files make install # Execute the full test suite with 4 concurrent processes ctest -j4 .. _example_python_driver: Python driver with a given executable ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example assumes the user has a fully functional OpenFAST executable available along with any necessary libraries, but does not have the source code repository downloaded. This might be the case when executables are distributed within an organization or downloaded from an `OpenFAST Release `__. Nothing will be compiled, but the test suite will be executed with the included Python driver. .. code-block:: bash # Download the source code from GitHub # Note: The default branch is 'main' git clone --recursive https://github.com/openfast/openfast.git cd openfast # If necessary, switch to another target branch and update r-test git checkout dev git submodule update # Execute the Python driver cd reg_tests python manualRegressionTest.py -h # usage: manualRegressionTest.py [-h] [-p [Plotting-Flag]] [-n [No-Execution]] # [-v [Verbose-Flag]] [-case [Case-Name]] [-module [Module-Name]] # Executable-Name Relative-Tolerance Absolute-Tolerance # # Executes OpenFAST or driver and a regression test for a single test case. # # positional arguments: # Executable-Name path to the executable # Relative-Tolerance Relative tolerance to allow the solution to deviate; expressed as order of magnitudes less than baseline. # Absolute-Tolerance Absolute tolerance to allow small values to pass; expressed as order of magnitudes less than baseline. # # optional arguments: # -h, --help show this help message and exit # -p [Plotting-Flag], -plot [Plotting-Flag] # bool to include plots in failed cases # -n [No-Execution], -no-exec [No-Execution] # bool to prevent execution of the test cases # -v [Verbose-Flag], -verbose [Verbose-Flag] # bool to include verbose system output # -case [Case-Name] single case name to execute # -module [Module-Name], -mod [Module-Name] # name of module to execute python manualRegressionTest.py ..\build\bin\openfast_x64_Double.exe 2.0 1.9 .. _reg_test_windows: Detailed example of running on Windows ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The :ref:`example_python_driver` example can be used for running the regression tests on a Windows computer. However, a more detailed, step-by-step description is given in :ref:`regression_test_windows`. .. toctree:: :maxdepth: 1 :hidden: regression_test_windows.rst .. _new_regression_test_case: Adding test cases ----------------- In all modes of execution, the regression tests are ultimately driven by a series of Python scripts located in the ``openfast/reg_tests`` directory with the naming scheme ``executeRegressionTest.py``. The first step to adding a new regression test case is to verify that a script exists for the target module. If it does not, an issue should be opened in `OpenFAST Issues `_ to coordinate with the NREL team on creating this script. The next step is to add the test case in the appropriate location in the `r-test` submodule. The directory structure in r-test mirrors the directory structure in OpenFAST, so module-level tests should be placed in their respective module directories and glue-code tests go in ``r-test/glue-codes/openfast``. Note the naming scheme of files for existing tests and adapt the new test case files accordingly. Specifically, the main input file and output file names may be expected in a particular convention by the Python scripts. Also, consider that any relative paths within the input deck for the new test case must work within the r-test directory structure. Once the test directory exists, the test case must be registered with the appropriate drivers. For OpenFAST glue-code tests, this happens both in CMake and a standalone list of test cases. For CMake, edit the file ``openfast/reg_tests/CTestList.cmake``. The additional test should be added in the section corresponding to the module or driver at the bottom of that file. For the Python driver, the new test case must be added to ``openfast/reg_tests/r-test/glue-codes/openfast/CaseList.md``. At this point, the registration with CTest can be verified: .. code-block:: bash # Move into the build directory cd openfast/build # Run CMake to take the new changes to the test list cmake .. -DBUILD_TESTING=ON # If the BUILD_TESTING flag was previously enabled, this can be left off # List the registered tests, but don't run them ctest -N For module regression tests, the only option for execution is with the CMake driver, so follow the instructions above to edit ``CTestList.cmake``. Finally, the new test cases in the r-test submodule must be added to the r-test repository. To do this, open a new issue in `r-test Issues `_ requesting for support from the NREL team to commit your test.