toolbox_schema.yaml not copied to venv site-packages, .yaml reading bug, reading old openfast .fst files

[ghylander]

I'm trying to run example 04 to tune a controller to a platform:
(I had a pre-existing conda env "rosco-env" from the now old ROSCO_toolbox repo)

1. I activate rosco-env

2. I navigate to the new ROSCO dir and run python run_setup.py install, exits with no errors

3. I go to /Examples and run example_04.py

4. I get the following error:

File ".\example_04.py", line 20, in <module>
from ROSCO_toolbox.inputs.validation import load_rosco_yaml
ModuleNotFoundError: No module named 'ROSCO_toolbox.inputs'

[ghylander]

I'm sorry for the confusion, i get different errors when using the now deprecated ROSCO_toolbox repo and the ROSCO 2.4.0 repo and I confused them

[nikhar-abbas]

@ghylander - I'm not sure I understand the issue here? Can this be closed?

If you are having environment issues, I would recommend starting clean from a new environment. The examples should all successfully run on windows, linux, and macos if the install was done properly.

[ghylander]

@ghylander - I'm not sure I understand the issue here? Can this be closed?

If you are having environment issues, I would recommend starting clean from a new environment. The examples should all successfully run on windows, linux, and macos if the install was done properly.

Sorry, i have two separate problems.

Problem 1

When running example_04.py i get:

File ".\example_04.py", line 20, in <module>
from ROSCO_toolbox.inputs.validation import load_rosco_yaml
ModuleNotFoundError: No module named 'ROSCO_toolbox.inputs'

As i stated in the OP, i followed the installation instructions from the docs:

1. I created a conda virtual env and activate it
2. I run python setup.py install, installation ends with no errors
3. I navigate to the /ROSCO/Examples dir and run example_04.py, the above error appears

What's weird is that i dont have this problem when i use the old ROSCO_toolbox scripts, i have been running the run_Testing.py file for a few weeks now with no problem. I can even run example_04.py from the ROSCO_toolbox repo.

Problem 2

Not related to ROSCO.

Sometimes i have to work with OpenFAST v2.2.0, which has different .fst structure to OpenFAST v3.0.0. In specific, the following lines are missing in OF v2.2 .fst file:
CalcSteady
TrimCase
TrimTol
TrimGain
Twr_Kdmp
Bld_Kdmp
When i try to run example_04 (from the old ROSCO_toolbox), i got an error that i traced back to the file FAST_reader.py, line 489:

        # Fst
        f.readline()
        self.fst_vt['Fst']['Linearize']  = f.readline().split()[0]
        self.fst_vt['Fst']['CalcSteady'] = f.readline().split()[0] <--
        self.fst_vt['Fst']['TrimCase']   = f.readline().split()[0] <--
        self.fst_vt['Fst']['TrimTol']    = f.readline().split()[0] <--
        self.fst_vt['Fst']['TrimGain']   = f.readline().split()[0] <--
        self.fst_vt['Fst']['Twr_Kdmp']   = f.readline().split()[0] <--
        self.fst_vt['Fst']['Bld_Kdmp']   = f.readline().split()[0] <--
        self.fst_vt['Fst']['NLinTimes']  = f.readline().split()[0]
        self.fst_vt['Fst']['LinTimes']   = re.findall(r'[^,\s]+', f.readline())[0:2]
        self.fst_vt['Fst']['LinInputs']  = f.readline().split()[0]
        self.fst_vt['Fst']['LinOutputs'] = f.readline().split()[0]
        self.fst_vt['Fst']['LinOutJac']  = f.readline().split()[0]
        self.fst_vt['Fst']['LinOutMod']  = f.readline().split()[0]

Here, the function read_MainInput is expecting to find the lines I specified above, which are not present in OF v2.2. Does ROSCO require those lines to function properly, or can i safely remove them when they are not present in the .fst file?

[ghylander]

Problem 1 (cont. fixes and new issues)

I created a new venv and repeated the process, it seems that the file toolbox_schema.yaml was not copied from "/ROSCO/ROSCO_toolbox/inputs" dir to the "\rosco-env2\Lib\site-packages\rosco-2.4.0-py3.8.egg\ROSCO_toolbox\inputs" dir:

Copying the file to the venv dir allowed me to run the file. This lead me to a new bug.
In the .yaml file I use to run example_04, I set Fl_mode = 1 in controller_params, and in the turbine_params I set the twr_freq and ptfm_freq to my platform's:

(the actual values of the frequencies are censored)

Despite this, when running example_04, the script exits with error:

Traceback (most recent call last):
  File "C:\Users\gus_h\miniconda3\envs\rosco-env2\lib\site-packages\rosco-2.4.0-py3.8.egg\ROSCO_toolbox\controller.py", line 98, in __init__
    self.twr_freq   = controller_params['twr_freq']
KeyError: 'twr_freq'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".\example_04_Test.py", line 35, in <module>
    controller      = ROSCO_controller.Controller(controller_params)
  File "C:\Users\gus_h\miniconda3\envs\rosco-env2\lib\site-packages\rosco-2.4.0-py3.8.egg\ROSCO_toolbox\controller.py", line 101, in __init__
    raise Exception('ROSCO_toolbox:controller: twr_freq and ptfm_freq must be set if Fl_Mode > 0')
Exception: ROSCO_toolbox:controller: twr_freq and ptfm_freq must be set if Fl_Mode > 0

Changing Fl_Mode = 0 "fixes" the issue.

However, this still leads me back to my previous problem 2:

Problem 2

Not related to ROSCO.

Sometimes i have to work with OpenFAST v2.2.0, which has different .fst structure to OpenFAST v3.0.0. In specific, the following lines are missing in OF v2.2 .fst file: CalcSteady TrimCase TrimTol TrimGain Twr_Kdmp Bld_Kdmp When i try to run example_04 (from the old ROSCO_toolbox), i got an error that i traced back to the file FAST_reader.py, line 489:

[dzalkind]

Hi @ghylander, we can only support one version of OpenFAST, so we try to support the latest version, which is 3.0.0. You can either edit the files in ofTools/ or update to OpenFAST 3.0.0 to fix your problem 2.

For problem 1 and #82, I think you should start with a fresh conda environment. A few other users have had similar issues and that resolved them.

[ghylander]

You can either edit the files in ofTools/

This was the point of my question. I'm no t asking you to support any OF other than the latest release, but can I simply comment out those lines? Are they required by ROSCO to function properly?

For problem 1 and #82, I think you should start with a fresh conda environment. A few other users have had similar issues and that resolved them.

I've tried this multiple times, I can make a screen recording if you want to check. The process I'm following is conda create -y --name rosco-env python=3.8 > conda activate rosco-env > conda install --y wisdem > python setup.py install > try to run example_04.py > it fails

This raises the error that requires me to manually copy the toolbox_schema.yaml to the env directory

Besides this, what about the KeyError: 'twr_freq' when Fl_Mode=1? is it confirmed to be a bug?

[ghylander]

I'm not an expert on .egg, but the file in rosco.egg-info/SOURCES.txt is missing the line ROSCO_toolbox/inputs/toolbox_schema.yaml, which means the file is not being copied to the venv .egg

Some short googling revealed .yaml files are not picked up by setuptools.find_packages as that method only finds python files
You will need to add a line to the setup script (in the metadata dict I suppose), i think it should be something like package_data={'': ['*.yaml']} (which includes all .yamls in all packages) or

packages=['inputs'],
package_data={'inputs': ['toolbox_schema.yaml']}

which inlcudes the specified file of the specified package
(https://setuptools.pypa.io/en/latest/userguide/datafiles.html)

[dzalkind]

Hi @ghylander,

Thanks for clarifying. Typically, we set up the ROSCO_toolbox using python setup.py develop. We are not egg experts either, so if you find a solution that works for you, please submit a PR to help other users like yourself.

Commenting out lines in ofTools should be fine, but you won't know until you try it :-)

[nikhar-abbas]

Okay, just to follow up on this.

I made a few updates to #83 to address the install issue, hopefully bugs are fixed. I added your suggested package_data={'': ['*.yaml']} line to the setup.py and modified the CI testing so that it does a python setup.py install when installing ROSCO to run the examples. This seems to run all of the examples as expected. You can see that example 04 is running here:
https://github.com/NREL/ROSCO/runs/4082909802?check_suite_focus=true#step:8:341

Note that the tests are failing because the examples that use OpenFAST are not running right now. This is because of a known issue with the conda-forge distribution of OpenFAST which is currently being fixed. Once this is sorted, #83 and #84 will all get merged into develop and then into main as v2.4.1.

As far as the twr_freq issue you are seeing - I would not say this is a known bug. I realized now that there is some inconsistency in the example tuning yamls as to which section the twr_freq and ptfm_freq parameters are in - please note that they are now in the controller_params section as shown here. Sorry about any confusion caused by this shiftt. I have updated the yamls in #83 as well.

As long as the input lines that you comment out in FAST_reader.py are not for parameters specifically used in the tuning process, it should be fine.

@ghylander, thanks for your patience and feedback on all this. Again, please keep in mind that we are doing as much as we can to make this open-source tool available, for free, to the research and development community. We highly encourage pull-requests with clearly written and well tested code that improves the functionality for all use cases. We only have limited time (and an even more limited--if any!--budget) to put towards sustained development and user-support for everyone using ROSCO around the globe.

[ghylander]

Great news about the install workflow, I didn't know what else could be the reason it wasn't working for me.

Regarding the examples yaml, it was confusing indeed, thanks for the clarification

I will try to comment out the lines and see the results

Lastly, thanks to you, Daniel and Rafael (and any other researcher I have not been in contact with so far) for developing and supporting ROSCO, I think it's a great tool and I'm more than happy to help streamline it.