[BUG] Dependencies don't specify python version

[psaiz]

Describe the bug

There are several python scripts deployed with opensearch-dashboards, and it does not specify if they are python2 or python3

To Reproduce
Steps to reproduce the behavior:

1. curl https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.2.0/opensearch-dashboards-1.2.0-linux-x64.tar.gz -o opensearch-dashboards-1.2.0-linux-x64.tar.gz
2. tar zxf opensearch-dashboards-1.2.0-linux-x64.tar.gz
3. head opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/MSVSSettings_test.py -n 1
   #!/usr/bin/env python

Expected behavior
Those files should specify if it is python2 or python3

OpenSearch Version
Not needed

Dashboards Version
1.2.0

Host/Environment (please complete the following information):

• OS: [e.g. iOS]

Additional context
I'm running into this issue while creating an rpm of opensearch-dashboards with the content of the tar file. The full list of files suffering from this issue is:

*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/re2/vendor/re2/make_unicode_casefold.py: #!/usr/bin/python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/re2/vendor/re2/make_unicode_groups.py: #!/usr/bin/python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/gyp_main.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/setup.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/tools/graphviz.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/tools/pretty_sln.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/tools/pretty_vcproj.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/tools/pretty_gyp.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/test_gyp.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/easy_xml_test.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/__init__.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/win_tool.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/common_test.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/generator/msvs_test.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/input_test.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/flock_tool.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/mac_tool.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.
*** ERROR: ambiguous python shebang in /usr/share/opensearch-dashboards/opensearch-dashboards-1.2.0-linux-x64/node_modules/node-gyp/gyp/pylib/gyp/MSVSSettings_test.py: #!/usr/bin/env python. Change it to python3 (or python2) explicitly.

For the record, I'll be happy to send the spec file that I'm using in case that helps. I followed the same procedure for opensearch, and in that one, it worked like a charm.

[boktorbb]

Hey @psaiz, it seems to me like the issue is that your python configuration is not python 3. If you're using virtualenv, you can set the default python version to 3 and it will default to that if the script doesn't specify

[psaiz]

Hi Bishoy,

Thanks for looking into this. I don't think that the issue is on the client side. Please, take a look at [1]. My understanding of this is that any file distributed should specify which version of python it requires.

Note that it could be done with the pathfix.py while creating the rpm, as mentioned on that link. At the same time, if the files in the tar file already included the proper shebangs, we wouldn't need the pathfix.py.

Cheers,
pablo

[1] https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error

[boktorbb]

Hey Pablo, It seems like most of these errors are coming from dependencies of Dashboards in our node_modules directory and we own very few python files in our repo. If you want to try fixing this issue and opening a PR, that would be great.

[psaiz]

Hi again,

Thanks for your reply, @boktorbb-amzn. I'd be happy to help. I might need a bit more info, if you don't mind. I was looking at the Release procedures, and I'm wondering if the best place to fix this would be in this project, or in opensearch-build. I will look first into that one. If you think that's not the correct approach, please let me know.

Cheers,
pablo

[psaiz]

Hi @boktorbb-amzn ,

Could you please take a look at the pull request? That should address the issue. Note that it requires one more package on the build nodes. I'm not sure where to specify that.

Best regards,
pablo

[tmarkley]

@psaiz did you check if newer versions of re2 and node-gyp have proper shebangs in their python files?

$ npm ls node-gyp
opensearch-dashboards@2.0.0 /home/ubuntu/ws/OpenSearch-Dashboards
├─┬ @osd/optimizer@1.0.0 -> ./packages/osd-optimizer
│ └─┬ node-sass@6.0.1
│   └── node-gyp@7.1.2
└─┬ re2@1.15.4
  └── node-gyp@7.0.0

Upgrading might be a better and/or easier approach to this problem. Which OS are you using?

[psaiz]

Hi Tommy,

Thanks for the remark. Indeed, node-gyp@8.0.0 fixes this issue. As far as I can see, newer versions of re2 do not help.

I'll check if increasing the version of node-gyp works, leaving the patch for the re2. I'll open a ticket on the re2 to see if they can fix the shebangs.

I'm running on Linux (CentOS).

Cheers,
pablo

[psaiz]

Hi again,

I've updated the MR, requiring the version 8.2.0 of node-gyp. Higher versions of that package failed with the message:

error @npmcli/fs@1.1.0: The engine "node" is incompatible with this module. Expected version "^12.13.0 || ^14.15.0 || >=16". Got "10.24.1"

Let's see if that helps.

Cheers,
pablo

[tmarkley]

I'll open a ticket on the re2 to see if they can fix the shebangs.

It looks like you got a response on that question: uhop/node-re2#121

[psaiz]

Thanks Tommy. Indeed, that one was closed, and the issue has been sent to re2 . Let's see if that one fixes it

[psaiz]

Hi again,

The dependencies have been fixed, so I think that this ticket can be closed. Thanks!

[tmarkley]

Thanks @psaiz!