refactor: use pipenv to manage dependencies #150

4141done · 2022-05-16T23:40:17Z

What

Using pip and requirements.txt led to issues where dependent library versions were
unpredictable and led to errors in some deployment environments.

This will also allow us to use semantic version properly without needing to explicitly
pin dependencies at certain versions but perform updates to libraries intentionally.

Proposed changes

Install and use Pipenv which will generate a Pipfile.lock detailing the exact versions of dependencies and their dependencies. This will help us have more reproducible builds since each dependency and subdependency will be installed at the exact version it was tested with.
Changed the code that determines the pulumi cli version to use the output of pip rather than reading from the requirements file (since that's not an exact version).

Pipenv Usage

pipenv is compatible with any commands that can be given to pip. There are some niceties that are good to know too:

graph

Prints the dependency graph with the Pipfile specs and the actual version resolved to.

$ pipenv graph
awscli==1.22.101
  - botocore [required: ==1.24.46, installed: 1.24.46]
    - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.0]
    - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
      - six [required: >=1.5, installed: 1.16.0]
    - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.9]
  - colorama [required: >=0.2.5,<0.4.4, installed: 0.4.3]
  - docutils [required: >=0.10,<0.16, installed: 0.15.2]
  - PyYAML [required: >=3.10,<5.5, installed: 5.4.1]
  - rsa [required: >=3.1.2,<4.8, installed: 4.7.2]
    - pyasn1 [required: >=0.1.3, installed: 0.4.8]
  - s3transfer [required: >=0.5.0,<0.6.0, installed: 0.5.2]
    - botocore [required: >=1.12.36,<2.0a.0, installed: 1.24.46]
      - jmespath [required: >=0.7.1,<2.0.0, installed: 1.0.0]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.2]
        - six [required: >=1.5, installed: 1.16.0]
      - urllib3 [required: >=1.25.4,<1.27, installed: 1.26.9]
fart==0.1.5
  - pyperclip [required: Any, installed: 1.8.2]
kic-pulumi-utils==1.0.0.post37+git.fb325311.dirty
  - GitPython [required: >=3.1.18,<3.2.0, installed: 3.1.27]
    - gitdb [required: >=4.0.1,<5, installed: 4.0.9]
      - smmap [required: >=3.0.1,<6, installed: 5.0.0]
  - passlib [required: >=1.7.4,<2.0.0, installed: 1.7.4]
  - pyyaml [required: >=5.3.1,<6.0, installed: 5.4.1]

... output truncated

Figuring out updates

You can use pipenv update --outdated to display a list of packages that have newer versions in the upstream.
From there, you can choose to update them one by one with pipenv update <package name> which will update the package according to the rules specified in the Pipfile. You may need to change the specification in the Pipfile to get the latest version.

https://pipenv-fork.readthedocs.io/en/latest/basics.html#example-pipenv-upgrade-workflow

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have written my commit messages in the Conventional Commits format.
I have read the CONTRIBUTING doc
I have added tests (when possible) that prove my fix is effective or that my feature works
I have checked that all unit tests pass after adding my changes
I have updated necessary documentation
I have rebased my branch onto master
I will ensure my PR is targeting the master branch and pulling from my branch from my own fork

4141done · 2022-05-17T00:23:31Z

bin/setup_venv.sh

+# Install certain utility packages like `nodeenv` and `wheel` that aid
+# in the installation of other build tools and dependencies
+# required by the other python packages.
+PIPENV_VERBOSITY=-1 PIPENV_PIPFILE="${script_dir}/../pulumi/python/Pipfile" pipenv install --dev


annotation:
Here we are:

Silencing an informational message telling us that pipenv is going to use the current venv with PIPENV_VERBOSITY=-1

Pointing pipenv at our Pipfile which is not in the current dir with PIPENV_PIPFILE

Only installing dev depenencies which are currently nodeenv and wheel

4141done · 2022-05-17T00:23:45Z

bin/setup_venv.sh

@@ -195,12 +194,15 @@ else
 fi

 # Install general package requirements
-pip3 install --requirement "${script_dir}/../pulumi/python/requirements.txt"
+PIPENV_VERBOSITY=-1 PIPENV_PIPFILE="${script_dir}/../pulumi/python/Pipfile" pipenv install


annotation:
Here we are:

Silencing an informational message telling us that pipenv is going to use the current venv with PIPENV_VERBOSITY=-1

Pointing pipenv at our Pipfile which is not in the current dir with PIPENV_PIPFILE

Installing all packages

4141done · 2022-05-17T00:24:48Z

bin/setup_venv.sh

+pip3 install "${script_dir}/../pulumi/python/utility/kic-pulumi-utils"
+
+rm -rf "${script_dir}/../pulumi/python/utility/kic-pulumi-utils/.eggs" \
+  "${script_dir}/../pulumi/python/utility/kic-pulumi-utils/build" \
+  "${script_dir}/../pulumi/python/utility/kic-pulumi-utils/kic_pulumi_utils.egg-info"
+


annotation:
Removing the && prevents an issue where if the pip install does not work the script would not fail. The rm operation should not fail either way.

4141done · 2022-05-17T00:25:13Z

bin/setup_venv.sh

 echo "Downloading Pulumi CLI into virtual environment"
-PULUMI_VERSION="$(grep '^pulumi~=.*$' "${script_dir}/../pulumi/python/requirements.txt" | cut -d '=' -f2 || true)"
+PULUMI_VERSION="$(pip list | grep 'pulumi ' | sed -nre 's/^[^0-9]*(([0-9]+\.)*[0-9]+).*/\1/p')"


annotation:
Here we read the pulumi version from pip to get the version that was actually installed.

dekobon

This is a great change! 🍥

dekobon · 2022-05-17T00:47:43Z

bin/setup_venv.sh

 pip3 install --upgrade pip
+pip install pipenv


Perhaps use pip3 here to bring it in harmony with the above line.

qdzlug · 2022-05-18T15:53:43Z

@4141done - this looks good to me; out of an abundance of caution I just kicked off all the deploy tests, but once they are clear I'm happy to have this merged whenever you are ready (my PR for #152 only touches the Jenkinsfile so it's going to be fine with yours).

qdzlug

Looks good to me...

Using pip and `requirements.txt` led to issues where dependent library versions were unpredictable and led to errors in some deployment environments. This will also allow us to use semantic version properly without needing to explicitly pin dependencies at certain versions but perform updates to libraries intentionally.

4141done commented May 17, 2022

View reviewed changes

dekobon approved these changes May 17, 2022

View reviewed changes

4141done force-pushed the pipenv branch from 5340d62 to fb32531 Compare May 17, 2022 16:14

4141done marked this pull request as ready for review May 17, 2022 16:31

qdzlug self-requested a review May 18, 2022 15:53

qdzlug approved these changes May 18, 2022

View reviewed changes

4141done force-pushed the pipenv branch from 5d4a517 to cc9a924 Compare May 18, 2022 22:23

4141done force-pushed the pipenv branch from 26428bf to 58b0fa9 Compare May 18, 2022 22:23

qdzlug merged commit 8c9a299 into nginxinc:master May 18, 2022

qdzlug mentioned this pull request May 20, 2022

feat: build on apple m1 #44

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: use pipenv to manage dependencies #150

refactor: use pipenv to manage dependencies #150

4141done commented May 16, 2022 •

edited

Loading

4141done May 17, 2022

4141done May 17, 2022

4141done May 17, 2022

4141done May 17, 2022

dekobon left a comment

dekobon May 17, 2022

qdzlug commented May 18, 2022

qdzlug left a comment

refactor: use pipenv to manage dependencies #150

refactor: use pipenv to manage dependencies #150

Conversation

4141done commented May 16, 2022 • edited Loading

What

Proposed changes

Pipenv Usage

graph

Figuring out updates

Checklist

4141done May 17, 2022

Choose a reason for hiding this comment

4141done May 17, 2022

Choose a reason for hiding this comment

4141done May 17, 2022

Choose a reason for hiding this comment

4141done May 17, 2022

Choose a reason for hiding this comment

dekobon left a comment

Choose a reason for hiding this comment

dekobon May 17, 2022

Choose a reason for hiding this comment

qdzlug commented May 18, 2022

qdzlug left a comment

Choose a reason for hiding this comment

4141done commented May 16, 2022 •

edited

Loading