Make requirements from Pipfile.lock, instead of Pipfile.

If _Pipfile_ is altered manually after running the last
series of tests, the generated _requirements.txt_ is not in
line with the latest run of the environment against which
was tested. The module `pipenv-to-requirements` solves
exactly that problem.
Remark that this seems to be the thought of people over at
the Python Packaging Authority, see the last paragraph of
the issue reported in
pypa/pipfile#7 (comment)

> Tools work with Pipfile.lock exclusively. A build system should never
automatically execute a Pipfile to generate a missing or outdated
Pipfile.lock

Pipfile

@@ -9,6 +9,7 @@ name = "pypi"
 pyspark = "==2.4.0"
 ipython = "*"
 "flake8" = "*"
+pipenv-to-requirements = "*"
 
 [requires]
 python_version = "3.6"

README.md

@@ -67,7 +67,7 @@ This package, together with any additional dependencies referenced within it, mu
 2. formally package and upload `dependencies` to somewhere like the `PyPI` archive (or a private version) and then run `pip3 install dependencies` on each node; or,
 3. a combination of manually copying new modules (e.g. `dependencies`) to the Python path of each node and using `pip3 install` for additional dependencies (e.g. for `requests`).
 
-Option (1) is by far the easiest and most flexible approach, so we will make use of this for now. To make this task easier, especially when modules such as `dependencies` have additional dependencies (e.g. the `requests` package), we have provided the `build_dependencies.sh` bash script for automating the production of `packages.zip`, given a list of dependencies documented in `Pipfile` and managed by the `pipenv` python application (discussed below).
+Option (1) is by far the easiest and most flexible approach, so we will make use of this for now. To make this task easier, especially when modules such as `dependencies` have additional dependencies (e.g. the `requests` package), we have provided the `build_dependencies.sh` bash script for automating the production of `packages.zip`, given a list of dependencies documented in `Pipfile.lock` and managed by the `pipenv` python application (discussed below).
 
 ## Running the ETL job
 
@@ -171,7 +171,7 @@ If you're wondering what the `pipenv` command is, then read the next section.
 
 ## Managing Project Dependencies using Pipenv
 
-We use [pipenv](https://docs.pipenv.org) for managing project dependencies and Python environments (i.e. virtual environments). All direct packages dependencies (e.g. NumPy may be used in a User Defined Function), as well as all the packages used during development (e.g. PySpark, flake8 for code linting, IPython for interactive console sessions, etc.), are described in the `Pipfile`. Their **precise** downstream dependencies are described in `Pipfile.lock`.
+We use [pipenv](https://docs.pipenv.org) for managing project dependencies and Python environments (i.e. virtual environments). All direct package dependencies (e.g. NumPy may be used in a User Defined Function), as well as all the packages used during development (e.g. PySpark, flake8 for code linting, IPython for interactive console sessions, etc.), are described in the `Pipfile`. Their **precise** downstream dependencies (“concrete requirements” or “frozen resolve”) are described in `Pipfile.lock`.
 
 ### Installing Pipenv
 

build_dependencies.sh

@@ -3,22 +3,28 @@
 # check to see if pipenv is installed
 if [ -x "$(which pipenv)" ]
 then
-    # check that there Pipfile exists in root directory
-    if [ ! -e Pipfile ]
+    # check that Pipfile.lock exists in root directory
+    if [ ! -e Pipfile.lock ]
     then
-        echo 'ERROR - cannot find Pipfile'
+        echo 'ERROR - cannot find Pipfile.lock'
         exit 1
     fi
 
-    # use Pipenv to create a requirement.txt file
-    echo '... creating requirements.txt from Pipfile'
-    pipenv lock -r > requirements.txt
+    # Ensure pipenv-to-requirements is installed
 
+
+    # use Pipenv.lock to create a requirements.txt file
+    echo '... creating requirements.txt from Pipfile.lock'
+    if ! pipenv run pipenv_to_requirements --freeze ; then
+        echo 'ERROR - `pipenv-to-requirements` is not installed. Install it with `pipenv install pipenv-to-requirements --dev`'
+        exit 1
+    fi
     # install packages to a temporary directory and zip it
+    touch requirements.txt  # safeguard in case there are no packages
     pip3 install -r requirements.txt --target ./packages
 
     # check to see if there are any external dependencies
-    # if not then create an empty file to see zip with
+    # if not then create an empty file to seed zip with
     if [ -z "$(ls -A packages)" ]
     then
         touch packages/empty.txt
@@ -32,20 +38,19 @@ then
     fi
 
     cd packages
-    zip -9mrv packages.zip .
+    zip -9 --move --recurse-paths --verbose packages.zip .
     mv packages.zip ..
     cd ..
 
     # remove temporary directory and requirements.txt
-    rm -rf packages
-    rm requirements.txt
+    rm --recursive --force packages requirements*.txt
 
     # add local modules
     echo '... adding all modules from local utils package'
-    zip -ru9 packages.zip dependencies -x dependencies/__pycache__/\*
+    zip --recurse-paths --update -9 packages.zip dependencies -x dependencies/__pycache__/\*
 
     exit 0
 else
-    echo 'ERROR - pipenv is not installed --> run pip3 install pipenv to load pipenv into global site packages or install via a system package manager.'
+    echo 'ERROR - pipenv is not installed --> run `pip3 install pipenv` to load pipenv into global site packages or install it via a system package manager.'
     exit 1
 fi