add: update folder structure and setup files so python package is aut…

…omatically installable

* add auto generating changlog to workflow

* add auto generating changlog to workflow

* changed to checkoutv3 in changlog workflow

* adding initial package files for user

* changed file structure for package and changed package name to repo name

* change expected values for repo_name

* change expected values for repo_name

* change expected values for repo_name

* change expected values for repo_name

* changed file structure for package and changed package name to repo name

* changed file structure for package and changed package name to repo name

* changed file structure and changed directory tests in env

* changed directory tests in env

* changed directory tests in env

* changing ignored subfolders in env test

* changing ignored subfolders and env variable names

* changing ignored subfolders and env variable names

* changing name change in env test

* added expected paths for folders in src

* added expected paths for folders in src

* changed src and tests env directories

* changed src and tests env directories and hid repo name in expected paths

* added bug fix to make expected repo name upper

* adding environment variable for package dir

* adding environment variable for package dir

* removing brackets from directory variables to match env file

* removing brackets and spaces from directory variables to match env file

* removing brackets and spaces from directory variables to match env file

* fix bug in remove bracket and space function

* change format of env variables

* refactoring test_env

* updating initial requiremnets files and package location

* updating typo in testing injected variables

* updated main readme to include package setup details

* changed: updated setup and requirement files

* docs: update structure in readmes

* docs: remove src package reference

* comment out src package in docs

* comment out src package in docs

* chore: remove fast fail from workflow

* chore: remove fast fail

* Revert "comment out src package in docs"
revert document changes
This reverts commit 23f23ab.

* revert "comment out src package in docs"

This reverts commit 5a80482.

* Revert "docs: remove src package reference"

This reverts commit 24c1358.

* Revert "docs: update structure in readmes"

This reverts commit d523e81.

* docs: change src module to {{ cookiecutter.repo_name }}

* fix: update injected variable count

* fix: update injected variable count

* Revert "changed to checkoutv3 in changlog workflow"

This reverts commit 10ac908.

* fix: remove changelog workflow

* docs: move package install instructions to template readme

* fix: remove changelog workflow

* docs: removed src and test docs in structure documentation and replaced with package name

* fix: update version of isort to 5.12.0 for poetry fix

* fix: update python versions in build wf's as isort no longer compatible with 3.7

* fix: fix python version 3.10 in build with '

* fix: fix accessability link

* fix: update isort on template build

* fix: remove r version 4.0.4

* fix: remove file exclusion from pre commit

* fix: remove ubuntu from template build

* fix: remove R version 4.05

* fix: update R versions in template build

* fix: add ubuntu os to template build

* chore: remove lower version of R for test

* fix: update dependencies for ubuntu in template build

* fix: add dependency to ubuntu packages

* fix: remove dependency to test if test passes

* fix: update os in template build

* docs: update docs to mention python installable and R not

.github/workflows/govcookiecutter-build.yml

@@ -9,7 +9,7 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest, macos-latest, windows-latest ]
-        python: [ 3.7, 3.8, 3.9, '3.10' ]
+        python: [ 3.8, 3.9, '3.10', 3.11 ]
 
     steps:
       - name: Checkout the revision

.github/workflows/govcookiecutter-template-build.yml

@@ -10,12 +10,11 @@ jobs:
       matrix:
         # TODO: Fix R pre-commit hook issues with Windows
         os: [ ubuntu-latest, macos-latest ]
-        python: [ 3.7, 3.8, 3.9, '3.10' ]
+        python: [ 3.8, 3.9, '3.10' ]
         R:
           - {using_R: No, version: N/A}
-          - {using_R: Yes, version: 4.0.4}
-          - {using_R: Yes, version: 4.0.5}
-          - {using_R: Yes, version: 4.1.0}
+          - {using_R: Yes, version: 4.1.3}
+          - {using_R: Yes, version: 4.2.2}
 
     steps:
       - name: Checkout the revision
@@ -33,7 +32,9 @@ jobs:
         if: ${{ matrix.R.using_r == 'Yes' }}
       - name: Install other ${{ matrix.os }} R dependencies
         run: |
+          sudo apt-get update
           sudo apt-get install libcurl4-openssl-dev
+          sudo apt-get install libgit2-dev
           sudo apt-get install libharfbuzz-dev
           sudo apt-get install libfribidi-dev
         if: ${{ matrix.os == 'ubuntu-latest' && matrix.R.using_R == 'Yes' }}

.pre-commit-config.yaml

@@ -12,7 +12,7 @@ repos:
       - id: trailing-whitespace
         name: Check for trailing whitespaces (auto-fixes)
   - repo: https://github.com/pycqa/isort
-    rev: 5.8.0
+    rev: 5.12.0
     hooks:
       - id: isort
         name: isort - Sort Python imports (auto-fixes)

README.md

@@ -4,7 +4,8 @@ A cookiecutter template for analytical, Python-, or Python and R-based projects
 His Majesty's Government, and wider public sector.
 
 This template helps to set up standardised project structures, and [includes security
-features using pre-commit hooks][docs-pre-commit].
+features using pre-commit hooks][docs-pre-commit]. This cookiecutter template also acts
+as an installable template (python projects only).
 
 It also provides an Agile, centralised, and lightweight analytical quality assurance
 (AQA) process. Pull or merge request templates are used to nudge users to complete this

docs/accessibility.md

@@ -104,7 +104,7 @@ We plan to fix the accessibility issues in the content by the end of December 20
 
 This statement was prepared on 30 June 2021. It was last reviewed on 20 July 2021.
 
-[abilitynet]: https://mcmw.abilitynet.org.uk/
+[abilitynet]: https://abilitynet.org.uk/
 [accessibility-legislation]: https://www.legislation.gov.uk/uksi/2018/952/regulation/4/made
 [eass]: https://www.equalityadvisoryservice.com/
 [email]: mailto:gsshelp@statistics.gov.uk

tests/test_env.py

@@ -14,6 +14,45 @@
 EXCLUDE_SUB_DIR_IN_PARENTS_NAMES = [*EXCLUDE_ROOT_DIR_NAMES, "docs"]
 
 
+def remove_brackets_and_spaces(path_name: str) -> str:
+    """Removes spaces and brackets from a string.
+
+    Args:
+        path_name : A path name as a string.
+
+    Returns:
+        The path name with spaces and brackets removed.
+    """
+    if "{" in path_name or "}" in path_name or " " in path_name:
+        path_name = path_name.replace("{", "")
+        path_name = path_name.replace("}", "")
+        path_name = path_name.replace(" ", "")
+
+    return path_name
+
+
+def loop_directories_children(
+    dir: Path, env_expected_dir_variable: Dict[str, Path]
+) -> Dict[str, Path]:
+    """Loop through a directories children and add their paths to a dictionary.
+
+    Args:
+        dir: directory containing Children locations
+        env_expected_dir_variable : Dictionary where the keys are directory
+            variables and values are their paths.
+
+    Returns:
+        Dictionary of directory keys and their paths with the children of the
+        given directory added.
+    """
+    for child in dir.iterdir():
+        if child.is_dir():
+            dir_name = remove_brackets_and_spaces(dir.name.upper())
+            child_name = remove_brackets_and_spaces(child.name.upper())
+            env_expected_dir_variable[f"DIR_{dir_name}_{child_name}"] = child
+    return env_expected_dir_variable
+
+
 def get_actual_env_variables(path_env: Path) -> Dict[str, Path]:
     """Get the environment variables and values for directories in the `.env` file of
      the `govcookiecutter` template.
@@ -61,7 +100,7 @@ def define_expected_env_variables(
     # upper case in the format "DIR_<<<DIRECTORY_NAME>>>". Ignore any directories
     # with a name in `exclude_root_folders`
     env_expected_dir_variable = {
-        f"DIR_{d.name.upper()}": d
+        f"DIR_{remove_brackets_and_spaces(d.name.upper())}": d
         for d in folder.glob("*")
         if d.is_dir() and d.name not in exclude_root_folders
     }
@@ -77,9 +116,14 @@ def define_expected_env_variables(
             and d.name not in exclude_folders
             and d.parent.name not in exclude_sub_folders_in_parent_folders
         ):
-            env_expected_dir_variable[
-                f"DIR_{d.parent.name.upper()}_{d.name.upper()}"
-            ] = d
+            if d.name.upper() == "SRC":
+                env_expected_dir_variable = loop_directories_children(
+                    d, env_expected_dir_variable
+                )
+
+            parent = remove_brackets_and_spaces(d.parent.name.upper())
+            name = remove_brackets_and_spaces(d.name.upper())
+            env_expected_dir_variable[f"DIR_{parent}_{name}"] = d
     return env_expected_dir_variable
 
 

tests/test_govcookiecutter_injected_variables.py

@@ -30,13 +30,16 @@
 }
 REPO_NAME_COUNT = {
     '"{{ cookiecutter.repo_name }}",': 0,
-    "`{{ cookiecutter.repo_name }}`": 7,
+    "`{{ cookiecutter.repo_name }}`": 15,
     "`{{ cookiecutter.repo_name }}`,": 1,
     '"{{ cookiecutter.repo_name }}.tex",': 0,
     '"{{ cookiecutter.repo_name }}doc"': 1,
-    "{{ cookiecutter.repo_name }}": 0,
+    "{{ cookiecutter.repo_name }}": 10,
+}
+OVERVIEW_COUNT = {
+    '"{{ cookiecutter.overview }}",': 0,
+    "{{ cookiecutter.overview }}": 1,
 }
-OVERVIEW_COUNT = {'"{{ cookiecutter.overview }}",': 0, "{{ cookiecutter.overview }}": 1}
 PROJECT_VERSION_COUNT = {
     '"{{ cookiecutter.project_version }}"': 2,
     "{{ cookiecutter.project_version }}": 0,
@@ -162,7 +165,7 @@ def recursive_open_and_count_search_terms(
     (
         "repo_name",
         "repo_2",
-        {**REPO_NAME_COUNT, "{{ cookiecutter.repo_name }}": 1},
+        {**REPO_NAME_COUNT, "{{ cookiecutter.repo_name }}": 11},
         {"using_R": "Yes"},
     ),
     ("overview", "overview_1", OVERVIEW_COUNT, {"using_R": "No"}),

{{ cookiecutter.repo_name }}/.env

@@ -38,13 +38,16 @@ DIR_NOTEBOOKS=./notebooks
 # Add environment variables for the `outputs` directory
 DIR_OUTPUTS=./outputs
 
+# Add environment variable for package directory
+DIR_COOKIECUTTER.REPO_NAME = ./{{ cookiecutter.repo_name }}
+
 # Add environment variables for the `src` directories
-DIR_SRC=./src
-DIR_SRC_MAKE_DATA=./src/make_data
-DIR_SRC_MAKE_FEATURES=./src/make_features
-DIR_SRC_MAKE_MODELS=./src/make_models
-DIR_SRC_MAKE_VISUALISATIONS=./src/make_visualisations
-DIR_SRC_UTILS=./src/utils
+DIR_COOKIECUTTER.REPO_NAME_SRC=./{{ cookiecutter.repo_name }}/src
+DIR_SRC_MAKE_DATA=./{{ cookiecutter.repo_name }}/src/make_data
+DIR_SRC_MAKE_FEATURES=./{{ cookiecutter.repo_name }}/src/make_features
+DIR_SRC_MAKE_MODELS=./{{ cookiecutter.repo_name }}/src/make_models
+DIR_SRC_MAKE_VISUALISATIONS=./{{ cookiecutter.repo_name }}/src/make_visualisations
+DIR_SRC_UTILS=./{{ cookiecutter.repo_name }}/src/utils
 
 # Add environment variables for the `tests` directory
-DIR_TESTS=./tests
+DIR_COOKIECUTTER.REPO_NAME_TESTS=./{{ cookiecutter.repo_name }}/tests

{{ cookiecutter.repo_name }}/.pre-commit-config.yaml

@@ -21,7 +21,7 @@ repos:
       - id: trailing-whitespace
         name: Check for trailing whitespaces (auto-fixes)
   - repo: https://github.com/pycqa/isort
-    rev: 5.8.0
+    rev: 5.12.0
     hooks:
       - id: isort
         name: isort - Sort Python imports (auto-fixes)

{{ cookiecutter.repo_name }}/README.md

@@ -32,6 +32,21 @@ To install the Python requirements, open your terminal and enter:
 pip install -r requirements.txt
 ```
 
+## Installing your package (Python Only)
+
+Whilst in the root folder, in the command prompt, you can install your package using:
+
+```shell
+pip install -e .
+```
+
+This installs an editable version of your package. Meaning, when you update your
+package code, you do not have to reinstall it for the changes to take effect.
+(This saves a lot of time when you test your code)
+
+Remember to update the setup and requirement files inline with any changes to your
+package. The inital files contain the bare minimum to get you started.
+
 ## Required secrets and credentials
 
 To run this project, [you need a `.secrets` file with secrets/credentials as

{{ cookiecutter.repo_name }}/docs/contributor_guide/writing_sphinx_documentation.md

@@ -47,12 +47,12 @@ the next subsection.
 
 ### Automatically creating docstring documentation (ReST)
 
-Let us say that `src/__init__.py` has functions called `hello` and `world` imported
+Let us say that `{{ cookiecutter.repo_name }}/__init__.py` has functions called `hello` and `world` imported
 into it, and both have docstrings. To automatically generate docstring documentation,
-create a ReST file, and add the following line to reference the `src` module:
+create a ReST file, and add the following line to reference the `{{ cookiecutter.repo_name }}` module:
 
 ```rest
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 
 Then, elsewhere in the body, [call the `autosummary` directive to generate the
@@ -82,13 +82,13 @@ it. We will cover some of the more widely used elements in the following subsect
 
 ### Automatically creating docstring documentation (MyST Markdown)
 
-Let us say that `src/__init__.py` has functions called `hello` and `world` imported
+Let us say that `{{ cookiecutter.repo_name }}/__init__.py` has functions called `hello` and `world` imported
 into it, and both have docstrings. To automatically generate docstring documentation,
-create a Markdown file, and add the following line to reference the `src` module:
+create a Markdown file, and add the following line to reference the `{{ cookiecutter.repo_name }}` module:
 
 ````md
 ```{eval-rst}
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 ````
 

{{ cookiecutter.repo_name }}/docs/reference/make_data.md

@@ -1,9 +1,9 @@
 # Data generation
 
-These `src` package functions generate data.
+These `{{ cookiecutter.repo_name }}` package functions generate data.
 
 ```{eval-rst}
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 
 ## Function heading

{{ cookiecutter.repo_name }}/docs/reference/make_features.md

@@ -1,9 +1,9 @@
 # Feature generation
 
-These `src` package functions create features.
+These `{{ cookiecutter.repo_name }}` package functions create features.
 
 ```{eval-rst}
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 
 ## Function heading

{{ cookiecutter.repo_name }}/docs/reference/make_models.md

@@ -1,9 +1,9 @@
 # Model generation
 
-These `src` package functions create models.
+These `{{ cookiecutter.repo_name }}` package functions create models.
 
 ```{eval-rst}
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 
 ## Function heading

{{ cookiecutter.repo_name }}/docs/reference/make_visualisations.md

@@ -1,9 +1,9 @@
 # Create visualisations
 
-These `src` package functions create visualisations.
+These `{{ cookiecutter.repo_name }}` package functions create visualisations.
 
 ```{eval-rst}
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 
 ## Function heading

{{ cookiecutter.repo_name }}/docs/reference/src.md

@@ -1,9 +1,9 @@
 # Package functions
 
-These are miscellaneous functions in the `src` package.
+These are miscellaneous functions in the `{{ cookiecutter.repo_name }}` package.
 
 ```{eval-rst}
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 
 ## Function heading

{{ cookiecutter.repo_name }}/docs/reference/utils.md

@@ -1,9 +1,9 @@
 # Create utility functions
 
-These `src` package functions create utility functions.
+These `{{ cookiecutter.repo_name }}` package functions create utility functions.
 
 ```{eval-rst}
-.. currentmodule:: src
+.. currentmodule:: {{ cookiecutter.repo_name }}
 ```
 
 ## Function heading

{{ cookiecutter.repo_name }}/docs/structure/{{ cookiecutter.repo_name }}.md

@@ -0,0 +1,3 @@
+```{include} ../../{{ cookiecutter.repo_name }}/README.md
+:relative-docs: ../docs/structure
+```

{{ cookiecutter.repo_name }}/pyproject.toml

@@ -1,7 +1,7 @@
 # `coverage` configurations
 [tool.coverage.run]
 source = [
-    "./src"
+    "./"
 ]
 
 [tool.coverage.report]
@@ -21,5 +21,5 @@ addopts = [
 ]
 doctest_optionflags = "NORMALIZE_WHITESPACE"
 testpaths = [
-    "./tests"
+    "./{{ cookiecutter.repo_name }}/tests"
 ]

{{ cookiecutter.repo_name }}/setup.cfg

@@ -0,0 +1,27 @@
+[metadata]
+name = {{ cookiecutter.repo_name }}
+description = a brief description
+author = {{ cookiecutter.organisation_handle }}
+platforms = win32
+classifiers =
+    Programming Language :: Python :: 3.6
+    Programming Language :: Python :: 3.7
+    Programming Language :: Python :: 3.8
+    Programming Language :: Python :: 3.9
+
+[options]
+packages =
+    {{ cookiecutter.repo_name }}
+install_requires =
+    coverage
+    detect-secrets == 1.0.3
+    myst-parser
+    pre-commit
+    pytest
+    detect-secrets
+    python-dotenv
+    Sphinx
+    toml
+python_requires = >=3.6
+
+zip_safe = no

{{ cookiecutter.repo_name }}/setup.py

@@ -0,0 +1,4 @@
+from setuptools import setup
+
+if __name__ == "__main__":
+    setup()