Merge pull request #9041 from netbox-community/develop

Release v3.2.0

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -14,7 +14,7 @@ body:
     attributes:
       label: NetBox version
       description: What version of NetBox are you currently running?
-      placeholder: v3.1.11
+      placeholder: v3.2.0
     validations:
       required: true
   - type: dropdown

.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -14,7 +14,7 @@ body:
     attributes:
       label: NetBox version
       description: What version of NetBox are you currently running?
-      placeholder: v3.1.11
+      placeholder: v3.2.0
     validations:
       required: true
   - type: dropdown

.github/workflows/ci.yml

@@ -3,10 +3,12 @@ on: [push, pull_request]
 jobs:
   build:
     runs-on: ubuntu-latest
+    env:
+      NETBOX_CONFIGURATION: netbox.configuration_testing
     strategy:
       matrix:
-        python-version: [3.7, 3.8, 3.9]
-        node-version: [14.x]
+        python-version: ['3.8', '3.9', '3.10']
+        node-version: ['14.x']
     services:
       redis:
         image: redis
@@ -57,7 +59,6 @@ jobs:
         python -m pip install --upgrade pip
         pip install -r requirements.txt
         pip install pycodestyle coverage tblib
-        ln -s configuration.testing.py netbox/netbox/configuration.py
 
     - name: Build documentation
       run: mkdocs build

.readthedocs.yaml

@@ -0,0 +1,10 @@
+version: 2
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.9"
+mkdocs:
+  configuration: mkdocs.yml
+python:
+   install:
+   - requirements: requirements.txt

base_requirements.txt

@@ -1,6 +1,6 @@
 # The Python web framework on which NetBox is built
 # https://github.com/django/django
-Django<4.0
+Django
 
 # Django middleware which permits cross-domain API requests
 # https://github.com/OttoYiu/django-cors-headers
@@ -68,7 +68,8 @@ gunicorn
 
 # Platform-agnostic template rendering engine
 # https://github.com/pallets/jinja
-Jinja2
+# Pin to v3.0 for mkdocstrings
+Jinja2<3.1
 
 # Simple markup language for rendering HTML
 # https://github.com/Python-Markdown/markdown
@@ -82,6 +83,10 @@ markdown-include
 # https://github.com/squidfunk/mkdocs-material
 mkdocs-material
 
+# Introspection for embedded code
+# https://github.com/mkdocstrings/mkdocstrings
+mkdocstrings<=0.17.0
+
 # Library for manipulating IP prefixes and addresses
 # https://github.com/netaddr/netaddr
 netaddr
@@ -113,3 +118,7 @@ svgwrite
 # Tabular dataset library (for table-based exports)
 # https://github.com/jazzband/tablib
 tablib
+
+# Timezone data (required by django-timezone-field on Python 3.9+)
+# https://github.com/python/tzdata
+tzdata

docs/configuration/dynamic-settings.md

@@ -66,6 +66,22 @@ CUSTOM_VALIDATORS = {
 
 ---
 
+## DEFAULT_USER_PREFERENCES
+
+This is a dictionary defining the default preferences to be set for newly-created user accounts. For example, to set the default page size for all users to 100, define the following:
+
+```python
+DEFAULT_USER_PREFERENCES = {
+    "pagination": {
+        "per_page": 100
+    }
+}
+```
+
+For a complete list of available preferences, log into NetBox and navigate to `/user/preferences/`. A period in a preference name indicates a level of nesting in the JSON data. The example above maps to `pagination.per_page`.
+
+---
+
 ## ENFORCE_GLOBAL_UNIQUE
 
 Default: False

docs/configuration/index.md

@@ -1,6 +1,11 @@
 # NetBox Configuration
 
-NetBox's local configuration is stored in `$INSTALL_ROOT/netbox/netbox/configuration.py`. An example configuration is provided as `configuration.example.py`. You may copy or rename the example configuration and make changes as appropriate. NetBox will not run without a configuration file.  While NetBox has many configuration settings, only a few of them must be defined at the time of installation: these are defined under "required settings" below.
+NetBox's local configuration is stored in `$INSTALL_ROOT/netbox/netbox/configuration.py` by default. An example configuration is provided as `configuration_example.py`. You may copy or rename the example configuration and make changes as appropriate. NetBox will not run without a configuration file.  While NetBox has many configuration settings, only a few of them must be defined at the time of installation: these are defined under "required settings" below.
+
+!!! info "Customizing the Configuration Module"
+    A custom configuration module may be specified by setting the `NETBOX_CONFIGURATION` environment variable. This must be a dotted path to the desired Python module. For example, a file named `my_config.py` in the same directory as `settings.py` would be referenced as `netbox.my_config`.
+
+    For the sake of brevity, the NetBox documentation refers to the configuration file simply as `configuration.py`.
 
 Some configuration parameters may alternatively be defined either in `configuration.py` or within the administrative section of the user interface. Settings which are "hard-coded" in the configuration file take precedence over those defined via the UI.
 

docs/configuration/optional-settings.md

@@ -13,6 +13,23 @@ ADMINS = [
 
 ---
 
+## AUTH_PASSWORD_VALIDATORS
+
+This parameter acts as a pass-through for configuring Django's built-in password validators for local user accounts. If configured, these will be applied whenever a user's password is updated to ensure that it meets minimum criteria such as length or complexity. An example is provided below. For more detail on the available options, please see [the Django documentation](https://docs.djangoproject.com/en/stable/topics/auth/passwords/#password-validation).
+
+```python
+AUTH_PASSWORD_VALIDATORS = [
+    {
+        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
+        'OPTIONS': {
+            'min_length': 10,
+        }
+    },
+]
+```
+
+---
+
 ## BASE_PATH
 
 Default: None
@@ -49,6 +66,21 @@ CORS_ORIGIN_WHITELIST = [
 
 ---
 
+## CSRF_TRUSTED_ORIGINS
+
+Default: `[]`
+
+Defines a list of trusted origins for unsafe (e.g. `POST`) requests. This is a pass-through to Django's [`CSRF_TRUSTED_ORIGINS`](https://docs.djangoproject.com/en/4.0/ref/settings/#std:setting-CSRF_TRUSTED_ORIGINS) setting. Note that each host listed must specify a scheme (e.g. `http://` or `https://).
+
+```python
+CSRF_TRUSTED_ORIGINS = (
+    'http://netbox.local',
+    'https://netbox.local',
+)
+```
+
+---
+
 ## DEBUG
 
 Default: False
@@ -140,6 +172,66 @@ EXEMPT_VIEW_PERMISSIONS = ['*']
 
 ---
 
+## FIELD_CHOICES
+
+Some static choice fields on models can be configured with custom values. This is done by defining `FIELD_CHOICES` as a dictionary mapping model fields to their choices. Each choice in the list must have a database value and a human-friendly label, and may optionally specify a color. (A list of available colors is provided below.)
+
+The choices provided can either replace the stock choices provided by NetBox, or append to them. To _replace_ the available choices, specify the app, model, and field name separated by dots. For example, the site model would be referenced as `dcim.Site.status`. To _extend_ the available choices, append a plus sign to the end of this string (e.g. `dcim.Site.status+`).
+
+For example, the following configuration would replace the default site status choices with the options Foo, Bar, and Baz:
+
+```python
+FIELD_CHOICES = {
+    'dcim.Site.status': (
+        ('foo', 'Foo', 'red'),
+        ('bar', 'Bar', 'green'),
+        ('baz', 'Baz', 'blue'),
+    )
+}
+```
+
+Appending a plus sign to the field identifier would instead _add_ these choices to the ones already offered:
+
+```python
+FIELD_CHOICES = {
+    'dcim.Site.status+': (
+        ...
+    )
+}
+```
+
+The following model fields support configurable choices:
+
+* `circuits.Circuit.status`
+* `dcim.Device.status`
+* `dcim.PowerFeed.status`
+* `dcim.Rack.status`
+* `dcim.Site.status`
+* `extras.JournalEntry.kind`
+* `ipam.IPAddress.status`
+* `ipam.IPRange.status`
+* `ipam.Prefix.status`
+* `ipam.VLAN.status`
+* `virtualization.VirtualMachine.status`
+
+The following colors are supported:
+
+* `blue`
+* `indigo`
+* `purple`
+* `pink`
+* `red`
+* `orange`
+* `yellow`
+* `green`
+* `teal`
+* `cyan`
+* `gray`
+* `black`
+* `white`
+
+---
+
 ## HTTP_PROXIES
 
 Default: None

docs/core-functionality/device-types.md

@@ -37,4 +37,5 @@ Once component templates have been created, every new device that you create as
 {!models/dcim/interfacetemplate.md!}
 {!models/dcim/frontporttemplate.md!}
 {!models/dcim/rearporttemplate.md!}
+{!models/dcim/modulebaytemplate.md!}
 {!models/dcim/devicebaytemplate.md!}

docs/core-functionality/devices.md

@@ -17,6 +17,7 @@ Device components represent discrete objects within a device which are used to t
 {!models/dcim/interface.md!}
 {!models/dcim/frontport.md!}
 {!models/dcim/rearport.md!}
+{!models/dcim/modulebay.md!}
 {!models/dcim/devicebay.md!}
 {!models/dcim/inventoryitem.md!}
 

docs/core-functionality/modules.md

@@ -0,0 +1,4 @@
+# Modules
+
+{!models/dcim/moduletype.md!}
+{!models/dcim/module.md!}

docs/core-functionality/services.md

@@ -1,3 +1,4 @@
 # Service Mapping
 
+{!models/ipam/servicetemplate.md!}
 {!models/ipam/service.md!}

docs/customization/reports.md

@@ -95,7 +95,7 @@ The following methods are available to log results within a report:
 
 The recording of one or more failure messages will automatically flag a report as failed. It is advised to log a success for each object that is evaluated so that the results will reflect how many objects are being reported on. (The inclusion of a log message is optional for successes.) Messages recorded with `log()` will appear in a report's results but are not associated with a particular object or status. Log messages also support using markdown syntax and will be rendered on the report result page.
 
-To perform additional tasks, such as sending an email or calling a webhook, after a report has been run, extend the `post_run()` method. The status of the report is available as `self.failed` and the results object is `self.result`.
+To perform additional tasks, such as sending an email or calling a webhook, before or after a report is run, extend the `pre_run()` and/or `post_run()` methods, respectively. The status of a completed report is available as `self.failed` and the results object is `self.result`.
 
 By default, reports within a module are ordered alphabetically in the reports list page. To return reports in a specific order, you can define the `report_order` variable at the end of your module. The `report_order` variable is a tuple which contains each Report class in the desired order. Any reports that are omitted from this list will be listed last.
 

docs/development/adding-models.md

@@ -2,7 +2,7 @@
 
 ## 1. Define the model class
 
-Models within each app are stored in either `models.py` or within a submodule under the `models/` directory. When creating a model, be sure to subclass the [appropriate base model](models.md) from `netbox.models`. This will typically be PrimaryModel or OrganizationalModel. Remember to add the model class to the `__all__` listing for the module.
+Models within each app are stored in either `models.py` or within a submodule under the `models/` directory. When creating a model, be sure to subclass the [appropriate base model](models.md) from `netbox.models`. This will typically be NetBoxModel or OrganizationalModel. Remember to add the model class to the `__all__` listing for the module.
 
 Each model should define, at a minimum:
 

docs/development/getting-started.md

@@ -11,17 +11,25 @@ Getting started with NetBox development is pretty straightforward, and should fe
 
 ### Fork the Repo
 
-Assuming you'll be working on your own fork, your first step will be to fork the [official git repository](https://github.com/netbox-community/netbox). (If you're a maintainer who's going to be working directly with the official repo, skip this step.) You can then clone your GitHub fork locally for development:
+Assuming you'll be working on your own fork, your first step will be to fork the [official git repository](https://github.com/netbox-community/netbox). (If you're a maintainer who's going to be working directly with the official repo, skip this step.) Click the "fork" button at top right (be sure that you've logged into GitHub first).
+
+![GitHub fork button](../media/development/github_fork_button.png)
+
+Copy the URL provided in the dialog box.
+
+![GitHub fork dialog](../media/development/github_fork_dialog.png)
+
+You can then clone your GitHub fork locally for development:
 
 ```no-highlight
-$ git clone https://github.com/youruseraccount/netbox.git
+$ git clone https://github.com/$username/netbox.git
 Cloning into 'netbox'...
-remote: Enumerating objects: 231, done.
-remote: Counting objects: 100% (231/231), done.
-remote: Compressing objects: 100% (147/147), done.
-remote: Total 56705 (delta 134), reused 145 (delta 84), pack-reused 56474
-Receiving objects: 100% (56705/56705), 27.96 MiB | 34.92 MiB/s, done.
-Resolving deltas: 100% (44177/44177), done.
+remote: Enumerating objects: 85949, done.
+remote: Counting objects: 100% (4672/4672), done.
+remote: Compressing objects: 100% (1224/1224), done.
+remote: Total 85949 (delta 3538), reused 4332 (delta 3438), pack-reused 81277
+Receiving objects: 100% (85949/85949), 55.16 MiB | 44.90 MiB/s, done.
+Resolving deltas: 100% (68008/68008), done.
 $ ls netbox/
 base_requirements.txt  contrib          docs         mkdocs.yml  NOTICE     requirements.txt  upgrade.sh
 CHANGELOG.md           CONTRIBUTING.md  LICENSE.txt  netbox      README.md  scripts
@@ -33,7 +41,7 @@ The NetBox project utilizes three persistent git branches to track work:
 * `develop` - All development on the upcoming stable release occurs here
 * `feature` - Tracks work on an upcoming major release
 
-Typically, you'll base pull requests off of the `develop` branch, or off of `feature` if you're working on a new major release. **Never** merge pull requests into the `master` branch, which receives merged only from the `develop` branch.
+Typically, you'll base pull requests off of the `develop` branch, or off of `feature` if you're working on a new major release. **Never** merge pull requests into the `master` branch: This branch only ever merges pull requests from the `develop` branch, to effect a new release.
 
 For example, assume that the current NetBox release is v3.1.1. Work applied to the `develop` branch will appear in v3.1.2, and work done under the `feature` branch will be included in the next minor release (v3.2.0).
 
@@ -60,7 +68,7 @@ $ python3 -m venv ~/.venv/netbox
 This will create a directory named `.venv/netbox/` in your home directory, which houses a virtual copy of the Python executable and its related libraries and tooling. When running NetBox for development, it will be run using the Python binary at `~/.venv/netbox/bin/python`.
 
 !!! info "Where to Create Your Virtual Environments"
-    Keeping virtual environments in `~/.venv/` is a common convention but entirely optional: Virtual environments can be created almost wherever you please.
+    Keeping virtual environments in `~/.venv/` is a common convention but entirely optional: Virtual environments can be created almost wherever you please. Also consider using [`virtualenvwrapper`](https://virtualenvwrapper.readthedocs.io/en/stable/) to simplify the management of multiple venvs.
 
 Once created, activate the virtual environment:
 
@@ -85,7 +93,7 @@ Collecting Django==3.1 (from -r requirements.txt (line 1))
 
 ### Configure NetBox
 
-Within the `netbox/netbox/` directory, copy `configuration.example.py` to `configuration.py` and update the following parameters:
+Within the `netbox/netbox/` directory, copy `configuration_example.py` to `configuration.py` and update the following parameters:
 
 * `ALLOWED_HOSTS`: This can be set to `['*']` for development purposes
 * `DATABASE`: PostgreSQL database connection parameters
@@ -99,12 +107,13 @@ Within the `netbox/netbox/` directory, copy `configuration.example.py` to `confi
 Django provides a lightweight, auto-updating HTTP/WSGI server for development use. It is started with the `runserver` management command:
 
 ```no-highlight
-$ python netbox/manage.py runserver
+$ ./manage.py runserver
+Watching for file changes with StatReloader
 Performing system checks...
 
 System check identified no issues (0 silenced).
-November 18, 2020 - 15:52:31
-Django version 3.1, using settings 'netbox.settings'
+February 18, 2022 - 20:29:57
+Django version 4.0.2, using settings 'netbox.settings'
 Starting development server at http://127.0.0.1:8000/
 Quit the server with CONTROL-C.
 ```
@@ -122,26 +131,36 @@ The demo data is provided in JSON format and loaded into an empty database using
 
 ## Running Tests
 
-Prior to committing any substantial changes to the code base, be sure to run NetBox's test suite to catch any potential errors. Tests are run using the `test` management command. Remember to ensure the Python virtual environment is active before running this command. Also keep in mind that these commands are executed in the `/netbox/` directory, not the root directory of the repository.
+Prior to committing any substantial changes to the code base, be sure to run NetBox's test suite to catch any potential errors. Tests are run using the `test` management command, which employs Python's [`unittest`](https://docs.python.org/3/library/unittest.html#module-unittest) library. Remember to ensure the Python virtual environment is active before running this command. Also keep in mind that these commands are executed in the `netbox/` directory, not the root directory of the repository.
 
-When running tests, it's advised to use the special testing configuration file that ships with NetBox. This ensures that tests are run with the same configuration parameters consistently. To override your local configuration when running tests, set the `NETBOX_CONFIGURATION` environment variable to `netbox.configuration_testing`.
+To avoid potential issues with your local configuration file, set the `NETBOX_CONFIGURATION` to point to the packaged test configuration at `netbox/configuration_testing.py`. This will handle things like ensuring that the dummy plugin is enabled for comprehensive testing.
 
 ```no-highlight
-$ NETBOX_CONFIGURATION=netbox.configuration_testing python manage.py test
+$ export NETBOX_CONFIGURATION=netbox.configuration_testing
+$ cd netbox/
+$ python manage.py test
 ```
 
-In cases where you haven't made any changes to the database (which is most of the time), you can append the `--keepdb` argument to this command to reuse the test database between runs. This cuts down on the time it takes to run the test suite since the database doesn't have to be rebuilt each time. (Note that this argument will cause errors if you've modified any model fields since the previous test run.)
+In cases where you haven't made any changes to the database schema (which is typical), you can append the `--keepdb` argument to this command to reuse the test database between runs. This cuts down on the time it takes to run the test suite since the database doesn't have to be rebuilt each time. (Note that this argument will cause errors if you've modified any model fields since the previous test run.)
 
 ```no-highlight
 $ python manage.py test --keepdb
 ```
 
-You can also limit the command to running only a specific subset of tests. For example, to run only IPAM and DCIM view tests:
+You can also reduce testing time by enabling parallel test execution with the `--parallel` flag. (By default, this will run as many parallel tests as you have processors. To avoid sluggishness, it's a good idea to specify a lower number of parallel tests.) This flag can be combined with `--keepdb`, although if you encounter any strange errors, try running the test suite again with parallelization disabled.
+
+```no-highlight
+$ python manage.py test --parallel <n>
+```
+
+Finally, it's possible to limit the run to a specific set of tests, specified by their Python path. For example, to run only IPAM and DCIM view tests:
 
 ```no-highlight
 $ python manage.py test dcim.tests.test_views ipam.tests.test_views
 ```
 
+This is handy for instances where just a few tests are failing and you want to re-run them individually.
+
 ## Submitting Pull Requests
 
 Once you're happy with your work and have verified that all tests pass, commit your changes and push it upstream to your fork. Always provide descriptive (but not excessively verbose) commit messages. When working on a specific issue, be sure to prefix your commit message with the word "Fixes" or "Closes" and the issue number (with a hash mark). This tells GitHub to automatically close the referenced issue once the commit has been merged.