Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ✨Add jinja settings support for golden config plugin #527

Merged
merged 13 commits into from
Aug 29, 2023
Merged

feat: ✨Add jinja settings support for golden config plugin #527

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d3941b8
feat: 🔧 Add jinja settings support for golden config plugin
cablesquirrel Jul 17, 2023
444b7fa
feat: ✨ Add support for jinja environment options
cablesquirrel Jul 17, 2023
47d96d0
fix: 🐛 Change default setting for lstrip_blocks to False
cablesquirrel Jul 17, 2023
c0287ac
fix: 🐛 Correct environment variable name for lstrip_blocks option
cablesquirrel Jul 17, 2023
d655eea
docs: 📝 Update docs to add jinja environment options
cablesquirrel Jul 17, 2023
bfa531b
docs: 💡 Correct spelling of 'django' in comment
erhansen-cox Aug 28, 2023
a450d6c
refactor: ♻️ Move Jinja ENV defaults
erhansen-cox Aug 28, 2023
5162ea9
feat: ✨ Allow setting of all Jinja2 Environment options
erhansen-cox Aug 29, 2023
7c75303
fix: 📝 Correct code fence in markdown
erhansen-cox Aug 29, 2023
1f201ac
Merge branch 'next' into feat-jinja2-env-settings
Aug 29, 2023
6710c5a
docs: 📝 Indent code block in notes
erhansen-cox Aug 29, 2023
a460597
docs: 📝 Correct example plugin config for Jinja2 environment settings
erhansen-cox Aug 29, 2023
f35ef16
fix: 🐛 Use jinja variable names in place of the nautobot settings names
erhansen-cox Aug 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion development/nautobot_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import sys

from nautobot.core.settings import * # noqa: F403
from nautobot.core.settings_funcs import parse_redis_connection
from nautobot.core.settings_funcs import is_truthy, parse_redis_connection


#
Expand Down Expand Up @@ -174,6 +174,8 @@
"enable_postprocessing": is_truthy(os.environ.get("ENABLE_POSTPROCESSING", True)),
"postprocessing_callables": os.environ.get("POSTPROCESSING_CALLABLES", []),
"postprocessing_subscribed": os.environ.get("POSTPROCESSING_SUBSCRIBED", []),
"jinja_env_trim_blocks": is_truthy(os.getenv("NAUTOBOT_JINJA_ENV_TRIM_BLOCKS", True)),
"jinja_env_lstrip_blocks": is_truthy(os.getenv("NAUTOBOT_JINJA_ENV_LSTRIP_BLOCKS", False)),
# The platform_slug_map maps an arbitrary platform slug to its corresponding parser.
# Use this if the platform slug names in your Nautobot instance don't correspond exactly
# to the Nornir driver names ("arista_eos", "cisco_ios", etc.).
Expand Down
8 changes: 6 additions & 2 deletions docs/admin/admin_install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ PLUGINS_CONFIG = {
"postprocessing_callables": [],
"postprocessing_subscribed": [],
"platform_slug_map": None,
"jinja_env_trim_blocks": True,
"jinja_env_lstrip_blocks": False,
itdependsnetworks marked this conversation as resolved.
Show resolved Hide resolved
# "get_custom_compliance": "my.custom_compliance.func"
},
}
Expand Down Expand Up @@ -93,14 +95,16 @@ The plugin behavior can be controlled with the following list of settings.
| enable_compliance | True | True | A boolean to represent whether or not to run the compliance process within the plugin. |
| enable_intended | True | True | A boolean to represent whether or not to generate intended configurations within the plugin. |
| enable_sotagg | True | True | A boolean to represent whether or not to provide a GraphQL query per device to allow the intended configuration to provide data variables to the plugin. |
| enable_postprocessing | True | False | A boolean to represent whether or not to generate intended configurations to push, with extra processing such as secrets rendering. |
| enable_postprocessing | True | False | A boolean to represent whether or not to generate intended configurations to push, with extra processing such as secrets rendering. |
| postprocessing_callables | ['mypackage.myfunction'] | [] | A list of function paths, in dotted format, that are appended to the available methods for post-processing the intended configuration, for instance, the `render_secrets`. |
| postprocessing_subscribed | ['mypackage.myfunction'] | [] | A list of function paths, that should exist as postprocessing_callables, that defines the order of application of during the post-processing process. |
| platform_slug_map | {"cisco_wlc": "cisco_aireos"} | None | A dictionary in which the key is the platform slug and the value is what netutils uses in any "network_os" parameter within `netutils.config.compliance.parser_map`. |
| platform_slug_map | {"cisco_wlc": "cisco_aireos"} | None | A dictionary in which the key is the platform slug and the value is what netutils uses in any "network_os" parameter within `netutils.config.compliance.parser_map`. |
| sot_agg_transposer | "mypkg.transposer" | None | A string representation of a function that can post-process the graphQL data. |
| per_feature_bar_width | 0.15 | 0.15 | The width of the table bar within the overview report |
| per_feature_width | 13 | 13 | The width in inches that the overview table can be. |
| per_feature_height | 4 | 4 | The height in inches that the overview table can be. |
| jinja_env_trim_blocks | True | True | A boolean to represent whether the jinja2 option for "trim_blocks" should be enabled for intended config rendering |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After getting internal feedback, I think the point @jeffkala and @electro47h is that there are many variables one could set for Jinja, I think the list is:

  • block_start_string
  • block_end_string
  • variable_start_string
  • variable_end_string
  • comment_start_string
  • comment_end_string
  • line_statement_prefix
  • line_comment_prefix
  • trim_blocks
  • lstrip_blocks
  • newline_sequence
  • keep_trailing_newline
  • extensions
  • optimized
  • undefined
  • finalize
  • autoescape
  • loader
  • cache_size
  • auto_reload
  • bytecode_cache
  • enable_async

It probably makes sense to support them, as a dictionary, with the 3 variables you have set as defaults. So, there should likely be one variable called jinja_env that is set. If you want to have a magic method of some sort such that anything set with jinja_env_* automagically gets into the dictionary, I could buy that as well, as long as it is in addition to the dictionary method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. None of those were possible to set before, but since they will be now, they should be included. This will take a good bit of rework on my end. I'm going to draft this PR and work on it later this week if time allows.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question for the masses here. For an option such as 'undefined', how would that look in the nautobot_config.py file for a setting?

from jinja2 import StrictUndefined

Environment(undefined=StrictUndefined, lstrip_blocks=...)

It appears most of the settings are boolean or string values, but a few of the options listed for Jinja are enumerations or classes. If the user intended to customize that setting in their nautobot_config.py file, would they have to add the import, or is there another way to represent that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the user intended to customize that setting in their nautobot_config.py file, would they have to add the import

Yes, they would have to add that to the import within nautobot_config.py. We could work around it, but the complexity is not really worth it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I have committed what my interpretation of the request is.

I'm out of the office tomorrow and will be unable to test. We also have to get our environment upgraded from 1.5.24 to >=1.6.1 to install this version of the plugin. I will work to get this tested when I can.

Please let me know if this is the correct architecture that you're looking for, or if I need to make changes. Happy to keep plugging away at this.

| jinja_env_lstrip_blocks | True | False | A boolean to represent whether the jinja2 option for "lstrip_blocks" should be enabled for intended config rendering |

!!! note
Over time the compliance report will become more dynamic, but for now allow users to configure the `per_*` configs in a way that fits best for them.
Expand Down
21 changes: 20 additions & 1 deletion nautobot_golden_config/nornir_plays/config_intended.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
from nornir.core.task import Result, Task

from django.template import engines
from jinja2 import StrictUndefined
from jinja2.sandbox import SandboxedEnvironment

from nornir_nautobot.exceptions import NornirNautobotException
from nornir_nautobot.plugins.tasks.dispatcher import dispatcher
Expand All @@ -18,6 +20,7 @@
from nautobot_plugin_nornir.constants import NORNIR_SETTINGS
from nautobot_plugin_nornir.utils import get_dispatcher

from nautobot_golden_config.utilities.constant import PLUGIN_CFG
from nautobot_golden_config.utilities.db_management import close_threaded_db_connections
from nautobot_golden_config.models import GoldenConfigSetting, GoldenConfig
from nautobot_golden_config.utilities.helper import (
Expand All @@ -32,7 +35,22 @@
InventoryPluginRegister.register("nautobot-inventory", NautobotORMInventory)
LOGGER = logging.getLogger(__name__)

jinja_env = engines["jinja"].env
# Use a custom Jinja2 environment instead of Django's to avoid HTML escaping
# Trim_blocks option defaulted to True to match nornir's default environment
OPTION_LSTRIP_BLOCKS = False
OPTION_TRIM_BLOCKS = True
itdependsnetworks marked this conversation as resolved.
Show resolved Hide resolved
if PLUGIN_CFG.get("jinja_env_trim_blocks"):
OPTION_TRIM_BLOCKS = PLUGIN_CFG.get("jinja_env_trim_blocks")
if PLUGIN_CFG.get("jinja_env_lstrip_blocks"):
OPTION_LSTRIP_BLOCKS = PLUGIN_CFG.get("jinja_env_lstrip_blocks")

jinja_env = SandboxedEnvironment(
undefined=StrictUndefined,
trim_blocks=OPTION_TRIM_BLOCKS,
lstrip_blocks=OPTION_LSTRIP_BLOCKS,
)
# Retrieve filters from the Djanog jinja template engine
jinja_env.filters = engines["jinja"].env.filters
itdependsnetworks marked this conversation as resolved.
Show resolved Hide resolved


@close_threaded_db_connections
Expand Down Expand Up @@ -85,6 +103,7 @@ def run_template( # pylint: disable=too-many-arguments
output_file_location=output_file_location,
default_drivers_mapping=get_dispatcher(),
jinja_filters=jinja_env.filters,
jinja_env=jinja_env,
itdependsnetworks marked this conversation as resolved.
Show resolved Hide resolved
)[1].result["config"]
intended_obj.intended_last_success_date = task.host.defaults.data["now"]
intended_obj.intended_config = generated_config
Expand Down