Inventory Data Plugin

[ubaumann]

Would it be useful for someone else to have a way to replace the default data store mechanism in an inventory?

I want to be able to use Pydantic Models and not a dictionary for the Host.data, Group.data and Defaults.data. The WIP PR #888 should explain how I would implement such a plugin system. This is many to have something to discuss about the idea.

The question would be if this is interesting for other users and if the change should be done in a way to avoid breaking changes or not.

[ubaumann]

This feature is not a show-stopper for me, and I can work around it by using transform functions and storing multiple pydantic models in the first level of the data dictionary. Still, it may be helpful for someone else to implement this as a plugin system.

[dbarrosop]

I am not sure if this should be a new plugin system on itself or a transform_function like you suggest or even something that inventory plugins do. The benefit of putting it in the core directly is that it becomes independent to other pieces but it adds some complexity.

My gut feeling is that this is so trivial to achieve with a transform function that it feels a bit unnecessary to add a plugin system but I may be missing something. I like the idea of data validaton though. Any thoughts @ktbyers?

[ubaumann]

The reasons why I was thinking of adding a new plugin systems are more or less the following:

1. Separation from Inventory (I can use a existing inventory without replacing the data store or with "any" data store)
2. Data can be converted / passed when the Host, Group, Default objects are created. I can after creating the options loop over the inventory (using transform function or just by my self) but could get issues with huge inventories. Updating Group and Default would be easier in a classic loop and not in the transform function but could also be done in the transform function if needed to.
3. Plugins could be registered and used by the community (in the case of Pydantic models the created and used models would still be needed to be specified in the configuration but maybe someone would like to store the data in Redis and writes a plugin and it would be easy to use)

However, if I am the only one using it, it does not make sense to add a new plugin. I can definitely work with transformer functions or self implemented inventories. I have done it in the past and besides of the mypy error it works well.

I would be happy to implement it if it brings value to the community and does not make nornir more complex for a small gain.

[dbarrosop]

if I am the only one using it

well, someone has to be the first user :) What we need to evaluate is the complexity vs gains vs alternatives

Separation from Inventory (I can use a existing inventory without replacing the data store or with "any" data store)

This is true as well with a transform_function though

Plugins could be registered and used by the community

This is the part I am not so sure about, unless we are talking about models like openconfig or the IETF ones, I really wouldn't expect users to agree on reusable models :P Maybe if the models are tied to an inventory plugin that could be a thing but then that plugin is where things should live I think.

I really think this is a useful feature but I am unsure what's the best path forward. I'd love to hear from maintainers/large_users_of netbox/nsot/nautobot and/or users of opencofig/ietf models to see what they think.

Also, one more thing, how does this play with the typing system? Do we expect to use generics in some way? Or just blind casting? You mentioned templates so maybe you don't really care but I think it's an important point as we can't assume the methods added to these models will be used exclusively in that context.

[ubaumann]

This is the part I am not so sure about, unless we are talking about models like openconfig or the IETF ones, I really wouldn't expect users to agree on reusable models :P Maybe if the models are tied to an inventory plugin that could be a thing but then that plugin is where things should live I think.

Finding a standard data structure is not realistic for the moment. I will take the Pydantic example to explain my idea. Users can write and create their own models and define the data structure. This depends, of course, on the input data from the inventory plugin, but it would be possible to change the data structure in the model initialization. For example, a User defines the following simplified pseudo model:

from typing import Literal, Union
from ipaddress import IPv4Interface, IPv6Interface
from pydantic import BaseModel, Field, EmailStr
from pydantic.networks import IPvAnyAddress

class RotedInterface(BaseModel):
    kind: Literal["routed"]
    ipv4: IPv4Interface
    ipv6: IPv6Interface
    shutdown: bool

    def no_shutdown(self):
        return "RPC/JSON or other payload to execute no shutdown" 

class Switchport(BaseModel):
    kind: Literal["sitchport"]
    vlan: int
    shutdown: bool

    def no_shutdown(self):
        return "RPC/JSON or another payload to execute no shutdown." 

class MyCustHostData(BaseModel):
    interfaces: Union[RoutedInterface, Switchport] = Field(..., discriminator="kind")
    snmp_contact: EmailStr
    snmp_location: str

Now, in the configuration, the model needs to be specified.

---
inventory:
  plugin: SimpleInventory
  options:
    host_file: "inventory/hosts.yaml"
    group_file: "inventory/groups.yaml"
    defaults_file: "inventory/defaults.yaml"
inventory_data:
    plugin: PydanticDataStore
    options:
        hosts: "mycode.models.MyCustHostData"
        groups: "mycode.models.MyCustGroupData"
        default: "mycode.models.MyCustDefaultData"

The plugin must implement the logic to find the proper object, load the model and store it in the Host. It should be flexible and support different models for Host, Group and Default inventory objects to be useful for many people. Two significant benefits I see are the enforcement of a schema for all data in the inventory and working with objects, which is an excellent way to reduce logic in templates and make it easier to test.

My vision would be to put some effort into the project "Pydantify", which can generate Pydantic models from YANG (not v2 at the moment) and store them in Nornir directly. With some development, creating NETCONF, RESTconf, or gNMI payload should be possible without templates. This still needs a lot of work from my side, tbh.

Also, one more thing, how does this play with the typing system? Do we expect to use generics in some way? Or just blind casting? You mentioned templates so maybe you don't really care but I think it's an important point as we can't assume the methods added to these models will be used exclusively in that context.

We currently have data = dict[str, Any]. Typing is important and will be one of the challenges. In the WIP PR, I added a new Protocol object:
https://github.com/nornir-automation/nornir/pull/888/files#diff-05c67254f84f8fe7478dea5c3d081018582963f6f0a7678f533aebd5d3aef3e3R18-R59

As a plugin (or a User) can store anything in the inventory, I think we need to keep the Any.