[Bug]: Enumerated DEFVAL with TEXTUAL-CONVENTION errors with Jinja template rendering error

[simonryf]

Expected behavior

mibdump compiles successfully (e.g. DOCS-IF3-MIB)

Actual behavior

Failed MIBs: DOCS-IF3-MIB (Jinja template rendering error: 'collections.OrderedDict object' has no attribute 'default' at MIB DOCS-IF3-MIB)

Detailed steps

This is a follow up from etingof/pysmi#72

Using only MIBs from https://github.com/lextudio/mibs.pysnmp.com/tree/master/asn1 I could successfully compile DOCS-IF3-MIB. However taking the latest DOCS-IF3-MIB from http://mibs.cablelabs.com/MIBs/DOCSIS/DOCS-IF3-MIB-2024-07-05.txt

I noticed a difference in DOCS-IF3-MIB, see specifically line 3242 DEFVAL { { bonding } }:

...
AttributeMask ::= TEXTUAL-CONVENTION
     STATUS      current
     DESCRIPTION
        "This data type consists of a sequence of 32-bit positions used
        to select the bonding group or the channel to which a service
        flow is assigned. DOCSIS defines three types of Attribute Masks
        for which this type applies: The Provisioned Attribute Mask
        that is configured to a Bonding Group or a single-channel,
        whereas the Required Attribute and the Forbidden Attribute
        Mask are part of the Service Flow QOS Parameter Set to be
        matched with the Provisioned Attribute Mask of CMTS-configured
        Bonding Groups or single-channels. DOCSIS reserves the
        assignment of the meaning of the first 8 bit positions
        (left to right) as follows:
            Bit 0: 'bonding'
            Bit 1: 'lowLatency'
            Bit 2: 'highAvailability'
            Bit positions 3-15 are reserved.

        Bit positions 16-31 are freely assigned by operators to
        represent their own constraints on the channel(s) selection
        for a particular service flow."
     REFERENCE
         "DOCSIS 3.0 MAC and Upper Layer Protocols Interface
         Specification CM-SP-MULPIv3.0-I08-080522, Service Flow
         Assignment section."
     SYNTAX      BITS {
                       bonding(0),
                       lowLatency(1),
                       highAvailability(2)
                      }
...

docsIf3BondingGrpCfgSfProvAttrMask OBJECT-TYPE
     SYNTAX      AttributeMask
     MAX-ACCESS  read-create
     STATUS      current
     DESCRIPTION
        "This attribute represents the Provisioned Attribute
        Mask encoding for the bonding group."
     DEFVAL { { bonding } }
     ::= { docsIf3BondingGrpCfgEntry 4 }

changing this line back (as it was in the version from mibs.pysnmp.com) to:

DEFVAL { '80000000'H }

it compiles successfully again. Hope this helps to track the issue down. Of course I'm not 100% confident that Enumerated DEFVAL with TEXTUAL-CONVENTION is actually something which is by the spec.

Python package information

pysmi-lextudio = "1.4.3"

Operating system information

macOS

Python information

3.12.4

(Optional) Contents of your test script

mkdir snmp-test && cd snmp-test
pyenv local 3.12.4
pyenv virtualenv snmp-test
pyenv local snmp-test

mkdir -p ./mibs && mkdir -p ./mib_cache

# get mibs from lextudio
mibs=(
    "CLAB-DEF-MIB"
    "DIFFSERV-DSCP-TC"
    "DIFFSERV-MIB"
    "DOCS-CABLE-DEVICE-MIB"
    "DOCS-IETF-BPI2-MIB"
    "DOCS-IF-MIB"
    "IANAifType-MIB"
    "IF-MIB"
    "INET-ADDRESS-MIB"
    "INTEGRATED-SERVICES-MIB"
    "RFC-1212"
    "RFC1155-SMI"
    "RFC1213-MIB"
    "RMON-MIB"
    "RMON2-MIB"
    "SNMP-FRAMEWORK-MIB"
    "SNMP-TARGET-MIB"
    "SNMPv2-CONF"
    "SNMPv2-MIB"
    "SNMPv2-SMI"
    "SNMPv2-TC"
)
url_prefix="https://raw.githubusercontent.com/lextudio/mibs.pysnmp.com/master/asn1/"
output_dir="./mibs"
mkdir -p "$output_dir"
for mib in "${mibs[@]}"; do
    wget "${url_prefix}${mib}" -O "${output_dir}/${mib}.mib"
done

# get DOCS-IF3-MIB from cablelabs
wget http://mibs.cablelabs.com/MIBs/DOCSIS/DOCS-IF3-MIB-2024-07-05.txt -O ./mibs/DOCS-IF3-MIB.mib

# compile mibs
mibdump --mib-source ./mibs --destination-directory ./mib_cache DOCS-IF3-MIB

Relevant log output

mibdump --mib-source ./mibs --destination-directory ./mib_cache DOCS-IF3-MIB
Source MIB repositories: ./mibs
Borrow missing/failed MIBs from: https://mibs.pysnmp.com:443/mibs/notexts/@mib@
Existing/compiled MIB locations: pysnmp.smi.mibs, pysnmp_mibs
Compiled MIBs destination directory: ./mib_cache
MIBs excluded from code generation: INET-ADDRESS-MIB, PYSNMP-USM-MIB, RFC-1212, RFC-1215, RFC1065-SMI, RFC1155-SMI, RFC1158-MIB, RFC1213-MIB, SNMP-FRAMEWORK-MIB, SNMP-TARGET-MIB, SNMPv2-CONF, SNMPv2-SMI, SNMPv2-TC, SNMPv2-TM, TRANSPORT-ADDRESS-MIB
MIBs to compile: DOCS-IF3-MIB
Destination format: pysnmp
Custom destination template: None
Parser grammar cache directory: not used
Also compile all relevant MIBs: yes
Rebuild MIBs regardless of age: no
Dry run mode: no
Create/update MIBs: yes
Byte-compile Python modules: yes (optimization level no)
Ignore compilation errors: no
Generate OID->MIB index: no
Generate texts in MIBs: no
Keep original texts layout: no
Try various file names while searching for MIB module: yes
MIBs analyzed 22, MIBs failed 0
Created/updated MIBs:
Pre-compiled MIBs borrowed:
Up to date MIBs: INET-ADDRESS-MIB, RFC-1212, RFC1155-SMI, RFC1213-MIB, SNMP-FRAMEWORK-MIB, SNMP-TARGET-MIB, SNMPv2-CONF, SNMPv2-SMI, SNMPv2-TC
Missing source MIBs:
Ignored MIBs: CLAB-DEF-MIB, DIFFSERV-DSCP-TC, DIFFSERV-MIB, DOCS-CABLE-DEVICE-MIB, DOCS-IETF-BPI2-MIB, DOCS-IF-MIB, IANAifType-MIB, IF-MIB, INTEGRATED-SERVICES-MIB, RMON-MIB, RMON2-MIB, SNMPv2-MIB
Failed MIBs: DOCS-IF3-MIB (Jinja template rendering error: 'collections.OrderedDict object' has no attribute 'default' at MIB DOCS-IF3-MIB)

[seimnseimn]

Also SIP-COMMON-MIB fails for the same reason, however there SYNTAX is BITS which is definitely by the spec (https://www.rfc-editor.org/rfc/rfc2578.txt -> 7.9. Mapping of the DEFVAL clause)

SIP-COMMON-MIB:

sipCommonCfgServiceNotifEnable OBJECT-TYPE
    SYNTAX      BITS {
                  sipCommonServiceColdStart(0),
                  sipCommonServiceWarmStart(1),
                  sipCommonServiceStatusChanged(2)
                }
    MAX-ACCESS  read-write
    STATUS      current
    DESCRIPTION
       "This object specifies which SIP service related notifications
        are enabled.  Each bit represents a specific notification.  If
        a bit has a value 1, the associated notification is enabled and
        will be generated by the SIP entity at the appropriate time.

        Support for these notifications is OPTIONAL: either none or all
        notification values are supported.  If an implementation does
        not support this object, it should return a 'noSuchObject'
        exception to an SNMP GET operation.  If notifications are
        supported, this object's default value SHOULD reflect
        sipCommonServiceColdStart and sipCommonServiceWarmStart enabled
        and sipCommonServiceStatusChanged disabled.

        This object value SHOULD persist across reboots."
    DEFVAL { { sipCommonServiceColdStart,
               sipCommonServiceWarmStart } }
    ::= { sipCommonCfgEntry 7 }

reproduce:

mkdir snmp-test && cd snmp-test
pyenv local 3.12.4
pyenv virtualenv snmp-test
pyenv local snmp-test

pip install pysmi-lextudio

mkdir -p ./mibs && mkdir -p ./mib_cache

# get mibs from lextudio
mibs=(
    "SIP-COMMON-MIB"
    "SNMPv2-SMI"
    "SNMPv2-TC"
    "SNMPv2-CONF"
    "SNMP-FRAMEWORK-MIB"
    "SIP-TC-MIB"
    "NETWORK-SERVICES-MIB"
    "INET-ADDRESS-MIB"
)
url_prefix="https://raw.githubusercontent.com/lextudio/mibs.pysnmp.com/master/asn1/"
output_dir="./mibs"
for mib in "${mibs[@]}"; do
    wget "${url_prefix}${mib}" -O "${output_dir}/${mib}.mib"
done

# compile mibs
mibdump --mib-source ./mibs --destination-directory ./mib_cache SIP-COMMON-MIB

[lextm]

The parser does understand the DEFVAL definition properly (so JSON output is correct), but the template is incorrect so it cannot output proper Python.

[lextm]

Should have been fixed in PySMI release 1.5.0.