Expanding the Python modules to allow M8 files to be created by all tools

[davidjwbbc]

Introduction

As discussed briefly on the call a few weeks ago, we need to allow other Application Provider tools to recreate the m8.json file. Thinking about future changes we may also want to have a similar mechanism in place for DVB-I xml files to be published via M8. With this in mind I've been doing some thinking around a possible architecture for the Python modules to accomodate this.

Existing architecture

At present the only tool which outputs m8.json is the msaf-configuration app and the formatting and output code is held within the app code as it is directly derived from the app's streams.json configuration file.

C4Context
    title Application Provider Python Modules (current state)
    Person(webuser,"Web UI User")

    Container_Boundary(webui, "Web UI") {
        Container(webpages, "Web Pages")
        Container(webssc, "Server Side Components")
    }

    Container_Boundary(rtm1cli,"rt_m1_client") {
        Container(m1sess, "M1Session")
        Container(m1cli, "M1Client")
        Container(m1conf, "Configuration")
        Container_Boundary(m1certs, "certificates") {
            Container(m1certacme, "ACMECertificateSigner")
            Container(m1certlocal, "LocalCACertificateSigner")
        }
    }

    Container_Boundary(rtm1apps,"rt_m1_apps") {
        Container(appsess, "m1-session")
        Container(appcli, "m1-client")
        Container(appsync, "msaf-configuration")
    }

    Enterprise_Boundary(5gms, "5GMS") {
        Container(afunc, "Application Function")
        Container(aserv, "Application Server")
    }

    Boundary(a, "Local file system") {
        SystemDb(conf,"Configuration")
        SystemDb(mediaconf,"streams.json")
        Container_Boundary(dstore, "Data Store") {
            SystemDb(storpsid, "Provisioning Session IDs")
            SystemDb(storcerts, "Certificates Metadata")
        }
        Container_Boundary(docroot, "Web Docroot") {
            SystemDb(m8json,"m8.json")
        }
    }

    BiRel(m1certlocal, storcerts, "")
    BiRel(m1certacme, storcerts, "")
    BiRel(m1sess, m1certlocal, "")
    BiRel(m1sess, m1certacme, "")
    BiRel(m1sess, storpsid, "")
    BiRel(m1sess, m1cli, "")
    BiRel(m1cli, afunc, "")
    BiRel(m1conf, conf, "")
    BiRel(afunc, aserv, "")
    BiRel(appsess, m1sess, "")
    BiRel(appsess, m1conf, "")
    BiRel(appcli, m1cli, "")
    BiRel(appsync, m1sess, "")
    BiRel(appsync, m1conf, "")
    BiRel(appsync, m8json, "")
    BiRel(appsync, mediaconf, "")
    BiRel(webssc, m1sess, "")
    BiRel(webssc, m1conf, "")
    BiRel(webuser, webpages, "")
    BiRel(webuser, webssc, "")

Loading

Component	Notes
rt_m1_client	Python module which provides classes to manage the 5GMS Application Function via the interface at M1.
rt_m1_client.client.M1Client	Simple implementation of M1 API as specified in TS 26.512.
rt_m1_client.session.M1Session	Class to handle a communication session with the 5GMS Application Function that uses M1Client for its underlying communications. This will cache data read from the 5GMS Application Function based on Cache-Control headers to avoid unnecessary communications. Provisioning Sessions created via this class are remembered by their IDs in the persistent data store so that subsequent access knows about previously created provisioning sessions. Provision Session IDs are purged from the data store if the 5GMS Application Function reports Not Found when a provisioning session is requested.
rt_m1_client.configuration.Configuration	Class to handle the reading and modification of the persistent application configuration for applications using M1Session to access the 5GMS Application Function
rt_m1_apps.m1_client	Command line application to execute individual M1 API calls.
rt_m1_apps.m1_session	Command line application to manage the 5GMS Application Function via the interface at M1 using the M1Session class. Application configuration is obtained via the Configuration class.
rt_m1_apps.msaf_configuration	Command line application which reads in a streams.json configuration which describes the preferred state of the 5GMS Application Function and uses the M1Session class to interrogate the current state and modify it to match the configuration in streams.json. Communication preferences are shared with the m1-session app via the Configuration class. This was initially intended as an example of an Application Provider using the M1Session class to set configuration and developed to reconfigure the 5G-MAG linode VM on reboot.

Proposed architecture

In order to enable the other components, m1-session and the server side components of the Web UI, to also manage the data presented via M8 we need to do two things. The output formatting for m8.json needs teasing out from msaf-configuration and placing into a reusable Python module so that multiple components can all use it as necessary. Also the model of media assets needs teasing out from msaf-configuration, into its own Python module, as this will be needed to feed the output formatting. Please note that the media assets model in msaf-configuration goes beyond the simple model in the 5GMS AF (and therefore the model handled by the rt_m1_client Python module) by understanding media assets which share a single provisioned root media node. While we are doing this it seems a good opportunity to make these new modules a little more generic and expandable, especially since there is ambition to provide a DVB-I M8 service rather than (or as well as) our proprietary m8.,json format file.

So I would propose that we create two new Python modules based on the current msaf-configuration tool. One for modelling media assets that can then use the rt_m1_client.M1Session class to reflect that model on 5GMS AF/AS and also call a second Python module which contains M8 output formatters. We can then use a local configuration to determine which output formatters should be used, possibly making the default the m8.json formatter and adding the DVB-I formatter once it has been developed.

It would also be a good opportunity to introduce a new Python module for the proprietary Ref-tools 5GMS AF "MAF-Management" API interface, which would allow the Web UI access to the full list of configured provisioning session as known by the AF.

My proposed new architecture would look like this:

C4Context
    title Proposed Application Provider architecture
    Person(webuser,"Web UI User")

    Container_Boundary(webui, "Web UI") {
        Container(webpages, "Web Pages")
        Container(webssc, "Server Side Components")
    }

    Container_Boundary(rtmafmgmt, "rt_maf_management")  {
        Container(mafmgmt, "MAFManagement")
    }

    Container_Boundary(rtm1cli,"rt_m1_client") {
        Container(m1sess, "M1Session")
        Container(m1cli, "M1Client")
        Container(m1conf, "Configuration")
        Container_Boundary(m1certs, "certificates") {
            Container(m1certacme, "ACMECertificateSigner")
            Container(m1certlocal, "LocalCACertificateSigner")
        }
    }

    Container_Boundary(rtm1apps,"rt_m1_apps") {
        Container(appsess, "m1-session")
        Container(appcli, "m1-client")
        Container(appsync, "msaf-configuration")
    }

    Enterprise_Boundary(5gms, "5GMS") {
        Container(afunc, "Application Function")
        Container(aserv, "Application Server")
    }

    Container_Boundary(rtmediaconf, "rt_media_configuration") {
        Container(mediaconf, "MediaConfiguration")
    }

    Boundary(a, "Local file system") {
        SystemDb(conf,"Configuration")
        SystemDb(streamsconf,"streams.json")
        Container_Boundary(dstore, "Data Store") {
            SystemDb(storpsid, "Provisioning Session IDs")
            SystemDb(storcerts, "Certificates Metadata")
        }
        Container_Boundary(docroot, "Web Docroot") {
            SystemDb(m8json,"m8.json")
            SystemDb(dvbifile,"DVB-I XML")
        }
    }

    Container_Boundary(rtm8output,"rt_m8_output") {
        Container(m8output, "M8Output")
        Container(m8jsonformatter, "M8JSONFormatter")
        Container(dvbiformatter, "DVBIFormatter")
    }

    Rel(m8output, m8jsonformatter, "")
    Rel(m8output, dvbiformatter, "")
    BiRel(m1certlocal, storcerts, "")
    BiRel(m1certacme, storcerts, "")
    BiRel(m1sess, m1certlocal, "")
    BiRel(m1sess, m1certacme, "")
    BiRel(m1sess, storpsid, "")
    BiRel(m1sess, m1cli, "")
    BiRel(m1cli, afunc, "")
    BiRel(m1conf, conf, "")
    BiRel(m8jsonformatter, m8json, "")
    Rel(dvbiformatter, dvbifile, "")
    Rel(mediaconf, m8output, "")
    Rel(streamsconf, mediaconf, "")
    BiRel(mediaconf, m1sess, "")
    BiRel(mediaconf, m1conf, "")
    BiRel(afunc, aserv, "")
    BiRel(appsess, m1sess, "")
    BiRel(appsess, m1conf, "")
    BiRel(appsess, mediaconf, "")
    BiRel(appcli, m1cli, "")
    BiRel(appsync, mediaconf, "")
    BiRel(webssc, m1sess, "")
    BiRel(webssc, m1conf, "")
    BiRel(webssc, mafmgmt, "")
    BiRel(webssc, mediaconf, "")
    BiRel(webuser, webpages, "")
    BiRel(webuser, webssc, "")

Loading

Component	Notes
rt_m1_client	Python module which provides classes to manage the 5GMS Application Function via the interface at M1.
rt_m1_client.client.M1Client	Simple implementation of M1 API as specified in TS 26.512.
rt_m1_client.session.M1Session	Class to handle a communication session with the 5GMS Application Function that uses M1Client for its underlying communications. This will cache data read from the 5GMS Application Function based on Cache-Control headers to avoid unnecessary communications. Provisioning Sessions created via this class are remembered by their IDs in the persistent data store so that subsequent access knows about previously created provisioning sessions. Provision Session IDs are purged from the data store if the 5GMS Application Function reports Not Found when a provisioning session is requested.
rt_m1_client.configuration.Configuration	Class to handle the reading and modification of the persistent application configuration for applications using M1Session to access the 5GMS Application Function
rt_media_configuration	Python module which models a set of media assets.
rt_media_configuration.MediaConfiguration	Python class to store the media assets model and understands how to use rt_m1_client.M1Session to read the current 5GMS AF configuration and push changes to that configuration to reflect the modelled media assets. It can also use rt_m8_output.M8Output to write model changes as M8 published data files.
rt_m8_output	Python module to publish M8 data files.
rt_m8_output.M8Output	Python class to write out M8 data files according to an M8 publication configuration. It will use formatter classes to perform file outputs in different formats.
rt_m8_output.M8JSONFormatter	A Python class which implements the formatter for the 5G-MAG m8.json format file.
rt_m8_output.DVBIFormatter	(future development) A Python class which implements a formatter for a DVB-I XML format M8 file.
rt_maf_management	Python module to implement the proprietary 5G-MAG Reftools 5GMS AF MAF Management API.
rt_maf_management.MAFManagement	Python class to implement a MAF Management API client.
rt_m1_apps.m1_client	Command line application to execute individual M1 API calls.
rt_m1_apps.m1_session	Command line application to manage the 5GMS Application Function via the interface at M1 using the M1Session class. Application configuration is obtained via the Configuration class.
rt_m1_apps.msaf_configuration	Command line application which reads in a streams.json configuration which describes the preferred state of the 5GMS Application Function and uses the MediaConfiguration class to synchronise this configuration with the 5GMS AF.

[davidjwbbc]

These are my initial thoughts on how to re-architect the Python modules in the Application Provider so that multiple tools can manage the M8 published files and also provide ways to expand this to formats like DVB-I service lists.

It may also be possible to split the work for this into multiple work items so that multiple people can work on this at once.

The diagrams above are my first attempt at using the C4 diagramming language so are likely to need improvement, feel free to edit. I've been using the Mermaid live editor to prototype the diagrams, but this has limits on the C4 functions that can be used.

Please provide any feedback on my proposed design.

[stojkovicv]

Hi David I wouldn't split Python module in rt_m1_client and rt_m1_apps. I suggest to merge those, since the rt_m1_apps are essentially executable wrappers around the main classes. This will simplify some internal connections. To avoid arrows overlapping, which makes reading the diagram very difficult, I just reordered modules and merged executable tool with the rest of Python classes in the following (draft) diagram:

[davidjwbbc]

rt_m1_client is the set of modules that implement the M1 API and is deliberately kept distinct from the tools that use them.

rt_m1_apps is the implementation of the Python command line tools which use rt_m1_client classes for the M1 communication, but these apps also include command line parsing and output formatting which is not part of the M1 communication.

rt_m1_client is reusable by other Python based applications such as your Web UI, rt_m1_apps is the non-reusable parts that are specific to those tools.

It is therefore a bad idea to merge these two.

[davidjwbbc]

Also note that the diagrams in the main description above are automatically generated from C4 UML descriptions, so there's not much control over placement. The only control seems to be by the order in which the elements of the diagram are declared, which can cause the layout engine to rearrange items. I tried to get them in an order that avoided a lot of crossing over of lines, but it's difficult to avoid without doing what you've done and using a tool to manually place things and just post the image file of the result. The problem there is no one else can then easily edit the diagram.

[stojkovicv]

All right then.
In case that we are willing to actually use this reorered diagram, I have saved draft file for simpler online diagram editing tool (https://app.diagrams.net/), and I'll be happy to update this version. If you're willing to to edit it by yourself, just let me know.

[stojkovicv]

Creation of m8.json requires Configured Provisioning Session data with media assests and entry points. AF MAF Management API is there to fetch the list of (configured) provisioning sessions, right? However, it will retrieve also the provisioning session IDs without any configuration or any other feature enabled? So the only data that can be used in this case is provisioning session ID? From the diagram, there is a direct interaction between UI Server Side Component and Media Configuration, I am not sure about the purpose of MAFManagement API...?

[davidjwbbc]

MAF Management is a debug/admin entry point to gain a list of provisioned sessions regardless of which tools did the provisioning. It could be used to regain access to provisioning sessions that have been forgotten or to provide an admin interface which can manage Provisioning Sessions regardless of who the original Application Provider was. Access to this was requested in a previous 5G-MAG meeting as a way of getting to Provisioning Sessions that had been created via other tools like Postman.

I would suggest a simple menu option to either rebuild the Provisioning Session list from the MAF Management API, which can then be used when the Web UI is out of sync with the AF, or a option check box to include other Provisioning Sessions on the Web UI interface, possibly with some sort of marker to indicate they were generated externally. But I would suggest that it is not just used automatically as we may wish to use multiple AP interfaces with their own sets of Provisioning Sessions and want to keep that separation.

[davidjwbbc]

@stojkovicv and I have had a discussion via a Zoom meeting to talk through this proposed design and we're both fairly happy that this will accomplish what we want.

A proposal was put forward to split this design down into 3 implementation steps:

1. Separate out the Media Configuration and resynchronisation logic from msaf-configuration into the new MediaConfigure class and separate out the m8.json output into the new M8JSONFormatter class and rework msaf-configuration to use these new classes.
2. Rework m1-session to manage ProvisioningSession information via the new MediaConfiguration class so that the tool will cause rewriting of M8 data files.
3. Rework the Web UI and backend server to use the new MediaConfiguration object for provisioning session manipulation.

Item 1 must be done first, but the other two can then be done in parallel. To this end I will raise 3 new issues to separate the work into these three areas.

[dsilhavy]

@davidjwbbc @stojkovicv Thank you for driving this forward, this looks excellent.

I agreed with @stojkovicv that the rt-5gms-application-provider.management-ui should synchronize itself against the AF. Otherwise, different users create provisioning sessions from different machines, the data is saved in local storage and not visible to anyone except the user who created the session.

Some additional comments / discussion items:

• I agree with @davidjwbbc that we should split rt_m1_client and rt_m1_apps. rt_m1_apps are one way to use the Python modules in rt_m1_client. Another way to use them is the web ui. There might be additional implementations/applications in the future that use the rt_m1_client Python modules.
• From my understanding, the web-ui uses the local-storage of the browser to save provisioning sessions that have been created. I assume this is covered by data store in the description/illustration above
• rt_maf_management.MAFManagement does not seem to be connected to the AF. From my understanding, we use this class
  to for instance fetch all provisioning sessions via our "custom" interface: {{maf_url}}/5gmag-rt-management/v1/provisioning-sessions. @stojkovicv This is the module/class we need to implement the functionality described above then
• I like the introduction of a rt_m8_output module. M8Output contains the generic functionality to write an M8 output while the logic to create specific formats like our current JSON format or the DVB-I service list are handled in specific classes. It would be nice if M8Output allows the configuration of multiple output formats at the same time, e.g. writing a DVB-I service XML and a JSON output.

@davidjwbbc Do you want to create the Github issues as mentioned in your last post? Otherwise I can take care of it and you can add missing information.

[davidjwbbc]

@davidjwbbc Do you want to create the Github issues as mentioned in your last post? Otherwise I can take care of it and you can add missing information.

Created project https://github.com/orgs/5G-MAG/projects/40/views/1 to track progress of these tickets and started creating draft issues for these 3 steps. I'll flesh these out when I get a moment.