CDM - Serialization Target State

[dschwartznyc]

Soliciting feedback, suggestions, and challenge, if any, on a proposal put forward in the Technology Architecture WG (TAWG) to address how CDM will be serialized going forward. The proposal was made in the 8th June 2023 TAWG meeting by the Task Force examining CDM's Build and Release Management. Please comment below and join the Task Force.

The proposal:

• Use the CDM model to directly define what is serialized.
• Include a reference to the entity (i.e., TradeEvent) and which version is used (i.e., 4.0.0)
• Ensure that there is sufficient information to allow users and/or developers to translate between supported versions

In principle, the proposal had consensus support but needs to be fleshed out and turned into action.

There are two central questions:

1. What should be CDM’s serialization format?
2. How should serialization support Model versioning?

Why is this an issue?

• Exchange of information and the mutual common understanding of what’s exchanged are central to meeting CDM’s goal for interoperability and collaboration
• There will be different versions of the model and libraries in use at any given time. New (major) versions may change Entity definitions

Open questions:

• Do we continue to use JSON or select a different format for serialization?
  -- JSON is a lightweight standard
  -- Alternatives such as XML more readily accommodate the inclusion of metadata such as reference to an entity or version
• What level of model granularity does the version represent? Do we version the model itself, each of the entities of the model, or something else?
  -- How do we balance the flexibility of innovation versus the complexity of development and maintenance?
  -- More granularity implies greater flexibility
  -- Less granularity implies less complexity
• Do we create a systematic means for translation between versions?
  -- Would the translation be direct between versions or against a canonical model representation such as a version of FPML?
• How will the change impact the build process?

[chrisisla]

A few thoughts from my side on this...

I would version against the model and not down to the entity level. I think trying to go too granular will make life very very difficult. Also I don't think we'll ever be in a position where we'd want to deliver pieces of the model separately e.g. deliver a new version of party for example, these will always be tied to a model version/release, so we only need to version at that level.

My suggestion would also be to keep using JSON as the serialization method for the CDM. It would be too large an undertaking to change it now. Having said that though, JSON was really designed for use with web applications so are we using it correctly with the CDM? It fits for where we're using API calls (so like calling an event function like Create_BusinessEvent) but does it really fit for where we're describing a trade object?

So I guess for API use cases JSON works well - so like as input to a function or receiving results back from a function. For messaging it also kind of works, although generally these are expressed as XML. As data dictionaries, again, XML is used predominantly. This infers that we need to consider what the actual use cases of the CDM are if we really want to determine what the native serialization format for the CDM should be. This type of decision is more something people on the CDM Steering Working Group should get involved in I feel?

One other point on versioning which you mention - conversion/translation between versions. This is an interesting comment. Lets assume that we start using the CDM on version 4.0.0 and we store the JSON created by the CDM from that version. Time moves on and we're now on version 10.0.0 and the JSON has changed considerably. If I have been storing the CDM JSON from version 4.0.0, should I expect backwards compatibility between versions 10.0.0 and 4.0.0? Or will scripts/documentation/functions be provided to facilitate the conversion of one version to another? In this scenario is JSON still our preferred format?

Sorry Dan, more questions than answers here I think 😂

[chrisisla]

@iansloyan, @davidshone, @gabriel-ICMA

Off the back of these technical discussions I think we need to think about what the target use cases for the CDM actually are. Once we have these agreed then we can determine the best way to support them from a technical perspective. I think the best forum for this would be the CDM Steering WG which I believe is due to be meeting soon. Could we add this as an agenda item?

To get the ball rolling the use cases we (ISLA) have been promoting are:

CDM as a common processing engine - if all parties to a trade call the same functions in the CDM then you should get the same result and reduce the number of fails and the need for reconciliation. This would be where the CDM is accessed via API calls where JSON would be a good choice for data format.

CDM as a negotiation/messaging standard - with the work we've been doing on pre-trade there is a need to transfer data from one party to another. This is where the CDM data would be used as a messaging standard - JSON will work for this but in general XML has historically been used for this use case.

CDM as a data dictionary - we are always promoting the ability the CDM has to provide a common representation of a financial object. This is not a particularly good use case of JSON, and again is more in line with how XML is used.

Any other use cases I have missed?

[brianlynn2]

Requirements

I think that a question the SWG needs to answer is whether we want separate implementations of CDM (e.g. at different firms) to be able to communicate using JSON serialized objects (API calls or messages), or do we require firms to always first translate to XML (e.g. FpML or other standard) before communicating to another firm.

Another related question we want to answer is whether we want JSON-serialized objects saved in an older version of CDM to be retrievable by newer versions of CDM.

If we want to allow JSON to be used for communication or persistence like this (and I think we should, given the ubiquity of JSON), we need to define how compatibility will work between separate implementations, because we know that they won't always be upgraded simultaneously. By compatibility I will define the following terms (I'm open to suggestion on these names):

• backward-processing compatibility means :
  a newer version of a CDM implementation can correctly deserialize JSON produced by an older version and turn it into objects usable with the current CDM.
• backward-generation compatibility means:
  a newer version of a CDM implementation can correctly serialize CDM objects to an older format/version of the CDM JSON
• forward-processing compatibility means:
  an older version a a CDM implementation can correctly deserialize JSON produced by a newer version of CDM, as long as the relevant content hasn't changed.

We don't necessarily need all of the above. For instance, we could mandate that if two CDM implementations want to interoperate, they need to fall back to the oldest supported JSON format, for communication in both directions. That would mean that forward-processing compatibility isn't required. But it may be helpful to have a limited amount of forward processing compatibility, e.g. by ignoring or flagging new content that doesn't match expectations.

Now, assuming we want any of these things, we'll need to define some kind of domain of support for the compatibility. In other words, how far back (how many versions back) do we guarantee compatibility? And do we want to guarantee that by making test cases that demonstrate it? These are things that the SWG should decide on, based on feedback from the TAWG on feasibility.

Deciding on the above will be hard, and will be based in part on technical feasibility. I suggest that we have the SWG discuss this, give some kind of steer on direction/requirements, and then task the TAWG with evaluating implementation options and their feasibility, and reporting back to the SWG with a recommended direction.

Implementation

Assuming that we want to support any of the above requirements, we'll need a strategy for doing that. That's something the TAWG should discuss.

First point is that we need a version numbering system for the model that has been serialized into JSON, and always include that in any generated JSON. I think for simplicity it should just be based on the CDM model version, a single number. Then users of the JSON can determine whether than model version supports the versions of each type they need / have.

Second is that we need some kind of standardized wrapper for the JSON to contain the version number and an entry point (and possibly other things). Unless something has changed recently, serialized JSON doesn't by default contain the name of the root type, just the contents, so you have to know what you're loading. We should have a standardized way of wrapping all that, so that implementations can just load a lump of CDM JSON and get a useful CDM object, without having to know what it is before loading it.

Having this standardized versioning will give us a starting point for interoperating with JSON. (Note that we have something similar with FpML and it is crucial to interoperating across implementations.)

Then, if we want to be able to provide backward-processing compatibility, some options we could consider (roughly from crudest to most sophisticated) include

• develop custom Java deserialization code in JSON deserializers (e.g. FasterXML Jackson) to map from the old JSON format to the new objects. Note that this probably only works for Java implementations. You might need to do something similar for other languages like Python or C#.
• develop some kind of translator module that includes different versions of the CDM jar files and has code to map from the old to the new.
• develop some kind of Rosetta syntax that allows representing multiple versions of a type (e.g QuantityPrice_v2, QuantityPrice_v3) and a mapping between the old and the new version (similar to the XML ingestion synonym mappings). This could potentially allow the JSON to be deserialized into the old model and then mapped automatically to the new model. It could be language independent if the mapping synonyms could be made to generate translation code, similar to XML ingestion.

I assume that the implementation options for backward-generation would be similar.

For forward-processing, it may be enough to find ways to keep the deserializer from barfing if the JSON code doesn't exactly match the current model, and to track any resulting exceptions.

Commentary

Developing this kind of mapping between versions of the model seems like it could be a time-consuming effort. However, the alternative is to strongly limit the changes that are allowed to the model, and force everybody to upgrade whenever changes are allowed. In the case of FpML, this made sense, particularly once the core products were stable. (In practice the FpML IR Swaps model hasn't changed much in nearly 20 years, except for some additions.)

In the case of CDM, I think that given the rapid evolution of the model that is likely to continue for a while, and the fact that there is a shared code base for working with the model, putting the translation between versions of a model somehow into the shared code base is worth the effort. In practice most changes to the model are likely to to be relatively easily mappable, and doing that once in the shared code base instead of for every implementation makes a lot more sense.

[dschwartznyc]

Of the types of use cases @chrisisla cites above, CDM is currently most frequently used as a data dictionary. There is no doubt that a shared standard industry lexicon is a necessary condition for increasing efficiency. It is insufficient without the exchange of data whether by messaging or API both of which serialization will need to support as the use of CDM expands.

Secondly, the simultaneous in-production use of different versions of CDM mentioned by @brianlynn2 seems inevitable, especially given the emerging usage of the standard. The stated goal of interoperability and collaboration demands that users of supported (I.E., non-deprecated major) versions mutually understand the data they exchange at least to the minimum extent available between the versions used by each party. The implications are that the model and version must be part of the serialized data and that later versions of the model must include backward processing and generation. The good news is that the CDM's well-defined modeling and tightly coupled generated code may provide the basis for enabling the understanding of supported earlier versions.

The need to provide the context of the model and version reasonably opens the question of whether to continue to use JSON as a serialization format, shift to something more self-described such as XML, or simultaneously support both. JSON historically has been used as the input or output of a process where the interaction with an API implies the model and version. As a result, some hold that including the model and version in the data itself is inconsistent with JSON's "ethos" and, therefore, more suited to a more verbose format such as XML. The argument against using JSON is understandable but a bit impractically purest in nature and the reasoning for using XML comes with its own suitability questions since it would require synchronization between the schema and CDM. Said another way, since Rosetta is used to define CDM, tight binding to XML implies that the schema should be the result of a new code generation process.

Separately, the discussion of version management and lessons learned from FPML led to a question about the best way to include product definition data which might rapidly evolve. When it is directly embedded, every change implies the need to create a new version of the model. The recommendation is to make reference to data external to the model itself.

[dschwartznyc]

Recapping the agenda for the first meeting held to discuss how to move this forward:

Overall mission: Define and realize a serialization format for CDM. Adoption of the to-be-proposed format may be one of the most critical issues in the adoption of the new standard.

Meeting Goal: Define and develop a plan to realize CDM's target state serialization format.

Agenda:

1. Process / Governance

1. This group will define supported use cases
2. This group will define a target state to enable the use cases
3. The proposal will include a request for resources
4. The recommendation will be presented for ratification by the Steering WG and the Trade Associations

2. Assumptions

1. CDM should have a defined format for data exchange
2. Supported data exchange use cases should include "out of process" activity such as messaging and/or "in process" communications such as reporting.
3. There needs to be interoperability between applications that provide support for non-deprecated versions of the model.
4. The solution should minimize the upgrade requirements for applications that support in production versions of the models. I.E., other than for true bug fixes, upgrades should not be required until the model version in use is deprecated.
5. There should be support for the inclusion of externally sourced reference data.

3. Functional considerations (see Brian's comments in the discussion above)

1. To what extent should we support backward-processing compatibility?
2. To what extent should we support backward-generation compatibility?
3. To what extent should we support forward processing?

4. Technical considerations

1. The format should include which entity and which model version was used when creating the data
2. Does serialization have a conical format (JSON, XML, FPML, Protobuf)? A combination?

5. Discuss the next steps

1. Documentation of proposed supported use cases
2. Investigation into the proposed approach

[dschwartznyc]

Recap of Kick-off meeting held: Aug 14

In attendance: @brianlynn2, @chrisisla, @eacunaISDA, @iansloyan, @manel-martos, @minesh-s-patel, @dschwartznyc
Apologies: Adrian Hutusoru, @mgratacos

Agreed:

• Focus on asynchronous / messaging as a proxy for all the use cases. Test the proposed solution on “in-process” / API-driven use cases to confirm its viability.
• Areas for drill down:

What do we mean by interoperability?
Definition/clarification of supported use cases
How and to what extent will there be support for interoperability across versions of Rosetta-defined functionality?
Technical considerations including solutions that may already be under development.

• Out of scope for serialization:

Versioning questions even though the two have a strong connection.
Support for customized models.
Modeling support for external reference data (2.5 from the agenda).

• Use GitHub discussions to facilitate the community’s open-source awareness and engagement

Note the suggestion that we refrain from long multi-topic entries. The recommendation is separate complex issues into several “short and sweet” postings.

Next steps

• Confirm support from CDM’s sponsors (ICMA, ISDA, ISLA) for the project to define and realize expectations for serialization.
• Develop a short presentation to outline the problem in practical, non-technical terms – Dan to organize with input from the group.
• Request and schedule an ad hoc Steering WG meeting -Ian.
• Schedule a follow-up meeting to begin the drill-down discussions – Dan.
• Create a template for the use cases – Minesh.

[minesh-s-patel]

@dschwartznyc @brianlynn2, @chrisisla, @eacunaISDA, @iansloyan, @manel-martos, @dschwartznyc
@mgratacos

I put some simplified use-cases into a PowerPoint as discussed. Take a look can we can discuss.
CDMSerialisationUseCases.pptx

[dschwartznyc]

Output from brainstorming session discussing functional/technical challenges to consider in designing a solution for serialization

[JBZ-Fragmos]

please make sure Fragmos-Chain CTO, namely Adrian Hutusoru is definitly involved in business rationale, use cases, validation process and any potential release, related to this "Serialisation" project

thanks

[dschwartznyc]

Reiterating that Adrian is already an active participant in the discussion.

[JBZ-Fragmos]

hi Dan
yes, indeed, thank you
regards

[dschwartznyc]

Update from Sep 1 through Oct 3
Presentation to the SWG: "A call to arms"
FINOS - CDM Serialization Call to Arms - Final.pdf
From meeting #6
FINOS - Serialization #6.pdf
Selection Framework
Selection Framework.xlsx

[dschwartznyc]

Potential Solutions reviewed 10/3

Objective: determine the recommended approach for serialization of CDM or present key alternatives w/pros and cons

Meeting Agenda: brainstorm/outline potential solutions and alternatives. The intent is to list out materially different approaches. The following is intended as a starter and not comprehensive by any means.

1. Two-part solution: leverages generated code for production and consumption of serialized data and offers a transformation framework to deal with conversions.
   1.1 The first is part of the generated code.
   1.1.i. It expects and produces JSON (must include type and version)
   1.1.ii. It produces data that matches the CDM definition used in creating the code.
   1.1.iii. It will consume serialized data that matches its definition and will do its best if the data does not directly conform. It has no mechanism for transformations.
   1.2 The second part is a framework that can support conversions.
   1.2.i. The framework ingests data and produces an instance in the targeted version (to the extent possible)
   1.2.ii. The framework takes an instance and serializes it to a targeted format
   1.2.iii. Specific transformations are “extensions” of the framework
   1.2.iv From or to formats other than JSON (XML) to JSON
   1.2.v From or to different versions to the extent possible
   1.3 Open questions (of the many)
   1.3.i. Does the core library rely upon a format other than JSON?
   1.3.ii. Does the solution support compression of a large data object?
   1.3.iii. Does the solution support collections whether homogeneous or heterogeneous?
   1.3.iv. Is the code generation process the vehicle to support proprietary implementations?
   1.4 Evaluation parking lot
   1.4.i. The solution leverages a logical partition with serialized data closely aligned to the model
   1.4.ii. Generated code is lightweight and stable by version. There is no need to make a change as long as the version in use is still supported (not deprecated) and its features match the requirements
   1.4.iii. The transformation process is extensible with a mechanism to support more input and output types
   1.4.iv. Reliance on generated code may make support for proprietary implementations compute-intensive (need to transform to and from)
2. Integrated approach: each version of the generated code includes transformations to the extent possible
   a. Code generated to support new versions of CDM can ingest and produce earlier versions
   b. The code generation process can also update code generated for earlier versions to support new versions
   c. Open questions (of the many)
   2.c.i. Is data serialized in human-readable form (JSON, XML)?
   2.c.ii. Is data serialized in a binary format?
   2.c.iii. How are formats that are not part of the baseline supported?
   2.d Evaluation parking lot
   2.d.i. The solution embeds transformations but …
   2.d.ii. The core solution is more heavyweight, very dynamic, and may require frequent upgrades
   2.d.iii. Serialized data may not be closely aligned with the logical model
   2.d.iv. This approach can more easily embed compression

[dschwartznyc]

Recap from Serialization Taskforce meeting on Oct 5, 0223

Attending: Adrian, Chris, Dhruva, Eleonora, Jason, Manuel, Minesh, Tom, Dan
Apologies: Brian, Marc, David
Unknown: Harsha, Raja

Thank you again to Dhruva for walking us through JPMorgan’s approach to exchanging information including the reasoning behind its decision to partition its reading and writing libraries.

Decisions:

• Subject to further validation, JSON will be the serialization format.
• There will be two parts to the serialization implementation.
• The generated code will have the ability to produce a data stream that matches the model and version that produced it and will be able to read streams provided to it to the extent possible. To achieve minimal interoperability this will be the first targeted release.
• There will be a separate TBD library that will provide transformations (e.g., from version to version and data type to JSON).

Open questions:

• Should serialization leverage additional technology whether to support compression or other concerns? JSON-LD, AVRO, …., et al.
• Should serialization directly support the exchange of collections of entities (homogenous or heterogeneous)?
• Is a POC necessary prior to making a recommendation?

The objective for the next meeting (the weekly on Thursday), is to sort out the Open Questions listed above.

Issues identified but outside of the scope for serialization (directed to the TAWG):

• Is there need for CDM to include some mode of signing (securing) data streams? An “object” hash? Encryption?
• Should CDM consider partitioning the model to limit the impact of updates? IE to facilitate selective upgrades aligned to usage.
• Should the model have a default option (“UNKNOWN”) for ENUMS?

Actions:

• Provide challenges, if any, to the decisions by EOD Monday: All
• Update to the SWG (Tuesday): Dan

[dschwartznyc]

Subject: Serialization #11 - agenda - 11/9
Apologies for the cancelation of the last two meetings
Meeting #11 - Agenda

1. Review the compression analysis below.
2. Review the recommendation deck (attached) with a focus on estimating the effort.
3. AOB

Java results:

• Methodology: uses Jackson object mapper to do the job to be close to the actual target implementation and a normal gzip compression on the JSON itself.

10k runs:
------- Size Comparison -------
JSON Size: 10928 Kbytes
BSON Size: 4640 Kbytes (size reduction of 58%)
Compressed JSON Size: 111 Kbytes (size reduction of 99%)
==========================
-------  Read Time Comparison (ms) -------
Avg JSON Read Time: 27 ms
AVG BSON Read Time: 49 ms (read time increase by 81%)
AVG Compressed JSON Read Time: 40 ms (read time increase by 48%)
[7:27](https://isda-digital-cdm.slack.com/archives/D05MF56R70S/p1699489679735319)
some more file sizes:------- Size Comparison -------
JSON Size: 49 Kbytes
BSON Size: 16 Kbytes (size reduction of 67%)
Compressed JSON Size: 4 Kbytes (size reduction of 91%)
==========================
-------  Read Time Comparison (ms) -------
Avg JSON Read Time: 1 ms
AVG BSON Read Time: 1 ms (read time increase by 0%)
AVG Compressed JSON Read Time: 1 ms (read time increase by 0%)
[7:28](https://isda-digital-cdm.slack.com/archives/D05MF56R70S/p1699489704015919)
------- Size Comparison -------
JSON Size: 383 Kbytes
BSON Size: 107 Kbytes (size reduction of 73%)
Compressed JSON Size: 14 Kbytes (size reduction of 97%)
==========================
-------  Read Time Comparison (ms) -------
Avg JSON Read Time: 2 ms
AVG BSON Read Time: 3 ms (read time increase by 50%)
AVG Compressed JSON Read Time: 4 ms (read time increase by 100%)

Python compression comparison analysis

• Methodology – read JSON or compressed file into a dict (hash table). Compress when necessary and write to file.

 
test run 1000 times 
+-------------+------------+-----------+---------+--------+
|             |    json    |    bson   |   lz4   |  zlib  |
+-------------+------------+-----------+---------+--------+
|  file size  | 10,927,572 | 4,778,368 |  67,111 | 30,302 |
| compression |            |   -56.3%  | -99.4 % | -99.7% |
+-------------+------------+-----------+---------+--------+
+--------------+---------+---------+---------+---------+
|     read     |   json  |   bson  |   lz4   |   zlib  |
+--------------+---------+---------+---------+---------+
|   avg read   | 0.08946 | 0.09019 | 0.08508 | 0.08431 |
|  delta read  |         |   0.8%  |  -4.9%  |  -5.8%  |
| std dev read | 0.06047 | 0.06464 | 0.06045 | 0.05987 |
|   min read   | 0.02814 | 0.02498 | 0.02415 | 0.02366 |
|   max read   | 0.16402 | 0.17423 | 0.16387 | 0.16453 |
+--------------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+
|     write     |   json  |   bson  |   lz4   |   zlib  |
+---------------+---------+---------+---------+---------+
|   avg write   | 0.03388 | 0.01518 | 0.03366 | 0.05113 |
|  delta write  |         |  -55.2% |  -0.6%  |  50.9%  |
| std dev write | 0.00059 | 0.00075 | 0.00110 | 0.00067 |
|   min write   | 0.03328 | 0.01448 | 0.03223 | 0.04997 |
|   max write   | 0.04268 | 0.02649 | 0.03856 | 0.05486 |
+---------------+---------+---------+---------+---------+

[dschwartznyc]

Serialization #11 - recap - 11/9
Attendees: Adrian, @brianlynn2, @chrisisla, @dschwartznyc, Dhruva, Diana, @eacunaISDA, Eric, Jason, @manel-martos, @minesh-s-patel, @tomhealey-icma

Apologies: David, Marc

Comments and corrections are welcome.

1. Compression

• Agreed that the inclusion of a compression methodology is out of scope. Applications that use CDM can compress from outside their usage.
• The recommendation is that the test harnesses and findings be shared.
  Python test harness

2. Effort Estimate (t-shirt sizing): 4 person months

• The transformation between versions is out of scope.
• Java dev, 1 Python dev, each for 6 weeks. PM – 2 weeks total effort, 2 weeks testing
• Assumes that the implementation will identify the following when reading:
  -- Faulty JSON vs well-formed JSON
  -- Whether the referenced version is supported in the library
  -- Whether the fully qualified referenced entity is supported in the library
  -- Whether the required fields are available in the expected parts of the data stream. Extra fields will be ignored.
  -- Whether the validation rules pass or not
• If sufficient resources are dedicated at the very beginning of Jan, delivery can happen by the end of Feb.

3. Revised recommendations to be made at the next SWG meeting. Updated deck to be posted in the GitHub issue.
   FINOS - CDM SWG - Serialization Proposal - 20231114.pdf