Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow defining payloads with non-JSON standards (XSD, Protobuf, etc.) #881

Closed
3 tasks
jonaslagoni opened this issue Dec 8, 2022 · 11 comments
Closed
3 tasks
Labels
💡 Proposal (RFC 1) RFC Stage 1 (See CONTRIBUTING.md) stale

Comments

@jonaslagoni
Copy link
Member

jonaslagoni commented Dec 8, 2022

I choose to start a new issue to map out more precisely what is needed to support this, without any specific sight on one approach or the other. This is a generic solution that would work with all non-JSON standards regardless of which one.

This was first pursued for V3 of the spec, but I don't believe that it will be possible for that deadline. Therefore I gathered the current progress to have one place to discuss it so it can be removed from the progress list.

Steps currently being taken:

Related resources:

@jonaslagoni jonaslagoni added the 💡 Proposal (RFC 1) RFC Stage 1 (See CONTRIBUTING.md) label Dec 8, 2022
@derberg
Copy link
Member

derberg commented Dec 19, 2022

I looked into the linked draft PR and the long discussion there. It was hard to follow as there was a lot of OpenAPI and JSON Schema "history" involved and I got lost.

just to make it clear: for me this is clearly only about the message payload referencing so referencing of schemas only, not referencing other parts of AsyncAPI

what I understand is:

  • the community needs a way to reference the payload of schema in xsd or proto
  • I assume we will need also something for GraphQL at some point of time
  • sometimes it is about referencing just a file, sometimes part of the file, as both in xsd and proto you can have some specific parts to reference too, like you can have multiple messages in one proto file
  • $ref cannot be used as it is for JSON

How about solving it by having another way of referencing things:

  • so alongside Schema Object, Reference Object we could have something like Schema Reference Object
  • ServerlessWorkflow spec has it already (different structure, pointing them out as example of custom pointers) and it makes lots of sense -> https://github.com/serverlessworkflow/specification/blob/main/specification.md#workflow-functions and the structure is pretty clear
  • long term Reference Object could be deprecated (AsyncAPI 5.0 - although I do not have strong opinion on deprecation need) and people would always use Schema Reference Object

So in short $ref is just for bundling multiple JSON/YAML AsyncAPI structures into one, referencing inside document different structures. It is better and more intuitive for users to reference schemas in a more defined way when you can say that whatever comes after # is a proto message:

"reference": "https://gist.githubusercontent.com/iamsurajgharat/1d43d8181f286a58617adebce1714c6f/raw/343d91d8e3f09ab717f50bffa3213a7fd8fb0e13/sumservice.proto#SumInput"

or xsd element:

"reference": "https://gist.githubusercontent.com/iamsurajgharat/1d43d8181f286a58617adebce1714c6f/raw/343d91d8e3f09ab717f50bffa3213a7fd8fb0e13/sumservice.proto#ldml"

the only problem I see is with the schemaFormat. I'm not 100% sure yet how that would play. The more I think about it, my view is that schemaFormat should not be defined outside payload as it is now.

PS. I do not worry that we will not have out-of-the-box tooling support as we have for $ref. We will just make sure to have current Studio and IDEs plugins support it and that should be really enough.

@jonaslagoni
Copy link
Member Author

jonaslagoni commented Jan 9, 2023

just to make it clear: for me this is clearly only about the message payload referencing so referencing of schemas only, not referencing other parts of AsyncAPI

Agreed, I think it will just over complicate things otherwise.

So in short $ref is just for bundling multiple JSON/YAML AsyncAPI structures into one, referencing inside document different structures. It is better and more intuitive for users to reference schemas in a more defined way when you can say that whatever comes after # is a proto message:

That would definitely be possible yes.

Do you know if serverlessworkflow has a core tool for solving references we could maybe use if this became the case?

PS. I do not worry that we will not have out-of-the-box tooling support as we have for $ref. We will just make sure to have current Studio and IDEs plugins support it and that should be really enough.

That is a contradicting sentence, cause Studio and IDE plugins can't support it without the core tool in place 🤷 Unless they implement the functionality themself of course, but then it might as well be in a separate tool everyone can integrate. Or did I misunderstand you? 🤔

@jonaslagoni
Copy link
Member Author

So started specific discussions for the 2 proposals in JSON Schema referencing repository and answered the outstanding clarification questions in separate issues for each standard

They seem to both support what we want, and leave it up to the context specification (AsyncAPI) to dictate the behavior necessary for referencing non-JSON Standards.

Copy link
Member

derberg commented Jan 12, 2023

Do you know if serverlessworkflow has a core tool for solving references we could maybe use if this became the case?

I do not think they have any standard there. Their needs are also super specific, like for example in case of referencing AsyncAPI, reference points to operationId, so they for sure just pull out the message payload of that operation "manually" without any standard in use.

Regarding JRI or JRef -> I could not find any reference on internet, what are these standards all about? and are they useful if there are no reference implementations anyway?

@GreenRover
Copy link
Collaborator

+1
I am interested in protobuff support as well.

@GreenRover
Copy link
Collaborator

@derberg i dont see GraphQL support.
Because as i understand GraphQL it is HTTP based only. And dont know concepts like channels.
Or differently sayed: channels/operations are all parts of the GraphQL schema.
So GraphQL is more a competitor of AsyncAPI as something that could be integreated

@derberg
Copy link
Member

derberg commented Mar 15, 2023

@GreenRover well, maybe not competitor, but yeah it is different spec for different use cases, and it has 2 things in one, synchronous queries and mutations and asynchronous subscriptions.

What I meant by mentioning GraphQL is specifically GraphQL schemas. Just like with OpenAPI, we support that someone can reference OpenAPI Schema in AsyncAPI document. Can be that someone already has schemas provided for GraphQL API, and they could reference them in AsyncAPI docs for other service. But there is nothing custom here, whatever solution we will have for XSD or Proto will work with GraphQL as it is very similar to Proto as far as I remember (didn't use it for over 3 year 😅 )

@rober15
Copy link

rober15 commented May 15, 2023

I think we have as well to consider another type of async messaging added now as well to Open API:
https://swagger.io/docs/specification/callbacks/

So i wonder how to handle that Callbacks which can be used for webhooks but shown under Open API which most people associate more with synchronous messaging and not as well async.

For the message other message formats a fast help could be a reference to other schemas.
That at least would allow companies to show all async apis in one place no matter what format they use to exchange.

Maybe similar to #ref we could have #extref

Copy link
Member

derberg commented May 15, 2023

@rober15 we will "soon" release AsyncAPI 3.0 where one of features is support for request/reply in EDA, so you can specify that operation B is a reply to operation A or specify that reply to operation A should go to dynamically specified target, through the header info, for example

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale label Sep 13, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 11, 2024
@AnimeshKumar923
Copy link
Contributor

Is it still valid? @jonaslagoni @derberg

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💡 Proposal (RFC 1) RFC Stage 1 (See CONTRIBUTING.md) stale
Projects
None yet
Development

No branches or pull requests

5 participants