Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add googlepubsub bindings #141

Merged
merged 3 commits into from
Sep 27, 2022
Merged

feat: add googlepubsub bindings #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5c7fd1e
feat: add googlepubsub bindings
whitlockjc Aug 1, 2022
885fdef
Merge branch 'master' into master
derberg Sep 20, 2022
4b18405
Merge branch 'master' into master
derberg Sep 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions CODEOWNERS
View file
Open in desktop
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@

* @derberg @fmvilas @asyncapi-bot-eve

/anypointmq/ @GeraldLoeffler
/ibmmq/ @rcoppen
/kafka/ @lbroudoux @dalelane
/solace/ @damaru-inc @CameronRushton
*.json @KhudaDad414
/anypointmq/ @GeraldLoeffler
/ibmmq/ @rcoppen
/kafka/ @lbroudoux @dalelane
/googlepubsub/ @whitlockjc
/solace/ @damaru-inc @CameronRushton
*.json @KhudaDad414
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This repository contains the specifications for each AsyncAPI protocol binding.

* [AMQP binding](./amqp)
* [AMQP 1.0 binding](./amqp1)
* [Google Cloud Pub/Sub binding](./googlepubsub)
* [HTTP binding](./http)
* [IBM MQ binding](./ibmmq)
* [JMS binding](./jms)
Expand Down
170 changes: 170 additions & 0 deletions googlepubsub/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# Google Cloud Pub/Sub Bindings

This document defines how to describe Google Cloud Pub/Sub specific information with AsyncAPI.

<a name="version"></a>

## Version

Current version is `0.1.0`.

<a name="channel"></a>

## Channel Binding Object

The `Channel Bindings Object` is used to describe the Google Cloud Pub/Sub specific
[Topic](https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics/create) details with AsyncAPI.

Field Name | Type | Description
---|---|---
`bindingVersion`|String|The current version is `0.1.0`
`labels`|Object|An object of key-value pairs _(These are used to categorize Cloud Resources like Cloud Pub/Sub Topics.)_
`messageRetentionDuration`|String|Indicates the minimum duration to retain a message after it is published to the topic _(Must be a valid [Duration](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration).)_
whitlockjc marked this conversation as resolved.
Show resolved Hide resolved
`messageStoragePolicy`|[Message Storage Policy Object](#message-storage-policy-object)|Policy constraining the set of Google Cloud Platform regions where messages published to the topic may be stored
whitlockjc marked this conversation as resolved.
Show resolved Hide resolved
`schemaSettings`|[Schema Settings Object](#schema-settings-object)|Settings for validating messages published against a schema
`topic`|String|The Google Cloud Pub/Sub Topic name

<a name="message-storage-policy-object"></a>

### Message Storage Policy Object

The `Message Storage Policy Object` is used to describe the Google Cloud Pub/Sub
[MessageStoragePolicy](https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics#MessageStoragePolicy)
Object with AsyncAPI.

Field Name | Type | Description
---|---|---
`allowedPersistenceRegions`|String[]|A list of IDs of GCP regions where messages that are published to the topic may be persisted in storage

<a name="schema-settings-object"></a>

### Schema Settings Object

The `Schema Settings Object` is used to describe the Google Cloud Pub/Sub
[SchemaSettings](https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics#SchemaSettings) Object with
AsyncAPI.

Field Name | Type | Description
---|---|---
`encoding`|String|The encoding of the message _(Must be one of the possible [Encoding](https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics#encoding) values.)_
`firstRevisionId`|String|The minimum _(inclusive)_ revision allowed for validating messages
`lastRevisionId`|String|The maximum _(inclusive)_ revision allowed for validating messages
`name`|String|The name of the schema that messages published should be validated against _(The format is `projects/{project}/schemas/{schema}`.)_

<a name="channel-binding-example"></a>

### Example

```yaml
# ...
channels:
topic-avro-schema:
bindings:
googlepubsub:
topic: projects/your-project/topics/topic-avro-schema
schemaSettings:
encoding: json
name: projects/your-project/schemas/message-avro
# ...
topic-proto-schema:
bindings:
googlepubsub:
topic: projects/your-project/topics/topic-proto-schema
messageRetentionDuration: 86400s
messageStoragePolicy:
allowedPersistenceRegions:
- us-central1
- us-central2
- us-east1
- us-east4
- us-east5
- us-east7
- us-south1
- us-west1
- us-west2
- us-west3
- us-west4
schemaSettings:
encoding: binary
name: projects/your-project/schemas/message-proto
# ...
```

<a name="message"></a>

## Message Binding Object

The `Message Binding Object` is used to describe the Google Cloud Pub/Sub specific
[PubsubMessage](https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage) details, alongside with pertintent
parts of the Google Cloud Pub/Sub
[Schema](https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.schemas#Schema)
Object, with AsyncAPI.

Field Name | Type | Description
---|---|---
`bindingVersion`|String|The current version is `0.1.0`
`attributes`|Object|Attributes for this message _(If this field is empty, the message must contain non-empty data. This can be used to filter messages on the subscription.)_
`orderingKey`|String|If non-empty, identifies related messages for which publish order should be respected _(For more information, see [ordering messages](https://cloud.google.com/pubsub/docs/ordering).)_
`schema`|[Schema Definition Object](#schema-definition-object)|Describes the schema used to validate the payload of this message

<a name="schema-definition-object"></a>

### Schema Definition Object
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite a workaround 😄
Shouldn't we try to fix it on a spec level rather than add such workaround gates on a binding level?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it's really not. The whole purpose behind this is to describe the schema Cloud Pub/Sub uses to validate messages. While AsyncAPI does have support for this already in the messages themselves, there is metadata on the schema object from a Cloud Pub/Sub perspective that is still useful for documenting the API. Below is an example of how AsyncAPI's support for Avro and how its lack of Protobuf support would be documented:

# ...
components:
    messages:
        messageAvroAsJSON:
            bindings:
                googlepubsub:
                    schema:
                        definition: |
                            {
                              "type" : "record",
                              "name" : "Message",
                              "fields" : [
                                {
                                  "name" : "message",
                                  "type" : "string"
                                }
                              ]
                            }
                        name: projects/jaydub-testing/schemas/message-avro
                        type: avro
            contentType: application/json
            name: projects/jaydub-testing/schemas/message-avro
            payload:
                fields:
                    - name: message
                      type: string
                name: Message
                type: record
            schemaFormat: application/vnd.apache.avro+yaml;version=1.9.0
        messageProto:
            bindings:
                googlepubsub:
                    schema:
                        definition: |
                            syntax = "proto2";

                            message Message {
                              required string message = 1;
                            }
                        name: projects/jaydub-testing/schemas/message-proto
                        type: protobuf
            contentType: application/octet-stream
            name: projects/jaydub-testing/schemas/message-proto
            payload: true
# ...

name is metadata that could/should be present, but I could see a case where if Protobuf were supported by AsyncAPI natively for Message payloads, maybe it wouldn't need to be present in the binding. Thoughts?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @derberg: I think we absolutely have to avoid duplicating things in bindings when they should be defined in the message itself. Protobuf support should ideally follow the same approach as for Avro: having a dedicated schemaFormat entry for it. However due to proto syntax, we'll probably not be able to embed it in the AsyncAPI spec itself and use a $ref to external file.

The value you put in name (that should be human-readable according the spec) seems similar to the one put into schemaSettings.name at the Channel level. Is it just a duplication? Does it serve a specific purpose to have it structured like the schema name in your GCP project?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When it comes to embedding versus using a $ref, to me it's six in one hand and half another in the other. Ultimately, we want the API consumer to know as much as they can about the API contract and having a visual representation of unsupported schema types could serve some purpose. As for duplication, I'm all for removing it but if it means inconsistency (duplication is removed for use case X but not use case Y), I personally would rather allow duplication and have consistency. Finally, for name, its structured is by design and a GCP concept.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, agree that's really a workaround. Let's wait until we have Protobuf support at the spec level using schemaFormat.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we do if/when GCP supports a new schema format? Do we disallow any sort of representation within the binding in hopes that AsyncAPI supports it? What do you think about making this optional instead of unnecessary?

Copy link
Collaborator

@lbroudoux lbroudoux Aug 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding name: I missed the point that I referred to schemaSettings.name and that it was a different thing than the topic name...

That said I realize that topic technical name was directly in the channel path. However this one should be human-readable, independent of binding (in case of multi-bindings supports) and may also contain parameters!

In other bindings, we decided to add a specific property within the channel binding in order to add the technical name of the destination (look at amqp where we have queue.name or exchange.name ; we also have similar things in ibmmq). I think we should do the same here so that all the technical things related to a binding are kept into this binding.

We'd ended up with something like:

channels:
  my-channel-name-with-business-meaning:
    bindings:
      googlepubsub:
        topic: projects/your-project/topics/topic-avro-schema
        schemaSettings:
          encoding: avro
          name: projects/your-project/schemas/message-avro

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we do if/when GCP supports a new schema format? Do we disallow any sort of representation within the binding in hopes that AsyncAPI supports it?

@whitlockjc that sounds like a perfect use case for an extension.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can go either way, but ultimately I need your agreement to get this submitted so I don't feel I've got much choice here. Requiring AsyncAPI to support a schema format formally versus allowing a binding to store potentially duplicate information might not be my first choice, the whole point of x-* is to fill the gaps so I don't care enough to push back.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the binding to add topic and to remove definition from schemaSettings per the discussion above.


The `Schema Definition Object` is used to describe the Google Cloud Pub/Sub
[Schema]([Schema](https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.schemas#Schema)) Object with AsyncAPI.
While some of this information could be, or is, described using native AsyncAPI, for consistency it makes sense to
provide this information here at all times, especially for cases where AsyncAPI does not natively support describing
payloads using a supported Google Cloud Pub/Sub schema format like Protobuf.

Field Name | Type | Description
---|---|---
`name`|String|The name of the schema
`type`|String|The type of the schema

<a name="message-binding-example"></a>

### Example

```yaml
# ...
components:
messages:
messageAvro:
bindings:
googlepubsub:
schema:
name: projects/your-project/schemas/message-avro
type: avro
contentType: application/json
name: MessageAvro
payload:
fields:
- name: message
type: string
name: Message
type: record
schemaFormat: application/vnd.apache.avro+yaml;version=1.9.0
messageProto:
bindings:
googlepubsub:
schema:
name: projects/your-project/schemas/message-proto
type: protobuf
contentType: application/octet-stream
name: MessageProto
payload: true
# ...
```

<a name="operation"></a>

## Operation Binding Object

This object MUST NOT contain any properties. Its name is reserved for future use.

<a name="server"></a>

## Server Binding Object

This object MUST NOT contain any properties. Its name is reserved for future use.
82 changes: 82 additions & 0 deletions googlepubsub/json_schemas/channel.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
{
whitlockjc marked this conversation as resolved.
Show resolved Hide resolved
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "http://asyncapi.com/bindings/googlepubsub/channel.json",
"title": "Cloud Pub/Sub Channel Schema",
"description": "This object contains information about the channel representation for Google Cloud Pub/Sub.",
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^x-[\\w\\d\\.\\-\\_]+$": {
"$ref": "https://raw.githubusercontent.com/asyncapi/spec-json-schemas/v2.14.0/schemas/2.4.0.json#/definitions/specificationExtension"
}
},
"properties": {
"bindingVersion": {
"type": "string",
"enum": [
"0.1.0"
],
"description": "The version of this binding."
},
"labels": {
"type": "object"
},
"messageRetentionDuration": {
"type": "string"
},
"messageStoragePolicy": {
"type": "object",
"additionalProperties": false,
"properties": {
"allowedPersistenceRegions": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"schemaSettings": {
"type": "object",
"additionalItems": false,
"properties": {
"encoding": {
"type": "string"
},
"firstRevisionId": {
"type": "string"
},
"lastRevisionId": {
"type": "string"
},
"name": {
"type": "string"
}
},
"required": ["encoding", "name"]
},
"topic": {
"type": "string"
}
},
"required": ["schemaSettings", "topic"],
"examples": [
{
"labels": {
"label1": "value1",
"label2": "value2"
},
"messageRetentionDuration": "86400s",
"messageStoragePolicy": {
"allowedPersistenceRegions": [
"us-central1",
"us-east1"
]
},
"schemaSettings": {
"encoding": "json",
"name": "projects/your-project-id/schemas/your-schema"
}
}
]
}
55 changes: 55 additions & 0 deletions googlepubsub/json_schemas/message.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
whitlockjc marked this conversation as resolved.
Show resolved Hide resolved
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "http://asyncapi.com/bindings/googlepubsub/message.json",
"title": "Cloud Pub/Sub Channel Schema",
"description": "This object contains information about the message representation for Google Cloud Pub/Sub.",
"type": "object",
"additionalProperties": false,
"patternProperties": {
"^x-[\\w\\d\\.\\-\\_]+$": {
"$ref": "https://raw.githubusercontent.com/asyncapi/spec-json-schemas/v2.14.0/schemas/2.4.0.json#/definitions/specificationExtension"
}
},
"properties": {
"bindingVersion": {
"type": "string",
"enum": [
"0.1.0"
],
"description": "The version of this binding."
},
"attributes": {
"type": "object"
},
"orderingKey": {
"type": "string"
},
"schema": {
"type": "object",
"additionalItems": false,
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
}
},
"required": ["name", "type"]
}
},
"examples": [
{
"schema": {
"name": "projects/your-project-id/schemas/your-avro-schema-id",
"type": "avro"
}
},
{
"schema": {
"name": "projects/your-project-id/schemas/your-protobuf-schema-id",
"type": "protobuf"
}
}
]
}