Add 'sequence' attribute #291

duglin · 2018-08-16T15:44:43Z

Fixes: #191

Actually, it picks up the work from #191 since that one appears to be stale.

I didn't address any of the comments in #191 because I wasn't sure how
the WG in general felt about them. So please speak up in this PR if you'd like
to see a change.

I'm not advocating for, or against, this proposal - I'm just moving it to a possible extension attribute per our F2F meeting so the WG can decide if we want to include it.

Signed-off-by: Doug Davis dug@us.ibm.com

BenBeattieHood · 2018-08-21T05:13:30Z

Many thanks for picking this up @duglin - I've been away for a bit and unable to respond. I agree that this is better suited as an extension than a core feature if IoC equipment is unable to provide state for incrementing an ordinal field. I do believe this extension is key to designing for events, as otherwise you get forward-coupling between services; but am happy to let time work this out.

ultrasaurus · 2018-08-30T03:12:39Z

extensions/ordinal.md

@@ -0,0 +1,15 @@
+# Ordinal


Suggest name: Sequence

"ordinal" implies a number; however, this is a string "expressing the relative order of the event" -- this solves a broader range of use cases than just a number, and is definitely a good approach. Propose we adjust the naming to match the definition.

BenBeattieHood · 2018-08-30T05:21:26Z

Good idea. I'm looking for a name that indicates uniqueness and contiguity. I feel in software we're used to ordinals being unique and continuous, but I agree it traditionally has meant a number which I also agree would be a mistake for this use. Sequence seems better but does it imply uniqueness and contiguity?

…

On Thu, 30 Aug 2018 at 13:12 Sarah Allen ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In extensions/ordinal.md <#291 (comment)>: > @@ -0,0 +1,15 @@ +# Ordinal Suggest name: Sequence "ordinal" implies a number; however, this is a string "expressing the relative order of the event" -- this solves a broader range of use cases than just a number, and is definitely a good approach. Propose we adjust the naming to match the definition. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#291 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFXUxzeSeDSPVZ_5CHnpIiqKg2c8q6oAks5uV1grgaJpZM4WAHW1> .

--

Tapppi · 2018-08-30T20:23:11Z

I'm looking for a name that indicates uniqueness and contiguity. I feel in software we're used to ordinals being unique and continuous, -- Sequence seems better but does it imply uniqueness and contiguity?

They both indicate "a particular way to order a number of things", except:

ordinal numbers are adopted from set theory, meaning there should be no repetition
sequences are a more general concept, allowing repetition within the sequence.

I prefer sequence, as it is the more general approach to the problem not limited by confusion with terms from set theory. I also think it sounds nicer and more understandable as a relative order for things.

Fixes: cloudevents#191 I didn't address any of the comments in cloudevents#191 because I wasn't sure how the WG in general felt about them. So please speak up in this PR if you'd like to see a change. Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin · 2018-09-01T20:06:28Z

Do we need to define the values of this sting or is it expected that all consumers will know what values each producer will use? As currently written, w/o some out-of-band communication with the producer, I wouldn't be able to use this extension. I'd like to add some text to the PR telling people about this if that's the case - but I need some guidance on what people's expectations are on this point.

BenBeattieHood · 2018-09-02T22:58:09Z

@Tapppi I'm ok with repetition, so long as the values are contiguous and therefore assertable.
@duglin Yes, if we were constrained to base-10 integers, it would be more obvious; but if the producer used a wider base (eg base 128) this would need to be communicated in order for the consumer to predict the next value.

BenBeattieHood · 2018-09-02T23:12:26Z

Fwiw, I still feel it's worth discussing that this sequence number, when combined with the source value, makes the eventID unnecessary. eg. one could synthesize an 'eventid' from [source]+[sequence], eg IoT/LightSensor5/Event4, which would be unique.

duglin · 2018-09-03T09:51:51Z

I switched this to use "sequence" and added some text about how to know what the possible values are. What do people think?

Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin · 2018-09-05T02:46:59Z

In your example, wouldn't the consumer also stop due to it waiting for 'b' to arrive - which it never will according to your example?

Added: RECOMMENDED as contiguous

cneijenhuis · 2018-09-05T08:21:17Z

In ecommerce, processing events in order and without gaps is often important. Our API enables the event consumers to do so by having a field called sequenceNumber.

The feedback I'm often getting is that it is tiresome to re-implement the in-order processing. This is especially true for an (otherwise) stateless consumer: State has to be introduced only for saving the source/sequence pairs. You're forced to go from FaaS to FaaS + Storage.

Personally, I'd prefer a more opinionated approach that does not require out-of-band communication. Or support for both standardized sequences and non-standardized ones that require out-of-band communication.

This would allow some middleware (either the FaaS itself, or an event gateway) to order the events, and the consumer to process them in order without having to re-implement it and keep state.

Example for an opinionated approach

Call the field intSequence or int32Sequence (to allow the definition of other sequences).
MUST start at 1, MUST be contiguous, MUST wrap around after 2,147,483,647 etc.

Example for a less opinionated approach

In addition to the field sequence, add an optional field sequenceType.
If the sequenceType field is set to Integer (or Int32), the sequence field MUST start at 1, MUST be contiguous, MUST wrap around after 2,147,483,647 etc.

The details need to be worked out, but both of these examples make out-of-band communication unnecessary.

duglin · 2018-09-05T13:48:54Z

@cneijenhuis I like the idea of two fields (sequence and sequenceType). We can then allow for some well-defined sequenceType values and allow people to define new ones - so its an extensibility point. What do other people think?

BenBeattieHood · 2018-09-05T23:06:01Z

I agree with including the sequenceType (in addition to sequence) - good suggestion.

(@duglin re 'b': sorry about my example (appears to have disappeared with the updated commit) - you're right I should've used tokens more obviously contiguous - I'm grateful you picked up on the main point I was trying to say.)

duglin · 2018-09-05T23:55:47Z

@cneijenhuis @BenBeattieHood do one of you guys want to suggest some text for what these two attributes would look like?

cneijenhuis · 2018-09-06T15:27:24Z

Here is my first try. Sorry, not wrapped at 80 characters yet 😉

I can also make a PR towards this branch (including wrapping), if that is easier.

Sequence

This extension defines an attribute that can be included within a CloudEvent to represent the position of the event relative to the other events from the same event source.

The exact value and meaning of this attribute is defined by the sequence type.
If the sequence type is missing, or not defined in this specification, event consumers will need to
have some out-of-band communication with the event producer to understand how to interpret the value of the attribute.

Attributes

sequence

Type: String
Description: Value expressing the relative order of the event. This enables
interpretation of data supercedence.
Constraints
- REQUIRED
- MUST be a non-empty lexicographically-orderable string
- RECOMMENDED as contiguous

sequenceType

Type: String
Description: Defines the semantics of the sequence attribute. The value "Integer" is defined in this specification. Other values are allowed.
Constraints
- OPTIONAL

Integer

If the sequenceType is set to "Integer", the sequence attribute has the following semantics:

The values are string-encoded, signed 32-bit Integers.
The first CloudEvent in an Integer sequence has the value 1, the second 2 etc.
The sequence is contiguous. In other words, if the event consumer receives two CloudEvents with sequence values of 1 and 3, another CloudEvent with a sequence value of 2 exists, but has not been received yet.
The sequence wraps around from 2,147,483,647 (2³¹ - 1) to −2,147,483,648 (-2³¹). The sequence contains repetitions if there are more than 2³² CloudEvents.

duglin · 2018-09-10T12:06:36Z

@cneijenhuis thanks! I just added your suggested text. All - please see what you think

duglin · 2018-09-13T15:20:08Z

oh darn, I just realized that we don't say what the default sequenceType value is if its not present. I'm assuming Integer. Will add that

cneijenhuis · 2018-09-13T15:46:52Z

@duglin I left it out intentionally - if it is missing, the consumer should treat it as an unknown value.

See https://github.com/cloudevents/spec/pull/291/files#diff-7fb2227cb146a7ff3164326d6b46f5a1R10

But I'm also totally fine with making Integer the default, and forcing producers to declare some other value. We should change the upper paragraph then, though.

Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin · 2018-09-13T15:51:25Z

@cneijenhuis oh sorry, I missed that line. I reverted my change. While I think defaulting to Integer might be nice, just to provide some base level of interop, as long as we say what's expected when its missing (even if its "use out of band communication") that addresses my biggest concern. I'll leave it to you experts to decide if "out of band" is ok or if we should default to Integer.

erikerikson · 2018-09-13T23:49:33Z

extensions/sequence.md

+* Constraints
+  * REQUIRED
+  * MUST be a non-empty lexicographically-orderable string
+  * RECOMMENDED as contiguous


Possibly:
MUST be produced in a monotonically increasing manner

at least monotonic

added "monotonically". Wasn't sure about the "increasing" part since that might imply a numeric based sequence and I'm assuming there might be some that don't fit that model. Is that a correct assumption?

Answering because you asked. You've already added "increasing".

I feel a little uncertain and would defer to a mathematician but the lexicographically orderable requirement gives an absolute ordering across all unique values. I would expect "increasing" to make sense in that context and mean "each subsequent value would be ordered/sorted after all preceding values".

For the record, my initial question would disallow the reasonable circular counter described in the Integer sequence type and therefore could not be a MUST.

erikerikson

This is an important extension, thank you for adding it.

erikerikson · 2018-09-14T04:38:29Z

extensions/sequence.md

+# Sequence
+
+This extension defines two attributes that can be included within a CloudEvent
+to describe the position of the event relative to the other events from the


Perhaps?
to describe the position of an event in the ordered sequence of events produced by a unique event source

erikerikson · 2018-09-14T05:48:44Z

extensions/sequence.md

+
+### sequenceType
+* Type: `String`
+* Description: Defines the semantics of the sequence attribute.


Nit: Specifies over defines since the actual definition is external or listed in the section below

erikerikson · 2018-09-14T05:51:18Z

extensions/sequence.md

+the following semantics:
+* The values of `sequence` are string-encoded signed 32-bit Integers
+* The first CloudEvent in an `Integer` sequence MUST have the value
+  `1`, the second `2`, etc.


the , the second `2`, etc. seems duplicative of the following line while specifying the first has value 1 seems properly independent.

I tried to tweak it, let me know if I went too far

erikerikson · 2018-09-14T05:57:34Z

extensions/sequence.md

+### sequence
+* Type: `String`
+* Description: Value expressing the relative order of the event. This enables
+  interpretation of data supercedence.


idempotency?

I wonder if this would be better done by checking the EventID instead? If not, have any suggested text?
Do we need to worry about a single event being part of more than one sequence?

I wonder if this would be better done by checking the EventID instead?

That's a great question. As per the eventID definition there is no monotinicity requirement. Without a value constraint, one would have to keep track of every eventID that has been processed (or at the very least a significant scope of them) to ensure no duplicate processing as opposed to checking the relationship of the currently being processed eventID to the last successful checkpoint eventID. I'm not sure that the assumptions I like to make would be appropriate to push onto all users fo the spec so I haven't advocated for that there but would have them adopted in my more-perfect world.

Do we need to worry about a single event being part of more than one sequence?

I'm taking an event as the tangible message sent by a unique producer (as opposed to the occurrence in reality being read by some sensor). As I've read this change, it seemed that there was a one-to-one mapping between unique producers of events with this attribute and the series of numbers representing the singular sequence that they produce. That's a long way to say I don't expect a unique producer to produce multiple sequences OR for multiple unique producers to produce the "same" event (though distinct events could be causally interlinked and contain the exact same content [sans producer id] by the shared reality producing them - e.g. redundant emitters of metrics coming from a single sensor). I could definitely be reading incorrectly.

erikerikson · 2018-09-14T05:58:06Z

extensions/sequence.md

+* Description: Value expressing the relative order of the event. This enables
+  interpretation of data supercedence.
+* Constraints
+  * REQUIRED


can extensions be required?

This may be something we need to add more text on... but the point here is that while the extension itself is optional, IF it is being used then this property is REQUIRED. See the documented-extension.md doc, where it says "Support for any extension is OPTIONAL. When an extension definition uses RFC 2199 keywords (e.g. MUST, SHOULD, MAY), this usage only applies to events that use the extension."

Thank you for clarifying. This makes perfect sense and clears my confusion.

cneijenhuis · 2018-09-17T07:56:08Z

extensions/sequence.md

@@ -20,11 +20,11 @@ event producer to understand how to interpret the value of the attribute.
 * Constraints
  * REQUIRED
  * MUST be a non-empty lexicographically-orderable string
-  * RECOMMENDED as contiguous
+  * RECOMMENDED as montomically contiguous


There is a typo, maybe monotonically increasing and contiguous reads better?

My math is quite rusty, but I'm not sure if a char-based sequence can legitimately be called monotonically increasing (e.g. a, b, ...z, aa, ab, ...). But, as it is only a recommendation, I think it's fine to get the point across.

erikerikson · 2018-09-18T21:59:07Z

extensions/sequence.md

+the following semantics:
+* The values of `sequence` are string-encoded signed 32-bit Integers.
+* The sequence MUST be contiguous, monotonically increasing by `1` and
+  start with a value of `1`.


Sorry to nit. It seems like it might read more naturally as

The sequence MUST start with a value of `1` and increase by `1` for each subsequent value (i.e. be contiguous and monotonically increasing).

works for me

Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin · 2018-09-20T12:14:50Z

Based on the discussions in #308, can those who are interested in possibly using this extension please speak up via a comment in this PR? @BenBeattieHood I assumed you would be - @Tapppi ? @cneijenhuis ?

cneijenhuis · 2018-09-23T21:04:07Z

Yes, commercetools is strongly interested in this. We already provide an Integer-based sequence for our proprietary events (called sequenceNumber). For CloudEvents, we'd like to cover this use-case in a non-proprietary way.

In fact, my hope is that an attribute for ordering events makes it into the main spec until 1.0. I think the extension proposed in this PR is a good first step towards that.

BenBeattieHood · 2018-09-24T06:26:19Z

You're not wrong @duglin :) I can't see using events without transmitting order - it brings too much logical coupling between services otherwise.

duglin · 2018-09-26T01:15:18Z

Is this one ready for review? @BenBeattieHood @cneijenhuis you guys ok with the current state?

BenBeattieHood · 2018-09-26T05:08:54Z

Yes, thanks :)

cneijenhuis · 2018-09-26T08:07:00Z

Yes 👍 Thanks for pushing this PR forward!

duglin · 2018-09-27T17:32:52Z

Approved on 9/27 call

Voters who voiced support (per our new "extension bar"): Christoph(commercetools), Jesse(Oracle)

Tapppi · 2018-09-28T11:18:03Z

Thanks for pushing this forward, I'm a bit swamped with product launches at the moment and can barely make it to the calls :) Definitely worthy extension though.

* Add 'ordinal' attribute Fixes: cloudevents#191 I didn't address any of the comments in cloudevents#191 because I wasn't sure how the WG in general felt about them. So please speak up in this PR if you'd like to see a change. Signed-off-by: Doug Davis <dug@us.ibm.com> * use sequence and add more clarifying text Signed-off-by: Doug Davis <dug@us.ibm.com> * grab Christoph's suggested text Signed-off-by: Doug Davis <dug@us.ibm.com> * more tweaks Signed-off-by: Doug Davis <dug@us.ibm.com> Signed-off-by: Clemens Vasters <clemensv@microsoft.com>

duglin mentioned this pull request Aug 16, 2018

Add event 'ordinal' field #191

Closed

duglin force-pushed the pr191 branch from ff31df8 to e5d502c Compare August 16, 2018 19:54

ultrasaurus reviewed Aug 30, 2018

View reviewed changes

Add 'ordinal' attribute

169b1e8

Fixes: cloudevents#191 I didn't address any of the comments in cloudevents#191 because I wasn't sure how the WG in general felt about them. So please speak up in this PR if you'd like to see a change. Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin force-pushed the pr191 branch from e5d502c to 169b1e8 Compare September 1, 2018 20:03

duglin changed the title ~~Add 'ordinal' attribute~~ Add 'sequence' attribute Sep 3, 2018

use sequence and add more clarifying text

008f598

Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin force-pushed the pr191 branch from 86356d1 to 008f598 Compare September 5, 2018 02:46

duglin force-pushed the pr191 branch from 3d9ffe2 to f1090dd Compare September 10, 2018 12:12

duglin force-pushed the pr191 branch from f1090dd to 70b7c40 Compare September 13, 2018 15:21

grab Christoph's suggested text

bfcaae1

Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin force-pushed the pr191 branch from 70b7c40 to bfcaae1 Compare September 13, 2018 15:49

erikerikson reviewed Sep 13, 2018

View reviewed changes

erikerikson reviewed Sep 14, 2018

View reviewed changes

cneijenhuis reviewed Sep 17, 2018

View reviewed changes

duglin force-pushed the pr191 branch from 048611b to d37bceb Compare September 17, 2018 12:35

erikerikson reviewed Sep 18, 2018

View reviewed changes

more tweaks

f000004

Signed-off-by: Doug Davis <dug@us.ibm.com>

duglin force-pushed the pr191 branch from d37bceb to f000004 Compare September 19, 2018 00:24

BenBeattieHood mentioned this pull request Sep 24, 2018

Proposal for an event key field as an extension #218

Merged

duglin merged commit 115493f into cloudevents:master Sep 27, 2018

duglin deleted the pr191 branch September 27, 2018 17:33

duglin mentioned this pull request Nov 30, 2018

v0.2 #358

Merged

duglin mentioned this pull request Jun 16, 2022

"sequence" extension - integer encoded but lexicographically-orderable string? #923

Closed

Add 'sequence' attribute #291

Add 'sequence' attribute #291

Conversation

duglin commented Aug 16, 2018

BenBeattieHood commented Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

BenBeattieHood commented Aug 30, 2018 via email

Tapppi commented Aug 30, 2018

duglin commented Sep 1, 2018

BenBeattieHood commented Sep 2, 2018

BenBeattieHood commented Sep 2, 2018 • edited Loading

duglin commented Sep 3, 2018

duglin commented Sep 5, 2018

cneijenhuis commented Sep 5, 2018 • edited Loading

Example for an opinionated approach

Example for a less opinionated approach

duglin commented Sep 5, 2018

BenBeattieHood commented Sep 5, 2018

duglin commented Sep 5, 2018

cneijenhuis commented Sep 6, 2018 • edited Loading

Sequence

Attributes

sequence

sequenceType

Integer

duglin commented Sep 10, 2018

duglin commented Sep 13, 2018

cneijenhuis commented Sep 13, 2018

duglin commented Sep 13, 2018

erikerikson Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikerikson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented Sep 20, 2018

cneijenhuis commented Sep 23, 2018 • edited Loading

BenBeattieHood commented Sep 24, 2018

duglin commented Sep 26, 2018

BenBeattieHood commented Sep 26, 2018

cneijenhuis commented Sep 26, 2018

duglin commented Sep 27, 2018

Tapppi commented Sep 28, 2018

BenBeattieHood commented Aug 21, 2018 •

edited

Loading

BenBeattieHood commented Sep 2, 2018 •

edited

Loading

cneijenhuis commented Sep 5, 2018 •

edited

Loading

cneijenhuis commented Sep 6, 2018 •

edited

Loading

erikerikson Sep 13, 2018 •

edited

Loading

cneijenhuis commented Sep 23, 2018 •

edited

Loading