add v3 version of nats jetstream protocol with integration tests and samples #1095
Conversation
There was a problem hiding this comment.
Regarding the samples, please take a look at how we use a protocol in https://github.com/cloudevents/sdk-go/blob/main/samples/kafka_confluent/sender/main.go
| func WithConsumerOptions(opts ...ConsumerOption) ProtocolOption { | ||
| return func(p *Protocol) error { | ||
| p.consumerOptions = opts | ||
| return nil | ||
| } | ||
| } | ||
|
|
||
| func WithSenderOptions(opts ...SenderOption) ProtocolOption { | ||
| return func(p *Protocol) error { | ||
| p.senderOptions = opts | ||
| return nil | ||
| } | ||
| } | ||
|
|
||
| // WithPublishOptions configures the Sender | ||
| func WithPublishOptions(publishOpts []jetstream.PublishOpt) SenderOption { | ||
| return func(s *Sender) error { | ||
| s.PublishOpts = publishOpts | ||
| return nil | ||
| } | ||
| } | ||
|
|
||
| // WithConsumerConfig configures the Consumer with the given config | ||
| func WithConsumerConfig(consumerConfig *jetstream.ConsumerConfig) ConsumerOption { |
There was a problem hiding this comment.
I'm still confused why we need different options/types for publisher/sender and consumer. Is there no way to hide more implementation details and just have one option type for either publisher/consumer?
There was a problem hiding this comment.
I was able to combine the SenderOptions, CustomerOptions, and JetstreamOptions into the one and only ProtocolOptions.
I was not able to combine the NatsOptions because the NatsOptions are used early in the process to create the connection.
stephen-totty-hpe
force-pushed
the
totty/nats-v3
branch
5 times, most recently
from
b5e1c62
to
16961e8
Compare
…samples Signed-off-by: stephen-totty-hpe <stephen.totty@hpe.com>
stephen-totty-hpe
force-pushed
the
totty/nats-v3
branch
from
16961e8
to
6f1ff3d
Compare
|
|
||
| // NatsOptions is a helper function to group a variadic nats.ProtocolOption into | ||
| // []nats.Option that can be used by either Sender, Consumer or Protocol | ||
| func NatsOptions(opts ...nats.Option) []nats.Option { |
There was a problem hiding this comment.
nit: IMHO this helper can either by internal if we need it or removed because it increases the API for no real benefit
| type Protocol struct { | ||
| conn *nats.Conn | ||
|
|
||
| Consumer *consumer |
There was a problem hiding this comment.
question: why do we need to keep those still public?
There was a problem hiding this comment.
in integration test:
cleanup, s, r := testProtocol(ctx, t, conn, tt.args.consumerConfig, tt.args.opts...)
the sender and receiver needs to be accesible from the protocol in order to call
test.SendReceive(t, binding.WithPreferredEventEncoding(context.TODO(), tt.args.bindingEncoding), in, s, r, ...
I can add a GetSender() or GetConsumer() in order to get the values from the Protocol object.
| } | ||
|
|
||
| // NewProtocol creates a new NATS protocol. | ||
| func NewProtocol(ctx context.Context, url, sendSubject string, natsOpts []nats.Option, opts ...ProtocolOption) (*Protocol, error) { |
There was a problem hiding this comment.
question: is natsOpts optional (can be nil/0 len) or required? If not, please make it a ProtocolOption otherwise check for its validity and return error early
There was a problem hiding this comment.
Similarly, is url and sendSubject required or can it be "" ?
There was a problem hiding this comment.
url is required by both Consumer and Sender. sendSubject is required by Sender.
There was a problem hiding this comment.
natsOpts is optional, but is highly used for things like credentials for security.
There was a problem hiding this comment.
The problem with natsOpts, is that I need it before applying the options in order to get a *nats.Conn. So a sort of chicken-egg problem. I need the *nats.Conn to create a consumer, but I need a consumer to apply an option.
There was a problem hiding this comment.
I could set the *nats.Conn on the consumer after applying the options, but I would have to see if any other option or logic needs the *nats.Conn sooner.
There was a problem hiding this comment.
If you like, I can add three more options:
func WithURL(url string, natsOpts []nats.Option) ProtocolOption
func WithConnection(conn *nats.Conn) ProtocolOption
func WithSenderSubject(sendSubject string) ProtocolOption
And then I could reduce to only one constructor that takes (ctx context.Context, opts ...ProtocolOption)
Of course, there would be more errors needed in case a Sender was created without sendSubject.
Also "one and only one" of WithURL/WithConnection must be provided.
| } | ||
|
|
||
| // NewProtocolFromConn creates a new NATS protocol with the given nats connection. | ||
| func NewProtocolFromConn(ctx context.Context, conn *nats.Conn, sendSubject string, opts ...ProtocolOption) (*Protocol, error) { |
There was a problem hiding this comment.
question: should we accept an interface for nats.Conn here to simplify mocking or hardcode it to this type?
There was a problem hiding this comment.
unfourtunately, the jetstream library explictly uses a "pointer to struct"
As been discussed previously, there has been a desire to implement the CloudEventSDK nats jetstream protocol using the newer jetstream package. The newer jetstream package exposes more jetstream specific functionality that should make it more flexible to use.
Much of this code is stolen from the v2 implementation and changed where necessary. It builds upon comments made in:
#1083
Under the covers, this uses a jetstream.Consumer, which can be "normal" or "ordered". This is done using consumer options WithConsumerConfig and WithOrderedConsumerConfig.
Also, the need to know the stream names upfront is not needed. Many of the internal options are available if needed, although the only required option is the ConsumerOption with the correct config.