What is ErrNoResponders exactly?

[umutozd]

Hello 👋🏻. Me and my company have been using JetStream as our internal messaging system for a few months now. Lately, we have spared some time to dig into solidifying our client implementation that uses the nats.go package.

TL;DR

What exactly is ErrNoResponders? In what cases can it occur (I need the exact list of cases)? Why is it necessary for the internal retry logic to be triggered?

Our Use Case

Our JetStream servers run on Kubernetes as a 3-instance cluster.

Our client code publishes messages through

nats.go/js.go

Line 36 in 6c6add8

Publish(subj string, data []byte, opts ...PubOpt) (*PubAck, error)

We intend to use a retry logic here when client cannot publish for some reason. Below is our intended use:

// jsc is the acronym for our client object
opts := []nats.PubOpt{
	nats.MsgId(jsc.nuid.Next()),
	nats.RetryWait(time.Second),  // 1-second wait between retry attempts
	nats.RetryAttempts(-1),       // unlimited retry-attempts
	nats.Context(ctx),            // a context with a deadline that controls the overall max duration of the Publish call
}
if _, err := jsc.js.Publish(topic, value, opts...); err != nil {
	return fmt.Errorf("error producing message: %v", err)
}

When tested with a network controller (Network Link Conditioner for our case), this call yields the following result:

• No network limit: Immediate delivery and successful result from the first request call.
• 100% packet loss: Call to the first request returns with a DeadlineExceeded error. Retry loop is not used.
• 10% packet loss: Call to the first request returns roughly after 10 seconds with a successful response.
• No network limit, but stream doesn't exist on server: In this case the first request call returns immediately and the retry loop is entered because the server returned the ErrNoResponders error. Until context deadline, loop runs every second (as we intended with nats.RetryWait(time.Second)) and finally returns with a DeadlineExceeded error.

Conclusion

Based on the above observation, I can conclude that the internal RequestMsgWithContext has some sort of retry logic because it delivers the message successfully when some of the packets are dropped. However, we don't quite understand other possible errors. For instance, what is ErrNoResponders and why is that one necessary for the retry loop to run? From our perspective, when an error is returned from the server and there is still time until the timeout, a retry should be made no matter what error is received.

Thanks to everyone in advance 🙏🏻. I'd be glad to provide any more details.

[wallyqs]

The client would get a ErrNoResponders error whenever the servers has not mounted yet the JetStream API in the server to serve a stream (for example during boot). The same would happen if the stream does not exist, since there is no JetStream API NATS subject ready for the client to send requests to in order to persists messages from that stream, so this also becomes a ErrNoResponders error.

[umutozd]

Thank you for your detailed answer @wallyqs 🙏🏻