Does the NATS source drain messages when shutdown signal is given?

[benjamin-awd]

Hi, I've been looking into the NATS source and was wondering if Vector executes a "graceful" shutdown when the SIGTERM signal occurs. Based on my understanding, the subscriber is killed after a certain point in time after the signal is given and continues to take messages until then.

Should there be an additional unsubscribe() call somewhere? (or otherwise _connection.close())

[jszwedko]

Hi @benjamin-awd !

The expectation is that sources stop consuming when a SIGTERM is received. Any in-flight data continues to be processed until there is none or the graceful shutdown timeout is hit. For the nats source, this is handled by

vector/src/sources/nats.rs

Line 229 in 6d1d521

let stream = subscriber.take_until(shutdown);

which stops consuming the subscriber channel when a shutdown is received.

It is possible that an additional unsubscribe() call should be in there, though, to tell NATs to stop forwarding messages, but we may be relying on the Drop behavior of the Subscriber which calls unsubscribe as you can see here: https://docs.rs/async-nats/0.37.0/src/async_nats/lib.rs.html#1266-1280. An explicit unsubscribe would be clearer, in my opinion.

Are you observing the source continuing to consume messages after the shutdown signal is received?

Hopefully that helps!

[benjamin-awd]

Thanks @jszwedko, your response was super helpful -- I was looking at Grafana and noticed that the amount of messages received/sent was noticeably dropping when spinning down pods, but maybe that makes sense since we're only scraping the Prometheus metrics every 60s.

I ran a few tests locally and I didn't notice any strange behavior, so I think no issue there. I agree that it would be nice to explicitly call unsubscribe, but it may be better to wait for the PR implementing drain() to be merged on the async_nats side: nats-io/nats.rs#1325.

This seems to be slightly more robust since it reduces the chance of in-flight messages being dropped.