-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support for key-value store #23
Conversation
Hi Piotr, Ironically, right now at my daily job I'm also working on NATS KV support, but there I have another kind of Tcl client that is a wrapper over the C client with C++/Qt - so, very different implementation, but the same questions about how Tcl API should look like. This time I'd like to avoid my mistake that I did back in 2021, when I accepted the JetStream contribution from ANT, left it as-is for many months, and then came back and rearranged many things for the 2.0 release, with many API changes, while you already had plenty of code running in production that would break after update. Now I have time for a proper review and work together with you for a good implementation, and hopefully you don't have much production code yet using KV :) Generally it's better to do this kind of major development as a Github issue, so that I can help and guide you - this way you can save time and effort. While I'm reviewing it, I'll post my thoughts in further comments. |
by the way, did you notice the functions to create streams and consumers from JSON files? I thought, it would be convenient for your team, since you have clients in different languages, and you can generate these JSON configs with NATS CLI and then keep them in git, instead of passing a long list of options to the Tcl client? So, if you think there's potential in this, it would be nice to have the same for creating buckets. I can add it later myself. |
I really appreciate that you wrote tests, docs and examples as well 👍 IMHO using NATS CLI is good for the first introduction to KV, but still - it is an interactive tool, not a client library. When I design the Tcl API, I look first of all at nats.py, and when in doubt - at nats.c and nats.go (I don't really know Go, but it's readable). So, here are the things I'd like you to change in the PR, with numbering for later reference:
And inside
Same function names are used in nats.go |
|
I suggest adding a convenience method, let's call it
|
That's all I have to say for now. I know I'm asking for a lot :) all this JetStream stuff turned out to be much more time-consuming for me than I thought in the beginning. It's objectively hard, and information is scattered. Please let me know if you have any deadlines for this PR to be merged. |
next week I'll try to sketch the Tcl KV API, because TBH I'm not 100% happy with naming by Synadia. E.g. in Go they have |
Thanks for reviewing changes! These are very good suggestions - I will try to implement them this or next week. We have no production code yet, so there is no problem in changing things now - as you said it is a lot better to do it now, than later change the API. |
I've drafted the KV API here , looking at the ADR, nats.py and nats.go. I think, for you it will be more or less trivial rearrangement. By the way, you can keep the new methods for bucket management in Removing And regarding the The The biggest question for me is the watcher. For my daily job I've just implemented the Tcl watcher based on nats.c, and I really didn't like that the C API doesn't provide a callback, like I expected - after all, this is just a push subscriber. nats.go uses channels, and I don't know Go, but probably it is non-blocking API, and same for nats.py - For this pure Tcl client it might be enough to have So, while doing it I realized that the watcher is basically a local cache of the whole bucket or its portion, depending on a wildcard that you're watching. So, I've added extra methods to get values directly from the watcher - without a roundtrip to NATS, like kv_bucket does. No official clients have it. It's just a small convenience, because a user can make such array themselves, while receiving updates from the watcher. So, it would be useful to understand your use case for watchers. Will you use it only to receive new values with |
For example, JS My general approach is lazy: if something doesn't HAVE to be coded - don't code it :) |
I have rearranged most of the API according to proposed changes. You can check it now if it fits other clients or there are any other changes to be made. Also some deadline occurred - it would be nice if this PR was merged until next Thursday (24.08) (I have two week vacation so it would be nice to close it till then). Referring to creating streams and KVs from JSON ( As it comes to But you are right that if user wants to use it as
So there are my thoughts for now - I will probably start implementing it tomorrow unless you think it is not necessary for now. If you have any suggestions let me know. |
Thanks! I looked very briefly, it's much better now 👍 |
and don't worry about the watcher, your input is enough for me to do it :) |
P.S. you are really fast! 😮 |
Thanks! 😄 |
The |
@piotr-karas also could you tell your opinion on #24 ? It matters if you replicate streams and need good performance. |
oh no I've noticed this only now by accident https://nats.io/blog/preview-release-new-jetstream-client-api/ @piotr-karas I think, this is something to discuss in your team btw the post was written by your old colleague Tomasz :) |
Regarding Regarding Oh, and sorry for such a long response time - as I mentioned I was on holiday |
thanks, I'll open a separate issue for I can imagine it's not easy for you to update dependencies like nats-tcl in a restricted factory environment, so need to plan ahead. I'm still working on KV, making good progress. |
Hi!
Here is huge pull request with full support for key-value store based on JetStreams.
There is a lot of code, but core nats and jet_streams are almost untouched - beside some changes to
_dict2json
which now enablesobjects
to be formatted - it was necessary in order to enablemirror
andsources
options inadd_stream
method (which were necessary forkv
to enable mirroring).Most of this code was based on behaviour and API of
nats-cli
client with some small changes. I also looked at ADR which are linked in source code. There is also documentation written with examples as well as a lot of tests.Hope you find it interesting :)
Kindest regards