Word Count Modernization

[creese]

This PR updates the "Word Count" example.

Highlights:

• Removes the ad hoc start/stop functions in favor of James Reeves's Integrant.
• Adds a dev alias.
• Adds separate Integrant configs for dev and prod.
• Adds a topic metadata "fake" for use in development.
• Adds a namespace with REPL helpers to create topics, and produce and consume records.
• Copies some parsing logic from the Topology Grapher.

When writing code, you can treat the topic metadata fake just like a map. When used with a "getter", it returns the topic metadata for the name supplied, EDN serdes for the key and value, and a partition count of one.

(def topic-metadata (new-fake-topic-metadata))
(:foo topic-metadata)
;; => {:topic-name "foo", 
       :partition-count 1,
       :replication-factor 1,
       :key-serde  #object[org.apache.kafka.common.serialization.Serdes ... ],
       :value-serde #object[org.apache.kafka.common.serialization.Serdes ... ]}

When in development, the topics are inferred from the topology. The dev workflow is largely unchanged. Simply jack in and type:

(reset)
;; => :reloading (word-count user)
;; => :resumed
(publish (:input topic-metadata) nil "all streams lead to kafka")
(publish (:input topic-metadata) nil "hello kafka streams")
(get-keyvals (:output topic-metadata))
;; => (["all" 1] ["streams" 1] ["lead" 1] ["to" 1] ["kafka" 1] ["hello" 1] ["kafka" 2] ["streams" 2])

The user namespace is part of the dev alias and is not loaded automatically. To apply the dev alias, add the following to your Emacs config:

(setq cider-clojure-cli-global-options "-A:server:client:dev")

or type C-u -A:server:client:dev when jacking in.

[creese]

I commented out this namespace while development. I plan to put this back.

[codecov]

Codecov Report

Merging #191 into master will decrease coverage by 2.68%.
The diff coverage is 18.69%.

@@            Coverage Diff             @@
##           master     #191      +/-   ##
==========================================
- Coverage   81.43%   78.74%   -2.69%     
==========================================
  Files          40       41       +1     
  Lines        2391     2498     +107     
  Branches      149      149              
==========================================
+ Hits         1947     1967      +20     
- Misses        295      382      +87     
  Partials      149      149

Impacted Files	Coverage Δ
src/jackdaw/streams/describe.clj	18.69% <18.69%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f404699...8c9e685. Read the comment docs.

[gphilipp]

I think that we should use a string serde for the key to make it easier to join with other topics. See https://github.com/FundingCircle/jackdaw-repl/commit/1ddd902e3ec325c862bdb2ef06f01050255b665e

[creese]

Keep in mind this is just for prototyping. We shouldn't use this or EDN keys in production.

[creese]

I was thinking we could add a new arity that takes an argument so the user can set their own default. Would that address your use case?

[gphilipp]

I started with that, and then separated out the cleanup or resources because sometimes you just want to stop the app but not delete everything so you can inspect the topics and state stores. https://github.com/FundingCircle/jackdaw-repl/commit/0ba5f6f70ab2a08f8f4d11819c5540d3e093574c

[creese]

Seems like you'd want a full reset to be able to start clean. Could we halt just the app with a different config structure?

[DaveWM]

I think we can just use a single instance of FakeTopicMetadata, so we can drop this function and update topic-metadata to be:

(def topic-metadata (FakeTopicMetadata.))

Then we can just use topic-metadata where new-fake-topic-metadata is currently used.

[creese]

Could do. Does this still make sense if we add an arity to allow the user to set their own default?

[DaveWM]

I'd move this logic to where the config map is instantiated, and get rid of this init-key method

[DaveWM]

When using this function from the repl, I've always needed to call seek-to-beginning-eager on the consumer before using it to consume messages. This may just be my setup though 🤷‍♂️

[creese]

This config sets a random UUID for the consumer group. It will always start from the beginning. If we were to stop doing that, I think we would need to do as you suggest. I'm not sure which is preferable.

[99-not-out]

It might be nice not to print every line, but some other (smaller) progress indicator. I use a fast terminal and it took a while - in an emacs buffer I imagine its a bit painful ...

[99-not-out]

Nice - works for me with the tweak for clj -A:dev in the CLI based instructions.