feat(wadm)!: Switch wadm events to limits-based stream and create a new sourcing stream for EventConsumer

[joonas]

Feature or Problem

This switches from our custom mirroring code to NATS Stream Sourcing.

It also changes the wadm events to follow the pattern established for wasmbus events where they are now published under wadm.evt.<lattice-id>.<event-type> to make the sourcing consistent.

Finally, it also includes code to clean up any existing wadm_(multitenant)_mirror streams on startup so that we don't end up with conflicting subjects (which would prevent the new streams from being created in the first place).

This effectively accomplishes the same thing as #295 using slightly different strategy, while also making the wadm_events (and thus turning wadm.evt.<lattice-id>.<event-type> into a stream-based API).

Related Issues

Release Information

Consumer Impact

Testing

Unit Test(s)

Acceptance or Integration

Manual Verification

[thomastaylor312]

Is this going to cause issues if someone has created their own stream with their requirements (such as more replicas or other things) or if something else is already consuming these downstream? I'd rather just sort the list of subjects and compare that way if possible

[joonas]

Is this going to cause issues if someone has created their own stream with their requirements (such as more replicas or other things) or if something else is already consuming these downstream?

For downstream this will not create issues, I tested this locally and the net-net on this is that once the Stream is re-created, NATS will continue where it left off, meaning that the only messages that could be lost are the ones that are in the stream exactly at the moment when it's being re-created.

As for letting someone (pre-)create this stream but with different configuration, I'm not sure how easy that would be to support. It feels like if that's the path we want to go down, a better strategy would be to explicitly expose the knobs for configuring the various parts of these stream as flags/environment variables that could be passed down to code that's (re)creating the streams.

WDYT?

[thomastaylor312]

We already let people specify buckets, so letting them specify streams seems ok. My principal concern is with the stream capturing all the wasmbus.evt stuff since that is the one I consider most likely an advanced wasmCloud user might want to configure to have more retention or replicas. This is fine for now though