add ability to deploy an observable nats without stan stack

[jwtodd]

objective

deploy a contemporary nats+surveyor stack

see: https://docs.google.com/document/d/1Um5bgu6CY1MapEb-VOy5OAECDkrr9RsgijjGiqCQTTQ/edit

not included in this patch are the following:

• nats auto-scaling

multi-tenancy

further going with this approach we will want to deploy the stans pods outside the central nats namespace, optimally collocated within the mds pod.

alternatively we can consider centralizing the stan cluster, leaving the [tenant-id] message prefix in place but this feels suboptimal to me (unnecessarily co-mingling tenant infrastructure).

profiles / tunables

when running locally it is encouraged to specify one of the non-default local profiles in that they are tuned to tighten the streaming message durability policies tuned for resource constrained environments.

on this note, the message default message retention policy for non-local environments is:

• --max_age=72h
• --max_bytes=10GB

note: best guessing these values which we can/will adjust as we gain more operational context

additionally included in this patch is nats-authentication which should work by default storing nats-credentials-information in the ~/.nkeys and ~/.nsc directories by default. this is overridable such that we can leverage shared operational credentials, namely from the cloud-ops repository, with an mds override, eg:

% MDS_CREDENTIALS=[path-to-credentials] ./bin/mdsctl ...

observability

nats provides surveyor which is the expected prometheus+grafana stack. as such we will gain more operational insight into nats.

deploy local

% ./bin/mdsctl -p local bootstrap
% ./bin/mdsctl build
% ./bin/mdsctl -p local install:surveyor
% ./bin/mdsctl -p processors -c namespace-nats=mds install:mds
% ./bin/mdsctl -c namespace-nats-box=mds install:natsbox

deploy remote

% ./bin/mdsctl bootstrap
% ./bin/mdsctl build
% ./bin/mdsctl -c image-repository=[account-id].dkr.ecr.us-west-2.amazonaws.com install:images
% ./bin/mdsctl -c sets-mds+=registry=[account-id].dkr.ecr.us-west-2.amazonaws.com install:mds

verify

% kubectl -n mds exec -it nats-box -- /usr/local/bin/nats-sub \
   --server nats-cluster-mgmt.nats.svc.cluster.local \
  --cluster mds test
% kubectl -n mds exec -it nats-box -- /usr/local/bin/nats-pub \
  --server nats-cluster-mgmt.nats.svc.cluster.local \
  --cluster mds test hello
# [todo] run simulators

cleanup

% ./bin/mdsctl uninstall:surveyor,natsbox,nats,mds uninstall

[avatarneil]

Few comments, once resolved and I've had a chance to test the changes on my machine I'll approve

[jwtodd]

@avatarneil @invertigo @ysingh-lacuna what say you?

[avatarneil]

./bin/mdsctl bootstrap modifies the package.json and installs a new dependency mochawesome on my machine @jwtodd

[avatarneil]

Also looks like the istio-system container is being created during bootstrap but not populated, I need to run a subsequent ./bin/mdsctl reinstall:istio to bring it up.

[avatarneil]

Also noticing that the redis and postgresql pods get tagged pretty strangely... mds-mds-postgresql-0

[avatarneil]

Besides those things, NATS seems to be working E2E, so good besides that. Will approve once the previous comment are addressed.

[jwtodd]

@avatarneil thx for the comments ... will go through them shortly. ball is in my court(tm)

[jwtodd]

Also noticing that the redis and postgresql pods get tagged pretty strangely... mds-mds-postgresql-0

this is to support service-management via helm-release-names vs pod/namespace existence

[jwtodd]

Also looks like the istio-system container is being created during bootstrap but not populated, I need to run a subsequent ./bin/mdsctl reinstall:istio to bring it up.

./bin/mdsctl bootstrap brought up istio for me ... as a follow-on patch we will add in verification [ cc @ysingh-lacuna ]

q: did you happen to capture the error? was it a timeout?

[jwtodd]

./bin/mdsctl bootstrap modifies the package.json and installs a new dependency mochawesome on my machine @jwtodd

this has been in mdsctl:bootstrap() for awhile now to support the js smoke-tests