Unable to create new domain

[wozuo]

Describe the bug
I'm trying to create a new Cadence domain with:

docker run --network=host --rm ubercadence/cli:master --domain sample-domain domain register

and I get the following error:

Error: Register Domain operation failed.
Error Details: BadRequestError{Message: Invalid cluster name: }
('export CADENCE_CLI_SHOW_STACKS=1' to see stack traces)

To Reproduce
Is the issue reproducible?
Yes

Steps to reproduce the behavior:
Start new Cadence Docker setup with:

• ubercadence/web:v3.22.2
• ubercadence/server:0.18.0-auto-setup
• graphiteapp/graphite-statsd
• cassandra:3.11.9

and then run the domain creation command mentioned above.

Expected behavior
Domain is created.

Hope somebody can help me out. I only started getting these errors after upgrading to Go 1.16 and the versions of the Cadence Docker components.

[DannyHinshaw]

I ran into this yesterday as well, what will get you to a working state (temporarily) is to add the cluster name to your command, ie:

docker run --network=host --rm ubercadence/cli:master --domain sample-domain domain register --active_cluster active

I had to dig through the Cadence Slack channel to figure that one out.

Now I say, "temporarily" because as of now you're not going to be able to use Go v1.16 because of this bug: uber-go/cadence-client#1056

So, yea...

[wozuo]

Hey thanks a lot for letting me know about this, it works! I think the documentation needs upgrading.

Regarding the 1.16 .gen issue, I'm using this branch to get around that until it is merged.

[richard-rance]

I'm not familiar with the internals but think there is a bug in this change. The default value for that flag is now an empty string instead of a nil pointer.

This flag should not be required for if you don't have any data center replication setup. Even if you do there should probably be some smart defaults for picking which cluster is initially active.

[vytautas-karpavicius]

There are defaults that picks active cluster. However logic changed from checking nil pointer to checking for empty string. You are using older server release (0.18.0) which still checks for nil, while your cli (latest master) already passes empty string. This will get resolved with upcoming 0.20 release. For now please specify --active_cluster flag.

[longquanzheng]

This bug is fixed in 0.18.2 .

[dragosrosculete]

This is not fixed. Unable to register a domain using the latest version. If I pass the --ac active argument it complains it doesn't exist. If I don't pass it , the domain is registered in secondary not in active cluster...

[longquanzheng]

This is not fixed. Unable to register a domain using the latest version. If I pass the --ac active argument it complains it doesn't exist. If I don't pass it , the domain is registered in secondary not in active cluster...

I think this is a different issue— it’s because of your config changes. Maybe not directly from you. I know Cadence helmchart changed it recently:

Can you show me your clusterMetadata config?

cadence/config/development_active.yaml

Line 71 in bec009a

clusterMetadata:

By default domain will be registered under primary(master) cluster.

and can you show me the domain info by “domain describe” command?

Also what is the issue of using secondary cluster?

[dragosrosculete]

Hi @longquanzheng , thank you for answering .
I am installing it using helm from here : https://github.com/banzaicloud/banzai-charts/tree/master/cadence

I don't have the "clusterMetadata" you've mentioned in the helm values....
Not sure why by default they are registered in secondary cluster . I even did a clean install and does the same

Describing a domain that was registered gives me the following:

Name: campaigns
UUID: bc2f1b63-9388-4a42-bea1-80b2b63b21a8
Description:
OwnerEmail:
DomainData: map[]
Status: REGISTERED
RetentionDays: 1
EmitMetrics: true
ActiveClusterName: secondary
Clusters: secondary
HistoryArchiveStatus: DISABLED
VisbilityArchieveStatus: DISABLED

• Also what is the issue of using secondary cluster?
  Well , if I do an upgrade I end up with things on active cluster ( according to information in the DB , or domain describe) and new domains end up on secondary cluster. I am not sure this will be ok.
  Only if I do a clean install I will have them all in the secondary cluster.
  I've also read that workflow only work on active cluster, but I might be wrong ..

[longquanzheng]

https://github.com/banzaicloud/banzai-charts/blob/031c49f4f1f542f60a88bb9f5fc56c22a518d773/cadence/templates/server-configmap.yaml#L158

If you pass this env as true then the cluster will be “Primary”

So it’s not a problem at all that a cluster name is not “active”. Active is a bad word to describe cluster name as we want to use “active”/ passive(standby) to describe if a domain is processing write request in that cluster under XDC (replication) mode.

Cadence repo also update the config example recently to use “primary/secondary “ instead.

cadence/docker/config_template.yaml

Line 208 in fbc79f9

currentClusterName: "secondary"

[dragosrosculete]

Cool, thank you. Adding the following into my helm cadence values made it work on a fresh install .

Env:
 PRIMARY: true

The cadence stuff is created under cluster active, and the domain registration without passing addtional parameters is also created under active.
I am afraid in the future, as you've mentioned that active will be renamed into primary, we will start again...

[longquanzheng]

Cool, thank you. Adding the following into my helm cadence values made it work on a fresh install .

Env:
 PRIMARY: true

The cadence stuff is created under cluster active, and the domain registration without passing addtional parameters is also created under active.
I am afraid in the future, as you've mentioned that active will be renamed into primary, we will start again...

Where do you get this PRIMARY?

I feel like something is off:
By default, the chart defines the clusterMetadata like this:
by default IS_NOT_PRIMARY is not set so it should be

masterClusterName: "primary"
currentClusterName: "primary"

https://github.com/banzaicloud/banzai-charts/blob/031c49f4f1f542f60a88bb9f5fc56c22a518d773/cadence/templates/server-configmap.yaml#L158

Did I miss something? Or are in on a different branch or commit?

[longquanzheng]

Created an issue to improve it: banzaicloud/banzai-charts#1274

[dragosrosculete]

I think it had to do with this part

    {{- if `{{ .Env.IS_NOT_PRIMARY }}` }}
      currentClusterName: "secondary"
    {{- else }}
      currentClusterName: "primary"
    {{- end }}

So when I specifid the the Env.PRIMARY , the current cluster name was set to primary so it resulted in active ?
I am not really sure to be honest but the result was that evertyhing was under "active" afterwards ...
And registering a domain without passing any extra parameters like global domain or cluster ended up on the active cluster .

[longquanzheng]

@dragosrosculete yeah I believe that's a bug then. Another user also reported the same thing that by default the template would apply to "secondary" as currentClusterName, which won't allow to create domain. Because only "masterCluster" can register domains.

I reported a bug here: banzaicloud/banzai-charts#1275