Multi-master syndic is not documented in all examples hot-hot, multimaster and multi syndic w multimaster

[whiteinge]

As the title says. I can't help but feel this is somehow @jacksontj's fault! ;-)

[jacksontj]

That's fair... ;) I can probably get that put together over the next week
or so
On Dec 4, 2014 2:43 PM, "Seth House" notifications@github.com wrote:

As the title says. I can't help but feel this is somehow @jacksontj
https://github.com/jacksontj's fault! ;-)

—
Reply to this email directly or view it on GitHub
#18751.

[whiteinge]

@jacksontj I'm happy to get that added too...but I don't know how.
Does syndic_master
just take a list like master does or is there more to it?

On Thu Dec 04 2014 at 5:44:58 PM Thomas Jackson notifications@github.com
wrote:

That's fair... ;) I can probably get that put together over the next week
or so
On Dec 4, 2014 2:43 PM, "Seth House" notifications@github.com wrote:

As the title says. I can't help but feel this is somehow @jacksontj
https://github.com/jacksontj's fault! ;-)

—
Reply to this email directly or view it on GitHub
#18751.

—
Reply to this email directly or view it on GitHub
#18751 (comment).

[UtahDave]

I'm quite interested in this, as well!

[whiteinge]

I misunderstood this addition. I thought it was to have a single syndic connected to two hot master-masters (e.g., how vanilla multi-master works). @cro set me straight. Would still be nice to have docs, but I see now there are no new config options for this.

[jacksontj]

@whiteinge Yea, this is basically just a way for you to list out a bunch of "syndic_master"s. There are a few caveats in the current implementation (which is what I'd like to document), but as you said there are no new config options.

[whiteinge]

Thank you for the clarification!

On Fri, Dec 5, 2014, 9:19 AM Thomas Jackson notifications@github.com
wrote:

@whiteinge https://github.com/whiteinge Yea, this is basically just a
way for you to list out a bunch of "syndic_master"s. There are a few
caveats in the current implementation (which is what I'd like to document),
but as you said there are no new config options.

—
Reply to this email directly or view it on GitHub
#18751 (comment).

[UtahDave]

@whiteinge would you mind clarifying your clarification?

[whiteinge]

@UtahDave A syndic cannot currently have a connection to multiple masters. This ticket is for the following scenario which provides more redundancy than just a master -> syndic-master connection but does not provide full HA.

Given Master A and Master B at main HQ, and syndic-masters a & b in one data center and syndic masters c & d in another data center:

• syndic-masters a & b are configured as multi-masters and minions e-k under them are connected to both as normal multi-master
• syndic-masters c & d are configured as multi-masters and minions l-r under them are connected to both as normal multi-master
• syndic-master a is connected to Master A; syndic-master c is connected to Master A.
• syndic-master b is connected to Master B; syndic-master d is connected to Master B.

If either Master A or Master B goes down, you still have a connection left to at least one syndic in each data center. But if the wrong syndic goes down at the same time as the wrong master, then you can still lose connection to a data center.

[jacksontj]

To clarify more, since the addition of MultiSyndic you can put a list (instead of a single string) as the master list for a syndic-- and the syndic will connect to all of them-- similar to multiminion. The code is a bit messy, but it does work ;)

[cachedout]

We need to completely overhaul the topology section of the docs to explain Salt's deployment options:

http://docs.saltstack.com/en/latest/topics/development/topology.html

[jacksontj]

+1

[whiteinge]

Holy cow, @jacksontj, that's awesome! I was previously told multi-connections had specifically not been added.

Definite +1 to getting MultiSyndic documented and also adding this to the topology doc.

[jacksontj]

The only reason I haven't pushed it a lot is that some of the failure cases are a bit odd... I might just go back and thread it to work around them. Basically in a failure scenario (one master unavailable) you have to let zmq attempt to re-connect and we have to re-auth through it-- which means that the syndic will lock up for however long that connect timeout is. In general slow is better than unavailable-- but it can be worked around, I'll try to get back to that after our 2014.7 release internally

[whiteinge]

Odd failure cases with ZeroMQ, you say? Hm. ;-)

Thanks for mentioning that. I can think of one or two large sites just offhand that will be interested in getting this set up.

[jacksontj]

We are one of them ;)

On Wed, Feb 18, 2015 at 8:33 AM, Seth House notifications@github.com
wrote:

Odd failure cases with ZeroMQ, you say? Hm. ;-)

Thanks for mentioning that. I can think of one or two large sites just
offhand that will be interested in getting this set up.

—
Reply to this email directly or view it on GitHub
#18751 (comment).

[Colstuwjx]

Hi, @jacksontj, I setup 4 syndic-master that the one would connect to the other three, just like #6 arch you introduced:

\#syndic 01 config:
syndic_master:
  -  [ syndic_02_ip ]
  -  [ syndic_03_ip ]
  -  [ syndic_04_ip ]

After all setup, I fired one job on syndic01, and I actually find out a duplicate of event loop problem, syndic 02\03\04 will duplicate to forward job and re-publish to its localmaster. so it will be:

\# on syndic_master 01
salt 'salt-test*' cmd.run 'echo 1TRY >> /tmp/run.log';

\# and all minions duplicate to add 1TRY to run.log

I checked out the MultiSyndic code, it seems that you only not forward event when the master_id match the origin one, but it didn't work in >2 syndics env.

After I check the 2014.7.5 & 2015.5.0 version, the setup cluster still doesn't work, still loop、loop and loop out...

Help~~~~~

[mimianddaniel]

so multisyndic is not working as it suppose to?

[Colstuwjx]

Yeah, Multi-Master with Multi-Syndic ( cluster mode ) didn't work in my test case, but Multi-Syndic does.

[mimianddaniel]

@jacksontj
could you enlighten us?

Multi-syndic setup has missing return issues from master of master for me.
While runing multi-syndics on the syndic nodes, I see missing returns often against wildcard matches against (only 300 minions) from the master node. Without multisyndic, no issues with the returns from the master node.

i have master_id set at the both master nodes and have a list of masters listed as syndic_master on the syndic nodes. Am i missing anything??

Seems like increasing the number of timeout fixes this problem - I did noticed that all the ret events arrive accordingly in each master of masters, but not all returns are showing up in --out-file but if i look up the JID afterwards, it has correct number of returns.

[jinglejengel]

I'm having a super interesting issue with this set up actually.

I have the following servers:

master1.domain.com
master2.domain.com
syndic1.domain.com
syndic2.domain.com
minion1.domain.com
minion2.domain.com

When I run a test.ping on master1 I get the following:

[root@master1 ~]# salt '*' test.ping
syndic2.domain.com:
    Minion did not return. [No response]
syndic1.domain.com:
    Minion did not return. [No response]

Yet when I run the same on master2, I get the following:

[root@master2 ~]# salt '*' test.ping
minion1.domain.com:
    True
minion2.domain.com:
    True
syndic2.domain.com:
    Minion did not return. [No response]

You'll notice on master2, syndic2 isn't responding, despite the key being accepted and transferred correctly. The syndic logs keep having the same message repeating with regards to master1:

[INFO    ] Syndic still not connected to master1.domain.com

When I start the salt-syndics from scratch, keys are exchanged on both master1 and master2 however?

[root@master1 ~]# salt-key -L
Accepted Keys:
syndic1.domain.com
syndic2.domain.com
Denied Keys:
Unaccepted Keys:
Rejected Keys:

[root@master2 ~]# salt-key -L
Accepted Keys:
syndic1.domain.com
syndic2.domain.com
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Digging down, I found that when I started the minions, syndic1 has the keys sent to it, but syndic2 does not, despite the configuration being the same on both minions with regards to having the syndic pub keys on them, here's what my salt-keys -L looks like:

syndic1

[root@syndic1 ~]# salt-key -L
Accepted Keys:
minion1.domain.com
minion2.domain.com
Denied Keys:
Unaccepted Keys:
Rejected Keys:

syndic2

[root@syndic2 ~]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Here's what i have for my config files on the syndics for the masters:

syndic_master:
  - master1.domain.com
  - master2.domain.com

And here's what I have on the minions for the syndics:

master:
   - syndic1.domain.com
   - syndic2.domain.com

master_type: failover

master_alive_interval: 30

Going a bit deeper, I can definitely confirm on the network level all instances should be able to talk each other:

Here's an example telnet from syndic2->master1

[root@syndic2 ~]# !telnet
telnet master1.domain.com 4506
Trying 10.1.4.231...
Connected to master1.domain.com.
Escape character is '^]'.

And for those counting at home master1->syndic2

[root@master1 ~]# telnet syndic1.domain.com 4506
Trying 10.10.2.217...
Connected to syndic1.domain.com.
Escape character is '^]'.

I know it's a lot of information so I'm happy to confirm anything needed from above, but I REALLY need a setup like this to work with syndics and masters for HA if I'm going to further proceed with SaltStack.

[mimianddaniel]

Make sure to have master_id set on the mom if you have not yet. With
multisyndic setup I see inconsistent showing of returns, returns arrived
properly but do not show up promptly at times.
On Sep 1, 2015 3:23 PM, "Joe Engel" notifications@github.com wrote:

I'm having a super interesting issue with this set up actually.

I have the following servers:

master1.domain.commaster2.domain.comsyndic1.domain.comsyndic2.domain.comminion1.domain.comminion2.domain.com

When I run a test.ping on master1 I get the following:

[root@master1 ~]# salt '*' test.pingsyndic2.domain.com:
Minion did not return. [No response]syndic1.domain.com:
Minion did not return. [No response]

Yet when I run the same on master2, I get the following:

[root@master2 ~]# salt '*' test.pingminion1.domain.com:
Trueminion2.domain.com:
Truesyndic2.domain.com:
Minion did not return. [No response]

You'll notice on master2, syndic2 isn't responding, despite the key being
accepted and transferred correctly. The syndic logs keep having the same
message repeating with regards to master1:

[INFO ] Syndic still not connected to master1.domain.com

When I start the salt-syndics from scratch, keys are exchanged on both
master1 and master2 however?

[root@master1 ~]# salt-key -L
Accepted Keys:syndic1.domain.comsyndic2.domain.com
Denied Keys:
Unaccepted Keys:
Rejected Keys:

[root@master2 ~]# salt-key -L
Accepted Keys:syndic1.saltysentinel.comsyndic2.saltysentinel.com
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Digging down, I found that when I started the minions, syndic1 has the
keys sent to it, but syndic2 does not, despite the configuration being the
same on both minions with regards to having the syndic pub keys on them,
here's what my salt-keys -L looks like:

syndic1

[root@syndic1 ~]# salt-key -L
Accepted Keys:minion1.domain.comminion2.domain.com
Denied Keys:
Unaccepted Keys:
Rejected Keys:

syndic2

[root@syndic2 ~]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Here's what i have for my config files on the syndics for the masters:

syndic_master:

• master1.domain.com
• master2.domain.com

And here's what I have on the minions for the syndics:

master:

• syndic1.saltysentinel.com
• syndic2.saltysentinel.com

master_type: failover

master_alive_interval: 30

Going a bit deeper, I can definitely confirm on the network level all
instances should be able to talk each other:

Here's an example telnet from syndic2->master1

[root@syndic2 ~]# !telnet
telnet master1.domain.com 4506
Trying 10.1.4.231...
Connected to master1.domain.com.
Escape character is '^]'.

And for those counting at home master1->syndic2

[root@master1 ~]# telnet syndic1.domain.com 4506
Trying 10.10.2.217...
Connected to syndic1.domain.com.
Escape character is '^]'.

I know it's a lot of information so I'm happy to confirm anything needed
from above, but I REALLY need a setup like this to work with syndics and
masters for HA if I'm going to further proceed with SaltStack.

—
Reply to this email directly or view it on GitHub
#18751 (comment).

[jinglejengel]

@mimianddaniel thanks for the suggestion, just gave that a go and still in the same spot. That was something I meant to have set anyways so at least I've got it covered now ;)

[jinglejengel]

Been over a month since any activity on this thread, just wondering if anyone from SaltStack has been able to plug at this.

[rallytime]

Doesn't look like we've had time to get to documenting this yet.

@jacobhammons Can you have a look here when you get a moment?

[waynew]

Had unexpected issues pop up, this one has to slip to the next release

[anilsil]

Closing as Won't fix. Please follow Syndic deprecation work https://github.com/orgs/saltstack/projects/42 due for 3007.0

[Oloremo]

@anilsil I get 404 for https://github.com/orgs/saltstack/projects/42
Any doc I can read about that?

[anilsil]

Still in WIP but you can refer this #64941