Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LDAP Missing Background Sync Options #23423

Open
HammerAdmin86 opened this issue Oct 9, 2021 · 27 comments
Open

LDAP Missing Background Sync Options #23423

HammerAdmin86 opened this issue Oct 9, 2021 · 27 comments

Comments

@HammerAdmin86
Copy link

HammerAdmin86 commented Oct 9, 2021

Deployment
Version
4.0.1

Apps Engine Version
1.28.0

Node Version
v12.22.1

MongoDB
5.0.3 / wiredTiger (oplog Enabled)

Commit Details
HEAD: (accdbc6)
Branch: HEAD

Ubuntu 20.04.03 (manual install)

In the web admin interface LDAP is missing to fields to configure background sync. LDAP is also missing the buttons to trigger a manual sync.

@HammerAdmin86
Copy link
Author

HammerAdmin86 commented Oct 9, 2021

rocketchat-ldap-001

The test LDAP connection works though.

It seems in the last 1-2 weeks the latest version has had the LDAP menus reshuffled. I have searched over all of the admin panels for the background sync options and trigger background sync button.

@HammerAdmin86
Copy link
Author

HammerAdmin86 commented Oct 9, 2021

Also, the documentation seems confusing.

One page seems to indicate that background sync is a non-enterprise feature (only Enhanced Sync is for EE):
https://docs.rocket.chat/guides/administration/settings/ldap/settings

Another docs page seems to indicate that background sync is for both CE and EE.
https://docs.rocket.chat/quick-start/identity-management-ee-vs-ce

I definitely had background sync working in CE last week.

@reetp
Copy link

reetp commented Oct 9, 2021

Manual sync should still be there, unless the goalposts have been moved, but it was promised that nothing else would be removed.

The rest is explained here.

https://forums.rocket.chat/t/upcoming-changes-to-identity-management-integrations/11994/95

@HammerAdmin86
Copy link
Author

HammerAdmin86 commented Oct 9, 2021

"...The community edition LDAP feature will allow workspaces to connect to an LDAP server and import user names and identifiers,..."

Seems they removed the manual sync button by accident then? I can deal without the background sync. But manually triggering a sync seems in scope.

There is no way currently to trigger a manual sync from the admin panel.

Therefore, this is still a bug.

@magnetic5355
Copy link

Stay on 3.18.1 and hope for a good fork while the software continues to be stripped. 4.01 only introduced bugs from hastily removing features users have relied on for years.

@reetp
Copy link

reetp commented Oct 11, 2021

"...The community edition LDAP feature will allow workspaces to connect to an LDAP server and import user names and identifiers,..."

Seems they removed the manual sync button by accident then? I can deal without the background sync. But manually triggering a sync seems in scope.

There is no way currently to trigger a manual sync from the admin panel.

Did you notice if this was in 4.0.0?

I have no idea if this was by accident or design but it should not have been removed.

We were promised that there would be no other features removed beyond group features, and as detailed in the forums posts and here in the docs. This clearly shows it should still be there:

https://docs.rocket.chat/quick-start/identity-management-ee-vs-ce

Screenshot_2021-10-11_14-49-11

@reetp
Copy link

reetp commented Oct 12, 2021

Probably related...

#23296 (comment)

@marcusquinn
Copy link

Erm, how about putting back the stolen Sync button please?

Open-source, doesn't mean open-season to sabotage something that's working fine, for the sake of now holding it to ransom for money.

I can think of many new developments that would be payment-worthy, this sort of stunt just makes me want to invest in a competitor on a pure moral basis.

@debdutdeb
Copy link
Member

Hi everyone - sorry for the late response. To clarify the situation, basic user data sync is still in CE. Background sync along with manual sync in EE only.

So in CE users are updated on demand (when the login happens).

Can someone please confirm whether basic user data sync is broken on 4.x or not?

@talheim-it
Copy link

talheim-it commented Oct 27, 2021

@debdutdeb okay thanks for clarification, moving to zulip my customers and me.

Thank you!

https://zulip.com/

@reetp
Copy link

reetp commented Oct 28, 2021

Debdut,

Can someone please confirm whether basic user data sync is broken on 4.x or not?

Yes because it doesn't background sync as per this bug. Which is not a bug per se but a feature grab, and not what was agreed.

This is very poor as per my comments here:

https://open.rocket.chat/channel/support?msg=PtoPEDDWt6v5QxTgn

@reetp
Copy link

reetp commented Oct 28, 2021

@debdutdeb okay thanks for clarification, moving to zulip my customers and me.

I don't blame you - I don't think you are alone.

@talheim-it
Copy link

@reetp noone here to blame, I just think that removing the sync button and telling us "That is part of the EE edition" when they first said only background sync is in EE makes it hard to trust.

And since RC gave my data to others without my permission my trust into them is gone.

@reetp
Copy link

reetp commented Oct 28, 2021

@reetp noone here to blame

Ha..... well yes, and no.

I am well aware as I worked at Rocket as Community Manager during the period this was discussed - Debdut works at Rocket and is one of my old colleagues - and I was closely involved with what was agreed.

I got the original plan changed. CE was meant to have basic user sync the same as Gitlab - which includes basic background sync.

The only features that should have gone to EE were advanced things such as group/team management, custom options etc. I had no reason to believe that sync was one of them.

The problem is Rocket made a rod for their own back due to the way it runs its own user database.

Because of that the LDAP DB and the Rocket DB need to be kept in sync (which gitlab does) or you can get a situation where an admin changes a password that is is not synced with the Rocket user DB.

If the connection is broken then the system falls back and the old password is still there.

I am sure there are other issues that I haven't thought of.

There are reasons to remove the background sync that have not been publicly explained.

This whole issue is known and understood internally all the way to the top. Any claims to the contrary are nonsense, and I am disappointed in my former colleagues that no one internally has come out and just been honest about it. They are just hunkering down and hoping the storm will blow over.

I will be moving my instances elsewhere - probably Matrix - as I now have zero confidence that, despite vague promises, they won't remove something else next year when they need more sales.

@talheim-it
Copy link

talheim-it commented Oct 28, 2021

@reetp noone here to blame

Ha..... well yes, and no.

I am well aware as I worked at Rocket as Community Manager during the period this was discussed - Debdut works at Rocket and is one of my old colleagues - and I was closely involved with what was agreed.

I got the original plan changed. CE was meant to have basic user sync the same as Gitlab - which includes basic background sync.

The only features that should have gone to EE were advanced things such as group/team management, custom options etc. I had no reason to believe that sync was one of them.

The problem is Rocket made a rod for their own back due to the way it runs its own user database.

Because of that the LDAP DB and the Rocket DB need to be kept in sync (which gitlab does) or you can get a situation where an admin changes a password that is is not synced with the Rocket user DB.

If the connection is broken then the system falls back and the old password is still there.

I am sure there are other issues that I haven't thought of.

There are reasons to remove the background sync that have not been publicly explained.

This whole issue is known and understood internally all the way to the top. Any claims to the contrary are nonsense, and I am disappointed in my former colleagues that no one internally has come out and just been honest about it. They are just hunkering down and hoping the storm will blow over.

I will be moving my instances elsewhere - probably Matrix - as I now have zero confidence that, despite vague promises, they won't remove something else next year when they need more sales.

There are valid reasons but for an open source project changes should be discussed with the community and not just overruled by RC without information.

As I read the posts it was never meant first to remove the manual sync button, now the button is deactivated in CE which had to be communicated in anyway first.

I just think that this is bad handled from RC and to wait till the storm is over is the worst they can do.

We used Matrix in the past, I just not really happy since there was no Admin UI when I used it for synapse. But with Zulip I am completely happy now.

Just hope RC thinks again what Open Source means especially what the community gave back in the past to make it grow.

@rodrigok
Copy link
Member

The problem is Rocket made a rod for their own back due to the way it runs its own user database.

Because of that the LDAP DB and the Rocket DB need to be kept in sync (which gitlab does) or you can get a situation where an admin changes a password that is is not synced with the Rocket user DB.

If the connection is broken then the system falls back and the old password is still there.

@reetp we never synced passwords from LDAP, it's not even possible. What we do is cache it since the password pass through us, so there is no way to update it when it's updated on LDAP, and this option to cache is disabled by default. Even the settings to disable accounts based on account being disabled on LDAP was always a EE feature.

@rodrigok
Copy link
Member

@reetp you, probably, misunderstood what "background sync" was, it is the ability to sync all users in the background, either manually called or automatically via a recurrence configuration. "User data sync" is part of the background sync that happens for each user and part of the login flow that imports data from LDAP for the specific user that is loginin.

So, user-by-user data sync by demand (via login) was never meant to be EE only. The ability to sync and import all the users from LDAP in the background was what we pointed that would be improved and be EE only.

@reetp
Copy link

reetp commented Nov 1, 2021

The ability to sync and import all the users from LDAP in the background was what we pointed that would be improved and be EE only.

This is marketing nonsense. There is no improvement - you either sync or you do not. It seemed to work perfectly before and I see no evidence of an enhancement (oooohhhh - it's in typescript now).

And after all, it was you that said that you couldn't keep manual sync in CE because you were worried that people would use a cron job to sync. So this decision is something that you were personally involved in.

This was never meant to go to EE because it is basic user information, not groups or teams. And you know that if I still worked at Rocket I would have fought this. Enough has been stripped from CE for profit already.

Hey ho.

What goes next when you need to grab more sales next year?

Guess this may as well be closed as it is clear it is not going to be fixed, and we can all move on elsewhere.

@Gummikavalier
Copy link

Gummikavalier commented Nov 3, 2021

@rodrigok Time for honest talk from my part, no malice here. I've been following the scene all the way from IRC to XMPP to RC and its competition.

Things are in a brewing state.

RC CE version is something we currently advertise to anyone asking us. We endorse to get licenses when the need for EE is real and they have decided whether they choose between RC, Mattermost, Zulip or even the ever elusive Matrix.

But we won't greenlight EE version to our partners until a bunch of particular usability bugs have been killed (yeah, I'm talking about the lack of real names in lots of places in RC). We simply cannot, as those bugs are a major nuisance for the users that maintainers can currently counter only with the answer "Hey, as long as its free and paying for EE wouldn't help you there.".

This removal of one time LDAP sync button just pushes the workload of admins to do bootstrap creation of accounts by using ldapsearch and API instead (and then wiping the passwords required by API from the DB). I even used couple of days to learn AD proxying with OpenLDAP to get around the initial removal of filter options limitation that thankfully didn't materialize in RC 4.x.

We can always find a way to do these things the ugly way. If API gets crippled we have to do it by writing directly to the database. Although at that point I definitely will be running any of the competition. And worst of all for you, when we've used time to learn and set these workarounds up, there is even less reason to go for EE, because if the only difference usability wise is how the accounts are getting synced, we've already crossed that particular desert, and the solution is repeatable.

My personal advice to you is that as long as your EE product is not pitch perfect from an average joe point of view, you should not try to enforce move from CE to EE. The damage is only to your own reputation, and it just makes people angry and look for alternatives. Bugs can exist in other places, but the UX experience is what sells your product. (And unfortunately currently does not. But you are close.)

Ps. I'd like also to add to above that when we originally chose RC over Mattermost for our own use almost five years ago, RC was selected because it did two things correctly that Mattermost failed at: Working active presence information (green bulb) and the support for real names with nordic characters in them. They mattered most (pun intended). And yes, we are a paying customer today. But there could be others shouldn't the latter of these two had faltered during the following years of development.

@reetp
Copy link

reetp commented Nov 4, 2021

If API gets crippled

I would imagine that will occur somehow, sometime soon, so you can't get around the manual sync like you are doing. They definitely don't want that.

I guess there is a watching brief on the number of complaints, and how many of their targets they have squeezed to sales so far, before they decide to remove the API options - that will be tricky as that would affect EE too so I am not sure how they'll manage it, but the $$$ says they probably will somehow

@reetp
Copy link

reetp commented Nov 9, 2021

FYI

RocketChat/feature-requests#431 (comment)

This feature has been implemented in RC 4.x. Endpoint is /api/v1/ldap.syncNow (requires "ldap-enterprise" license).

@Gummikavalier
Copy link

I see. Thanks for info @reetp !

For the devs:

I'd much prefer that Enterprise license would be required for the new stuff like teams-functionality that becomes important later when a new instance gains more traction. Crippling the system at the root level like this effectively prevents adoption at the very earliest stage of the lifecycle of the RC instance.

The very first step to build "a demonstration system" for anybody is to bootstrap it with accounts, and people can start making channels and adding members to them immediately, instead of wondering why they cannot find anyone in the system yet (chicken-egg-problem). Then this originally supposedly just a demo system lives for a year or two, and becomes a tool you cannot live without. At that point you are supposed to start extracting money from them with more advanced features. Not the basic ones.

But this has been discussed in the release of RC 4.0, so I refrain myself from commenting more about it from now on. I just hope that my 20 years of experience of building IT-systems not only for us, but actual customers of ours, is considered worth something in input. You already have us. But we bring in potential customers apart from just ourselves.

@ankar84
Copy link

ankar84 commented Nov 30, 2021

Looks like manual sync is back in 4.2.0!
With one addition of asking of users password to start (security purposes I guess)
image
image

So, that and related to manual sync issues can be closed now

@solune
Copy link

solune commented Dec 9, 2021

In the CE edition, is it possible to launch ldap sync from REST API ? (so i could run it with cron ! ).

@reetp
Copy link

reetp commented Dec 10, 2021

In the CE edition, is it possible to launch ldap sync from REST API ? (so i could run it with cron ! ).

Don't believe so no. Disabled on purpose. Manual only sync via admin interface with ridiculous 2FA enabled so your cron can't run.

#23458 (comment)

See also:

#23423

@MarkVorkosigan
Copy link

MarkVorkosigan commented Dec 20, 2021

In the CE edition, is it possible to launch ldap sync from REST API ? (so i could run it with cron ! ).

Don't believe so no. Disabled on purpose. Manual only sync via admin interface with ridiculous 2FA enabled so your cron can't run.

#23458 (comment)

So, correct me if I'm wrong:

  • LDAP sync doesn't work in 4.2.2 as a background process;
  • Only manually pressed button in admin interface gets you synchronization;
  • Users are synchronized when they sign in.

P.S. Why wasn't user deleted from MongoDB after I deleted him from AD and manually synced?

@UAnton
Copy link

UAnton commented Jan 4, 2022

LDAP sync doesn't work in 4.3 as background and manual from Admin Interface

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests