Greenlight v2 -> v3 migration script migrates no external users

[drudgede]

I solved my original problem but now I have the problem that I can only migrate local users, see post below. Any help is appreciated.

Yesterday, I set up a fresh BBB server along with greenlight v3 according to the docs and both components are present in the latest version. In general, the setup works fine.

Today, I wanted to migrate all users from BBB 2.4 with greenlight 2.14.7 to the new server with BBB 2.6 and greenlight v3. All of my users (with the exception of one local admin user) are LDAP users and I understand that LDAP is not natively supported anymore in greenlight v3, however, I want to keep the association between the LDAP users and their created rooms and settings in the past.

When I run the migration script, however, nothing is actually migrated. For instance, I have several roles (e.g. normal users, users with the right to record etc.) in my old environment and the migration script for the roles is executed as follows:

docker exec -it greenlight-v2 bundle exec rake migrations:roles
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initiali
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initial
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initial
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.

Roles migration complete.

The V3_* parameters are set correctly according to the docs.

As you can see, the roles migration script states that the migration is completed. Unfortunately, no roles are listed during the execution and they are not transferred to the new greenlight v3 instance. According to the docs, the script lists all roles (users, rooms, settings), which are currently migrated by the script. However, in my case, no roles are listed. I have the tested it for users, rooms and settings as well and I have the same behaviour. I also tried to download the migration script manually according to the docs for greenlight v2 version which are not present in the latest version and I have the same result.

So in general, my questions are:

• Is it possible to migrate the association between LDAP users and their old rooms in greenlight v2 or is this only possible for local users? At least, I expected the script to migrate the roles since they seem to be greenlight-local settings.
• Is it possible to migrate the users between greenlight v2.14.7 with BBB 2.4 to 2.6 directly or do I have to migrate the roles, users, rooms, and settings to 2.5 first and then to 2.6?

[drudgede]

Okay, I found out by manually checking the old postgres database that during a failed migration process in the past, it seems that all entities were deleted. I restored a snapshot of the machien and now it works in general.

I successfully migrated the roles. However, I have the problems that only local users are migrated:

docker exec -it greenlight-v2 bundle exec rake migrations:users
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Successfully migrated User:
  UID: gl-quuokcehwzsa
  Name: Administrator
  Provider: greenlight

Users migration completed.
docker exec -it greenlight-v2 bundle exec rake migrations:users
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: already initialized constant Net::ProtocRetryError
/usr/local/lib/ruby/2.7.0/net/protocol.rb:66: warning: previous definition of ProtocRetryError was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: already initialized constant Net::BufferedIO::BUFSIZE
/usr/local/lib/ruby/2.7.0/net/protocol.rb:206: warning: previous definition of BUFSIZE was here
/usr/src/app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: already initialized constant Net::NetPrivate::Socket
/usr/local/lib/ruby/2.7.0/net/protocol.rb:503: warning: previous definition of Socket was here
warning: parser/current is loading parser/ruby27, which recognizes2.7.6-compliant syntax, but you are running 2.7.7.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Successfully migrated User:
  UID: gl-quuokcehwzsa
  Name: Administrator
  Provider: greenlight

Users migration completed.

Although the documentations states: Both local and external users will be migrated. I need to keep the associations between the LDAP users and their rooms.

However, I can see als the users in the database:

docker exec -it a /bin/bash  # greenlight v2 docker machine
bash-5.1# su postgres
\c greenlight_production
You are now connected to database "greenlight_production" as user "postgres".
select * from users where id=1 or id=2; # 1 is the local admin and 2 is my LDAP user 

 id | room_id |  provider  |       uid       |       name        | username |                email                 |              social_uid               | image |                       password_digest                        | accepted_terms |         created_at         |         updated_at         | email_verified | language | reset_digest | reset_sent_at | activation_digest | activated_at | deleted | role_id |         last_login         | failed_attempts | last_failed_attempt | last_pwd_update
----+---------+------------+-----------------+-------------------+----------+--------------------------------------+---------------------------------------+-------+--------------------------------------------------------------+----------------+----------------------------+----------------------------+----------------+----------+--------------+---------------+-------------------+--------------+---------+---------+----------------------------+-----------------+---------------------+-----------------
  2 |       2 | ldap       | gl-xwnlnpxgpisk | Mark              | r10ad059 | emailaddress@company.tld             | cn=myuser,ou=myou,o=mycompany         |       |                                                              | f              | 2021-06-17 12:03:34.421188 | 2023-06-21 18:29:49.297479 | t              | default  |              |               |                   |              | f       |       6 | 2023-06-21 18:29:49.295827 |                 |                     |
  1 |       1 | greenlight | gl-quuokcehwzsa | Administrator     |          | marksadminmail@company.tld           |                                       |       | $2a$12$0/csrbpYaC0HZrz8L1fg..rYM.DW3pPh2fs0ngiMBgjm2iE7sYqAi | t              | 2021-06-17 11:13:15.489085 | 2022-01-07 03:32:37.785148 | t              | default  |              |               |                   |              | f       |       2 | 2021-06-17 11:13:40.802726 |                 |                     |
(2 rows)

(redacted some property values but the entities are there)

Any ideas how I can also migrate the external users?

[drudgede]

I inspected the migration rake script and saw that there is a parameter called provider which is by default greenlight. As apparent above, the LDAP users come with an ldap-provider.

I re-run the script with the parameter:

docker exec -it greenlight-v2 bundle exec rake migrations:users\[ldap\]
docker exec -it greenlight-v2 bundle exec rake migrations:rooms\[ldap\]
docker exec -it greenlight-v2 bundle exec rake migrations:settings\[ldap\]

And it worked perfectly. Everything is now imported.

Afterwards, I only had to adapt the users in the new postgres database because the new users.external_id attribute was set to the full LDAP DN instead to the CN and therefore, the login did not work.

So to migrate values in users.external_id the format cn=myusername,ou=myorgunit,ou=myparentorgunit,o=myorg, only to myusername , I used the following script:

Select new values which will be used by by the next command to check if they will be migrated correctly:

select split_part(split_part(external_id,'=',2),',',1) from users where external_id like '%o=myorg';

Now the correct usernames are printed. Update the values:

update users set external_id=split_part(split_part(external_id,'=',2),',',1) where external_id like '%o=myorg';

Now, the login usernames were modified and everything works like a charm.

I'll leave this here for users with the same problem.

However, I leave the ticket open because I want to propose that the additional parameter for the provider is added to the documentation. I cannot find it anywhere on the migration doc page. The pages only states that internal and external users are migrated, whereby the default provider value is set to greenlight and I did not find any indication that this additional parameter can be used and I guess that reading the rake scripts might be complicated for unexperienced users.

Apart from that, the ticket can be closed 😄

[farhatahmad]

Hmm - this change was accidental. Working on a hotfix

[danci1973]

I'm currently in a similar process, migrating from BBB 2.2 to 2.6.

Since using LDAP directly is no longer supported (at least as far as I could find out), I installed and configured Keycloak + OpenID (in Keycloak I used 'User federation), which seems to work as LDAP users can log in and a Greenlight user profile gets created.

There are also some local users on old BBB which were invited to test BBB, but normally our users are authenticated through LDAP.

Migration scripts successfully migrated the few local users (I think these were external -as in not our staff- users, just for testing BBB), but when I run the user migration with [ldap] provider, I get an error that 'Provider does not exists':

Unable to migrate User:
UID: gl-qprpiezescxd
Name: First Last
Provider: ldap
Errors: Provider does not exist
...

And it shows that for all LDAP users in Greenlight v2, so it appears that the problem is in the new instance of Greenlight v3.

Any ideas why Keycloak / LDAP is not recognized as 'ldap' provider?

[drudgede]

@farhatahmad thanks for providing a quick hotfix. I also encountered another problem which I recognized today. However, since all of my users are still migrated, I cannot test if the hotfix covers this issue.

I noticed that Keycloak creates the username (e.g. for LDAP users) always in lowercase. If one chooses CN as the username identifier in Keykcloak, for example the username u123 is internally created, no matter if the original CN was CN=u123 or CN=U123.

In my case, the external_id in my case was migrated and not migrated to lowercase. I had login issues for users, whose usernames are in lowercase in Keycloak but not in the greenlight database (e.g. Greenlight username is U123 and Keycloak username is u123).

When the user logs in, the logins succeeds in general (in Keycloak you can see a succesful login if logging is enabled), however, Greenlight tells you that you cannot be authenticated.

I updated my users by executing

update users set external_id = lower(external_id)

and now the login works fine for all.

I'm not deep enough into rake to see if this an actual problem, but I wanted you to know that there is an edge case which may cause trouble.

[farhatahmad]

Thanks for the info - this should be working in the latest releases