Data-verse fails to support upload in multiple user configured project groups

[jmurugan-fzj]

Describe the bug
By default the data-verse implementation uses 'defaultProjectGroup' which is supposed to be the 'research' database. All the models are created within this DB and also the projects existing in 'defaultProjectGroup' will only be populated in dataverse UI

Steps To Reproduce
Steps to reproduce the behavior:

1. Run Pasta App
2. Go to "System | project groups" and add a new project group (my_project_group_name, database:test) and save it successfully
3. 
4. Go to "System | Change project group" and change to "my_project_group_name"
5. Add a new project named, "Test Pasta Project" and save it in successfully
6. Go to "System | Upload to dataverse" and see the error been thrown below:
7. 

Expected behavior
Dataverse must work consistently across different project groups without any errors.

Screenshots
Given above.

Desktop (please complete the following information):

• OS: 22.04.1-Ubuntu
• Browser: chrome
• PASTA ELN Version: Revision Number f9985e5

[jmurugan-fzj]

@SteffenBrinckmann I logged this as an issue and would like to discuss with you face to face before the implementation. Since you know the project groups feature very well, I would like to discuss it with you and need to know all the possibilities before going for a particular implementation, whether to go for a dedicated db itself, etc...

[SteffenBrinckmann]

@jmurugan-fzj I am officially not working for some time, but I can try to find some time.
Here in short: the content of each project group is stored in a separate database within the one couchdb server, for protection, ... with no ability to interact. By design.
If you want to have some function that works for all project groups, it cannot be in any of those databases. For your dataverse content: just create a new database on the couchdb server, and you only use that database. You can instatiate that database whenever you want and re-creating the example in any of the project groups does not affect your database.
Let me know if this is enough information to go on, otherwise I have to schedule some time. Best, Steffen

[jmurugan-fzj]

@SteffenBrinckmann I can move everything related to data-verse to a separate database, but the problem is that the data-verse UI also needs to show the projects appropriately, so for me it should be also possible to get the currently configured projectgroup and the corresponding database name from somewhere, is this the default project group or this information is stored somewhere else? So in my API, I have to use configure project group database name while fetching the projects list and all the other operations should be done within the newly created dedicated dataverse db. I am not sure if this is really a neat approach, but this is what I could think of quickly with the present implementation! Please let me know your opinion on this..

Meanwhile, I will try the above approach and see if I stumble upon any major problems..!

[SteffenBrinckmann]

@jmurugan-fzj yes, it is the 'defaultProjectGroup' from the configuration file .pastaEln.json. The database infromation is then in the corresponding entry: under local -> database.
I think this is a neat approach; you get the project data from the default project group and then store all your dataverse items in their own database.

[jmurugan-fzj]

@SteffenBrinckmann What about the '-dataHierarchy-' document, will this be always created in "research" database or in 'defaultProjectGroup', how does it work in general when the user switches the project groups? I am a bit confused, will this have to be created for every 'defaultProjectGroup' or just "research" alone?

Another point is about the history, the "show completed" will display the finished items from all the project groups, is this fine?

[SteffenBrinckmann]

Each project group has it's own database and each database has it's own Datahierarchy.

[jmurugan-fzj]

Each project group has it's own database and each database has it's own Datahierarchy.

@SteffenBrinckmann Okay, so now that if use 'defaultProjectGroup' db for projects and a "dedicated db" for dataverse, this can make the logic a bit more complicated that I have to keep track of two databases in parallel, one for the dataverse alone and the other one for reading the projects list alone. So internally the API needs to be adapted to fetch data for different db_names. Another problem is that as I mentioned earlier the history of completed items will show all the items from different project groups in a single window. If we keep track of a single database, when all the data-verse related items are also stored in 'defaultProjectGroup' database itself, then the history will also be shown appropriately as per the switched DB. Otherwise if we need to display the history too as per the project group, then they have to be also filtered in this manner. I think, the second approach (everything created and stored in switched 'defaultProjectGroup' database) is a bit more neat and concise, what are your thoughts?

[jmurugan-fzj]

@SteffenBrinckmann Another problem is that I take the user name and password for the db from the .pastaELN.json file, for the newly created projects with "Save encrypted" option, a "cred" is assosciated and no "username" & "password" available for the couchdb client, how should this scenario be handled? do I need to extract the user & pass combination somehow or can I use this "cred" directly in the couchdb client?

Note: Found the solution for this problem already, I found the function in codebase to decrypt the "cred" value. Please take a look at the review and provide your feedback, if this is done correctly and as expected!