(Server Mode) How do I make a new project visible in the UI?

[guillaumededrie]

Hello,
I'm trying to add a completely new project, but I end up without seeing only the default one initialized in the interface.
I think, with the use of --defaultProject, I'm may mistaken the procedure to create a new project (even an empty one).

I'm using the following docker-compose.yml file in a Docker Swarm environment:

---
version: "3.8"


networks:
  datashare_backend:
    name: "datashare_backend"

volumes:
  redis-data:
  elasticsearch-data:
  postgresql-data:

services:
  redis:
    image: "redis:7.2.3-bookworm@sha256:d4c84914b872521e215f77d8845914c2268a96b0e35bacd5691e1f5e1f88b500"
    volumes:
      - "redis-data:/data"
    networks:
      - datashare_backend
    deploy:
        <DOCKER SWARM_REDACTED>

  elasticsearch:
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.17.16-amd64"
    volumes:
      - "elasticsearch-data:/usr/share/elasticsearch/data"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    networks:
      - datashare_backend
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=0.0.0.0"
      - "cluster.name=datashare"
      - "discovery.type=single-node"
      - "discovery.zen.minimum_master_nodes=1"
      - "xpack.license.self_generated.type=basic"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
      - "http.cors.allow-methods=OPTIONS, HEAD, GET, POST, PUT, DELETE"
    deploy:
        <DOCKER SWARM_REDACTED>

  postgresql:
    image: "postgres:15.5-bookworm@sha256:ee2f170c46df225310c923010230434e269238a65307539f9aced9da6ca44fad"
    environment:
      - "POSTGRES_USER=datashare"
      - "POSTGRES_PASSWORD=password"
      - "POSTGRES_DB=datashare"
    volumes:
        - "postgresql-data:/var/lib/postgresql/data"
    networks:
      - datashare_backend
    command: postgres -c 'max_connections=200'
    deploy:
        <DOCKER SWARM_REDACTED>


  datashare_web:
    image: "icij/datashare:13.9.0@sha256:44325432bb02adcfbf09171087ab92f74e2c4fd27a8a68fc024e6ea54fd90844"
    volumes:
      - "/home/datashare/projects:/home/datashare/Datashare"
    networks:
      - datashare_backend
    environment:
      - "DS_DOCKER_MOUNTED_DATA_DIR=/home/datashare/Datashare"
    command: >-
      --mode SERVER
      --dataDir /home/datashare/Datashare
      --pluginsDir /home/datashare/plugins
      --extensionsDir /home/datashare/extensions
      --authFilter org.icij.datashare.session.YesCookieAuthFilter
      --busType REDIS
      --batchQueueType REDIS
      --dataSourceUrl jdbc:postgresql://postgresql/datashare?user=datashare\&password=password
      --defaultProject default
      --elasticsearchAddress http://elasticsearch:9200
      --messageBusAddress redis://redis:6379
      --queueType REDIS
      --redisAddress redis://redis:6379
      --rootHost <redacted>
      --sessionStoreType REDIS
      --sessionTtlSeconds 43200
      --tcpListenPort 8080

Then, after deploying, I've run the equivalent datashare_create_project for the --defaultProject default:

$ sudo docker exec -it 421c0553838f /entrypoint.sh --mode CLI --stage INDEX --elasticsearchAddress http://elasticsearch:9200 --defaultProject default

Then, using the Web Interface, I can see the default project and there are no errors.

Now, I want to add a new project, that will received new documents (as it is done on https://datashare-demo.icij.org), so I've tried --createIndex (then also with --defaultProject instead, but that wasn't making sense to me…):

$ sudo docker exec -it 421c0553838f /entrypoint.sh --mode CLI --elasticsearchAddress http://elasticsearch:9200 --createIndex newproject

2024-01-04 14:50:40,370 [main] INFO  Main - Running datashare
2024-01-04 14:50:40,371 [main] INFO  Main - JVM version 11.0.16.1
2024-01-04 14:50:40,372 [main] INFO  Main - JVM charset encoding UTF-8
2024-01-04 14:50:40,461 [main] WARN  PropertiesProvider - no null file found, using default values
2024-01-04 14:50:40,464 [main] INFO  PropertiesProvider - adding properties from env vars {mountedDataDir=/home/datashare/Datashare}
2024-01-04 14:50:40,464 [main] INFO  PropertiesProvider - properties set to {mountedDataDir=/home/datashare/Datashare}
2024-01-04 14:50:40,464 [main] INFO  PropertiesProvider - merged properties (without override) with {parserParallelism=1, batchDownloadDir=/home/datashare/app/tmp, cors=no-cors, dataDir=/home/datashare/data, para
llelism=16, batchQueueType=MEMORY, scrollSlices=1, elasticsearchDataPath=/home/datashare/es, elasticsearchAddress=http://elasticsearch:9200, protectedUriPrefix=/api/, redisPoolSize=2, mode=CLI, digestProjectName=
local-datashare, clusterName=datashare, nlpParallelism=1, busType=MEMORY, defaultProject=local-datashare, noDigestProject=false, scrollSize=1000, ocr=true, embeddedDocumentDownloadMaxSize=1G, messageBusAddress=re
dis://redis:6379, dataSourceUrl=jdbc:sqlite:file:memorydb.db?mode=memory&cache=shared, defaultUserName=local, batchDownloadMaxNbFiles=10000, digestAlgorithm=SHA-384, createIndex=newproject, tcpListenPort=8080,
sessionStoreType=MEMORY, queueName=extract:queue, batchDownloadTimeToLive=24, redisAddress=redis://redis:6379, queueType=MEMORY, browserOpenLink=false, batchDownloadMaxSize=100M, sessionTtlSeconds=43200}
2024-01-04 14:50:40,767 [main] INFO  ExtensionLoader - read directory ./extensions and found jars (executable): []
2024-01-04 14:50:40,771 [main] INFO  HikariDataSource - HikariPool-1 - Starting...
2024-01-04 14:50:40,813 [main] INFO  HikariDataSource - HikariPool-1 - Start completed.
2024-01-04 14:50:42,886 [main] INFO  JdbcExecutor - SELECT COUNT(*) FROM DATABASECHANGELOGLOCK
2024-01-04 14:50:42,892 [main] INFO  JdbcExecutor - CREATE TABLE DATABASECHANGELOGLOCK (ID INTEGER NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TEXT, LOCKEDBY VARCHAR(255), CONSTRAINT PK_DATABASECHANGELOGLOCK P
RIMARY KEY (ID))
2024-01-04 14:50:42,892 [main] INFO  JdbcExecutor - SELECT COUNT(*) FROM DATABASECHANGELOGLOCK
2024-01-04 14:50:42,895 [main] INFO  JdbcExecutor - DELETE FROM DATABASECHANGELOGLOCK
[…]
2024-01-04 14:50:43,908 [main] INFO  JdbcExecutor - CREATE INDEX document_user_mark_read_user_id ON document_user_recommendation(user_id)
2024-01-04 14:50:43,909 [main] INFO  ChangeSet - Columns creation_date(datetime) added to document_user_recommendation
2024-01-04 14:50:43,909 [main] INFO  ChangeSet - ChangeSet liquibase/changelog/changes/034-adds-creation-date-column-to-document-user-recommendation.yml::62::promera ran successfully in 30ms
2024-01-04 14:50:43,909 [main] INFO  JdbcExecutor - INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYME
NT_ID) VALUES ('62', 'promera', 'liquibase/changelog/changes/034-adds-creation-date-column-to-document-user-recommendation.yml', CURRENT_TIMESTAMP, 60, '8:32b98343fe32d4accaf39cce3c301ea7', 'addColumn tableName=d
ocument_user_recommendation', '', 'EXECUTED', NULL, NULL, '3.8.9', '4379843355')
2024-01-04 14:50:43,909 [main] INFO  StandardLockService - Successfully released change log lock
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/home/datashare/lib/datashare-dist-13.9.0-all.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,by
te[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-01-04 14:50:43,995 [main] INFO  Indexer - indexer defined with cfg{indexJoinField='join', docTypeField='type', shards=1, replicas=1}
2024-01-04 14:50:43,996 [main] INFO  CliApp - found 0 CLI extension(s)

I can see that:

1. both default and newproject ES indexes are available:

   root@ce5f83988add:/usr/share/elasticsearch# curl -XGET 127.0.0.1:9200/_cat/shards?
   newproject                                                  0 p STARTED     0   227b 10.0.30.6 ce5f83988add
   newproject                                                  0 r UNASSIGNED
   .geoip_databases                                              0 p STARTED    41 38.5mb 10.0.30.6 ce5f83988add
   default                                                       0 p STARTED     0   227b 10.0.30.6 ce5f83988add
   default                                                       0 r UNASSIGNED
   .ds-.logs-deprecation.elasticsearch-default-2024.01.04-000001 0 p STARTED              10.0.30.6 ce5f83988add
   .ds-ilm-history-5-2024.01.04-000001                           0 p STARTED              10.0.30.6 ce5f83988add

2. That there aren't any record in the project Postgresql table:

   datashare=# select * from project;
   id | path | allow_from_mask | label | publisher_name | maintainer_name | source_url | logo_url | creation_date | 
   update_date | description
   ----+------+-----------------+-------+----------------+-----------------+------------+----------+---------------+-------------+-------------
   (0 rows)

I've check the online documentation, GitHub Issues and Discussions, but I wasn't able to make any progress.
Could you point me in the right part of the documentation in order to be able to manage multiple projects in Datashare?

Regards.

[ancailliau]

You need to use an authentication provider and allow your user to view the project. See https://icij.gitbook.io/datashare/server-mode/authentication-providers