Add support for LionWeb 2024.1 #98

joswarmer · 2024-12-29T13:44:06Z

Changes in Version 0.3.0

Support for LionWeb 2024.1
Removed init request from dbAdmin
Request createRepository in dbAdmin has additional mandatory parameter lionWebVersion
- All tests and applications need to add this parameter
- The server config section for creating repositories also has this additional field
All requests fail if the LionWeb version of the chunk is not the same LionWeb version as the repository.

Signed-off-by: Jos Warmer <jos.warmer@openmodeling.nl>

dslmeinte

npm reports several vulnerabilities that would need looking it, especially since this is a server component. (I know it's not the subject of this PR.)

packages/common/src/apiutil/functions.ts

packages/bulkapi/src/database/LionWebQueries.ts

packages/dbadmin/src/controllers/DBAdminApi.ts

packages/dbadmin/src/tools/init-schema-with-history.ts

packages/dbadmin/src/tools/init-schema-without-history.ts

packages/dbadmin/src/tools/init-schema-with-history.ts

packages/dbadmin/src/tools/init-schema-without-history.ts

Still need to validate requests, currently only the admin part is done.

…s parameter

joswarmer · 2025-01-06T19:44:58Z

npm reports several vulnerabilities that would need looking it, especially since this is a server component. (I know it's not the subject of this PR.)

Will take a look at them.

Now back to 0 vulnerablities.

enikao · 2025-01-07T09:27:41Z

packages/bulkapi/src/controllers/BulkApi.ts

+import { getRepositoryData, validateLionWebVersion } from "@lionweb/repository-dbadmin"
+import { getLanguageRegistry } from "@lionweb/repository-languages"
+import { LionWebJsonChunk, LionWebValidator } from "@lionweb/validation"
+// - unpack the request


Is this at the right place?

No, moved to line 1 and rephrased.

enikao · 2025-01-07T09:31:47Z

packages/dbadmin/src/controllers/DBAdminApi.ts

@@ -2,15 +2,23 @@ import e, { Request, Response } from "express"
 import {
    getRepositoryParameter,
    HttpSuccessCodes,
-    RepositoryData,
+    // RepositoryData,


enikao · 2025-01-07T09:33:44Z

packages/history/src/controllers/HistoryApi.ts

@@ -1,4 +1,5 @@
 // functions implementing the LionWeb bulk API
+import { getRepositoryData } from "@lionweb/repository-dbadmin"


order of comments / imports correct?

enikao · 2025-01-07T09:34:29Z

packages/inspection/src/controllers/InspectionApi.ts

        let clientId = getStringParam(request, "clientId")
+        // TODO Change using new repository structure


still open?

Yes, and fixed

enikao · 2025-01-07T09:35:33Z

packages/test/src/test/tests-issue19.ts

@@ -23,6 +30,7 @@ describe("Repository tests", () => {
                "./data/add-new-nodes/Disk-add-new-nodes-single-node.json",
                "./data/Disk_A.json",
            ])
+            // await t.dbAdmin.deleteRepository("default")


commented code

ftomassetti · 2025-01-10T17:05:45Z

@joswarmer
Today during the meeting you mention that we need to update the bulk import feature to the new support for multiple lionweb versions.

My understanding is that with this PR you add a column in the repositories table to record which lionweb version is used for a certain repository, but besides that the db schema does not change.

What the bulk import does is to skip the normal path for node insertions (which requires essentially calculating a diff with the current state) as we are dealing with a special situation: we are adding millions of nodes and they are all new, so we skip the diff, and we write directly to the DB, using the special COPY operation.

Now, given that the tables we are writing have not changed, my understanding is that these bulk import operations should still work and that we do not need to do anything (which is always ideal :D).

Does this make sense?

Edit: Maybe we just need to check all nodes are of the lionweb version we want too. We could do that as a preliminary check before starting the insertion, similarly as we do when checking that all sent nodes are new

ftomassetti

I just added minor comments

ftomassetti · 2025-01-10T17:07:13Z

configuration.md

-        "history": false
+        "history": false,
+        // Values can be: "2023.1" | "2024.1"
+        lionWebVersion: "2023.1"


I think 99% of users will go for the latest version, so I would use 2024.1 in this example

Ok, but that makes 2024.1 the default value from now on, as this document describes the default values.

ftomassetti · 2025-01-10T17:10:14Z

packages/bulkapi/src/database/InitializedMapToArray.ts

    get(key: KEY): VALUE[] {
        const keyString = JSON.stringify(key)
        let existingValue = this.map.get(keyString)
        if (existingValue === undefined) {
-           existingValue = []
-           this.map.set(keyString, existingValue) 
+            existingValue = []


Isn't this a weird indentation (3 spaces)?

Reformatted

ftomassetti · 2025-01-10T17:11:21Z

packages/bulkapi/src/database/QueryNode.ts

@@ -98,7 +98,7 @@ export const nodesForQueryQuery = (nodesQuery: string): string => {
            )
        containments
        FROM node_properties n1
-        LEFT JOIN ${CONTAINMENTS_TABLE} con  ON con.node_id  = n1.id 
+        LEFT JOIN ${CONTAINMENTS_TABLE} con  ON con.node_id  = n1.id  


extra white space at the end of the line

ftomassetti · 2025-01-12T13:10:18Z

configuration.md

@@ -36,7 +36,9 @@ Below is the server-config.json with all default values
        // Values are "always" | "never" | "if-not-exists"
        create: "if-not-exists"


Suggested change

create: "if-not-exists"

"create": "if-not-exists",

…un mode

joswarmer · 2025-01-13T09:08:36Z

@joswarmer Today during the meeting you mention that we need to update the bulk import feature to the new support for multiple lionweb versions.

My understanding is that with this PR you add a column in the repositories table to record which lionweb version is used for a certain repository, but besides that the db schema does not change.

That is correct.

What the bulk import does is to skip the normal path for node insertions (which requires essentially calculating a diff with the current state) as we are dealing with a special situation: we are adding millions of nodes and they are all new, so we skip the diff, and we write directly to the DB, using the special COPY operation.

Now, given that the tables we are writing have not changed, my understanding is that these bulk import operations should still work and that we do not need to do anything (which is always ideal :D).

Indeed, they should still work.

Does this make sense?

Edit: Maybe we just need to check all nodes are of the lionweb version we want too. We could do that as a preliminary check before starting the insertion, similarly as we do when checking that all sent nodes are new

I do not think it is possible to check a node for its LionWeb version, this information is not available. In a chunk of LionWeb nodes the only way to find the LionWeb version is the SerializationFormatVersion.

What want to avoid are scenarios like

I am using 2024.1 and store nodes using the bulk import feature
the repository is for 2023.1 nodes, but it cannot check the bulk import call and accepts the nodes
I try to change nodes using the regular bulk API, using seralizationFormatVersion 2024.1 and cannot access my nodes because the repository refuses them.

One way of doing the check is to provide the LionWeb version as e.g. a parameter of the bulk import call, then this can be checked once (so it won't affect the performance). This way in step 1 I can state my expectation that the repo should support 2024.1

ftomassetti · 2025-01-13T09:56:24Z

packages/common/src/apiutil/functions.ts

@@ -245,9 +261,8 @@ export function nodesToChunk(nodes: LionWebJsonNode[]): LionWebJsonChunk {
    return {
        serializationFormatVersion: "2023.1",


Here the serialization format should depend on the lionWebVersion of the repository

Indeed, changes this (at several places)

ftomassetti · 2025-01-13T10:10:27Z

packages/common/src/apiutil/functions.ts

@@ -245,9 +261,8 @@ export function nodesToChunk(nodes: LionWebJsonNode[]): LionWebJsonChunk {
    return {
        serializationFormatVersion: "2023.1",
        languages: collectUsedLanguages(nodes),
-        nodes: nodes,
+        nodes: nodes
    }
 }

 export const EMPTY_CHUNK = nodesToChunk([])


Suggested change

export const EMPTY_CHUNK = nodesToChunk([])

export const EMPTY_CHUNKS: { [K in LionWebVersion]: LionWebJsonChunk } = {

"2023.1": nodesToChunk([], "2023.1"),

"2024.1": nodesToChunk([], "2024.1"),

};

I would suggest changing EMPTY_CHUNK into a map and update usages from EMPTY_CHUNK to EMPTY_CHUNKS[repositoryData.repository.lionweb_version]

Done as suggested

ftomassetti · 2025-01-13T14:51:55Z

configuration.md

-        "history": false
+        "history": false,
+        // Values can be: "2023.1" | "2024.1"
+        lionWebVersion: "2024.1"


Suggested change

lionWebVersion: "2024.1"

"lionWebVersion": "2024.1"

joswarmer · 2025-01-13T15:03:59Z

npm reports several vulnerabilities that would need looking it, especially since this is a server component. (I know it's not the subject of this PR.)

Done, 0 left

packages/additionalapi/src/database/ImportLogic.ts

joswarmer added 3 commits December 28, 2024 17:53

Add lionweb version, make some strings (like table names) constants

6d7fced

Add repository admin in server

021a7bc

remove logs

dd033d5

Signed-off-by: Jos Warmer <jos.warmer@openmodeling.nl>

dslmeinte requested changes Dec 29, 2024

View reviewed changes

joswarmer added 11 commits December 29, 2024 15:21

remove commented logs

11227a2

Ensure repositories has a context

7504499

remove commented logs, changed

99d64ac

Ensure repositories has a context

c9de792

Merge remote-tracking branch 'origin/lionweb-2024.1' into lionweb-2024.1

5f4ab44

Take LionWeb version into account for all repositories.

cfac433

Still need to validate requests, currently only the admin part is done.

Version 0.2.0 => 0.3.0

4a602d2

Add validation of lionweb version to all requests that have a chunk a…

0ce02ce

…s parameter

return in memory repositories for listRepositories request.

dcd36a8

Code cleanup

f80112f

Fix error

d88c1a0

joswarmer changed the title ~~[WIP] Add lionweb version, make some strings (like table names) constants~~ Add support for LionWeb 2024.1 Jan 6, 2025

joswarmer requested review from ftomassetti and enikao January 6, 2025 15:25

review comments

862e2a8

Formatting sources.

c031fdc

enikao reviewed Jan 7, 2025

View reviewed changes

ftomassetti reviewed Jan 10, 2025

View reviewed changes

ftomassetti reviewed Jan 12, 2025

View reviewed changes

fix createRepository

a1aeb0c

ftomassetti added 4 commits January 12, 2025 14:39

improve logging

adca66d

Fix deletion of schema

19762ae

Wip

f932e4d

Ensure we read the repository store when starting the repository in r…

0673096

…un mode

ftomassetti reviewed Jan 13, 2025

View reviewed changes

ftomassetti and others added 13 commits January 13, 2025 11:11

Ensure to serialize the chunk using the right version

cbf5875

Update dependencies to have no vulnerabilities

de38c2a

Update dependencies to have no vulnerabilities

cca9dc2

Linting

5df0044

Adding DBAdminApiWorker.addRepositoryToTable

714e796

Adding DBAdminApiWorker.addRepositoryToTable

fd3aa35

perform setup before run, when running tests

1815b79

Update configuration.md

7e55df9

review comments

d286a47

Merge branch 'lionweb-2024.1' into bugfix/repoinit

4ee45de

Fix incorrect imports

8afbd03

Merge branch 'lionweb-2024.1' into bugfix/repoinit

a0a7516

Fix merge errors

c40dd0d

ftomassetti reviewed Jan 13, 2025

View reviewed changes

Review comment

9563835

dslmeinte approved these changes Jan 13, 2025

View reviewed changes

ftomassetti reviewed Jan 14, 2025

View reviewed changes

packages/additionalapi/src/database/ImportLogic.ts Outdated Show resolved Hide resolved

Fix use of repository and schema names

7c6dfa1

This was referenced Jan 15, 2025

Introduce support for LW 2024.1 LionWeb-io/lionweb-kotlin#5

Merged

Update to LW 2024.1 Strumenta/kolasu#387

Merged

joswarmer merged commit cbf8e0c into main Jan 16, 2025
2 checks passed

joswarmer deleted the lionweb-2024.1 branch January 16, 2025 11:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for LionWeb 2024.1 #98

Add support for LionWeb 2024.1 #98

joswarmer commented Dec 29, 2024 •

edited

Loading

dslmeinte left a comment

joswarmer commented Jan 6, 2025 •

edited

Loading

enikao Jan 7, 2025

joswarmer Jan 13, 2025

enikao Jan 7, 2025

joswarmer Jan 13, 2025

enikao Jan 7, 2025

joswarmer Jan 13, 2025

enikao Jan 7, 2025

joswarmer Jan 13, 2025

enikao Jan 7, 2025

joswarmer Jan 13, 2025

ftomassetti commented Jan 10, 2025 •

edited

Loading

ftomassetti left a comment

ftomassetti Jan 10, 2025

joswarmer Jan 13, 2025

ftomassetti Jan 10, 2025

joswarmer Jan 13, 2025

ftomassetti Jan 10, 2025

joswarmer Jan 13, 2025

ftomassetti Jan 12, 2025

joswarmer Jan 13, 2025

joswarmer commented Jan 13, 2025 •

edited

Loading

ftomassetti Jan 13, 2025

joswarmer Jan 13, 2025

ftomassetti Jan 13, 2025

joswarmer Jan 13, 2025

ftomassetti Jan 13, 2025

joswarmer Jan 13, 2025

joswarmer commented Jan 13, 2025

		@@ -1,4 +1,5 @@
		// functions implementing the LionWeb bulk API
		import { getRepositoryData } from "@lionweb/repository-dbadmin"

		let clientId = getStringParam(request, "clientId")
		// TODO Change using new repository structure

		@@ -36,7 +36,9 @@ Below is the server-config.json with all default values
		// Values are "always" \| "never" \| "if-not-exists"
		create: "if-not-exists"

		@@ -245,9 +261,8 @@ export function nodesToChunk(nodes: LionWebJsonNode[]): LionWebJsonChunk {
		return {
		serializationFormatVersion: "2023.1",

-export const EMPTY_CHUNK = nodesToChunk([])
+export const EMPTY_CHUNKS: { [K in LionWebVersion]: LionWebJsonChunk } = {
+    "2023.1": nodesToChunk([], "2023.1"),
+    "2024.1": nodesToChunk([], "2024.1"),
+};

Add support for LionWeb 2024.1 #98

Add support for LionWeb 2024.1 #98

Conversation

joswarmer commented Dec 29, 2024 • edited Loading

dslmeinte left a comment

Choose a reason for hiding this comment

joswarmer commented Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ftomassetti commented Jan 10, 2025 • edited Loading

ftomassetti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joswarmer commented Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joswarmer commented Jan 13, 2025

joswarmer commented Dec 29, 2024 •

edited

Loading

joswarmer commented Jan 6, 2025 •

edited

Loading

ftomassetti commented Jan 10, 2025 •

edited

Loading

joswarmer commented Jan 13, 2025 •

edited

Loading