Minilcm diff API #1181

hahn-kev · 2024-10-30T09:53:59Z

uses the new harmony version to put a commit id on each object
creates a new UpdateEntry(Entry entry) api for updating using the version to ensure we don't stomp on changes
write some fuzzing tests using randomly generated entries for create and update tests

was current/previous now before after

# Conflicts: # backend/FwLite/FwDataMiniLcmBridge/Api/UpdateProxy/UpdateSenseProxy.cs # backend/FwLite/FwLiteProjectSync.Tests/UpdateDiffTests.cs # backend/FwLite/LcmCrdt/CrdtMiniLcmApi.cs # backend/FwLite/MiniLcm/Models/Entry.cs

# Conflicts: # .idea/.idea.LexBox/.idea/dataSources.xml # backend/FwLite/FwLiteProjectSync.Tests/Fixtures/SyncFixture.cs

…recreated with the same Id

…nit twice

hahn-kev · 2024-10-30T09:55:01Z

backend/FwLite/FwDataMiniLcmBridge.Tests/MiniLcmTests/BasicApiTests.cs

+
+        update2.LexemeForm["es"] = "updated again";
+        Func<Task> action = async () => await Api.UpdateEntry(update2);
+        await action.Should().ThrowAsync<VersionInvalidException>();


this is different from the CRDT version, is that ok?

That's what CRDT's are all about right?! 🙂 👍

github-actions · 2024-10-30T09:56:14Z

C# Unit Tests

75 tests ±0 75 ✅ ±0 5s ⏱️ ±0s
13 suites ±0 0 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit e402294. ± Comparison against base commit a84c509.

♻️ This comment has been updated with latest results.

hahn-kev · 2024-10-30T10:03:03Z

backend/FwLite/LcmCrdt.Tests/MiniLcmTests/BasicApiTests.cs

+    }
+
+    [Fact]
+    public async Task UpdateEntry_CanUseSameVersionMultipleTimes()


as with a similar version in FW, this is different behavior from the CRDT Version

hahn-kev · 2024-10-30T10:05:53Z

backend/FwLite/LcmCrdt/CrdtMiniLcmApi.cs

+            .SingleAsyncEF();
+        //todo this will not work for properties not in the snapshot, but we don't have a way to get the full snapshot yet
+        //todo also, add api for getting an entity at a specific commit
+        var snapshot = await dataModel.GetEntitySnapshotAtTime(commitHybridDate.DateTime.AddSeconds(1), entry.Id);


I plan to fix this up so we can pass in the commit Id and get back the entry

hahn-kev · 2024-10-30T10:06:07Z

backend/FwLite/LcmCrdt/CrdtMiniLcmApi.cs

+        var snapshot = await dataModel.GetEntitySnapshotAtTime(commitHybridDate.DateTime.AddSeconds(1), entry.Id);
+        var before = snapshot?.Entity.DbObject as Entry;
+        ArgumentNullException.ThrowIfNull(before);
+        //workaround to avoid syncing senses, which are not in the snapshot


I'm not sure how to resolve this issue

So we basically have to replicate this?

hahn-kev · 2024-10-30T10:08:11Z

backend/FwLite/MiniLcm.Tests/AutoFakerHelpers/EntryFakerHelper.cs

+        if (createComplexFormTypes) await CreateComplexFormTypes(entry.ComplexFormTypes, api);
+        foreach (var sense in entry.Senses)
+        {
+            sense.EntryId = entry.Id;


I don't want to have to think about this when using our faker, but we should write a test and decide what we want to happen if the EntryId does not match on the sense, maybe if it's empty then we set it, but if it's not the same should we throw?

hahn-kev · 2024-10-30T10:08:56Z

backend/FwLite/MiniLcm.Tests/AutoFakerHelpers/EntryFakerHelper.cs

+                //generated entries won't have the expected ids, so fix them up here
+                if (isComponent)
+                {
+                    complexFormComponent.ComplexFormEntryId = entryId;


same for complex form stuff, if the id doesn't match what it should be do throw?

hahn-kev · 2024-10-30T10:09:35Z

backend/FwLite/MiniLcm.Tests/AutoFakerHelpers/ObjectWithIdOverride.cs

+    {
+        if (context.Instance is IObjectWithId obj)
+        {
+            obj.DeletedAt = null;


if someone calls update with a deleted object what should we do? throw? delete it?

hahn-kev · 2024-10-30T10:10:19Z

backend/FwLite/MiniLcm.Tests/CreateEntryTestsBase.cs

+        //todo limitation of fwdata prevents us from specifying the complex form type ahead of time
+        foreach (var entryComplexFormType in entry.ComplexFormTypes)
+        {
+            entryComplexFormType.Id = Guid.Empty;


pending sillsdev/liblcm#313

…han using a HashSet of versions

backend/FwLite/FwDataMiniLcmBridge/Api/FwDataMiniLcmApi.cs

myieye

Made a bit of progress

myieye · 2024-10-30T15:03:44Z

backend/FwLite/FwDataMiniLcmBridge/Api/FwDataMiniLcmApi.cs

+                var lexEntry = EntriesRepository.GetObject(complexFormComponent.ComplexFormEntryId);
+                AddComplexFormComponent(lexEntry, complexFormComponent);
+            });
+        return Task.FromResult(ToComplexFormComponents(EntriesRepository.GetObject(complexFormComponent.ComplexFormEntryId)).Single(c => c.ComponentEntryId == complexFormComponent.ComponentEntryId));


A complex form can reference both an entry itself as well as any number of senses of that entry. That's how the FW UI works at least, so the FWL UI does the same.

Suggested change

return Task.FromResult(ToComplexFormComponents(EntriesRepository.GetObject(complexFormComponent.ComplexFormEntryId)).Single(c => c.ComponentEntryId == complexFormComponent.ComponentEntryId));

return Task.FromResult(ToComplexFormComponents(EntriesRepository.GetObject(complexFormComponent.ComplexFormEntryId)).Single(c => c.ComponentEntryId == complexFormComponent.ComponentEntryId && c.ComponentSenseId == complexFormComponent.ComponentSenseId));

backend/FwLite/FwDataMiniLcmBridge/Api/FwDataMiniLcmApi.cs

myieye · 2024-10-30T15:15:56Z

backend/FwLite/FwDataMiniLcmBridge/Api/UpdateProxy/UpdateSenseProxy.cs

@@ -64,14 +64,14 @@ public override IList<SemanticDomain> SemanticDomains
            new UpdateListProxy<SemanticDomain>(
                semanticDomain =>
                {
+#pragma warning disable VSTHRD002
                    if (semanticDomain.Id != default && lexboxLcmApi.GetLcmSemanticDomain(semanticDomain.Id) is { } lcmSemanticDomain)
                        sense.SemanticDomainsRC.Add(lcmSemanticDomain);


Why does Remove use the lexboxLcmApi, but Add doesn't?

myieye · 2024-10-30T15:19:59Z

backend/FwLite/FwLiteProjectSync.Tests/EntrySyncTests.cs

+                }
+            ]
+        });
+        Entry after = (Entry) component.Copy();


Getting a lot of mileage out of .Copy() 😀

myieye · 2024-10-30T15:21:55Z

backend/FwLite/FwLiteProjectSync.Tests/Fixtures/SyncFixture.cs

@@ -49,16 +49,15 @@ public async Task InitializeAsync()
            .ProjectsFolder;
        if (Path.Exists(projectsFolder)) Directory.Delete(projectsFolder, true);
        Directory.CreateDirectory(projectsFolder);
-        _services.ServiceProvider.GetRequiredService<IProjectLoader>()
+        var lcmCache = _services.ServiceProvider.GetRequiredService<IProjectLoader>()


This looks unintentional as I don't think it's being used.

myieye · 2024-10-30T16:01:09Z

backend/FwLite/MiniLcm/SyncHelpers/EntrySync.cs

+            beforeComponents,
+            afterComponents,
+            //we can't use the ID as there's none defined by Fw so it won't work as a sync key
+            component => (component.ComplexFormEntryId, component.ComponentEntryId, component.ComponentSenseId),


So, this will never result in a replace operation, because the identity is equivalent to the value, so the collection differ will call replace for all unchanged components, bceause the id is the same and then the replace logic starting on L83 will do nothing, because the value is the same.

You probably know that and it's the simplest way to keep everything happy for now.

This is where a more sophisticated list differ would be nice in the future. E.g. if the before and after lists are the same length and only the nth object changed, then a replace probably makes sense even if the IDs are different. But then we'd be disregarding Guids that the client generated, so that might be bad.

myieye · 2024-10-30T16:03:39Z

backend/FwLite/MiniLcm/SyncHelpers/EntrySync.cs

+    public static UpdateObjectInput<Entry>? EntryDiffToUpdate(Entry beforeEntry, Entry afterEntry)
+    {
+        JsonPatchDocument<Entry> patchDocument = new();
+        patchDocument.Operations.AddRange(MultiStringDiff.GetMultiStringDiff<Entry>(nameof(Entry.LexemeForm), beforeEntry.LexemeForm, afterEntry.LexemeForm));


What are your thoughts on using reflection for this?

myieye · 2024-10-30T16:06:52Z

backend/FwLite/MiniLcm/SyncHelpers/SenseSync.cs

+        patchDocument.Operations.AddRange(MultiStringDiff.GetMultiStringDiff<Sense>(nameof(Sense.Definition),
+            beforeSense.Definition,
+            afterSense.Definition));
+        if (beforeSense.PartOfSpeech != afterSense.PartOfSpeech)


Having both PartofSpeech and PartOfSpeechId is code debt, right?
I feel like it was a quick hack somewhere along the line. Should there be an issue to clean that up?

myieye · 2024-10-30T16:12:35Z

backend/FwLite/LcmCrdt/CrdtMiniLcmApi.cs

+                if (complexFormComponent.ComplexFormEntryId == default) complexFormComponent.ComplexFormEntryId = entry.Id;
+                if (complexFormComponent.ComponentEntryId == complexFormComponent.ComplexFormEntryId)
+                {
+                    throw new InvalidOperationException($"Complex form component {complexFormComponent} has the same component id as its complex form");


All these exceptions are being thrown in GetEntry and are testing for things we don't expect to be possible right?
They're also slow, because they're behind db queries.

Shouldn't this validation essentially be dev-mode asserts?

myieye · 2024-10-30T16:23:47Z

backend/FwLite/LcmCrdt/CrdtMiniLcmApi.cs

+        var snapshot = await dataModel.GetEntitySnapshotAtTime(commitHybridDate.DateTime.AddSeconds(1), entry.Id);
+        var before = snapshot?.Entity.DbObject as Entry;
+        ArgumentNullException.ThrowIfNull(before);
+        //workaround to avoid syncing senses, which are not in the snapshot


So we basically have to replicate this?

hahn-kev and others added 25 commits October 17, 2024 14:54

refactor diff code out of the sync service

63a7a80

change terminiology

f39073f

was current/previous now before after

write test trying to sync changes between an entrys components

99a8524

implement Component API

0a01487

implement semantic domain API

33eb096

GitButler WIP Commit

c111134

Merge branch 'develop' into Prep-diff-apis

5947210

# Conflicts: # backend/FwLite/FwDataMiniLcmBridge/Api/UpdateProxy/UpdateSenseProxy.cs # backend/FwLite/FwLiteProjectSync.Tests/UpdateDiffTests.cs # backend/FwLite/LcmCrdt/CrdtMiniLcmApi.cs # backend/FwLite/MiniLcm/Models/Entry.cs

being making API to update an entry by simply passing it in

a411b00

add logging from MiniLcmApiFixture to xunit

e03f1e7

comment out version stuff and make a 2 entry update api for now

f889baa

Merge branch 'develop' into Prep-diff-apis

a4fb38c

# Conflicts: # .idea/.idea.LexBox/.idea/dataSources.xml # backend/FwLite/FwLiteProjectSync.Tests/Fixtures/SyncFixture.cs

remove version from entry

804c8fa

fix some test bugs related to complex forms getting deleted and then …

2ccf0e2

…recreated with the same Id

use locking to prevent a race condition from calling Project loader i…

f2c3eac

…nit twice

support syncing Entry.ComplexForms in addition to Components

6228935

define AddComplexFormType and RemoveComplexFormType on MiniLcmApi

80fb4c3

preseed complex form types in crdt projects

90c44b4

implement syncing entry complex form types

be5fa9e

make complex form type a record

9d4611e

test creating an entry with a generated entry

f783ca1

write sync tests for a generated entry, fix bugs

f04794e

update harmony to include the new BeforeSaveObject option

fb05ae7

add version to miniLcm models using CrdtConfig.BeforeSaveObject

2a4e139

fix bug trying to use Commit.DateTime in a query

2c4119d

implement UpdateEntry(Entry entry) on FwDataMiniLcmApi, then fix bugs

8d7f5a6

hahn-kev commented Oct 30, 2024

View reviewed changes

remove unused fields

356679d

hahn-kev commented Oct 30, 2024

View reviewed changes

use the version to check for changes having been invalidated rather t…

e402294

…han using a HashSet of versions

hahn-kev commented Oct 30, 2024

View reviewed changes

backend/FwLite/FwDataMiniLcmBridge/Api/FwDataMiniLcmApi.cs Show resolved Hide resolved

myieye reviewed Oct 30, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minilcm diff API #1181

Minilcm diff API #1181

hahn-kev commented Oct 30, 2024

hahn-kev Oct 30, 2024

myieye Oct 30, 2024

github-actions bot commented Oct 30, 2024 •

edited

Loading

hahn-kev Oct 30, 2024

hahn-kev Oct 30, 2024

hahn-kev Oct 30, 2024

myieye Oct 30, 2024

hahn-kev Oct 30, 2024

hahn-kev Oct 30, 2024

hahn-kev Oct 30, 2024

hahn-kev Oct 30, 2024

myieye left a comment

myieye Oct 30, 2024

myieye Oct 30, 2024

myieye Oct 30, 2024

myieye Oct 30, 2024

myieye Oct 30, 2024

myieye Oct 30, 2024

myieye Oct 30, 2024

myieye Oct 30, 2024

myieye Oct 30, 2024

	return Task.FromResult(ToComplexFormComponents(EntriesRepository.GetObject(complexFormComponent.ComplexFormEntryId)).Single(c => c.ComponentEntryId == complexFormComponent.ComponentEntryId));
	return Task.FromResult(ToComplexFormComponents(EntriesRepository.GetObject(complexFormComponent.ComplexFormEntryId)).Single(c => c.ComponentEntryId == complexFormComponent.ComponentEntryId && c.ComponentSenseId == complexFormComponent.ComponentSenseId));

Minilcm diff API #1181

Are you sure you want to change the base?

Minilcm diff API #1181

Conversation

hahn-kev commented Oct 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 30, 2024 • edited Loading

C# Unit Tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

myieye left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 30, 2024 •

edited

Loading