DataStore update queue order not respected

[raybotha]

Before opening, please confirm:

• I have searched for duplicate or closed issues.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React Native

Amplify APIs

DataStore

Amplify Categories

api

Environment information

aws-amplify: ^3.3.26 => 3.3.26

Describe the bug

The queue for updates consistently doesn't seem to work, updates to a record seem to be disregarded completely if a previous update/save is being synced to the server.

Example is of updating a record on text field change (on each change query by ID and save).

Expected behavior

With a single network-connected client there should be no conflicts between server and client store. The latest update to a record should take priority and not be discarded or overwritten by previous updates, especially not from the same client.

Reproduction steps

Update a record rapidly and query the results as in code snippet.

Code Snippet

  useEffect(() => {
    if (active.id) {
      DataStore.query(Record, active.id).then(loaded => {
        setName(loaded.name)
        console.log(`Querying: ${loaded.name}`)
      })
    }
  }, [active.id])

  const save = useCallback(async () => {
    const newName = name || `Record ${new Date().toLocaleDateString()}`

    if (active.id) {
      const record = await DataStore.query(Record, active.id)
      if (record.name !== newName) {
        const new = await DataStore.save(
          Record.copyOf(record, updated => {
            updated.name = newName
          })
        )
        console.log(`Updating: ${record.name} -> ${new.name}`)
      }
    } else {
      const new = await DataStore.save(
        new Record({ name: newName })
      )
      active.setId(new.id)
      console.log(`Created ${new.name}`)
    }
  }, [active, name])

  useEffect(() => {
    if (name) {
      save()
    }
  }, [name, save])

Log output

 LOG  Querying: John Smith
 LOG  Updating: John Smith -> John Smit
 LOG  Updating: John Smit -> John Smi
 LOG  Updating: John Smi -> John Sm
 LOG  Updating: John Sm -> John S
 LOG  Updating: John S -> John
 LOG  Updating: John  -> John
# Pausing for a few seconds
 LOG  Updating: John Smit -> JohnD
 LOG  Updating: JohnD -> JohnDo
 LOG  Updating: JohnDo -> JohnDoe
# Pausing for a few seconds
 LOG  Updating: JohnD -> JohnDo
 LOG  Updating: JohnDo -> JohnD
 LOG  Updating: JohnD -> JohnDe
 LOG  Updating: JohnDe -> JohnDee
 LOG  Updating: JohnDee -> JohnDeer
 LOG  Updating: JohnDeer -> JohnDeere
# Pausing for a few seconds
 LOG  Updating: JohnDo -> JohnDeer

[iartemiev]

Hey @raybotha, thanks for opening the issue and providing a detailed description. This will be fixed in #8000

[iartemiev]

I was able to reproduce this issue with https://www.npmjs.com/package/aws-amplify/v/3.3.27-unstable.19 specifically with the code you have here, where name is the value for a controlled component/input. Rapidly typing does not always result in the final value being persisted.

We'll need to explore this a bit more. Could you please elaborate on this use case so we can better understand it? Normally, I wouldn't expect to send the input with each stroke but would be debouncing or going off of another event altogether before sending the network request.

[raybotha]

I was able to reproduce this issue with https://www.npmjs.com/package/aws-amplify/v/3.3.27-unstable.19 specifically with the code you have here, where name is the value for a controlled component/input. Rapidly typing does not always result in the final value being persisted.

We'll need to explore this a bit more. Could you please elaborate on this use case so we can better understand it? Normally, I wouldn't expect to send the input with each stroke but would be debouncing or going off of another event altogether before sending the network request.

We could try debouncing or using an event such as changing screens to save, however our users can often see 2-3s or upwards of latency. The concern is then that a user will modify a record within the few seconds it takes for the network request to complete. This would update the local store temporarily and give the impression that a record was updated without the change actually being persisted. Or if connectivity is lost, we haven't tested this but presumably it would also result in lost data.

Rapidly typing does not always result in the final value being persisted.

Yes only the first typed letter (or backspace) is actually persisted, even though DataStore shows the change as persisted initially (local store presumably), another query after a few seconds reveals only the first request's change to have been effected.

[raybotha]

I was able to reproduce this issue with https://www.npmjs.com/package/aws-amplify/v/3.3.27-unstable.19 specifically with the code you have here, where name is the value for a controlled component/input. Rapidly typing does not always result in the final value being persisted.

We'll need to explore this a bit more. Could you please elaborate on this use case so we can better understand it? Normally, I wouldn't expect to send the input with each stroke but would be debouncing or going off of another event altogether before sending the network request.

Adding some latency artificially helps with demonstrating the potential problems, if you happen to be close to the AWS region. I'm quite far from the AWS region we're running against, our users are distributed too, and I'm not aware of a way to reduce latency to accomodate quicker updates.

[iartemiev]

I will explore solutions for this use case specifically too, but let me know if debouncing or performing a DataStore.save on another event, e.g., onEndEditing on the clientside helps

[iartemiev]

I want to get some more context about your application and usage patterns to see if we can help you come up with a working solution.

Will the same Record ever be modified by multiple users at the same time? Or just a single user that owns the record? How many such fields will your application have? How many models in the schema? Roughly how many users are you expecting (hundreds? thousands? hundreds of thousands?) etc. Please share whatever info you feel comfortable sharing publicly.

[raybotha]

I want to get some more context about your application and usage patterns to see if we can help you come up with a working solution.

Will the same Record ever be modified by multiple users at the same time? Or just a single user that owns the record? How many such fields will your application have? How many models in the schema? Roughly how many users are you expecting (hundreds? thousands? hundreds of thousands?) etc. Please share whatever info you feel comfortable sharing publicly.

One record would likely only be modified by one user. There's about ten models linked with keys at the moment, and each has about 5 to 10 fields. We're expecting hundreds of users in the short terms, scaling to thousands after several months.

[raybotha]

I want to get some more context about your application and usage patterns to see if we can help you come up with a working solution.
Will the same Record ever be modified by multiple users at the same time? Or just a single user that owns the record? How many such fields will your application have? How many models in the schema? Roughly how many users are you expecting (hundreds? thousands? hundreds of thousands?) etc. Please share whatever info you feel comfortable sharing publicly.

One record would likely only be modified by one user. There's about ten models linked with keys at the moment, and each has about 5 to 10 fields. We're expecting hundreds of users in the short terms, scaling to thousands after several months.

Roughly half of those fields would be liable to be updated frequently/quickly.

[iartemiev]

@raybotha have you tried changing the conflict resolution strategy to Optimistic Concurrency? I think that will work much better for your use case, especially if you're not expecting multiple users to be updating the same records.

You can change the conflict resolution strategy for your entire API or on a per-model basis via the Amplify CLI

> amplify update api
? GraphQL
? Walkthrough all configurations
(Select the same auth config that you already have)
? Do you want to configure advanced settings for the GraphQL API? 
    Yes, I want to make some additional changes.
? Configure additional auth types? No
? Enable conflict detection? **Yes**
? Select the default resolution strategy **Optimistic Concurrency**
? Do you want to override default per model settings? (y/N) 

I still suggest debouncing changes in your inputs before performing a DataStore.save to reduce the number of network requests, but it should work either way.

[raybotha]

@raybotha have you tried changing the conflict resolution strategy to Optimistic Concurrency? I think that will work much better for your use case, especially if you're not expecting multiple users to be updating the same records.

You can change the conflict resolution strategy for your entire API or on a per-model basis via the Amplify CLI

> amplify update api
? GraphQL
? Walkthrough all configurations
(Select the same auth config that you already have)
? Do you want to configure advanced settings for the GraphQL API? 
    Yes, I want to make some additional changes.
? Configure additional auth types? No
? Enable conflict detection? **Yes**
? Select the default resolution strategy **Optimistic Concurrency**
? Do you want to override default per model settings? (y/N) 

I still suggest debouncing changes in your inputs before performing a DataStore.save to reduce the number of network requests, but it should work either way.

Thanks for the update, using optimistic concurrency does seem to work as desired. It would likely solve the issue for our current use case. We'll add deboucing anyway as a matter of good practice.

[iartemiev]

Great! Is it ok to close this issue?

[raybotha]

Great! Is it ok to close this issue?

I think so yes.

[raybotha]

This issue randomly returned and I had to run the CLI command again, it could be related to using the Amplify Admin UI.

[raybotha]

Is the admin UI incompatible with optimistic concurrency?

[iartemiev]

@raybotha AdminUI will currently revert your conflict resolution strategy back to Automerge after you perform changes to the data model. The team is aware of this issue and is tracking it internally. We will be modifying the behavior of AdminUI to keep whichever conflict resolution strategy has been configured on that API.

As a temporary workaround, you can manually set the conflict resolution back to Optimistic Concurrency via the CLI following a change to the data model in AdminUI. Alternatively, you can update the conflict resolution strategy in the AWS AppSync Console in each mutation resolver.

[raybotha]

@raybotha AdminUI will currently revert your conflict resolution strategy back to Automerge after you perform changes to the data model. The team is aware of this issue and is tracking it internally. We will be modifying the behavior of AdminUI to keep whichever conflict resolution strategy has been configured on that API.

As a temporary workaround, you can manually set the conflict resolution back to Optimistic Concurrency via the CLI following a change to the data model in AdminUI. Alternatively, you can update the conflict resolution strategy in the AWS AppSync Console in each mutation resolver.

Thanks, I thought that's what's happening. Is there an issue or somewhere to keep track of when it's fixed?

[iartemiev]

We're tracking it internally at the moment, but I'll make sure to circle back around and post an update here once we've fixed it

[raybotha]

@iartemiev Hey, any news or public issue yet for this Admin UI issue?

[github-actions]

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.