[Research] Data persistence in current dashboard plugins

[abbyhu2000]

Data persistence

There are four plugins that currently have data persistence ability in opensearch dashboard: dashboard, discover, timeline, and visualize. They are using the following services and mechanisms from opensearch_dashboard_utils plugin to achieve data persistence.

State syncing utils are a set of helpers to sync application state with URL or browser storage(when turning state: storeInSessionStore on in advanced setting, in case of overflowed URL):

1. syncState(): subscribe to state changes and push them to state storage; subscribe to state storage and push them to state container
2. storages that are compatible with syncState()

• OsdUrlStateStorage: serialize state and persist it to URL's query param in rison format; listen for state change in URL and update them back to state
• SessionStorageStateStorage: serialize state and persist it to URL's query param in session storage

3. state containers: redux-store like objects to help manage states and provide a central place to store state

Two type of persistence

There are two parts for data persistence:

1. app state (example from visualization plugin)

   1. app state storage key: '_a'

   2. app state is persistent only within the specific app, values will persist when we refresh the page, values will not be persist when we navigate away from the app

   3. for visualize app, the params are:

      1. query
      
      2. filters
      
      3. vis
      4. ui state
2. global query state

   1. global state storage key: '_g'

   2. global query state is persistent across the entire application, values will persist when we refresh the page, or when we navigate across visualize, discover, timeline or dashboard page. For example, if we set time range to last 24 hours, and refresh intervals to every 30 min, the same time range and refresh intervals will be applied if we navigate to any of the other pages.

   3. params:

      1. filters
      2. refresh intervals
      
      3. time range
      

URL breakdown & example

Global state persistence

1. In plugin.ts, during plugin setup, call createOsdUrlTracker(), listen to history changes and global state changes, then update the nav link URL. This also returns function such as onMountApp(), onUnmountedApp()

   const {
    appMounted,
    appUnMounted,
    ...
   } = createOsdUrlTracker({
    baseUrl: core.http.basePath.prepend('/app/visualize'),
    defaultSubUrl: '#/',
    storageKey: `lastUrl:${core.http.basePath.get()}:visualize`,
    navLinkUpdater$: this.appStateUpdater,
    stateParams: [
      {
        osdUrlKey: '_g',
        stateUpdate$: data.query.state$.pipe(
          filter(
            ({ changes }) => !!(changes.globalFilters || changes.time || changes.refreshInterval)
          ),
          map(({ state }) => ({
            ...state,
            filters: state.filters?.filter(opensearchFilters.isFilterPinned),
          }))
        ),
      },
    ],
   ....
   

   • when we enter the app and app is mounted, it initialize nav link by getting previously stored URL from storage instance: const storedUrl = storageInstance.getItem(storageKey). (Storage instance is a browser wide session storage instance.) Then it unsubscribes to global state$ and subscribes to URL$. The current app actively listens to history location changes. If there are changes, set the updated URL as the active URL

     function onMountApp() {
         unsubscribe();
         ...
         // track current hash when within app
               unsubscribeURLHistory = historyInstance.listen((location) => {
                     ...
                     setActiveUrl(location.hash.substr(1));
                  }
               });
            }
   

   • when we are leaving the app and app is unmounted, unsubscribe URL$ and subscribe to global state$. If the global states are changed in another app, the global state listener will still get triggered in this app even though it is unmounted, it will set the updated URL in storage instance, so next time when we enter the app, it gets the URL from the storage instance thus the global state will persist.

          function onUnmountApp() {
               unsubscribe();
               // propagate state updates when in other apps
                    unsubscribeGlobalState = stateParams.map(({ stateUpdate$, osdUrlKey }) =>
                          stateUpdate$.subscribe((state) => {
                          ...
                          const updatedUrl = setStateToOsdUrl( ... );
                          ...
                         storageInstance.setItem(storageKey, activeUrl);
                 })
               );
          }
     

2. In app.tsx, call syncQueryStateWithUrl(query, osdUrlStateStorage) to sync '_g' portion of url with global state params

   • when we first enter the app, there is no initial state in the URL, then we initialize and put the _g key into url

     if (!initialStateFromUrl) {
          osdUrlStateStorage.set<QueryState>(GLOBAL_STATE_STORAGE_KEY, initialState, {
          replace: true,
       });
     }
     

   • when we enter the app, if there is some initial state in the URL(the previous saved URL in storageInstance), then we retrieve global state from '_g' URL

   // retrieve current state from `_g` url
  const initialStateFromUrl = osdUrlStateStorage.get<QueryState>(GLOBAL_STATE_STORAGE_KEY);
  // remember whether there was info in the URL
  const hasInheritedQueryFromUrl = Boolean(
    initialStateFromUrl && Object.keys(initialStateFromUrl).length
  );
  // prepare initial state, whatever was in URL takes precedences over current state in services
  const initialState: QueryState = {
    ...defaultState,
    ...initialStateFromUrl,
  };

• if we make some changes to the global state
  1. stateUpdate$ get triggered for all other unmounted app(if we made the change in visualize plugin, then the stateUpdate$ will get triggered for dashboard, discover, timeline), then it will call setStateToOsdUrl() to set updatedURL in storageInstance so global state get updated for all unmounted app
  2. updateStorage() get triggered for currentApp to update current URL state storage, then global query state container will also be in sync with URL state storage

const { start, stop: stopSyncingWithUrl } = syncState({
  stateStorage: osdUrlStateStorage,
  stateContainer: {
    ...globalQueryStateContainer,
    set: (state) => {
      if (state) {
        // syncState utils requires to handle incoming "null" value
        globalQueryStateContainer.set(state);
      }
    },
  },
  storageKey: GLOBAL_STATE_STORAGE_KEY,
});
start();

App state persistence

• we use useVisualizeAppState() hook to instantiate the visualize app state container, which is in sync with '_a' URL

const { stateContainer, stopStateSync } = createVisualizeAppState({
        stateDefaults,
        osdUrlStateStorage: services.osdUrlStateStorage,
        byValue,
      });

• when we first enter the app, there is no app state in the URL, so we set the default states into URL in createDefaultVisualizeAppState()
  osdUrlStateStorage.set(STATE_STORAGE_KEY, initialState, { replace: true });

• when we make changes to the app state, the dirtyStateChange event emitter will get triggered, then osd state container will call updateStorage() to update the URL state storage, then state container(appState) will also be in sync with URL state storage

  const onDirtyStateChange = ({ isDirty }: { isDirty: boolean }) => {
       if (!isDirty) {
         // it is important to update vis state with fresh data
         stateContainer.transitions.updateVisState(visStateToEditorState(instance, services).vis);
       }
       setHasUnappliedChanges(isDirty);
     };
     eventEmitter.on('dirtyStateChange', onDirtyStateChange);
   ... 
   const { start, stop: stopSyncingWithUrl } = syncState({
             stateStorage: osdUrlStateStorage,
             stateContainer: {
                 ...globalQueryStateContainer,
                 set: (state) => {
                     if (state) {
                         globalQueryStateContainer.set(state);
                      }
                 },
           },
           storageKey: GLOBAL_STATE_STORAGE_KEY,
      });
      // start syncing the appState with the ('_a') url
       startStateSync();
  

• in useEditorUpdates(), we use the saved appState to load the visualize editor

Refresh

When we refresh the page, both app state and global state should persist:

1. appMounted() gets triggered for the current app, so current app subscribe to URL$
2. syncQueryStateWithUrl() gets called within app.tsx for the current app, and we are getting the global states from URL '_g', and then connectToQueryState() gets called to sync global states and state container for the current app so the current app load the saved global states in top nav
3. stateUpdate$ will get triggered for every other unmounted app, so the global states are updated for their URL in storage instance as well by calling setStateOsdUrl()
4. when we load the visualize editor, createDefaultVisualizeAppState() gets called, and it gets app state from URL '_a', and it updates appState based on URL
5. in useEditorUpdates(), it uses the updated appState to load the visualization with previous saved states

Navigate to another app

When we navigate to another app from the current app, global state should persist:

1. appUnmounted() triggered for the current app, unsubscribe to URLHistory$, and subscribe to stateUpdate$
2. appMounted() triggered for the app that we navigated to, so it unsubscribe its stateUpdate$, and subscribe to URLHistory$
3. syncQueryStateWithUrl is triggered, it then gets the saved global state from the osdurlstatestorage and set the top nav global states by using globalQueryStateContainer

Diagram

1. When first navigate to the visualize app, initialize and sync state storage and state containers

2. When we navigate to another app, the browser wide storage instance stores the last active URL for each app and also updates the URL if there are any new global state values. This ensure global data persistence across different apps. For example, if we navigate from visualize app to discover app:

Example storage instance:

3. When we made some changes to the global params, the global query state container will receive updates and push the updates to osd url state storage. Other unmounted app will update their last active URL in instance storage as well.

4. When we made some changes to the app params, the app state container will receive updates and also push the updates to osd url state storage.

5. When we refresh the page, we parse the information from the URL(_g for global state, _a for app state). We use the saved information to create new state containers and set up synchronization between state containers and state storage.

[ashwin-pc]

@abbyhu2000 This is an amazing deep-dive! I can already think of a lot of good ways to put this system to use.

@kroosh @kgcreative while this is a technical deep-dive, I just want to put this on your radar when you think about how we want make the experience across Dashboard's and its plugins to feel unified and when we want to persist information as the user navigates form one app to another.

[joshuarrrr]

@abbyhu2000 What's the plan for where we're going to add this info to make it discoverable by other devs?

[abbyhu2000]

@abbyhu2000 What's the plan for where we're going to add this info to make it discoverable by other devs?

@joshuarrrr will raise a PR and put this into root level doc folder.

[abbyhu2000]

Since the readme on the current plugin persistence implementation is merged, will close this issue.