Version 4.0.1 - endless loading - manual refresh is needed to show dashboard

[Slaheddine]

Bug description

After migrating to the app version 4.0.1 and Helm Chart 0.12.11, our dashboards are not showing any more and stay in loading state without showing the result. We are obliged to manually refresh the page to have the results.

How to reproduce the bug

using docker images :

• superset 4.0.1
• superset 4.0.1-dockerize
• superset 4.0.1-websocket
  and Helm Chart : 0.12.11

Screenshots/recordings

Superset version

4.0.1

Python version

3.9

Node version

Not applicable

Browser

Not applicable

Additional context

the only logs we have :

│ superset [2024-06-03 11:31:49,932: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:507: CP ││ endingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine │

│ superset whether broker connection retries are made during startup in Celery 6.0 and above. ││ superset If you wish to retain the existing behavior for retrying connections on startup, │

│ superset you should set bro

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

@Slaheddine Merhaba! I'm here to help you with any bugs, questions, or contributions you have for the repository. Let me know how I can assist you!

To resolve the issue of dashboards not displaying correctly and remaining in a loading state indefinitely after migrating to app version 4.0.1 and Helm Chart 0.12.11, you should ensure that the hydrateDashboard function is being called correctly and that the necessary data (dashboard, charts, datasets) is being fetched and processed properly.

Here are the key points to check and ensure:

1. Data Fetching: Ensure that the useDashboard, useDashboardCharts, and useDashboardDatasets hooks are correctly fetching the required data. Check for any errors in these hooks and handle them appropriately.

2. Hydration: Ensure that the hydrateDashboard function is being called with the correct parameters. This function should be called only when readyToRender is true and isDashboardHydrated.current is false.

3. Error Handling: Ensure that any errors in fetching data are properly handled and displayed using addDangerToast.

4. Loading State: Ensure that the loading state is correctly managed and that the Loading component is displayed only when necessary. Additionally, verify that the isDashboardLoading function is correctly determining the loading state of the charts.

Here is the relevant part of the code that handles these aspects:

useEffect(() => {
  // eslint-disable-next-line consistent-return
  async function getDataMaskApplied() {
    const permalinkKey = getUrlParam(URL_PARAMS.permalinkKey);
    const nativeFilterKeyValue = getUrlParam(URL_PARAMS.nativeFiltersKey);
    const isOldRison = getUrlParam(URL_PARAMS.nativeFilters);

    let dataMask = nativeFilterKeyValue || {};
    // activeTabs is initialized with undefined so that it doesn't override
    // the currently stored value when hydrating
    let activeTabs: string[] | undefined;
    if (permalinkKey) {
      const permalinkValue = await getPermalinkValue(permalinkKey);
      if (permalinkValue) {
        ({ dataMask, activeTabs } = permalinkValue.state);
      }
    } else if (nativeFilterKeyValue) {
      dataMask = await getFilterValue(id, nativeFilterKeyValue);
    }
    if (isOldRison) {
      dataMask = isOldRison;
    }

    if (readyToRender) {
      if (!isDashboardHydrated.current) {
        isDashboardHydrated.current = true;
      }
      dispatch(
        hydrateDashboard({
          history,
          dashboard,
          charts,
          activeTabs,
          dataMask,
        }),
      );
    }
    return null;
  }
  if (id) getDataMaskApplied();
  // eslint-disable-next-line react-hooks/exhaustive-deps
}, [readyToRender]);

Ensure that the readyToRender variable is correctly set to true only when both dashboard and charts are available:

const readyToRender = Boolean(dashboard && charts);

Additionally, verify that the isDashboardLoading function is correctly determining the loading state of the charts:

export default function isDashboardLoading(charts) {
  return Object.values(charts).some(
    chart => chart.chartUpdateStartTime > (chart.chartUpdateEndTime || 0),
  );
}

If the issue persists, you may need to debug further to ensure that all necessary data is being fetched and processed correctly, and that there are no issues with the hydrateDashboard function or the data it relies on ^[1][2].

To continue talking to Dosu, mention @dosu.

[Slaheddine]

We do not have this code in our project, we are only using your docker images and your chart helm

[Slaheddine]

Any news please ?

[dosubot]

To resolve the issue of Superset dashboards staying in a loading state indefinitely after migrating to app version 4.0.1 and Helm Chart 0.12.11, you should ensure that the ON_REFRESH_SUCCESS and ON_FILTERS_REFRESH_SUCCESS actions are properly dispatched and handled in the dashboardStateReducer. These actions are responsible for setting the isRefreshing and isFiltersRefreshing states to false, which indicates that the loading process has completed.

Here is the relevant part of the dashboardStateReducer that handles these actions:

[ON_REFRESH_SUCCESS]() {
  return {
    ...state,
    isRefreshing: false,
  };
},
[ON_FILTERS_REFRESH_SUCCESS]() {
  return {
    ...state,
    isFiltersRefreshing: false,
  };
},

Ensure that these actions are correctly dispatched in your application when the data fetching or refreshing process completes. If these actions are not dispatched, the isRefreshing and isFiltersRefreshing states will remain true, causing the dashboard to stay in a loading state indefinitely.

Additionally, verify that the DashboardPage component is correctly handling the loading state and dispatching the necessary actions. The DashboardPage component should ensure that the dashboard data is properly hydrated and that the loading state is updated accordingly. Here is an excerpt from the DashboardPage.tsx file that shows how the component handles the loading state and data hydration:

useEffect(() => {
  dispatch(setDatasetsStatus(status));
}, [dispatch, status]);

useEffect(() => {
  async function getDataMaskApplied() {
    const permalinkKey = getUrlParam(URL_PARAMS.permalinkKey);
    const nativeFilterKeyValue = getUrlParam(URL_PARAMS.nativeFiltersKey);
    const isOldRison = getUrlParam(URL_PARAMS.nativeFilters);

    let dataMask = nativeFilterKeyValue || {};
    let activeTabs: string[] | undefined;
    if (permalinkKey) {
      const permalinkValue = await getPermalinkValue(permalinkKey);
      if (permalinkValue) {
        ({ dataMask, activeTabs } = permalinkValue.state);
      }
    } else if (nativeFilterKeyValue) {
      dataMask = await getFilterValue(id, nativeFilterKeyValue);
    }
    if (isOldRison) {
      dataMask = isOldRison;
    }

    if (readyToRender) {
      if (!isDashboardHydrated.current) {
        isDashboardHydrated.current = true;
      }
      dispatch(
        hydrateDashboard({
          history,
          dashboard,
          charts,
          activeTabs,
          dataMask,
        }),
      );
    }
  }
  if (id) getDataMaskApplied();
}, [readyToRender]);

Ensure that the getDataMaskApplied function is correctly fetching and applying the data mask, and that the hydrateDashboard action is dispatched when the dashboard is ready to render. This will help ensure that the dashboard loading state is properly managed and that the dashboard does not stay in a loading state indefinitely ^[1][2].

To continue talking to Dosu, mention @dosu.

[rusackas]

Nope, no updates, and seemingly nobody else experiencing this. Are you getting any errors or showing any additional logs at all?

[Slaheddine]

Changing the GLOBAL_ASYNC_QUERIES_TRANSPORT from "ws" to "polling" and downgrading to version 2.1.1 resolved the issu ..
I think there is a problem with the websocket mode.

Also, when doing the database downgrade, it gets blocked at version 90139bf715e4 because the downgrade script is trying to remove a table "metrics" which doesn't exist. We were obliged to manually create a table "metrics" so that the database downgrade script works until this revision ID: 9c2a5681ddfd.

[rusackas]

Oooh, interesting. I'm not sure who's using the websocket mode in day-to-day ops presently. Not sure if @craig-rueda @villebro or @michael-s-molina have tried it out lately.

[MalavikaN1]

We are also facing the same issue, using superset image version 3.1.1.
Any fixes for this?

[pasqualtroncone]

I'm facing somewhat the same problem after upgrading from 3.1.2 to 3.1.3. In my case, the dashboard will load the first time, but some charts do not load when filters are applied. This will happen only in Firefox.

[rusackas]

@pasqualtroncone are you getting any errors in the JS console, or in the server logs?

[max-viafoura]

We are also seeing this after upgrading from 3.1.1 to 4.0.1
Using docker to deploy it on EKS. Not seeing errors on front end but seeing a lot of Received unexpected request: GET /" on websocket pod

[max-viafoura]

Update - the above was issue with infrastructure on our end - restarting the pod resolved the messaging but did not solve a problem. Right now we can select filters and dashboard loads initially but once anyone apply filter - we see loading icon but no errors in web-dev console or on server. And data requests are getting pending result with nothing else happening:

If we reload the page - we get filtered out results

[pasqualtroncone]

@pasqualtroncone are you getting any errors in the JS console, or in the server logs?

@rusackas, nothing on the JS console or server logs, but it seems that my case is related to Firefox. It takes way too much time compared to Chrome to give back some results. At first, I thought it was an endless loop, but then the browser advised closing the tab, so the problem was clear. I solved my issue by not using Firefox. I think the cause of this comes from Pivot Table component.
For the same dashboard, Chrome takes 3 seconds, while Firefox takes over 40 seconds.

Tested on 3.1.3 and 4.0.2

[rusackas]

Hm, interesting! @betodealmeida have you run into this? You're the only avid Firefox user I know :D

[padbk]

I am using websockets and I see this issue only when SLACK_ENABLE_AVATARS is enabled..

[sadpandajoe]

I am using websockets and I see this issue only when SLACK_ENABLE_AVATARS is enabled..

@mistercrunch thoughts on this as the creator of: #27849

[mistercrunch]

No direct/quick intuition on how SLACK_ENABLE_AVATARS relates to websocket/firefox.

Have we identified this is specific to GLOBAL_ASYNC_QUERIES_TRANSPORT + FIREFOX?

I'd qualify GLOBAL_ASYNC_QUERIES_TRANSPORT as bleeding edge currently as I think few large install based use it in production but that's just a guess, we don't have hard data on it. I can say for sure we don't use GLOBAL_ASYNC_QUERIES_TRANSPORT at Preset currently but intend to do so in the nearish future as the related SIP and work takes place.

[padbk]

It was consistent for both chrome and firefox for me.

I was unaware you do not use GLOBAL_ASYNC_QUERIES_TRANSPORT for Preset. I therefore assume this means GLOBAL_ASYNC_QUERIES_TRANSPORT is not tested as part of Preset/Superset CI?

[padbk]

Can confirm this issue exists in both chrome and firefox when GLOBAL_ASYNC_QUERIES_TRANSPORT is set to either polling or ws and SLACK_ENABLE_AVATARS is set.

When SLACK_ENABLE_AVATARS is not set, both polling and ws work fine for me.

[mistercrunch]

I do not know the details about the test coverage of GLOBAL_ASYNC_QUERIES_TRANSPORT, but hear @villebro use it in his production cluster. Preset currently uses polling in its installation, and SLACK_ENABLE_AVATARS is getting rolled out progressively through a feature flag (we use split.io behind the scene to support dynamic feature flag configuration and progressive rollouts). It may be a good idea to label GLOBAL_ASYNC_QUERIES_TRANSPORT as beta or alpha.

[jorgehn52]

Hello all.

I am facing the same issue for charts loading forever with version 4.0.1 and 4.0.2, on Superset installed with helm 0.12.11.

I already created another issue for the same in this link
Also noticing that some configs require several DB inside Redis but the helm chart is only creating:

REDIS_DB=1
REDIS_CELERY_DB=0

Is this a limitation?

[gulivan]

Can confirm that disabling GLOBAL_ASYNC_QUERIES resolved the issue with infinite loading after applying filters on 4.0.2.