[SIP-127] User Preferences

[villebro]

[SIP] Proposal for User Preferences

Motivation

Currently it's possible to customize various aspects of Superset:

• Color palettes
• Enable/disable thumbnails
• Default viz type
• Number formats and currencies
• etc

However, these are configured per deployment, meaning users can't override the globally defined defaults (note: Thumbnails can be enabled/disabled via a toggle switch in the chart/dashboard page when the FF is enabled, which isn't really the right place for it). In addition, the lack of user-specific configurability has made it difficult to introduce typically available customizability, like

• Timezone support (local vs fixed timezone)
• Light/dark mode
• Replacing colors in charts with decals/patterns for visually impaired

Proposed Change

To make Superset slightly less one-size-fits-all, we propose introducing a settings page where new user-specific settings can be easily introduced and changed per user. This would make it possible to change the functionality and appearance of Superset per user, without having to impose the same for all.

New or Changed Public Interfaces

A new config would be introduced which defines the default user settings. These could then be overridden per user, via a User Preferences page. The preferences would be stored in the metastore, and would be exposed via the API for normal CRUD operations. The API would be hosted on the /api/v1/user_preferences/ resource with typical CRUD verbs, and could be queried to retrieve a user's full set of preferences during page initialization, and then stored in Redux. This would be a move away from bootstrap data to make the initial html payload leaner. This also makes it easier to update the preference state during runtime without having to refresh the page.

Due to the simple data structure and limited need for querying individual preference keys, the full preference data would be stored in the key_value table, with a version id:

{
  "version": 1,
  "revision": 10,
  "preferences": {
    "timeZone": "PST",
    "somePreferenceState": {
      "key1": "value1",
      "key2": 123,
      "booleanKey": true,
    },
  }
}

The payload structure would be a TypedDict in the backend, making the key-value pairs strongly typed:

class SomePreferenceState(TypedDict):
    key1: str,
    key2: int
    booleanKey: bool

class UserPreferences(TypedDict):
    version: int
    revision: int
    timeZone: str
    somePreferenceState: SomePreferenceState

The version key would leave open the option to migrate keys in the future, if breaking changes need to be made. The revision key would indicate the revision of the preference state (this would be incremented on each update). The other keys (timeZone and somePreferenceState in the example) would be reserved for individual preferences. To protect against race conditions on updates, the preference object would be locked with KeyValueDistributedLock. This would ensure that only a single thread is updating it at a time. Note, that collisions in this state are highly unexpected, but are annoying to debug as they can cause weird side-effects.

Draft of User Preferences page

We should add the settings that we already know we'll need, even if we don't implement them in the POC. This will help us plan and organize the interface to be extended with additional settings later.

Migration Plan and Compatibility

The feature would be hidden behind a new feature flag until the user preferences page becomes stable. After that, the original feature flag would be removed, with new experimental preferences being introduced via an experimental flag to ensure they're not exposed to users before they're stable.

Rejected Alternatives

Adding light/dark/auto mode, timezone etc to the config, which takes effect for all users.

Pilot

As a pilot for this, I propose implementing one of the following:

1. Accessibility through decals

As discussed in #25379, ECharts has good support for adding decals to chart series to improve readability for users for whom the current color coding scheme doesn't work well (e.g. color blindness). By adding an option to the User Preferences page, we could render at least all ECharts charts with decals, making them more widely accessible to users.

2. Timezone support

a first version of the UI, and add user customizable timezone support, as Superset currently only works with naive or single timezone timestamps. As many databases work in UTC, it's often painful for users to convert UTC to whichever timezone they're in. While it's possible to force a certain timezone for a dataset/chart, enforcing a single timezone for all users can be difficult, especially if the users are distributed across multiple timezones.

I did some hacking on adding timezone support to the chart data endpoint a while ago, but decided against implementing it until it can be made generally available via a per-user customizable setting. See here: #23511

[rusackas]

Another needed example would be enabling accessibility features (e.g. ECharts decals/patterns) since few would want to turn this on deployment-wide, but any org would want their users to opt into such capabilities.

[villebro]

Another needed example would be enabling accessibility features (e.g. ECharts decals/patterns) since few would want to turn this on deployment-wide, but any org would want their users to opt into such capabilities.

Oh, super good example - I'll add it to the summary 👍

[michael-s-molina]

Thank you for the SIP @villebro. This is a much needed feature.

I'm always trying to dive deeper during analysis phase, even if our first version will be just a subset of the whole. I think it's important to have a vision of where we want to go and avoid structural decisions that block that vision because they were made considering only the subset. For that reason, I think it would be good if the SIP includes:

• A draft of the User Preferences page. We could add the settings that we already know we'll need, even if we don't implement them in the POC. This will help us plan and organize the interface to be extended with additional settings later. @kasiazjc
• The proposed API changes. We generally don't do this upfront and that's why many of our APIs are poorly designed. This is simply specifying the new or changed endpoints, params, payload, response, etc. Something like this.
• How the frontend should access this information. I believe we're going to use Redux here, so it would be good to see where this information will be placed and how it will be modeled in the Redux state.

I'm happy to collaborate with you and @kasiazjc on this effort as I think it's a very valuable feature for users.

[villebro]

thanks @michael-s-molina - I added an initial proposal for the API, state management and frontend handling of the preference object.

[rusackas]

Think this is ready for a DISCUSS thread?

[michael-s-molina]

@kasiazjc @villebro Can we add the following to the SIP? I'm available if we want to schedule a meeting to do this.

A draft of the User Preferences page. We could add the settings that we already know we'll need, even if we don't implement them in the POC. This will help us plan and organize the interface to be extended with additional settings later. @kasiazjc

[villebro]

@kasiazjc @villebro Can we add the following to the SIP? I'm available if we want to schedule a meeting to do this.

A draft of the User Preferences page. We could add the settings that we already know we'll need, even if we don't implement them in the POC. This will help us plan and organize the interface to be extended with additional settings later. @kasiazjc

@michael-s-molina I've added a section for this in the proposal. I also added the decal-based accessibility feature as an alternative for a POC, as it feels like it would be fairly easy to do, and would help us reach new users that may previously have been unable to use Superset.

@kasiazjc do you think you'd be able to work on a draft for the Preferences page? This would be a huge help, as that's probably the main thing we're missing right now.

[john-bodley]

@villebro the one issue I have with custom user preferences—depending on the actual preference—is that a chart or dashboard shared amongst users could be perceived differently based on their user preferences.

For example if users are using the local time zone and discussing an anomaly in a chart, then the time of the anomaly would be different if they were based in different time zones. Similarly different color palettes could result in misinterpretation of the data when two or more users are discussing the "blue line" (or similar).

I think—from a data transparency/community perspective—consistency is vital, thus only a subset of the proposed user preferences are actually applicable. Maybe it makes more sense if the actual settings are discussed individually.

[kasiazjc]

@kasiazjc @villebro Can we add the following to the SIP? I'm available if we want to schedule a meeting to do this.

A draft of the User Preferences page. We could add the settings that we already know we'll need, even if we don't implement them in the POC. This will help us plan and organize the interface to be extended with additional settings later. @kasiazjc

@michael-s-molina I've added a section for this in the proposal. I also added the decal-based accessibility feature as an alternative for a POC, as it feels like it would be fairly easy to do, and would help us reach new users that may previously have been unable to use Superset.

@kasiazjc do you think you'd be able to work on a draft for the Preferences page? This would be a huge help, as that's probably the main thing we're missing right now.

Actually settings page is something that I've been trying to advocate for in Preset, but never had the resources. So happy to contribute :)

[villebro]

Update from design meeting today (@michael-s-molina @rusackas @kasiazjc @justinpark @villebro present):

@kasiazjc shared first mocks:

Proposal 1 (user/global tabs on top):

Proposal 2 (nav on left hand side):

Quick summary of discussion:

• Regular users would only see the user preferences
• Admins should be able to edit global defaults through this UI rather than having to make the changes in superset_config.py. These would be shown in a separate section with same options as user preferences.
• As we're expecting lots of preferences to be added over time, we should introduce a global search that makes it possible to quickly find the relevant preferences
• For the UI, it would be preferable to create a new React component that can be used to design the new version of the Explore Control Panel. This way we could introduce global search in the Control Panel, too.
• The schema for preferences should probably be stored in a big blob, either in the key-value store or the user object, and support the following features:
  • versioning: make it possible to migrate to new versions with ease, similar to how the chart migration tool works
  • be able to mark deprecated preferences
  • support deep nesting. This will be especially important on the Explore Control Panel, where complex schemas will make it possible to fully leverage the ECharts schema for high customizability of charts

The work can roughly be split up in the following major tasks:

• Design: next steps to see if we can add deep nesting to the design @kasiazjc
• Component design: See if a JSON schema can be created that makes it possible to render the preference section/control panel. @rusackas @michael-s-molina @justinpark
• Backend design: User preference blob schema, migration + persistence in backend. @villebro

Ping @anamitraadhikari who may be interested in also contributing to the FE work.

[rusackas]

@villebro think this is ready for a VOTE?

[mattitoo]

Sorry If I am late to the conversation, but I have this question: Is there a way to disable a preference in the user preferences? Like, we do absolutely not want people to set "Thumbnail", because this means creating screenshots of all dashboards (right?). This would create a very heavy load on the database (and presumably costs).
Other such cases might appear, e.g. where an instance does not want to have ppl change the timezone.

[villebro]

@mattitoo thumbnails will still remain a deployment-wide config. So it will only be available if it's been enabled. However, similar to how it currently works, users would be able to disable thumbnails from the User Preferences if it's available if they don't want them. So in other words, the Thumbnails switch would be removed from the Welcome page, and moved to User Preferences.