Runtime: Set instance config with variables

[begelundmuller]

• Adds support for setting instance-wide config using variables.
• Variables for instance config are prefixed with rill.. For example, rill.models.default_materialize configures the instance's default model materialization.
• Only config properties that are safe for users to customize can be set with variables.
• Makes previously hard-coded config constants configurable. The new list of configurable variables is:
  • rill.download_row_limit (int, default: 200k)
  • rill.pivot_cell_limit (int, default: 2m)
  • rill.interactive_sql_row_limit (int, default: 10k)
  • rill.stage_changes (bool, default: true)
  • rill.models.default_materialize (bool, default: false)
  • rill.models.materialize_delay_seconds (int, default: 0)
  • rill.alerts.default_streaming_refresh_cron (string, default: every 10 mins)

Closes #4426

[k-anshul]

LGTM.
But I am not sure about the usage of rill. prefix here.
In general either we should allow all configuration(including connector related) to be appended with rill. as in the case of spark configurations : https://spark.apache.org/docs/latest/configuration.html#available-properties
or nothing should be appended and as in the case of duckdb.
As a user I would prefer if configurations are either rill.models.default_materialize,rill.connector.duckdb.path or models.default_materialize, connector.duckdb.path.

[begelundmuller]

LGTM. But I am not sure about the usage of rill. prefix here. In general either we should allow all configuration(including connector related) to be appended with rill. as in the case of spark configurations : https://spark.apache.org/docs/latest/configuration.html#available-properties or nothing should be appended and as in the case of duckdb. As a user I would prefer if configurations are either rill.models.default_materialize,rill.connector.duckdb.path or models.default_materialize, connector.duckdb.path.

I understand this point. One issue is that users can also provide their own variables (for example, see https://github.com/rilldata/margin-metering/blob/main/rill.yaml). So to avoid conflicts, having a fixed prefix like rill. is necessary.

Then about connector. vs. rill.connector., yeah it might make sense to change it. One thing that makes it a little tricky is that the connectors are somewhat user defined – for example rill.connector.ezcommerce-slack.bot_token=... looks a little weird with the rill. prefix, since ezcommerce-slack is not Rill system related.

[k-anshul]

I understand this point. One issue is that users can also provide their own variables (for example, see https://github.com/rilldata/margin-metering/blob/main/rill.yaml). So to avoid conflicts, having a fixed prefix like rill. is necessary.

Makes sense.

Then about connector. vs. rill.connector., yeah it might make sense to change it. One thing that makes it a little tricky is that the connectors are somewhat user defined – for example rill.connector.ezcommerce-slack.bot_token=... looks a little weird with the rill. prefix, since ezcommerce-slack is not Rill system related.

May be we can consider connector.ezcommerce-slack.bot_token as dynamically generated rill variables.