Closes #2396: Add finer-grained scheduling. #2426

emtwo · 2018-03-29T21:04:14Z

This patch includes the following:

UI changes, including a datepicker for stop date (unlike the design initially presented in issue Add Finer-grained scheduling #2396
New database fields and data migration for them
- Note that I don't delete the old schedule field in the migration in case we discover a bug after this change is deployed. Maybe it would be safer to remove that field after this feature is released and stable for a bit.
Update to scheduling logic based on new fields
Existing test fixes and new tests
Manually tested data migration
Refactored most suggestions by code climate

emtwo · 2018-04-05T21:38:12Z

@arikfr I'd say this PR is ready for review now. Are you the right person to ask?

Note that I didn't make all of the code climate suggested changes because some of them didn't seem necessary or useful. But I'm happy to make the changes if they are a requirement.

emtwo · 2018-04-05T21:41:34Z

Adding some more up-to-date screenshots here.

kocsmy · 2018-04-05T21:57:40Z

Looks very good at first glance, I think the UI is much more intuitive now. Thank you for this!

A suggestion for better picker for the day:

<div class="btn-toolbar">
  <div class="btn-group" data-toggle="buttons">
    <label class="btn btn-xs btn-default"><input type="radio" name="options" id="option1">Mon</label>
    <label class="btn btn-xs btn-default"><input type="radio" name="options" id="option2">Tue</label>
  </div>
</div>

emtwo · 2018-04-09T18:40:59Z

Thanks for the recommendation @kocsmy! I've updated the day picker to be more similar to your suggestion

arikfr

Thank you for this great feature and for the detailed pull request :-)

I did a quick review to touch a few fundamental things. I will do another pass at this when I'm done with a few other things, but wanted to start the conversation.

arikfr · 2018-04-12T13:16:27Z

client/app/components/queries/schedule-dialog.js

    link($scope) {
-      $scope.refreshOptions =
-        clientConfig.queryRefreshIntervals.map(interval => ({ value: String(interval), name: `Every ${durationHumanize(interval)}` }));


We need to maintain this functionality... the Redash admin should be able to define what intervals are acceptable.

If it makes it complex with the new method to support arbitrary intervals, we can settle for letting the admin define whether minutes and/or hours based frequencies are OK.

I think it would be possible and not too complicated to only show allowable intervals specified by the admin. The only concern is that the way it works now, the admin would have a much longer list to show that 1, 2, 3....X hours/days/weeks are acceptable here: https://github.com/getredash/redash/blob/master/redash/settings/__init__.py#L205

I think it would be easier to use a set of allowable frequencies as you suggested.

arikfr · 2018-04-12T13:20:24Z

migrations/versions/abba12491b72_add_new_scheduling_columns.py

+    op.add_column('queries', sa.Column('schedule_until', sa.DateTime(timezone=True), nullable=True))
+    op.add_column('queries', sa.Column('schedule_day_of_week', sa.String(length=10), nullable=True))
+    op.add_column('queries', sa.Column('schedule_interval', sa.String(length=10), nullable=True))
+    op.add_column('queries', sa.Column('schedule_time', sa.String(length=5), nullable=True))


Have you considered using the options field of the query or to turn the schedule field into a dictionary (JSON) as well to hold the extra information?

This will save us the need to run a downtime requiring migrations (adding columns) and also give flexibility in the future to change how things are stored without much "drama".

emtwo · 2018-04-19T14:49:08Z

@arikfr Thanks for the feedback! I've addressed the two comments you've made and added new commits.

jezdez · 2018-06-19T18:55:27Z

@emtwo Would you mind rebasing this on top of master?

emtwo · 2018-10-04T17:00:12Z

@arikfr This PR has been rebased and updated to use React. It's ready for another lookover. Thanks!

arikfr · 2018-10-05T13:51:00Z

client/app/components/queries/ScheduleDialog.js

+import { react2angular } from 'react2angular';
+import React from 'react';
+import PropTypes from 'prop-types';
+import { Modal } from 'react-bootstrap';


We're standardizing on using Ant's components instead of Bootstrap (already using them for date inputs and in @washort's pull requests). In this case you should probably use Ant's Modal.

emtwo · 2018-10-09T19:36:30Z

@arikfr updated to Ant

* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes #927, #928: Schema refresh improvements. * Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes #939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Co-authored-by: Alison <github@bankofknowledge.net> Schema Improvements Part 2: Add data source config options. Adding BigQuery schema drawer with data types and samples.

* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes #927, #928: Schema refresh improvements. * Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes #939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Schema Improvements Part 2: Add data source config options. Adding BigQuery schema drawer with data types and samples. Add empty migration to replace the removed schedule_until migration Add merge migration. Co-authored-by: Alison <github@bankofknowledge.net>

* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes #927, #928: Schema refresh improvements. * Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes #939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Schema Improvements Part 2: Add data source config options. Adding BigQuery schema drawer with data types and samples. Add empty migration to replace the removed schedule_until migration Add merge migration. Co-authored-by: Alison <github@bankofknowledge.net> Co-authored-by: Jannis Leidel <jannis@leidel.info>

* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes #927, #928: Schema refresh improvements. * Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes #939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Schema Improvements Part 2: Add data source config options. Adding BigQuery schema drawer with data types and samples. Add empty migration to replace the removed schedule_until migration Add merge migration. Fix spacing issue with data scanned value in query execution metadata. Increase schema refresh timeout. Co-authored-by: Alison <github@bankofknowledge.net> Co-authored-by: Jannis Leidel <jannis@leidel.info>

* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes #927, #928: Schema refresh improvements. * Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes #939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Schema Improvements Part 2: Add data source config options. Adding BigQuery schema drawer with data types and samples. Add empty migration to replace the removed schedule_until migration Add merge migration. Fix spacing issue with data scanned value in query execution metadata. Increase schema refresh timeout. Remove old migrations. Co-authored-by: Alison <github@bankofknowledge.net> Co-authored-by: Jannis Leidel <jannis@leidel.info>

emtwo force-pushed the schedule branch 5 times, most recently from 10b95a1 to c8ebe77 Compare April 5, 2018 21:26

emtwo changed the title ~~[WIP - Not ready for review] Closes #2396: Add finer-grained scheduling.~~ Closes #2396: Add finer-grained scheduling. Apr 5, 2018

RichardLitt added Frontend Backend labels Apr 6, 2018

emtwo mentioned this pull request Apr 9, 2018

Add Finer-grained scheduling mozilla/redash#187

Closed

arikfr reviewed Apr 12, 2018

View reviewed changes

emtwo force-pushed the schedule branch 4 times, most recently from 110a5fe to 50242e6 Compare April 18, 2018 02:30

emtwo force-pushed the schedule branch from 50242e6 to 473ba20 Compare October 3, 2018 15:17

emtwo mentioned this pull request Oct 3, 2018

Port schedule_until feature to upstream mozilla/redash#427

Closed

emtwo force-pushed the schedule branch 2 times, most recently from d20811f to 7b116dc Compare October 4, 2018 15:31

arikfr reviewed Oct 5, 2018

View reviewed changes

jezdez assigned emtwo Oct 9, 2018

emtwo force-pushed the schedule branch from 7b116dc to 7222616 Compare October 9, 2018 19:17

emtwo mentioned this pull request Oct 10, 2018

Closes #427: Migrate schedule_until into new json schedule field. mozilla/redash#591

Closed

oliviertassinari mentioned this pull request Feb 14, 2020

[RFC] Integrate the pickers repository back into the main mono-repository mui/material-ui#19706

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Closes #2396: Add finer-grained scheduling. #2426

Closes #2396: Add finer-grained scheduling. #2426

emtwo commented Mar 29, 2018 •

edited

Loading

emtwo commented Apr 5, 2018 •

edited

Loading

emtwo commented Apr 5, 2018

kocsmy commented Apr 5, 2018 •

edited

Loading

emtwo commented Apr 9, 2018 •

edited

Loading

arikfr left a comment

arikfr Apr 12, 2018

emtwo Apr 17, 2018

arikfr Apr 12, 2018

emtwo commented Apr 19, 2018

jezdez commented Jun 19, 2018

emtwo commented Oct 4, 2018

arikfr Oct 5, 2018

emtwo commented Oct 9, 2018

Closes #2396: Add finer-grained scheduling. #2426

Closes #2396: Add finer-grained scheduling. #2426

Conversation

emtwo commented Mar 29, 2018 • edited Loading

emtwo commented Apr 5, 2018 • edited Loading

emtwo commented Apr 5, 2018

kocsmy commented Apr 5, 2018 • edited Loading

emtwo commented Apr 9, 2018 • edited Loading

arikfr left a comment

Choose a reason for hiding this comment

arikfr Apr 12, 2018

Choose a reason for hiding this comment

emtwo Apr 17, 2018

Choose a reason for hiding this comment

arikfr Apr 12, 2018

Choose a reason for hiding this comment

emtwo commented Apr 19, 2018

jezdez commented Jun 19, 2018

emtwo commented Oct 4, 2018

arikfr Oct 5, 2018

Choose a reason for hiding this comment

emtwo commented Oct 9, 2018

emtwo commented Mar 29, 2018 •

edited

Loading

emtwo commented Apr 5, 2018 •

edited

Loading

kocsmy commented Apr 5, 2018 •

edited

Loading

emtwo commented Apr 9, 2018 •

edited

Loading