Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transform bug fixes #109

Merged
merged 46 commits into from
Oct 19, 2021
Merged

Conversation

annie3431
Copy link
Contributor

@annie3431 annie3431 commented Oct 14, 2021

Signed-off-by: Annie Lee leeyun@amazon.com

Description

  • When creating transform jobs, support the options to aggregate by min and max for fields with data type of date.
  • Fix the bug during create transform job: When adding data filter to source index, the preview data for transformation does not apply the filter.
  • Add verification to search source index with data filter before moving to step 2.
  • Update transform name verification process to check if it is empty before checking for duplicate name.
  • Update the add transform using "Group by date histogram" option to separate into "fixed interval" and "calendar interval" so that fixed interval time units can take custom interval values.
  • Update the "Source index" field in create transform and create rollup page so that when user enters value including the wildcard character *, the suggested option will not add 1 more * character.
  • Move some shared constants and helper methods between rollup and transform to a higher level file.
  • Enlarge the code editor components for source index filter and scripted metrics.

Screenshots

Options to aggregate by min and max for fields with data type of date.

image

Applying source index filter example using customer_gender = FEMALE

Filter content
image

Sample data applying filter
image

Verification of source index filter

Invalid filter
image

Inline error and danger toast message
image

Verify that name is defined before moving to step 2

image

Update to "Group by date histogram" workflow

When clicking on "add" button of a date type field (no change)
image

Click on "Group by date histogram"
image

View of "Fixed interval" and timeunit options
image
image

View of "Calendar interval" and timeunit options
image
image

Source index wildcard

image

image

Enlarge code editor components

Source index filter
image

Scripted metrics
image

Issues Resolved

#96

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Annie and others added 27 commits April 30, 2021 16:46
* Release 1.1.0.0 (opensearch-project#83)

* Release 1.1.0.0

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

* Correct copyright notices

* UI fixes for new ISM UI (opensearch-project#84)

* Removes X icon from action/transition flyout footer next to cancel

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds tooltips to edit/delete icon buttons on the draggable action/transition components

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Removes overlay for flyout so clicking outside doesn't close flyout and removes X close button on top right

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds edit/delete tooltips for state component and removes underline from hovering states

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Changes state flyout from X Close to Cancel

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds JSON editor for allocation action

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds isValid to action interface and implements in actions

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Correctly removes unused rollover keys from rollover object, handles uncontrolled inputs and NaN minDocs

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds error for state name already existing in policy

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes duplicated actions when editing as previously we had a new ID and matching logic didn't find an action

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes other action inputs that can return NaN and creates a no conditions option that is the default

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates snapshot

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Correctly show danger toast on update/create failures

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes rollup action nesting multiple ism_template keys

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes timeout/retry settings turning into uncontrolled inputs and NaN value

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Removes text transformation on inputs and updates isValid methods and passes to form labels

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes allocation and rollup adding UIAction properties to policy JSON on updates

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates release notes w/ new PR changes

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Wraps actions/transitions in states component and updates default rollup to include ism_rollup key

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes some small UI touchups/issues for new ISM UI (opensearch-project#85)

* Moves cancel secondary button next to primary on action/transition flyout

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Removes default ISM template and increases width of empty prompt

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds punctuation

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds back X icon next to cancel for state flyout

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Makes inputs in flyout full width, updates help text, and adds some spacing

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates transition default to not include conditions empty object, and on change transition to delete conditions when selecting none

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Moves edit button out of policy settings content panel and on to global page for view policy

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates release note

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes broken link

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes small issues on new ISM UI (opensearch-project#88)

* Fixes small issues on new ISM UI

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates snapshots

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates release notes

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates workflows to trigger on 1.* branches

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com>
@codecov-commenter
Copy link

codecov-commenter commented Oct 14, 2021

Codecov Report

Merging #109 (2eaaaeb) into main (b0b7143) will increase coverage by 1.15%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #109      +/-   ##
==========================================
+ Coverage   50.05%   51.20%   +1.15%     
==========================================
  Files         154      155       +1     
  Lines        4387     4388       +1     
  Branches      682      685       +3     
==========================================
+ Hits         2196     2247      +51     
+ Misses       2035     1985      -50     
  Partials      156      156              
Impacted Files Coverage Δ
...gement-dashboards-plugin/public/utils/constants.ts 100.00% <0.00%> (ø)
...lugin/public/pages/CreateRollup/utils/constants.ts 100.00% <0.00%> (ø)
...eRollup/components/RollupIndices/RollupIndices.tsx 81.57% <0.00%> (ø)
...m/components/DefineTransforms/DefineTransforms.tsx 23.07% <0.00%> (ø)
...m/components/TransformIndices/TransformIndices.tsx 12.24% <0.00%> (ø)
...m/components/TransformOptions/TransformOptions.tsx 0.00% <0.00%> (ø)
...mponents/IndexFilterPopover/IndexFilterPopover.tsx 0.00% <0.00%> (ø)
...ainers/CreateTransformForm/CreateTransformForm.tsx 13.55% <0.00%> (ø)
...ners/DefineTransformsStep/DefineTransformsStep.tsx 9.09% <0.00%> (ø)
...nels/ScriptedMetricsPanel/ScriptedMetricsPanel.tsx 0.00% <0.00%> (ø)
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0b7143...2eaaaeb. Read the comment docs.

Signed-off-by: Annie Lee <leeyun@amazon.com>
@annie3431 annie3431 marked this pull request as ready for review October 15, 2021 23:25
@annie3431 annie3431 requested a review from downsrob October 18, 2021 18:37
thalurur
thalurur previously approved these changes Oct 18, 2021
Signed-off-by: Annie Lee <leeyun@amazon.com>
@annie3431 annie3431 merged commit 6f304e3 into opensearch-project:main Oct 19, 2021
@annie3431 annie3431 deleted the transform-bug-fix branch October 19, 2021 21:34
lobdelle added a commit that referenced this pull request Nov 5, 2021
* Fixes editing rollup delay (#82)

* Fixes editing the delay amount and displaying the schedule for rollups

Signed-off-by: Clay Downs <downsrob@amazon.com>

* Bumps version to 1.2 (#103)

* Bumps version to 1.2

* Changes test workflows to follow Dashboards 1.x

Signed-off-by: Clay Downs <downsrob@amazon.com>

* Fixes flakey tests (#105)

* Fixes flakey cypress test 'rollup enable and disable test'

* Fixes flakey unit test 'can delete a rollup job'

Signed-off-by: Robert Downs <downsrob@amazon.com>

* Fixed Transforms geo_point bug and boolean render (#93)

* Fixed Transforms geo_point bug and boolean render
* Set default DefineTransforms table render behavior to JSON.stringify
* Updated DefineTransforms null cell value check

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Transform bug fixes (#109)

* Rename plugin helper file

* Update TransformIndices.tsx

* Backport commits from main (#90)

* Release 1.1.0.0 (#83)

* Release 1.1.0.0

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

* Correct copyright notices

* UI fixes for new ISM UI (#84)

* Removes X icon from action/transition flyout footer next to cancel

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds tooltips to edit/delete icon buttons on the draggable action/transition components

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Removes overlay for flyout so clicking outside doesn't close flyout and removes X close button on top right

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds edit/delete tooltips for state component and removes underline from hovering states

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Changes state flyout from X Close to Cancel

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds JSON editor for allocation action

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds isValid to action interface and implements in actions

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Correctly removes unused rollover keys from rollover object, handles uncontrolled inputs and NaN minDocs

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds error for state name already existing in policy

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes duplicated actions when editing as previously we had a new ID and matching logic didn't find an action

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes other action inputs that can return NaN and creates a no conditions option that is the default

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates snapshot

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Correctly show danger toast on update/create failures

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes rollup action nesting multiple ism_template keys

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes timeout/retry settings turning into uncontrolled inputs and NaN value

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Removes text transformation on inputs and updates isValid methods and passes to form labels

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes allocation and rollup adding UIAction properties to policy JSON on updates

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates release notes w/ new PR changes

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Wraps actions/transitions in states component and updates default rollup to include ism_rollup key

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes some small UI touchups/issues for new ISM UI (#85)

* Moves cancel secondary button next to primary on action/transition flyout

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Removes default ISM template and increases width of empty prompt

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds punctuation

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Adds back X icon next to cancel for state flyout

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Makes inputs in flyout full width, updates help text, and adds some spacing

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates transition default to not include conditions empty object, and on change transition to delete conditions when selecting none

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Moves edit button out of policy settings content panel and on to global page for view policy

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates release note

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes broken link

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Fixes small issues on new ISM UI (#88)

* Fixes small issues on new ISM UI

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates snapshots

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates release notes

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

* Updates workflows to trigger on 1.* branches

Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>

Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com>

* Support data filter when viewing sample data

* Update TransformOptions.tsx

* Add TransformService.test.ts

* Draft of date histogram panel

* Add helper method and move import

* Split calendar and fixed interval to 2 panels

* Add rendering test

* Add close popover test

* Update cypress-workflow.yml

* Update links.yml

* Update unit-tests-workflow.yml

* Update TransformOptions.tsx

* Update tests

* Update DateHistogramPanel.test.tsx

* Make scripted metrics editor larger

* Expand code editor related panels

* Undo changes to workflow

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Update TimeAggregation.tsx

* Update DefineTransforms.tsx

undo changes to style

* Clean up code

* Update snapshot

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Refactor the order of checking if name is defined before checking for duplicate name

* Update CreateTransformForm.tsx

* Update DefineTransforms.tsx

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Update DefineTransforms.tsx

Signed-off-by: Annie Lee <leeyun@amazon.com>

* Update DefineTransforms.tsx

* Update DefineTransforms.tsx

* Update DefineTransforms.tsx

Co-authored-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>
Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com>

* Fix the bugs in visual UI for legacy notification (#111)

* Transforms unit and cypress tests (#92)

* Transform cypress tests (#21)

* Added Cypress tests for Transforms UI

* License update

* Unit tests initial commit

* Added jest tests for Transforms, details, edit and create

* Sample transform fix

* Fixed cypress test transform and sample data load order

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Test fix debug

* Simplified unit test

* Updated transforms cypress test comments and success criteria

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Fixed quotation mark issue in test

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Removed debug mapping and index objects from unit test

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Cypress test improvements and completed edit transform unit test

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Integ test delete fix and EditTransforms jest test fix

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Restoring enable disable page refresh delay to fix failing integ test

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Resolve test constants conflict

Signed-off-by: Eric Lobdell <lobdelle@amazon.com>

* Fix pagination of managed indices page (#113)

* Disables jitter for managed_indices_spec Cypress tests (#122)

Signed-off-by: Robert Downs <downsrob@amazon.com>

Co-authored-by: Clay Downs <89109232+downsrob@users.noreply.github.com>
Co-authored-by: Annie Lee <71157062+leeyun-amzn@users.noreply.github.com>
Co-authored-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>
Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com>
Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants