opendistro-for-elasticsearch-release-notes-1.12.0.md

Open Distro for Elasticsearch 1.12.0 Release Notes

Open Distro for Elasticsearch 1.12.0 is now available for download.

The release consists of Apache 2 licensed Elasticsearch version 7.10.0 and Kibana version 7.10.0. Plugins in the distribution include Alerting, Index Management, Performance Analyzer (with Root Cause Analysis Engine), Security, SQL, Machine Learning with k-NN, Job Scheduler, Anomaly Detection, Kibana Notebooks, Reports, and Gantt Chart. Also, SQL JDBC/ODBC driver, SQL CLI Client, and PerfTop (a client for Performance Analyzer) are available for download now.

Release Highlights

• You can now share reports in PDF, CSV, and PNG format with Kibana Reports. You can schedule or immediately generate reports from the Dashboard, Visualize, and Discover panels.
• We introduced a new visualization in Kibana that provides you with an ability to embed Gantt charts into dashboards to enable visualization of events, steps and tasks as horizontal bars.
• As time-series data grows to considerable sizes over time, it can slow down your aggregations, and incur a substantial storage cost. Also, the usefulness of high granularity data reduces with time. Rollups lets you summarize the data and helps you preserve useful aggregations that can be leveraged for analytics while drastically reducing storage costs.
• Hamming distance can now be used as a distance metric for similarity search. Hamming distance measures similarity by comparing two binary data strings and is commonly used in image retrieval, fraud detection, and recognizing duplicate webpages.

Release Details

Open Distro for Elasticsearch 1.12.0 includes the following breaking changes, features, enhancements, bug fixes, infrastructure, documentation, maintenance, and refactoring updates.

BREAKING CHANGES

• We have renamed all the Kibana Plugins with camelCase naming convention. You can find the full list of names here.

FEATURES

Alerting

• Allow for http method selection in custom webhook (#101)

Alerting Kibana Plugin

• Allow for http method selection in custom webhook (#90)

Index Management

• Adds support for Rollup feature (#319, #320, #321, #322, #323, #324, #336, #337, #338, #339, #340, #341, #342, #343, #344, #345, #346, #347, #348)

Index Management Kibana Plugin

• Rollup Kibana (#128)

Kibana Reports

• Lock Edit report source and Input Validation (#225)
• Use puppeteer-core with custom chromium instead of puppeteer (#222)
• Added Loading Modal to Details pages (#221)
• Add Multiselect for Reports Table (#218)
• Add Icon to Refresh Button on Landing Page (#216)
• Using request tenant info from common-utils to filter (#215)
• Adding filtering the reports based on tenants. (#214)
• Use new API to Generate Reports from Existing Definitions (#213)
• Disable api calls that uses Kibana default internal user (#212)
• More polish to visual report(pdf and png) (#211)
• Refine pdf/png report generated by puppeteer (#209)
• Remove Persistent Permissions Error Toast (#208)
• Adding support for filter by Roles (#204)
• [reports-scheduler] Add support for Elasticsearch 7.10.0 (#203)
• Not returning access details for non-admin users (#202)
• disable notification feature from UI (#198)
• Add Permissions Error Toasts and Input Validation Errors (#196)
• Updated Time Range Parsing in Report Details (#195)
• Call updateReportStatus and notification API as Kibana user (#194)
• Remove create report logic for background job (#193)
• Fixed Base Url Formation for Visualizations/Saved Searches (#192)
• Adding Validation Modal for Deleting Report Definition (#190)
• Updated logging : reduced logging size where not required to be verbose (#189)
• Updated Toast Notifications (#188)
• Fixed poller permission check (#186)
• Align url format with all other Open Distro plugins (#185)
• Add Punctuation to All Toasts (#184)
• Added user and backend role based access control to APIs (#183)
• Add unit testing for model converters (#182)
• Improve query_url and base_url input validation (#181)
• Removed deprecated APIs and cleaned up code. (#180)
• Improve typing and remove deprecated code (#179)
• Pagination support for list APIs (#178)
• Changed "roles" to "access" to support multiple access patterns. (#175)
• Added transport actions for all the APIs (#174)
• Update Monthly Report Trigger UI (#171)
• Cypress Edit Test (#170)
• Cypress Tests for Details Pages (#169)
• Using IndexManager operation directly through variables (#168)
• Added model for all REST request/response (#167)
• Add Cypress Test Framework & Create tests (#166)
• Setup github action for reports-scheduler (#164)
• Remove Modal Elements (#162)
• Improve Main Test Coverage (#161)
• Improve Coverage for Homepage Tables (#160)
• Add github action CI/CD for kibana-reports plugin (#159)
• Improve Delivery Test Coverage (#158)
• [Testing] Update existing visual/data report helper test suite (#157)
• Fix Warning on Report Details Test (#156)
• Improved Test Coverage for Report Settings (#155)
• Report definition details/Report details test coverage increase (#154)
• Remove "refresh interval" option from time selector (#153)
• Disable/Hide related UI to leave only CSV report (#152)
• Removed baseUrl as it can be created from SourceType and Source.id (#151)
• Report Trigger Jest Test Coverage (#150)
• Added JobSchedular integration to index operation APIs (#149)
• Preserve Pre-error Edit state on Invalid Update (#147)
• Adding polling and job locking API (#143)
• Update report definition UI styles (#142)
• Add email body template & Optimize notification setting UI (#141)
• Update editor selected tab and list preview style (#138)
• Update Kibana nav menu order (#137)
• Report Definition Details Display Change (#136)
• ReportDefinition and ReportInstance index operations and REST APIs cr… (#135)
• Change Selector for Visualization Reports (#133)
• Add download to reportDetails and reportDefinitionDetails (#131)
• Use session cookie for puppeteer to access url of security-enabled domain (#129)
• Improve server side input validation (#128)
• Update Jest Snapshots (#127)
• Main_utils Test Suite (#126)
• Sanitize header and footer user input (#125)
• Added settings/configuration support to plugin (#124)
• Converted some of the required classes to Kotlin (#122)
• Add context menu UI on discover (#121)
• Header & Footer Plaintext Support (#120)
• Kotlin support to the plugin added (#119)
• Added Toasts for all API actions (#116)
• Input Validation for Create Report Definition (#115)
• Update the ES query result size from default 100 to default max value 10000 (#114)
• Remove Unused Filters (#113)
• Removed test_data dependency (#112)
• Removed Unnecessary Columns/Filters (#111)
• Integrate notification to kibana-reports (#109)
• Change Trigger Types in Edit page (#107)
• Add Functionality to Report Definition Details Buttons (#105)
• Make Container Width Responsive (#104)
• UI Changes after Version Upgrade (#102)
• CSV Integration for On-demand Reports (#100)
• Use kibana server hostname for puppeteer to access pages (#99)
• Add In-Context Reporting Menu (#97)
• Create Report Definition Final UI Changes (#82)
• Report Definition Details UI Changes (#80)
• Moving the notification code to new repository (#79)
• Report Details UI Final Changes (#74)
• UI Homepage Final Review Items (#71)
• Added REST input parser respod success in JSON (#70)
• UI Fit & Finish Pre-check Changes (#69)
• Initial commit to notification plugin. (#67)
• Connect Kibana-reports plugin to reports-scheduler plugin (#63)
• Added .vscode/ to gitignore list (#60)
• Add Trimming and Header & Footer Functionality (#59)
• Hooked Edit report definition APIs (#58)
• Add Routes for Get Visualizations & Saved Search (#57)
• Build Reports Scheduler ES plugin (#56)
• Connect Report Definition Details Backend (#53)
• Connect Report Details Backend (#52)
• Connect Front & Back End for Homepage & Create report definition (#51)
• APIs endpoints for data reports. (#50)
• Add Edit Report Definition UI (#48)
• CRUD APIs for report and report definition (#47)
• Update Kibana-Reporting-Design-Proposal.md (#44)
• Add Report Definition Details UI (#43)
• Create Report Final UI (#42)
• Update Landing Page UI (#37)
• update image headers (#35)
• Stream reports to client (#30)
• Update UI Snapshot (#29)
• Add Report Details UI (#28)
• API to generate report in PDF/PNG format (#16)
• First Typescript Migration (#13)
• First UI Unit Test (#10)
• Reporting UI (#9)

Kibana Visualizations

• Rebase gantt chart to main as a sub directory (#1)

k-NN

• Support for hamming bit distance in custom scoring (#267)

Performance Analyzer

• Publish shard state metrics (#212)
• Add Master Throttling Collector Metrics (#227)
• Publish fault detection metrics (#218)
• Adding new GC info collector to the scheduled metrics collector(#225)

SQL

• Add count() support for PPL (#894)
• Add like, isnull, isnotnull function in PPL (#893)
• Revert Error Messages for Run/Explain (#889)
• Extend the PPL identifier defintion (#888)
• Add Error Message for Explain and Run (#872)
• Remove workbench side nav logo and fix download link (#869)
• For ODFE 1.12 change position for sql workbench plugin (remove DEFAULT_APP_CATEGORIES) (#857)
• For ODFE 1.12 change position for sql workbench plugin (#855)
• Add support for HH:mm:ss (#850)
• Support NULLS FIRST/LAST in new engine (#843)
• Support subquery in FROM clause in new engine (#822)
• Support CASE clause in new engine (#818)
• Support COUNT star and literal in new engine (#802)
• Adding example of nested() for more complex nested queries (#801)
• Adding example of nested() for more complex nested queries (#799)
• Support HAVING in new SQL engine (#798)
• Add ppl request log (#796)

Release Engineering

• Add CodeQL for scanning vulnerabilities for Pull Request in opendistro-build repository (#466)

ENHANCEMENTS

Alerting

• Run /_execute in User context (#312)
• Support filterBy in update/delete destination/monitor APIs (#311)

Alerting Kibana Plugin

• Change the position of the plugin in the side bar (#214)
• Remove an unused import after the side bar change (#216)
• Add toast notifications for more backend errors (#219)

Anomaly Detection

• Improve profile API (#298)
• Add checkpoint index retention for multi entity detector (#283)
• Stashing context for Stats API to allow users to query from RestAPI (#300)
• Add HC detector request/failure stats (#307)

Anomaly Detection Kibana Plugin

• Fix vulnerability caused by old version formik. Issue: #333 (#334)
• Change side bar position for anomly detection plugin (#335)
• Use plotly.js-dist to reduce artifact size (#340)

Index Management

• Adds support for Elasticsearch 7.10.0 (#349)

Index Management Kibana Plugin

• Change position of index-management in kibana side bar (#140)
• Kibana migration (#142)

Performance Analyzer

• Updating Default Port web-server to 9600 (#233)

Security

• Adding support for SSL dual mode (#712)
• When replacing .kibana index with multi-tenant index, create index with alias if one already does not exist (#765)
• Demo Config : Adding AD Indices to system index and creating pre-defined roles (#776)
• Add user & roles to the thread context (#798)
• Security configuration for reporting and notification plugins (#836)
• Support user injection for transport requests (#763)
• Support ES 7.10.0 (#840)
• Support certs with separate Extended Key Usage (#493)
• Adding requested tenant to the thread context transient info for consumption (#850)

SQL

• Sort field push down (#848)
• Seperate the logical plan optimization rule from core to storage engine (#836)

Release Engineering

• All roles in one statefulset, allows to choose between dedicated/non-dedicated pods for lightweight deployments in helm charts, thanks @everythings-gonna-be-alright (#453)

BUG FIXES

Alerting

• Get user info from threadcontext (#289)
• Fix filter by user.backendroles and integ tests for it (#290)
• Check empty user object for the AD monitor (#304)
• Add security system property for integTest task (#319)
• Fix failed IT cases for AD (#320)

Alerting Kibana Plugin

• Fix 2 bugs in Anomaly Detection monitor (#215)

Anomaly Detection

• Fix edge case where entities found for preview is empty (#296)
• Fix null user in detector (#301)
• Fix fatal error of missing method parseString (#302)
• Remove clock Guice binding (#305)
• Filter out empty value for entity features (#306)
• Fix for upgrading mapping (#309)
• Fix double nan error when parse to json (#310)
• Fix issue where data hole exists for Preview API (#312)
• Fix delete running detector bug (#320)
• Fix detector and feature serialization (#322)
• Moving common-utils to 1.12.0.2 (#323)

Anomaly Detection Kibana Plugin

• Show second for heatmap cell hover timestamp (#327)
• Fix sunburst chart height is 0 (#332)
• Fix two issues during recent upgrade (#339)

Index Management

• Correctly handles remote transport exceptions in rollover (#325)
• Accept request parameters in RestGetRollupAction and fix flakey tests (#353)

Index Management Kibana Plugin

• Bug fix for duplicate dimension/metrics items and deletion error (#145)

Kibana Notebooks

• Correct URL link (#55)

Kibana Reports

• Use default max size to call getAll ES API (#224)
• In-context menu download UI (#219)
• ReportInstance is missing id field (#207)
• Permission denied error for background job when security is disabled (#191) refine error handler (#187)
• Use relative url for href attribute of report source link (#173)
• Context menu doesn't show up when switching between tabs from sidebar (#172)
• Fix Timezone selection and bugs (#144)
• Create/Edit Bug Fixes (#140)
• Improve logging and error handling; Fix edit report bug; Fix header/footer rendering (#123)
• Fix small display issues in report details and report table (#85)
• Configure fixed timezone for jest testing (#163)
• Fix "open in Kibana" link issue in embedded html of email body (#148)
• Fix email recipients render in edit report definition page (#146)
• Report Details UI Fixes (#145)
• Workaround to fix table refresh, disable links for pending reports (#139)
• Landing Page Minor Issues Fix (#132)
• Create from Context Menu Fix (#130)
• Time Range Fixes (#118)
• Context Menu Fixes (#117)
• Fix enable/disable after editing schedule type (#110)
• Small UI Fixes (#108) Refactor (#103)
• Apply workaround to partially fix the plugin build issue (#96)
• Add trigger type check for delete report definition API (#77)
• Fix CSV Test Cases (#62)

Performance Analyzer

• Cache Size metric: Using the Actual Cache Values and not the Delta (#231)

Security

• Fix missing trim when parsing roles in proxy authenticator (#766)
• Fix empty password issue in upgrade from 6x to 7x (#816)
• Reject empty password in internal user creation (#818)
• Use reflection to get reduceOrder, termBytes and format due to java.lang.IllegalAccessError (#866)
• Fix for java.io.OptionalDataException that is caused by changes to User object after it is put on thread context. (#869)
• Catch and respond invalid_index_name_exception when an index with invalid name is mentioned (#865)

SQL

• Fix workbench version number for ODFE 1.12.0.0 (#903)
• Disable sorting on workbench (#900)
• Fix select all from subquery issue (#902)
• Fix for ExprValueFactory construct issue (#898)
• Fix issue result table in workbench not displaying values of boolean type (#891)
• Fix key "id" in result table issue of the workbench (#890)
• Fix workbench bugs from plugin platform upgrade (#886)
• Fix ExprCollectionValue serialization bug (#859)
• Fix issue: sort order keyword is case sensitive (#853)
• Config the default locale for gradle as en_US (#847)
• Fix bug of nested field format issue in JDBC response (#846)
• Fix symbol error and Fix SSLError when connect es. (#831)
• Bug fix, using Local.Root when format the string in DateTimeFunctionIT (#794)

Release Engineering

• Fix description of "maxUnavailable" in helm charts, thanks @webwurst (#489)

INFRASTRUCTURE

Alerting

• Add integ security tests with a limited user (#313)

Anomaly Detection

• Add multi node integration testing into CI workflow (#318)

Index Management Kibana Plugin

• Updates to github action workflows and some bug fix after upgrading to Kibana 7.10 (#139)

Kibana Notebooks

• Update plugin id to use camelcase (#51)

Performance Analyzer

• Set up jacoco for code coverage (#234)

SQL

• Revert java version in jdbc release workflow (#871)
• Fix for odbc build failure (#885)
• Fix release workflow for workbench (#868)
• Add workflow to rename and upload odbc to s3 (#865)
• Add codecov for sql plugin (#835)
• Update odbc workflow (#828)
• Updated workbench snapshots to fix broken workflow (#823)
• Updated Mac version for GitHub action build (#804)
• Fix unstable integration tests (#793)
• Update cypress tests and increase delay time (#792)

DOCUMENTATION

Alerting Kibana Plugin

• Update release notes for 1.12.0.2 (#226)

Anomaly Detection Kibana Plugin

• Update release notes (#326)
• Update release notes (#336)

Kibana Reports

• Add UX documentation, userflows, screens and mocks (#34)
• Update README with dev guide (#23)
• Update user stories (#22)
• Update Design Proposal (#18)
• Design Proposal Update (#15)

SQL

• Add identifier and datatype documentation for PPL (#873)
• Add doc for ODFE SQL demo (#826)
• Update out-of-date documentation (#820)
• Add release notes for ODFE 1.12 (#841)

MAINTENANCE

Alerting

• Adds support for Elasticsearch 7.10.0 (#300)
• Move to common-utils-1.12.0.2 version (#314)

Alerting Kibana Plugin

• Add support for Kibana 7.10.0 (#212)

Anomaly Detection

• Support ES 7.10.0 (#313)

Anomaly Detection Kibana Plugin

• Migrate entire plugin to new platform (#328)
• Upgrade to Kibana 7.10.0 (#329)

Index Management

• Uploads elasticsearch.log files from failed CI runs (#336)
• Adds support for running local cluster with security plugin enabled (#322)
• Updates integration tests to not wipe indices between each test to help reduce tests bleeding into each other (#342)
• Changes set-env command in github workflow (* Adds support for Elasticsearch 7.10.0 (#349))

Job Scheduler

• Support Elasticsearch 7.10 (#82)

Kibana Notebooks

• Upgrade to ODFE 1.12.0 (#56)

Kibana Reports

• Position change for reporting plugin in side bar (#223)
• Position change for reporting plugin in side bar (#217)
• Add support for Kibana 7.10.0 (#205)
• Migrate all Kibana server side APIs to call ES-reporting APIs (#177)
• Upgrade to ES/Kibana version v7.9.1 (#101)
• Migrate the project to be under a fixed Kibana version 7.8.0 (#55)
• Migrate Client-Side to New Platform (#41)
• Migrate server side to new kibana plugin platform (#38)
• Migrate create_report/ to Typescript (#17)
• Migrate public/main to Typescript (#14)

Kibana Visualizations

• Upgrade to ODFE 1.12.0 (#3)

k-NN

• k-NN plugin support for Elasticsearch version 7.10.0 (#271)
• Bump odfe version to 1.12 (#273)

Security

• Create release drafter (#769)
• Upgrade junit to 4.13.1 (#835)
• Updating static_roles.yml (#838)
• Security configuration cleanup for static and test resources (#841)
• Change version to 1.12.0.0 (#860)
• Upgrade github CD action to using Environment Files (#862)
• Refactor getUserInfoString (#864)
• Update 1.12 release notes (#867)
• Update 1.12 release notes (#872)
• Use StringJoiner instead of (Immutable)List builder (#877)

Security Kibana Plugin

• Add support for elasticsearch 7.10.0 (#626)

SQL

• SQL release for Elasticsearch 7.10 (#834)
• Migrate Query Workbench to new Platform (#812)
• Migrate Query Workbench to 7.10 (#840)
• Bump version number to 1.12.0.0 for [JDBC, ODBC, SQL-CLI] (#838)

REFACTORING

Alerting Kibana Plugin

• Migrate Alerting Kibana plugin to the new Kibana Platform (#209)
• Create a constant for the size of query results used for drop-down menus (#213)
• Remove Kibana icon for the ODFE category in side bar (#218)
• Fix issue that nothing is in 'Time field' dropdown when defining monitors by visual graph after the migration (#222)
• Remove 'JSON.stringify()' from constructing the body of API calls in request handlers (#223)
• Correct variable names of the response for 'getEmailAccount' and 'getEmailGroup' in the request handler (#224)
• Passing 'core.notifications' to 'Email' and 'EmailRecipients' components (#225)

Kibana Reports

• [UI] Align create report definition UI to schema & API refactor (#83)
• [UI] Align UI to the schema & API refactor - table and detail page (#78)
• Refactoring saved search reporting APIs maintenance (#73)
• Refactor schema and API [backend] (#72)

You can also track upcoming features in Open Distro for Elasticsearch by watching the code repositories or checking the project website.