Master to Dev Post Release 2023-11

* Fix inline aggregations

* Fix inline aggregations

…es in data-broker (#4235)

master into dev

* RN-728 Create basic data table (#4313)

* Update redux dev tools config

Required after Chrome update
See https://github.com/reduxjs/redux-devtools/tree/main/extension#installation

* Re-generate types

* Add DataTable create route

* Allow EditModal to have custom content, extra dialog props

* Add DataTable create modal

* Use snake_case in @types pkg

* Fix test mock

* RN-729 RN-731 RN-732 Data table frontend components (#4312)

* RN-730 edit data table (#4336)

* Simplify redux actions around edit modal

* Data Table editing

* Update create data table backend after permission group id -> name change

* RN-733 Add preview to datatable (#4338)

* Add fetch preview data endpoint

* Reuse getColumns

* Constrain permission_groups and type

* Fetch preview data endpoint generates columns and rows

* Add QueryClientProvider

* Adjust CSS style

* Add preview panel

* Preview fetch button

* Update yarn.lock

* Wrap limit and count into query
Enhance fetching preview query performance

* Add migration to alter constraints

* Remove BES permission check
Data table server check permissions when fetching

* Add DataTablePreviewRequest type
It can be used by: admin-panel-server, data-table-server, and api-client

* Apply DataTablePreviewRequest type to FetchPreview route

* extracted reusable functions

* Refactor fetch preview endpoints in admin-panel-server

* Minor refactor

* RN-628: Convert 'fetch' to be a viz-builder transform step (#4214)

* RN-628: Override default mathjs config to treat array like data as arrays rather than matrix

* RN-628: Converted 'fetch' step to the 'fetchData' transform

* RN-628: Reworked viz-builder UI to just use transforms

* RN-628: Added migration to convert reports fetch step to a fetchData transform

* RN-628: Fixed/updated tests

* RN-628: Added unit tests for expressions in fetchData

* RN-628: Fixed up yarn.lock

* RN-628: Fixed tests after rebase

* RN-628: Allowing multi-joins

* RN-628: Fixed up tests after rebase

* RN-628: Fixed up tests after rebase

* RN-628: Renamed migration to be more recent

* RN-644: Rework report-server to pull from data-tables (#4341)

* RN-644: Switch report-server to fetch from data-table-server

* RN-644: Add migration to convert reports to using data-table-server

* RN-644: Implemented converting project entities to child countries in Analytics and Events DataTableServices
- Was previously handled by QueryBuilder in the report-server

* RN-644: Updated tests in report-server

* RN-644: Implemented pulling events for all data-elements if no data-elements supplied

* RN-644: PR Cleanups
- Improved tests
- Updated buildContext to just take Row[] as new data

* RN-644: Fixed up tests

* RN-644: Renamed migration to a later date

* No Issue  - Fix project to country mapping bug (#4348)

* Fix mapProjectEntitiesToCountries

* Clarify project code with comment

* Update http example

---------

Co-authored-by: Chris Pollard <therealchrispollard@gmail.com>

* Add valid property names to new config (#4354)

* RN-782: Fix dataTableCode Migration (#4355)

* Merge branch 'dev' into data-tables-ph1

* Fix: Wrap aggregations into object (#4361)

* Remove 'data-table-api' from url when creating apiClient baseUrls

* Convert data group code to string (#4362)

* Fix data table content overflow (#4367)

* RN-768 RN-769 RN-770 RN-771 RN-783 Add data element data group metadata and rewrite vizs(#4350)

* Fix data table ph1 (#4366)

* Convert external database connections to drop list (#4371)

* RN-688: Remove Output Context (#4375)

* RN-688: Remove Output Context

* update rawDataExport test

* Fix issues found in RN-729  (#4379)

* Add permission check on externalDatabaseConnections endpoint

As discussed, users who have permissions to the external database connection should have read access to view the connections, but edit/delete permissions for an external database connection are BES Admin only

* Use Autocomplete from ui-components
The difference is the custom Autocomplete component in admin-panel will fetch by search term for options, which didn't allow to show the list when an option is selected.

* Remove mock up externalDatabaseConnectionCode

* Add delete button for external database connections

* Fix RN-783 testing issues (#4376)

* Set lookup even with no row
Previously if no row, the lookup won't set, which will block fetching metadata using '=@all.dataElements'. Link to issue: https://linear.app/bes/issue/RN-783#comment-744f2fe2

* Remove unused options metadata
Link to issue: https://linear.app/bes/issue/RN-783#comment-984ca258

* Fix unit tests

* RN-802 Default parameters on preview (#4387)

* RN-818 Diff default startdate (#4389)

* Restore default startDate behaviour

* Update tests

* Use defaults again

* Update packages/data-table-server/src/dataTableService/services/AnalyticsDataTableService.ts

Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com>

* Update packages/data-table-server/src/dataTableService/services/AnalyticsDataTableService.ts

Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com>

* Update

* DRY

---------

Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com>

* Assign new aggregations in migration (#4391)

* RN-773 Rewrite viz using entity relations (#4374)

* Organise files for clarity

* Minor: rethrow same error to keep stack

* Minor: add more detail to error messages

* Refactor: move tsutils/types.ts into @tupaia/types

* Add ajvValidate convenience fn

* Refactor: use ajv validation in middleware

* Fix: context applies on fully joined dataset rather than new data from fetch

* Remove insertNumberOfFacilitiesColumn transform

* Remove minItems: 1 requirement

Was a passing thought on an improvement, but the tests tell me otherwise

* Migrate reports using insertNumberOfFacilitiesColumn

* RN-809 Rewrite WISH Vizes (#4386)

* Update wish baseline reports

* Rewrite WISH matrix reports

* Rewrite matrices with multiple data elements

* Update migration

* RN-810 Convert RH orgUnitCodeToName Reports (#4392)

* Add entities table fetch to matrix reports

* Include more reports

* Update 20230222233428-RewriteVizsUsedDataElementCodeToName-modifies-schema.js (#4393)

* Prevent fetchData error when entityCodes undefined (#4395)

* Prevent fetchData error when entityCodes undefined

* Fix expression

* Fix some bugs for data tables found in demo (#4396)

* Add missing import JsonEditor

* Make ancestorType parameter optional in entity_relations data-table

* Fix issue-11 in rn-783 (#4398) & Empty joins should still join tables

* Add exitOnNoData flag to fetchData (#4399)

* Fix request schema (#4400)

* RN-783 [fix]: Reworked fetchDataElementMetadata in DhisService to pull category option combos and combine with data element name (#4403)

* fix orgunit replace for events (#4401)

* Add column sort to fix Palau matrix viz (#4407)

* Fixup broken date offset vizes (#4406)

* update individual transforms (#4405)

* Fix hierarchy field on change error

* RN-831 Fix wrong variable name on OrganisationUnitCodesField

* RN-819 Convert additional reports (#4412)

* Refactor: rename type for clarity

* Add entity_attributes data table type

* Cleanup: remove orgUnitAttribute function

* Cleanup: remove orgUnitCodeToId function

* Fix type issue

* Add entity_attributes DT

* RN-156: Added startDate and endDate selectors to the viz-builder (#4408)

---------

Co-authored-by: Biao Li <lbit123@outlook.com>
Co-authored-by: Biao Li <31789355+biaoli0@users.noreply.github.com>
Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com>
Co-authored-by: Chris Pollard <therealchrispollard@gmail.com>
Co-authored-by: Rohan Port <rohan@beyondessential.com.au>

…#4419)

* Add options back to metadata

* Fix unit test

* Convert week to date for period in psss

* minor changes

…4432)

…chData transform (#4440)

…lds in the viz-builder (#4442)