Data Tables Phase 1 (#4318)

* RN-728 Create basic data table (#4313) * Update redux dev tools config Required after Chrome update See https://github.com/reduxjs/redux-devtools/tree/main/extension#installation * Re-generate types * Add DataTable create route * Allow EditModal to have custom content, extra dialog props * Add DataTable create modal * Use snake_case in @types pkg * Fix test mock * RN-729 RN-731 RN-732 Data table frontend components (#4312) * RN-730 edit data table (#4336) * Simplify redux actions around edit modal * Data Table editing * Update create data table backend after permission group id -> name change * RN-733 Add preview to datatable (#4338) * Add fetch preview data endpoint * Reuse getColumns * Constrain permission_groups and type * Fetch preview data endpoint generates columns and rows * Add QueryClientProvider * Adjust CSS style * Add preview panel * Preview fetch button * Update yarn.lock * Wrap limit and count into query Enhance fetching preview query performance * Add migration to alter constraints * Remove BES permission check Data table server check permissions when fetching * Add DataTablePreviewRequest type It can be used by: admin-panel-server, data-table-server, and api-client * Apply DataTablePreviewRequest type to FetchPreview route * extracted reusable functions * Refactor fetch preview endpoints in admin-panel-server * Minor refactor * RN-628: Convert 'fetch' to be a viz-builder transform step (#4214) * RN-628: Override default mathjs config to treat array like data as arrays rather than matrix * RN-628: Converted 'fetch' step to the 'fetchData' transform * RN-628: Reworked viz-builder UI to just use transforms * RN-628: Added migration to convert reports fetch step to a fetchData transform * RN-628: Fixed/updated tests * RN-628: Added unit tests for expressions in fetchData * RN-628: Fixed up yarn.lock * RN-628: Fixed tests after rebase * RN-628: Allowing multi-joins * RN-628: Fixed up tests after rebase * RN-628: Fixed up tests after rebase * RN-628: Renamed migration to be more recent * RN-644: Rework report-server to pull from data-tables (#4341) * RN-644: Switch report-server to fetch from data-table-server * RN-644: Add migration to convert reports to using data-table-server * RN-644: Implemented converting project entities to child countries in Analytics and Events DataTableServices - Was previously handled by QueryBuilder in the report-server * RN-644: Updated tests in report-server * RN-644: Implemented pulling events for all data-elements if no data-elements supplied * RN-644: PR Cleanups - Improved tests - Updated buildContext to just take Row[] as new data * RN-644: Fixed up tests * RN-644: Renamed migration to a later date * No Issue - Fix project to country mapping bug (#4348) * Fix mapProjectEntitiesToCountries * Clarify project code with comment * Update http example --------- Co-authored-by: Chris Pollard <therealchrispollard@gmail.com> * Add valid property names to new config (#4354) * RN-782: Fix dataTableCode Migration (#4355) * Merge branch 'dev' into data-tables-ph1 * Fix: Wrap aggregations into object (#4361) * Remove 'data-table-api' from url when creating apiClient baseUrls * Convert data group code to string (#4362) * Fix data table content overflow (#4367) * RN-768 RN-769 RN-770 RN-771 RN-783 Add data element data group metadata and rewrite vizs(#4350) * Fix data table ph1 (#4366) * Convert external database connections to drop list (#4371) * RN-688: Remove Output Context (#4375) * RN-688: Remove Output Context * update rawDataExport test * Fix issues found in RN-729 (#4379) * Add permission check on externalDatabaseConnections endpoint As discussed, users who have permissions to the external database connection should have read access to view the connections, but edit/delete permissions for an external database connection are BES Admin only * Use Autocomplete from ui-components The difference is the custom Autocomplete component in admin-panel will fetch by search term for options, which didn't allow to show the list when an option is selected. * Remove mock up externalDatabaseConnectionCode * Add delete button for external database connections * Fix RN-783 testing issues (#4376) * Set lookup even with no row Previously if no row, the lookup won't set, which will block fetching metadata using '=@all.dataElements'. Link to issue: https://linear.app/bes/issue/RN-783#comment-744f2fe2 * Remove unused options metadata Link to issue: https://linear.app/bes/issue/RN-783#comment-984ca258 * Fix unit tests * RN-802 Default parameters on preview (#4387) * RN-818 Diff default startdate (#4389) * Restore default startDate behaviour * Update tests * Use defaults again * Update packages/data-table-server/src/dataTableService/services/AnalyticsDataTableService.ts Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com> * Update packages/data-table-server/src/dataTableService/services/AnalyticsDataTableService.ts Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com> * Update * DRY --------- Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com> * Assign new aggregations in migration (#4391) * RN-773 Rewrite viz using entity relations (#4374) * Organise files for clarity * Minor: rethrow same error to keep stack * Minor: add more detail to error messages * Refactor: move tsutils/types.ts into @tupaia/types * Add ajvValidate convenience fn * Refactor: use ajv validation in middleware * Fix: context applies on fully joined dataset rather than new data from fetch * Remove insertNumberOfFacilitiesColumn transform * Remove minItems: 1 requirement Was a passing thought on an improvement, but the tests tell me otherwise * Migrate reports using insertNumberOfFacilitiesColumn * RN-809 Rewrite WISH Vizes (#4386) * Update wish baseline reports * Rewrite WISH matrix reports * Rewrite matrices with multiple data elements * Update migration * RN-810 Convert RH orgUnitCodeToName Reports (#4392) * Add entities table fetch to matrix reports * Include more reports * Update 20230222233428-RewriteVizsUsedDataElementCodeToName-modifies-schema.js (#4393) * Prevent fetchData error when entityCodes undefined (#4395) * Prevent fetchData error when entityCodes undefined * Fix expression * Fix some bugs for data tables found in demo (#4396) * Add missing import JsonEditor * Make ancestorType parameter optional in entity_relations data-table * Fix issue-11 in rn-783 (#4398) & Empty joins should still join tables * Add exitOnNoData flag to fetchData (#4399) * Fix request schema (#4400) * RN-783 [fix]: Reworked fetchDataElementMetadata in DhisService to pull category option combos and combine with data element name (#4403) * fix orgunit replace for events (#4401) * Add column sort to fix Palau matrix viz (#4407) * Fixup broken date offset vizes (#4406) * update individual transforms (#4405) * Fix hierarchy field on change error * RN-831 Fix wrong variable name on OrganisationUnitCodesField * RN-819 Convert additional reports (#4412) * Refactor: rename type for clarity * Add entity_attributes data table type * Cleanup: remove orgUnitAttribute function * Cleanup: remove orgUnitCodeToId function * Fix type issue * Add entity_attributes DT * RN-156: Added startDate and endDate selectors to the viz-builder (#4408) --------- Co-authored-by: Biao Li <lbit123@outlook.com> Co-authored-by: Biao Li <31789355+biaoli0@users.noreply.github.com> Co-authored-by: Rohan Port <59544282+rohan-bes@users.noreply.github.com> Co-authored-by: Chris Pollard <therealchrispollard@gmail.com> Co-authored-by: Rohan Port <rohan@beyondessential.com.au>
beyondessential · Apr 4, 2023 · 8529846 · 8529846
1 parent bbef7a6
commit 8529846
Show file tree

Hide file tree

Showing 268 changed files with 7,657 additions and 4,221 deletions.
diff --git a/.eslintrc b/.eslintrc
@@ -47,7 +47,8 @@
         "packages/report-server/**",
         "packages/server-boilerplate/**",
         "packages/supserset-api/**",
-        "packages/tsutils/**"
+        "packages/tsutils/**",
+        "packages/types/**"
       ],
       "extends": "@beyondessential/ts",
       "parserOptions": {

diff --git a/.prettierignore b/.prettierignore
@@ -0,0 +1,3 @@
+# Ignore auto-generated files
+packages/types/src/schemas/schemas.ts
+packages/types/src/types/models.ts
diff --git a/packages/admin-panel-server/examples.http b/packages/admin-panel-server/examples.http
@@ -174,3 +174,33 @@ Authorization: {{authorization}}
         }
     }
 }
+
+### Fetch Data Table Preview Data
+POST {{baseUrl}}/fetchDataTablePreviewData
+content-type: {{contentType}}
+Authorization: {{authorization}}
+
+{
+  "previewConfig": {
+    "code": "data_lake_db_test",
+    "type": "sql",
+    "config": {
+      "sql": "SELECT * FROM analytics WHERE entity_code LIKE :entityCode",
+      "externalDatabaseConnectionCode": "DATA_LAKE_DB",
+      "additionalParams": [
+        {
+          "name": "entityCode",
+          "config": {
+            "type": "string"
+          }
+        }
+      ]
+    },
+    "runtimeParams": {
+      "entityCode": "AU_SA%"
+    },
+    "permission_groups": [
+      "*"
+    ]
+  }
+}
diff --git a/.../src/__tests__/viz-builder/dashboardVisualisation/DashboardVisualisationExtractor.test.ts b/.../src/__tests__/viz-builder/dashboardVisualisation/DashboardVisualisationExtractor.test.ts
@@ -33,33 +33,9 @@ describe('DashboardVisualisationExtractor', () => {
     });
 
     describe('getReport() - draftReport', () => {
-      it('throws error if viz does not have data.fetch', () => {
-        const extractor = new DashboardVisualisationExtractor(
-          { code: 'viz', data: { transform: [] }, presentation: {} },
-          yup.object(),
-          draftReportValidator,
-        );
-
-        const getReport = () => extractor.getReport();
-
-        expect(getReport).toThrow('fetch is a required field');
-      });
-
-      it('throws error if viz does not have data.fetch.dataElements or data.fetch.dataGroups', () => {
-        const extractor = new DashboardVisualisationExtractor(
-          { code: 'viz', data: { fetch: {}, transform: [] }, presentation: {} },
-          yup.object(),
-          draftReportValidator,
-        );
-
-        const getReport = () => extractor.getReport();
-
-        expect(getReport).toThrow('Requires "dataGroups" or "dataElements"');
-      });
-
       it('throws error if viz does not have data.transform', () => {
         const extractor = new DashboardVisualisationExtractor(
-          { code: 'viz', data: { fetch: { dataElements: ['BCD1'] } }, presentation: {} },
+          { code: 'viz', data: {}, presentation: {} },
           yup.object(),
           draftReportValidator,
         );
@@ -72,7 +48,7 @@ describe('DashboardVisualisationExtractor', () => {
       it('throws error if viz does not have code', () => {
         const extractor = new DashboardVisualisationExtractor(
           {
-            data: { fetch: { dataElements: ['BCD1'] }, transform: [] },
+            data: { transform: [] },
             presentation: {},
           },
           yup.object(),
@@ -88,7 +64,7 @@ describe('DashboardVisualisationExtractor', () => {
         const extractor = new DashboardVisualisationExtractor(
           {
             code: 'viz',
-            data: { fetch: { dataElements: ['BCD1'] }, transform: [] },
+            data: { transform: [] },
             presentation: {},
           },
           yup.object(),
@@ -100,9 +76,6 @@ describe('DashboardVisualisationExtractor', () => {
         expect(report).toEqual({
           code: 'viz',
           config: {
-            fetch: {
-              dataElements: ['BCD1'],
-            },
             transform: [],
           },
         });
@@ -296,11 +269,6 @@ describe('DashboardVisualisationExtractor', () => {
           code: 'viz',
           name: 'My Viz',
           data: {
-            fetch: {
-              dataElements: ['BCD1', 'BCD2'],
-              organisationUnits: ['$requested', 'TO'],
-              startDate: '20210101',
-            },
             aggregate: ['SUM_EACH_WEEK'],
             transform: ['keyValueByDataElementName'],
           },
@@ -324,12 +292,6 @@ describe('DashboardVisualisationExtractor', () => {
       expect(report).toEqual({
         code: 'viz',
         config: {
-          fetch: {
-            dataElements: ['BCD1', 'BCD2'],
-            organisationUnits: ['$requested', 'TO'],
-            startDate: '20210101',
-            aggregations: ['SUM_EACH_WEEK'],
-          },
           transform: ['keyValueByDataElementName'],
           output: {
             type: 'bar',
@@ -347,9 +309,6 @@ describe('DashboardVisualisationExtractor', () => {
           code: 'viz',
           name: 'My Viz',
           data: {
-            fetch: {
-              dataElements: ['BCD1', 'BCD2'],
-            },
             transform: ['keyValueByDataElementName'],
           },
           presentation: {
@@ -372,9 +331,6 @@ describe('DashboardVisualisationExtractor', () => {
       expect(report).toEqual({
         code: 'viz',
         config: {
-          fetch: {
-            dataElements: ['BCD1', 'BCD2'],
-          },
           transform: ['keyValueByDataElementName'],
           output: {
             type: 'bar',
@@ -392,9 +348,6 @@ describe('DashboardVisualisationExtractor', () => {
           code: 'viz',
           name: 'My Viz',
           data: {
-            fetch: {
-              dataElements: ['BCD1', 'BCD2'],
-            },
             transform: ['keyValueByDataElementName'],
           },
           presentation: {
@@ -417,9 +370,6 @@ describe('DashboardVisualisationExtractor', () => {
       expect(report).toEqual({
         code: 'viz',
         config: {
-          fetch: {
-            dataElements: ['BCD1', 'BCD2'],
-          },
           transform: ['keyValueByDataElementName'],
           output: {
             type: 'rowsAndColumns',
@@ -437,9 +387,6 @@ describe('DashboardVisualisationExtractor', () => {
           code: 'viz',
           name: 'My Viz',
           data: {
-            fetch: {
-              dataElements: ['BCD1', 'BCD2'],
-            },
             transform: ['keyValueByDataElementName'],
           },
           presentation: {
@@ -477,9 +424,6 @@ describe('DashboardVisualisationExtractor', () => {
           code: 'viz',
           name: 'My Viz',
           data: {
-            fetch: {
-              dataElements: ['BCD1', 'BCD2'],
-            },
             transform: ['keyValueByDataElementName'],
           },
           presentation: {
@@ -501,9 +445,6 @@ describe('DashboardVisualisationExtractor', () => {
         report: {
           code: 'viz',
           config: {
-            fetch: {
-              dataElements: ['BCD1', 'BCD2'],
-            },
             transform: ['keyValueByDataElementName'],
             output: {
               type: 'bar',

diff --git a/packages/admin-panel-server/src/app/createApp.ts b/packages/admin-panel-server/src/app/createApp.ts
@@ -25,6 +25,8 @@ import {
   FetchMapOverlayVisualisationRoute,
   FetchReportPreviewDataRequest,
   FetchReportPreviewDataRoute,
+  FetchDataTablePreviewDataRequest,
+  FetchDataTablePreviewDataRoute,
   ImportDashboardVisualisationRequest,
   ImportDashboardVisualisationRoute,
   SaveDashboardVisualisationRequest,
@@ -36,10 +38,10 @@ import {
   UserRoute,
   ImportMapOverlayVisualisationRequest,
   ImportMapOverlayVisualisationRoute,
-  FetchAggregationOptionsRequest,
-  FetchAggregationOptionsRoute,
   FetchTransformSchemasRequest,
   FetchTransformSchemasRoute,
+  FetchDataTableBuiltInParamsRequest,
+  FetchDataTableBuiltInParamsRoute,
 } from '../routes';
 import { authHandlerProvider } from '../auth';
 
@@ -82,6 +84,14 @@ export function createApp() {
       'dashboardVisualisation/:dashboardVisualisationId',
       handleWith(FetchDashboardVisualisationRoute),
     )
+    .post<FetchDataTablePreviewDataRequest>(
+      'fetchDataTablePreviewData',
+      handleWith(FetchDataTablePreviewDataRoute),
+    )
+    .get<FetchDataTableBuiltInParamsRequest>(
+      'fetchDataTableBuiltInParams',
+      handleWith(FetchDataTableBuiltInParamsRoute),
+    )
     .get(
       'export/dashboardVisualisation/:dashboardVisualisationId',
       handleWith(ExportDashboardVisualisationRoute),
@@ -117,10 +127,6 @@ export function createApp() {
       'mapOverlayVisualisation/:mapOverlayVisualisationId',
       handleWith(FetchMapOverlayVisualisationRoute),
     )
-    .get<FetchAggregationOptionsRequest>(
-      'fetchAggregationOptions',
-      handleWith(FetchAggregationOptionsRoute),
-    )
     .get<FetchTransformSchemasRequest>(
       'fetchTransformSchemas',
       handleWith(FetchTransformSchemasRoute),

diff --git a/packages/admin-panel-server/src/routes/FetchAggregationOptionsRoute.ts b/packages/admin-panel-server/src/routes/FetchAggregationOptionsRoute.ts
diff --git a/packages/admin-panel-server/src/routes/FetchDataTableBuiltInParamsRoute.ts b/packages/admin-panel-server/src/routes/FetchDataTableBuiltInParamsRoute.ts
@@ -0,0 +1,24 @@
+/*
+ * Tupaia
+ * Copyright (c) 2017 - 2021 Beyond Essential Systems Pty Ltd
+ *
+ */
+
+import { Request } from 'express';
+
+import { Route } from '@tupaia/server-boilerplate';
+
+export type FetchDataTableBuiltInParamsRequest = Request<
+  Record<string, unknown>,
+  Record<string, unknown>,
+  Record<string, unknown>,
+  { dataTableType: string }
+>;
+
+export class FetchDataTableBuiltInParamsRoute extends Route<FetchDataTableBuiltInParamsRequest> {
+  public async buildResponse() {
+    const { dataTableType } = this.req.query;
+
+    return this.req.ctx.services.dataTable.getBuiltInParameters(dataTableType);
+  }
+}
diff --git a/packages/admin-panel-server/src/routes/FetchDataTablePreviewDataRoute.ts b/packages/admin-panel-server/src/routes/FetchDataTablePreviewDataRoute.ts
@@ -0,0 +1,27 @@
+/*
+ * Tupaia
+ * Copyright (c) 2017 - 2021 Beyond Essential Systems Pty Ltd
+ *
+ */
+
+import { Request } from 'express';
+
+import { Route } from '@tupaia/server-boilerplate';
+import type { DataTablePreviewRequest } from '@tupaia/types';
+
+export type FetchDataTablePreviewDataRequest = Request<
+  Record<string, never>,
+  Record<string, unknown>,
+  {
+    previewConfig: DataTablePreviewRequest;
+  },
+  Record<string, never>
+>;
+
+export class FetchDataTablePreviewDataRoute extends Route<FetchDataTablePreviewDataRequest> {
+  public async buildResponse() {
+    const { previewConfig } = this.req.body;
+
+    return this.req.ctx.services.dataTable.fetchPreviewData(previewConfig);
+  }
+}
diff --git a/packages/admin-panel-server/src/routes/FetchReportPreviewDataRoute.ts b/packages/admin-panel-server/src/routes/FetchReportPreviewDataRoute.ts
@@ -29,6 +29,8 @@ export type FetchReportPreviewDataRequest = Request<
   {
     entityCode: string;
     hierarchy: string;
+    startDate?: string;
+    endDate?: string;
     permissionGroup?: string;
     previewMode?: PreviewMode;
     dashboardItemOrMapOverlay: DashboardItemOrMapOverlayParam;
@@ -39,22 +41,20 @@ export class FetchReportPreviewDataRoute extends Route<FetchReportPreviewDataReq
   public async buildResponse() {
     this.validate();
 
-    const { entityCode, hierarchy, permissionGroup } = this.req.query;
+    const { entityCode, hierarchy, startDate, endDate, permissionGroup } = this.req.query;
     const { testData = null } = this.req.body;
 
     const reportConfig = this.getReportConfig();
 
-    return this.req.ctx.services.report.testReport(
-      {
-        organisationUnitCodes: entityCode as string,
-        hierarchy: hierarchy as string,
-        permissionGroup: permissionGroup as string,
-      },
-      {
-        testData,
-        testConfig: reportConfig,
-      },
-    );
+    const parameters: Record<string, string> = { hierarchy, organisationUnitCodes: entityCode };
+    if (startDate) parameters.startDate = startDate;
+    if (endDate) parameters.endDate = endDate;
+    if (permissionGroup) parameters.permissionGroup = permissionGroup;
+
+    return this.req.ctx.services.report.testReport(parameters, {
+      testData,
+      testConfig: reportConfig,
+    });
   }
 
   private validate = () => {

diff --git a/packages/admin-panel-server/src/routes/index.ts b/packages/admin-panel-server/src/routes/index.ts
@@ -8,7 +8,8 @@ export * from './dashboardVisualisations';
 export * from './mapOverlayVisualisations';
 export * from './FetchHierarchyEntitiesRoute';
 export * from './FetchReportPreviewDataRoute';
-export * from './FetchAggregationOptionsRoute';
 export * from './FetchReportSchemasRoute';
+export * from './FetchDataTablePreviewDataRoute';
+export * from './FetchDataTableBuiltInParamsRoute';
 export * from './UploadTestDataRoute';
 export * from './UserRoute';