feat: custom calculations (DHIS2-13871)

[martinkrulltott]

Implements DHIS2-13871

Design spec

Relates to dhis2/data-visualizer-app#2232

---

Key features

1. Adds support for listing, creating and editing calculations in the Data dimension modal
2. Fetches data elements that can be searched and filtered - @edoardo 🙏
3. Supports drag-n-drop from the dimension list and within the formula field - @jenniferarnesen 🙏
4. Avoids stale cached results from the backend by generating and using a new edi_cache prop - @janhenrikoverland 🙏

---

New lingo

EDI: Expression Dimension Item, commonly referred to as a "calculation".
formula: The actual math formula for the EDI, known in the backend as expression.

---

Description

The DataDimension component now supports listing, creating and editing EDIs. This consists of several parts:

The Data Dimension modal

1. Adds a new + Calculation button at the bottom of the modal for creating new EDIs (opens the new calculations modal),
2. The dimension list includes EDIs and can filter to view only EDIs by using the new Calculations option in the data type filter.
3. Listed EDIs (both selected and non-selected) have an edit button that opens the new calculations modal

The new Calculation modal

1. Fetches and lists available data elements and data element operands for selection, which can also be searched and filtered, to be used in a formula (curtesy of @edoardo)
2. Supports drag-and-drop from the dimension list into the formula field and within the formula field itself (curtesy of @jenniferarnesen)
3. The formula supports client-side validation for some basic mistakes with user-friendly error messages, which is followed by a more rigorous backend validation (that however unfortunately doesn't provide useful error message)

Metadata strategy

All metadata needed for the separate parts of the formula in an EDI are handled by the CalculationModal internally, i.e. no separate metadata needs to be passed in when the CalculationModal is opened. This alleviates the consuming apps (namely DV) as they won't need to fetch and store this metadata (which was initially the idea, but turned out to be quite messy).

Parsing a formula / expression

As a formula is created as separate parts in the CalculationsModal (as an array) and the backend format for expression is a simple string, there are two new helper functions for handling this: parseExpressionToArray and parseArrayToExpression. The former uses a regex (curtesy of ChatGPT!) for splitting the expression into different parts and maps the parts with provided metadata (which is fetched internally, as per the previous section above).

Cache handling / edi_cache

The backend caches requests based on the provided params and thus disregards if an EDI has been updated. To avoid returning a stale cached response after updating an EDI a special edi_cache param are sent with each request that changes with the content of the EDI's formula. This is created by sorting a visualisation's expressions, joining them and hashing it all using crypto-js/sha1 (curtesy of @janhenrikoverland)

---

BREAKING CHANGE

The data dimension modal now return Expression Dimension Items, which require the consuming app to provide the correct metadata for these items.

---

TODO

• remove formula validation when the formula is dirty
• refactor ’apiValidateExpression’
• error handling for all requests, show the alert bar + wrap everything after await to only continue if the response is ok
  - [ ] Handle deleting an EDI that is used in an AO - delete is handled by the backed and should always be allowed
• New design
• Fetch data elements
• Search / filters
• Infinite scrolling
• Parse string -> components (opening) and components -> string (saving)
• Fetch metadata for the data elements used within the expression
• Display names for dimensions in the calculations in the formula field
• Double-click to add
• Drag-n-drop
• <number> - dnd with "edit mode"

---

Known issues

Numbered list for internal use. italic = fixed

1. The admin user can't save EDIs (missing permissions, temporarily using user system now instead) - enable F_EXPRESSION_DIMENSION_ITEM in the user app
2. Requests after the expression of an EDI is changed will return values based on the old/outdated expression - fixed with edi_cache
3. EDIs are missing dimensionItemType when opening an AO (temporarily hardcoded for now instead) - fixed by backend
4. EDIs are missing name when the shortName setting is used - fixed by backend
5. Data elements used in an expression is missing metadata (i.e. no way to map an id to a name) - metadata has been added
6. expression value missing from metadata when opening an AO - metadata has been added
7. Trying to delete an EDI that is used by a visualization throws an error - delete is now allowed
8. Opening a saved AO with an EDI and changing the name of the EDI results in the old name still being displayed in the Transfer list (due to outdated metadata in the consuming app's store)
9. Dragging a data element downwards also triggers scrolling in the data element list. A similar problem was fixed in LL.

---

Screenshots

New Calculations data type and new + Calculation button

New calculations modal that opens on top of the data dimension modal

Creating a formula with drag-n-drop and/or click/double-click

creating.a.formula.mov

Client-side validation

Searching and filtering

[dhis2-bot]

🎉 This PR is included in version 25.0.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀