Ensure History Changes only if Resource Changes

This change introduces a new transaction command called "keep" that just
means: "Keep the resource of it is not changed in between." Unlike
"put", "keep" will not introduce a new history entry. The command "keep"
is always conditional because the detection of no changes will be done
outside of the transaction.

Closes: #777

.github/scripts/patient-identical-update.sh

@@ -0,0 +1,49 @@
+#!/bin/bash -e
+
+#
+# This script tests that an update without changes of the resource content
+# doesn't create a new history entry.
+#
+
+SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
+. "$SCRIPT_DIR/util.sh"
+
+bundle() {
+cat <<END
+{
+  "resourceType": "Bundle",
+  "type": "transaction",
+  "entry": [
+    {
+      "resource": $1,
+      "request": {
+        "method": "PUT",
+        "url": "Patient/$2"
+      }
+    }
+  ]
+}
+END
+}
+
+BASE="http://localhost:8080/fhir"
+PATIENT_IDENTIFIER="X79746011X"
+PATIENT=$(curl -sH "Accept: application/fhir+json" "$BASE/Patient?identifier=$PATIENT_IDENTIFIER" | jq -cM '.entry[0].resource')
+ID="$(echo "$PATIENT" | jq -r .id)"
+VERSION_ID="$(echo "$PATIENT" | jq -r .meta.versionId)"
+
+# Update Interaction
+RESULT=$(curl -sXPUT -H "Content-Type: application/fhir+json" -d "$PATIENT" "$BASE/Patient/$ID")
+RESULT_VERSION_ID="$(echo "$RESULT" | jq -r .meta.versionId)"
+
+test "update versionId" "$RESULT_VERSION_ID" "$VERSION_ID"
+
+# Transaction Interaction
+RESULT=$(curl -sH "Content-Type: application/fhir+json" -H "Prefer: return=representation" -d "$(bundle "$PATIENT" "$ID")" "$BASE")
+RESULT_VERSION_ID="$(echo "$RESULT" | jq -r '.entry[0].resource.meta.versionId')"
+
+test "transaction versionId" "$RESULT_VERSION_ID" "$VERSION_ID"
+
+HISTORY_TOTAL=$(curl -sH "Accept: application/fhir+json" "$BASE/Patient/$ID/_history" | jq -r '.total')
+
+test "history total" "$HISTORY_TOTAL" "1"

.github/workflows/build.yml

@@ -58,6 +58,7 @@ jobs:
         - kv
         - luid
         - metrics
+        - module-base
         - openid-auth
         - operation-graphql
         - operation-measure-evaluate-measure
@@ -620,6 +621,9 @@ jobs:
     - name: Patient Everything
       run: .github/scripts/patient-everything.sh
 
+    - name: Patient Identical Update
+      run:  .github/scripts/patient-identical-update.sh
+
   not-enforcing-referential-integrity-test:
     needs: build
     runs-on: ubuntu-22.04
@@ -1519,6 +1523,9 @@ jobs:
     - name: Patient Everything
       run: .github/scripts/patient-everything.sh
 
+    - name: Patient Identical Update
+      run:  .github/scripts/patient-identical-update.sh
+
     - name: Docker Stats
       run: docker stats --no-stream
 

.gitignore

@@ -2,5 +2,4 @@ target
 classes
 .cpcache
 .cache
-blaze-load-tests/node_modules
 .nrepl-port

docs/api.md

@@ -2,6 +2,12 @@
 
 Blaze exposes a [FHIR RESTful API][1] under the default context path of `/fhir`. The [CapabilityStatement][2] exposed under `/fhir/metadata` can be used to discover the capabilities of Blaze. Everything stated there can be considered to be implemented correctly. If not please [file an issue][3]. 
 
+## Interactions
+
+### Update
+
+Blaze keeps track over the history of all updates of each resource. However if the content of the resource update is equal to the current version of the resource, no new history entry is created. Usually such identical content updates will only cost a very small amount of transaction handling storage but no additional resource or index storage.
+
 ## Operations
 
 The following Operations are implemented:

docs/implementation/database.md

@@ -176,10 +176,10 @@ After concatenation, the strings are hashed with the [Murmur3][7] algorithm in i
 For this example, we don't use the hashed versions of the key parts except for the content-hash.
 
 | Key (search-param, type, value, id, content-hash) |
-|---|
-| gender, Patient, female, 1, 6744ed32 |
-| gender, Patient, female, 2, b7e3e5f8 |
-| gender, Patient, male, 0, ba9c9b24 |
+|---------------------------------------------------|
+| gender, Patient, female, 1, 6744ed32              |
+| gender, Patient, female, 2, b7e3e5f8              |
+| gender, Patient, male, 0, ba9c9b24                |
 
 In case one searches for female patients, Blaze will seek into the index with the key prefix (gender, Patient, female) and scan over it while the prefix stays the same. The result will be the `[id, hash]` tuples:
 * `[1, 6744ed32]` and
@@ -213,7 +213,43 @@ That tuples are further processed against the `ResourceAsOf` index in order to c
 * this node submits the transaction commands to the central transaction log
 * all nodes (inkl. the transaction submitter) receive the transaction commands from the central transaction log
 
-**TODO: continue...**
+### Transaction Commands
+
+### Create
+
+**TODO**
+
+### Put
+
+The `put` command is used to create or update a resource.
+
+#### Properties
+
+| Name          | Required | Data Type     | Description                                 |
+|---------------|----------|---------------|---------------------------------------------|
+| type          | yes      | string        | resource type                               |
+| id            | yes      | string        | resource id                                 |
+| hash          | yes      | string        | resource content hash                       |
+| refs          | no       | list          | references to other resources               |
+| if-match      | no       | number        | the t the resource to update has to match   |
+| if-none-match | no       | "*" or number | the t the resource to update must not match |
+
+### Keep
+
+The `keep` command can be used instead of a `put` command if it's likely that the update of the resource will result in no changes. In that sense, the `keep` command is an optimization of the `put` command that has to be retried if it fails.   
+
+#### Properties
+
+| Name     | Required | Data Type | Description                                                   |
+|----------|----------|-----------|---------------------------------------------------------------|
+| type     | yes      | string    | resource type                                                 |
+| id       | yes      | string    | resource id                                                   |
+| hash     | yes      | string    | the resource content hash the resource to update has to match |
+| if-match | no       | number    | the t the resource to update has to match                     |
+
+### Delete
+
+**TODO**
 
 [1]: <https://www.datomic.com>
 [2]: <https://xtdb.com>

docs/monitoring.md

@@ -4,6 +4,28 @@ It's recommended to use [Prometheus][1] and [Grafana][2] to monitor the runtime
 
 ![](monitoring/prometheus.png)
 
+## Prometheus Config
+
+A basic Prometheus config looks like this:
+
+```yml
+global:
+  scrape_interval: 15s
+
+scrape_configs:
+- job_name: 'node'
+  static_configs:
+  - targets: ['<server-ip-addr>:9100']
+  - labels:
+      instance: 'blaze'
+
+- job_name: 'blaze'
+  static_configs:
+  - targets: ['<server-ip-addr>:8081']
+  - labels:
+      instance: 'blaze'
+```
+
 ## Import the Blaze Dashboard
 
 In order to import the Blaze dashboard into your Grafana instance, please copy the contents of [blaze.json](monitoring/blaze.json) and pate it into the import dialog on the Import dashboard site:

docs/performance/fhir-search.md

@@ -46,8 +46,8 @@ curl -s "http://localhost:8080/fhir/Observation?code=http://loinc.org|$CODE&_sum
 | CCX42  | 100k    | 8310-5  |  115 k |     0.07 |  0.005 | 0.62 ² |
 | CCX42  | 100k    | 55758-7 |  1.0 M |     0.53 |  0.062 |   0.52 |
 | CCX42  | 100k    | 72514-3 |  2.7 M |     1.31 |  0.017 |   0.47 |
-| LE1080 | 1M      | 8310-5  |  1.1 M |     0.60 |  0.004 |   0.51 |
-| LE1080 | 1M      | 55758-7 | 10.1 M |     5.59 |  0.020 |   0.55 |
+| LE1080 | 1M      | 8310-5  |  1.1 M |     0.59 |  0.008 |   0.50 |
+| LE1080 | 1M      | 55758-7 | 10.1 M |     5.46 |  0.097 |   0.53 |
 | LE1080 | 1M      | 72514-3 | 27.3 M |    14.55 |  0.039 |   0.53 |
 
 ¹ time in seconds per 1 million resources, ² Because the measurement of the duration includes the whole HTTP stack, durations of small results like the 115 k includes an overhead that results in a duration more like 0.6 seconds per 1 million hits.
@@ -72,8 +72,8 @@ blazectl download --server http://localhost:8080/fhir Observation -q "code=http:
 | CCX42  | 100k    | 8310-5  |  115 k |     2.46 |  0.044 |   21.34 |           
 | CCX42  | 100k    | 55758-7 |  1.0 M |    19.74 |  0.237 |   19.60 |         
 | CCX42  | 100k    | 72514-3 |  2.7 M |    52.95 |  0.484 |   19.26 |         
-| LE1080 | 1M      | 8310-5  |  1.1 M |    26.17 |  0.186 |   22.57 |         
-| LE1080 | 1M      | 55758-7 | 10.1 M |   237.82 |  1.534 |   23.45 |         
+| LE1080 | 1M      | 8310-5  |  1.1 M |    21.26 |  0.289 |   18.34 |         
+| LE1080 | 1M      | 55758-7 | 10.1 M |   209.43 |  2.313 |   20.65 |         
 | LE1080 | 1M      | 72514-3 | 27.3 M |  1038.65 | 61.478 | 37.98 ² |
 
 ¹ time in seconds per 1 million resources, ² resource cache size (11 million) is smaller than the number of resources returned (27.3 million)
@@ -100,7 +100,7 @@ blazectl download --server http://localhost:8080/fhir Observation -q "code=http:
 | CCX42  | 100k    | 8310-5  |  115 k |     1.78 |  0.052 |   15.41 |           
 | CCX42  | 100k    | 55758-7 |  1.0 M |    14.46 |  0.177 |   14.35 |           
 | CCX42  | 100k    | 72514-3 |  2.7 M |    37.82 |  0.107 |   13.76 |
-| LE1080 | 1M      | 8310-5  |  1.1 M |    19.88 |  0.112 |   17.15 |          
+| LE1080 | 1M      | 8310-5  |  1.1 M |    14.50 |  0.161 |   12.50 |          
 | LE1080 | 1M      | 55758-7 | 10.1 M |   178.90 |  0.786 |   17.64 |          
 | LE1080 | 1M      | 72514-3 | 27.3 M |   808.84 | 55.742 | 29.58 ² |
 

modules/anomaly/src/blaze/anomaly.clj

@@ -28,6 +28,10 @@
   (identical? ::anom/not-found (::anom/category x)))
 
 
+(defn conflict? [x]
+  (identical? ::anom/conflict (::anom/category x)))
+
+
 (defn fault? [x]
   (identical? ::anom/fault (::anom/category x)))
 
@@ -70,8 +74,11 @@
    (anomaly* ::anom/fault msg kvs)))
 
 
-(defn busy [msg & {:as kvs}]
-  (anomaly* ::anom/busy msg kvs))
+(defn busy
+  ([]
+   (busy nil))
+  ([msg & {:as kvs}]
+   (anomaly* ::anom/busy msg kvs)))
 
 
 (defn- format-exception

modules/anomaly/src/blaze/anomaly_spec.clj

@@ -26,6 +26,11 @@
   :ret boolean?)
 
 
+(s/fdef ba/conflict?
+  :args (s/cat :x any?)
+  :ret boolean?)
+
+
 (s/fdef ba/fault?
   :args (s/cat :x any?)
   :ret boolean?)
@@ -67,7 +72,7 @@
 
 
 (s/fdef ba/busy
-  :args (s/cat :msg (s/nilable string?) :kvs (s/* (s/cat :k keyword? :v any?)))
+  :args (s/cat :msg (s/? (s/nilable string?)) :kvs (s/* (s/cat :k keyword? :v any?)))
   :ret ::anom/anomaly)
 
 

modules/anomaly/test/blaze/anomaly_test.clj

@@ -62,6 +62,17 @@
     (is (not (ba/anomaly? nil)))))
 
 
+(deftest conflict?-test
+  (testing "a conflict anomaly has to have the right category"
+    (is (ba/conflict? {::anom/category ::anom/conflict})))
+
+  (testing "anomalies with other categories are no conflict anomalies"
+    (is (not (ba/conflict? {::anom/category ::anom/fault}))))
+
+  (testing "nil is no conflict anomaly"
+    (is (not (ba/anomaly? nil)))))
+
+
 (deftest fault?-test
   (testing "a fault anomaly has to have the right category"
     (is (ba/fault? {::anom/category ::anom/fault})))
@@ -152,7 +163,6 @@
   (testing "without message"
     (is (= (ba/fault) {::anom/category ::anom/fault})))
 
-
   (testing "with nil message"
     (is (= (ba/fault nil) {::anom/category ::anom/fault})))
 
@@ -169,6 +179,9 @@
 
 
 (deftest busy-test
+  (testing "without message"
+    (is (= (ba/busy) {::anom/category ::anom/busy})))
+
   (testing "with nil message"
     (is (= (ba/busy nil) {::anom/category ::anom/busy})))