CORE-18698 Store and find filtered transactions in the database #5485

nkovacsx · 2024-01-22T18:28:59Z

This PR contains the following changes:

Move FilteredTransaction and FilteredTransactionFactory and their implementations to a separate module called ledger-filtered-transaction-api and ledger-filtered-transaction. This is needed because the common flow module didn't allow us to use these in the persistence worker.
Compile and test changes related to this new module
A new interface called MerkleProofInternal which has a merge function on it so we don't have to use MerkleProofImpl everywhere.
A new interface called UtxoFilteredTransactionInternal so we can access the its FilteredTransaction.
FilteredTransaction now exposes privacySalt
Remove previously public API related to Merkle Proofs from the UtxoLedgerPersistenceService
New functions called persistFilteredTransactions and findFilteredTransactions on the UtxoPersistenceService API (persistence worker side)

Draft PR for the requests that I used for testing:
corda/corda-api#1456
#5490

...dger-persistence/src/main/kotlin/net/corda/ledger/persistence/utxo/UtxoPersistenceService.kt

blsemo · 2024-01-23T15:32:51Z

I was wondering whether we need new modules for filtered transactions? Couldn't they be moved into libs/ledger/ledger-utxo-data? We tend to have too many modules which causes gradle slowness and OSGi overhead...

dickon · 2024-01-25T07:59:09Z

I think we should split out moving the code around and get that merged first so it's easier to read the behaviour changes.

blsemo · 2024-01-25T10:15:24Z

...sistence/src/main/kotlin/net/corda/ledger/persistence/utxo/impl/AbstractUtxoQueryProvider.kt

-                AND utmp.group_idx = :groupIndex"""
+            WHERE utmp.transaction_id IN (:transactionIds)
+            AND utc.leaf_idx IN (
+                SELECT utmpl.leaf_index


Can you get input from @ifrankrui on the best way to write those queries?

...istence/src/main/kotlin/net/corda/ledger/persistence/utxo/impl/UtxoPersistenceServiceImpl.kt

libs/crypto/cipher-suite/build.gradle

...istence/src/main/kotlin/net/corda/ledger/persistence/utxo/impl/UtxoPersistenceServiceImpl.kt

…o ledger-common-flow (#5526) In prepration for using FilteredTransaciton from UtxoLedgerService move the interface and add a stub. Refactoring broken out from #5485.

… ledger libs (#5528) Move filtered transactions implementation to libs. This is in preparation for using filtered transactions more widely. Refactoring broken out from #5485.

…me-os into nandor/CORE-18698/persist-and-find-ftx # Conflicts: # gradle.properties

corda-jenkins-ci02 · 2024-02-01T15:00:18Z

Jenkins build for PR 5485 build 32

Build Successful:
Jar artifact version produced by this PR: 5.2.0.0-alpha-1706883968777
Helm chart version produced by this PR: 5.2.0-alpha.1706883968777
Helm chart pushed to: oci://corda-os-docker-dev.software.r3.com/helm-charts/pr-5485/corda

…me-os into nandor/CORE-18698/persist-and-find-ftx # Conflicts: # components/ledger/ledger-persistence/src/main/kotlin/net/corda/ledger/persistence/utxo/UtxoPersistenceService.kt # components/ledger/ledger-utxo-flow/src/main/kotlin/net/corda/ledger/utxo/flow/impl/persistence/UtxoLedgerPersistenceService.kt # components/ledger/ledger-utxo-flow/src/main/kotlin/net/corda/ledger/utxo/flow/impl/persistence/UtxoLedgerPersistenceServiceImpl.kt

…me-os into nandor/CORE-18698/persist-and-find-ftx

vlajos

I added a few small things potentially for a later PR.

vlajos · 2024-02-02T10:42:39Z

...grationTest/kotlin/net/corda/ledger/persistence/utxo/tests/UtxoPersistenceServiceImplTest.kt

-            listOf(0, 1),
-            emptyList()
+        persistenceService.persistFilteredTransactions(
+            mapOf(filteredTransactionToStore to emptyList()),


It would be good to verify the signature persistence's execution path.
But that can happen in a separate PR/ticket.

vlajos · 2024-02-02T10:54:10Z

...grationTest/kotlin/net/corda/ledger/persistence/utxo/tests/UtxoPersistenceServiceImplTest.kt

+            signedTransaction.wireTransaction,
+            componentGroupFilterParameters = listOf(
+                ComponentGroupFilterParameters.AuditProof(
+                    0,


UtxoComponentGroup.OUTPUTS_INFO.ordinal would be a bit nicer instead of the constant.
Or potentially using the LedgerService.filterSignedTransaction to replace the creation steps. (if that works in integration tests.)

vlajos · 2024-02-02T11:02:14Z

...istence/src/main/kotlin/net/corda/ledger/persistence/utxo/impl/UtxoPersistenceServiceImpl.kt

+            )
+
+            // 6. Map the transaction id to the filtered transaction object and signatures
+            filteredTransaction.id.toString() to Pair(filteredTransaction, ftxDto.signatures)


transactionId is already the same in string.

vlajos · 2024-02-02T11:08:56Z

...a/src/main/kotlin/net/corda/ledger/common/data/transaction/Extensions.TransactionMetadata.kt

+import net.corda.v5.crypto.extensions.merkle.MerkleTreeHashDigestProvider
+import net.corda.v5.crypto.merkle.HashDigestConstants
+import net.corda.v5.ledger.common.transaction.TransactionMetadata
+import java.util.*


Is detekt OK with wildcard import?

rick-r3

Re-approve

gradle.properties

mbrkic-r3

LGTM (flow related code)

sonarqubecloud · 2024-02-02T14:29:12Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

2 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

wzur-r3

LGTM

nkovacsx added 3 commits January 22, 2024 17:08

CORE-18698 Store and retrieve filtered transactions

053ada4

fix test compile

c1d63d4

vbump

ea7c864

nkovacsx requested review from a team as code owners January 22, 2024 18:28

nkovacsx added 3 commits January 23, 2024 09:07

Minor cleanups, KDocs

f45a7da

detekt

0ab4e9f

import cleanup

1d1eafb

jennyang-r3 reviewed Jan 23, 2024

View reviewed changes

...dger-persistence/src/main/kotlin/net/corda/ledger/persistence/utxo/UtxoPersistenceService.kt Outdated Show resolved Hide resolved

nkovacsx added 2 commits January 24, 2024 10:27

Move filtered transaction API/impl to ledger-common-data

7f2f6b7

Make findFilteredTransactions private

4200fcf

nkovacsx requested a review from jennyang-r3 January 24, 2024 10:29

blsemo reviewed Jan 25, 2024

View reviewed changes

jennyang-r3 reviewed Jan 25, 2024

View reviewed changes

...istence/src/main/kotlin/net/corda/ledger/persistence/utxo/impl/UtxoPersistenceServiceImpl.kt Outdated Show resolved Hide resolved

blsemo mentioned this pull request Jan 26, 2024

CORE-18698: move FilteredTransaction interface and its data class to ledger-common-flow #5526

Merged

nkovacsx mentioned this pull request Jan 26, 2024

CORE-18698: move filtered transaction implementation to ledger libs #5528

Merged

dickon mentioned this pull request Jan 26, 2024

CORE-18691 Implementation of store and fetch merkle proofs (#5407) #5534

Closed

nkovacsx added 7 commits January 26, 2024 13:50

Store top level MP, add digest provider for nonce

3b5aa33

Bugfixes

ee97a20

merge 5.2

a796979

Fix merge problems

68fc7a3

Merge branch 'release/os/5.2' of https://github.com/corda/corda-runti…

e98ed86

…me-os into nandor/CORE-18698/persist-and-find-ftx # Conflicts: # gradle.properties

Fix merge problems

eb82087

detekt

abf1ba7

nkovacsx added 2 commits February 1, 2024 14:22

vbump

05b12b5

Merge branch 'release/os/5.2' of https://github.com/corda/corda-runti…

7cba865

…me-os into nandor/CORE-18698/persist-and-find-ftx # Conflicts: # gradle.properties

nkovacsx added 3 commits February 1, 2024 17:13

Add verify to test

4d4f283

Merge branch 'release/os/5.2' of https://github.com/corda/corda-runti…

fbc94dd

…me-os into nandor/CORE-18698/persist-and-find-ftx

rick-r3 previously approved these changes Feb 2, 2024

View reviewed changes

vbump

8b65d70

nkovacsx dismissed rick-r3’s stale review via 8b65d70 February 2, 2024 10:00

nkovacsx requested a review from rick-r3 February 2, 2024 10:00

vlajos previously approved these changes Feb 2, 2024

View reviewed changes

rick-r3 previously approved these changes Feb 2, 2024

View reviewed changes

nkovacsx removed the request for review from a team February 2, 2024 11:55

wzur-r3 requested changes Feb 2, 2024

View reviewed changes

gradle.properties Outdated Show resolved Hide resolved

mbrkic-r3 previously approved these changes Feb 2, 2024

View reviewed changes

vbump

9ff3f24

nkovacsx dismissed stale reviews from mbrkic-r3, rick-r3, and vlajos via 9ff3f24 February 2, 2024 14:25

nkovacsx requested a review from a team as a code owner February 2, 2024 14:25

nkovacsx requested review from wzur-r3, vlajos, mbrkic-r3 and rick-r3 February 2, 2024 14:25

rick-r3 approved these changes Feb 2, 2024

View reviewed changes

vlajos approved these changes Feb 2, 2024

View reviewed changes

wzur-r3 approved these changes Feb 2, 2024

View reviewed changes

nkovacsx merged commit 2352f31 into release/os/5.2 Feb 2, 2024
4 checks passed

nkovacsx deleted the nandor/CORE-18698/persist-and-find-ftx branch February 2, 2024 15:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CORE-18698 Store and find filtered transactions in the database #5485

CORE-18698 Store and find filtered transactions in the database #5485

nkovacsx commented Jan 22, 2024 •

edited

Loading

blsemo commented Jan 23, 2024

dickon commented Jan 25, 2024

blsemo Jan 25, 2024

corda-jenkins-ci02 bot commented Feb 1, 2024 •

edited

Loading

vlajos left a comment

vlajos Feb 2, 2024

vlajos Feb 2, 2024

vlajos Feb 2, 2024

vlajos Feb 2, 2024

rick-r3 left a comment

mbrkic-r3 left a comment

sonarqubecloud bot commented Feb 2, 2024

wzur-r3 left a comment

CORE-18698 Store and find filtered transactions in the database #5485

CORE-18698 Store and find filtered transactions in the database #5485

Conversation

nkovacsx commented Jan 22, 2024 • edited Loading

blsemo commented Jan 23, 2024

dickon commented Jan 25, 2024

blsemo Jan 25, 2024

Choose a reason for hiding this comment

corda-jenkins-ci02 bot commented Feb 1, 2024 • edited Loading

vlajos left a comment

Choose a reason for hiding this comment

vlajos Feb 2, 2024

Choose a reason for hiding this comment

vlajos Feb 2, 2024

Choose a reason for hiding this comment

vlajos Feb 2, 2024

Choose a reason for hiding this comment

vlajos Feb 2, 2024

Choose a reason for hiding this comment

rick-r3 left a comment

Choose a reason for hiding this comment

mbrkic-r3 left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 2, 2024

Quality Gate passed

wzur-r3 left a comment

Choose a reason for hiding this comment

nkovacsx commented Jan 22, 2024 •

edited

Loading

corda-jenkins-ci02 bot commented Feb 1, 2024 •

edited

Loading