-
Notifications
You must be signed in to change notification settings - Fork 722
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FR] - Query utxo indexer service #4678
Comments
There already exists quite a few indexers out there. And I can recommend one from a former colleague of mine: https://github.com/CardanoSolutions/kupo |
|
@abailly-iohk is our head of architecture. He'll be providing some feedback on how he wants this to be architected. |
I'd like to petition to keep the UTxO query for the case of local testnets. We have tests in cardano-node that utilize the UTxO query and in this instance performance is not an issue since the UTxO is small. The UTxO query is also useful for quick and dirty debugging especially when integrating a new era. Not having to setup an additional indexer would save myself and QA additional work. |
The goal is to keep the UTxO query in the |
@disassembler We would need some numbers on the current vs. forecasted performance of this (and possibly other) kind of queries, and a better understanding of what's the target/requirement from users' perspective. I think @dnadales or @jasagredo have already run or will run some benchmarks for the utxo-hd case. |
Also, we really would want to have some some kind of impact study which implies talking to various node users to understand what would be an acceptable solution. |
@disassembler Shouldn't we reclassify this issue as |
While being stuck on ouroboros-network, we have explored the alternative approach of working on the rpc command integration to cardano-cli. A non-production version for the marconi-mamba with rpcs and update of cardano-cli seems doable within this week. By rpc we are refering to json rpc simlar to bitcoin and ethereum
Meanwhile, we can also explore the ouroboros-network for the proxying part where a meeting with ouroboros-network team would greatly help. Marconi
|
We we were looking at ways to intercept and interpret cli-to-node communication and return query result using different service. But it seems that the setting up an intercepting service was harder than we thought. Can somebody point out how to setup an Also, all the query apis in |
@disassembler The non-production demo is available here. |
I've created a PR. #4810 |
Can we have an output, that replicates the
one? With that i mean a plaintext output like the original one. Many tools on the cli rely on this kind of output, because of the still existing bug in jq to work with really large numbers. For that reason also f.e. koios sends the amounts as strings. The current output from |
The addition from PR seems a bit less than ideal to my eyes - essentially proposal is using
|
@gitmachtl @rdlrt Based on the above feedback I have listed following points to be considered for final implementation
Given the considerations outlined, do you believe that this is the right way to go forward with the proposed changes? Regards, |
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days. |
Internal/External
Internal if an IOHK staff member.
Area
Other Any other topic (Delegation, Ranking, ...).
Describe the feature you'd like
Do not use ledger state for
query utxo
.The utxo on-disk work will make querying utxo from ledger state really slow (it's already really slow and getting slower as utxo size increases). What we need is a separate service that talks to the node socket to keep track of all utxos (or maybe a subset of interesting ones that the user defines). What we want to do is provide a similar interface, e.g.
But not use the ledger state queries to get this information. Essentially, this will need to something like
foldBlocks
ormarconi
does indexing state transitions on the chain from genesis and keeping it separate from the node itself.Describe alternatives you've considered
Additional context / screenshots
Add any other context or screenshots about the feature request here.
From @dcoutts
@abailly-iohk yes, the only way to make it fast is to use an index. The original feature was a quick convenient hack that got totally out of hand. (My bad.) And yes, sqlite would be perfectly good as an implementation of such and index, and the right place to do that is in an indexing client, and not within the node process itself.
One of our general architectural principles for the node and its external clients is that we make all chain information available to clients, but we do not store information in the node (and provide client access) that the node does not itself need. The node is part of the trusted base of the system. It is complex and has strong security and performance requirements. It is the wrong place to build database features. External clients are the perfect place to build database features.
The right solution is a client that maintains an index and provides query access. There are then multiple choices: provide a minimal one that just replicates what the CLI provided before (via the node), or use some existing more general purpose indexer, or some combo to satisfy different use cases.
The text was updated successfully, but these errors were encountered: