Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync improvements #37

Merged
merged 10 commits into from
May 25, 2024
Merged

Conversation

LordTermor
Copy link
Collaborator

@LordTermor LordTermor commented May 24, 2024

This PR improves overall previously half-baked sync service. It also includes

  • dbcli dev utility to look through the LMDB packages database
  • Sync exclude list support
  • Fix for Sync triggers twice #36
  • Package version parsing logic follows the same logic as ALPM's one (logic I used from ArchWiki is incomplete, e.g. it doesn't say it allows + in package version)
  • Improving package validation by checking it's descfile on adding

Those all are somewhat connected to the sync process but other subsystems will benefit as well.

@LordTermor LordTermor added bug Something isn't working enhancement New feature or request Daemon Changes related to the backend labels May 24, 2024
@LordTermor LordTermor requested a review from romangg May 24, 2024 09:46
@LordTermor LordTermor self-assigned this May 24, 2024
@LordTermor LordTermor added this to the Initial testing (MVP) milestone May 24, 2024
This was linked to issues May 24, 2024
Copy link
Contributor

@romangg romangg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commits need better commit messages. There are some commits rewriting 100s lines of code and it's difficult to understand what they do.

dbcli/dbcli.cpp Outdated

int main(int argc, char** argv) {
auto lmdbenv = std::make_shared<bxt::Utilities::LMDB::Environment>(
std::make_shared<coro::io_scheduler>());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these shared pointer necessary? Value types are when possible preferable.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LMDB's Enviroment is indeed intended to be used with a shared global io_scheduler (to manage transactions and concurrency) and also be shared with DBs itself for the same purpose, that's why it's needed.

In this particular case this doesn't really make sense but I just reused the daemon's components. I can rewrite using plain LMDB(++) if preferable.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

dbcli/CMakeLists.txt Show resolved Hide resolved
@LordTermor
Copy link
Collaborator Author

Commits need better commit messages. There are some commits rewriting 100s lines of code and it's difficult to understand what they do.

Will do, boss.

Implements a basic database CLI for looking through the LMDB
Drastically improves sync service, including:

- The service now retries failed downloads and has a connection timeout
- Improved error handling and logging on all steps
- Use scope guard for event dispatching on errors
- Templated hash functions
- More verbose variables
Async synchronization needs more testing.
@LordTermor LordTermor merged commit d7a30cb into anydistro:master May 25, 2024
4 checks passed
@LordTermor LordTermor deleted the sync-improvements branch May 25, 2024 09:20
@LordTermor LordTermor restored the sync-improvements branch May 25, 2024 09:21
@LordTermor LordTermor deleted the sync-improvements branch July 9, 2024 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Daemon Changes related to the backend enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sync triggers twice ALPM DB exporter fails sometimes
2 participants