MessageCache and LightNode #59

MHHukiewitz · 2023-09-05T12:48:29Z

Reopened with smaller scope from #31. Depends on #54.

Problem

Some apps require access to the same messages over and over again. This puts unnecessary strain on the CCN API and increases the latency of apps using Aleph Messages.

Solution

This pull request proposes a solution by introducing a message cache implemented with Peewee and an SQLite database in the Aleph SDK. The message cache effectively stores frequently accessed messages, eliminating the need for repetitive CCN API calls. With the message cache in place, apps utilizing the Aleph SDK will experience enhanced efficiency and responsiveness in accessing messages.

MHHukiewitz · 2023-09-06T19:49:38Z

I started using v1 posts instead of v0 posts. I wonder why we did not earlier, as it seems a cleaner, albeit minimalistic, endpoint for getting posts. There should be no problem in changing back to v0 posts though.

MHHukiewitz · 2023-09-27T17:23:26Z

A few points on my ToDo list:

DomainNode: Should keep a synchronized cache with a subset of all Aleph messages.
- Only allow messages to be posted, that fall under the scope of the Domain, meaning they would not be filtered by given initialization parameters.
- Should remove messages from the cache, if they not reappear as confirmed in the watch_messages stream, after some pre-defined timeout.
Make BaseAlephClient and BaseAuthenticatedAlephClient pure interfaces.
Add interfaces for UserSessionSync and AuthenticatedUserSessionSync
- Let MessageCache implement UserSessionSyncInterface instead of BaseAlephClient, as all provided methods can be sync.
- Implement DomainNode.__enter__() with a DomainNodeSync class.
Add parameter classes like in Refactor: Filters were messy with duplication #58 for all of the above.

odesenfans

Reviewed ~70% of it, some changes and questions.

src/aleph/sdk/conf.py

src/aleph/sdk/models.py

src/aleph/sdk/node/__init__.py

src/aleph/sdk/node/common.py

tests/integration/itest_forget.py

MHHukiewitz · 2023-10-25T21:19:57Z

Implemented all of the suggested changes by @odesenfans, highlights:

Broadcasted messages are now being polled up to 50 seconds for their status every 5 seconds. If they fail to become processed, they are removed from the cache.
Validating that the database schema is the same as defined in the SDK's model, raising an error and asking the user to delete or migrate the existing DB.
Aggregates are now correctly being aggregated and materialized in an Aggregate table.
Forget messages are now taken into consideration and they delete entries in the cache.
File downloads are now cached, too.

Addressed issues, but Olivier can't review them anymore.

github-actions · 2023-11-13T13:28:55Z

Failed to retrieve llama text: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

MHHukiewitz · 2023-11-13T13:32:51Z

Failed to retrieve llama text: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Context size not enough, I rate this PR manually as BLACK

A LightNode can synchronize on a subset, or domain, of aleph.im messages. It relies on the MessageCache, which manages a message database with peewee.

…ust `create_program` methods on `LightNode`

hoh · 2024-02-01T12:26:45Z

This PR adds +1,937 and -16 lines of code to the project.

What is the impact of this PR on the test coverage ?

MHHukiewitz · 2024-02-01T12:47:25Z

What is the impact of this PR on the test coverage ?

It actually improved relatively, as the new code has more coverage than the existing one.

ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate when using the function: AuthenticatedAlephHttpClient I searched on the internet for a way to solve this problem, but all the commands/advice given didn't work. So I thought it would be a good idea to give the user the option of specifying a specific SSL certificate if they wish. This worked in my case and gave me the option of continuing to use the SDK provided by Aleph.

I had the following problem: ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate when using the function: AuthenticatedAlephHttpClient I searched on the internet for a way to solve this problem, but all the commands/advice given didn't work. So I thought it would be a good idea to give the user the option of specifying a specific SSL certificate if they wish. This worked in my case and gave me the option of continuing to use the SDK provided by Aleph.

Co-authored-by: Mike Hukiewitz <70762838+MHHukiewitz@users.noreply.github.com>

The problem was with the connector type: src/aleph/sdk/client/http.py:55:25 : error : Incompatible types in assignment (expression has type "UnixConnector", variable has type "Optional[TCPConnector]") I have therefore declared the connector as a union of possible BaseConnector or None types. This means it can be any aiohttp BaseConnector or None.

Users could not easily import or migrate accounts using their mnemonic representation. Solution: Add a static method `from_mnemonic` on the `ETHAccount` class. Discussion: This is a first step and this behaviour can be extended to more chains in the future.

Problem: Instances don't use program encodings. This argument was left from refactoring. Solution: Drop the argument.

Problem: Users of the SDK could not create instances with a specific payment method such as token streams. Solution: Add a new argument, `payment`, to `AuthenticatedAlephClient` and use it in the "Instance" messages generated. The argument is optional and defaults to "hold" on "ETH" for backward compatibility. Discussion: The argument is added just after mandatory arguments so it can be made mandatory in the future. This may however break backward compatibility with existing code that does not call the function using keywords arguments.

Problem: too strict aleph-message dependency Solution: loosen it to accept compatible versions to 0.4.2 Co-authored-by: mhh <mike.hukiewitz@robotter.ai>

Problem: too strict aleph-message dependency Solution: loosen it to accept compatible versions to 0.4.2 Co-authored-by: Hugo Herter <git@hugoherter.com>

Psycojoker · 2024-06-20T15:38:12Z

src/aleph/sdk/__init__.py

@@ -1,6 +1,6 @@
 from pkg_resources import DistributionNotFound, get_distribution

-from aleph.sdk.client import AlephHttpClient, AuthenticatedAlephHttpClient
+from aleph.sdk.client import AlephHttpClient, AuthenticatedAlephHttpClient, LightNode


Current discussion with @hoh : we might want to use another name for this class

MHHukiewitz requested a review from odesenfans September 5, 2023 12:48

MHHukiewitz force-pushed the mhh-domain-node branch from 26146b7 to 603afef Compare September 6, 2023 19:05

MHHukiewitz requested a review from hoh September 19, 2023 08:05

MHHukiewitz mentioned this pull request Sep 19, 2023

Refactor: Filters were messy with duplication #58

Closed

MHHukiewitz mentioned this pull request Sep 27, 2023

AlephClientBase and AuthenticatedAlephClientBase #54

Merged

MHHukiewitz force-pushed the mhh-domain-node branch from d2d3264 to a99fa66 Compare September 27, 2023 17:04

MHHukiewitz mentioned this pull request Sep 28, 2023

Refactor: Add MessageFilter & PostFilter #63

Closed

odesenfans previously requested changes Oct 3, 2023

View reviewed changes

MHHukiewitz mentioned this pull request Oct 6, 2023

Refactor AlephClient classes #65

Merged

MHHukiewitz force-pushed the mhh-domain-node branch from 778ae95 to 06fb6f9 Compare October 25, 2023 21:11

MHHukiewitz changed the title ~~MessageCache and DomainNode~~ MessageCache and LightNode Oct 25, 2023

MHHukiewitz force-pushed the mhh-domain-node branch from 06fb6f9 to cd62d4c Compare October 25, 2023 21:16

MHHukiewitz force-pushed the mhh-domain-node branch 3 times, most recently from a146f5a to 0e3102f Compare October 25, 2023 21:51

MHHukiewitz requested a review from odesenfans October 25, 2023 21:56

MHHukiewitz force-pushed the mhh-domain-node branch from 0e3102f to 59d361d Compare November 13, 2023 13:25

MHHukiewitz closed this Nov 13, 2023

MHHukiewitz reopened this Nov 13, 2023

MHHukiewitz added the BLACK This PR has critical implications and must be reviewed by a senior engineer. label Nov 13, 2023

MHHukiewitz removed the request for review from odesenfans November 17, 2023 11:08

MHHukiewitz force-pushed the mhh-domain-node branch from 59d361d to 6f286fc Compare November 27, 2023 15:55

MHHukiewitz added 2 commits December 8, 2023 11:24

Feature: Add LightNode and MessageCache

af01b9a

A LightNode can synchronize on a subset, or domain, of aleph.im messages. It relies on the MessageCache, which manages a message database with peewee.

Add DateTimeField to cache DB models; Add create_instance and adj…

ae7309e

…ust `create_program` methods on `LightNode`

MHHukiewitz added 2 commits December 8, 2023 11:24

Add __init__.py files for db and query modules

b6dab52

Adjust light_node.submit() to fit new interface

77bdf1b

MHHukiewitz force-pushed the mhh-domain-node branch from ac6a28c to 77bdf1b Compare December 8, 2023 10:36

MHHukiewitz added 2 commits December 8, 2023 11:36

Fix test using new submit()

bf508d8

Fix formatting

b1ad1a5

nesitor previously approved these changes Feb 1, 2024

View reviewed changes

Merge branch 'main' into mhh-domain-node

22e22dd

MHHukiewitz dismissed nesitor’s stale review via 22e22dd February 1, 2024 12:46

Reformat with new black

0f6f8ca

MHHukiewitz requested a review from nesitor February 8, 2024 10:15

Antonyjin and others added 12 commits February 8, 2024 19:50

Update src/aleph/sdk/client/http.py

302e9fc

Co-authored-by: Mike Hukiewitz <70762838+MHHukiewitz@users.noreply.github.com>

fixup! Fix: An ETH account could not be initialized from its mnemonic

03de4b3

Fix: create_instance required a program encoding

fed1d95

Problem: Instances don't use program encodings. This argument was left from refactoring. Solution: Drop the argument.

Fix: Loosen aleph-message dependency (#92)

a3e120a

Problem: too strict aleph-message dependency Solution: loosen it to accept compatible versions to 0.4.2 Co-authored-by: mhh <mike.hukiewitz@robotter.ai>

Feature: Add Deprecation Message (#103)

b6c3714

Problem: too strict aleph-message dependency Solution: loosen it to accept compatible versions to 0.4.2 Co-authored-by: Hugo Herter <git@hugoherter.com>

Merge branch 'main' into mhh-domain-node

042bfc8

Add payment argument to LightNode.create_instance()

12ac7d5

hoh force-pushed the main branch from f60cedc to 4ee3380 Compare February 22, 2024 21:55

Psycojoker reviewed Jun 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MessageCache and LightNode #59

MessageCache and LightNode #59

MHHukiewitz commented Sep 5, 2023

MHHukiewitz commented Sep 6, 2023

MHHukiewitz commented Sep 27, 2023 •

edited

Loading

odesenfans left a comment

MHHukiewitz commented Oct 25, 2023

github-actions bot commented Nov 13, 2023

MHHukiewitz commented Nov 13, 2023

hoh commented Feb 1, 2024

MHHukiewitz commented Feb 1, 2024

Psycojoker Jun 20, 2024

MessageCache and LightNode #59

Are you sure you want to change the base?

MessageCache and LightNode #59

Conversation

MHHukiewitz commented Sep 5, 2023

Problem

Solution

MHHukiewitz commented Sep 6, 2023

MHHukiewitz commented Sep 27, 2023 • edited Loading

odesenfans left a comment

Choose a reason for hiding this comment

MHHukiewitz commented Oct 25, 2023

github-actions bot commented Nov 13, 2023

MHHukiewitz commented Nov 13, 2023

hoh commented Feb 1, 2024

MHHukiewitz commented Feb 1, 2024

Psycojoker Jun 20, 2024

Choose a reason for hiding this comment

MHHukiewitz commented Sep 27, 2023 •

edited

Loading