Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mediator) : Add TTL for message collection #290

Merged
merged 1 commit into from
Apr 23, 2024

Conversation

mineme0110
Copy link
Contributor

@mineme0110 mineme0110 commented Apr 19, 2024

Description:

Storing every message received by the mediator in the messages collection can lead to significant growth in the collection size. However, not all received messages need to be kept once the relevant action is completed, except for those intended for delivery to the recipient. Therefore, implementing a TTL index for the messages collection, with a filter to expire mediator messages, would optimize storage usage and efficiency.

Alternatives Considered (optional):

Checklist:

  • My PR follows the contribution guidelines of this project
  • My PR is free of third-party dependencies that don't comply with the Allowlist
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked the PR title to follow the conventional commit specification

Copy link

github-actions bot commented Apr 19, 2024

Integration Test Results

 5 files  ±0   5 suites  ±0   12s ⏱️ -1s
17 tests ±0  17 ✅ ±0  0 💤 ±0  0 ❌ ±0 
20 runs  ±0  20 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit be8a59b. ± Comparison against base commit 83d82db.

♻️ This comment has been updated with latest results.

initdb.js Outdated Show resolved Hide resolved
Copy link
Contributor

@FabioPinheiro FabioPinheiro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the documentation in README I think it looks good

README.md Outdated Show resolved Hide resolved
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl
@mineme0110 mineme0110 force-pushed the Add-TTL-For-Message-Collection branch from f15a602 to be8a59b Compare April 23, 2024 09:53
@mineme0110 mineme0110 merged commit a22c1a9 into main Apr 23, 2024
3 checks passed
@mineme0110 mineme0110 deleted the Add-TTL-For-Message-Collection branch April 23, 2024 10:13
#### Mediator storage
- The `messages` collection contains two types of messages: `Mediator` and `User`.
1. **Mediator Messages**:
- These messages received by mediator for any interactions with the mediator.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The mediator receives these messages for any interactions with the mediator.

1. **Mediator Messages**:
- These messages received by mediator for any interactions with the mediator.
- Examples include messages for setting up mediation, requesting mediation, or picking up messages from the mediator.
- These messages stored in collection can be used for debugging purpose mediator functionality and interactions with the mediator. Hence they can be deleted after a period of time.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The messages stored in the collection are usable for debugging purposes, mediator functionality, and interactions with the mediator. Hence, after a predetermined period, deleting them is possible.

- These messages received by mediator for any interactions with the mediator.
- Examples include messages for setting up mediation, requesting mediation, or picking up messages from the mediator.
- These messages stored in collection can be used for debugging purpose mediator functionality and interactions with the mediator. Hence they can be deleted after a period of time.
- This message type `Mediator` can be setup to have a configurable Time-To-Live (TTL) value, after which they can expire.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- This message type `Mediator` can be set up to have a configurable Time-To-Live (TTL) value, after which they can expire.

- Examples include messages for setting up mediation, requesting mediation, or picking up messages from the mediator.
- These messages stored in collection can be used for debugging purpose mediator functionality and interactions with the mediator. Hence they can be deleted after a period of time.
- This message type `Mediator` can be setup to have a configurable Time-To-Live (TTL) value, after which they can expire.
- This is how the TTL can be configured for the collection messages [initdb.js](initdb.js)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This is how the TTL is configurable for the collection messages initdb.js

- This message type `Mediator` can be setup to have a configurable Time-To-Live (TTL) value, after which they can expire.
- This is how the TTL can be configured for the collection messages [initdb.js](initdb.js)
2. **User Messages**:
- These are the actual messages e.g like the Forward message from the mediator, contain a `User` message inside. This inside message is stored as type `User` to be delivered to user.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • These are the actual messages, e.g. the Forward message from the mediator, containing a User message inside. This inside message gets stored as User deliverable to the user.

- This is how the TTL can be configured for the collection messages [initdb.js](initdb.js)
2. **User Messages**:
- These are the actual messages e.g like the Forward message from the mediator, contain a `User` message inside. This inside message is stored as type `User` to be delivered to user.
- They do not have a TTL, and will persist in the system until the user retrieves them using a pickup protocol and deletes them.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- They do not have a TTL and will persist in the system until the user retrieves them using a pickup protocol and deletes them.

- They do not have a TTL, and will persist in the system until the user retrieves them using a pickup protocol and deletes them.
- The mediator is responsible for storing and making these user messages available for delivery to the intended recipients.

ℹ️ For existing users, please utilize the migration script [migration_mediator_collection.js](migration_mediator_collection.js) to migrate the collection.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

::: info
For existing users, please utilize the migration script migration_mediator_collection.js to migrate the collection.
:::

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not valid in markdown file :::info

Copy link
Contributor

@petevielhaber petevielhaber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Requested changes

@mineme0110
Copy link
Contributor Author

https://github.com/input-output-hk/atala-prism-mediator/pull/292/files
@petevielhaber please this PR for fixes Since I had already merged the PR

mineme0110 added a commit that referenced this pull request Apr 30, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl
mineme0110 added a commit that referenced this pull request Apr 30, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl

Signed-off-by: Shailesh <Patil>
mineme0110 added a commit that referenced this pull request May 1, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl

Signed-off-by: Shailesh <Patil>
mineme0110 added a commit that referenced this pull request May 1, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl

Signed-off-by: Shailesh Patil <53746241+mineme0110@users.noreply.github.com>
mineme0110 added a commit that referenced this pull request May 1, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl

Signed-off-by: Shailesh Patil <53746241+mineme0110@users.noreply.github.com>
Signed-off-by: mineme0110 <shailesh.patil@iohk.io>
mineme0110 added a commit that referenced this pull request May 1, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl

Signed-off-by: Shailesh Patil <53746241+mineme0110@users.noreply.github.com>

Signed-off-by: Shailesh Patil <shailesh.patil@iohk.io>
mineme0110 added a commit that referenced this pull request May 1, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl

Signed-off-by: Shailesh Patil <53746241+mineme0110@users.noreply.github.com>

Signed-off-by: Shailesh Patil <shailesh.patil@iohk.io>
mineme0110 added a commit that referenced this pull request May 1, 2024
Add TTL for message collection to do house keeping
update time stamp field from String to Intant
Update initdb.js
Add migration script for ttl

Signed-off-by: Shailesh Patil <53746241+mineme0110@users.noreply.github.com>

Signed-off-by: Shailesh Patil <shailesh.patil@iohk.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants