Verify the features which got affected from Mirror node changes

[prernaadev01]

Problem description

As there's changes in hashgrapn-owned Mirror node behaviour with an implementation of 60 days data visbility as mentioned in the below Slack message:

To ensure the resilience and performance of the APIs, the Mirror Node team is introducing a 60-day limitation to the following RestAPIs:

/api/v1/accounts/{id}
/api/v1/contracts/results
/api/v1/contracts/results/logs
/api/v1/transactions

The above APIs will not show transactions older than 60 days when there is no timestamp in the URL because the rest of the API server will add an implicit timestamp range [now-60d, now] to query the database.

This has been implemented on September 20th 2024.

Call to action

• When using the specified Hedera APIs to access history greater than 60 days, users should either:
  • Specify specific historical time ranges of no more than 60 days in their API request
  • Leverage the links.next or RFC5988 functionality to page through the history with 2 conditions:
    • Results returned will be within a 60-day time range
    • No more than 100 results per return

Requirements

1. Ensure that Guardian (including Indexer) works as expected following the introduction of 60-days limit to the Mirror Nodes API. Implement the recommendations of using links.next or RFC5988 functionality where required.
2. Ensure that Guardian (including Indexer) usage of Mirror Nodes via API is optimised, i.e. that all opportunities to restrict/limit the scope of returned data or searching by formulating the API request with specific relevant time-ranges (where possible) are utilized. Change the requests accordingly wherever necessary.

Definition of done

We are confident that Guardian uses mirror node API in the most optimal way.

Acceptance criteria

• Guardian functionality is restored following the introduction of the 60-day data limits for Mirror Nodes.
• Guardian usage of Mirror Node API is optimised, Guardian always uses strictly the resources that are unavoidable from Mirror Nodes.

[anvabr]

@artembuslaev in collaboration with the rest of the dev and QA team have done impact assessment of this change on Guardian by examining Guardian codebase and performing tests where necessary.

Among the code changes and improvements which have be done for this ticket the method for requesting information from the Hedera Rest API has been changed not to rely on the number of returned result but to use the presence of next URL as an indicator of more results available.

'60 days' limit

From the list of affected end-points:

/api/v1/accounts/{id}
/api/v1/contracts/results
/api/v1/contracts/results/logs
/api/v1/transactions

Guardian use only the following (for retrying mint operation):/api/v1/transactions

Guardian code steps transaction by transaction in the ASC order, using pagination (next), our tests indicates that this approach works across 60-day boundary.

Smart contracts are synchronised using endpoint which was unaffected by the '60 days' change:
/api/v1/contracts/{contractIdOrAddress}/results/logs. So no code modifications are required here.

Optimisation requirement.

Existing Guardian, Indexer and Analytics requires are as optimised as possible, most requests use: /api/v1/topics/{topicId}/messages

Endpoint optimisation in Guardian:

/api/v1/transactions
/api/v1/contracts/{contractIdOrAddress}/results/logs

The requests start from the latest unprocessed transaction, e.g. at the beginning of the mint - the latest user transaction is recorded and if retry is triggered transactions are searched using gt filter, using pagination. For smart contracts synchronisation the latest log entry of synchronisation is recorded, which serves as a starting point for the next iteration.