Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prod Release 16/07/24 #869

Merged
merged 14 commits into from
Jul 18, 2024
Merged

Prod Release 16/07/24 #869

merged 14 commits into from
Jul 18, 2024

Commits on Jul 4, 2024

  1. feat: Implement Historical Backfill using Bitmap Indexer (#803)

    Replaces Delta Lake Stream with Bitmap Indexer Stream as the method for
    backfilling matching block heights before switching to real time
    processing. Bitmap Indexer resolves many issues the Delta Lake approach
    had:
    1. Can be extended more easily.
    2. Can be queried much faster thanks to storage in a database as well as
    in a compressed format.
    3. Queries can be customized, enabling better matching. 
    4. Supports wildcard contract filters much better than the previous
    approach.
    5. Is much harder to overload with too much data, which was a problem
    with the previous approach.
    
    ---------
    
    Co-authored-by: Morgan Mccauley <morgan@mccauley.co.nz>
    darunrs and morgsmccauley authored Jul 4, 2024
    Configuration menu
    Copy the full SHA
    c64f8ff View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. test: Cancel block_stream task to workaround infinite lake loop (#856)

    `near-lake-framework` will infinitely retry, making our tests infinitely
    hang. This PR monitors the `last_processed_ block`, and cancels the task
    from the outside when the last desired block is reached.
    morgsmccauley authored Jul 9, 2024
    Configuration menu
    Copy the full SHA
    8c40b2e View commit details
    Browse the repository at this point in the history
  2. refactor: Unify logic for generating Redis keys (#807)

    Both the `IndexerState` and `IndexerConfig` structs are used
    individually to access Redis, and therefore have duplicate code for
    generating the required keys. This is problematic for the following
    reasons:
    1. Redis access is limited to these structs, anything else requiring
    access needs to duplicate the logic again
    2. Updates need to be reflected across both structs, creating more room
    for error
    
    This PR creates a common Redis `KeyProvider` interface/trait for
    accessing Indexer Redis data/keys. Now the required structs, i.e.
    `IndexerState` and `IndexerConfig`, can implement this trait, and
    automatically get access to the methods which generate the keys.
    
    Also did some minor clean up, removing the old Coordinator migration
    code.
    morgsmccauley authored Jul 9, 2024
    Configuration menu
    Copy the full SHA
    680c6c9 View commit details
    Browse the repository at this point in the history
  3. fix: remove the root cause of multiple rerenders on editor lifecycle (#…

    …857)
    
    Cause of multiple rerendering on page load
    Kevin101Zhang authored Jul 9, 2024
    Configuration menu
    Copy the full SHA
    889fba0 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. feat: Enhance Schema Error Visualization with New Icons (#855)

    Optimized Editor Lifecycle methods and removed the 'janky' Alert that
    would insert itself between the developer tools and Editor component in
    favor of a small icon that is in the fileswitcher.
    Kevin101Zhang authored Jul 10, 2024
    Configuration menu
    Copy the full SHA
    1318cb6 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. fix: Filter receiver blocks before starting block (#863)

    As we fetch blocks for a given day, there is possibility for that day to
    include blocks lesser than the provided `start_block_height`. This PR
    ensures that those blocks are filtered out.
    morgsmccauley authored Jul 11, 2024
    Configuration menu
    Copy the full SHA
    e697f37 View commit details
    Browse the repository at this point in the history
  2. doc: Document bitmap implementation (#862)

    In an effort to understand the bitmap implementation, I've added
    documentation comments. Let me know if it's incorrect!
    
    ---------
    
    Co-authored-by: Darun Seethammagari <darun@near.org>
    morgsmccauley and darunrs authored Jul 11, 2024
    Configuration menu
    Copy the full SHA
    8c37b78 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. fix: Support array values for queries and string type for inserts (#864)

    Context DB in the frontend was showing an error for inserts and select.
    For inserts, some types such as `date` would expect `Date` type when
    string type also works. I updated the type gen to allow string type
    input for these options. This ensures string or number input is
    acceptable for essentially all types.
    <img width="955" alt="image"
    src="https://github.com/near/queryapi/assets/22734869/3dd9ae4b-fa95-44de-824e-b026c525c5d1">
    
    
    In addition, selection queries allow for array input to match muliple
    values for a certain column. However, the frontend was not updated to
    accept this input. I've added a new interface called Query which allows
    for array values in input objects.
    darunrs authored Jul 12, 2024
    Configuration menu
    Copy the full SHA
    3655d91 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. feat: display the number of queries in the past 7 days on each indexe…

    …r card in the dashboard (#837)
    
    BOS component indexercard now queries past 7 day query count.
    Kevin101Zhang authored Jul 15, 2024
    Configuration menu
    Copy the full SHA
    1a2d419 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Configuration menu
    Copy the full SHA
    9aa9500 View commit details
    Browse the repository at this point in the history
  2. feat!: Return Option rather than panic (#870)

    Option is more convenient as we don't need to manually inspect the error
    to determine whether or not the Indexer exists. This is a breaking
    change, but there aren't any consumers of this so it is safe.
    morgsmccauley authored Jul 16, 2024
    Configuration menu
    Copy the full SHA
    0870f25 View commit details
    Browse the repository at this point in the history
  3. fix: Do not eat bitmap backfill errors (#871)

    If the stream yielding block heights from the bitmap indexer fails, its
    error will be eaten by the while loop as the loop will simply end if any
    error takes place. This PR ensures that if the result from the stream is
    an error, it logs the error before proceeding with Lake.
    darunrs authored Jul 16, 2024
    Configuration menu
    Copy the full SHA
    d25763d View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. fix: fetch BOS indexerdata from near RPC call and conditionally rende…

    …r metadata if available in databricks (#873)
    
    BOS component data is fetched from NEAR RPC call 'list_all'. We use a
    map to set a k/v pair for indexer metadata recieved on databricks. When
    we render each card we check the map to see if the indexer to be
    rendered from NEAR RPC exist in the map. If it does exist we
    conditionally render metadata.
    Kevin101Zhang authored Jul 17, 2024
    Configuration menu
    Copy the full SHA
    6643318 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    27ac9a1 View commit details
    Browse the repository at this point in the history