Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Orderbook Mid Price Cache #2289

Merged
merged 2 commits into from
Sep 23, 2024
Merged

Add Orderbook Mid Price Cache #2289

merged 2 commits into from
Sep 23, 2024

Conversation

adamfraser
Copy link
Contributor

@adamfraser adamfraser commented Sep 18, 2024

Changelist

  • Adds a redis cache to keep a 5 second sample of orderbook mid prices
  • Adds a roundtable task to sample orderbook mid price every second for each market
  • Use 5 second median from orderbook cache for candle mid price

Test Plan

Tested in dev
Added unit tests

Summary by CodeRabbit

  • New Features

    • Introduced a caching mechanism for order book mid prices using Redis.
    • Added functionality to set and retrieve median prices for various tickers.
    • Implemented new Lua scripts for managing market prices and calculating medians.
    • New configuration options for enabling and controlling caching intervals.
  • Bug Fixes

    • Improved handling of undefined prices and error scenarios in caching logic.
  • Tests

    • Added comprehensive unit tests for the new caching functionality and price management.
    • Updated existing tests to utilize the new caching methods for price updates.

Copy link
Contributor

coderabbitai bot commented Sep 18, 2024

Walkthrough

The changes introduce a new caching mechanism for order book mid prices using Redis, including the implementation of two primary functions: setPrice and getMedianPrice. A suite of unit tests is added to ensure the functionality works as intended. Additionally, Lua scripts for adding prices and retrieving medians are introduced, along with configuration parameters to manage the caching process. The overall structure enhances the management and retrieval of mid prices across various components.

Changes

Files Change Summary
indexer/packages/redis/__tests__/caches/orderbook-mid-prices-cache.test.ts Added unit tests for setPrice and getMedianPrice functions, verifying price storage and median calculation.
indexer/packages/redis/src/caches/orderbook-mid-prices-cache.ts Introduced setPrice and getMedianPrice functions for managing order book mid prices in Redis.
indexer/packages/redis/src/caches/scripts.ts Added two Lua scripts: addMarketPriceScript for adding prices and getMarketMedianScript for retrieving median prices.
indexer/packages/redis/src/index.ts Exported the OrderbookMidPricesCache module to enhance available caching functionalities.
indexer/packages/redis/src/scripts/add_market_price.lua Implemented a Lua script to add market prices to a Redis sorted set with timestamp management.
indexer/packages/redis/src/scripts/get_market_median_price.lua Created a Lua script to calculate and return the median price from a sorted set in Redis.
indexer/services/ender/__tests__/lib/candles-generator.test.ts Updated tests to replace updatePriceLevel with OrderbookMidPricesCache.setPrice, enhancing price update logic.
indexer/services/ender/src/lib/candles-generator.ts Changed import from OrderbookLevelsCache to OrderbookMidPricesCache for retrieving mid prices.
indexer/services/roundtable/__tests__/tasks/cache-orderbook-mid-prices.test.ts Introduced tests for the runTask function that caches mid prices, covering various scenarios including error handling.
indexer/services/roundtable/src/config.ts Added configuration parameters for enabling and setting intervals for caching order book mid prices.
indexer/services/roundtable/src/index.ts Integrated a new task for caching order book mid prices based on configuration settings.
indexer/services/roundtable/src/tasks/cache-orderbook-mid-prices.ts Defined runTask function to update the OrderbookMidPricesCache with current market prices.

Possibly related PRs

  • Candles HLOC Data #1887: This PR introduces new fields for order book mid prices (orderbookMidPriceOpen and orderbookMidPriceClose) in the candle data structure, which directly relates to the main PR's focus on testing and managing order book mid prices.
  • Adam/ct 1059 address issue with orderbookmidprice calculation #2012: This PR addresses issues with the calculation of orderbookMidPrice, which is directly relevant to the main PR's changes involving the orderbook-mid-prices-cache functionality.
  • Adam/add candles hloc #2047: This PR adds new fields for order book mid prices to the API documentation, aligning with the main PR's focus on the orderbook-mid-prices-cache and its associated functionalities.

Suggested labels

protocol

Suggested reviewers

  • dydxwill
  • Christopher-Li

Poem

🐇 In the land of code where prices play,
A cache was born to save the day.
With Lua scripts and tests galore,
Mid prices dance, forevermore!
So hop along, let’s celebrate,
For order books now feel great! 🎉


Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 18e10e4 and 7e9675d.

Files selected for processing (12)
  • indexer/packages/redis/tests/caches/orderbook-mid-prices-cache.test.ts (1 hunks)
  • indexer/packages/redis/src/caches/orderbook-mid-prices-cache.ts (1 hunks)
  • indexer/packages/redis/src/caches/scripts.ts (2 hunks)
  • indexer/packages/redis/src/index.ts (1 hunks)
  • indexer/packages/redis/src/scripts/add_market_price.lua (1 hunks)
  • indexer/packages/redis/src/scripts/get_market_median_price.lua (1 hunks)
  • indexer/services/ender/tests/lib/candles-generator.test.ts (7 hunks)
  • indexer/services/ender/src/lib/candles-generator.ts (2 hunks)
  • indexer/services/roundtable/tests/tasks/cache-orderbook-mid-prices.test.ts (1 hunks)
  • indexer/services/roundtable/src/config.ts (2 hunks)
  • indexer/services/roundtable/src/index.ts (2 hunks)
  • indexer/services/roundtable/src/tasks/cache-orderbook-mid-prices.ts (1 hunks)
Files skipped from review as they are similar to previous changes (12)
  • indexer/packages/redis/tests/caches/orderbook-mid-prices-cache.test.ts
  • indexer/packages/redis/src/caches/orderbook-mid-prices-cache.ts
  • indexer/packages/redis/src/caches/scripts.ts
  • indexer/packages/redis/src/index.ts
  • indexer/packages/redis/src/scripts/add_market_price.lua
  • indexer/packages/redis/src/scripts/get_market_median_price.lua
  • indexer/services/ender/tests/lib/candles-generator.test.ts
  • indexer/services/ender/src/lib/candles-generator.ts
  • indexer/services/roundtable/tests/tasks/cache-orderbook-mid-prices.test.ts
  • indexer/services/roundtable/src/config.ts
  • indexer/services/roundtable/src/index.ts
  • indexer/services/roundtable/src/tasks/cache-orderbook-mid-prices.ts

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    -- I pushed a fix in commit <commit_id>, please review it.
    -- Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    -- @coderabbitai generate unit testing code for this file.
    -- @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    -- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    -- @coderabbitai read src/utils.ts and generate unit testing code.
    -- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    -- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@adamfraser adamfraser marked this pull request as ready for review September 23, 2024 16:12
@adamfraser adamfraser requested a review from a team as a code owner September 23, 2024 16:12
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post

Actionable comments posted: 13

Outside diff range and nitpick comments (11)
indexer/packages/redis/src/scripts/add_market_price.lua (2)

1-8: LGTM! Consider adding input validation.

The variable declarations and initializations look good. The use of KEYS[1] for the Redis key and ARGV[1] and ARGV[2] for the price and timestamp is correct. Converting the arguments to numbers using tonumber() is a good practice.

Consider adding input validation to ensure that both price and nowSeconds are valid numbers. For example:

if not price or not nowSeconds then
  return redis.error_reply("Invalid input: price and timestamp must be numbers")
end

1-17: Overall, the implementation looks solid. Consider adding documentation.

The script effectively implements a 5-second cache for market prices using Redis sorted sets, aligning well with the PR objectives. The use of Redis commands is appropriate and should perform efficiently for high-frequency updates.

Consider adding a brief comment at the beginning of the script to explain its purpose and expected inputs. For example:

--[[
  add_market_price.lua
  
  This script adds a new market price to a Redis sorted set and removes entries older than 5 seconds.
  
  Input:
    KEYS[1]: The Redis key for the sorted set storing price data
    ARGV[1]: The price to be added (number)
    ARGV[2]: The current timestamp in seconds (number)
  
  Returns:
    The number of old entries removed from the sorted set
]]

This documentation would make the script more maintainable and easier for other developers to understand and use.

indexer/packages/redis/src/scripts/get_market_median_price.lua (1)

1-5: LGTM! Consider adding a comment for clarity.

The input handling and data retrieval look correct. The script properly uses KEYS[1] for the sorted set key and retrieves all prices using 'zrange'.

Consider adding a comment explaining the expected format of the sorted set, e.g.:

-- Retrieve all prices from the sorted set (format: score = timestamp, member = price)

This would provide clarity on the data structure being used.

indexer/services/roundtable/src/tasks/cache-orderbook-mid-prices.ts (1)

1-40: Unit tests are needed for this file

As mentioned in a previous review comment, unit tests should be added to ensure the reliability and correctness of this implementation. Tests should cover:

  1. Successful price updates for multiple markets
  2. Handling of undefined prices
  3. Error cases (e.g., database errors, cache errors)

Would you like me to provide a skeleton for these unit tests or create a GitHub issue to track this task?

indexer/packages/redis/src/caches/scripts.ts (1)

66-67: LGTM! Consider adding comments for clarity.

The new Lua scripts for adding market price and getting market median price are correctly implemented using the newLuaScript function, consistent with other script definitions in the file.

Consider adding brief comments explaining the purpose of these new scripts, similar to the comments for other scripts in this file. For example:

// Lua Scripts for managing market prices
export const addMarketPriceScript: LuaScript = newLuaScript('addMarketPrice', '../scripts/add_market_price.lua');
export const getMarketMedianScript: LuaScript = newLuaScript('getMarketMedianPrice', '../scripts/get_market_median_price.lua');
indexer/services/roundtable/__tests__/tasks/cache-orderbook-mid-prices.test.ts (1)

29-43: LGTM: Test suite setup and teardown are comprehensive.

The setup and teardown processes ensure a clean state for each test, which is crucial for reliable testing. The handling of database migrations, data clearing, Redis cache clearing, and mock resetting is well-implemented.

Consider adding a afterEach block to reset mocks after each test, in addition to the afterAll block. This can help isolate tests further and prevent any potential inter-test dependencies. For example:

afterEach(() => {
  jest.resetAllMocks();
});
indexer/services/roundtable/src/index.ts (2)

269-275: LGTM: New loop for caching orderbook mid prices.

The implementation follows the established pattern for starting loops and aligns with the PR objectives. The use of configuration variables allows for flexible deployment.

For consistency with other loops in this file, consider adding a lock multiplier configuration:

 if (config.LOOPS_ENABLED_CACHE_ORDERBOOK_MID_PRICES) {
   startLoop(
     cacheOrderbookMidPrices,
     'cache_orderbook_mid_prices',
     config.LOOPS_INTERVAL_MS_CACHE_ORDERBOOK_MID_PRICES,
+    config.CACHE_ORDERBOOK_MID_PRICES_LOCK_MULTIPLIER,
   );
 }

This change would require adding the CACHE_ORDERBOOK_MID_PRICES_LOCK_MULTIPLIER to the configuration file.


Missing Unit Tests for cacheOrderbookMidPrices Task

The cacheOrderbookMidPrices task is implemented, and the relevant configuration variables are properly defined. However, no unit tests have been found for this task. It is recommended to add unit tests to ensure the functionality is correctly implemented and to prevent potential regressions.

  • Add unit tests for cacheOrderbookMidPrices in the test suite.
Analysis chain

Line range hint 1-276: Summary: Changes align well with PR objectives.

The modifications to this file successfully implement the caching of orderbook mid prices as described in the PR objectives. The changes are minimal, focused, and follow existing patterns, which reduces the risk of introducing bugs. The use of configuration variables allows for flexible deployment and easy adjustment of the new feature.

To fully validate these changes:

  1. Ensure that the cacheOrderbookMidPrices task is implemented correctly in the ./tasks/cache-orderbook-mid-prices file.
  2. Verify that the necessary configuration variables (LOOPS_ENABLED_CACHE_ORDERBOOK_MID_PRICES and LOOPS_INTERVAL_MS_CACHE_ORDERBOOK_MID_PRICES) are properly defined in the configuration file.
  3. Consider adding unit tests for the new loop initialization if not already present.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Verify the implementation of cacheOrderbookMidPrices task
echo "Checking implementation of cacheOrderbookMidPrices task:"
cat ./tasks/cache-orderbook-mid-prices.ts

# Verify the configuration variables
echo "Checking configuration variables:"
grep -E "LOOPS_ENABLED_CACHE_ORDERBOOK_MID_PRICES|LOOPS_INTERVAL_MS_CACHE_ORDERBOOK_MID_PRICES" ./config.ts

# Check for existing unit tests
echo "Checking for unit tests:"
grep -r "cacheOrderbookMidPrices" ./test

Length of output: 608


Script:

#!/bin/bash
# Search for the cacheOrderbookMidPrices task implementation
echo "Searching for 'cache-orderbook-mid-prices.ts' in the repository:"
fd "cache-orderbook-mid-prices.ts"

# Search for the config.ts file
echo "Searching for 'config.ts' in the repository:"
fd "config.ts"

# Search for unit tests referencing 'cacheOrderbookMidPrices'
echo "Searching for references to 'cacheOrderbookMidPrices' in test files:"
rg "cacheOrderbookMidPrices" --type test

Length of output: 1455


Script:

#!/bin/bash
# Search for references to 'cacheOrderbookMidPrices' in test files with common test extensions
echo "Searching for references to 'cacheOrderbookMidPrices' in test files (.test.ts and .spec.ts):"
rg "cacheOrderbookMidPrices" --glob "*.test.ts" --glob "*.spec.ts"

# Verify the relevant configuration variables in the correct config.ts
echo "Verifying configuration variables in 'indexer/services/roundtable/src/config.ts':"
grep -E "LOOPS_ENABLED_CACHE_ORDERBOOK_MID_PRICES|LOOPS_INTERVAL_MS_CACHE_ORDERBOOK_MID_PRICES" indexer/services/roundtable/src/config.ts

Length of output: 726

indexer/services/roundtable/src/config.ts (1)

137-139: LGTM: New parameter for orderbook mid price caching interval

The addition of LOOPS_INTERVAL_MS_CACHE_ORDERBOOK_MID_PRICES is appropriate and aligns with the PR objectives. The default value of one second matches the requirement to sample the orderbook mid price every second.

For consistency with other similar parameters in the file, consider removing the line breaks:

-  LOOPS_INTERVAL_MS_CACHE_ORDERBOOK_MID_PRICES: parseInteger({
-    default: ONE_SECOND_IN_MILLISECONDS,
-  }),
+  LOOPS_INTERVAL_MS_CACHE_ORDERBOOK_MID_PRICES: parseInteger({ default: ONE_SECOND_IN_MILLISECONDS }),
indexer/packages/redis/src/caches/orderbook-mid-prices-cache.ts (2)

10-10: Simplify constant declaration

The type annotation : string is redundant for a const variable initialized with a string literal. TypeScript can infer the type automatically. Removing the explicit type improves code readability.

Apply this minor change:

-export const ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX: string = 'v4/orderbook_mid_prices/';
+export const ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX = 'v4/orderbook_mid_prices/';

17-19: Consider exporting getOrderbookMidPriceCacheKey if used externally

If there's a need to generate the cache key outside of this module (e.g., for testing or other caches), you might want to export the getOrderbookMidPriceCacheKey function.

Export the function as follows:

-function getOrderbookMidPriceCacheKey(ticker: string): string {
+export function getOrderbookMidPriceCacheKey(ticker: string): string {
   return `${ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX}${ticker}`;
 }
Comments failed to post (13)
indexer/packages/redis/src/scripts/add_market_price.lua (1)

13-17: LGTM! Consider returning more informative data.

The removal of old entries is implemented correctly using ZREMRANGEBYSCORE. The cutoff time calculation is accurate.

Consider returning the number of removed entries instead of just true. This could provide useful information for monitoring and debugging. You can modify the last line as follows:

-return true
+return redis.call("zremrangebyscore", priceCacheKey, "-inf", cutoffTime)

This change would return the number of elements removed from the sorted set.

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

-- 2. Remove any entries older than 5 seconds
local cutoffTime = nowSeconds - fiveSeconds
return redis.call("zremrangebyscore", priceCacheKey, "-inf", cutoffTime)
indexer/packages/redis/src/scripts/get_market_median_price.lua (2)

1-23: LGTM! Consider adding error handling for robustness.

The script effectively calculates the median price from a Redis sorted set, aligning well with the PR objectives. It handles both even and odd number of prices correctly and deals with the empty data set case.

For increased robustness, consider adding error handling:

  1. Check if KEYS[1] is provided:
if #KEYS < 1 then
  return redis.error_reply("Missing required key")
end
  1. Wrap the main logic in a pcall to catch any runtime errors:
local ok, result = pcall(function()
  -- Existing script logic here
end)

if not ok then
  return redis.error_reply("Error: " .. tostring(result))
end

return result

This will make the script more resilient to unexpected inputs or runtime errors.


12-23: LGTM! Consider ensuring compatibility with older Lua versions.

The median calculation logic is correct for both even and odd number of prices. The use of tonumber() and tostring() is appropriate for type conversion.

To ensure compatibility with Lua 5.2 and earlier versions, consider modifying the average calculation:

- local median = (tonumber(prices[middle]) + tonumber(prices[middle + 1])) / 2
+ local median = (tonumber(prices[middle]) + tonumber(prices[middle + 1])) * 0.5

This avoids potential issues with integer division in older Lua versions.

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

-- Calculate the middle index
local middle = math.floor(#prices / 2)

-- Calculate median
if #prices % 2 == 0 then
  -- If even, return the average of the two middle elements
  local median = (tonumber(prices[middle]) + tonumber(prices[middle + 1])) * 0.5
  return tostring(median)
else
  -- If odd, return the middle element
  return prices[middle + 1]
end
indexer/services/roundtable/src/tasks/cache-orderbook-mid-prices.ts (1)

19-31: Main logic looks good, but consider rate limiting

The implementation correctly fetches all markets and updates the cache for each. However, consider adding a small delay between iterations to prevent potential rate limiting or high load, especially if there are many markets.

Consider adding a delay between iterations:

import { sleep } from '@dydxprotocol-indexer/base';

// Inside the for loop
await sleep(100); // 100ms delay
indexer/services/roundtable/__tests__/tasks/cache-orderbook-mid-prices.test.ts (1)

45-98: LGTM: Test cases cover main scenarios, but could be improved.

The test cases effectively cover the main functionality, edge cases, and error scenarios for the cache-orderbook-mid-prices task. They verify caching behavior, handling of undefined prices, and error logging.

Consider the following improvements to enhance test isolation and coverage:

  1. Use beforeEach to reset mocks instead of resetting them only in afterAll. This ensures each test starts with fresh mocks.

  2. In the "caches mid prices for all markets" test, consider adding assertions for the number of times getOrderBookMidPrice was called to ensure it's not called more than expected.

  3. Add a test case for when getOrderBookMidPrice returns a valid string that's not a number, to ensure proper error handling.

  4. In the "handles errors" test, consider adding an assertion to verify that the price was not cached in the error scenario.

Here's an example of how you might implement some of these suggestions:

beforeEach(() => {
  jest.resetAllMocks();
});

it('caches mid prices for all markets', async () => {
  // ... existing test code ...

  expect(mockGetOrderBookMidPrice).toHaveBeenCalledTimes(2);
});

it('handles non-numeric prices', async () => {
  const market = testConstants.defaultPerpetualMarket;
  const mockGetOrderBookMidPrice = jest.spyOn(OrderbookLevelsCache, 'getOrderBookMidPrice');
  mockGetOrderBookMidPrice.mockResolvedValueOnce('not a number');

  await runTask();

  const price = await OrderbookMidPricesCache.getMedianPrice(redisClient, market.ticker);
  expect(price).toBeNull();
  expect(jest.requireMock('@dydxprotocol-indexer/base').logger.error).toHaveBeenCalled();
});

it('handles errors', async () => {
  // ... existing test code ...

  const price = await OrderbookMidPricesCache.getMedianPrice(redisClient, testConstants.defaultPerpetualMarket.ticker);
  expect(price).toBeNull();
});

These changes will improve the robustness and coverage of your test suite.

indexer/packages/redis/src/caches/orderbook-mid-prices-cache.ts (5)

37-68: Refactor duplicated code for evalAsync into a reusable helper function

The evalAsync function in both setPrice and getMedianPrice is nearly identical, leading to code duplication. This can make maintenance harder and increases the risk of inconsistencies. Consider abstracting this logic into a reusable helper function to improve modularity and maintainability.

Apply the following refactor to create a generic helper function:

+async function evalAsync<T>(
+  client: RedisClient,
+  scriptHash: string,
+  numKeys: number,
+  keys: string[],
+  args: any[],
+): Promise<T> {
+  return new Promise<T>((resolve, reject) => {
+    client.evalsha(
+      scriptHash,
+      numKeys,
+      ...keys,
+      ...args,
+      (err: Error | null, result: T) => {
+        if (err) {
+          return reject(err);
+        }
+        return resolve(result);
+      },
+    );
+  });
+}

Update setPrice to use the helper function:

 export async function setPrice(
   client: RedisClient,
   ticker: string,
   price: string,
 ): Promise<void> {
-  const numKeys: number = 1;
-  let evalAsync: (marketCacheKey: string) => Promise<void> = (marketCacheKey) => {
-    return new Promise<void>((resolve, reject) => {
-      const callback: Callback<void> = (err: Error | null) => {
-        if (err) {
-          return reject(err);
-        }
-        return resolve();
-      };
-      const nowSeconds = Math.floor(Date.now() / 1000);
-      client.evalsha(
-        addMarketPriceScript.hash,
-        numKeys,
-        marketCacheKey,
-        price,
-        nowSeconds,
-        callback,
-      );
-    });
-  };
-  evalAsync = evalAsync.bind(client);
-  return evalAsync(getOrderbookMidPriceCacheKey(ticker));
+  const marketCacheKey = getOrderbookMidPriceCacheKey(ticker);
+  const nowSeconds = Math.floor(Date.now() / 1000);
+  await evalAsync<void>(
+    client,
+    addMarketPriceScript.hash,
+    1,
+    [marketCacheKey],
+    [price, nowSeconds],
+  );
 }

Similarly, update getMedianPrice:

 export async function getMedianPrice(
   client: RedisClient,
   ticker: string,
 ): Promise<string | null> {
-  let evalAsync: (marketCacheKey: string) => Promise<string> = (marketCacheKey) => {
-    return new Promise((resolve, reject) => {
-      const callback: Callback<string> = (err: Error | null, results: string) => {
-        if (err) {
-          return reject(err);
-        }
-        return resolve(results);
-      };
-      client.evalsha(
-        getMarketMedianScript.hash,
-        1,
-        marketCacheKey,
-        callback,
-      );
-    });
-  };
-  evalAsync = evalAsync.bind(client);
-  return evalAsync(getOrderbookMidPriceCacheKey(ticker));
+  const marketCacheKey = getOrderbookMidPriceCacheKey(ticker);
+  const result = await evalAsync<string>(
+    client,
+    getMarketMedianScript.hash,
+    1,
+    [marketCacheKey],
+    [],
+  );
+  return result;
 }

Also applies to: 77-107


95-99: Add fallback for missing Lua scripts in Redis

If Redis restarts, the cached Lua scripts might be cleared, causing evalsha to fail with a NOSCRIPT error. Consider adding fallback logic to load the script using eval if evalsha fails.

Implement a fallback mechanism:

 client.evalsha(
   scriptHash,
   numKeys,
   ...keys,
   ...args,
   (err: Error | null, result: T) => {
+    if (err && err.message.includes('NOSCRIPT')) {
+      // Fallback to eval if script is not cached
+      client.eval(
+        scriptContent, // The Lua script content
+        numKeys,
+        ...keys,
+        ...args,
+        (evalErr: Error | null, evalResult: T) => {
+          if (evalErr) {
+            return reject(evalErr);
+          }
+          return resolve(evalResult);
+        },
+      );
+    } else if (err) {
       return reject(err);
     }
     return resolve(result);
   },
 );

Ensure that scriptContent is accessible where needed.

Also applies to: 53-59


77-107: Simplify the getMedianPrice function for clarity

Similar to setPrice, you can simplify getMedianPrice by removing unnecessary bind usage and defining evalAsync as a straightforward function.

Simplify getMedianPrice as follows:

 export async function getMedianPrice(
   client: RedisClient,
   ticker: string,
 ): Promise<string | null> {
-  let evalAsync: (
+  const evalAsync = (
     marketCacheKey: string,
   ): Promise<string> => {
     return new Promise((resolve, reject) => {
       client.evalsha(
         getMarketMedianScript.hash,
         1,
         marketCacheKey,
         (err: Error | null, results: string) => {
           if (err) {
             return reject(err);
           }
           return resolve(results);
         },
       );
     });
   };
-  evalAsync = evalAsync.bind(client);

   return evalAsync(getOrderbookMidPriceCacheKey(ticker));
 }
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

export async function getMedianPrice(client: RedisClient, ticker: string): Promise<string | null> {
  const evalAsync = (
    marketCacheKey: string,
  ): Promise<string> => {
    return new Promise((resolve, reject) => {
      const callback: Callback<string> = (
        err: Error | null,
        results: string,
      ) => {
        if (err) {
          return reject(err);
        }
        return resolve(results);
      };

      client.evalsha(
        getMarketMedianScript.hash,
        1,
        marketCacheKey,
        callback,
      );
    });
  };

  return evalAsync(
    getOrderbookMidPriceCacheKey(ticker),
  );
}

29-68: Simplify the setPrice function by removing unnecessary bind

The use of bind on evalAsync is unnecessary because the client is already accessible within the scope. You can simplify the function by removing bind and redefining evalAsync as a regular function.

Simplify setPrice as follows:

 export async function setPrice(
   client: RedisClient,
   ticker: string,
   price: string,
 ): Promise<void> {
   const numKeys: number = 1;

-  let evalAsync: (
+  const evalAsync = (
     marketCacheKey: string,
   ): Promise<void> => {
     return new Promise<void>((resolve, reject) => {
       const nowSeconds = Math.floor(Date.now() / 1000);
       client.evalsha(
         addMarketPriceScript.hash,
         numKeys,
         marketCacheKey,
         price,
         nowSeconds,
         (err: Error | null) => {
           if (err) {
             return reject(err);
           }
           return resolve();
         },
       );
     });
   };
-  evalAsync = evalAsync.bind(client);

   return evalAsync(getOrderbookMidPriceCacheKey(ticker));
 }
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

export async function setPrice(
  client: RedisClient,
  ticker: string,
  price: string,
): Promise<void> {
  // Number of keys for the lua script.
  const numKeys: number = 1;

  const evalAsync = (
    marketCacheKey: string,
  ): Promise<void> => {

    return new Promise<void>((resolve, reject) => {
      const callback: Callback<void> = (
        err: Error | null,
      ) => {
        if (err) {
          return reject(err);
        }
        return resolve();
      };

      const nowSeconds = Math.floor(Date.now() / 1000); // Current time in seconds
      client.evalsha(
        addMarketPriceScript.hash,
        numKeys,
        marketCacheKey,
        price,
        nowSeconds,
        callback,
      );

    });
  };

  return evalAsync(
    getOrderbookMidPriceCacheKey(ticker),
  );
}

85-87: Check for null in results before returning

In the callback of getMedianPrice, the results variable might be null if no median price is found. Ensure that this case is handled properly to prevent unexpected errors.

Modify the callback as follows:

 client.evalsha(
   getMarketMedianScript.hash,
   1,
   marketCacheKey,
   (err: Error | null, results: string | null) => {
     if (err) {
       return reject(err);
     }
+    if (results === null) {
+      return resolve(null);
+    }
     return resolve(results);
   },
 );

Committable suggestion was skipped due to low confidence.

indexer/packages/redis/__tests__/caches/orderbook-mid-prices-cache.test.ts (3)

24-31: Correct the usage of Redis client's zrange method

The client.zrange method returns a Promise and does not accept a callback. Using a callback here is incorrect and could lead to unexpected behavior or errors.

Apply this diff to fix the issue:

-await client.zrange(
-  `${ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX}${ticker}`,
-  0,
-  -1,
-  (_: any, response: string[]) => {
-    expect(response[0]).toBe('50000');
-  },
-);
+const response = await client.zrange(
+  `${ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX}${ticker}`,
+  0,
+  -1
+);
+expect(response[0]).toBe('50000');
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

      const response = await client.zrange(
        `${ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX}${ticker}`,
        0,
        -1
      );
      expect(response[0]).toBe('50000');

41-49: Correct the usage of Redis client's zrange method

The client.zrange method returns a Promise and does not accept a callback. Using a callback here is incorrect and could lead to unexpected behavior or errors.

Apply this diff to fix the issue:

-await client.zrange(
-  `${ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX}${ticker}`,
-  0,
-  -1,
-  (_: any, response: string[]) => {
-    expect(response).toEqual(['49000', '50000', '51000']);
-  },
-);
+const response = await client.zrange(
+  `${ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX}${ticker}`,
+  0,
+  -1
+);
+expect(response).toEqual(['49000', '50000', '51000']);
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

      const response = await client.zrange(
        `${ORDERBOOK_MID_PRICES_CACHE_KEY_PREFIX}${ticker}`,
        0,
        -1
      );
      expect(response).toEqual(['49000', '50000', '51000']);
    });

81-104: Ensure timers are properly restored after using fake timers

When using jest.useFakeTimers(), it's important to restore the real timers after the test completes, even if an error occurs, to avoid side effects on other tests.

Consider wrapping the test code in a try...finally block to ensure jest.useRealTimers() is always called:

 it('returns the correct median price after 5 seconds', async () => {
+  try {
     jest.useFakeTimers();

     // ... rest of the test code ...

     const result = await getMedianPrice(client, ticker);
     expect(result).toBe('52500');
-    jest.useRealTimers();
+  } finally {
+    jest.useRealTimers();
+  }
 });
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    it('returns the correct median price after 5 seconds', async () => {
      try {
        jest.useFakeTimers();

        const nowSeconds = Math.floor(Date.now() / 1000);
        jest.setSystemTime(nowSeconds * 1000);

        await Promise.all([
          setPrice(client, ticker, '50000'),
          setPrice(client, ticker, '51000'),
        ]);

        jest.advanceTimersByTime(6000); // Advance time by 6 seconds
        await Promise.all([
          setPrice(client, ticker, '49000'),
          setPrice(client, ticker, '48000'),
          setPrice(client, ticker, '52000'),
          setPrice(client, ticker, '53000'),
        ]);

        const result = await getMedianPrice(client, ticker);
        expect(result).toBe('52500');
      } finally {
        jest.useRealTimers();
      }
    });

@adamfraser adamfraser merged commit 637c237 into main Sep 23, 2024
16 checks passed
@adamfraser adamfraser deleted the adam/candles-fix branch September 23, 2024 17:24
adamfraser added a commit that referenced this pull request Sep 24, 2024
mergify bot pushed a commit that referenced this pull request Sep 24, 2024
adamfraser added a commit that referenced this pull request Sep 24, 2024
Co-authored-by: Adam Fraser <adamfraser0@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

2 participants