Question for different Data Prep AND Example Codes

[pakornavakul]

Hello, I'll try to keep this concise. Thanks much in advance!

Data Prep

I'm trying to use another data source (traditional assets) for this libraries as I don't have long historical data as messages, but I do have Order book snapshots at ticks events and trades events.

Currently, I convert my LOB snapshot into state changes for each price level that has its quantity change or sign changes
If price is bid side, side will be 1 while ask side -1

For example ,
time 0: bid price 100 has 50 qty
time 1: bid price 100 has 25 qty

Event	exch_timestamp	local_timestamp	side	price	qty
1	xxxx	yyyy	1	100	25

time 0: bid price 100 has 50 qty
time 1: bid price 100 has 60qty

Event	exch_timestamp	local_timestamp	side	price	qty
1	xxxx	yyyy	1	100	60

I will do this for all my LOB updates, where if price disappears from the snapshot, I assume nothing happen until the price level appears in the snapshot again.

Then I convert my trades into your format and join the two table and sort by exchage ts

Is this wrong? Or should I just use event type 4 and put in the prices and qty?

Backtesting Example

Referencing your GLFT example,

From what I understand, you are requoting every 100ms, at around 10-25 halfspread (depending on the coef and vol) ticks. it just surprises me that it trades that much with such a fast quoting around mid price, but it's possible for ETH with delays included.

Hopeful

[nkaz001]

If I correctly understand what you intend, this is the right approach. You can refer to this method, which converts the snapshot into incremental changes. You may need to check if the conversion is correct.

Regarding the second point, the GLFT model is derived from a high-frequency grid-trading model that fixes the quoting price within a given grid. Therefore, even though the quoting interval is short, it does not update orders at every interval since the quoting price remains fixed within the grid.

[pakornavakul]

Ah I did not know you had this script, will be going through them! Thanks.

As for the GLFT, I found that it is calculating bid,ask price that you want to quote and placing order every 100ms while canceling the previous ones that are at different prices. The quotes are based on mid_price +/- half_spread and some skew. This will, not all the time, move the quoted price every time mid price moves? If this is incorrect, I apologize in advance.

[nkaz001]

That's correct. I may have misunderstood your point. The GLFT model is based on order arrival rates for each depth (price), which are measured in 100ms intervals. When you examine the order arrival rates around the mid-price, you can see that, even within a 100ms interval, orders can arrive at deeper prices for many crypto pairs. Therefore, even though quotes update around the mid-price every 100ms, there is still a chance for the orders to be filled.

[pakornavakul]

@nkaz001 Hello,

I wanted to revisit this question regarding creating valid data from market snapshot (orderbook) + trades data. I tried running simple live data model and compared the bid/ask mid against the backtest and I found some discrepancies. I wanted to revisit your code, but It appears that in the new version, you have taken /hftbacktest/data/utils/difforderbooksnapshot.py out. Was there something that prompt it to be removed? Much Appreciated!

[nkaz001]

Hello,

That moved to https://github.com/nkaz001/hftbacktest/blob/master/py-hftbacktest/hftbacktest/data/utils/difforderbooksnapshot.py

I didn't test it comprehensively, as the desired data is at least an L2 Market-By-Price feed. Please let me know if you find anything that needs to be fixed.

[pakornavakul]

Hi,

I just saw that you are updating to v2, which looks amazing. Thanks for the update. Now, I want to convert this difforderbooksnapshot.py to fit the new data requirements.

I am unsure what to do when OUT_OF_BOOK_DELETION_BELOW, OUT_OF_BOOK_DELETION_ABOVE, IN_THE_BOOK_DELETION occurs. Let's say that our asks are
[
[100, 10],
[101, 10],
[102, 5],
[103, 3],
[106, 5]
]
and for case OUT_OF_BOOK_DELETION_ABOVE
let's say
[
[100, 10],
[101, 10],
[102, 5],
[103, 3],
[104, 3],
]
level 106 is deleted, what combination of event flag is supposed to be used here and how to create the data row for this event?

And for others' OUT_OF_BOOK_DELETION_BELOW, IN_THE_BOOK_DELETION

Much Appreciated!

[nkaz001]

The fundamental approach is to treat all DELETION flags as DEPTH_EVENT with a quantity of 0. If I remember correctly, the purpose of adding these DELETION flags is to support users who wish to retain a kind of afterimage of the market depth. For example, in your example, the market depth is limited to the top 5 levels. But, you might want to infer a more extended market depth beyond these levels. A simple method to achieve this is by maintaining the market depth beyond the visible top 5 levels. We cannot confirm whether the ask at 106 is deleted or not, as the information falls outside the top 5 levels. By preserving this information, users can create an extended depth by retaining an afterimage of the market depth. Ignoring OUT_OF_BOOK_DELETION_ABOVE in the ask and OUT_OF_BOOK_DELETION_BELOW in the bid preserves the market depth beyond the top 5 levels, thereby retaining an afterimage.