V1 API /trades Missing Data

[matthewdowney]

I'm not sure if this is the right place put API issues — let me know if I should move this somewhere more appropriate!

I'm gathering public trade data from the trades rest endpoint and it seems like there are situations in which the API omits trades during pagination.

Every few minutes I'm hitting the endpoint asking for trades at or after the most recent trade that I've seen and paginating forward in time until I see the page with the most recent trades, but there are times when the API excludes some block of trades directly after the trade I am querying from.

Missing Trades

For example, in a previous request, I had seen the trade

{"timestamp":1557677028,
 "tid":356105007,
 "price":"6957.6",
 "amount":"0.16",
 "exchange":"bitfinex",
 "type":"sell"}

I subsequently queried https://api.bitfinex.com/v1/trades/btcusd?limit_trades=25&timestamp=1557677028 and saw that the first trade was

{"timestamp":1557677283,"tid":356106715,"price":"7000.0","amount":"0.01","exchange":"bitfinex","type":"sell"}

Since I had already seen the above trade with tid 356105007, I am sure there is at least one trade at the timestamp 1557677028, but the second request starts at 1557677283, 255 seconds later. Trades are missing!

Attempts to Resolve

Once I discovered the issue, I tried several things to figure out what was going on.

My first thought was that maybe the API docs were mistaken when they say that the timestamp parameter prompts it to "Only show trades at or after this timestamp". Perhaps the timestamp was exclusive. So subtracting one from the previous timestamp, we'd request https://api.bitfinex.com/v1/trades/btcusd?limit_trades=25&timestamp=1557677027, which still does not include the original trade.

I tried with many different limit_trades values to see if it might be a page size issue, but I saw the same problem with sizes of 25, 250, 500, and 999.

Interestingly, I was only seeing this problem on the BTC/USD book, so it might have something to do with the amount of activity taking place on the exchange.

Pagination Broken?

As I'm writing this, I'm checking the response of https://api.bitfinex.com/v1/trades/btcusd?limit_trades=25&timestamp=1557677028 periodically. It changes every time.

Is the pagination on the endpoint just broken? Is it taking limit_trades starting now, rather than at timestamp?

[matthewdowney]

I definitely think I'm seeing this issue more, now that the price is popping off! :)

I'm looking at switching to the V2 API but since it's still in beta I'm kind of reluctant, plus it doesn't report trade side which is really necessary for what I'm doing.

[tenninebt]

I'm facing it with the Websocket API (v1), a lot of missing trades, I'm using the api on a plain simple example, but subscribing to all of the margin symbols. Comparing to market trades directly on the web app, I can see a lot of discrepancies. Didn't try to switch to the V2 yet.

[matthewdowney]

Wow that's really interesting that you're seeing it on the websocket as well! I assumed it was some sort of caching issue, but that doesn't seem as likely if it's also affecting the socket.

[tenninebt]

Just realised there was maybe an issue on my side, I need to do more tests to confirm.