You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is a bug in the code (and a simple fix as well ) in dealing with limit in table.py:
# Avoid round-trip when exhausted
if len(items) < how_many:
break
To provide some context, items stores the rows for current batch returned from the HBase server (according to a scan iterator) . how_many indicates the number of rows fetched for the batch. The idea of the above statements is to avoid keeping fetching if the number of returned rows in items is less than how_many.
However, note that the how_many is only a suggestion to the server, not a requirement: the size of the batch returned from the server may well be less than the suggested number (how_many) despite there could still be more to return. This cause the scan to return immaturely. The fix is easy:
# if len(items) < how_many:
# break
if len(items) == 0:
break
The text was updated successfully, but these errors were encountered:
I have used the following very simple code to debug table.py. Basically, I just print the number of items returned by inserting the following in the table.py:
print '===:', len(items)
Certainly, I need to apply the aforementioned fix to avoid premature exit. Then, I used the following to test against my table:
import happybase
conn=happybase.Connection(thrift)
tb=conn.table(tbname)
scanner=tb.scan(row_prefix='com.google', limit=1000, batch_size=100)
for (k,v) in scanner:
pass
Despite the batch_size=100, we see the items returned during the scan having the length less than 100 (i.e. 48, 8, 7, 10). I have not looked in to the HBase code yet, so I can not confirm whether it is intended or accidental from HBase side. But I can understand if cache_size in scan is only a suggestion number.
BTW, this is on MapR Table implementation of Big Table.
There is a bug in the code (and a simple fix as well ) in dealing with limit in
table.py
:To provide some context,
items
stores the rows for current batch returned from the HBase server (according to ascan
iterator) .how_many
indicates the number of rows fetched for the batch. The idea of the above statements is to avoid keeping fetching if the number of returned rows initems
is less thanhow_many
.However, note that the
how_many
is only a suggestion to the server, not a requirement: the size of the batch returned from the server may well be less than the suggested number (how_many
) despite there could still be more to return. This cause the scan to return immaturely. The fix is easy:The text was updated successfully, but these errors were encountered: