bencoded item count limit exceeded #3037
Comments
|
According to the libtorrent docs, the bencoded dict was too large to decode (ref http://www.rasterbar.com/products/libtorrent/manual.html#lazy-bdecode). Since this call is hidden in |
|
As it seems that the fix for this would be to write our own |
|
It looks to me that this error is triggered by reaching |
|
@pimlie yes, but this is only available in the C++ backend: the Python binding to control this is no longer available. We would have to recompile libtorrent to expose this functionality (which is a fair bit of work and something we can't realistically support in the long run). |
|
@qstokkink Ah right, sorry. Couldn't we just submit an upstream issue for this with libtorrent then? I would think if its possible in the C++ backend it should be possible in the Python binding, or is that too simple? ;) |
|
@pimlie Once again correct, but that would take months, if not years, to trickle down into the package managers of different platforms. Short term, we would still need a workaround for this. |
|
@qstokkink unless you see more benefits from developing your own solution, I am not sure why you would go to all that trouble if this issue is the only reason for that. Probably that limit has been in libtorrent like forever and it seems nobody (also me) ever had problems with that limit before. So if I would need a specific lib version to be able to change that limit, I wouldnt regard that as a problem in this case. You just cant have it all ;) That said, being dependent on package managers of different platforms is never a good solution. Probably one of the reasons why so many open source applications provide users with their own repositories (like ppa's for ubuntu). As an organisation it gives you full control and makes it easier to eg introduce new features also for (maybe) out-dated platforms. Probably for this exact reason is why eg Deluge already has a ppa which includes their own libtorrent binary: https://launchpad.net/~deluge-team/+archive/ubuntu/develop/+packages |
|
@pimlie having a torrent which Tribler cannot handle, but other platforms can, is a thorn in my side. This hurts my pride. Nevertheless, I have been urged to delay fixing this until after the 7.0 release. But, yes, you are probably right. Considering the issues we had in the past, just like Deluge, it would probably be best to start maintaining our own libtorrent binary as well. |
|
I looked into this more. The only realistic option is to submit a PR to libtorrent to expose the bencoding limit. |
drew2a
modified the milestones:
7.6.0 November: Stability, usability, performance,
Next-next release
|
Is this issue outdated? In Tribler we use libtorrent version Line 26 in 932f51a Also, I don't see this error in Sentry:
|
|
This should still be an issue. Looking at the libtorrent source code, the default depth limit is Perhaps we can make our own huge torrent file for testing? |
|
Reproduced using the following script: # pip install bencode.py
import bencodepy
FILE_COUNT = 1000000
output = bencodepy.encode({
'nodes': [],
'info':
{
'files':
[
{'length': 1, 'path': [f'{i}.txt']} for i in range(FILE_COUNT)
],
'name': 'longtorrent',
'piece length': 1,
'pieces': b"\x01" * 20 * FILE_COUNT
}
})
with open("longtorrent.torrent", "wb") as f:
f.write(output)
f.flush()Output file: longtorrent.7z.gz I confirmed that a
If you click "download" anyway, you get this:
|
|
The following is maybe even more interesting: torrent of death 😃. Tribler crashes and auto-closes before you can submit a report: # pip install bencode.py
import bencodepy
PATH_DEPTH = 10000
output = bencodepy.encode({
'nodes': [],
'info':
{
'files':
[
{'length': 1, 'path': [f"{i}" for i in range(PATH_DEPTH)] + [f'file.txt']}
],
'name': 'deeptorrent',
'piece length': 1,
'pieces': b"\x01" * 20
}
})
with open("deeptorrent.torrent", "wb") as f:
f.write(output)
f.flush()Output file: deeptorrent.torrent.gz Digging into the logs, this is causing it: |
|
@qstokkink great investigation! I think now I can fix the issue (at least the torrent of local death). |
I have read CONTRIBUTING.rst
I have tried with the latest pre-release version and I still can reproduce the issue.
Tribler version/branch+revision:
$ git status
On branch next
Your branch is up-to-date with 'origin/next'.
nothing to commit, working directory clean
$ git show
commit bc8991f
Merge: fe5e6be 521a0af
Date: Wed Jun 21 13:30:50 2017 +0200
Operating system and version:
Ubuntu 16.04
Could not find any issues yet with this message. I was adding a torrent and suddenly Tribler crashed with the following error. The torrent is from a private tracker which I cant share unfortunately, but it consists of a lot of small files (the torrent file is ~25MB in size).
The text was updated successfully, but these errors were encountered: