OverflowError: Python int too large to convert to C long

[valentina-bec]

when importing the model:

from aitextgen import aitextgen

i got the error:

csv.field_size_limit(sys.maxsize)
OverflowError: Python int too large to convert to C long

solved by changing
csv.field_size_limit(sys.maxsize)
in TokenDataset.py

by :

maxInt = sys.maxsize
try:
    csv.field_size_limit(maxInt)
except OverflowError:
    maxInt = int(maxInt/10)

Does this change affect the model?

[MarcusLlewellyn]

So, I'm out of my depth here. But this looks like it is expecting a C long, but sys.maxsize returns the value for a C long long or 9223372036854775807 on my system (Python 3.8.7 64-bit on Windows 10). I think that the maximum value in this case might actually need to be 2^32 - 1 or 2,147,483,647.

[minimaxir]

This was added with #67

That change will not affect the model/dataset construction at all.

What OS are you using? I see 9223372036854775807 as well. I have no issues explicitly setting 2**32 - 1.

[MarcusLlewellyn]

@minimaxir Windows 10 build 19042 64-bit with Python 3.8.7 64-bit.

Edit: And it still fails when I plug 2**32 -1 in as a magic number. :(
Edit2: 2**16 -1 works. I get a whole new error, but that's probably for another issue.

[smt923]

I have this issue also, Windows 10 20H2 (19042.804), using Python 3.9.2, the change in TokenDataset.py also fixed it for me and allows me to import and generate

for what it's worth I also see 9223372036854775807 for sys.maxsize, and just out of curiosity I tried it on WSL2 Ubuntu and it does work fine there without any changes

[minimaxir]

So this is apparently a complicated quirk. From that, it seems 2 ** 31 - 1 is fine.