Kernel dies for when reading beyond 4 million-th row of dataframe

[hjazz6]

Hi, sorry for all these questions. Really appreciate the help I've gotten here.

I was able to read multiple HDF5 files into a single vaex dataframe containing 1.12 billion rows. However, when I tried to do a groupby operation on the entire dataframe, the kernel dies.

I tried decreasing the size of the dataframe and found that when I'm trying to access beyond the 4 million-th row of the dataframe, the kernel usually dies. But the operation is always successful for the first 4 million rows.

# Executes successfully
df_short = df[:4000000]
df_count = df.groupby(df['columnA'], agg='count')
df.sort(by='count',ascending=False).head(10)

# Kernel dies
df_short = df[4000000:4500000]
df_count = df.groupby(df['columnA'], agg='count')
df.sort(by='count',ascending=False).head(10)

It does not seem related to the size of the dataframe, because the kernel fails when I'm trying to read 10 random rows beyond the 4 million-th row, e.g. df[9001000:9001010], though there are a few times when reading some other 10 rows is successful. Basically, when reading beyond the 4 million-th row, the kernel dies about 70% of the time, but operation is always successful when reading up to the 4 million-th row. I'm not sure why this is happening.

Thank you.

[maartenbreddels]

Hi,

how many unique values are there for columnA, and what is the dtype of it?
This seems like a bug btw, if you filled in the issue form, we ask you to list the Vaex version, Python version and OS version, could you provide us with that?

Regards,

Maarten Breddels

[hjazz6]

There are fewer than 300 unique values for columnA, and it's of str type. I have submitted a bug report for this. Thanks.