dropmissing! creates weird memory bugs/errors on 1.7 and 1.6

[pcjentsch]

The MWE is available in this repository. It uses the publicly available Our World in Data COVID dataset. Let me know if this should be posted in the Julia language repository instead. I am using the latest DataFrames release.

This code creates a few different error messages.

On 1.7

corrupted double-linked list

another message

julia: malloc.c:4259: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed.

(then hangs)

On 1.6

julia-1.6: malloc.c:4259: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed.

signal (6): Aborted
in expression starting at /home/peterj/scratch.jl:45

Generally replacing the line

      dropmissing!(location_subset, [:new_cases_smoothed,:new_vaccinations_smoothed,:total_cases])

with

 location_subset =dropmissing(location_subset, [:new_cases_smoothed,:new_vaccinations_smoothed,:total_cases])

avoids this error.

[bkamins]

I have traced the source of the bug. Unfortunately it seems to be in Julia Base (which is unlikely), so this is strange.

The problem happens if rows that need to be removed from location_subset form a continuous block. In this case Julia Base calls a fast method for deleteat! (which takes a starting position and number of elements to remove). For some strange reason this sometimes fails.

The problem is fixed when in line

DataFrames.jl/src/dataframe/dataframe.jl

Line 978 in ab5ffd7

drop = _findall(inds)

we write instead:

drop = collect(_findall(inds))

(this forces Julia Base to use another internal implementation of deleteat!).

I will try asking Julia core devs for help.

[bkamins]

@pcjentsch - if you can/have time can you please test the code with the fix in JuliaLang/julia#41646 (if not I can do it later, but currently I do not have access to an appropriate machine to do this). Thank you!

Nevertheless I will add a fix to DataFrames.jl code and make a patch release.

Thank you for reporting.

[bkamins]

@pcjentsch - please update DataFrames.jl to 1.2.1 release and all will work OK.

[pcjentsch]

Great, thank you! I will not have time to test the new Julia branch for the next week or so but I appreciate your fast response time.