Reduce memory footprint of file-level statistics

[dispanser]

Description

I've been playing with a relatively large delta table recently, in the context of #425 (ignore tombstones for readers).

While the 5.3 million file paths themselves occupy about 600M, the overall memory consumption of a small rust program after loading the delta table adds up to 49.4G.

Here's the memory consumption for the various fields in action::Add, on that table with 5.3 million files, and 19 columns (each column has file statistics enabled).

field	mem ps
stats parsed	37.1G
stats	6.4G
partition values	2.1G
parsed partition values	1.9G
path	600M

I think we should be able to get that down considerably. The biggest offender is stats_parsed, which is of type Option<parquet::record::Row>. If I understand the concept correctly, it has minValues, maxValues, nullCounts and numRecords as top-level entries. Each of the first three then has a list / vec of tuples of (column name, value), so it effectively repeats each of my column names three times, for every individual file.

[mgill25]

I did some memory inspections. Just in case anyone else is working from a Mac (@dispanser uses Arch):

I tried to reproduce this and was very confused because my htop did not show high memory usage. I was expecting the program with the same large dataset to blow up/or at least be OOM-killed. But turns out both Memory Compression and Swap played a role in displaying an incomplete picture (as far as htop) was concerned.

_delta_log size: ~15G

htop after Table Load:

Activity Monitor shows the rest of the picture:

Here we indeed see ~44G memory utilization (post table load). We also observe heavy Swap and Memory Compression by the OS:

The numbers come down when the program is killed:

[houqp]

I also think there is room to reduce the memory usage by 10x. I would drop all of stats, stats_parsed, partition_values and parsed_partition_values in memory. Both stats and partition values can be stored in a columnar format, e.g. arrow record batch, which should also help with applying filter push down on table scan.

[mgill25]

@houqp @dispanser I'll try to do some refactoring and see what I can come up with. The First approach would be exactly as you suggest, just try to store the fields using record batches. I hope to make some progress (first time digging into the codebase) but I'll ask for advice here if I get stuck :)

[houqp]

Thanks @mgill25 , let us know if you need any help :)

[dispanser]

Tombstones have a similar structure and suffer from the memory consumption problem to some extend, having a field pub partition_values: HashMap<String, Option<String>> that's contributing to the overall memory consumption of tombstones considerably.

If we find an efficient way to represent Add operations, it would be great if that is also applied for tombstones in a later PR.

[houqp]

Good call @dispanser for tombstone, i think we can further optimize it by remove all other fields and only keep path and deleted_timestamp. I don't think we need access for other fields for vacuum.

[mgill25]

@houqp and @dispanser Just to provide an update here, I have some working code, with no doubt lots of enhancements to be done.

On the plus side, I can read stats_parsed , and then make record batches out of them for any provided columns. On the negative side, I'm still working with the assumptions of i64s as data types (to make the testing easy). I'll do some clean up tomorrow and post a gist for you to take a look at?

The annoying part so far has mostly been trying to parse the extremely deeply nested structure and trying to re-arrange them into some familiar data structures. I suspect things will get better with time :)

[houqp]

@mgill25 please feel free to send a draft PR directly if you prefer, we can comment and iterate there.

[roeap]

@dispanser - do you still have access to that table. If so, I'd be very curious to see what it looks like after the move to an arrow backend :).

[ion-elgreco]

Closing this one as it's potentially outdated and we have already made improvements here, if people perceive new issues with memory usage please create a new issue for tracking :)