feat[rust, python]: if datetime format not specified for "write_csv", infer the required precision

[alexander-beedie]

If the datetime_format parameter is not set by the caller, the CSVWriter defaults to nanosecond precision, even when not required. This patch allows the write_csv call to infer the maximum required output precision so that unnecessary decimals aren't written, and the precision of the output matches the precision of the frame.

(If the datetime_format parameter is set, it is of course respected, and no inference is done).

Example

from datetime import datetime, date
import polars as pl

# frame with millisecond-precision datetime cols
df = pl.DataFrame(
    data = {
        "colx": [datetime(2022,9,4,10,30,45,123000)],
        "coly": [datetime(2022,9,4,10,30,45,123000)],
    },
    columns = [
        ("colx", pl.Datetime("ms")),
        ("coly", pl.Datetime("ms")),
    ],
)
print( df.write_csv() )

Before (forces "ns" precision on output)

colx,coly
2022-09-04T10:30:45.123000000,2022-09-04T10:30:45.123000000

After (infers actual "ms" precision: 20% more compact)

colx,coly
2022-09-04T10:30:45.123,2022-09-04T10:30:45.123

FYI: I did consider trying per-column precision on output, but it felt like an unnecessarily heavyweight approach, and was more intrusive in the code; given that most frames are likely to have consistent time units this seemed more reasonable.

[matteosantama]

@alexander-beedie you mention that your approach is a compromise since per-column formatting was too heavyweight. Accommodating per-column formatting would be pretty surgical, though -- you could simply change these lines to

            None => {
                let dt = match tu {
                    TimeUnit::Nanoseconds => temporal_conversions::timestamp_ns_to_datetime(v),
                    TimeUnit::Microseconds => temporal_conversions::timestamp_us_to_datetime(v),
                    TimeUnit::Milliseconds => temporal_conversions::timestamp_ms_to_datetime(v),
                };
                let fmt = match &options.datetime_format {
                    None => match tu {
                        TimeUnit::Nanoseconds => "%FT%H:%M:%S.%9f".to_string(),
                        TimeUnit::Microseconds => "%FT%H:%M:%S.%6f".to_string(),
                        TimeUnit::Milliseconds => "%FT%H:%M:%S.%3f".to_string(),
                    },
                    Some(fmt) => fmt,
                }
                write!(f, "{}", dt.format(fmt))
            }

Are you afraid of a performance penalty this way?

[matteosantama]

Actually, I think this would be faster than your way -- primarily because it requires no additional looping

[ritchie46]

This would be a heap alloc to string every value written. That would be slow.

I don't think we need the string alloc. But I don't think we can make this performance claims, as this hits every written value.

[matteosantama]

Ah right, this is on a per-value basis, not per-column. Fair enough

[matteosantama]

You are right -- the .to_string() bit is unnecessary.

There isn't much of a performance penalty to my approach

df = pl.Series(pd.date_range("2000-01-01", "2100-01-01")).to_frame()

%timeit df.write_csv()

# master
46.5 ms ± 1.38 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# my approach
46.5 ms ± 777 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

[ritchie46]

This is not really a good benchmark. On writing such a small DataFrame, we are only benchmarking virtual dispatch, and likely python interop. Try measuring different row sizes up to a millions rows or so, such that we benchmark the hot loop, not the setup code.

This different row sizes tells us more about how it deals with caches.

[alexander-beedie]

@matteosantama

you mention that your approach is a compromise since per-column formatting was too heavyweight

Mostly because on my first approach I found myself making it unnecessarily complicated (I was allowing a different user-specified format per time-unit and it became... unruly ;). So, I pulled back to a cleaner implementation that would slide into the existing code with minimal fuss. Would be nice to do per-timeunit formatting though... (I added something similar for cast(str) datetimes last week).

@ritchie46

try measuring different row sizes up to a millions rows or so, such that we benchmark the hot loop

Done; the per-time unit approach above becomes ~5-7% slower on larger data (I benched on ~63,000,000 rows, as created by: df = pl.Series(pl.date_range(date(2000,1,1), date(2001,12,31), "1s")).to_frame()). So not much slower, but not faster - at least, not on my home setup (vanilla M1).

[ritchie46]

Yep, I'd expect something like that.

I think we should go for your approach. It is simple.

We can explore something more fine grained later, but I think that should require some effort to not loose performance.

[matteosantama]

@alexander-beedie thanks for checking!

[ritchie46]

Thanks @alexander-beedie. That will save a lot of future bytes written. :)