Writing dates with toArrow seems broken

[bmesuere]

When processing a csv file with 8M rows containing Date objects, I noticed the dates in many rows (from a certain point over halfway the dataset) went missing after writing the arrowBuffer to file and reading it again. I managed to construct a minimal example using only 10 rows:

import * as aq from "arquero"
import { writeFile } from "fs/promises";

let data = aq.table({ "time": ["2021-06-14 19:32:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00", "2021-06-14 19:33:00"]});

data = data.select("time").derive({time: aq.escape(d => new Date(d.time))});
data.print();

let data2 = aq.fromArrow(data.toArrow());
data2.print();

The first print results in the correct output:

Table: 1 col x 10 rows. Showing 10 rows.
┌─────────┬──────────────────────────┐
│ (index) │           time           │
├─────────┼──────────────────────────┤
│    0    │ 2021-06-14T17:32:00.000Z │
│    1    │ 2021-06-14T17:33:00.000Z │
│    2    │ 2021-06-14T17:33:00.000Z │
│    3    │ 2021-06-14T17:33:00.000Z │
│    4    │ 2021-06-14T17:33:00.000Z │
│    5    │ 2021-06-14T17:33:00.000Z │
│    6    │ 2021-06-14T17:33:00.000Z │
│    7    │ 2021-06-14T17:33:00.000Z │
│    8    │ 2021-06-14T17:33:00.000Z │
│    9    │ 2021-06-14T17:33:00.000Z │
└─────────┴──────────────────────────┘

The second has missing dates:

Table: 1 col x 10 rows. Showing 10 rows.
┌─────────┬──────────────────────────┐
│ (index) │           time           │
├─────────┼──────────────────────────┤
│    0    │ 2021-06-14T17:32:00.000Z │
│    1    │ 2021-06-14T17:33:00.000Z │
│    2    │ 2021-06-14T17:33:00.000Z │
│    3    │ 2021-06-14T17:33:00.000Z │
│    4    │ 2021-06-14T17:33:00.000Z │
│    5    │ 2021-06-14T17:33:00.000Z │
│    6    │ 2021-06-14T17:33:00.000Z │
│    7    │ 2021-06-14T17:33:00.000Z │
│    8    │       Invalid Date       │
│    9    │       Invalid Date       │
└─────────┴──────────────────────────┘

I am using node 16.3.

There is of course always the possibility I made an embarrassing mistake.

edit: an even more minimal example

[jheer]

No, the embarrassing mistake is on this end :) Fix coming soon.

[bmesuere]

That was quick, thanks! 🙏