Type coercion / ignored format on export?

[rjbudzynski]

In my table I have a column where the values are strings of two digits, like "02", "04", ..."12", "22" and so on.
If I call table.toJSON() values in this column are coerced to numbers. This is most certainly not what I want, and it's very undesirable when dealing with codes which are strings of digits which might have leading zeroes.
I even tried table.toJSON({ format: { WOJ: d => String(d).padStart(2,"0") }}) but the format seems to be ignored.
A similar issue arises when exporting to Arrow, specifying the arrow type of the column as Utf8 has no effect.

[jheer]

The output methods appear to be working fine:

aq.table({ foo: ["02", "04", "12", "22"] }).toCSV()
// foo
// 02
// 04
// 12
// 22

aq.table({ foo: ["02", "04", "12", "22"] }).toJSON()
// {"schema":{"fields":[{"name":"foo"}]},"data":{"foo":["02","04","12","22"]}}

I'm guessing the problem is that the values are being parsed as strings upon load. So I'd look at the data loading and possibly make changes there. Below is an example that adds options to the CSV parser to avoid coercing digit strings.

// standard auto-type parsing results in number coercion
aq.fromCSV(`foo\n02\n04\n12\n22`).toJSON()
// {"schema":{"fields":[{"name":"foo"}]},"data":{"foo":[2,4,12,22]}}

// using custom identify function parser for column foo
aq.fromCSV(`foo\n02\n04\n12\n22`, { parse: { foo: x => x } }).toJSON()
// {"schema":{"fields":[{"name":"foo"}]},"data":{"foo":["02","04","12","22"]}}

[rjbudzynski]

This (observable framework) page displays alike behavior when exporting to apache-arrow:
https://budzynski.xyz/arquero-test/

[jheer]

Hmm. I'm not able to replicate the issue. I added the following cells in an Observable notebook, and the end result remains string values, as expected. I also tested locally (MacOS, node 18, Arrow v16) and see the same behavior. I also tested with and without explicit types in the toArrowBuffer call, and get strings either way. Not sure what is different on your end...

dataStr = `WOJ;POW;GMI;RODZ;NAZWA;NAZWA_DOD;STAN_NA\r
02;;;;DOLNOŚLĄSKIE;województwo;2024-01-01\r
02;01;;;bolesławiecki;powiat;2024-01-01\r
02;01;01;1;Bolesławiec;gmina miejska;2024-01-01\r
02;01;02;2;Bolesławiec;gmina wiejska;2024-01-01\r
02;01;03;2;Gromadka;gmina wiejska;2024-01-01\r
02;01;04;3;Nowogrodziec;gmina miejsko-wiejska;2024-01-01\r
02;01;04;4;Nowogrodziec;miasto;2024-01-01\r
02;01;04;5;Nowogrodziec;obszar wiejski;2024-01-01\r
02;01;05;2;Osiecznica;gmina wiejska;2024-01-01\r
02;01;06;2;Warta Bolesławiecka;gmina wiejska;2024-01-01\r
02;02;;;dzierżoniowski;powiat;2024-01-01\r
02;02;01;1;Bielawa;gmina miejska;2024-01-01\r
02;02;02;1;Dzierżoniów;gmina miejska;2024-01-01\r
02;02;03;3;Pieszyce;gmina miejsko-wiejska;2024-01-01\r
02;02;03;4;Pieszyce;miasto;2024-01-01\r
02;02;03;5;Pieszyce;obszar wiejski;2024-01-01\r
02;02;04;1;Piława Górna;gmina miejska;2024-01-01\r
02;02;05;2;Dzierżoniów;gmina wiejska;2024-01-01\r
02;02;06;2;Łagiewniki;gmina wiejska;2024-01-01\r
02;02;07;3;Niemcza;gmina miejsko-wiejska;2024-01-01\r
02;02;07;4;Niemcza;miasto;2024-01-01\r
02;02;07;5;Niemcza;obszar wiejski;2024-01-01\r
`;

nullIfEmpty = s => s.length ? s : null

TERC = aq.fromCSV(dataStr, { 
    delimiter: ";",
    parse: {
        WOJ: String,
        POW: nullIfEmpty,
        GMI: nullIfEmpty,
        RODZ: nullIfEmpty,
        NAZWA: nullIfEmpty,
        NAZWA_DOD: nullIfEmpty,
        STAN_NA: d => new Date(Date.parse(d))
    } 
}).select(aq.not("STAN_NA"))

// ["02", "02", "02", "02", "02", "02", "02", "02", ... ]
aq.fromArrow(TERC.toArrowBuffer()).array('WOJ')