The strptime datetime parser doesn't support %s

[magnalite]

Data I want to ingest has ended up with timestamps formatted as seconds.milliseconds eg 1686764678.9197. I could transform these into ints however for large volumes of data that can be painful. I would instead like to parse it using the strptime parser via "%s.%f" however it doesn't recognise %s (a unix timestamp in seconds).

[fulmicoton]

Also, I'd love to have a sandbox in the UI for people to copy and paste their stuff and get visual feedback of how it is interpreted by quickwit.

[fulmicoton]

Ok so after investigating, ideally this means adding support for "%s" in time-fmt.

Just to be sure, can you confirm that your data is fed as a JsonString (as opposed to JsonNumber)

[magnalite]

Just to be sure, can you confirm that your data is fed as a JsonString (as opposed to JsonNumber)

I believe it is a JsonNumber. My data looks like this in its raw form
{ "timestamp":1686764741.2561002, ... }

I now realise that this is actually also beyond millisecond resolution too which is unclear to me if %f is happy to deal with or if it can only be milliseconds, eg needs to be truncated to 1686764741.256.

I am wondering if perhaps it would be more appropriate to add a further datetime type unix_timestamp_f64 which accepts unix time in seconds as a f64 (but then internally converted/used by quickwit as i64 ofcourse), maybe in addition to support for %s. However it would be good to add a warning in documentation to avoid using floats for timestamps as float precision is not guaranteed in json and can lead to data inaccuracies. For example if they have a json library which for whatever reason used floats instead of a doubles it would be wildly inaccurate but valid as per the json spec.

This is not something I had truly considered the repercussions of before now as I thought every json library would respect them being doubles, and in practice most do, but this is where I am at. As quickwit sees wider adoption I can see more people ending up in this scenario and instead of saying "you should just use ints" I believe it would be better to give guidance along the lines of "you really shouldn't use floats because xyz but if you really want to you can use unix_timestamp_f64".

[guilload]

Closed via #3639.