SO import/export: Improve JSON import experience

[PhaedrusTheGreek]

Currently Kibana supports newline delimited JSON as a valid "Upload a File" import, however the usability experience had some friction.

1. No more than 999 lines was supported, which maybe overly conservative.
2. A JSON array was not recognized as a valid file format. e.g.:

[
  {
    "attachment_created": "2021-08-30 14:29:52 UTC",
    "from_address": "redacted",
    "case_number": "redacted",
    "case_created": "2021-08-27 20:21:00 UTC",
    "diff_case_creation_minutes": "3968",
    "sub_id": "redacted",
    "sub_created": "2016-06-06 16:55:09 UTC",
    "diff_sub_creation_minutes": "2751694"
  },
  {
    "attachment_created": "2021-08-27 21:24:11 UTC",
    "from_address": "redacted",
    "case_number": "redacted",
    "case_created": "2021-08-27 20:21:00 UTC",
    "diff_case_creation_minutes": "63",
    "sub_id": "redacted",
    "sub_created": "2016-06-06 16:55:09 UTC",
    "diff_sub_creation_minutes": "2747789"
  }
]

In addition to newline delimited JSON, it seems logical to support array format as well.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

No more than 999 lines was supported

Hum, are you sure this is a object count limit and not a payload size limit? The default max number of objects is 10k

kibana/src/core/server/saved_objects/saved_objects_config.ts

Lines 48 to 51 in 393505a

	const soSchema = schema.object({
	maxImportPayloadBytes: schema.byteSize({ defaultValue: 26_214_400 }),
	maxImportExportSize: schema.number({ defaultValue: 10_000 }),
	});

In addition to newline delimited JSON, it seems logical to support array format as well.

Could you elaborate on why this would be logical, given that the export API can only provide export files in NDJSON?

[PhaedrusTheGreek]

I think it would be unreasonable to support arbitrary formats, however collections of JSON documents generally only come in these 2 formats (newline delimited and array), so I see it as being a controlled decision. Additionally, common tools export results in array format. For example, Google BigQuery's default export format is array, where each array element = 1 record result from query.

[pgayvallet]

I think it would be unreasonable to support arbitrary formats, however collections of JSON documents generally only come in these 2 formats (newline delimited and array), so I see it as being a controlled decision

To be totally honest, I personally think using ndjson as our export format was a terrible decision, especially due to it being a rather non-standard extension / format (e.g it's not even in some mimetype registries such as the one used by HAPI), and due to it's limitations compared to a plain json or json-extended format (e.g when working with non-consistent entries, when in need to add metadatas to the export...).

Now, the reason behind not allowing json files in an array format is that this file extension/format is already used by what we're currently calling the legacy import format, and adding logic to identify the format (legacy or new) of of a given json file would be quite complex, and impacting both server and client-side code.

Also, the main argument in favor of ndjson is also that is easily supports streaming, as you can process an element on the server-side as soon as you encountered a line separator, which is not the case for plain json (you need to await the whole payload to parse it). Sames goes for the export, where you can (more easily) write each entries individually in NDJSON than in JSON.

Now, this legacy format support is going to be dropped for 8.0 (#110738), so we could, potentially, open the discussion about using JSON format again. We're currently in the phase of designing what we currently call the 'new import/export', as we're planning to extend/refactor this API, probably shortly after 8.0, so this format discussion would probably wait until then.

[pgayvallet]

We realistically won't be implementing an additional import/export format anytime soon.

Closing.