CSV reports missing metadata fields even when selected

[zhongnansu]

ref: metaData fields

When user selected metadata fields like "_index", "_type" in Discover, it will not appear in csv reports, without throwing any error.

Root cause

https://github.com/opensearch-project/dashboards-reports/blob/56008de1dd447b3b4967e9abdea692e1b4036119/dashboards-reports/server/routes/utils/dataReportHelpers.ts?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L194

let result = traverse(data._source, report._source.selectedFields)

The traverse logic aims to find the data fields by matching the selected fields. Selected metaData fields will be saved in selected fields, this part is fine. The issue is with the data._source

The following is an example of the data object, and you can see metadata fields are in the data object, but the data._source field we passed to traverse()for matching will not include the metaData fields anyways.

{
  "_index": "opensearch_dashboards_sample_data_ecommerce",
  "_type": "_doc",
  "_id": "j89pyHwBt6Wu9HvV9AcN",
  "_version": 1,
  "_score": null,
  "_source": {
    "category": [
      "Women's Clothing"
    ]
  },
  "fields": {
    "order_date": [
      "2021-10-28T19:27:50.000"
    ],
    "products.created_on": [
      "2016-12-15T19:27:50.000",
      "2016-12-15T19:27:50.000"
    ]
  },
  "sort": [
    1635449270000
  ]
}

[zhongnansu]

@joshuali925 I notice that before your PR to fix the csv rendering nested fields, it was using data not data.source when traverse(), which contained the metadata fields. Is it straightforward to add it back?

https://github.com/opensearch-project/dashboards-reports/blame/f517f3f616e0f2a1ebec7d5db9731bc5d0275e89/dashboards-reports/server/routes/utils/dataReportHelpers.ts#L184

[joshuali925]

@zhongnansu I see what the issue is, you can revert it to data, and change this line https://github.com/opensearch-project/dashboards-reports/blob/56008de1dd447b3b4967e9abdea692e1b4036119/dashboards-reports/server/routes/utils/dataReportHelpers.ts#L232

to

      result[prefix.replace(/^_source\./, '') + key] = value;