[BUG]Discover: Some Fields Aren't Displayed in the Left Navigation Bar

[MadaniKK]

Describe the bug
In Discover some fields are not being displayed in the left Field Nav

To Reproduce

• step 1: go to dev tools. create an index and add nested field.

PUT test-2/_doc/1
{
  "products": [
    {
      "base_price": 28.99,
      "discount_percentage": 0,
      "quantity": 1,
      "manufacturer": "Tigress Enterprises"
    },
    {
      "base_price": 32.99,
      "discount_percentage": 0,
      "quantity": 1,
      "manufacturer": "Oceanavigations"
    }
  ]
}

PUT test-2/_doc/2
{
  "products": [
    {
      "base_price": 31.14,
      "discount_percentage": 5,
      "quantity": 1,
      "manufacturer": "Virora"
    },
    {
      "base_price": 35.69,
      "discount_percentage": 10,
      "quantity": 3,
      "manufacturer": "Oceanavigations"
    }
  ]
}

• step 2: go to Dashboards management --> Index patterns --> create index pattern

• step 3: enter test* to match test-2
• go back to discover. in the index pattern selector (on top of left nav), choose test*.

So in new Discover (2.11), if you check the left field nav, you could see we miss the nested fieldproducts. but the flyout and display are fine. It is just that there is no way customer could select nested field.

But if we do the above steps in 2.9 OSD with legacy discover, we could see products in left nav and once click + button to select it, you could see multiple lines in each row as customer shown.

I think we need to first solve the nested field not picked up by left nav then verify if we see multiline issue after select the nested field.

Originally posted by @ananzh in #5208 (comment)

Expected behavior
The Left field nav should display all these fields

OpenSearch Version
3.0.0

Dashboards Version
Discover 2.11

Plugins

Please list all plugins currently enabled.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Browser and version [e.g. 22]

Additional context

Add any other context about the problem here.

[MadaniKK]

Here is what @ananzh and I found after some digging:
the fields are rendered based on some conditions, which involve mostly two objects indexPattern and fieldCounts. What is causing the subfields of Product not to be displayed is the construction of fieldCounts. All the subfields exist in the object indexPattern but not in fieldCounts

We found that the flattenHit(row) function here never unnest the subfields inside Product, causing all the subfields missing in 'fieldCounts`

OpenSearch-Dashboards/src/plugins/discover/public/application/view_components/utils/use_search.ts

Lines 164 to 175 in eeb3251

	const hits = fetchResp.hits.total as number;
	const rows = fetchResp.hits.hits;
	let bucketInterval = {};
	let chartData;
	for (const row of rows) {
	const fields = Object.keys(indexPattern.flattenHit(row));
	for (const fieldName of fields) {
	fetchStateRef.current.fieldCounts[fieldName] =
	(fetchStateRef.current.fieldCounts[fieldName] || 0) + 1;
	}
	}

[MadaniKK]

I am also confused about the purpose of fieldCounts. We will do more research on this later.
And I wonder, when constructing the field counts, why can't we just use whatever existing keys in 'IndexPattern'. Because indexPattern must have gone through unnesting successfully every time we create/update a new fields.
Thus I thought const fields = Object.keys(indexPattern.flattenHit(row)); can be changed into const fields = indexPattern.fields.getAll().map((field) => field.displayName);

OpenSearch-Dashboards/src/plugins/discover/public/application/view_components/utils/use_search.ts

Lines 164 to 175 in eeb3251

	const hits = fetchResp.hits.total as number;
	const rows = fetchResp.hits.hits;
	let bucketInterval = {};
	let chartData;
	for (const row of rows) {
	const fields = Object.keys(indexPattern.flattenHit(row));
	for (const fieldName of fields) {
	fetchStateRef.current.fieldCounts[fieldName] =
	(fetchStateRef.current.fieldCounts[fieldName] || 0) + 1;
	}
	}

But Still right now we are still going to preserve the old codes and just fix the flattenHit() function

[MadaniKK]

We just discovered that if we don't construct the fields through the original way const fields = Object.keys(indexPattern.flattenHit(row)); , it would cause a new bug: cannot display nested object properly in Discover table
new bug:

expected:

So the best way to resolve this issue is still using flattenHitWrapper functions

[willie-hung]

@MadaniKK Thank you for sharing the findings.

[ananzh]

resolved by #5429