[event log] setting event log index mappings when creating initial index

[pmuellr]

Currently when a new set of event log elasticsearch resources are being created (because a new deployment, or deployment upgraded their stack version), we perform the following steps:

• create index template that includes mappings
• create initial index that should reference the index template for default settings, including the mappings

full steps:

kibana/x-pack/plugins/event_log/server/es/init.ts

Lines 28 to 35 in f4f51e6

	async function initializeEsResources(esContext: EsContext) {
	const steps = new EsInitializationSteps(esContext);
	await steps.setExistingAssetsToHidden();
	await steps.createIlmPolicyIfNotExists();
	await steps.createIndexTemplateIfNotExists();
	await steps.createInitialIndexIfNotExists();
	}

The assumption is that the mappings will be in the template, and then used when we create the initial index. So we don't set the mappings in the create index call:

kibana/x-pack/plugins/event_log/server/es/init.ts

Lines 204 to 216 in f4f51e6

	async createInitialIndexIfNotExists(): Promise<void> {
	const exists = await this.esContext.esAdapter.doesAliasExist(this.esContext.esNames.alias);
	if (!exists) {
	await this.esContext.esAdapter.createIndex(this.esContext.esNames.initialIndex, {
	aliases: {
	[this.esContext.esNames.alias]: {
	is_write_index: true,
	is_hidden: true,
	},
	},
	});
	}
	}

However, we do have an issue from a user where the default auto-mapping policy was used for the initial event log index, which ends up creating an event log index that we can't search over correctly (because the auto-mapped field names don't correspond to what we're using for keyword/text, etc).

It's not clear HOW this happened. But in case there was some kind of race condition going on (the user we saw this with had two Kibana's running / restarting after an upgrade simultaneously), it seems safer to just set the mappings in the create call itself - as well as in the template (for ILM-created indices once ILM rolls over an index).

I suspect we'll want to chat with Elasticsearch folks on the best approach here. Maybe it's safer to create the index with the mappings, and THEN create the template? The mappings will be same for both. And I'd guess we want to apply ALL the template settings, not just mappings. Here's what we currently set in the template:

kibana/x-pack/plugins/event_log/server/es/documents.ts

Lines 12 to 28 in f4f51e6

	export function getIndexTemplate(esNames: EsNames) {
	const indexTemplateBody = {
	index_patterns: [esNames.indexPatternWithVersion],
	template: {
	settings: {
	number_of_shards: 1,
	auto_expand_replicas: '0-1',
	'index.lifecycle.name': esNames.ilmPolicy,
	'index.lifecycle.rollover_alias': esNames.alias,
	'index.hidden': true,
	},
	mappings,
	},
	};
	return indexTemplateBody;
	}

It's possible the problem encountered by the user could have happened through human intervention. Perhaps the user removed some of the event log resources (like the template), then restored the actual event log index and with no template, the auto-mapping got applied. I think any solution we're looking for here should hopefully prevent the auto-mapping from occurring via scenarios like this.

related: #127029

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mikecote]

Closing in favour of #127029.