feat(operator): Add support for Loki OTLP limits config

[periklis]

What this PR does / why we need it:
The following PR adds support to the Loki Operator to handle OTLP configuration (global+per-tenant) on how to handle indexing fields from log streams ingested through Loki 3.x OTLP endpoint. The configuration options are provided in the LokiStack CRD below the LimitsSpec. Example LokiStack with this change:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: lokistack-dev
  namespace: openshift-logging
spec:
  limits:
    global:
      otlp:
        # Resource Attribtutes always treated as index labels by the distributor
        # Applied to the distributor.otlp_config.default_resource_attributes_as_index_labels
        indexedResourceAttributes: 
          - service.name
          - service.id
        # Anything below here are applied to the limits_config.otlp_config
        resourceAttributes:
          ignoreDefaults: true
          attributes:
            - action: structured_metadata
              attributes:
                - k8s.pod.name
                - k8s.container.name
        scopeAttributes:
          - action: structured_metadata
            attributes:
              - span.id
              - trace.id
        logAttributes:
          - action: drop
            attributes:
              - log.record.uid
    tenants:
      application:
        otlp:
          # Anything below here are applied to the limits_config.otlp_config
          # per tenant in the loki-runtime.yaml
          resourceAttributes:
            ignoreDefaults: true
            attributes:
              - action: structured_metadata
                attributes:
                  - k8s.pod.name
                  - k8s.container.name
                  - k8s.namespace.name
          scopeAttributes:
            - action: structured_metadata
              attributes:
                - span.id
                - trace.id
          logAttributes:
            - action: drop
              attributes:
                - log.record.uid
  size: 1x.demo
  storage:
    schemas:
    - version: v13
      effectiveDate: 2023-10-15
    secret:
      name: test
      type: s3
  storageClassName: gp3-csi
  tenants:
    mode: openshift-logging

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
It's worthwhile to consider the following Loki docs pages:

• Distributor: Default Resource Attributes indexed as Stream Labels
• OTLP ingestion: Send Data from OTELCollector

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[JoaoBraveCoding]

Didn't test, some comments

[JoaoBraveCoding]

Still have to test, some other "nits"

[xperimental]

First round of comments.

I noticed that IgnoreDefaults makes it easy to create a non-working configuration by specifying no index labels. Do we maybe need a validation for that?

Also this is another instance of the "array with objects where all attributes are optional"... maybe we also need a validation to guard against "all attributes empty"?

[xperimental]

Seems to work nicely in test cluster. 👍

Mostly corrected typos/wording. Only thing is a missing validation.

[xperimental]

Sorry for another round. Forgot to add one comment yesterday and also missed one thing in the config generation.

[xperimental]

Noticed yesterday, but forgot to add the comment... 🙄

Do we want to follow any existing convention with the enum values? Official Kubernetes is CamelCase, our convention in Loki Operator so far seems to be camelCase and not snake_case like in the Loki configuration.

Suggested change

	OTLPAttributeActionIndexLabel OTLPAttributeAction = "index_label"
	// OTLPAttributeActionStructuredMetadata stores an attribute as structured metadata with each log entry.
	OTLPAttributeActionStructuredMetadata OTLPAttributeAction = "structured_metadata"
	OTLPAttributeActionIndexLabel OTLPAttributeAction = "indexLabel"
	// OTLPAttributeActionStructuredMetadata stores an attribute as structured metadata with each log entry.
	OTLPAttributeActionStructuredMetadata OTLPAttributeAction = "structuredMetadata"

(needs to also update the validations if changed)