[Integration][Kubecost][Opencost] Add support for filters

[lordsarcastic]

Description

Docs PR

What:

• Added filter params to Opencost cloudcost and Kubecost cloud and kubesystem kind.
• Implemented separate config for v1 and v2 selectors for Kubecost cloud and kubesystem kind

Why:

• Opencost and Kubecost added the filter feature which is a much more powerful aggregation parameter as opposed to the existing fields, adding this field allows users to be much more flexible with what data they ingest into Port
• The Kubecost integration, while currently on v1 as set:

KUBECOST_API_VERSION_1 = "v1"

Uses the same selector to handle filters for both kubesystem and cloud kinds which should be separate. In addition, the params accepted by v1 is not the same as that on v2; this applies to both kinds.

How:
To address these needs, different selectors are defined for each kind and for each version and can be used when needed

Type of change

Please leave one option from the following and delete the rest:

• New feature (non-breaking change which adds functionality)
• Non-breaking change (fix of existing functionality that will not change current behavior)

Screenshots

Include screenshots from your environment showing how the resources of the integration will look.

API Documentation

Provide links to the API documentation used for this integration.

• Kubecost v1 Allocation API
• Kubecost v2 Allocation API
• Kubecost v1 Cloudcost API
• Kubecost v2 Cloudcost API
• Opencost Cloudcost API

Note: Kubecost docs have a slight error in their documentation, I have raised an issue on their repository for that.

Mapping config example for v1 kubesystem kind:

createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: kubesystem
    selector:
      query: "true"
      window: "7d"
      aggregate: service
      step: "1d" # this will result in 7 entries which is 1 day over the course of `window` time period: 7 days
      accumulate: false
      idle: true
      external: false
      filterLabels: "app:internal-service"
      filterServices: "notification,account,functions"
    port:
      entity:
        mappings:
          blueprint: '"kubecostResourceAllocation"'
          identifier: .name
          title: .name
          properties:
            cluster: .properties.cluster
            namespace: .properties.namespace
            startDate: .start
            endDate: .end
            cpuCoreHours: .cpuCoreHours
            cpuCost: .cpuCost
            cpuEfficiency: .cpuEfficiency
            gpuHours: .gpuHours
            gpuCost: .gpuCost
            networkCost: .networkCost
            loadBalancerCost: .loadBalancerCost
            pvCost: .pvCost
            pvBytes: .pvBytes
            ramBytes: .ramBytes
            ramCost: .ramCost
            ramEfficiency: .ramEfficiency
            sharedCost: .sharedCost
            externalCost: .externalCost
            totalCost: .totalCost
            totalEfficiency: .totalEfficiency

Mapping config example for v2 kubesystem kind:

createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: kubesystem
    selector:
      query: "true"
      window: "7d"
      aggregate: service
      step: "1d" # this will result in 7 entries which is 1 day over the course of `window` time period: 7 days
      accumulate: false
      idle: true
      filter: 'labels:"app:internal-service","app:service-2"+service:"notification","account","functions"' # replaces all `filter*` properties needed in v1
    port:
      entity:
        mappings:
          blueprint: '"kubecostResourceAllocation"'
          identifier: .name
          title: .name
          properties:
            cluster: .properties.cluster
            namespace: .properties.namespace
            startDate: .start
            endDate: .end
            cpuCoreHours: .cpuCoreHours
            cpuCost: .cpuCost
            cpuEfficiency: .cpuEfficiency
            gpuHours: .gpuHours
            gpuCost: .gpuCost
            networkCost: .networkCost
            loadBalancerCost: .loadBalancerCost
            pvCost: .pvCost
            pvBytes: .pvBytes
            ramBytes: .ramBytes
            ramCost: .ramCost
            ramEfficiency: .ramEfficiency
            sharedCost: .sharedCost
            externalCost: .externalCost
            totalCost: .totalCost
            totalEfficiency: .totalEfficiency

Mapping config example for v1 kubecostCloudAllocation kind

createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: cloud
    selector:
      query: "true"
      window: 7d
      aggregate: service
      filterAccountIDs: "984573291056,374859201234"
      filterLabel: label-one,label-two
      filterServices: notification,account,chat
    port:
      entity:
        mappings:
          blueprint: '"kubecostCloudAllocation"'
          identifier: .properties.provider + "/" + .properties.providerID + "/" + .properties.category + "/" + .properties.service | gsub("[^A-Za-z0-9@_.:\\\\/=-]"; "-")
          title: .properties.provider + "/" + .properties.service
          properties:
            provider: .properties.provider
            accountID: .properties.accountID
            invoiceEntityID: .properties.invoiceEntityID
            startDate: .window.start
            endDate: .window.end
            listCost: .listCost.cost
            listCostPercent: .listCost.kubernetesPercent
            netCost: .netCost.cost
            netCostPercent: .netCost.kubernetesPercent
            amortizedNetCost: .amortizedNetCost.cost
            amortizedNetCostPercent: .amortizedNetCost.kubernetesPercent
            invoicedCost: .invoicedCost.cost
            invoicedCostPercent: .invoicedCost.kubernetesPercent

Mapping config example for v2 kubecostCloudAllocation kind

createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: cloud
    selector:
      query: "true"
      window: 7d
      aggregate: service
      accumulate: false
      filter: 'labels:"app:internal-service","app:service-2"+service:"notification","account","functions"' # replaces all `filter*` properties needed in v1
    port:
      entity:
        mappings:
          blueprint: '"kubecostCloudAllocation"'
          identifier: .properties.provider + "/" + .properties.providerID + "/" + .properties.category + "/" + .properties.service | gsub("[^A-Za-z0-9@_.:\\\\/=-]"; "-")
          title: .properties.provider + "/" + .properties.service
          properties:
            provider: .properties.provider
            accountID: .properties.accountID
            invoiceEntityID: .properties.invoiceEntityID
            startDate: .window.start
            endDate: .window.end
            listCost: .listCost.cost
            listCostPercent: .listCost.kubernetesPercent
            netCost: .netCost.cost
            netCostPercent: .netCost.kubernetesPercent
            amortizedNetCost: .amortizedNetCost.cost
            amortizedNetCostPercent: .amortizedNetCost.kubernetesPercent
            invoicedCost: .invoicedCost.cost
            invoicedCostPercent: .invoicedCost.kubernetesPercent