Metricbeat collection of ES monitoring data is inconsistent for the license field

[chrisronline]

This was tested on the latest master of each product (ES and beats)

I have a CCS scenario setup with cluster A (prod) and cluster B (Monitoring). I have configured metricbeat to collect and ship monitoring data to cluster B. I have configured CCS from cluster A to cluster B.

This query:

POST Monitoring:.monitoring-es-7-2019.11.15/_search?filter_path=hits.hits._source.license
{
  "size": 1,
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

yields:

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "license" : {
            "status" : "active",
            "uid" : "ac93a382-2cb3-4e03-b14b-ac1664ce456c",
            "type" : "basic",
            "issue_date" : "2019-11-15T14:39:30.697Z",
            "issue_date_in_millis" : 1573828770697,
            "max_nodes" : 1000,
            "issued_to" : "MonitoringCluster",
            "issuer" : "elasticsearch",
            "start_date_in_millis" : -1
          }
        }
      }
    ]
  }
}

Note that there is no expiry_date_in_millis field.

Then, this query:

POST Monitoring:.monitoring-es-7-mb-2019.11.15/_search?filter_path=hits.hits._source.license
{
  "size": 1,
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

yields:

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "license" : {
            "max_nodes" : 1000,
            "issued_to" : "MonitoringCluster",
            "status" : "active",
            "issue_date_in_millis" : 1573828770697,
            "start_date_in_millis" : -1,
            "type" : "basic",
            "issuer" : "elasticsearch",
            "issue_date" : "2019-11-15T14:39:30.697Z",
            "expiry_date_in_millis" : 0,
            "cluster_needs_tls" : false,
            "id" : "ac93a382-2cb3-4e03-b14b-ac1664ce456c"
          }
        }
      }
    ]
  }
}

Note the addition of expiry_date_in_millis: 0

This is causing an issue in Kibana, as our check to see if a license can expire only checks for the presence of the expiry_date_in_millis field

cc @ycombinator

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[ycombinator]

Just tested and this is also happening on 7.x and 7.5 branches. But it's not happening on 7.4. This leads me to believe this regression was introduced by the change in #14378.

@chrisronline Given the impact of this change — Stack Monitoring UI incorrectly showing license as expired — I feel like this should be a blocker for 7.5.0, since it hasn't been released yet. WDYT?

[chrisronline]

@ycombinator I agree. With this bug, any user on a Basic license is not able to use the Stack Monitoring. If a user attempts to load the UI, they will be redirected to a page that shows: