[APM] Show services reporting only metrics in the APM UI - Reopen #92075

[martenls]

Issue #92075 which should have been fixed with #92378 is still not fixed in version 7.11.2 of Kibana.

The APM view shows the service when the time filter is set to the last hour.

The APM view does not show the service when the time filter is set to the last 30 minutes.

However there were metric events in the last 30 minutes.

Versions
Kibana: 7.11.2
APM Server: 7.11.2
Elasticsearch: 7.11.2

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[dgieselaar]

@martenls can you show more fields in the discover screenshot? Showing all fields for a couple of documents would be great. That should help us zoom in on possible causes.

[martenls]

Here a two documents as json. Is that enough?

{
  "_index": "apm-7.11.2-metric-000001",
  "_type": "_doc",
  "_id": "hZnNRngBdqqWqgeFNnFq",
  "_version": 1,
  "_score": null,
  "_source": {
    "jvm": {
      "memory": {
        "non_heap": {
          "committed": 70893568,
          "max": -1,
          "used": 67788328
        },
        "heap": {
          "committed": 536870912,
          "max": 8577351680,
          "used": 57012736
        }
      },
      "thread": {
        "count": 18
      },
      "gc": {
        "alloc": 748530200
      }
    },
    "observer": {
      "hostname": "apm-server",
      "id": "ddd4eaf4-74da-48c9-acc8-c2e0de5b659e",
      "ephemeral_id": "50c91607-a0db-4ea9-b03a-dc85a3100c77",
      "type": "apm-server",
      "version": "7.11.2",
      "version_major": 7
    },
    "process": {
      "pid": 22572,
      "title": "C:\\Program Files\\AdoptOpenJDK\\jdk-11.0.8.10-hotspot\\bin\\java.exe",
      "ppid": 1272
    },
    "agent": {
      "name": "java",
      "ephemeral_id": "16751094-fccd-4843-ae00-b6bb8fc0c5df",
      "version": "1.21.0"
    },
    "@timestamp": "2021-03-18T19:25:45.936Z",
    "system": {
      "process": {
        "memory": {
          "size": 737058816
        },
        "cpu": {
          "total": {
            "norm": {
              "pct": 0.00022786724634188562
            }
          }
        }
      },
      "memory": {
        "actual": {
          "free": 17287880704
        },
        "total": 34307977216
      },
      "cpu": {
        "total": {
          "norm": {
            "pct": 0.04321802015660148
          }
        }
      }
    },
    "ecs": {
      "version": "1.6.0"
    },
    "service": {
      "node": {
        "name": "Marten-PC"
      },
      "environment": "production",
      "name": "javatest",
      "runtime": {
        "name": "Java",
        "version": "11.0.8"
      },
      "language": {
        "name": "Java",
        "version": "11.0.8"
      }
    },
    "host": {
      "hostname": "Marten-PC",
      "os": {
        "platform": "Windows 10"
      },
      "ip": "***",
      "name": "Marten-PC",
      "architecture": "amd64"
    },
    "event": {
      "ingested": "2021-03-18T19:25:57.225820200Z"
    },
    "processor": {
      "name": "metric",
      "event": "metric"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-03-18T19:25:57.225Z"
    ],
    "@timestamp": [
      "2021-03-18T19:25:45.936Z"
    ]
  },
  "sort": [
    1616095545936
  ]
}

{
  "_index": "apm-7.11.2-metric-000001",
  "_type": "_doc",
  "_id": "h5nNRngBdqqWqgeFNnFq",
  "_version": 1,
  "_score": null,
  "_source": {
    "jvm": {
      "gc": {
        "count": 0,
        "time": 0
      }
    },
    "observer": {
      "hostname": "apm-server",
      "id": "ddd4eaf4-74da-48c9-acc8-c2e0de5b659e",
      "type": "apm-server",
      "ephemeral_id": "50c91607-a0db-4ea9-b03a-dc85a3100c77",
      "version": "7.11.2",
      "version_major": 7
    },
    "agent": {
      "name": "java",
      "ephemeral_id": "16751094-fccd-4843-ae00-b6bb8fc0c5df",
      "version": "1.21.0"
    },
    "process": {
      "pid": 22572,
      "title": "C:\\Program Files\\AdoptOpenJDK\\jdk-11.0.8.10-hotspot\\bin\\java.exe",
      "ppid": 1272
    },
    "@timestamp": "2021-03-18T19:25:45.936Z",
    "ecs": {
      "version": "1.6.0"
    },
    "service": {
      "node": {
        "name": "Marten-PC"
      },
      "environment": "production",
      "name": "javatest",
      "runtime": {
        "name": "Java",
        "version": "11.0.8"
      },
      "language": {
        "name": "Java",
        "version": "11.0.8"
      }
    },
    "host": {
      "hostname": "Marten-PC",
      "os": {
        "platform": "Windows 10"
      },
      "ip": "***",
      "name": "Marten-PC",
      "architecture": "amd64"
    },
    "event": {
      "ingested": "2021-03-18T19:25:57.226259425Z"
    },
    "processor": {
      "name": "metric",
      "event": "metric"
    },
    "labels": {
      "name": "G1 Old Generation"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-03-18T19:25:57.226Z"
    ],
    "@timestamp": [
      "2021-03-18T19:25:45.936Z"
    ]
  },
  "sort": [
    1616095545936
  ]
}

[martenls]

More on our specific issue:
We have one Java App which currently only publishes metric events. This service would not show up at all on the APM page.
After finding this issue we tried a Java Spring App which publishes, among others, transaction events and it showed up right away.
After we had shutdown the Java Spring App and then started our metric only app again and let it publish the events under the same service name we could analyze the metrics in the APM page but only if we set the timeframe large enough so that the old transaction events were also included.

[dgieselaar]

@martenls ah, I see the fix was backported to 7.12 only. I think that was a conscious decision, let me verify whether that was the case.

[dgieselaar]

@martenls I think backporting this to 7.11 was hairy, and because there is an upgrade path to fix this (to 7.12), and it being close to the release, we decided not to take the risk. Are you able to upgrade to 7.12?

[martenls]

Yes upgrading to 7.12 when its released should be no problem. We'll have to wait then... Thanks for the quick answer!

[dgieselaar]

Totally forgot that release is still a while out 😀 we are already a few weeks into the next cycle, my bad. And sure thing! I'll close this one then.