services not showing in services tab and otel-v1-apm-service-map is empty

[lgarvey]

I'm running collector -> dataprepper 1.0 -> AES 7.10

Looking at #45 I have reverted the data-prepper PR that changes the span schema and manually built a jar file. This has fixed the origiinal issue of seeing no traces. However, now I see no services. The otel-v1-apm-span-* indices contain data, but the otel-v1-apm-service-map is always empty.

I see nothing in the elasticsearch logs indicating mapping errors etc.

We're looking to migrate our org away from elastic-apm to opentelemetry/trace-analytics but need the ability to drill down based on service name.

The data-prepper shows that the service-map pipeline is processing events:

[service-map-pipeline-prepper-worker-3-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - service-map-pipeline Worker: Processing 1 records from buffer 
[entry-pipeline-prepper-worker-1-thread-1] INFO com.amazon.dataprepper.pipeline.ProcessWorker - entry-pipeline Worker: Processing 1 records from buffer 
[entry-pipeline-prepper-worker-1-thread-1] DEBUG com.amazon.dataprepper.pipeline.ProcessWorker - Pipeline Worker: Submitting 1 processed records to sinks

Data-prepper config:

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - otel_trace_raw_prepper:
  sink:
    - elasticsearch:
        hosts: ["${ELASTICSEARCH_HOST}" ]
        username: "${ELASTICSEARCH_USERNAME}"
        password: "${ELASTICSEARCH_PASSWORD}"
        trace_analytics_raw: true
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - service_map_stateful:
  sink:
    - elasticsearch:
        hosts: ["${ELASTICSEARCH_HOST}" ]
        username: "${ELASTICSEARCH_USERNAME}"
        password: "${ELASTICSEARCH_PASSWORD}"
        trace_analytics_service_map: true

Example span data:

{
    "name": "baz",
    "context": {
        "trace_id": "0x2bfe27e9786311bb5d417c957da798e5",
        "span_id": "0x48b3327412dfa8d5",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xf04395ec442bfcbb",
    "start_time": "2021-07-14T13:34:07.759899Z",
    "end_time": "2021-07-14T13:34:07.759978Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "service.name": "ocal-app",
        "space": "platform",
        "org": "local"
    }
}
{
    "name": "bar",
    "context": {
        "trace_id": "0x2bfe27e9786311bb5d417c957da798e5",
        "span_id": "0xf04395ec442bfcbb",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0xea98094123af9929",
    "start_time": "2021-07-14T13:34:07.759756Z",
    "end_time": "2021-07-14T13:34:07.760013Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "service.name": "local-app",
        "space": "platform",
        "org": "local"
    }
}
{
    "name": "foo",
    "context": {
        "trace_id": "0x2bfe27e9786311bb5d417c957da798e5",
        "span_id": "0xea98094123af9929",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2021-07-14T13:34:07.759644Z",
    "end_time": "2021-07-14T13:34:07.760028Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "service.name": "local-app",
        "space": "platform",
        "org": "local"
    }
}

[joshuali925]

Linking issue in data-prepper repo opendistro-for-elasticsearch/data-prepper#720