Telegraf config for parsing Juniper UDP / Protobuf (dial-out) data

[jryburn]

Please direct all support questsions to slack or the forums. Thank you.

I had a thread going over on the community site here.

I cannot seem to get the socket_listener plugin to properly decode the stream telemetry (dial-out) UDP protobuf data I am getting from a Juniper virtual switch. Here is a zipfile that contains the following:

• data.gpb - raw capture of the protobuf data coming in off the wire
• data.json - data coming converted to JSON using protoc command (protoc --decode TelemetryStream port.proto -I /usr/include/protos -I /usr/include/google/protobuf < /tmp/data.gpb > /tmp/data.json)

Archive.zip

Here is my telegraf.conf file:

# ===============================================================
[global_tags]

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  debug = true
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = true
# ===============================================================

[[inputs.file]]
  files = ["/tmp/data.gpb"]
  data_format = "xpath_protobuf"
  xpath_protobuf_file = "port.proto"
  xpath_protobuf_type = "TelemetryStream"
  xpath_print_document = true

  xpath_protobuf_import_paths = ["/usr/include/protos","/usr/include/google/protobuf"]

The resulting output from the xpath_print_document = true is:

<?xml version=\"1.0\"?>
<system_id>ST-Justin:10.55.60.125</system_id>
<sub_component_id>0</sub_component_id>
<sensor_name>ROUTER-INF:/junos/system/linecard/interface/:/junos/system/linecard/interface/:PFE</sensor_name>
<sequence_number>25715</sequence_number>
<timestamp>1719490700801</timestamp>
<version_major>1</version_major>
<version_minor>1</version_minor>
<enterprise></enterprise>
<component_id>0</component_id>

The proto files in the /usr/include/protos directory come from here. The proto files in the /usr/include/google/protobuf directory come from here.

[telegraf-tiger]

Hello! I recommend posting this question in our Community Slack or Community Forums, we have a lot of talented community members there who could help answer your question more quickly. You can also learn more about Telegraf by enrolling at InfluxDB University for free!

Heads up, this issue will be automatically closed after 7 days of inactivity. Thank you!

[srebhan]

Thanks @jryburn! Will take a look...

[srebhan]

@jryburn sorry for the late feedback! I tracked the issue down to the usage of "extensions" in the protocol buffer which is currently not handled by the underlying parser library. Want to let you know that I'm working on a solution but that might take a bit...

[srebhan]

@jryburn please test the binary in PR #15586, available as soon as CI finished the tests, and let me know if this fixes the issue. I've used

  xpath_print_document = true
  xpath_native_types = true
  xpath_protobuf_files = ["telemetry_top.proto", "port.proto"]
  xpath_protobuf_type = "TelemetryStream"
  xpath_protobuf_import_paths = [".", "./testcases/protobuf_issue_15571"]

  [[inputs.file.xpath]]
    metric_name = "'test'"
    field_selection = "* | //if_name"
    field_name_expansion = true
    timestamp = "//timestamp"
    timestamp_format = "unix_ms"

during tests. It is important to specify all used extensions in xpath_protobuf_files as otherwise the data is not resolved...

[jryburn]

This is working great. I will play around and grab the metrics I need from here. Thanks!

[srebhan]

@jryburn thanks for testing! Please keep the issue open until we merged the PR, which will close the issue automatically. This helps us to keep track of what is open and also helps others to find the issue more easily in cause they encounter similar problems.