Telemetry: local and global

[jhellerstein]

Code:

• Associate hydroflow+ functions with hydroflow syntax for flamegraphs: Pass Hydroflow+ call span info down to hydroflow surface syntax in order to trace flamegraphs #1268
• Dataflow edge cardinality counters #1269
• Measure networking overhead #1270
  • What kind of throughput are we getting
  • (source or sink op id, tick, bytes, messages), time elapsed per tick

Tasks:

• Figure out how perf flamegraphs work with idle time
• Measure overhead from profiling

---

• Need a way to get fine-grained telemetry and control over its aggregation/publishing within a transducer
• Need ways to collect/aggregate telemetry across transducers
  • write to S3/etc
  • a library like Anna KVS with telemetry keys, sketch lattices, etc.
• Telemetry standards and third-party tools
• Beware of self-measurement abberations

While designing, think about what would make this a differentiating and general-purpose value-add of Hydro. Compare to Dapper etc for collection/aggregation.

[MingweiSamuel]

Tiemo:

• How can I know when a batch of data is fully processed? (Diagnostics: ops need some ergonomic way of generating flow of debug info (e.g. flag when all work is done) #834)
  • Right now, have to implement a tombstone to signal EOS
  • Have design patterns for termination detection
  • (Have general-purpose termination detection as a std library? Hackathon: Examples of common distributed design patterns #346)

[MingweiSamuel]

#834 related

[rohitkulshreshtha]

Next steps: Check with David re: metrics.

[MingweiSamuel]

Closing, just about done, see small issues above