Add obsreport to receiverhelper scrapers #1961

james-bebbington · 2020-10-15T02:32:38Z

Add obsreport to receiverhelper scraper library

Added consumererror.PartialScrapeError for denoting that a scrape was partially successful and recording the number of metrics that failed to be scraped
Add receiverhelper.CombineScrapeErrors to support combining partial scrape errors
Updated obsreport with Trace & Metric observability support for Scrapers. Obsreport records the number of failed scrapes based on the partial scrape error count if relevant
Call obsreport in the receiverhelper scraper library 🎉

Blocks #1949

Sample Output: (see #1949)

Note in these examples we see quite a lot of errors scraping process data due to not running the Collector in "Administrative mode" on Windows:

Traces (from http://localhost:55679/debug/tracez?zspanname=scraper%2fprocess%2fMetricsScraped&ztype=2&zsubtype=0):

Metrics (from http://localhost:8888/metrics):

codecov · 2020-10-15T02:41:16Z

Codecov Report

Merging #1961 into master will decrease coverage by 0.02%.
The diff coverage is 97.27%.

@@            Coverage Diff             @@
##           master    #1961      +/-   ##
==========================================
- Coverage   91.88%   91.86%   -0.03%     
==========================================
  Files         276      279       +3     
  Lines       16495    16587      +92     
==========================================
+ Hits        15157    15238      +81     
- Misses        922      929       +7     
- Partials      416      420       +4

Impacted Files	Coverage Δ
consumer/consumererror/partialscrapeerror.go	`71.42% <71.42%> (ø)`
receiver/receiverhelper/scrapercontroller.go	`94.44% <95.00%> (-5.56%)`	⬇️
obsreport/obsreport.go	`94.66% <100.00%> (+0.38%)`	⬆️
obsreport/obsreport_scraper.go	`100.00% <100.00%> (ø)`
obsreport/obsreporttest/obsreporttest.go	`95.23% <100.00%> (+0.89%)`	⬆️
receiver/receiverhelper/errors.go	`100.00% <100.00%> (ø)`
receiver/receiverhelper/scraper.go	`100.00% <100.00%> (ø)`
translator/internaldata/resource_to_oc.go	`89.04% <0.00%> (-2.74%)`	⬇️
receiver/otlpreceiver/otlp.go	`89.74% <0.00%> (-2.57%)`	⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14618bd...539afad. Read the comment docs.

obsreport/obsreport_scraper.go

component/componenterror/errors.go

obsreport/obsreport_scraper.go

dashpole · 2020-10-16T16:37:13Z

obsreport/obsreport_scraper.go

+		}
+	}
+
+	span := trace.FromContext(scraperCtx)


Is this span guaranteed to always be the one created by StartMetricsScrapeOp? Should we explicitly return the span from StartMetricsScrapeOp, and accept it as an argument in EndMetricsScrapeOp instead?

Or alternatively, make StartMetricsScrapeOp return an additional struct which contains the span and has an End() or RecordResult() function.

func StartMetricsScrapeOp(...) (context.Context, *MetricsScrapeOp) { ... ctx, span := trace.StartSpan(scraperCtx, spanName) return ctx, &MetricsScrapeOp{ctx: ctx, span: span} } // or you could instead make this unexported, and add an interface. type MetricsScrapeOp struct { ctx context.Context span trace.Span } func (m *MetricsScrapeOp) End(numScrapedMetrics int, err error) { // use m.ctx and m.span }

That's a very good point. I like your idea of returning a struct with an End() function.

I don't want to change this here though as all the other obsreport_*.go files follow this same pattern. Probably worth doing this in a followup PR though.

consumer/consumererror/partialscrapeerror.go

component/componenterror/errors.go

jrcamp · 2020-10-21T17:35:39Z

Overall looks good, thanks for adding this!

jrcamp · 2020-10-21T17:48:27Z

obsreport/obsreport_scraper.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package obsreport


Should this go in obsreport (or must it for technical reasons?) or could the scraper stuff all be kept together?

It's much more convenient adding this here as can make use of many of the helper functions for testing, etc. I think it is also more consistent to put this here? (even thought Scrapers aren't a first class component)

bogdandrutu · 2020-10-28T20:02:07Z

@james-bebbington any updates on this?

james-bebbington · 2020-10-29T03:27:05Z

@james-bebbington any updates on this?

Sorry just got back to this now and addressed the feedback.

I also updated this so that for the generated traces & metrics, scrapers are scoped to the receiver they belong to, and the scraper name is optional, i.e. will not be included in traces & metrics if set to an empty string - this is useful in the case of a simple receiver with only one scraper.

james-bebbington · 2020-10-29T06:35:59Z

This PR now breaks contrib-test because it adds a Name() function to the Scraper interface (needed to populate obsreport info). It will be straightforward to fix this when updating Contrib.

jrcamp · 2020-10-29T15:18:14Z

consumer/consumererror/partialscrapeerror.go

+
+// PartialScrapeError can be used to signalize that a subset of metrics were failed
+// to be scraped
+type PartialScrapeError struct {


I could be wrong but it feels like this and obsreport stuff about scraping would ideally all be in the same package (or subpackages) instead of spread across core. @bogdandrutu any thoughts?

Yea I'm not sure about this one.

I felt the obsreport stuff should stay together, and since that package needs to reference PartialScrapeError to compute metric stats, I needed to put this error in a separate package as well. consumererror seems to be the most appropriate place.

let's leave it there for the moment. @james-bebbington you promised me an issue to discuss making Scraper a top level component.

Did I? 😅

After applying this library to the hostmetrics receiver, my thoughts were that we could make this library even more usable if we:

Add ScraperFactory interfaces

Add code to simplify creation of scrapers via factories to receiver/receiverhelper

At that point, the Scraper interfaces will have many similarities in structure with other components, and it may arguably make sense to:

Move interfaces from receiver/receiverhelper/scraper.go into component/scraper.go

I don't think that scrapers should be a "top level component" though in the sense that they should be configurable at the top level like receivers / processors / exporters. Here's a doc Jay and I were using a while back to discuss the idea of making Scraper a top level component which eventually led us to do none of the things in that doc and implement this as a library in receiverhelper instead: https://docs.google.com/document/d/15WKpYvw23rQ6WnBhoE2wdk4szfjlXEXg8WYYvBcKshA/edit

Please move this to an issue so we don't forget about this

* Add make tasks to update examples versions and run all examples to test the collector can at least start * Extend antoine's changes to make it more compatible with linux development box + split out examples portion of `Makefile` to `examples/` directory Co-authored-by: hughesjj <james@subject17.net>

james-bebbington requested review from bogdandrutu, dmitryax, nilebox, owais, pjanotti and tigrannajaryan as code owners October 15, 2020 02:32

james-bebbington mentioned this pull request Oct 15, 2020

Receiverhelper obsreport #1956

Closed

james-bebbington force-pushed the receiverhelper-obsreport-clean branch from af4e5f9 to 77f7248 Compare October 15, 2020 02:48

bogdandrutu closed this Oct 15, 2020

bogdandrutu reopened this Oct 15, 2020

james-bebbington closed this Oct 15, 2020

james-bebbington reopened this Oct 15, 2020

james-bebbington force-pushed the receiverhelper-obsreport-clean branch from 77f7248 to da21dde Compare October 15, 2020 21:26

james-bebbington mentioned this pull request Oct 15, 2020

Update host metrics receiver to use scraper controller #1949

Merged

tigrannajaryan assigned jrcamp Oct 16, 2020

dashpole reviewed Oct 16, 2020

View reviewed changes

jrcamp reviewed Oct 21, 2020

View reviewed changes

component/componenterror/errors.go Outdated Show resolved Hide resolved

jrcamp reviewed Oct 21, 2020

View reviewed changes

james-bebbington force-pushed the receiverhelper-obsreport-clean branch from da21dde to e45fccf Compare October 28, 2020 07:23

james-bebbington requested a review from a team as a code owner October 28, 2020 07:23

james-bebbington force-pushed the receiverhelper-obsreport-clean branch 2 times, most recently from 0895d5e to 7d433eb Compare October 29, 2020 03:23

james-bebbington force-pushed the receiverhelper-obsreport-clean branch 5 times, most recently from d9d1caf to 7b3f624 Compare October 29, 2020 09:48

Add obsreport to receiverhelper scrapers

539afad

james-bebbington force-pushed the receiverhelper-obsreport-clean branch from 7b3f624 to 539afad Compare October 29, 2020 10:46

jrcamp reviewed Oct 29, 2020

View reviewed changes

james-bebbington mentioned this pull request Oct 29, 2020

2/3 Add Windows Performance Counters Receiver skeleton implementation open-telemetry/opentelemetry-collector-contrib#1211

Merged

bogdandrutu approved these changes Oct 29, 2020

View reviewed changes

bogdandrutu merged commit 07d7a1f into open-telemetry:master Oct 29, 2020

Troels51 pushed a commit to Troels51/opentelemetry-collector that referenced this pull request Jul 5, 2024

Fixed copyright for every file under sdk. (open-telemetry#1961)

6925fc4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add obsreport to receiverhelper scrapers #1961

Add obsreport to receiverhelper scrapers #1961

james-bebbington commented Oct 15, 2020 •

edited

Loading

codecov bot commented Oct 15, 2020 •

edited

Loading

dashpole Oct 16, 2020

james-bebbington Oct 29, 2020 •

edited

Loading

jrcamp commented Oct 21, 2020

jrcamp Oct 21, 2020

james-bebbington Oct 29, 2020 •

edited

Loading

bogdandrutu commented Oct 28, 2020

james-bebbington commented Oct 29, 2020 •

edited

Loading

james-bebbington commented Oct 29, 2020 •

edited

Loading

jrcamp Oct 29, 2020

james-bebbington Oct 29, 2020

bogdandrutu Oct 29, 2020

james-bebbington Oct 29, 2020 •

edited

Loading

bogdandrutu Oct 29, 2020

james-bebbington Oct 30, 2020

Add obsreport to receiverhelper scrapers #1961

Add obsreport to receiverhelper scrapers #1961

Conversation

james-bebbington commented Oct 15, 2020 • edited Loading

codecov bot commented Oct 15, 2020 • edited Loading

Codecov Report

dashpole Oct 16, 2020

Choose a reason for hiding this comment

james-bebbington Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

jrcamp commented Oct 21, 2020

jrcamp Oct 21, 2020

Choose a reason for hiding this comment

james-bebbington Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

bogdandrutu commented Oct 28, 2020

james-bebbington commented Oct 29, 2020 • edited Loading

james-bebbington commented Oct 29, 2020 • edited Loading

jrcamp Oct 29, 2020

Choose a reason for hiding this comment

james-bebbington Oct 29, 2020

Choose a reason for hiding this comment

bogdandrutu Oct 29, 2020

Choose a reason for hiding this comment

james-bebbington Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

bogdandrutu Oct 29, 2020

Choose a reason for hiding this comment

james-bebbington Oct 30, 2020

Choose a reason for hiding this comment

james-bebbington commented Oct 15, 2020 •

edited

Loading

codecov bot commented Oct 15, 2020 •

edited

Loading

james-bebbington Oct 29, 2020 •

edited

Loading

james-bebbington Oct 29, 2020 •

edited

Loading

james-bebbington commented Oct 29, 2020 •

edited

Loading

james-bebbington commented Oct 29, 2020 •

edited

Loading

james-bebbington Oct 29, 2020 •

edited

Loading