[exporter/datasetexporter]: Initial Commit #20733

martin-majlis-s1 · 2023-04-06T11:04:57Z

Description: Adding exporter DataSetExporter - initial commit

Link to tracking Issue: #20660

Testing: This is initial commit without any implementation. It contains only the structure of the exporter.

Documentation: Initial README.md describing configuration options.

linux-foundation-easycla · 2023-04-06T11:05:01Z

The committers listed above are authorized under a signed CLA.

✅ login: martin-majlis-s1 / name: Martin Majlis (015ca51, 0368966, 35fa0a3, 7e59db9, 4d5ae44, 0d69b04, 3d67349, 5b2277e)

martin-majlis-s1 · 2023-04-06T11:37:09Z

The committers listed above are authorized under a signed CLA.

✅ login: martin-majlis-s1 / name: Martin Majlis (015ca51, 0368966, 35fa0a3, 7e59db9, 4d5ae44, 0d69b04, 3d67349, 5b2277e, 565dd48)

martin-majlis-s1 · 2023-04-06T11:40:34Z

@atoulme is the sponsor of the linked issue.

exporter/elasticsearchexporter/config.go

runforesight · 2023-04-06T20:39:10Z

Foresight Summary

Major Impacts

build-and-test duration(25 minutes 7 seconds) has decreased 21 minutes 29 seconds compared to main branch avg(46 minutes 36 seconds).

View More Details

⭕ build-and-test-windows workflow has finished in 6 seconds (30 minutes 28 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
windows-unittest-matrix	- 🔗	N/A	See Details
windows-unittest	- 🔗	N/A	See Details

✅ telemetrygen workflow has finished in 1 minute 2 seconds and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
build-dev	- 🔗	N/A	See Details
publish-latest	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details

✅ check-links workflow has finished in 1 minute 43 seconds (⚠️ 47 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
changed files	- 🔗	N/A	See Details
check-links	- 🔗	N/A	See Details

✅ changelog workflow has finished in 2 minutes and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
changelog	- 🔗	N/A	See Details

✅ prometheus-compliance-tests workflow has finished in 10 minutes 26 seconds (⚠️ 4 minutes 5 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
prometheus-compliance-tests	- 🔗	N/A	See Details

✅ e2e-tests workflow has finished in 16 minutes 21 seconds (⚠️ 2 minutes 17 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
kubernetes-test (v1.26.0)	- 🔗	N/A	See Details
kubernetes-test (v1.25.3)	- 🔗	N/A	See Details
kubernetes-test (v1.24.7)	- 🔗	N/A	See Details
kubernetes-test (v1.23.13)	- 🔗	N/A	See Details

✅ load-tests workflow has finished in 19 minutes 24 seconds (⚠️ 8 minutes 53 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

Job	Failed Steps	Tests
setup-environment	- 🔗	N/A	See Details
loadtest (TestIdleMode)	- 🔗	N/A	See Details
loadtest (TestBallastMemory\|TestLog10kDPS)	- 🔗	N/A	See Details
loadtest (TestMetric10kDPS\|TestMetricsFromFile)	- 🔗	N/A	See Details
loadtest (TestMetricResourceProcessor\|TestTrace10kSPS)	- 🔗	N/A	See Details
loadtest (TestTraceNoBackend10kSPS\|TestTrace1kSPSWithAttrs)	- 🔗	N/A	See Details
loadtest (TestTraceBallast1kSPSWithAttrs\|TestTraceBallast1kSPSAddAttrs)	- 🔗	N/A	See Details
loadtest (TestTraceAttributesProcessor)	- 🔗	N/A	See Details

❌ build-and-test workflow has finished in 25 minutes 7 seconds (21 minutes 29 seconds less than `main` branch avg.) and finished at 13th Apr, 2023. 4 jobs failed.

Job	Failed Steps	Tests
setup-environment	- 🔗	N/A	See Details
govulncheck	- 🔗	N/A	See Details
check-collector-module-version	- 🔗	N/A	See Details
check-codeowners	- 🔗	N/A	See Details
build-examples	- 🔗	N/A	See Details
checks	Gen genotelcontribcol 🔗	N/A	See Details
correctness-metrics	- 🔗	N/A	See Details
correctness-traces	- 🔗	N/A	See Details
integration-tests	- 🔗	N/A	See Details
lint-matrix (receiver-0)	- 🔗	N/A	See Details
lint-matrix (receiver-1)	- 🔗	N/A	See Details
lint-matrix (processor)	- 🔗	N/A	See Details
lint-matrix (exporter)	- 🔗	N/A	See Details
lint-matrix (extension)	- 🔗	N/A	See Details
lint-matrix (connector)	- 🔗	N/A	See Details
lint-matrix (internal)	- 🔗	N/A	See Details
lint-matrix (other)	- 🔗	N/A	See Details
unittest-matrix (1.20, receiver-0)	- 🔗	N/A	See Details
unittest-matrix (1.20, receiver-1)	- 🔗	N/A	See Details
unittest-matrix (1.20, processor)	- 🔗	N/A	See Details
unittest-matrix (1.20, exporter)	- 🔗	N/A	See Details
unittest-matrix (1.20, extension)	- 🔗	N/A	See Details
unittest-matrix (1.20, connector)	- 🔗	N/A	See Details
unittest-matrix (1.20, internal)	- 🔗	N/A	See Details
unittest-matrix (1.20, other)	Run Unit Tests 🔗	N/A	See Details
unittest-matrix (1.19, receiver-0)	- 🔗	N/A	See Details
unittest-matrix (1.19, receiver-1)	- 🔗	N/A	See Details
unittest-matrix (1.19, processor)	- 🔗	N/A	See Details
unittest-matrix (1.19, exporter)	- 🔗	N/A	See Details
unittest-matrix (1.19, extension)	- 🔗	N/A	See Details
unittest-matrix (1.19, connector)	- 🔗	N/A	See Details
unittest-matrix (1.19, internal)	- 🔗	N/A	See Details
unittest-matrix (1.19, other)	- 🔗	N/A	See Details
unittest (1.20)	Interpret result 🔗	N/A	See Details
unittest (1.19)	Interpret result 🔗	N/A	See Details
lint	- 🔗	N/A	See Details
build-package	- 🔗	N/A	See Details
cross-compile	- 🔗	N/A	See Details
windows-msi	- 🔗	N/A	See Details
publish-check	- 🔗	N/A	See Details
publish-stable	- 🔗	N/A	See Details
publish-dev	- 🔗	N/A	See Details
rotate-milestone	- 🔗	N/A	See Details

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}

atoulme · 2023-04-06T20:41:12Z

This is the right scope of change, thank you! We will need to deal with a few administrative changes to the repo:

add a changelog entry (see .chloggen/TEMPLATE)
run make gendependabot
run make genotelcontribcol
run make generate-gh-issue-templates

martin-majlis-s1 · 2023-04-10T15:52:36Z

@atoulme : I have reverted those changes in elastic search, since they are not related to dataset. It was automatically changed during refactoring and I have missed it.

martin-majlis-s1 · 2023-04-10T16:13:24Z

@atoulme : I have run those commands - and two of them looks problematic to me:

✅ Validate Changelog

make chlog-validate

cd .../opentelemetry-collector-contrib/internal/tools && go build -o .../opentelemetry-collector-contrib/.tools/chloggen -trimpath go.opentelemetry.io/build-tools/chloggen
.../opentelemetry-collector-contrib/.tools/chloggen validate
PASS: all files in .../opentelemetry-collector-contrib/.chloggen/ are valid

❓ Run Dependabot

make gendependabot

Recreating .github/dependabot.yml file
Add entry for "/" github-actions
Add entry for "/" docker
Add entry for "/" gomod
Add entry for "/cmd/configschema"
Add entry for "/cmd/mdatagen"
...
Add entry for "/receiver/webhookeventreceiver"
Add entry for "/receiver/windowseventlogreceiver"
Add entry for "/receiver/windowsperfcountersreceiver"
The following modules are not included in the dependabot file because it has a limit of 220 entries:
  - /receiver/zipkinreceiver
  - /receiver/zookeeperreceiver
  - /testbed
  - /testbed/mockdatareceivers/mockawsxrayreceiver
  - /testbed/mockdatasenders/mockdatadogagentexporter

2 existing receivers were removed and more of them in the future. This is known issue #19410

✅ Gen Contrib

make genotelcontribcol

cd .../opentelemetry-collector-contrib/internal/tools && go build -o .../opentelemetry-collector-contrib/.tools/builder -trimpath go.opentelemetry.io/collector/cmd/builder
.../opentelemetry-collector-contrib/.tools/builder --skip-compilation --config cmd/otelcontribcol/builder-config.yaml --output-path cmd/otelcontribcol
Flag --output-path has been deprecated, use config distribution::output_path
2023-04-10T18:04:06.219+0200    INFO    internal/command.go:126 OpenTelemetry Collector Builder {"version": "dev", "date": "unknown"}
2023-04-10T18:04:06.225+0200    INFO    internal/command.go:159 Using config file       {"path": "cmd/otelcontribcol/builder-config.yaml"}
2023-04-10T18:04:06.225+0200    INFO    builder/config.go:116   Using go        {"go-executable": "/opt/homebrew/bin/go"}
2023-04-10T18:04:06.233+0200    INFO    builder/main.go:76      Sources created {"path": "cmd/otelcontribcol"}
Error: failed to update go.mod: exit status 1. Output:
go: downloading github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datasetexporter v0.75.0
github.com/open-telemetry/opentelemetry-collector-contrib/cmd/otelcontribcol imports
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datasetexporter: reading github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datasetexporter/go.mod at revision exporter/datasetexporter/v0.75.0: unknown revision exporter/datasetexporter/v0.75.0

I have manually added replace line - 3d67349

✅ Gen GH Issue

make generate-gh-issue-templates

for FILE in bug_report feature_request other; do \
                YAML_FILE=".github/ISSUE_TEMPLATE/${FILE}.yaml"; \
                TMP_FILE=".github/ISSUE_TEMPLATE/${FILE}.yaml.tmp"; \
                cat "${YAML_FILE}" > "${TMP_FILE}"; \
                FILE="${TMP_FILE}" ./.github/workflows/scripts/add-component-options.sh > "${YAML_FILE}"; \
                rm "${TMP_FILE}"; \
        done

martin-majlis-s1 · 2023-04-12T09:24:26Z

Follow up PR is #20870

.github/CODEOWNERS

martin-majlis-s1 · 2023-05-03T18:37:46Z

I have again resolved all the merge conflicts and rerun following commands:

make gendependabot && \
make chlog-validate && \
make genotelcontribcol && \
make generate-gh-issue-templates && \
make generate && \
(cd exporter/datasetexporter; make) && \
(cd exporter/datasetexporter; make test) && \
make -j2 golint GROUP=other && \
.github/workflows/scripts/check-collector-module-version.sh && \
make -j2 goporto && \
make -j2 crosslink && \
make -j2 gotidy && \
make -j2 generate && \
 make multimod-verify && \
make for-all CMD="make tidy"

martin-majlis-s1 · 2023-05-03T18:43:34Z

It takes me 40 minutes to resolve all those conflicts:

martin-majlis-s1 · 2023-05-04T07:38:10Z

There is again merge conflict 🙈

atoulme · 2023-05-04T07:56:52Z

You're very close :)

martin-majlis-s1 · 2023-05-04T09:03:22Z

I have merged main and executed: make gendependabot to fix the merge conflict.

martin-majlis-s1 · 2023-05-04T09:06:46Z

@atoulme : I have feeling that I am very close for last 3 weeks. 🙈

enykeev · 2023-05-04T10:03:48Z

Can we enable auto-merge so that next time this PR is approved and the checks pass, it immediately get merged in?

exporter/datasetexporter/README.md

exporter/datasetexporter/config.go

martin-majlis-s1 · 2023-05-05T08:36:19Z

I have again executed these commands:

make gendependabot && \
make chlog-validate && \
make genotelcontribcol && \
make generate-gh-issue-templates && \
make generate && \
(cd exporter/datasetexporter; make) && \
(cd exporter/datasetexporter; make test) && \
make -j2 golint GROUP=other && \
.github/workflows/scripts/check-collector-module-version.sh && \
make -j2 goporto && \
make -j2 crosslink && \
make -j2 gotidy && \
make -j2 generate && \
 make multimod-verify && \
make for-all CMD="make tidy"

And all of them passes and no changes are made.

martin-majlis-s1 · 2023-05-05T10:04:14Z

There is problem in the step - cache go, but thanks to @mx-psi retries we have been able to get through.

I have created bug report to investigate, what is going on - #21506

martin-majlis-s1 · 2023-05-05T12:04:33Z

Thanks everyone for your time and feedback! I appreciate your patience and expertise, and I'm grateful for the opportunity to learn from you all. Looking forward to contributing more in the future.

DataSetExporter: Initial Commit

015ca51

martin-majlis-s1 requested review from a team and Aneurysm9 April 6, 2023 11:04

github-actions bot assigned evan-bradley Apr 6, 2023

github-actions bot added cmd/otelcontribcol otelcontribcol command exporter/elasticsearch labels Apr 6, 2023

github-actions bot requested a review from JaredTan95 April 6, 2023 11:05

martin-majlis-s1 mentioned this pull request Apr 6, 2023

New component: DataSet Exporter #20660

Closed

2 tasks

Cleanup go.mod

0368966

atoulme removed the exporter/elasticsearch label Apr 6, 2023

atoulme reviewed Apr 6, 2023

View reviewed changes

exporter/elasticsearchexporter/config.go Outdated Show resolved Hide resolved

atoulme added the Accepted Component New component has been sponsored label Apr 6, 2023

Reverse changes in elasticsearchexporter/config.go

35fa0a3

Run mentioned commands

7e59db9

martin-majlis-s1 added 4 commits April 10, 2023 18:35

Run make goporto

4d5ae44

Merge branch 'main' into issue-20660-datasetexporter-initial-commit

0d69b04

Add replace line manually.

3d67349

Add dummy comment to trigger EasyCLA check

5b2277e

martin-majlis-s1 requested a review from atoulme April 11, 2023 12:51

martin-majlis-s1 mentioned this pull request Apr 12, 2023

[exporer/datasetexporter]: Log & Traces - initial support #20870

Closed

Merge branch 'main' into issue-20660-datasetexporter-initial-commit

565dd48

atoulme reviewed Apr 12, 2023

View reviewed changes

.github/CODEOWNERS Show resolved Hide resolved

martin-majlis-s1 added 5 commits May 3, 2023 20:13

Restore go.mod from main

2639b3a

Fix license check

098d558

Fix version check

ef97bae

Fix check-collector-module-versionr

81db459

Rerun all the remaining commands

ae758ff

martin-majlis-s1 requested a review from atoulme May 3, 2023 18:38

Merge branch 'main' into issue-20660-datasetexporter-initial-commit

892cdf2

atoulme added the ready to merge Code review completed; ready to merge by maintainers label May 4, 2023

atoulme approved these changes May 4, 2023

View reviewed changes

TylerHelmuth reviewed May 4, 2023

View reviewed changes

exporter/datasetexporter/README.md Outdated Show resolved Hide resolved

exporter/datasetexporter/config.go Outdated Show resolved Hide resolved

martin-majlis-s1 added 3 commits May 5, 2023 10:03

Merge branch 'main' into issue-20660-datasetexporter-initial-commit

c5bad0e

Rename MaxDelayMs to maxDelayMs

f1268c2

Modify metadata.yaml => make generate => adjust code

24b9aa2

mx-psi merged commit 5f47f5b into open-telemetry:main May 5, 2023

github-actions bot added this to the next release milestone May 5, 2023

This was referenced May 5, 2023

[exporter/datasetexporter]: Add support for sending logs #21513

Closed

[exporter/datasetexporter]: Initial implementation of logs and traces #21812

Closed

[exporter/datasetexporter]: Initial implementation of logs and traces #21815

Merged

This was referenced May 22, 2023

[exporter/datasetexporter]: Mark as alpha #22154

Merged

[exporter/datasetexporter]: Enable in internal components #22700

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/datasetexporter]: Initial Commit #20733

[exporter/datasetexporter]: Initial Commit #20733

martin-majlis-s1 commented Apr 6, 2023 •

edited by atoulme

Loading

linux-foundation-easycla bot commented Apr 6, 2023 •

edited

Loading

martin-majlis-s1 commented Apr 6, 2023 •

edited by linux-foundation-easycla bot

Loading

martin-majlis-s1 commented Apr 6, 2023

runforesight bot commented Apr 6, 2023 •

edited

Loading

`build-and-test` duration(25 minutes 7 seconds) has decreased 21 minutes 29 seconds compared to main branch avg(46 minutes 36 seconds).

⭕ build-and-test-windows workflow has finished in 6 seconds (30 minutes 28 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

✅ telemetrygen workflow has finished in 1 minute 2 seconds and finished at 13th Apr, 2023.

✅ check-links workflow has finished in 1 minute 43 seconds (⚠️ 47 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ changelog workflow has finished in 2 minutes and finished at 13th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 10 minutes 26 seconds (⚠️ 4 minutes 5 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ e2e-tests workflow has finished in 16 minutes 21 seconds (⚠️ 2 minutes 17 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ load-tests workflow has finished in 19 minutes 24 seconds (⚠️ 8 minutes 53 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

❌ build-and-test workflow has finished in 25 minutes 7 seconds (21 minutes 29 seconds less than `main` branch avg.) and finished at 13th Apr, 2023. 4 jobs failed.

atoulme commented Apr 6, 2023 •

edited

Loading

martin-majlis-s1 commented Apr 10, 2023

martin-majlis-s1 commented Apr 10, 2023 •

edited

Loading

martin-majlis-s1 commented Apr 12, 2023

martin-majlis-s1 commented May 3, 2023

martin-majlis-s1 commented May 3, 2023

martin-majlis-s1 commented May 4, 2023

atoulme commented May 4, 2023

martin-majlis-s1 commented May 4, 2023

martin-majlis-s1 commented May 4, 2023

enykeev commented May 4, 2023

martin-majlis-s1 commented May 5, 2023

martin-majlis-s1 commented May 5, 2023

martin-majlis-s1 commented May 5, 2023

[exporter/datasetexporter]: Initial Commit #20733

[exporter/datasetexporter]: Initial Commit #20733

Conversation

martin-majlis-s1 commented Apr 6, 2023 • edited by atoulme Loading

linux-foundation-easycla bot commented Apr 6, 2023 • edited Loading

martin-majlis-s1 commented Apr 6, 2023 • edited by linux-foundation-easycla bot Loading

martin-majlis-s1 commented Apr 6, 2023

runforesight bot commented Apr 6, 2023 • edited Loading

Foresight Summary

build-and-test duration(25 minutes 7 seconds) has decreased 21 minutes 29 seconds compared to main branch avg(46 minutes 36 seconds).

⭕ build-and-test-windows workflow has finished in 6 seconds (30 minutes 28 seconds less than main branch avg.) and finished at 13th Apr, 2023.

✅ telemetrygen workflow has finished in 1 minute 2 seconds and finished at 13th Apr, 2023.

✅ check-links workflow has finished in 1 minute 43 seconds (⚠️ 47 seconds more than main branch avg.) and finished at 13th Apr, 2023.

✅ changelog workflow has finished in 2 minutes and finished at 13th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 10 minutes 26 seconds (⚠️ 4 minutes 5 seconds more than main branch avg.) and finished at 13th Apr, 2023.

✅ e2e-tests workflow has finished in 16 minutes 21 seconds (⚠️ 2 minutes 17 seconds more than main branch avg.) and finished at 13th Apr, 2023.

✅ load-tests workflow has finished in 19 minutes 24 seconds (⚠️ 8 minutes 53 seconds more than main branch avg.) and finished at 13th Apr, 2023.

❌ build-and-test workflow has finished in 25 minutes 7 seconds (21 minutes 29 seconds less than main branch avg.) and finished at 13th Apr, 2023. 4 jobs failed.

atoulme commented Apr 6, 2023 • edited Loading

martin-majlis-s1 commented Apr 10, 2023

martin-majlis-s1 commented Apr 10, 2023 • edited Loading

✅ Validate Changelog

❓ Run Dependabot

✅ Gen Contrib

✅ Gen GH Issue

martin-majlis-s1 commented Apr 12, 2023

martin-majlis-s1 commented May 3, 2023

martin-majlis-s1 commented May 3, 2023

martin-majlis-s1 commented May 4, 2023

atoulme commented May 4, 2023

martin-majlis-s1 commented May 4, 2023

martin-majlis-s1 commented May 4, 2023

enykeev commented May 4, 2023

martin-majlis-s1 commented May 5, 2023

martin-majlis-s1 commented May 5, 2023

martin-majlis-s1 commented May 5, 2023

martin-majlis-s1 commented Apr 6, 2023 •

edited by atoulme

Loading

linux-foundation-easycla bot commented Apr 6, 2023 •

edited

Loading

martin-majlis-s1 commented Apr 6, 2023 •

edited by linux-foundation-easycla bot

Loading

runforesight bot commented Apr 6, 2023 •

edited

Loading

`build-and-test` duration(25 minutes 7 seconds) has decreased 21 minutes 29 seconds compared to main branch avg(46 minutes 36 seconds).

⭕ build-and-test-windows workflow has finished in 6 seconds (30 minutes 28 seconds less than `main` branch avg.) and finished at 13th Apr, 2023.

✅ check-links workflow has finished in 1 minute 43 seconds (⚠️ 47 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ prometheus-compliance-tests workflow has finished in 10 minutes 26 seconds (⚠️ 4 minutes 5 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ e2e-tests workflow has finished in 16 minutes 21 seconds (⚠️ 2 minutes 17 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

✅ load-tests workflow has finished in 19 minutes 24 seconds (⚠️ 8 minutes 53 seconds more than `main` branch avg.) and finished at 13th Apr, 2023.

❌ build-and-test workflow has finished in 25 minutes 7 seconds (21 minutes 29 seconds less than `main` branch avg.) and finished at 13th Apr, 2023. 4 jobs failed.

atoulme commented Apr 6, 2023 •

edited

Loading

martin-majlis-s1 commented Apr 10, 2023 •

edited

Loading