Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add CloudQuery integration (ECOINT-11) #2451

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open

feat: Add CloudQuery integration (ECOINT-11) #2451

wants to merge 28 commits into from
+766 −0

Conversation

erezrokah
Copy link

What does this PR do?

Add CloudQuery integration

Motivation

Get listed on the integrations page per https://docs.datadoghq.com/developers/integrations/?tab=integrations

Review checklist

  • PR has a meaningful title or PR has the no-changelog label attached
  • Feature or bugfix has tests
  • Git history is clean
  • If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo
  • If this PR includes a log pipeline, please add a description describing the remappers and processors.

Additional Notes

Anything else we should know when reviewing?

I followed the guide in https://docs.datadoghq.com/developers/integrations/?tab=integrations as best as I can. Feedback welcomed!

@erezrokah erezrokah requested review from a team as code owners July 29, 2024 10:06
@erezrokah erezrokah force-pushed the feat/cloudquery branch 2 times, most recently from adf28ee to f7469b0 Compare July 29, 2024 10:31
@drichards-87
Copy link
Contributor

Created Jira card for Docs Team review.

@drichards-87 drichards-87 added the editorial review Waiting on a more in-depth review from a docs team editor label Jul 29, 2024
estherk15
estherk15 reviewed Jul 31, 2024
View reviewed changes
Copy link
Contributor

@estherk15 estherk15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some suggestions for documentation, let me know if you have any questions or would like a re-review!

cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/README.md Outdated Show resolved Hide resolved
cloudquery/assets/dashboards/cloudquery_syncs_insights.json Outdated Show resolved Hide resolved
erezrokah and others added 17 commits July 31, 2024 18:45
@erezrokah @estherk15
Update cloudquery/README.md
d55b12c 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
85dbe39 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
f437ee7 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
1409aff 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
0b8b86e 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
9d6091a 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
d75e399 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
f5dce6f 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
b1bee1d 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
b76dc88 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
221794b 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
c143193 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
fce9366 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/README.md
3f38830 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Apply suggestions from code review
e174a52 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah @estherk15
Update cloudquery/assets/dashboards/cloudquery_syncs_insights.json
939fe20 
Co-authored-by: Esther Kim <esthercodes2019@gmail.com>
@erezrokah
fix: Links numbering
087f84a
@erezrokah erezrokah requested a review from estherk15 July 31, 2024 15:56
@erezrokah
Copy link
Author

Thanks for the review @estherk15, great suggestions 🚀 I also followed up with 087f84a and 06a21ee to fix the links and lists numbering (as the lists numbering is not preserved after the code blocks).

@erezrokah
Copy link
Author

Thanks @estherk15, per a meeting with the team in charge of partner integrations, we'll submit this as a community integration until there's official OTEL support

@erezrokah erezrokah closed this Aug 5, 2024
@erezrokah erezrokah reopened this Aug 6, 2024
@erezrokah
Copy link
Author

erezrokah commented Aug 6, 2024 •
edited
Loading

we'll submit this as a community integration until there's official OTEL support

OK so I guess I got confused by something, since the instructions in https://docs.datadoghq.com/agent/guide/use-community-integrations/?tab=agentv721v621 say to open a PR to this repo.

Can we accept this PR as a community integration?

Please note this integration does not require any installation on the agent side

@erezrokah erezrokah requested a review from a team as a code owner August 6, 2024 17:56
@erezrokah
Copy link
Author

Moving to draft until I fix the log pipelines CI

@erezrokah erezrokah marked this pull request as draft August 7, 2024 07:39
@erezrokah erezrokah mentioned this pull request Aug 7, 2024
Merged
@erezrokah erezrokah marked this pull request as ready for review August 7, 2024 08:19
@erezrokah
Copy link
Author

Not sure how to fix these errors:
image

This is how I configured the filter in the Datadog UI for AWS:
image

We send a service field with name and version properties in the logs so we should be able to filter by those no?

kodiakhq bot pushed a commit to cloudquery/cloudquery that referenced this pull request Aug 7, 2024
@erezrokah
feat: Add logs to Datadog dashboard (#18867)
54c4be8 
#### Summary

This adds a new section to our Datadog dashboard with information on throttled APIs:
![image](https://github.com/user-attachments/assets/fda9fcfe-7d3c-4e42-a134-95890e64243a)

Clicking on a column will users open the logs in the relevant location:
![image](https://github.com/user-attachments/assets/e9b931b0-9074-467f-ad4a-b9ec4cd929f5)
![image](https://github.com/user-attachments/assets/06d2c975-df01-4a99-8446-317eb1b56c5d)

I've also added the the same change to DataDog/integrations-extras#2451.

DataDog/integrations-extras#2451 also adds custom log pipelines to extract the retry attempts into attributes instead of relying on the `retrying request` message.
However if we only distribute the dashboard (like we do on our website) we can't ship the pipelines with the dashboard (we need the integration for that).
@estherk15 estherk15 added docs/approved and removed editorial review Waiting on a more in-depth review from a docs team editor labels Aug 7, 2024
@erezrokah
Copy link
Author

erezrokah commented Aug 14, 2024 •
edited
Loading

OK, removed the log pipelines in the initial version, I'll add those later

@erezrokah
Copy link
Author

14209ec marks this as a community integration

@erezrokah erezrokah mentioned this pull request Aug 21, 2024
Merged
@mgashaj mgashaj changed the title feat: Add CloudQuery integration feat: Add CloudQuery integration (ECOINT-11) Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@estherk15 estherk15 estherk15 approved these changes

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
docs/approved
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@erezrokah @drichards-87 @estherk15 @emarsha94