Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Doppler integration (ECOINT-6) #2460

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

nmanoogian
Copy link

What does this PR do?

This PR creates the Doppler log collection integration. This is an API-based integration that uses OAuth to connect Doppler with Datadog. Once connected, Doppler will begin sending activity log data to Datadog.

Motivation

What inspired you to submit this pull request?

Review checklist

  • PR has a meaningful title or PR has the no-changelog label attached
  • Feature or bugfix has tests (Not Relevant)
  • Git history is clean
  • If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo
  • If this PR includes a log pipeline, please add a description describing the remappers and processors.

@neko-dd
Copy link
Contributor

neko-dd commented Aug 6, 2024

Created DOCS-8635 for docs review

Copy link

@michaelcretzman michaelcretzman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with some minor edits.

doppler/README.md Show resolved Hide resolved
doppler/README.md Show resolved Hide resolved
doppler/README.md Show resolved Hide resolved
@dd-dominic dd-dominic changed the title Add Doppler integration Add Doppler integration (ECOINT-6) Aug 23, 2024

## Overview

[Doppler][1] is a secrets manager designed with both security and developer productivity in mind.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overview should include:
1 What does your product do in 1-2 sentences
2 What will customers get from this integration (the value of the integration & why it's helpful)
3 What specific data your integration monitors and the value?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I've updated this section with more of those specifics.

This integration allows Activity Logs from [Doppler][1] to be streamed to Datadog, allowing you to monitor changes to your workplace.

## Setup

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add clear directions a user would take to install the integration? For example, go to the Doppler tile on the integrations page, click connect accounts, you will be redirected to doppler to log in, ..

This should walk through the full oauth flow a user will take. I recommend testing the oauth flow and creating the steps on the steps you take!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely, I re-tested the full oauth flow and provided more detail here.

Comment on lines 161 to 164
# The `result` field should be left blank to start. Once you submit your log asset files with
# your integration pull-request in a Datadog GitHub repository, Datadog's validations will
# run your raw logs against your pipeline and return the result. If the result output in the
# validation is accurate, take the output and add it to the `result` field in your test YAML file.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see here and follow steps above

],
"description": "Doppler API Client",
"id": "87fdb7f6-542c-11ef-abb5-da7ad0900002",
"scopes": ["api_keys_write", "agent_upgrade_write"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove the agent_upgrade write scope from your oauth client and re-publish the client

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Comment on lines 52 to 58
"Supported OS::Linux",
"Supported OS::Windows",
"Supported OS::macOS",
"Category::Security",
"Category::Developer Tools",
"Category::Log Collection",
"Offering::Integration"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is missing the submitted data type. Please add this https://docs.datadoghq.com/developers/integrations/check_references/#classifier-tags

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the logs datatype

@nmanoogian
Copy link
Author

Thanks all for the feedback!

A question for you: Do I need the files added in the "WIP: Add tests" commit? I re-ran ddev create and noticed that these came up.

I still need to let CI run and collect the test log results. I'll push again once that's done.

@nmanoogian
Copy link
Author

Another question: What should I provide for these manifest values if our integration doesn't produce metrics or provide service checks:

{
  "assets": {
    "integration": {
      "metrics": {
        "prefix": [
          "Missing data for required field."
        ],
        "metadata_path": [
          "Missing data for required field."
        ]
      },
      "service_checks": {
        "metadata_path": [
          "Missing data for required field."
        ]
      }
    }
  }
}

@nmanoogian
Copy link
Author

I've been able to resolve most of the validation errors with trial-and-error but I'm a little stumped on the outstanding issue with logs:

Facet path `type` conflicts with other integrations: crest_data_systems_fortigate, jfrog_platform, pan.firewall, vault. Please fix this conflict or change this facet definition to:
groups:
  - Event
name: Type
path: type
source: log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants