feat: Add CloudQuery integration (ECOINT-11)

.github/CODEOWNERS

@@ -37,6 +37,7 @@
 /census/                                 @sankalp04 support@getcensus.com @DataDog/ecosystems-review
 /cfssl/                                  @JeanFred
 /cloudnatix/                             @junm-cloudnatix @kenji-cloudnatix @somik-cloudnatix @rohit-cloudnatix
+/cloudquery/                             @cloudquery/cloudquery-framework
 /cloudsmith/                             @cloudsmith ccarey@cloudsmith.io @DataDog/ecosystems-review
 /cloudzero/                              @ben-dalton @mattyellen @egafford @alinaquinones
 /cockroachdb_dedicated/                  @DataDog/saas-integrations

cloudquery/CHANGELOG.md

@@ -0,0 +1,7 @@
+# CHANGELOG - CloudQuery
+
+## 1.0.0 / 2024-07-29
+
+***Added***:
+
+* Initial Release

cloudquery/README.md

@@ -0,0 +1,177 @@
+# CloudQuery
+
+![datadog-integration][1]
+
+## Overview
+
+[CloudQuery][2] is an [open-source][3], high-performance data integration framework built for developers, with support for a wide range of plugins.
+
+CloudQuery extracts, transforms, and loads configuration from cloud APIs to a variety of supported destinations such as databases, data lakes, or streaming platforms for further analysis.
+
+## Setup
+
+### Installation
+
+Install the Datadog Agent version >=6.48.0 or >=7.48.0, which supports ingesting OpenTelemetry traces, metrics, and logs from CloudQuery.
+Alternatively, you can use OpenTelemetry Collector and Datadog Exporter as described below.
+
+### Configuration
+
+CloudQuery supports [OpenTelemetry][5] traces, metrics and logs out of the box.
+There are multiple ways to configure OpenTelemetry with Datadog. We'll show only a subset of them here, and you can find more information [in the Datadog docs][6].
+
+- Using an OpenTelemetry collector
+
+To configure an OpenTelemetry collector with Datadog, you need to create a configuration file, for example `otel_collector_config.yaml` with the content below:
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      http:
+        endpoint: "0.0.0.0:4318"
+
+processors:
+  batch/datadog:
+    send_batch_max_size: 1000
+    send_batch_size: 100
+    timeout: 10s
+
+exporters:
+  datadog:
+    api:
+      site: ${env:DATADOG_SITE}
+      key: ${env:DATADOG_API_KEY}
+
+service:
+  pipelines:
+    metrics:
+      receivers: [otlp]
+      processors: [batch/datadog]
+      exporters: [datadog]
+    traces:
+      receivers: [otlp]
+      processors: [batch/datadog]
+      exporters: [datadog]
+    logs:
+      receivers: [otlp]
+      processors: [batch/datadog]
+      exporters: [datadog]
+```
+
+Then run the collector with the following command (replacing `DATADOG_SITE` and `DATADOG_API_KEY` with your own values):
+
+```bash
+docker run \
+    -p 4318:4318 \
+    -e DATADOG_SITE=$DATADOG_SITE \
+    -e DATADOG_API_KEY=$DATADOG_API_KEY \
+    --hostname $(hostname) \
+    -v $(pwd)/otel_collector_config.yaml:/etc/otelcol-contrib/config.yaml \
+    otel/opentelemetry-collector-contrib:0.104.0
+```
+
+> For additional ways to run the collector, please refer to the [official documentation][7].
+
+- Direct OTEL Ingestion by the Datadog Agent via a configuration file
+
+[Locate][8] your `datadog.yaml` file and add the following configuration:
+
+```yaml
+otlp_config:
+  receiver:
+    protocols:
+      http:
+        endpoint: 0.0.0.0:4318
+  logs:
+    enabled: true
+logs_enabled: true
+```
+
+[Restart][9] the Datadog agent for the change to take effect.
+
+- Direct OTEL ingestion by the Datadog Agent via environment variables
+
+Pass the `DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT` environment variable to the Datadog agent with a value of `0.0.0.0:4318`.
+If you're using Docker compose, you can find an example below:
+
+```yaml
+version: "3.0"
+services:
+  agent:
+    image: gcr.io/datadoghq/agent:7
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /proc/:/host/proc/:ro
+      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
+    environment:
+      DD_API_KEY: redacted
+      DD_SITE: "datadoghq.eu"
+      DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318"
+      DD_LOGS_ENABLED: "true"
+      DD_OTLP_CONFIG_LOGS_ENABLED: "true"
+    ports:
+      - "4318:4318"
+```
+
+[Restart][10] the Datadog agent for the change to take effect.
+
+> For additional ways to configure the Datadog agent, please refer to the [official documentation][11].
+
+Once you have the agent or collector ready, you can specify the endpoint in the source spec:
+
+```yaml
+kind: source
+spec:
+  name: "aws"
+  path: "cloudquery/aws"
+  registry: "cloudquery"
+  version: "VERSION_SOURCE_AWS"
+  tables: ["aws_s3_buckets"]
+  destinations: ["postgresql"]
+  otel_endpoint: "0.0.0.0:4318"
+  otel_endpoint_insecure: true
+  spec:
+```
+
+### Validation
+
+Run `cloudquery sync spec.yml`.
+Once ingestion starts you should be able to start seeing the traces in Datadog under APM->Traces->Explorer.
+You should see metrics and logs in the respective sections of the Datadog UI as well.
+
+## Data Collected
+
+### Metrics
+
+The CloudQuery does not include any metrics.
+
+### Service Checks
+
+The CloudQuery does not include any service checks.
+
+### Events
+
+The CloudQuery does not include any events.
+
+## Uninstallation
+
+If you use the OpenTelemetry collector, you can stop it by running `docker stop <container_id>`.
+IF you use the Datadog agent, remove the configuration or environment variables you added and [restart][10] the agent.
+Finally, delete the dashboard from your Datadog account.
+
+## Support
+
+Need help? [Contact us][4].
+
+[1]: https://raw.githubusercontent.com/DataDog/integrations-extras/master/cloudquery/images/cloudquery_logo_png_dark_background.png
+[2]: https://www.cloudquery.io/
+[3]: https://github.com/cloudquery/cloudquery
+[4]: https://www.cloudquery.io/pricing
+[5]: https://opentelemetry.io/
+[6]: https://docs.datadoghq.com/opentelemetry/
+[7]: https://docs.datadoghq.com/opentelemetry/collector_exporter/deployment#running-the-collector
+[8]: https://docs.datadoghq.com/agent/configuration/agent-configuration-files/
+[9]: https://docs.datadoghq.com/agent/configuration/agent-commands/#restart-the-agent
+[10]: https://docs.datadoghq.com/agent/configuration/agent-commands/#restart-the-agent
+[11]: https://docs.datadoghq.com/opentelemetry/interoperability/otlp_ingest_in_the_agent#enabling-otlp-ingestion-on-the-datadog-agent