diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 72d9c8a6d..339104024 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -37,6 +37,7 @@ /census/ @sankalp04 support@getcensus.com @DataDog/ecosystems-review /cfssl/ @JeanFred /cloudnatix/ @junm-cloudnatix @kenji-cloudnatix @somik-cloudnatix @rohit-cloudnatix +/cloudquery/ @cloudquery/cloudquery-framework /cloudsmith/ @cloudsmith ccarey@cloudsmith.io @DataDog/ecosystems-review /cloudzero/ @ben-dalton @mattyellen @egafford @alinaquinones /cockroachdb_dedicated/ @DataDog/saas-integrations diff --git a/cloudquery/CHANGELOG.md b/cloudquery/CHANGELOG.md new file mode 100644 index 000000000..e88300e92 --- /dev/null +++ b/cloudquery/CHANGELOG.md @@ -0,0 +1,7 @@ +# CHANGELOG - CloudQuery + +## 1.0.0 / 2024-07-29 + +***Added***: + +* Initial Release diff --git a/cloudquery/README.md b/cloudquery/README.md new file mode 100644 index 000000000..2e37e8cfb --- /dev/null +++ b/cloudquery/README.md @@ -0,0 +1,219 @@ +# CloudQuery + +![datadog-integration][1] + +## Overview + +[CloudQuery][2] is an [open-source][3], high-performance data integration framework built for developers, with support for a wide range of plugins. + +CloudQuery extracts, transforms, and loads configuration from cloud APIs to a variety of supported destinations such as databases, data lakes, or streaming platforms for further analysis. + +## Setup + +### Installation + +To ingest OpenTelemetry traces, metrics, and logs from CloudQuery, install the [Datadog Agent][11] version >=6.48.0 or >=7.48.0. +Alternatively, you can use OpenTelemetry Collector and Datadog Exporter as described below. + +### Configuration + +CloudQuery supports [OpenTelemetry][5] traces, metrics, and logs out of the box. +There are multiple ways to configure OpenTelemetry with Datadog: + +- [Using an OpenTelemetry collector](#opentelemetry-collector) +- [Direct OTEL Ingestion by the Datadog Agent through a configuration file](#datadog-agent-otel-ingestion-through-a-configuration-file) +- [Direct OTEL ingestion by the Datadog Agent through environment variables](#datadog-agent-otel-ingestion-through-environment-variables) + +For more information, see [OpenTelemetry in Datadog][6]. + +#### OpenTelemetry collector + +To configure an OpenTelemetry collector with Datadog: + +1. Create a configuration file. For example, create an `otel_collector_config.yaml` file with the content below: + +```yaml +receivers: + otlp: + protocols: + http: + endpoint: "0.0.0.0:4318" + +processors: + batch/datadog: + send_batch_max_size: 1000 + send_batch_size: 100 + timeout: 10s + +exporters: + datadog: + api: + site: ${env:DATADOG_SITE} + key: ${env:DATADOG_API_KEY} + +service: + pipelines: + metrics: + receivers: [otlp] + processors: [batch/datadog] + exporters: [datadog] + traces: + receivers: [otlp] + processors: [batch/datadog] + exporters: [datadog] + logs: + receivers: [otlp] + processors: [batch/datadog] + exporters: [datadog] +``` + +2. Run the collector with the following command (replace `DATADOG_SITE` and `DATADOG_API_KEY` with your own values): + +```bash +docker run \ + -p 4318:4318 \ + -e DATADOG_SITE=$DATADOG_SITE \ + -e DATADOG_API_KEY=$DATADOG_API_KEY \ + --hostname $(hostname) \ + -v $(pwd)/otel_collector_config.yaml:/etc/otelcol-contrib/config.yaml \ + otel/opentelemetry-collector-contrib:0.104.0 +``` +3. After you have the collector ready, specify the endpoint in the source spec: + +```yaml +kind: source +spec: + name: "aws" + path: "cloudquery/aws" + # Replace with the AWS source plugin version + version: "" + tables: ["aws_s3_buckets"] + destinations: ["postgresql"] + otel_endpoint: "0.0.0.0:4318" + otel_endpoint_insecure: true + spec: +``` + +For additional ways to run the collector, see [OpenTelemetry Deployment][7]. + +#### Datadog Agent OTEL ingestion through a configuration file + +1. Locate your [`datadog.yaml` Agent configuration file][8] and add the following configuration: + +```yaml +otlp_config: + receiver: + protocols: + http: + endpoint: 0.0.0.0:4318 + logs: + enabled: true +logs_enabled: true +``` + +2. [Restart][9] the Datadog agent for the change to take effect. + +3. After you have the Agent ready, specify the endpoint in the source spec: + +```yaml +kind: source +spec: + name: "aws" + path: "cloudquery/aws" + # Replace with the AWS source plugin version + version: "" + tables: ["aws_s3_buckets"] + destinations: ["postgresql"] + otel_endpoint: "0.0.0.0:4318" + otel_endpoint_insecure: true + spec: +``` + +#### Datadog Agent OTEL ingestion through environment variables + +1. Pass the `DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT` environment variable to the Datadog Agent with a value of `0.0.0.0:4318`. +If you're using Docker compose, see the example below: + +```yaml +version: "3.0" +services: + agent: + image: gcr.io/datadoghq/agent:7 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - /proc/:/host/proc/:ro + - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro + environment: + DD_API_KEY: redacted + DD_SITE: "datadoghq.eu" + DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318" + DD_LOGS_ENABLED: "true" + DD_OTLP_CONFIG_LOGS_ENABLED: "true" + ports: + - "4318:4318" +``` + +2. [Restart][9] the Datadog agent for the change to take effect. + +3. After you have the Agent ready, specify the endpoint in the source spec: +```yaml +kind: source +spec: + name: "aws" + path: "cloudquery/aws" + # Replace with the AWS source plugin version + version: "" + tables: ["aws_s3_buckets"] + destinations: ["postgresql"] + otel_endpoint: "0.0.0.0:4318" + otel_endpoint_insecure: true + spec: +``` + +For more ways to configure the Datadog Agent, see [OTLP Ingestion by the Datadog Agent][10]. + + +### Validation + +Run `cloudquery sync spec.yml`. +After ingestion starts, you should start seeing the traces in the Datadog [**APM Traces Explorer**][12]. +You can also validate metrics and logs in the [**Metrics Summary**][13] and [**Log Explorer**][14]. + +## Data Collected + +### Metrics + +The CloudQuery does not include any metrics. + +### Service Checks + +The CloudQuery does not include any service checks. + +### Events + +The CloudQuery does not include any events. + +## Uninstallation + +If you use the OpenTelemetry collector, you can stop it by running `docker stop `. +If you use the Datadog Agent, remove the configuration or environment variables you added and [restart][9] the agent. +Finally, delete the dashboard from your Datadog account. + +## Support + +For support, contact [CloudQuery][4]. + +[1]: https://raw.githubusercontent.com/DataDog/integrations-extras/master/cloudquery/images/cloudquery_logo_png_dark_background.png +[2]: https://www.cloudquery.io/ +[3]: https://github.com/cloudquery/cloudquery +[4]: https://www.cloudquery.io/pricing +[5]: https://opentelemetry.io/ +[6]: https://docs.datadoghq.com/opentelemetry/ +[7]: https://docs.datadoghq.com/opentelemetry/collector_exporter/deployment#running-the-collector +[8]: https://docs.datadoghq.com/agent/configuration/agent-configuration-files/ +[9]: https://docs.datadoghq.com/agent/configuration/agent-commands/#restart-the-agent +[10]: https://docs.datadoghq.com/opentelemetry/interoperability/otlp_ingest_in_the_agent#enabling-otlp-ingestion-on-the-datadog-agent +[11]: https://docs.datadoghq.com/agent/ +[12]: https://app.datadoghq.com/apm/traces +[13]: https://app.datadoghq.com/metric/summary +[14]: https://app.datadoghq.com/logs diff --git a/cloudquery/assets/dashboards/cloudquery_syncs_insights.json b/cloudquery/assets/dashboards/cloudquery_syncs_insights.json new file mode 100644 index 000000000..02a0e67d2 --- /dev/null +++ b/cloudquery/assets/dashboards/cloudquery_syncs_insights.json @@ -0,0 +1,392 @@ +{ + "title": "CloudQuery Syncs Insights", + "description": "", + "widgets": [ + { + "id": 6533183048352006, + "definition": { + "title": "test", + "banner_img": "", + "show_title": false, + "type": "group", + "layout_type": "ordered", + "widgets": [ + { + "id": 818019981090292, + "definition": { + "type": "note", + "content": "![CloudQueryLogo](https://docs.cloudquery.io/images/brand-logo/cloudquery_logo_png_light_background.png)", + "background_color": "white", + "font_size": "36", + "text_align": "center", + "vertical_align": "center", + "show_tick": false, + "tick_pos": "50%", + "tick_edge": "left", + "has_padding": true + }, + "layout": { "x": 0, "y": 0, "width": 6, "height": 2 } + }, + { + "id": 1803149444713820, + "definition": { + "type": "note", + "content": "Use this dashboard to get insights into CloudQuery syncs and optimize them. For more information on setting up CloudQuery with Datadog, see the [CloudQuery documentation](https://docs.cloudquery.io/docs/advanced-topics/monitoring#opentelemetry-and-datadog).", + "background_color": "white", + "font_size": "14", + "text_align": "left", + "vertical_align": "top", + "show_tick": false, + "tick_pos": "50%", + "tick_edge": "left", + "has_padding": true + }, + "layout": { "x": 0, "y": 2, "width": 3, "height": 2 } + }, + { + "id": 7570738384871020, + "definition": { + "type": "note", + "content": "## Links\n\n- [Home](https://www.cloudquery.io/)\n- [Download CLI](https://www.cloudquery.io/download)\n- [Register](https://www.cloudquery.io/auth/register)\n- [Pricing](https://www.cloudquery.io/pricing)", + "background_color": "white", + "font_size": "14", + "text_align": "left", + "vertical_align": "top", + "show_tick": false, + "tick_pos": "50%", + "tick_edge": "left", + "has_padding": true + }, + "layout": { "x": 3, "y": 2, "width": 3, "height": 2 } + } + ] + }, + "layout": { "x": 0, "y": 0, "width": 6, "height": 5 } + }, + { + "id": 7236287707805706, + "definition": { + "title": "Resources per table", + "title_size": "16", + "title_align": "left", + "type": "query_table", + "requests": [ + { + "queries": [ + { + "name": "query1", + "data_source": "metrics", + "query": "sum:sync.table.resources{$sync.table.name, $sync.client.id, $service} by {sync.table.name}.as_count()", + "aggregator": "sum" + } + ], + "response_format": "scalar", + "sort": { + "count": 500, + "order_by": [{ "type": "formula", "index": 0, "order": "desc" }] + }, + "formulas": [ + { + "cell_display_mode": "bar", + "alias": "Resources", + "formula": "query1" + } + ] + } + ], + "has_search_bar": "auto" + }, + "layout": { "x": 6, "y": 0, "width": 6, "height": 5 } + }, + { + "id": 7997006831764135, + "definition": { + "title": "Top 12 Slowest Tables (p95)", + "title_size": "16", + "title_align": "left", + "type": "split_group", + "source_widget_definition": { + "title": "", + "title_size": "16", + "title_align": "left", + "type": "timeseries", + "requests": [ + { + "formulas": [{ "formula": "p95" }], + "queries": [ + { + "data_source": "metrics", + "name": "p95", + "query": "p95:trace.io.cloudquery.internal{service:$service.value,resource_name:sync.table.$sync.table.name.value}" + } + ], + "response_format": "timeseries", + "style": { + "palette": "dog_classic", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "line" + } + ], + "markers": [] + }, + "split_config": { + "split_dimensions": [{ "one_graph_per": "resource_name" }], + "limit": 12, + "sort": { + "order": "desc", + "compute": { + "aggregation": "p95", + "metric": "trace.io.cloudquery.internal" + } + } + }, + "size": "sm", + "has_uniform_y_axes": true + }, + "layout": { "x": 0, "y": 5, "width": 12, "height": 3 } + }, + { + "id": 4618294854537207, + "definition": { + "title": "Throttled APIs detected in logs over time", + "title_size": "16", + "title_align": "left", + "show_legend": false, + "legend_layout": "auto", + "legend_columns": ["avg", "min", "max", "value", "sum"], + "type": "timeseries", + "requests": [ + { + "formulas": [{ "formula": "a" }], + "queries": [ + { + "data_source": "logs", + "name": "a", + "indexes": ["*"], + "compute": { "aggregation": "count" }, + "group_by": [], + "search": { + "query": "(@retry.attempt:>=2 OR \"retrying request\") @service.name:$service.value" + }, + "storage": "hot" + } + ], + "response_format": "timeseries", + "style": { "palette": "dog_classic" }, + "display_type": "bars" + } + ] + }, + "layout": { "x": 0, "y": 0, "width": 12, "height": 6 } + }, + { + "id": 5021998435653025, + "definition": { + "title": "Permission errors over time", + "title_size": "16", + "title_align": "left", + "show_legend": false, + "legend_layout": "auto", + "legend_columns": ["avg", "min", "max", "value", "sum"], + "type": "timeseries", + "requests": [ + { + "formulas": [{ "formula": "a" }], + "queries": [ + { + "data_source": "logs", + "name": "a", + "indexes": ["*"], + "compute": { "aggregation": "count" }, + "group_by": [], + "search": { + "query": "*:(*permissions OR denied*) -status:(info OR debug) @service.name:$service.value" + }, + "storage": "hot" + } + ], + "response_format": "timeseries", + "style": { "palette": "dog_classic" }, + "display_type": "bars" + } + ] + }, + "layout": { "x": 0, "y": 6, "width": 12, "height": 3 } + }, + { + "id": 2851755114443664, + "definition": { + "title": "Errors per table", + "title_size": "16", + "title_align": "left", + "type": "query_table", + "requests": [ + { + "queries": [ + { + "name": "query1", + "data_source": "metrics", + "query": "sum:sync.table.errors{$sync.table.name, $sync.client.id, $service} by {sync.table.name}.as_count()", + "aggregator": "sum" + } + ], + "response_format": "scalar", + "sort": { + "count": 500, + "order_by": [{ "type": "formula", "index": 0, "order": "desc" }] + }, + "formulas": [ + { + "cell_display_mode": "bar", + "alias": "Errors", + "formula": "query1" + } + ] + } + ], + "has_search_bar": "auto" + }, + "layout": { + "x": 0, + "y": 0, + "width": 5, + "height": 5, + "is_column_break": true + } + }, + { + "id": 8467667933609944, + "definition": { + "title": "Sync errors rate over time", + "title_size": "16", + "title_align": "left", + "show_legend": true, + "legend_layout": "auto", + "legend_columns": ["avg", "min", "max", "value", "sum"], + "type": "timeseries", + "requests": [ + { + "formulas": [{ "formula": "query1" }], + "queries": [ + { + "data_source": "metrics", + "name": "query1", + "query": "sum:sync.table.errors{$service, $sync.table.name, $sync.client.id}.as_rate()" + } + ], + "response_format": "timeseries", + "style": { + "palette": "dog_classic", + "order_by": "values", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "line" + } + ] + }, + "layout": { "x": 5, "y": 0, "width": 7, "height": 5 } + }, + { + "id": 4153628843582668, + "definition": { + "title": "Sync time per plugin over time", + "title_size": "16", + "title_align": "left", + "type": "split_group", + "source_widget_definition": { + "title": "", + "title_size": "16", + "title_align": "left", + "type": "timeseries", + "requests": [ + { + "formulas": [{ "formula": "query1" }], + "queries": [ + { + "data_source": "metrics", + "name": "query1", + "query": "sum:trace.io.cloudquery.internal{service:$service.value ,resource_name:sync} by {service}.as_count()" + } + ], + "response_format": "timeseries", + "style": { + "palette": "dog_classic", + "order_by": "values", + "line_type": "solid", + "line_width": "normal" + }, + "display_type": "bars" + } + ] + }, + "split_config": { + "split_dimensions": [{ "one_graph_per": "service" }], + "limit": 6, + "sort": { + "order": "desc", + "compute": { + "aggregation": "count", + "metric": "trace.io.cloudquery.internal" + } + } + }, + "size": "lg", + "has_uniform_y_axes": true + }, + "layout": { "x": 0, "y": 5, "width": 12, "height": 3 } + }, + { + "id": 4790101845729091, + "definition": { + "title": "Tables by sync duration", + "title_size": "16", + "title_align": "left", + "requests": [ + { + "response_format": "event_list", + "columns": [ + { "field": "service", "width": "auto" }, + { "field": "resource_name", "width": "auto" }, + { "field": "@duration", "width": "auto" }, + { "field": "@sync.client.id", "width": "auto" }, + { "field": "@sync.invocation.id", "width": "auto" } + ], + "query": { + "data_source": "trace_stream", + "query_string": "@_top_level:1 resource_name:sync.table.$sync.table.name.value service:$service.value @sync.client.id:$sync.client.id.value", + "indexes": [] + } + } + ], + "type": "list_stream" + }, + "layout": { "x": 0, "y": 0, "width": 12, "height": 10 } + } + ], + "template_variables": [ + { + "name": "service", + "prefix": "service", + "available_values": [], + "default": "*" + }, + { + "name": "sync.table.name", + "prefix": "sync.table.name", + "available_values": [], + "default": "*" + }, + { + "name": "sync.client.id", + "prefix": "sync.client.id", + "available_values": [], + "default": "*" + } + ], + "layout_type": "ordered", + "notify_list": [], + "reflow_type": "fixed" +} diff --git a/cloudquery/assets/service_checks.json b/cloudquery/assets/service_checks.json new file mode 100644 index 000000000..fe51488c7 --- /dev/null +++ b/cloudquery/assets/service_checks.json @@ -0,0 +1 @@ +[] diff --git a/cloudquery/images/cloudquery_logo_png_dark_background.png b/cloudquery/images/cloudquery_logo_png_dark_background.png new file mode 100644 index 000000000..c05f230b6 Binary files /dev/null and b/cloudquery/images/cloudquery_logo_png_dark_background.png differ diff --git a/cloudquery/images/cloudquery_logo_png_light_background.png b/cloudquery/images/cloudquery_logo_png_light_background.png new file mode 100644 index 000000000..aca120e33 Binary files /dev/null and b/cloudquery/images/cloudquery_logo_png_light_background.png differ diff --git a/cloudquery/images/cloudquery_logo_svg_dark_background.svg b/cloudquery/images/cloudquery_logo_svg_dark_background.svg new file mode 100644 index 000000000..f870834d4 --- /dev/null +++ b/cloudquery/images/cloudquery_logo_svg_dark_background.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/cloudquery/images/cloudquery_logo_svg_light_background.svg b/cloudquery/images/cloudquery_logo_svg_light_background.svg new file mode 100644 index 000000000..1daf7f67c --- /dev/null +++ b/cloudquery/images/cloudquery_logo_svg_light_background.svg @@ -0,0 +1,82 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/cloudquery/images/dashboard.png b/cloudquery/images/dashboard.png new file mode 100644 index 000000000..c4da221cc Binary files /dev/null and b/cloudquery/images/dashboard.png differ diff --git a/cloudquery/manifest.json b/cloudquery/manifest.json new file mode 100644 index 000000000..1d5a16948 --- /dev/null +++ b/cloudquery/manifest.json @@ -0,0 +1,62 @@ +{ + "manifest_version": "2.0.0", + "app_uuid": "9a7d20ab-d200-4c5a-ae7c-b3e4fa18407d", + "app_id": "cloudquery", + "display_on_public_website": true, + "tile": { + "overview": "README.md#Overview", + "configuration": "README.md#Setup", + "support": "README.md#Support", + "changelog": "CHANGELOG.md", + "description": "Monitor your CloudQuery syncs", + "title": "CloudQuery", + "media": [ + { + "media_type": "image", + "caption": "CloudQuery Syncs Insights", + "image_url": "images/dashboard.png" + } + ], + "classifier_tags": [ + "Supported OS::Linux", + "Supported OS::Windows", + "Supported OS::macOS", + "Category::Cost Management", + "Category::Data Stores", + "Category::Developer Tools", + "Category::Security", + "Offering::Integration", + "Submitted Data Type::Metrics", + "Submitted Data Type::Logs", + "Submitted Data Type::Traces" + ] + }, + "assets": { + "integration": { + "auto_install": true, + "source_type_id": 22006399, + "source_type_name": "CloudQuery", + "configuration": {}, + "events": { + "creates_events": false + }, + "metrics": { + "prefix": "cloudquery.", + "check": "", + "metadata_path": "metadata.csv" + }, + "service_checks": { + "metadata_path": "assets/service_checks.json" + } + }, + "dashboards": { + "CloudQuery": "assets/dashboards/cloudquery_syncs_insights.json" + } + }, + "author": { + "support_email": "support@cloudquery.io", + "name": "Community", + "homepage": "https://www.cloudquery.io/", + "sales_email": "hello@cloudquery.io" + } +} diff --git a/cloudquery/metadata.csv b/cloudquery/metadata.csv new file mode 100644 index 000000000..02cde5e98 --- /dev/null +++ b/cloudquery/metadata.csv @@ -0,0 +1 @@ +metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name,curated_metric,sample_tags