Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Metrics section to Scaling Airbyte doc #14224

Merged
merged 14 commits into from
Jun 29, 2022
Merged

Add Metrics section to Scaling Airbyte doc #14224

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
94ad925
Added metrics section to scaling airbyte doc
sophia-wiley Jun 27, 2022
7a00e30
Udpated URL in doc
sophia-wiley Jun 27, 2022
5d7745c
Delted link
sophia-wiley Jun 27, 2022
7d17e38
Added link
sophia-wiley Jun 27, 2022
d7d6b96
Added backslashes before brackets that aren't links
sophia-wiley Jun 27, 2022
fe82f88
Edited note about tagged metrics
sophia-wiley Jun 27, 2022
72489d0
Changed list
sophia-wiley Jun 28, 2022
269ba5b
Changed spacing
sophia-wiley Jun 28, 2022
3470d04
Changed spacing
sophia-wiley Jun 28, 2022
0c64ed2
Changed spacing
sophia-wiley Jun 28, 2022
01fa9dc
Deleted period
sophia-wiley Jun 28, 2022
4cd6480
Fixed broken firebolt link
sophia-wiley Jun 28, 2022
8f9fcb5
Added tables
sophia-wiley Jun 28, 2022
69ad126
Cleaned up wording in tables
sophia-wiley Jun 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs/operator-guides/scaling-airbyte.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,42 @@ The advice here is best-effort and by no means comprehensive. Please reach out o

If you've been running Airbyte in production and have more tips up your sleeve, we welcome contributions!

## Recommended Metrics
Airbyte supports exporting built-in metrics to Datadog or [OpenTelemetry](https://docs.airbyte.com/operator-guides/collecting-metrics/)

### Key Metrics
sophia-wiley marked this conversation as resolved.
Show resolved Hide resolved
* `oldest_pending_job_age_secs`
* This metric shows how long a pending job waits before it is scheduled. If a job is in pending state for a long time, more workers may be required.
sophia-wiley marked this conversation as resolved.
Show resolved Hide resolved
* `oldest_running_job_age_secs`
* This metric shows how long the oldest job has been running. A running job that is too large can indicate stuck jobs. This is relative to each job’s runtime.
* `job_failed_by_release_stage`
* This metric shows jobs that have failed in that release stage and is tagged as alpha, beta, or GA.

:::note

Metrics with `by_release_stage` in their name are tagged by connector release stage (alpha, beta, or GA). These tags allow you to filter by release stage. Alpha and beta connectors are less stable and have a higher failure rate than GA connectors, so filtering by those release stages can help you find failed jobs.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Amruta-Ranade, if you have any suggestions for wording here, I am all ears.


:::

:::code

**Example**

If a job was created for an Alpha source to a Beta destination, and the outcome of the job is a success, the following metrics are displayed:

`job_created_by_release_stage\[“alpha”\] = 1;`
`job_created_by_release_stage\[“beta”\] = 1;`
`job_failed_by_release_stage\[“alpha”\] = 1;`
`job_succeeded_by_release_stage\[“beta”\] = 1;`

**Note:** Each job has a source and destination, so each metric is counted twice — once for source and once for destination.

:::

### Additional Recommended Metrics
sophia-wiley marked this conversation as resolved.
Show resolved Hide resolved
* `num_running_jobs & num_pending_jobs`
* These two metrics show how many jobs are currently running and how many jobs are in pending state. These help you understand the general system state.
* `job_succeeded_by_release_stage`
* This metric shows successful jobs in that release stage and is tagged as alpha, beta, or GA.
* `job_created_by_release_stage`
* This metric shows the jobs created in that release stage and is tagged as alpha, beta, or GA.
98 changes: 93 additions & 5 deletions docusaurus/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,109 @@
"write-heading-ids": "docusaurus write-heading-ids"
},
"dependencies": {
"@babel/core": "7.18.6",
"@babel/helper-builder-binary-assignment-operator-visitor": "7.18.6",
"@babel/helper-explode-assignable-expression": "7.18.6",
"@babel/plugin-bugfix-safari-id-destructuring-collision-in-function-expression": "7.18.6",
"@babel/plugin-bugfix-v8-spread-parameters-in-optional-chaining": "7.18.6",
"@babel/plugin-proposal-async-generator-functions": "7.18.6",
"@babel/plugin-proposal-class-properties": "7.18.6",
"@babel/plugin-proposal-class-static-block": "7.18.6",
"@babel/plugin-proposal-dynamic-import": "7.18.6",
"@babel/plugin-proposal-export-namespace-from": "7.18.6",
"@babel/plugin-proposal-json-strings": "7.18.6",
"@babel/plugin-proposal-logical-assignment-operators": "7.18.6",
"@babel/plugin-proposal-nullish-coalescing-operator": "7.18.6",
"@babel/plugin-proposal-numeric-separator": "7.18.6",
"@babel/plugin-proposal-object-rest-spread": "7.18.6",
"@babel/plugin-proposal-optional-catch-binding": "7.18.6",
"@babel/plugin-proposal-private-methods": "7.18.6",
"@babel/plugin-proposal-private-property-in-object": "7.18.6",
"@babel/plugin-syntax-import-assertions": "7.18.6",
"@babel/plugin-syntax-typescript": "7.18.6",
"@babel/plugin-transform-arrow-functions": "7.18.6",
"@babel/plugin-transform-async-to-generator": "7.18.6",
"@babel/plugin-transform-block-scoped-functions": "7.18.6",
"@babel/plugin-transform-block-scoping": "7.18.6",
"@babel/plugin-transform-classes": "7.18.6",
"@babel/plugin-transform-computed-properties": "7.18.6",
"@babel/plugin-transform-destructuring": "7.18.6",
"@babel/plugin-transform-duplicate-keys": "7.18.6",
"@babel/plugin-transform-exponentiation-operator": "7.18.6",
"@babel/plugin-transform-for-of": "7.18.6",
"@babel/plugin-transform-function-name": "7.18.6",
"@babel/plugin-transform-literals": "7.18.6",
"@babel/plugin-transform-member-expression-literals": "7.18.6",
"@babel/plugin-transform-modules-amd": "7.18.6",
"@babel/plugin-transform-modules-commonjs": "7.18.6",
"@babel/plugin-transform-modules-systemjs": "7.18.6",
"@babel/plugin-transform-modules-umd": "7.18.6",
"@babel/plugin-transform-named-capturing-groups-regex": "7.18.6",
"@babel/plugin-transform-new-target": "7.18.6",
"@babel/plugin-transform-object-super": "7.18.6",
"@babel/plugin-transform-property-literals": "7.18.6",
"@babel/plugin-transform-react-display-name": "7.18.6",
"@babel/plugin-transform-react-jsx-development": "7.18.6",
"@babel/plugin-transform-react-pure-annotations": "7.18.6",
"@babel/plugin-transform-regenerator": "7.18.6",
"@babel/plugin-transform-reserved-words": "7.18.6",
"@babel/plugin-transform-runtime": "7.18.6",
"@babel/plugin-transform-shorthand-properties": "7.18.6",
"@babel/plugin-transform-spread": "7.18.6",
"@babel/plugin-transform-sticky-regex": "7.18.6",
"@babel/plugin-transform-template-literals": "7.18.6",
"@babel/plugin-transform-typeof-symbol": "7.18.6",
"@babel/plugin-transform-typescript": "7.18.6",
"@babel/plugin-transform-unicode-escapes": "7.18.6",
"@babel/plugin-transform-unicode-regex": "7.18.6",
"@babel/preset-env": "7.18.6",
"@babel/preset-react": "7.18.6",
"@babel/preset-typescript": "7.18.6",
"@babel/runtime-corejs3": "7.18.6",
"@cmfcmf/docusaurus-search-local": "^0.10.0",
"@docusaurus/core": "^2.0.0-beta.20",
"@docusaurus/plugin-client-redirects": "^2.0.0-beta.20",
"@docusaurus/plugin-google-gtag": "^2.0.0-beta.20",
"@docusaurus/preset-classic": "^2.0.0-beta.20",
"@docsearch/react": "3.1.0",
"@docusaurus/core": "^2.0.0-beta.21",
"@docusaurus/cssnano-preset": "2.0.0-beta.21",
"@docusaurus/module-type-aliases": "2.0.0-beta.21",
"@docusaurus/plugin-client-redirects": "2.0.0-beta.21",
"@docusaurus/plugin-debug": "2.0.0-beta.21",
"@docusaurus/plugin-google-analytics": "2.0.0-beta.21",
"@docusaurus/plugin-google-gtag": "2.0.0-beta.21",
"@docusaurus/plugin-sitemap": "2.0.0-beta.21",
"@docusaurus/preset-classic": "2.0.0-beta.21",
"@docusaurus/theme-classic": "2.0.0-beta.21",
"@docusaurus/theme-search-algolia": "2.0.0-beta.21",
"@docusaurus/types": "2.0.0-beta.21",
"@mdx-js/react": "^1.6.21",
"async": "2.6.4",
"autoprefixer": "10.4.7",
"clsx": "^1.1.1",
"copy-webpack-plugin": "11.0.0",
"core-js": "3.23.3",
"css-declaration-sorter": "6.3.0",
"css-minimizer-webpack-plugin": "4.0.0",
"cssnano": "5.1.12",
"cssnano-preset-advanced": "5.3.8",
"del": "6.1.1",
"docusaurus-plugin-hubspot": "^1.0.0",
"docusaurus-plugin-segment": "^1.0.3",
"nth-check": "2.0.1",
"postcss-convert-values": "5.1.2",
"postcss-discard-comments": "5.1.2",
"postcss-loader": "7.0.0",
"postcss-merge-longhand": "5.1.6",
"postcss-merge-rules": "5.1.2",
"postcss-minify-selectors": "5.2.1",
"postcss-normalize-positions": "5.1.1",
"postcss-normalize-repeat-style": "5.1.1",
"postcss-ordered-values": "5.1.3",
"prism-react-renderer": "^1.2.1",
"react": "^17.0.1",
"react-dom": "^17.0.1",
"trim": "0.0.3"
"react-router": "5.3.3",
"sockjs": "0.3.24",
"trim": "0.0.3",
"webpack-dev-server": "4.9.2"
},
"browserslist": {
"production": [
Expand Down
Loading