🎉 Create a Helm Chart For Airbyte #5891

jonstacks · 2021-09-07T18:33:51Z

What

See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case.
All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

This does not fall under any of the existing checklist items regarding connectors.

In EKS if you have a multi-az cluster volumes can get provisioned in seperate AZs and pods won't schedule

Prevents 'Throttling request took' error message

Due to the volume mounts, it needs to be recreated so volumes can be moved to a new node. Potentially, we could add an affinity so that the volume is already attached to the node for no downtime. Another alternative would be to use object storage or something for this.

marcosmarxm · 2021-09-08T20:21:22Z

thanks for the contribution @jonstacks this is amazing!

michel-tricot · 2021-09-09T02:45:08Z

Thank you @jonstacks !!

marcosmarxm · 2021-09-13T17:05:21Z

@jonstacks this is an incredible contribution and complex one! thanks a lot again. Davin and Jared will review this week your code. There is anything you feel needs attention before they took a look or I can help you?

jonstacks · 2021-09-13T19:45:56Z

@marcosmarxm, the only thing I can think of is that the environment variables which get stored in the configmap may not be correct for every possible combination of how a user is deploying minio, postgres, etc. Those should be easy fixes going forward though when they are encountered. When something new does come up, we can add a matrix of test cases(github actions) to make sure there are no regressions going forward once we fix an issue.

danieldiamond · 2021-09-14T06:07:08Z

Trying to run this but running into PV issues - is this missing a PersistentVolume configuration?

jonstacks · 2021-09-14T13:35:44Z

@danieldiamond,

Someone in the slack chat also brought this up. Looks like its caused by defaulting the server.persistence.stoargeClass to standard, when it should be ""(will use the default storage class). I'll get a fix out for it. In the meantime, you can supply:

server:
  persistence:
    storageClass: ""

to the current chart and it should use the default storage class. You might have to delete the current pvc if you get an error installing

jrhizor

This is fantastic!

I left a bunch of comments. All of the ones to @davinchia are forward-looking and can easily be done after this PR is merged.

The one thing that we really need to do before merging is add the couple files to bumpversion with the latest version already.

jrhizor · 2021-09-14T22:33:23Z

.github/workflows/helm.yaml

+      working-directory: ./charts/airbyte
+      run: make lint
+
+  generate-docs:


This step should check for docs that aren't in sync with the generated.

jrhizor · 2021-09-14T22:37:44Z

.github/workflows/helm.yaml

+      run: |
+        helm repo add bitnami https://charts.bitnami.com/bitnami
+        helm dep build
+    - name: Setup Kind Cluster


@davinchia we'll need to test how this works for the Github AMIs + set up running this step on external runners. Not sure what configuration this will need vs helm/kind-action@v1.2.0.

I ran this helm action in my fork and can try to move to helm/kind-action. This is just the one I was familiar with for bootstrapping a k8s cluster.

jrhizor · 2021-09-14T22:38:25Z

.github/workflows/helm.yaml

+      working-directory: charts/airbyte
+      run: make update-docs
+
+  install:


@davinchia We'll want to run acceptance tests at the end here and rename to test.

@jrhizor created #6131 to remind us to address some of these comments after we merge this in.

jrhizor · 2021-09-14T22:41:14Z

charts/airbyte/Chart.yaml

+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0


@davinchia any opinions on how we should handle chart versioning? Presumably we want to update this both manually and when we bump appVersion with bumpversion.

yeah I wonder if we can configure bumpversion to do this for us as well

jrhizor · 2021-09-14T22:42:28Z

charts/airbyte/Chart.yaml

+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "0.29.13-alpha"


We need to add this file to .bumpversion.cfg so it bumps the app version when we do a release

jrhizor · 2021-09-14T23:27:50Z

charts/airbyte/values.yaml

+
+server:
+  ## @param server.replicaCount Number of server replicas
+  replicaCount: 1


Same for the scheduler comment above. We're planning on making this horizontally scalable but haven't yet.

charts/airbyte/values.yaml

jrhizor · 2021-09-14T23:30:03Z

charts/airbyte/values.yaml

+  persistence:
+    accessMode: ReadWriteOnce
+    storageClass: standard
+    size: 1Gi


@davinchia we definitely want a larger storage here. Maybe that should be in a separate "production" values file?

From what I've seen just browsing the bitnami helm community, most projects usually leave the default pretty small(1-10Gi range). I think the reasoning is the same as not supplying resource request and limits so that it installs on things like mini kube or kind clusters. I think it would be up to the end user to maintain their own values files and apply them either with -f production.yaml or --set server.persistence.size when they install the chart.

charts/airbyte/templates/NOTES.txt

Addresses code review comment Co-authored-by: Jared Rhizor <me@jaredrhizor.com>

Due to pulling images, the install can sometimes take this long. Co-authored-by: Jared Rhizor <me@jaredrhizor.com>

davinchia

Nice work!

Some question to make sure I understand and two things from me:

can you make sure to add a newline at the end of every file?
can you make the variables in the configmap injectable from the values.yaml?

I'll merge this in once you do and make some changes so this better fits in with our build system.

Thank you for the contribution :)

davinchia · 2021-09-16T10:26:20Z

charts/airbyte/values.yaml

+    port: 80
+
+  ## Web app resource requests and limits
+  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/


@jonstacks are these comments generated or hand written?

@davinchia, hand written. I used a lot of the bitnami helm-charts as inspiration for writing the docs(ex: airflow). They might have something special for that, but I didn't see anything.

amazing - thought they were generated!

davinchia · 2021-09-16T10:42:35Z

charts/airbyte/Chart.yaml

+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0


yeah I wonder if we can configure bumpversion to do this for us as well

davinchia · 2021-09-16T10:43:10Z

.github/workflows/helm.yaml

+    - if: success()
+      name: Test airbyte chart
+      working-directory: ./charts/airbyte
+      run: ./ci.sh test


@jonstacks curious, what is this testing today?

@davinchia, This job in the workflow:

Spins up a kind cluster

Does a helm install of the chart with the default values.yaml file and waits(with the --wait 600s for all the pods to be ready according to the readiness probe. If there is a misconfig in the env-configmap, the pod should crash loop and the install will fail. I've seen this locally when it can't reach postgres or mino it just crash loops.

It then runs a helm test which spins up a test pod to make sure the web frontend index.html is able to be fetched. (If there are other good tests, like hitting specific routes on the server service, those can be added as well for a more comprehensive test)

You can find the an example and output of the action here(https://github.com/jonstacks/airbyte/runs/3613634997)

charts/airbyte/templates/NOTES.txt

davinchia · 2021-09-16T13:45:10Z

charts/airbyte/templates/env-configmap.yaml

@@ -0,0 +1,44 @@
+apiVersion: v1


Yes we should make this configurable.

I can handle splitting this out in a follow up PR.

davinchia · 2021-09-17T05:21:41Z

Spoke with Jon - he is going to follow up on the configmap changes in a second PR to keep things manageable.

* Update check connection method * #5796 silence printing full config when config validation fails (#5879) * - #5796 silence printing full config when config validation fails * fix unit tests after config validation check changes Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com> * Format google-search-console schemas (#6047) * Update ads_insights.json (#5946) fix ads_insights schema according to [facebook docs](https://developers.facebook.com/docs/marketing-api/reference/adgroup/insights/) and my own data * Bump connectors version + update docs (#6060) * 🐛 Source Facebook Marketing: Convert values' types according to schema types (#4978) * Convert values' types according to schema types * Put streams back to `configured_catalog.json` Put back `ads_insights` and `ads_insights_age_and_gender` streams. * Pickup changes from #5946 * Implement change request + fix previous PR * Update schema * Remove items_type from convert_to_schema_types() * Bump connectors version * add oauth to connector_base dependencies (#6064) * use spec when persisting source configs (#6036) * switch most usages of writing sources to using specs * fix other usages * fix test * only wait on the server in the scheduler, not the worker * fix * rephrase sanity check and remove stdout * 🎉 Source Stripe: Add `PaymentIntents` stream (#6004) * Add `PaymentIntents` stream * Update docs * Implement change request + few updates Split `source.py` file into `source.py` and `streams.py` files. Update `payment_intents.json` file. * Bump connectors version + update docs * Add skeleton for databricks destination (#5629) Co-authored-by: Liren Tu <tuliren.git@outlook.com> Co-authored-by: LiRen Tu <tuliren@gmail.com> * Revert "Add skeleton for databricks destination (#5629)" (#6066) This reverts commit 79256c4. * 🎉 New Destination: Databricks (#5998) Implement new destination connector for databricks delta lake. Resolves #2075. Co-authored-by: George Claireaux <george@claireaux.co.uk> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Source PostHog: add support for self-hosted instances (#6058) * publish #6058 (#6059) * Destination Kafka: correct spec json and data types in config (#6040) * correct spec json and data types in config * bump version * correct tests * correct config parser NPE * format files Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local> * Fix or delete broken links (#6069) * Fix more doc issues (#6072) * 🎉 Added optional platform flag for build image script (#6000) * Fix dependabot security alert. (#6073) * Pin set value to greater than 4.0.1 to fix security warning. * Format the rest of the connectors. * add coverage report (#6045) Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com> * Fix the format of the data returned by Google Ads oauth to match the config accepted by the connector (#6032) * update salesforce docs (#6081) * 🎉 Source Github: add caching for all streams (#5949) * Source Github: add checking for all streams * bump version, update changelogs * Disable automatic migration acceptance test (#5988) - The automatic migration acceptance test no longer works because of the new Flyway migration system. - The file-based migration system is being deprecated. * 🎉 CDK: Add requests native authenticator support (#5731) * Add requests native auth class * Update init file. Update type annotations. Bump version. * Update TokenAuthenticator implementation. Update Oauth2Authenticator implemetation. Add CHANGELOG.md record. * Update Oauth2Authenticator default value setting. Update CHANGELOG.md * Add requests native authenticator tests * Add CDK requests native __call__ method tests. Update CHANGELOG.md * Add outdated auth deprication messages * Update requests native auth __call__ method tests * Bump CDK version to 0.1.20 * Interface changes to support separating secrets from the config (#6065) * Interface changes to support separating secrets from the config * Cleanup from PR comments and whitespace * Update log message for empty env variable (#6115) Co-authored-by: Jared Rhizor <jared@dataline.io> * Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha (#6125) Co-authored-by: davinchia <davinchia@users.noreply.github.com> * return auth spec in the API when getting definition specification (#6121) * Ignore python test coverage files (#6144) * CDK: support nested refs resolving (#6044) Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com> * feat: path for nested fields (#6130) * feat: path for nested fields * fix: clipRule error * fix: remove field name * Fix request middleware for ConnectionService (#6148) * Jamakase/update onboarding flow (#5656) * Doc explains normalization full-refresh implications (#6097) * update docs * add info in quickstart connection page * update abhi comments Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local> * Fix migration validation issue (#6154) Resolves #6151. * Bump Airbyte version from 0.29.18-alpha to 0.29.19-alpha (#6156) Co-authored-by: tuliren <tuliren@users.noreply.github.com> * Add information on which destinations support Incremental - Deduped History in their docs (#6031) Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local> * Update Airbyte Spec acknowledgements. (#6155) Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local> * Update new integration request * Add back the migration acceptance test (#6163) * 🎉 Create a Helm Chart For Airbyte (#5891) See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case. All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install. * Fix OAuth Summary strings (#6143) Co-authored-by: Marcos Eliziario Santos <eliziario@users.noreply.github.com> Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com> Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com> Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Jared Rhizor <jared@dataline.io> Co-authored-by: George Claireaux <george@claireaux.co.uk> Co-authored-by: Liren Tu <tuliren.git@outlook.com> Co-authored-by: LiRen Tu <tuliren@gmail.com> Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local> Co-authored-by: Harsha Teja Kanna <h7kanna@users.noreply.github.com> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: Dmytro <avida.d3@gmail.com> Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com> Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com> Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com> Co-authored-by: davinchia <davinchia@users.noreply.github.com> Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com> Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com> Co-authored-by: tuliren <tuliren@users.noreply.github.com> Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io> Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local> Co-authored-by: Jonathan Stacks <jonstacks@users.noreply.github.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com>

* Add GET_FBA_INVENTORY_AGED_DATA data * Add GET_MERCHANT_LISTINGS_ALL_DATA stream support * Update schemas * Update configured_catalog.json * Update connector to airbyte-cdk * Add amazon seller partner test creds * Update state sample files * Apply code format * Update acceptance-test-config.yml * Add dummy integration test * Refactor auth signature. Update streams.py * Remove print_function import from auth.py * Refactor source class. Add pydantic spec. PR fixes. * Add dummy integration test * Typing added. Add _create_prepared_request docstring. * Add extra streams and schemas * Update docs and spec * Post merge code fixes * Fix test setup * Fix test setup * Add sample_state.json * Update reports streams logics. Update test and config files. * Update tests config. Small code style fixes. * Add reports stream slices. Update check_connection method. * Post review fixes. * Streams update * Add reports document retrieval and decrypting. Update schemas and configs. * Add CVS parsing into result rows * Update ReportsAmazonSPStream class to be the child of Stream class. Update GET_FLAT_FILE_OPEN_LISTINGS_DATA and GET_MERCHANT_LISTINGS_ALL_DATA schemas. * Schema updates * Source check method updated * Update ReportsAmazonSPStream retry report logics * Update check_connection source method * Update reports read_records method. Update report schemas. * Update streams.py * Update acceptance tests config. Add small code fixes. * Update report read_records logics * Add reports streams rate limit handling logics. Add rate limit unit tests. * Source Amazon SP: Update reports streams logics. (#5311) * Update check connection method * #5796 silence printing full config when config validation fails (#5879) * - #5796 silence printing full config when config validation fails * fix unit tests after config validation check changes Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com> * Format google-search-console schemas (#6047) * Update ads_insights.json (#5946) fix ads_insights schema according to [facebook docs](https://developers.facebook.com/docs/marketing-api/reference/adgroup/insights/) and my own data * Bump connectors version + update docs (#6060) * 🐛 Source Facebook Marketing: Convert values' types according to schema types (#4978) * Convert values' types according to schema types * Put streams back to `configured_catalog.json` Put back `ads_insights` and `ads_insights_age_and_gender` streams. * Pickup changes from #5946 * Implement change request + fix previous PR * Update schema * Remove items_type from convert_to_schema_types() * Bump connectors version * add oauth to connector_base dependencies (#6064) * use spec when persisting source configs (#6036) * switch most usages of writing sources to using specs * fix other usages * fix test * only wait on the server in the scheduler, not the worker * fix * rephrase sanity check and remove stdout * 🎉 Source Stripe: Add `PaymentIntents` stream (#6004) * Add `PaymentIntents` stream * Update docs * Implement change request + few updates Split `source.py` file into `source.py` and `streams.py` files. Update `payment_intents.json` file. * Bump connectors version + update docs * Add skeleton for databricks destination (#5629) Co-authored-by: Liren Tu <tuliren.git@outlook.com> Co-authored-by: LiRen Tu <tuliren@gmail.com> * Revert "Add skeleton for databricks destination (#5629)" (#6066) This reverts commit 79256c4. * 🎉 New Destination: Databricks (#5998) Implement new destination connector for databricks delta lake. Resolves #2075. Co-authored-by: George Claireaux <george@claireaux.co.uk> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Source PostHog: add support for self-hosted instances (#6058) * publish #6058 (#6059) * Destination Kafka: correct spec json and data types in config (#6040) * correct spec json and data types in config * bump version * correct tests * correct config parser NPE * format files Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local> * Fix or delete broken links (#6069) * Fix more doc issues (#6072) * 🎉 Added optional platform flag for build image script (#6000) * Fix dependabot security alert. (#6073) * Pin set value to greater than 4.0.1 to fix security warning. * Format the rest of the connectors. * add coverage report (#6045) Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com> * Fix the format of the data returned by Google Ads oauth to match the config accepted by the connector (#6032) * update salesforce docs (#6081) * 🎉 Source Github: add caching for all streams (#5949) * Source Github: add checking for all streams * bump version, update changelogs * Disable automatic migration acceptance test (#5988) - The automatic migration acceptance test no longer works because of the new Flyway migration system. - The file-based migration system is being deprecated. * 🎉 CDK: Add requests native authenticator support (#5731) * Add requests native auth class * Update init file. Update type annotations. Bump version. * Update TokenAuthenticator implementation. Update Oauth2Authenticator implemetation. Add CHANGELOG.md record. * Update Oauth2Authenticator default value setting. Update CHANGELOG.md * Add requests native authenticator tests * Add CDK requests native __call__ method tests. Update CHANGELOG.md * Add outdated auth deprication messages * Update requests native auth __call__ method tests * Bump CDK version to 0.1.20 * Interface changes to support separating secrets from the config (#6065) * Interface changes to support separating secrets from the config * Cleanup from PR comments and whitespace * Update log message for empty env variable (#6115) Co-authored-by: Jared Rhizor <jared@dataline.io> * Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha (#6125) Co-authored-by: davinchia <davinchia@users.noreply.github.com> * return auth spec in the API when getting definition specification (#6121) * Ignore python test coverage files (#6144) * CDK: support nested refs resolving (#6044) Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com> * feat: path for nested fields (#6130) * feat: path for nested fields * fix: clipRule error * fix: remove field name * Fix request middleware for ConnectionService (#6148) * Jamakase/update onboarding flow (#5656) * Doc explains normalization full-refresh implications (#6097) * update docs * add info in quickstart connection page * update abhi comments Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local> * Fix migration validation issue (#6154) Resolves #6151. * Bump Airbyte version from 0.29.18-alpha to 0.29.19-alpha (#6156) Co-authored-by: tuliren <tuliren@users.noreply.github.com> * Add information on which destinations support Incremental - Deduped History in their docs (#6031) Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local> * Update Airbyte Spec acknowledgements. (#6155) Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local> * Update new integration request * Add back the migration acceptance test (#6163) * 🎉 Create a Helm Chart For Airbyte (#5891) See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case. All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install. * Fix OAuth Summary strings (#6143) Co-authored-by: Marcos Eliziario Santos <eliziario@users.noreply.github.com> Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com> Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com> Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Jared Rhizor <jared@dataline.io> Co-authored-by: George Claireaux <george@claireaux.co.uk> Co-authored-by: Liren Tu <tuliren.git@outlook.com> Co-authored-by: LiRen Tu <tuliren@gmail.com> Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local> Co-authored-by: Harsha Teja Kanna <h7kanna@users.noreply.github.com> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: Dmytro <avida.d3@gmail.com> Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com> Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com> Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com> Co-authored-by: davinchia <davinchia@users.noreply.github.com> Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com> Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com> Co-authored-by: tuliren <tuliren@users.noreply.github.com> Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io> Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local> Co-authored-by: Jonathan Stacks <jonstacks@users.noreply.github.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com> * Bump source version. Update source docs. * Mock time.sleep in test_reports_stream_send_request_backoff_exception test * Acceptance test basic_read test disabled Co-authored-by: Marcos Eliziario Santos <eliziario@users.noreply.github.com> Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com> Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com> Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Jared Rhizor <jared@dataline.io> Co-authored-by: George Claireaux <george@claireaux.co.uk> Co-authored-by: Liren Tu <tuliren.git@outlook.com> Co-authored-by: LiRen Tu <tuliren@gmail.com> Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local> Co-authored-by: Harsha Teja Kanna <h7kanna@users.noreply.github.com> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: Dmytro <avida.d3@gmail.com> Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com> Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com> Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com> Co-authored-by: davinchia <davinchia@users.noreply.github.com> Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com> Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com> Co-authored-by: tuliren <tuliren@users.noreply.github.com> Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io> Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local> Co-authored-by: Jonathan Stacks <jonstacks@users.noreply.github.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com>

jonstacks added 7 commits September 7, 2021 12:51

Initial Helm chart creation

272e100

Fix ingress service name to match

367cc74

Use a single server volume with different subPaths

2ad7a3f

In EKS if you have a multi-az cluster volumes can get provisioned in seperate AZs and pods won't schedule

Add resources and podAnnotations

2fbb54f

Bitnami image needs a .kube folder that it can write to

6bac0d6

Prevents 'Throttling request took' error message

Add node selectors & tolerations to helm chart

ebbc323

jrhizor requested review from davinchia and jrhizor September 7, 2021 20:02

avaidyanatha changed the title ~~Create a Helm Chart For Airbyte~~ 🎉 Create a Helm Chart For Airbyte Sep 7, 2021

marcosmarxm added the community_pr label Sep 8, 2021

marcosmarxm self-assigned this Sep 8, 2021

davinchia mentioned this pull request Sep 9, 2021

Migrate Airbyte Cloud Helm charts over to use open source Helm chart #5934

Closed

jrhizor mentioned this pull request Sep 9, 2021

Create a helm chart/package #1868

Closed

jrhizor approved these changes Sep 14, 2021

View reviewed changes

jonstacks and others added 8 commits September 14, 2021 20:29

Configure server data pvc to default to using the default storage class

2df0ec4

Change namespace for Make install

595ca5a

Addresses code review comment Co-authored-by: Jared Rhizor <me@jaredrhizor.com>

Update timeout for chart install

8166e0f

Due to pulling images, the install can sometimes take this long. Co-authored-by: Jared Rhizor <me@jaredrhizor.com>

Fix typo in documentation

ab8a270

Change helm action for kind cluster setup to helm/kind-action

09828e6

Move from using make to a bash script

cbb0ff4

Fix minio endpoint

d6f2674

Fix postgres environment variable generation

c4a25d0

davinchia mentioned this pull request Sep 16, 2021

Clean up helm chart after we merge it in. #6131

Closed

davinchia approved these changes Sep 16, 2021

View reviewed changes

Add final newlines at end of files per PR feedback

fb59cdb

davinchia merged commit fe460ce into airbytehq:master Sep 17, 2021

jrhizor mentioned this pull request Sep 18, 2021

Bump Airbyte version from 0.29.19-alpha to 0.29.20-alpha #6243

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Create a Helm Chart For Airbyte #5891

🎉 Create a Helm Chart For Airbyte #5891

jonstacks commented Sep 7, 2021

marcosmarxm commented Sep 8, 2021

michel-tricot commented Sep 9, 2021

marcosmarxm commented Sep 13, 2021

jonstacks commented Sep 13, 2021

danieldiamond commented Sep 14, 2021

jonstacks commented Sep 14, 2021

jrhizor left a comment

jrhizor Sep 14, 2021

jrhizor Sep 14, 2021

jonstacks Sep 15, 2021

jrhizor Sep 14, 2021

davinchia Sep 16, 2021

jrhizor Sep 14, 2021

davinchia Sep 16, 2021

jrhizor Sep 14, 2021

jrhizor Sep 14, 2021

jrhizor Sep 14, 2021

jonstacks Sep 15, 2021

davinchia left a comment •

edited

Loading

davinchia Sep 16, 2021

jonstacks Sep 16, 2021

davinchia Sep 17, 2021

davinchia Sep 16, 2021

davinchia Sep 16, 2021

jonstacks Sep 16, 2021

davinchia Sep 16, 2021

davinchia commented Sep 17, 2021

🎉 Create a Helm Chart For Airbyte #5891

🎉 Create a Helm Chart For Airbyte #5891

Conversation

jonstacks commented Sep 7, 2021

What

Pre-merge Checklist

marcosmarxm commented Sep 8, 2021

michel-tricot commented Sep 9, 2021

marcosmarxm commented Sep 13, 2021

jonstacks commented Sep 13, 2021

danieldiamond commented Sep 14, 2021

jonstacks commented Sep 14, 2021

jrhizor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia commented Sep 17, 2021

davinchia left a comment •

edited

Loading