Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use correct job resource values instead of env vars #114

Merged
merged 1 commit into from
Sep 16, 2024

Conversation

abuchanan-airbyte
Copy link
Collaborator

@abuchanan-airbyte abuchanan-airbyte commented Sep 12, 2024

Hopefully this fixes: https://github.com/airbytehq/airbyte-internal-issues/issues/9712

I think the issue was that we were setting env vars like JOB_MAIN_CONTAINER_CPU_LIMIT, but the charts set them via config maps via global values.

I can now run install multiple times, changing between --low-resource-mode and not

Existing cluster:

docker exec -it airbyte-abctl-control-plane kubectl -n airbyte-abctl get configmap -oyaml --field-selector metadata.name=airbyte-abctl-airbyte-env | grep JOB_
    JOB_MAIN_CONTAINER_CPU_LIMIT: "3"
    JOB_MAIN_CONTAINER_CPU_REQUEST: ""
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: 4Gi
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: ""

After running abctl local install --low-resource-mode:

docker exec -it airbyte-abctl-control-plane kubectl -n airbyte-abctl get configmap -oyaml --field-selector metadata.name=airbyte-abctl-airbyte-env | grep JOB_
    JOB_MAIN_CONTAINER_CPU_LIMIT: "0"
    JOB_MAIN_CONTAINER_CPU_REQUEST: "0"
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: "0"
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: "0"

After running abctl local install:

docker exec -it airbyte-abctl-control-plane kubectl -n airbyte-abctl get configmap -oyaml --field-selector metadata.name=airbyte-abctl-airbyte-env | grep JOB_
    JOB_MAIN_CONTAINER_CPU_LIMIT: "3"
    JOB_MAIN_CONTAINER_CPU_REQUEST: ""
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: 4Gi
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: ""

@abuchanan-airbyte abuchanan-airbyte requested a review from a team as a code owner September 12, 2024 21:57
@abuchanan-airbyte abuchanan-airbyte merged commit 1106fb9 into main Sep 16, 2024
3 checks passed
@abuchanan-airbyte abuchanan-airbyte deleted the abuch/fix-jobs-resource-values branch September 16, 2024 14:22
@jomach
Copy link

jomach commented Dec 5, 2024

Airbyte completely ignores these values on k8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants