Improve JOB_POD variable naming + improve doc about memory management

[alafanechere]

What

• Some env var have POD in their name but are not specific to Kubernetes deployment: I suggest removing POD from their name.
• Our Scaling Airbyte documentation is not mentioning that we enable users to customize memory requests and limits for sync jobs.

How

• Rename JOB_POD_* to JOB_* env var when they are not strictly related to K8S deployments
• Update the Scaling airbyte documentation to mention JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT variable.

Recommended reading order

1. Double check my variable renaming is correct
2. Check if method names are consistent.
3. Check if the doc update is clear enough.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed.

[davinchia]

Thanks for taking this on. Agree leaving pod in the name was confusing.

Instead of only removing pod, what about introducing a convention where Kube only Job variables have the JOB_KUBE_ prefix? e.g. JOB_KUBE_NAMESPACE, JOB_KUBE_SOCAT_IMAGE etc. It would help differentiate between general job variables, and job variables that only apply to K8s.

[alafanechere]

Thanks for taking this on. Agree leaving pod in the name was confusing.

Instead of only removing pod, what about introducing a convention where Kube only Job variables have the JOB_KUBE_ prefix? e.g. JOB_KUBE_NAMESPACE, JOB_KUBE_SOCAT_IMAGE etc. It would help differentiate between general job variables, and job variables that only apply to K8s.

Agreed 🎯 Done

[davinchia]

Some more renames!

Please also make sure the methods match the variable names.

[davinchia]

Thanks!