Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support CPU resource configurable for Kubernates job under MoK Mode #16008

Conversation

fectrain
Copy link
Contributor

@fectrain fectrain commented Feb 29, 2024

Fixes #16007 .

Description

Make CPU core configurable for Kubernetes job(Peon) under MoK Mode

Currently the CPU core for Peon is fixed to 1, this PR is to support adjusting it by adding a Overlord property: druid.indexer.runner.cpuCoreInMicro=2000

Verification

截屏2024-02-29 下午5 42 59 image

Release note


Key changed/added classes in this PR
  • KubernetesTaskRunnerConfig
  • PeonCommandContext
  • K8sTaskAdapter
  • DruidK8sConstants

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@georgew5656
Copy link
Contributor

will take a look at this today, looks pretty reasonable

Copy link
Contributor

@georgew5656 georgew5656 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic looks good to me, left a couple small comments on docs and changing the name of a test

@@ -572,14 +576,72 @@ void testEphemeralStorageIsRespected() throws IOException
Assertions.assertEquals(expected, actual);
}

@Test
void testEphemeralResourceIsEspected() throws IOException
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this test name is wrong, seems like it should be more like testCPUResourceIsRespected

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, thanks!

|`druid.indexer.runner.annotations`| `JsonObject` | Additional annotations you want to add to peon pod |`{}`|No|
|`druid.indexer.runner.peonMonitors`| `JsonArray` | Overrides `druid.monitoring.monitors`. Use this property if you don't want to inherit monitors from the Overlord. |`[]`|No|
|`druid.indexer.runner.graceTerminationPeriodSeconds`| `Long` | Number of seconds you want to wait after a sigterm for container lifecycle hooks to complete. Keep at a smaller value if you want tasks to hold locks for shorter periods. |`PT30S` (K8s default)|No|
|`druid.indexer.runner.capacity`| `Integer` | Number of concurrent jobs that can be sent to Kubernetes. |`2147483647`|No|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use the intellij props from here so the whole table doesn't get replaced like this? https://github.com/apache/druid/blob/master/dev/druid_intellij_formatting.xml#L77-L80

intellij config instructions: https://github.com/apache/druid/blob/master/dev/intellij-setup.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

@georgew5656 georgew5656 merged commit 4e9b758 into apache:master Mar 4, 2024
83 checks passed
@adarshsanjeev adarshsanjeev added this to the 30.0.0 milestone May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make CPU config of Container configurable under MoK mode
3 participants