Koordinator YARN Copilot

English | 简体中文

Introduction

Koordinator has supported hybrid orchestration workloads on Kubernetes, so that batch jobs can use the requested but unused resource as koord-batch priority and BE QoS class to improve the cluster utilization. However, there are still lots of applications running beyond K8s such as Apache Haddop YARN. As a resource management platform in BigData ecosystem, YARN has supported numbers of computing engines including MapReduce, Spark, Flink, Presto, etc.

In order to extend the co-location scenario of Koordinator, now the community has provided Hadoop YARN extended suits Koordinator YARN Copilot in BigData ecosystem, supporting running Hadoop YARN jobs by koord-batch resources with other K8s pods. The Koordinator YARN Copilot has following characteristics:

• Open-Source native: implement against open-sourced version of Hadoop YARN; so there is no hack inside YARN modules.
• Unifed resource priority and QoS strategy: the suits aims to the koord-batch priority of Koordinator, and also managed by QoS strategies of koordlet.
• Resource sharing on node level: node resources of koord-batch priority can be requested by tasks of YARN or Batch pods both.
• Adaptive for multiple environments: the suits can be run under any environment, including public cloud or IDC.

Quick Start

You can view the full documentation from the Koordinator website.

• Install or upgrade Koordinator with the latest version.
• Referring to best practices, there will be detailed instructions for running Hadoop YARN jobs with Koordinator batch resources in K8s.

Code of conduct

The Koordinator community is guided by our Code of Conduct, which we encourage everybody to read before participating.

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Contributing

You are warmly welcome to hack on Koordinator. We have prepared a detailed guide CONTRIBUTING.md.

Community

The koordinator-sh/community repository hosts all information about the community, membership and how to become them, developing inspection, who to contact about what, etc.

We encourage all contributors to become members. We aim to grow an active, healthy community of contributors, reviewers, and code owners. Learn more about requirements and responsibilities of membership in the community membership page.

Active communication channels:

• Bi-weekly Community Meeting (APAC, Chinese):
  • Tuesday 19:30 GMT+8 (Asia/Shanghai)
  • Meeting Link(DingTalk)
  • Notes and agenda
• Slack(English): koordinator channel in Kubernetes workspace
• DingTalk(Chinese): Search Group ID 33383887 or scan the following QR Code

License

Koordinator is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.