diff --git a/rsts/administrator/install/index.rst b/rsts/administrator/install/index.rst index d6f8579492..16f3d7fc06 100644 --- a/rsts/administrator/install/index.rst +++ b/rsts/administrator/install/index.rst @@ -13,4 +13,5 @@ This doc explains how to install Flyte, starting with the simplest method, and e production multi_cluster optional_components + managing_customizable_resources configure/index diff --git a/rsts/administrator/install/managing_customizable_resources.rst b/rsts/administrator/install/managing_customizable_resources.rst new file mode 100644 index 0000000000..a9f4ad3823 --- /dev/null +++ b/rsts/administrator/install/managing_customizable_resources.rst @@ -0,0 +1,117 @@ +.. _managing_customizable_resources: + +################################## +Configuring customizable resources +################################## + +As the complexity of your user-base grows, you may find yourself tweaking resource assignments based on specific projects, domains and workflows. +This document walks through how to use MatchableResource attributes to customize your workflow execution environment. + +Flyteadmin allows for overrides of task resource request and limit defaults, kubernetes cluster resource configuration, +dynamic task execution queues and specifying executions on specific kubernetes clusters. These can all be overriden for specific combinations +of domain; domain and project; domain, project and workflow (name); and domain, project, workflow (name), and launch plan. + +*************************** +Configurable Resource Types +*************************** + +The proto definition is the definitive source of +`matchable attributes `_ +which can be customized. See below for a detailed explanation + +Task Resources +============== + +This includes setting default value for task resource requests and limits for the following resources: + +- cpu + +- gpu + +- memory + +- storage + +In the absence of an override the global +`default values `_ +in the flyteadmin config are used. + +The override values from the database are assigned at execution time. + + +Cluster Resources +================= + +These are free-form key-value pairs which are used when creating project-domain based resources on Flyte kubernetes clusters. + +The keys represent templatized variables in `clusterresource template yaml `_ + +In the absence of custom override values, templateData from the `flyteadmin config `_ is used as a default. + + +Execution Queues +================ + +Execution queues are use to determine where dynamic tasks run. + +Execution queues themselves are currently defined in the +`flyteadmin config `_. + +The **attributes** associated with an execution queue must match the **tags** for workflow executions. The tags are associated with configurable resources +stored in the admin database. + + +Execution Cluster Label +======================= + +This allows forcing a matching execution to always execute on a specific kubernetes cluster. + + +********* +Hierarchy +********* + +Increasing specifity defines how matchable resource attributes get applied. The available configurations, in order of decreasing specifity are: + + +#. Domain, project, workflow name and launch plan. + +#. Domain, project and workflow name + +#. Domain and project + +#. Domain + +Default values for all and per-domain attributes may be specified in the flyteadmin config as documented above. + + +Example +======= + +Let's say that our database includes the following + ++------------+--------------+----------+-------------+-----------+ +| Domain | Project | Workflow | Launch Plan | Tags | ++============+==============+==========+=============+===========+ +| production | widgetmodels | | | critical | ++------------+--------------+----------+-------------+-----------+ +| production | widgetmodels | Demand | | supply | ++------------+--------------+----------+-------------+-----------+ + +Any inbound CreateExecution requests with **[Domain: Production, Project: widgetmodels, Workflow: Demand]** for any launch plan would have a tag value of "supply". +Any inbound CreateExecution requests with **[Domain: Production, Project: widgetmodels]** for any workflow other than Deman and for any launch plan would have a tag value of "critical". + +All other inbound CreateExecution requests would use the default values specified in the flyteadmin config (if any). + +********* +Debugging +********* + +Use the `get `_ endpoint +to see if overrides exist for a specific resource. + +E.g. `https://example.com/api/v1/project_domain_attributes/widgetmodels/production?resource_type=2 `_ + +To get the global state of the world, use the list all endpoint, e.g. `https://example.com/api/v1/matchable_attributes?resource_type=2 `_. + +The resource type enum (int) is defined in the `proto `_.