Task Level HW Resource Requirements

[amiryi365]

Hkube 1.1.171V
The Problem:
Hkube enables to define HW resources (#cpu, #gpu, memory) for Algorithm, but sometimes
the required resources depends on the input, so that different tasks of the same Algorithm require
very different resource sets (e.g. table of 1K rows vs. table with 1M rows).
e.g. I have an Alg which requires 3 to 15 GB RSS memory (4 to 16 Virtual Memory), depends on the inupt.

Desired Solution:
For better HW optimization, enable to optionally define HW requirements per task (it would override
the Algorithm level definitions).

Alternatives:
If there are large differences in HW requirements between tasks of same Alg, I may consider to define several algorithms with the same code but different sets of HW requirements.

[nassiharel]

@amiryi365 @yehiyam
We define HW resources only at the algorithm level.
The feature you suggest, require also to define HW resources at the pipeline level.
This is not feasible in the way Hkube works.

The Alternatives you suggest can do the trick.