-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to use the allocation policy #1774
Comments
Hi @aq2013 and thanks for the issue.
GPU plugin works at node level so it can only affect the GPU selection among the GPUs that are in its control. Thus the "balanced" mode only applies within the node not over the whole cluster. e.g, when user deploys two Pods and they happen to be scheduled to the same node, GPU plugin will deploy one to GPU1 and the other to GPU2. Depending on your goal, I can think of two ways to get where you'd want to be:
Also, Flex 140 should have two GPUs per a physical card. With "shared-dev-num" == 2, there should be 4 EDIT: GAS doesn't solve the problem either. Its "balancedResource" also works at node level. |
Thanks for the reply. Now we added the anti-affinity rules to schedule Pods to different nodes. I will close this issue. |
Describe the support request
We created gpu device plugin with allocation-policy balanced and shared-dev-num 2. One node with one of Intel GPU Flex 140 GPU card. So the cluster have 2 node with Flex 140 GPU card and each node have 2 gpu.intel.com/i915.
Now we deployed 2 gpu applications, each requests 1 gpu.intel.com/i915. As I understand, when the allocation-policy is balanced ( " balanced mode spreads workloads among GPU devices"), these 2 applications should be scheduled to 2 different nodes with gpu cards. But we found all 2 applications were scheduled to the same node.
device-plugin:
worknode-104:
![截屏2024-07-08 下午2 43 41](https://private-user-images.githubusercontent.com/20430537/346454578-c2e5af47-af98-46d8-b55d-c654cf40a288.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1OTk3NjgsIm5iZiI6MTcyNDU5OTQ2OCwicGF0aCI6Ii8yMDQzMDUzNy8zNDY0NTQ1NzgtYzJlNWFmNDctYWY5OC00NmQ4LWI1NWQtYzY1NGNmNDBhMjg4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDE1MjQyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRjMjhmYzBmM2U5NTI0OWQ2MmUyMzYwNmJkYTQyY2VjYWI0NWQ4OTQ1MzllMjExMWFkYmQ1M2E0YTI5ZjVlYjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.PaWtm2BomfqbuTHOuBWcFWqoNxJns1KqDPesFefz8oM)
worknode-105:
![截屏2024-07-08 下午2 44 31](https://private-user-images.githubusercontent.com/20430537/346454844-94a1704b-607b-4b78-82a8-a2e96c8c77d3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1OTk3NjgsIm5iZiI6MTcyNDU5OTQ2OCwicGF0aCI6Ii8yMDQzMDUzNy8zNDY0NTQ4NDQtOTRhMTcwNGItNjA3Yi00Yjc4LTgyYTgtYTJlOTZjOGM3N2QzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDE1MjQyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdhOTBmOWQ5YjBlN2NhMzgxNzRhYzZhYTA5OGMyZmVlMTVmMTRhY2RmMDBmOGU0Y2M2ODcyNGE4ODA3MzMzODYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.7kv89uyvitKXS9eXoOmvQZLRO0zersy4rEfzaitq5ws)
2 applications on one node:
![截屏2024-07-03 下午4 32 02](https://private-user-images.githubusercontent.com/20430537/346455339-5a1999ba-ce00-49d0-baf2-e835904aa53f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1OTk3NjgsIm5iZiI6MTcyNDU5OTQ2OCwicGF0aCI6Ii8yMDQzMDUzNy8zNDY0NTUzMzktNWExOTk5YmEtY2UwMC00OWQwLWJhZjItZTgzNTkwNGFhNTNmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDE1MjQyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk2MjBkZjlmNGM4OWZhMTczOWI1NzIzM2U2ZmNhNTVkZTkwZTM3NjZhZGU0MDUxZjRhNzU5OGIxNjcxZjJkZDYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.A3V4BfkK9KGZVIy7aVknRGjqS1YTNpRiF9ANbsn6ZSs)
System (please complete the following information if applicable):
The text was updated successfully, but these errors were encountered: