[SPARK-4140] Document dynamic allocation #3731
Conversation
Signed-off-by: Tsuyoshi Ozawa <ozawa.tsuyoshi@gmail.com>
Signed-off-by: Tsuyoshi Ozawa <ozawa.tsuyoshi@gmail.com>
|
Test build #24575 has started for PR 3731 at commit
|
|
Test build #24575 has finished for PR 3731 at commit
|
|
Test PASSed. |
| configurations are described on the [configurations page](configuration.html#dynamic-allocation) | ||
| and in the subsequent sections in detail. | ||
|
|
||
| Additionally, your application must use an external shuffle service (described below). To enable |
There was a problem hiding this comment.
It would be nice to add a short clause explaining why this is the case
|
Super nice to have documentation at this level of detail. This mostly looks good, I left a few comments. |
|
Test build #24663 has started for PR 3731 at commit
|
|
Ok I'm merging this into master and 1.2 thanks guys |
Once the external shuffle service is also documented, the dynamic allocation section will link to it. Let me know if the whole dynamic allocation should be moved to its separate page; I personally think the organization might be cleaner that way. This patch builds on top of oza's work in #3689. aarondav pwendell Author: Andrew Or <andrew@databricks.com> Author: Tsuyoshi Ozawa <ozawa.tsuyoshi@gmail.com> Closes #3731 from andrewor14/document-dynamic-allocation and squashes the following commits: 1281447 [Andrew Or] Address a few comments b9843f2 [Andrew Or] Document the configs as well 246fb44 [Andrew Or] Merge branch 'SPARK-4839' of github.com:oza/spark into document-dynamic-allocation 8c64004 [Andrew Or] Add documentation for dynamic allocation (without configs) 6827b56 [Tsuyoshi Ozawa] Fixing a documentation of spark.dynamicAllocation.enabled. 53cff58 [Tsuyoshi Ozawa] Adding a documentation about dynamic resource allocation. (cherry picked from commit 15c03e1) Signed-off-by: Andrew Or <andrew@databricks.com>
|
Test build #24663 has finished for PR 3731 at commit
|
|
Test PASSed. |
|
@andrewor14 good job, thanks! |
|
@pwendell @andrewor14 Can we publish updated docs to the website? It looks like these changes aren't live yet: https://spark.apache.org/docs/latest/job-scheduling.html |
Once the external shuffle service is also documented, the dynamic allocation section will link to it. Let me know if the whole dynamic allocation should be moved to its separate page; I personally think the organization might be cleaner that way.
This patch builds on top of @oza's work in #3689.
@aarondav @pwendell