diff --git a/docs/configuration.md b/docs/configuration.md index 4bb5371cc2700..1f9fa7056697e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -398,6 +398,14 @@ Apart from these, the following properties are also available, and may be useful If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is recommended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations. + + spark.shuffle.file.buffer.kb + 100 + + Size of the in-memory buffer for each shuffle file output stream, in kilobytes. These buffers + reduce the number of disk seeks and system calls made in creating intermediate shuffle files. + + spark.shuffle.spill true