Skip to content

Commit

Permalink
SIMD: Use alloc_pages_node to force alignment
Browse files Browse the repository at this point in the history
fxsave and xsave require the target address to be 16-/64-byte aligned.

kmalloc(_node) does not (yet) offer such fine-grained control over
alignment[0,1], even though it does "the right thing" most of the time
for power-of-2 sizes. unfortunately, alignment is completely off when
using certain debugging or hardening features/configs, such as KASAN,
slub_debug=Z or the not-yet-upstream SLAB_CANARY.

Use alloc_pages_node() instead which allows us to allocate page-aligned
memory. Since fpregs_state is padded to a full page anyway, and this
code is only relevant for x86 which has 4k pages, this approach should
not allocate any unnecessary memory but still guarantee the needed
alignment.

0: https://lwn.net/Articles/787740/
1: https://lore.kernel.org/linux-block/20190826111627.7505-1-vbabka@suse.cz/

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#9608
Closes openzfs#9674
  • Loading branch information
Fabian-Gruenbichler authored and tonyhutter committed Dec 26, 2019
1 parent 1871ae2 commit 91d6a71
Showing 1 changed file with 17 additions and 6 deletions.
23 changes: 17 additions & 6 deletions include/linux/simd_x86.h
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,8 @@
*/
#if defined(HAVE_KERNEL_FPU_INTERNAL)

#include <linux/mm.h>

extern union fpregs_state **zfs_kfpu_fpregs;

/*
Expand All @@ -156,7 +158,8 @@ kfpu_fini(void)

for_each_possible_cpu(cpu) {
if (zfs_kfpu_fpregs[cpu] != NULL) {
kfree(zfs_kfpu_fpregs[cpu]);
free_pages((unsigned long)zfs_kfpu_fpregs[cpu],
get_order(sizeof (union fpregs_state)));
}
}

Expand All @@ -166,20 +169,28 @@ kfpu_fini(void)
static inline int
kfpu_init(void)
{
int cpu;

zfs_kfpu_fpregs = kzalloc(num_possible_cpus() *
sizeof (union fpregs_state *), GFP_KERNEL);
if (zfs_kfpu_fpregs == NULL)
return (-ENOMEM);

/*
* The fxsave and xsave operations require 16-/64-byte alignment of
* the target memory. Since kmalloc() provides no alignment
* guarantee instead use alloc_pages_node().
*/
unsigned int order = get_order(sizeof (union fpregs_state));
int cpu;

for_each_possible_cpu(cpu) {
zfs_kfpu_fpregs[cpu] = kmalloc_node(sizeof (union fpregs_state),
GFP_KERNEL | __GFP_ZERO, cpu_to_node(cpu));
if (zfs_kfpu_fpregs[cpu] == NULL) {
struct page *page = alloc_pages_node(cpu_to_node(cpu),
GFP_KERNEL | __GFP_ZERO, order);
if (page == NULL) {
kfpu_fini();
return (-ENOMEM);
}

zfs_kfpu_fpregs[cpu] = page_address(page);
}

return (0);
Expand Down

0 comments on commit 91d6a71

Please sign in to comment.