[VPlan] Implement VPReductionRecipe::computeCost(). NFC #107790

ElvisWang123 · 2024-09-09T00:19:27Z

Implementation of computeCost() function for VPReductionRecipe.

Note that in-loop and any-of reductions are not supported by VPlan-based cost model currently.

llvmbot · 2024-09-09T00:24:57Z

@llvm/pr-subscribers-llvm-transforms

Author: Elvis Wang (ElvisWang123)

Changes

Implementation of computeCost() function for VPReductionRecipe.

Full diff: https://github.com/llvm/llvm-project/pull/107790.diff

2 Files Affected:

(modified) llvm/lib/Transforms/Vectorize/VPlan.h (+4)
(modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+24)

diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index bd71dbffa929e7..68f8ea3dea0db3 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -2242,6 +2242,10 @@ class VPReductionRecipe : public VPSingleDefRecipe {
   /// Generate the reduction in the loop
   void execute(VPTransformState &State) override;
 
+  /// Return the cost of VPReductionRecipe.
+  InstructionCost computeCost(ElementCount VF,
+                              VPCostContext &Ctx) const override;
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
   /// Print the recipe.
   void print(raw_ostream &O, const Twine &Indent,
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 3d08e3cefbf633..12b6d2c7d06dd1 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -1897,6 +1897,30 @@ void VPReductionEVLRecipe::execute(VPTransformState &State) {
   State.set(this, NewRed, 0, /*IsScalar*/ true);
 }
 
+InstructionCost VPReductionRecipe::computeCost(ElementCount VF,
+                                               VPCostContext &Ctx) const {
+  RecurKind RdxKind = RdxDesc.getRecurrenceKind();
+  Type *ElementTy = RdxDesc.getRecurrenceType();
+  auto *VectorTy = dyn_cast<VectorType>(ToVectorTy(ElementTy, VF));
+  TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput;
+  unsigned Opcode = RdxDesc.getOpcode();
+
+  if (VectorTy == nullptr)
+    return InstructionCost::getInvalid();
+
+  // Cost = Reduction cost + BinOp cost
+  InstructionCost Cost =
+      Ctx.TTI.getArithmeticInstrCost(Opcode, ElementTy, CostKind);
+  if (RecurrenceDescriptor::isMinMaxRecurrenceKind(RdxKind)) {
+    Intrinsic::ID Id = getMinMaxReductionIntrinsicOp(RdxKind);
+    return Cost + Ctx.TTI.getMinMaxReductionCost(
+                      Id, VectorTy, RdxDesc.getFastMathFlags(), CostKind);
+  }
+
+  return Cost + Ctx.TTI.getArithmeticReductionCost(
+                    Opcode, VectorTy, RdxDesc.getFastMathFlags(), CostKind);
+}
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void VPReductionRecipe::print(raw_ostream &O, const Twine &Indent,
                               VPSlotTracker &SlotTracker) const {

ElvisWang123 · 2024-09-11T02:37:23Z

Gentle ping!

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

fhahn · 2024-09-25T13:17:58Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

+                                               VPCostContext &Ctx) const {
+  RecurKind RdxKind = RdxDesc.getRecurrenceKind();
+  Type *ElementTy = RdxDesc.getRecurrenceType();
+  auto *VectorTy = cast<VectorType>(ToVectorTy(ElementTy, VF));


infer the type for the recipe instead and assert that the type matches RdxDesc.getRecurrenceType() to ensure consistency?

Sure, thanks.

fhahn · 2024-09-25T13:20:43Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

@@ -1999,6 +1999,27 @@ void VPReductionEVLRecipe::execute(VPTransformState &State) {
  State.set(this, NewRed, 0, /*IsScalar*/ true);
 }

+InstructionCost VPReductionRecipe::computeCost(ElementCount VF,
+                                               VPCostContext &Ctx) const {
+  RecurKind RdxKind = RdxDesc.getRecurrenceKind();


This computes the cost for non-in loop and non-any-of reductions, correct? Would be good to add an assert an explanation why (for those the cost needs to be pre-computed at the moment)

Thanks, I will add an assertion to check it is not an any-of reduction.

I tried to compute the cost of in-loop reductions by the vplan-based cost model.

It would probably be better to add support for in-loop reductions separately, as it adds extra complexity and may introduce new divergences

Sure, I will open another PR to address in-loop reduction.

fhahn

LGTM, thanks!

Would be good to update the description to clarify that this is for non-in-loop reductions and non any-of?

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

fhahn · 2024-10-08T19:18:40Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

+  // TODO: Support any-of reduction and in-loop reduction
+  assert(!RecurrenceDescriptor::isAnyOfRecurrenceKind(RdxKind) &&
+         "Not support any-of reduction in VPlan-based cost model currently.");
+


also assert that it's not an in-loop reduction?

Add a new function isInLoopReduction() to check if the instruction is in the inLoopReduction chain.

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

fhahn · 2024-10-09T20:02:24Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

+       ForceTargetInstructionCost.getNumOccurrences() > 0) &&
+      "Any-of reduction not implemented in VPlan-based cost model currently.");
+  assert(
+      (!Ctx.isInLoopReduction(getUnderlyingInstr(), VF, VectorTy) ||


Do we need to introduce a lookup via the legacy cost model here? IIRC the first op of the reduction recipe is the reduction PHI recipe, which has an in-loop flag. Can we check that instead?

Fixed, thanks.

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

Implementation of `computeCost()` function for `VPReductionRecipe`.

fhahn

LGTM, thanks!

Spotted one more typo, please feel free to land with if fixed

fhahn · 2024-10-14T15:13:26Z

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

+      "In-loop reduction not implemented in VPlan-based cost model currently.");
+
+  assert(ElementTy->getTypeID() == RdxDesc.getRecurrenceType()->getTypeID() &&
+         "Infered type and recurrence type mismatch.");


Suggested change

"Infered type and recurrence type mismatch.");

"Inferred type and recurrence type mismatch.");

Implementation of `computeCost()` function for `VPReductionRecipe`. Note that `in-loop` and `any-of` reductions are not supported by VPlan-based cost model currently.

ElvisWang123 requested review from fhahn, arcbbb and ayalz September 9, 2024 00:19

llvmbot added vectorizers llvm:transforms labels Sep 9, 2024

ElvisWang123 changed the title ~~[VPlan] Implment VPReductionRecipe::computeCost(). NFC~~ [VPlan] Implement VPReductionRecipe::computeCost(). NFC Sep 9, 2024

fhahn reviewed Sep 23, 2024

View reviewed changes

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp Outdated Show resolved Hide resolved

ElvisWang123 force-pushed the impl-vp-reduction-recipe-cost branch 2 times, most recently from a42e18f to c5fb73a Compare September 25, 2024 11:50

fhahn reviewed Sep 25, 2024

View reviewed changes

ElvisWang123 force-pushed the impl-vp-reduction-recipe-cost branch 2 times, most recently from faa86e5 to dc5b400 Compare October 4, 2024 10:20

fhahn approved these changes Oct 8, 2024

View reviewed changes

ElvisWang123 force-pushed the impl-vp-reduction-recipe-cost branch from dc5b400 to 20e90e7 Compare October 9, 2024 03:05

fhahn reviewed Oct 9, 2024

View reviewed changes

fhahn approved these changes Oct 10, 2024

View reviewed changes

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp Outdated Show resolved Hide resolved

ElvisWang123 added 6 commits October 13, 2024 19:28

[VPlan] Implment VPReductionRecipe::computeCost(). NFC

324659a

Implementation of `computeCost()` function for `VPReductionRecipe`.

Address comments.

91fcf39

Address comments.

4f2cc46

Address comments and add isInLoopReduction().

28b82ca

Fixup! Remove legacy model query

041f41b

Fixup! Using cast to avoid warning when compiling without assertion.

7acd294

ElvisWang123 force-pushed the impl-vp-reduction-recipe-cost branch from 562ae34 to 7acd294 Compare October 14, 2024 02:45

fhahn approved these changes Oct 14, 2024

View reviewed changes

Fixup! typo

80faae5

ElvisWang123 merged commit 3c91a2f into llvm:main Oct 15, 2024
8 checks passed

ElvisWang123 deleted the impl-vp-reduction-recipe-cost branch October 15, 2024 07:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VPlan] Implement VPReductionRecipe::computeCost(). NFC #107790

[VPlan] Implement VPReductionRecipe::computeCost(). NFC #107790

ElvisWang123 commented Sep 9, 2024 •

edited

Loading

llvmbot commented Sep 9, 2024

ElvisWang123 commented Sep 11, 2024

fhahn Sep 25, 2024

ElvisWang123 Oct 4, 2024

fhahn Sep 25, 2024

ElvisWang123 Oct 4, 2024

fhahn Oct 4, 2024

ElvisWang123 Oct 4, 2024

fhahn left a comment

fhahn Oct 8, 2024

ElvisWang123 Oct 9, 2024

fhahn Oct 9, 2024

ElvisWang123 Oct 9, 2024

fhahn left a comment

fhahn Oct 14, 2024

	"Infered type and recurrence type mismatch.");
	"Inferred type and recurrence type mismatch.");

[VPlan] Implement VPReductionRecipe::computeCost(). NFC #107790

[VPlan] Implement VPReductionRecipe::computeCost(). NFC #107790

Conversation

ElvisWang123 commented Sep 9, 2024 • edited Loading

llvmbot commented Sep 9, 2024

ElvisWang123 commented Sep 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fhahn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fhahn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ElvisWang123 commented Sep 9, 2024 •

edited

Loading