1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
From fe8a27acd716a42667f5a572f52f2b04636010ff Mon Sep 17 00:00:00 2001
From: Florian Hahn <flo@fhahn.com>
Date: Tue, 29 Mar 2022 22:52:42 +0100
Subject: [PATCH] [LV] Handle zero cost loops in selectInterleaveCount.
In some case, like in the added test case, we can reach
selectInterleaveCount with loops that actually have a cost of 0.
Unfortunately a loop cost of 0 is also used to communicate that the cost
has not been computed yet. To resolve the crash, bail out if the cost
remains zero after computing it.
This seems like the best option, as there are multiple code paths that
return a cost of 0 to force a computation in selectInterleaveCount.
Computing the cost at multiple places up front there would unnecessarily
complicate the logic.
Fixes #54413.
(cherry picked from commit ecb4171dcbf1b433c9963fd605a74898303e850d)
---
.../Transforms/Vectorize/LoopVectorize.cpp | 22 ++++----
...ct-interleave-count-loop-with-cost-zero.ll | 50 +++++++++++++++++++
2 files changed, 62 insertions(+), 10 deletions(-)
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/pr54413-select-interleave-count-loop-with-cost-zero.ll
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 21c16f07e237..e1cc7946073e 100644
--- a/src/llvm-project/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/src/llvm-project/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -6035,6 +6035,18 @@ unsigned LoopVectorizationCostModel::selectInterleaveCount(ElementCount VF,
!(InterleaveSmallLoopScalarReduction && HasReductions && VF.isScalar()))
return 1;
+ // If we did not calculate the cost for VF (because the user selected the VF)
+ // then we calculate the cost of VF here.
+ if (LoopCost == 0) {
+ InstructionCost C = expectedCost(VF).first;
+ assert(C.isValid() && "Expected to have chosen a VF with valid cost");
+ LoopCost = *C.getValue();
+
+ // Loop body is free and there is no need for interleaving.
+ if (LoopCost == 0)
+ return 1;
+ }
+
RegisterUsage R = calculateRegisterUsage({VF})[0];
// We divide by these constants so assume that we have at least one
// instruction that uses at least one register.
@@ -6126,16 +6138,6 @@ unsigned LoopVectorizationCostModel::selectInterleaveCount(ElementCount VF,
assert(IC > 0 && "Interleave count must be greater than 0.");
- // If we did not calculate the cost for VF (because the user selected the VF)
- // then we calculate the cost of VF here.
- if (LoopCost == 0) {
- InstructionCost C = expectedCost(VF).first;
- assert(C.isValid() && "Expected to have chosen a VF with valid cost");
- LoopCost = *C.getValue();
- }
-
- assert(LoopCost && "Non-zero loop cost expected");
-
// Interleave if we vectorized this loop and there is a reduction that could
// benefit from interleaving.
if (VF.isVector() && HasReductions) {
|
