Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[clang] Do not substitute parameter pack while retaining the pack expansion #108197

Merged
merged 8 commits into from
Sep 12, 2024

Conversation

usx95
Copy link
Contributor

@usx95 usx95 commented Sep 11, 2024

Consider when Input[I] is a VarDecl with parameter pack. We would have already expanded the pack before the code change in the loopfor (unsigned I = 0; I != *NumExpansions; ++I) {.

Now in if (RetainExpansion) {, without this change, we continue to substitute the pack in the pattern even when we do not have meaningful ArgumentPackSubstitutionIndex set.

This leads to use of an invalid pack substitution index in TemplateInstantiator::TransformFunctionParmPackRefExpr in TransformedDecl = (*Pack)[getSema().ArgumentPackSubstitutionIndex];

This change sets ArgumentPackSubstitutionIndex to -1 while retaining expansion to instruct TransformFunctionParmPackRefExpr to build FunctionParmPackExpr instead of substituting the param pack.


There are other instances of RetainExpansion and IIUC, they should also unset the ArgumentPackSubstitutionIndex. It would be great if someone can verify my understanding. If this is correct then we could instead have a ArgumentPackSubstitutionIndexRAII as part of ForgetPartiallySubstitutedPackRAII.

EDIT: I have moved this to ForgetPartiallySubstitutedPackRAII.

Fixes #63819
Fixes #107560

@llvmbot llvmbot added clang Clang issues not falling into any other category clang:frontend Language frontend issues, e.g. anything involving "Sema" labels Sep 11, 2024
@llvmbot
Copy link
Member

llvmbot commented Sep 11, 2024

@llvm/pr-subscribers-clang

Author: Utkarsh Saxena (usx95)

Changes

Consider when Input[I] is a VarDecl with parameter pack. We would have already expanded the pack before the code change in the loopfor (unsigned I = 0; I != *NumExpansions; ++I) {.

Now in if (RetainExpansion) {, without this change, we continue to substitute the pack in the pattern even when we do not have meaningful ArgumentPackSubstitutionIndex set.

This leads to use of an invalid pack substitution index in TemplateInstantiator::TransformFunctionParmPackRefExpr containing TransformedDecl = (*Pack)[getSema().ArgumentPackSubstitutionIndex];

This change set ArgumentPackSubstitutionIndex to -1 while retaining expansion to instruct TransformFunctionParmPackRefExpr to build FunctionParmPackExpr instead of substituting the param pack.


There are other instances of RetainExpansion and IIUC, they should also unset the ArgumentPackSubstitutionIndex. It would be great if someone can verify my understanding. If this is correct then we could instead have a ArgumentPackSubstitutionIndexRAII as part of ForgetPartiallySubstitutedPackRAII.

Fixes #63819
Fixes #107560


Full diff: https://github.com/llvm/llvm-project/pull/108197.diff

1 Files Affected:

  • (modified) clang/lib/Sema/TreeTransform.h (+2-1)
diff --git a/clang/lib/Sema/TreeTransform.h b/clang/lib/Sema/TreeTransform.h
index 0daf620b4123e4..0de43d2127b12f 100644
--- a/clang/lib/Sema/TreeTransform.h
+++ b/clang/lib/Sema/TreeTransform.h
@@ -4361,7 +4361,8 @@ bool TreeTransform<Derived>::TransformExprs(Expr *const *Inputs,
       // forgetting the partially-substituted parameter pack.
       if (RetainExpansion) {
         ForgetPartiallySubstitutedPackRAII Forget(getDerived());
-
+        // Simple transform producing another pack expansion.
+        Sema::ArgumentPackSubstitutionIndexRAII SubstIndex(getSema(), -1);
         ExprResult Out = getDerived().TransformExpr(Pattern);
         if (Out.isInvalid())
           return true;

Copy link
Contributor

@ilya-biryukov ilya-biryukov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for not looking closer yet, but could we get a test case?
It would make it much easier to review this change.

@ilya-biryukov
Copy link
Contributor

Other than the test, the idea of the change does seem correct, here's the relevant comment from Sema.h:

/// The current index into pack expansion arguments that will be
/// used for substitution of parameter packs.
///
/// The pack expansion index will be -1 to indicate that parameter packs
/// should be instantiated as themselves. Otherwise, the index specifies
/// which argument within the parameter pack will be used for substitution.
int ArgumentPackSubstitutionIndex;

It looks clear that when RetainExpansion == true, we would like the behavior that ArgumentPackSubstitutionIndex == -1 produces. And given that we already forget the particular partially substituted pack, it's no surprise that using ArgumentPackSubstitutionIndex gives us an out-of-bounds access.

Maybe there are certain intricacies I am not getting, but I believe your direction to put this into ForgetPartiallySubstitutedPackRAII is the right way forward.

cc @zygoloid @shafik @cor3ntin in case they want to chime in.

@ilya-biryukov
Copy link
Contributor

I have spent some time poking at the code and looking at the debugger and came up with a smaller repro, see https://gcc.godbolt.org/z/6ccPPd6hz:

int bar(...);

template <int> struct Int {};

template <class ...T>
constexpr auto foo(T... x) -> decltype(bar(T(x)...)) { return 1; }
template <class ...T>
constexpr auto baz(Int<foo<T>(T())>... x) -> int { return 1; }

static_assert(baz<Int<1>, Int<2>, Int<3>>(Int<1>(), Int<2>(), Int<3>(), Int<4>()) == 1);

I hope this helps.

@usx95
Copy link
Contributor Author

usx95 commented Sep 11, 2024

Thanks. I have moved this to ForgetPartiallySubstitutedPackRAII.

@shafik shafik requested review from cor3ntin and Endilll September 12, 2024 04:39
Copy link
Contributor

@ilya-biryukov ilya-biryukov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side if all other tests pass.
Given that it's a small change and bugfix, I think it should be okay to land this without waiting for other reviewers. We could always follow up with more fixes if they have additional comments.

clang/lib/Sema/TreeTransform.h Show resolved Hide resolved
Copy link
Contributor

@cor3ntin cor3ntin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get a changelog entry?
LGTM otherwise

@usx95
Copy link
Contributor Author

usx95 commented Sep 12, 2024

Thanks for the review. Landing now. For other reviewers, feel free to drop more comments, and I would be happy to address in a followup.

@usx95 usx95 merged commit 849d1b8 into llvm:main Sep 12, 2024
6 of 8 checks passed
@llvm-ci
Copy link
Collaborator

llvm-ci commented Sep 12, 2024

LLVM Buildbot has detected a new failure on builder clang-armv8-quick running on linaro-clang-armv8-quick while building clang at step 5 "ninja check 1".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/154/builds/4277

Here is the relevant piece of the build log for the reference
Step 5 (ninja check 1) failure: stage 1 checked (failure)
******************** TEST 'lit :: googletest-timeout.py' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 9
not env -u FILECHECK_OPTS "/usr/bin/python3.10" /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/llvm/utils/lit/lit.py -j1 --order=lexical -v Inputs/googletest-timeout    --param gtest_filter=InfiniteLoopSubTest --timeout=1 > /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/Output/googletest-timeout.py.tmp.cmd.out
# executed command: not env -u FILECHECK_OPTS /usr/bin/python3.10 /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/llvm/utils/lit/lit.py -j1 --order=lexical -v Inputs/googletest-timeout --param gtest_filter=InfiniteLoopSubTest --timeout=1
# .---command stderr------------
# | lit.py: /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/llvm/utils/lit/lit/main.py:72: note: The test suite configuration requested an individual test timeout of 0 seconds but a timeout of 1 seconds was requested on the command line. Forcing timeout to be 1 seconds.
# `-----------------------------
# RUN: at line 11
FileCheck --check-prefix=CHECK-INF < /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/Output/googletest-timeout.py.tmp.cmd.out /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/googletest-timeout.py
# executed command: FileCheck --check-prefix=CHECK-INF /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/googletest-timeout.py
# RUN: at line 16
not env -u FILECHECK_OPTS "/usr/bin/python3.10" /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/llvm/utils/lit/lit.py -j1 --order=lexical -v Inputs/googletest-timeout   --param gtest_filter=InfiniteLoopSubTest  --param set_timeout=1   > /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/Output/googletest-timeout.py.tmp.cfgset.out
# executed command: not env -u FILECHECK_OPTS /usr/bin/python3.10 /home/tcwg-buildbot/worker/clang-armv8-quick/llvm/llvm/utils/lit/lit.py -j1 --order=lexical -v Inputs/googletest-timeout --param gtest_filter=InfiniteLoopSubTest --param set_timeout=1
# RUN: at line 19
FileCheck --check-prefix=CHECK-INF < /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/Output/googletest-timeout.py.tmp.cfgset.out /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/googletest-timeout.py
# executed command: FileCheck --check-prefix=CHECK-INF /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/googletest-timeout.py
# .---command stderr------------
# | /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/googletest-timeout.py:34:14: error: CHECK-INF: expected string not found in input
# | # CHECK-INF: Timed Out: 1
# |              ^
# | <stdin>:13:29: note: scanning from here
# | Reached timeout of 1 seconds
# |                             ^
# | <stdin>:37:2: note: possible intended match here
# |  Timed Out: 2 (100.00%)
# |  ^
# | 
# | Input file: <stdin>
# | Check file: /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/utils/lit/tests/googletest-timeout.py
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |             .
# |             .
# |             .
# |             8:  
# |             9:  
# |            10: -- 
# |            11: exit: -9 
# |            12: -- 
# |            13: Reached timeout of 1 seconds 
# | check:34'0                                 X error: no match found
# |            14: ******************** 
# | check:34'0     ~~~~~~~~~~~~~~~~~~~~~
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
clang:frontend Language frontend issues, e.g. anything involving "Sema" clang Clang issues not falling into any other category
Projects
None yet
6 participants