"Failed to inject profiler into <pid>"

[jakewins]

I'm running the following command:

$ ./profiler.sh list 5015

And am getting the following output:

Failed to inject profiler into 5015
	linux-vdso.so.1 (0x00007ffd4ddcb000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f62c2080000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f62c1e63000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f62c1ae1000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f62c17dd000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f62c15c6000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f62c1227000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f62c291a000)

The process I'm trying to attach to is running:

$ java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

The OS is:

$ uname -a
Linux ip-<snip> 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

[apangin]

Please, remove > /dev/null at profiler.sh:48 and rerun the script to see if jattach prints an error message.

[jakewins]

Well.. I did that, retried, and the attach worked just fine. Tried adding > /dev/null back, for sanity, but it seems to work now. I'm not entirely sure what I changed. I'll come back with updated output if I'm able to reproduce it again.

[krems]

I've got the same issue
After removing all >/dev/nulls the output is:

Connected to remote JVM
Response code = -1

Failed to inject profiler into 32698
	linux-vdso.so.1 =>  (0x00007fffa0d2d000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f27e9042000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f27e8e26000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f27e8b1f000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f27e889d000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f27e8687000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f27e82fa000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f27e98e5000)

Couldn't find any solution out there.
@apangin , please advise

[apangin]

@krems Are you building async-profiler from master without changes?

Please check what the following command prints.

java -agentpath:/path/to/libasyncProfiler.so=start,summary,flat -version

[krems]

Yes

$ java -agentpath:./build/libasyncProfiler.so=start,summary,flat -version
Started [cpu] profiling
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
--- Execution profile ---
Total samples:         10
Unknown (native):      3 (30.00%)

Frame buffer usage:    0.0079%

     2091384 (20.60%)       2  Method::make_jmethod_id(ClassLoaderData*, Method*)
     1136257 (11.19%)       1  java.io.BufferedWriter.write
     1015855 (10.00%)       1  java.lang.Thread.<init>
     1012302 ( 9.97%)       1  java.util.LinkedHashMap.linkNodeLast
     1007078 ( 9.92%)       1  ClassFileParser::parseClassFile(Symbol*, ClassLoaderData*, Handle, KlassHandle, GrowableArray<Handle>*, TempNewSymbol&, bool, Thread*)
     1006586 ( 9.91%)       1  SignatureStream::SignatureStream(Symbol*, bool)
      998909 ( 9.84%)       1  SymbolTable::lookup_only(char const*, int, unsigned int&)
      995460 ( 9.80%)       1  clock_gettime

[krems]

I should mention that I have to run profiler via sudo -u user ./profiler.sh

[apangin]

OK, the library seems to work.
Is anything printed in stderr of the target JVM when you run profiler.sh?

[krems]

No output, but I found out what was the problem.
I'd assume this is our administration settings issue.
Even though I was running profiler.sh under user, the fact that async-profiler/* files were not in that user's home dir prevented profiler from attaching to the JVM process.
Moving async-profiler dir to user's home fixed the problem

[hvanhovell]

I hit a similar error when I tried to attach the async profiler to something that was running inside a LXC container. Running it from inside the container did the trick for me.

[apangin]

Thank you for the details. The clue is that profiler library is loaded by the target JVM on behalf of the user of the JVM process. This implies that async-profiler binary must be accessible by that user by exactly the same absolute path. I'll update troubleshooting sections with this information.

[roy-20230331]

Dear @apangin,

I encounter this issue when I run async profiler in host, my application is running as container in this host.

[root@ip-1-1-1-1 async-profiler-2.9-linux-x64]# java -agentpath:/tmp/async-profiler-2.9-linux-x64/build/libasyncProfiler.so=start,flat -version
Profiling started
openjdk version "11.0.18" 2023-01-17 LTS
OpenJDK Runtime Environment Corretto-11.0.18.10.1 (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.18.10.1 (build 11.0.18+10-LTS, mixed mode)
 Execution profile 
Total samples       : 0
ns  percent  samples  top

[root@ip-1-1-1-1 async-profiler-2.9-linux-x64]# ./profiler.sh -d 30 -e mprotect -f /tmp/a.html 13751
Failed to inject profiler into 13751
./profiler.sh: line 116: /bin/true: No such file or directory
[root@ip-1-1-1-1 async-profiler-2.9-linux-x64]# ./profiler.sh -v
Async-profiler 2.9 built on Nov 26 2022
Copyright 2016-2021 Andrei Pangin
[oot@ip-1-1-1-1 async-profiler-2.9-linux-x64]#

I give all the permission(-rwxrwxrwx) to libasyncProfiler.so.

[root@ip-1-1-1-1 build]# ll
total 460
-rwxrwxrwx 1 root root   5015 Nov 26 22:40 async-profiler.jar
-rwxrwxrwx 1 root root  46172 Nov 26 22:40 converter.jar
-rwxrwxrwx 1 root root  18218 Nov 26 22:40 fdtransfer
-rwxrwxrwx 1 root root  24850 Nov 26 22:40 jattach
-rwxrwxrwx 1 root root 361312 Nov 26 22:40 libasyncProfiler.so
[root@ip-1-1-1-1 build]# 

Could u please kindly help to check?

[apangin]

@roy-20230331 libasyncProfiler.so should be available in the container at the same path as in the host. Please see the linked questions for details.

[roy-20230331]

I see, thx @apangin !