Import simpleperf profiles

[gregtatum]

We should support imported simpleperf profiles, as these are part of the Android profiling experience. They are close to the linux-perf profile format. When I took a stab at this, I got the generated HTML file that Android Studio saved for me, and worked on parsing the data out of the HTML file. I got stuck on updating the regex we have in our source to handle the slightly different data. This was awhile back, or else I'd share more details.

[julienw]

I'm just a bit surprised because we did some work in #2011. Or is this bug about that we want to import the output of report_sample.py instead of using jesup's modifications to report_sample_gecko.py?

Also from my earlier testing with just linux perf, I realized that our importer is very sensitive to the options we pass when starting perf (see #1792 (comment)). Indeed the number of columns will change, and as you showed in the channel our regex is very strict.
I think the main problem is I don't think that the perf format includes the requested columns themselves, so it's difficult to have a generic format. (but maybe an option can do it ?).

I'd be interested to know if the instructions in https://github.com/acreskeyMoz/simpleperf_for_geckoview_example still work for us. Also we could look at them and decide what we can do to simplify that.

[mstange]

I agree with Julien: The output of report_sample_gecko.py already works, we import it correctly.
So the other two formats we could consider importing are:

• the html output of report_sample.py
• the binary perf.data file.

For the perf.data format, the format is probably very similar to the Linux perf's perf.data format, which is documented in the linux kernel. I'm not sure at which points symbolication happens; it's possible that perf.data only contains addresses and that report_sample[_gecko].py perform symbolication. In that case, importing perf.data would mean that we would also need to handle symbolication in the Firefox profiler for these profiles. But on the other hand, if we had the raw frame addresses rather than just function names, we can show better information once we have assembly views and C++ source code views in the profiler.

[julienw]

I'm surprised there's nothing like perf script to make it possible to handle this data with the default linux perf tools (eg hotspot)... or is it because a lot of android developers aren't using Linux, so they didn't do it because it wouldn't be useful?

[mstange]

Here's the code from the android repo that can parse the raw simpleperf format: SimpleperfTraceParser.java.

It makes use of ProtoBuf generated code, generated from simplerperf_report.proto (lol, simplerperf, that must be a typo).

Piping that proto file through protoc (downloaded from here) with the command line
~/Downloads/protoc-3.11.4-osx-x86_64/bin/protoc --js_out=import_style=commonjs_strict,binary:. simplerperf_report.proto spits out this file: https://gist.github.com/mstange/2e31934083ba04bed8f2d88596a22a57

[mstange]

Nevermind, I'm not quite sure which format SimpleperfTraceParser parses - it's a format that starts with the magic header "SIMPLEPERF". But that's different from the perf.data format: The perf.data format has the magic header "PERFILE2".

Here's the simpleperf code that parses perf.data:
https://android.googlesource.com/platform/system/extras/+/refs/heads/master/simpleperf/record_file_format.h#25
https://android.googlesource.com/platform/system/extras/+/refs/heads/master/simpleperf/record_file.h#103
https://android.googlesource.com/platform/system/extras/+/refs/heads/master/simpleperf/record.h
https://android.googlesource.com/platform/system/extras/+/refs/heads/master/simpleperf/record.cpp
https://android.googlesource.com/platform/system/extras/+/refs/heads/master/simpleperf/record_file_reader.cpp

[mstange]

As for profiling vs symbolicating, the responsibilities seems to be as follows:

• First, you run app_profiler.py, which does the following:
  • Tell simpleperf on the phone to do profiling.
  • Wait until profiling is done
  • Copy perf.data from the phone to the host machine
  • Copy unstripped libraries to the binary_cache directory. No symbolication happens.
• Then you run report_html.py or one of its friends. This script does the symbolication, reading symbols from the libraries that are now ready in the binary_cache directory.

In other words, if we want to import the perf.data file, we'll get the raw addresses and library information, but we'll have to do symbolication ourselves.

[mstange]

Nevermind, I'm not quite sure which format SimpleperfTraceParser parses - it's a format that starts with the magic header "SIMPLEPERF".

This format is produced by the simpleperf report-sample command. For example, from the simpleperf github clone directory, running ./bin/darwin/x86_64/simpleperf report-sample --protobuf --show-callchain -o outfile creates a file called outfile that starts with the magic header "SIMPLEPERF".

[mstange]

I'm surprised there's nothing like perf script to make it possible to handle this data with the default linux perf tools (eg hotspot)...

Well, hotspot at least seems to use the perf.data file directly. So maybe in theory it could be adjusted to work with the simpleperf flavour of this file, too. But simply opening a perf.data file with hotspot gives:

bad feature data: 0x7ffef29941e0
No information about CPU architecture found. Cannot unwind.

[mstange]

... which is KDAB/hotspot#224 .

[mhansen]

FWIW, I have had success with importing simpleperf into Firefox Profiler. Using the simpleperf scripts from https://android.googlesource.com/platform/system/extras/+/master/simpleperf/doc/scripts_reference.md:

./app_profiler.py
./report_sample.py | gzip > perf.txt.gz

And importing the perf.txt.gz into Firefox Profiler.

So I think we might be able to close this issue?

[mstange]

Sounds good! Glad it's working.