Issue while loading long trace files in trace_processor

[rohanaggarwal7997]

Hi this is a follow up to issue 435. I was able to move past the earlier error on a large trace file but now run into this. @LalitMaganti Can you please take a look ? I would be really grateful.

`[232.452] ing_trace_parser.cc:110 Fuchsia trace detected
[411.819] bump_allocator.cc:68 PERFETTO_CHECK(LastChunkIndex() < kMaxChunkCount) (errno: 0, Undefined error: 0)

-----BEGIN PERFETTO PRE-CRASH LOG-----
trace_uuid: 00000000-0000-0000-d919-7e42f24
[232.452] ing_trace_parser.cc:110 Fuchsia trace detected
[411.819] bump_allocator.cc:68 PERFETTO_CHECK(LastChunkIndex() < kMaxChunkCount) (errno: 0, Undefined error: 0)

-----END PERFETTO PRE-CRASH LOG-----

------------------ BEGINNING OF CRASH ------------------
Signal: Illegal instruction (possibly unaligned access)
Fault addr: 000000010AE8B610

Backtrace:

#00 _sigtramp

#1 perfetto::trace_processor::BumpAllocator::Alloc(unsigned int)

#2 perfetto::trace_processor::TraceTokenBuffer::AllocAndResizeInternedVectors(unsigned int)

#3 perfetto::trace_processor::TraceTokenBuffer::Id perfetto::trace_processor::TraceTokenBuffer::Appendperfetto::trace_processor::FuchsiaRecord(perfetto::trace_processor::FuchsiaRecord)

#4 perfetto::trace_processor::TraceSorter::PushFuchsiaRecord(long long, perfetto::trace_processor::FuchsiaRecord)

#5 perfetto::trace_processor::FuchsiaTraceTokenizer::ParseRecord(perfetto::trace_processor::TraceBlobView)

#6 perfetto::trace_processor::FuchsiaTraceTokenizer::Parse(perfetto::trace_processor::TraceBlobView)

#7 perfetto::trace_processor::ForwardingTraceParser::Parse(perfetto::trace_processor::TraceBlobView)

#8 perfetto::trace_processor::TraceProcessorStorageImpl::Parse(perfetto::trace_processor::TraceBlobView)

#9 perfetto::trace_processor::TraceProcessorImpl::Parse(perfetto::trace_processor::TraceBlobView)

#0A perfetto::trace_processor::ReadTraceUnfinalized(perfetto::trace_processor::TraceProcessor*, char const*, std::__1::function<void (unsigned long long)> const&)

#0B perfetto::trace_processor::(anonymous namespace)::LoadTrace(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, double*)

#0C perfetto::trace_processor::(anonymous namespace)::TraceProcessorMain(int, char**)

#0D main

#0E start
------------------ END OF CRASH ------------------
Illegal instruction
`

[LalitMaganti]

Do you have a repro trace to share? This doesn't repro with any large trace I have.

[rohanaggarwal7997]

Hi @LalitMaganti Thank you for the fast response. Unfortunately I cannot share the trace as much as i want to, because it contains confidential stuff. But here is a stack trace of the error with debug symbols on. Does this help ? I can help you with other things, please let me know.

* thread # 1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) frame # 0: 0x0000000103a3d610 trace_processor_shellperfetto::trace_processor::BumpAllocator::Alloc(this=0x00006000012f8018, size=80) at bump_allocator.cc:68:3
65 chunks_.emplace_back(std::move(chunk));
66
67 // Ensure that we haven't exceeded the maximum number of chunks.
-> 68 PERFETTO_CHECK(LastChunkIndex() < kMaxChunkCount);
69
70 // This time the allocation should definitely succeed in the last chunk (which
71 // we just added).
Target 0: (trace_processor_shell) stopped.
(lldb) bt

• thread # 1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)

  • frame # 0: 0x0000000103a3d610 trace_processor_shell`perfetto::trace_processor::BumpAllocator::Alloc(this=0x00006000012f8018, size=80) at bump_allocator.cc:68:3

  • frame # 1: 0x0000000103a35ab1 trace_processor_shell`perfetto::trace_processor::TraceTokenBuffer::AllocAndResizeInternedVectors(this=0x00006000012f8018, size=80) at trace_token_buffer.cc:250:48

    frame # 2: 0x0000000103ded975 trace_processor_shell`perfetto::trace_processor::TraceTokenBuffer::Id perfetto::trace_processor::TraceTokenBuffer::Appendperfetto::trace_processor::FuchsiaRecord(this=0x00006000012f8018, object=FuchsiaRecord @ 0x00007ff7bcc3e318) at trace_token_buffer.h:80:33

    frame # 3: 0x0000000103decbee trace_processor_shell`perfetto::trace_processor::TraceSorter::PushFuchsiaRecord(this=0x00006000012f8000, timestamp=203889523, fuchsia_record=) at trace_sorter.h:114:45

    frame # 4: 0x0000000103deb451 trace_processor_shell`perfetto::trace_processor::FuchsiaTraceTokenizer::ParseRecord(this=0x00006000015f4000, tbv=TraceBlobView @ 0x00007ff7bcc3ed28) at fuchsia_trace_tokenizer.cc:394:15

    frame # 5: 0x0000000103dea5dc trace_processor_shell`perfetto::trace_processor::FuchsiaTraceTokenizer::Parse(this=0x00006000015f4000, blob=TraceBlobView @ 0x00007ff7bcc3eea0) at fuchsia_trace_tokenizer.cc:179:5

    frame # 6: 0x00000001034f6d62 trace_processor_shell`perfetto::trace_processor::ForwardingTraceParser::Parse(this=0x00006000029ddf80, blob=TraceBlobView @ 0x00007ff7bcc3efd0) at forwarding_trace_parser.cc:160:19

    frame # 7: 0x0000000103513ea9 trace_processor_shell`perfetto::trace_processor::TraceProcessorStorageImpl::Parse(this=0x00007f941d7046d0, blob=TraceBlobView @ 0x00007ff7bcc3f050) at trace_processor_storage_impl.cc:117:48

    frame # 8: 0x00000001032fdda3 trace_processor_shell`perfetto::trace_processor::TraceProcessorImpl::Parse(this=0x00007f941d7046d0, blob=TraceBlobView @ 0x00007ff7bcc3f1a0) at trace_processor_impl.cc:906:37

    frame # 9: 0x00000001032f3c6c trace_processor_shell`perfetto::trace_processor::ReadTraceUnfinalized(tp=0x00007f941d7046d0, filename="trace_13594_512mb.fxt", progress_callback=0x00007ff7bcc3f330)> const&) at read_trace_internal.cc:107:9

    frame # 10: 0x00000001032c4622 trace_processor_shell`perfetto::trace_processor::(anonymous namespace)::LoadTrace(trace_file_path="trace_13594_512mb.fxt", size_mb=0x00007ff7bcc3f550) at trace_processor_shell.cc:1032:30

    frame # 11: 0x00000001032c29e5 trace_processor_shell`perfetto::trace_processor::(anonymous namespace)::TraceProcessorMain(argc=3, argv=0x00007ff7bcc3fa70) at trace_processor_shell.cc:1595:5

    frame # 12: 0x00000001032c2494 trace_processor_shell`main(argc=3, argv=0x00007ff7bcc3fa70) at trace_processor_shell.cc:1689:17

    frame # 13: 0x00007ff80792f310 dyldstart + 2432

[rohanaggarwal7997]

Trace processor is built with debug symbols on the latest commit 52366ac

[LalitMaganti]

Without the trace, I can simply guess at what the problem is.

I've sent https://android-review.git.corp.google.com/c/platform/external/perfetto/+/2457489 to try to fix this. Would appreciate your help in checking if this works.

[rohanaggarwal7997]

Hi @LalitMaganti Thank you for the quick update. I am not able to access this link as it appears to be a google internal link. Can you please help me out with a public link. Id be happy to help test it. Thanks.

[rohanaggarwal7997]

Actually https://android-review.googlesource.com/c/platform/external/perfetto/+/2457489 appears to be working - I copied this from the old issue and changed the number - Let me try this

[rohanaggarwal7997]

This was amazing...! @LalitMaganti You are awesome....!!! It worked. Now I am able to load all the traces..!