Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Profile: make heap snapshots viewable in vscode viewer #53833

Merged
merged 3 commits into from
Apr 5, 2024

Conversation

IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Mar 24, 2024

Tested against https://github.com/microsoft/vscode-v8-heap-tools/tree/main/v8-heap-parser by copying a julia heapsnapshot into the test dir as basic.heapsnapshot and running cargo test

Update: This is working, except for the graph view, which appears to be an upstream issue microsoft/vscode-js-profile-visualizer#175


Original issues

With these changes, it still fails, but because of the middle line here

"codelocs",
"�\u0003",
"<generic memory - malloc>",

with

failures:

---- decoder::tests::test_basic_heapsnapshot stdout ----
thread 'decoder::tests::test_basic_heapsnapshot' panicked at v8-heap-parser/src/decoder.rs:574:33:
expect no errors: Error("invalid unicode code point", line: 6020770, column: 9)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- graph::tests::test_retained_sizes stdout ----
thread 'graph::tests::test_retained_sizes' panicked at v8-heap-parser/src/graph.rs:719:45:
expect no errors: Error("invalid unicode code point", line: 6020770, column: 9)


failures:
    decoder::tests::test_basic_heapsnapshot
    graph::tests::test_retained_sizes

If I manually empty that string the only test that fails is a content comparison against a fixed reference.
Does the string sanitizer need fixing or is this some other issue?

However.. the string-fixed file loads in neither chrome devtools, nor vscode's viewer.

I'm opening this in case you see the issue, @vilterp @NHDaly

@vtjnash
Copy link
Member

vtjnash commented Mar 24, 2024

JavaScript JSON strings differ slightly from Unicode encoding in that high code points are written as malformed UTF-16 surrogate pairs (when encoded as \x or \u escape sequences) instead of being encoded as their Unicode value (which is only supported in the very latest new version of ecmascript). Julia also may include arbitrary binary data in a string, while JSON only permits valid UTF-8 sequences (even as escape sequences)

@IanButterworth
Copy link
Member Author

IanButterworth commented Mar 24, 2024

I see. I just coincidentally pushed a change to only print unicode assigned chars. Does that sound reasonable?

With this they open in vscode now, although the graph view either takes longer than I've tried to build, or hangs. It's not clear

Screenshot 2024-03-24 at 11 48 27 AM

Screenshot 2024-03-24 at 11 50 04 AM

@IanButterworth
Copy link
Member Author

Note that vscode is on an older version of https://github.com/microsoft/vscode-v8-heap-tools/tree/1e1cbd86d9376995e459c31fbaacbd0f7a03adb9
Testing on the right commit helped..

@IanButterworth IanButterworth marked this pull request as ready for review March 24, 2024 19:40
@IanButterworth IanButterworth added backport 1.10 Change should be backported to the 1.10 release backport 1.11 Change should be backported to release-1.11 labels Mar 24, 2024
This was referenced Mar 27, 2024
@IanButterworth
Copy link
Member Author

It turns out that opening a heapsnapshot in the graph viewer via the dialog isn't valid and will be fixed (disabled) in microsoft/vscode-js-profile-visualizer#179

Instead, clicking on the graph icon where presented in the table view beside an object opens the graph successfully.

Screenshot 2024-04-04 at 4 30 50 PM

@vilterp @NHDaly I'll merge this tomorrow if I don't hear otherwise.

@vilterp
Copy link
Contributor

vilterp commented Apr 4, 2024

Oh, so it's just a 'retainer graph'? i.e. starting from a given object, go up to the roots? that's cool

Copy link
Contributor

@vilterp vilterp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@IanButterworth
Copy link
Member Author

Just remembered I needed to update stdlib/Manifest.toml because Profile took on Unicode as a dep.

@IanButterworth IanButterworth added the merge me PR is reviewed. Merge when all tests are passing label Apr 4, 2024
@IanButterworth IanButterworth merged commit 57bbff6 into JuliaLang:master Apr 5, 2024
8 checks passed
@IanButterworth IanButterworth deleted the ib/heap_vscode branch April 5, 2024 02:00
@IanButterworth IanButterworth removed the merge me PR is reviewed. Merge when all tests are passing label Apr 5, 2024
KristofferC pushed a commit that referenced this pull request Apr 9, 2024
KristofferC added a commit that referenced this pull request Apr 9, 2024
Backported PRs:
- [x] #53757 <!-- Add an IndexStyle example to the diagind docstring -->
- [x] #53809 <!-- Add missing GC_POP() in emit_cfunction -->
- [x] #53789 <!-- also check that UUID of project is non-null when
treating it as a package -->
- [x] #53805 <!-- precompilepkgs: simplify custom config printing if
only one -->
- [x] #53822 <!-- Bump libuv -->
- [x] #53837 <!-- update MPFR to 4.2.1 -->
- [x] #53862 <!-- precompilepkgs: fix error reporting -->
- [x] #53774 <!-- Remove some duplicates from emitted compilation traces
-->
- [ ] #53696 <!-- add invokelatest to on_done callback in bracketed
paste -->
- [x] #53383 <!-- Add `_unsetindex!` methods for `SubArray`s and
`CartesianIndex`es -->
- [x] #53475 <!-- Fix boundscheck in unsetindex for SubArrays -->
- [x] #53888 
- [x] #53870 <!-- Revert change to checksum for llvm-julia -->
- [x] #53906 <!-- Add `Base.isrelocatable(pkg)` -->
- [x] #53833 <!-- Profile: make heap snapshots viewable in vscode viewer
-->
- [x] #53961 <!-- `LazyString` in `LinearAlgebra.checksquare` error
message -->
- [x] #53962 <!-- Use StringMemory instead of StringVector where
possible -->
- [x] #53825 <!-- profile: doc: update the `Allocs.@profile` doc string
-->
- [x] #53975 <!-- `LazyString` in `DimensionMismatch` error messages in
broadcasting -->
- [x] #53905 <!-- Avoid repeated precompilation when loading from
non-relocatable cachefiles -->
- [x] #53896 <!-- Make reshape and view on Memory produce Arrays and
delete wrap -->
- [x] #53991 <!-- Test and fix non-int-length bug in `view(::Memory,
::Union{UnitRange, Base.OneTo})` -->
vtjnash pushed a commit that referenced this pull request Apr 16, 2024
Followup to #53833
Fixes a failure seen in #53974
(below)

I believe this is the more correct check to make?

The heapsnapshot generated from this PR is viewable in vscode.

```
2024-04-06 09:33:58 EDT	      From worker 7:	ERROR: Base.InvalidCharError{Char}('\xc1\xae')
2024-04-06 09:33:58 EDT	      From worker 7:	Stacktrace:
2024-04-06 09:33:58 EDT	      From worker 7:	  [1] throw_invalid_char(c::Char)
2024-04-06 09:33:58 EDT	      From worker 7:	    @ Base ./char.jl:86
2024-04-06 09:33:58 EDT	      From worker 7:	  [2] UInt32
2024-04-06 09:33:58 EDT	      From worker 7:	    @ ./char.jl:133 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [3] category_code
2024-04-06 09:33:58 EDT	      From worker 7:	    @ ./strings/unicode.jl:339 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [4] isassigned
2024-04-06 09:33:58 EDT	      From worker 7:	    @ ./strings/unicode.jl:355 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [5] isassigned
2024-04-06 09:33:58 EDT	      From worker 7:	    @ /cache/build/tester-amdci5-14/julialang/julia-master/julia-41d026beaf/share/julia/stdlib/v1.12/Unicode/src/Unicode.jl:138 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [6] print_str_escape_json(stream::IOStream, s::String)
2024-04-06 09:33:58 EDT	      From worker 7:	    @ Profile.HeapSnapshot /cache/build/tester-amdci5-14/julialang/julia-master/julia-41d026beaf/share/julia/stdlib/v1.12/Profile/src/heapsnapshot_reassemble.jl:239
2024-04-06 09:33:59 EDT	      From worker 7:	  [7] (::Profile.HeapSnapshot.var"#5#6"{IOStream})(strings_io::IOStream)
2024-04-06 09:33:59 EDT	      From worker 7:	    @ Profile.HeapSnapshot /cache/build/tester-amdci5-14/julialang/julia-master/julia-41d026beaf/share/julia/stdlib/v1.12/Profile/src/heapsnapshot_reassemble.jl:192
```
@vtjnash
Copy link
Member

vtjnash commented Apr 16, 2024

I had thought jl_static_show_string from #52799 would have already fixed this, but I didn't realize that this code used a different string encoder that is somewhat more specific to JSON output. That change might still affect the places where jl_static_show is used here (e.g. for printing some Type representations) but not commonly so

KristofferC pushed a commit that referenced this pull request Apr 17, 2024
Followup to #53833
Fixes a failure seen in #53974
(below)

I believe this is the more correct check to make?

The heapsnapshot generated from this PR is viewable in vscode.

```
2024-04-06 09:33:58 EDT	      From worker 7:	ERROR: Base.InvalidCharError{Char}('\xc1\xae')
2024-04-06 09:33:58 EDT	      From worker 7:	Stacktrace:
2024-04-06 09:33:58 EDT	      From worker 7:	  [1] throw_invalid_char(c::Char)
2024-04-06 09:33:58 EDT	      From worker 7:	    @ Base ./char.jl:86
2024-04-06 09:33:58 EDT	      From worker 7:	  [2] UInt32
2024-04-06 09:33:58 EDT	      From worker 7:	    @ ./char.jl:133 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [3] category_code
2024-04-06 09:33:58 EDT	      From worker 7:	    @ ./strings/unicode.jl:339 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [4] isassigned
2024-04-06 09:33:58 EDT	      From worker 7:	    @ ./strings/unicode.jl:355 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [5] isassigned
2024-04-06 09:33:58 EDT	      From worker 7:	    @ /cache/build/tester-amdci5-14/julialang/julia-master/julia-41d026beaf/share/julia/stdlib/v1.12/Unicode/src/Unicode.jl:138 [inlined]
2024-04-06 09:33:58 EDT	      From worker 7:	  [6] print_str_escape_json(stream::IOStream, s::String)
2024-04-06 09:33:58 EDT	      From worker 7:	    @ Profile.HeapSnapshot /cache/build/tester-amdci5-14/julialang/julia-master/julia-41d026beaf/share/julia/stdlib/v1.12/Profile/src/heapsnapshot_reassemble.jl:239
2024-04-06 09:33:59 EDT	      From worker 7:	  [7] (::Profile.HeapSnapshot.var"#5#6"{IOStream})(strings_io::IOStream)
2024-04-06 09:33:59 EDT	      From worker 7:	    @ Profile.HeapSnapshot /cache/build/tester-amdci5-14/julialang/julia-master/julia-41d026beaf/share/julia/stdlib/v1.12/Profile/src/heapsnapshot_reassemble.jl:192
```

(cherry picked from commit c557636)
@KristofferC KristofferC removed the backport 1.11 Change should be backported to release-1.11 label Apr 17, 2024
@KristofferC KristofferC mentioned this pull request May 8, 2024
23 tasks
KristofferC added a commit that referenced this pull request May 30, 2024
Backported PRs:
- [x] #54010 <!-- Overload `Base.literal_pow` for `AbstractQ` -->
- [x] #54143 <!-- Fix `make install` from tarballs -->
- [x] #54151 <!-- LinearAlgebra: Correct zero element in
`_generic_matvecmul!` for block adj/trans -->
- [x] #54233 <!-- set MAX_OS_WRITE on unix -->
- [x] #54251 <!-- fix typo in gc_mark_memory8 when chunking a large
array -->
- [x] #54363 <!-- typeintersect: fix another stack overflow caused by
circular constraints -->
- [x] #54497 <!-- Make TestLogger thread-safe (introduce a lock) -->
- [x] #53796 <!-- Add a missing doc -->
- [x] #54465 <!-- typeintersect: conservative typevar subtitution during
`finish_unionall` -->
- [x] #54514 <!-- typeintersect: followup cleanup for the nothrow path
of type instantiation -->

Need manual backport:
- [ ] #52505 <!-- fix alignment of emit_unbox_store copy -->
- [ ] #53373 <!-- fix sysimage-native-code=no option with pkgimages -->
- [ ] #53815 <!-- create phantom task for GC threads -->
- [ ] #53984 <!-- Profile: fix heap snapshot is valid char check -->
- [ ] #54276 <!-- Fix solve for complex `Hermitian` with non-vanishing
imaginary part on diagonal -->

Contains multiple commits, manual intervention needed:
- [ ] #52854 <!-- Change to streaming out the heap snapshot data -->
- [ ] #53218 <!-- Fix interpreter_exec.jl test -->
- [ ] #53833 <!-- Profile: make heap snapshots viewable in vscode viewer
-->
- [ ] #54303 <!-- LinearAlgebra: improve type-inference in
Symmetric/Hermitian matmul -->
- [ ] #52694 <!-- Reinstate similar for AbstractQ for backward
compatibility -->

Non-merged PRs with backport label:
- [ ] #54471 <!-- Actually setup jit targets when compiling
packageimages instead of targeting only one -->
- [ ] #53452 <!-- RFC: allow Tuple{Union{}}, returning Union{} -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
@KristofferC KristofferC mentioned this pull request Jun 19, 2024
46 tasks
KristofferC added a commit that referenced this pull request Aug 13, 2024
Backported PRs:
- [x] #51351 <!-- Remove boxing in pinv -->
- [x] #52678 <!-- Profile: Improve module docstring -->
- [x] #54201 <!-- Fix generic triangular solves with empty matrices -->
- [x] #54605 <!-- Allow libquadmath to also fail as it is not available
on all systems -->
- [x] #54634 <!-- Fix trampoline assembly for build on clang 18 on apple
silicon -->
- [x] #54635 <!-- Aggressive constprop in trevc! to stabilize triangular
eigvec -->
- [x] #54645 <!-- ensure we set the right value to gc_first_tid -->
- [x] #54671 <!-- Add boundscheck in bindingkey_eq to avoid OOB access
due to data race -->
- [x] #54672 <!-- make: Fix `sed` command for LLVM libraries with no
symbol versioning -->
- [x] #54704 <!-- LazyString in reinterpretarray error messages -->
- [x] #54713 <!-- make: use `readelf` for LLVM symbol version detection
-->
- [x] #54781 <!-- [LinearAlgebra] Improve resilience to unknown
libblastrampoline flags -->
- [x] #54837 <!-- Do not add type tag size to the `alloc_typed` lowering
for GC allocations -->
- [x] #54815 <!-- add sticky task warning to `@task` and `schedule` -->
- [x] #55141 <!-- Update the aarch64 devdocs to reflect the current
state of its support -->
- [x] #55178 <!-- Compat for `Base.@nospecializeinfer` -->
- [x] #55197 <!-- compat notice for a[begin] indexing -->
- [x] #55209 <!-- correction to compat notice for a[begin] -->
- [x] #55203 <!-- document mutable struct const fields -->
- [x] #54769 <!-- add missing compat entry to edit -->
- [x] #54791 <!-- Bump libblastrampoline to v5.10.1 -->
- [x] #55070 <!-- LinearAlgebra: LazyString in error messages for
Diagonal/Bidiagonal -->
- [x] #54624 <!-- more precise aliasing checks for SubArray -->
- [x] #54690 <!-- Fix assertion/crash when optimizing function with dead
basic block -->
- [x] #55084 <!-- Use triple quotes in TOML.print when string contains
newline -->


Need manual backport:
- [ ] #52505 <!-- fix alignment of emit_unbox_store copy -->
- [ ] #53373 <!-- fix sysimage-native-code=no option with pkgimages -->
- [ ] #53984 <!-- Profile: fix heap snapshot is valid char check -->
- [ ] #54276 <!-- Fix solve for complex `Hermitian` with non-vanishing
imaginary part on diagonal -->
- [ ] #54669 <!-- Improve error message in inplace transpose -->
- [ ] #54871 <!-- Make warn missed transformations pass optional -->

Contains multiple commits, manual intervention needed:
- [ ] #52854 <!-- Change to streaming out the heap snapshot data -->
- [ ] #53218 <!-- Fix interpreter_exec.jl test -->
- [ ] #53833 <!-- Profile: make heap snapshots viewable in vscode viewer
-->
- [ ] #54303 <!-- LinearAlgebra: improve type-inference in
Symmetric/Hermitian matmul -->
- [ ] #52694 <!-- Reinstate similar for AbstractQ for backward
compatibility -->
- [ ] #54737 <!-- LazyString in interpolated error messages involving
types -->
- [ ] #54738 <!-- serialization: fix relocatability bug -->
- [ ] #55052 <!-- Fix `(l/r)mul!` with `Diagonal`/`Bidiagonal` -->

Non-merged PRs with backport label:
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #55013 <!-- [docs] change docstring to match code -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
- [ ] #41244 <!-- Fix shell `cd` error when working dir has been deleted
-->
@KristofferC KristofferC mentioned this pull request Sep 12, 2024
63 tasks
@KristofferC KristofferC mentioned this pull request Oct 29, 2024
47 tasks
@KristofferC KristofferC removed the backport 1.10 Change should be backported to the 1.10 release label Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Heap snapshots aren't viewable in vscode viewer
5 participants