Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Unable to load global callgraph into memory #138

Closed
TorNATO-PRO opened this issue May 26, 2022 · 4 comments
Closed

[BUG] Unable to load global callgraph into memory #138

TorNATO-PRO opened this issue May 26, 2022 · 4 comments
Labels
bug Something isn't working

Comments

@TorNATO-PRO
Copy link

TorNATO-PRO commented May 26, 2022 •
edited
Loading

Description of bug
I am trying to load global callgraphs of programs represented in Graphviz dot format into memory. Networkx is able to handle this whereas Graphs.jl/GraphsIO.jl is not.

How to reproduce
Detail steps to reproduce the behavior.

  1. Download the awk program's global callgraph, from here.
  2. Open a Julia interpreter, type the following - make sure that you have the packages Graphs, GraphIO, ParserCombinator added to your global dependencies or local dependencies.
> using Graphs
> using GraphIO
> using ParserCombinator

> export sample_graph

> sample_graph() = loadgraph("path-to-awk", "code", GraphIO.DOT.DOTFormat())

> sample_graph()

Expected behavior
I would expect it to be loaded into memory, as it is in networkx.

Actual behavior

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x283fb83 -- subtype_unionall at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:796
in expression starting at REPL[3]:1
subtype_unionall at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:796
subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1260
subtype_tuple_tail at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1079 [inlined]
subtype_tuple at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1158 [inlined]
subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1299
exists_subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1395 [inlined]
forall_exists_subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1423
jl_subtype_env at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1878
jl_subtype_env at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1848 [inlined]
jl_subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1914 [inlined]
subtype_tuple_tail at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1076 [inlined]
subtype_tuple at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1158 [inlined]
subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1299
subtype_unionall at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:774
subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1260
exists_subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1395 [inlined]
forall_exists_subtype at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1423
jl_subtype_env at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1878
jl_subtype_env at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:1848 [inlined]
jl_type_intersection_env_s at /cygdrive/c/buildbot/worker/package_win64/build/src\subtype.c:3394
jl_typemap_intersection_node_visitor at /cygdrive/c/buildbot/worker/package_win64/build/src\typemap.c:459
jl_typemap_intersection_visitor at /cygdrive/c/buildbot/worker/package_win64/build/src\typemap.c:626
ml_matches at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2761
_gf_invoke_lookup at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2440
jl_mt_assoc_by_type at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:1195
jl_lookup_generic_ at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2400
jl_apply_generic at /cygdrive/c/buildbot/worker/package_win64/build/src\gf.c:2425
typed_vcat at .\abstractarray.jl:1619
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:713
vcat at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\SparseArrays\src\sparsevector.jl:1123
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:713
edges at C:\Users\waltz\.julia\packages\ParserCombinator\35p4P\src\dot\DOT.jl:234
_dot_read_one_graph at C:\Users\waltz\.julia\packages\GraphIO\x37Ru\src\DOT\Dot.jl:53
loaddot at C:\Users\waltz\.julia\packages\GraphIO\x37Ru\src\DOT\Dot.jl:68
loadgraph at C:\Users\waltz\.julia\packages\GraphIO\x37Ru\src\DOT\Dot.jl:83 [inlined]
#120 at C:\Users\waltz\.julia\packages\Graphs\zrMoC\src\persistence\common.jl:15 [inlined]
#open#355 at .\io.jl:330
open at .\io.jl:328 [inlined]
loadgraph at C:\Users\waltz\.julia\packages\Graphs\zrMoC\src\persistence\common.jl:14 [inlined]
sample_graph at C:\Users\waltz\Documents\PolymorphicGNN\src\callgraph_loader.jl:7
unknown function (ip: 000000005f7f4823)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
do_call at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:126
eval_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:215
eval_stmt_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:166 [inlined]
eval_body at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:583
jl_interpret_toplevel_thunk at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:731
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:885
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:830
jl_toplevel_eval at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:894 [inlined]
jl_toplevel_eval_in at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:944
eval at .\boot.jl:373 [inlined]
eval_user_input at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:150
repl_backend_loop at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:246start_repl_backend at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:231
#run_repl#47 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:364
run_repl at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:351
#936 at .\client.jl:394
jfptr_YY.936_22652.clone_1 at C:\Users\waltz\AppData\Local\Programs\Julia-1.7.3\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757
#invokelatest#2 at .\essentials.jl:716 [inlined]
invokelatest at .\essentials.jl:714 [inlined]
run_main_repl at .\client.jl:379
exec_options at .\client.jl:309
_start at .\client.jl:495
jfptr__start_34800.clone_1 at C:\Users\waltz\AppData\Local\Programs\Julia-1.7.3\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
true_main at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:559
jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:701
mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42
BaseThreadInitThunk at C:\Windows\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
Allocations: 31525714 (Pool: 31523944; Big: 1770); GC: 27

Code demonstrating bug
Demonstrated above.

Version information

Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 7 5800H with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, znver3)

     Project PolymorphicGNN v0.1.0
      Status `C:\Users\####\Documents\PolymorphicGNN\Project.toml`
  [86223c79] Graphs v1.7.0

Additional context
Loading a global callgraph of the awk program into memory. I wouldn't think that memory would be a problem since there are not so many nodes.
@TorNATO-PRO TorNATO-PRO added the bug Something isn't working label May 26, 2022
@etiennedeg
Copy link
Member

This is a GraphsIO.jl bug, not a Graphs.jl. I can reproduce it only when the file is encoded with LF (and not CRLF, otherwise I get a "cannot parse" error). The offender seems to be ParserCombinator.jl which generates an enormous object, and Julia, which can't handle it properly. I will file issues.

@etiennedeg etiennedeg mentioned this issue May 30, 2022
Open
@etiennedeg
Copy link
Member

ParseCombinator.jl seems also to be unmaintained, we should maybe find an alternative to it...

@KristofferC
Copy link

andrewcooke/ParserCombinator.jl#39 might fix it.

@TorNATO-PRO
Copy link
Author

@KristofferC Thanks so much, it was a problem with ParserCombinator.jl and now it works! Using reduce() solved the problem.

For others that might have a similar problem, here is what you do. Make sure that you are using the ParserCombinator.jl >=v2.2.1 from GitHub found at commit 5d477e1b563c3ea47eb6f430a810dcb63997a39c with UUIDfae87a5f-d1ad-5cf0-8f61-c941e1580b46, later versions will probably work too. Here is how I did it:

(@v1.7) pkg> activate .
  Activating project at `C:\Users\waltz\Documents\PolymorphicGNN`

(PolymorphicGNN) pkg> add https://github.com/andrewcooke/ParserCombinator.jl.git
     Cloning git-repo `https://github.com/andrewcooke/ParserCombinator.jl.git`
    Updating git-repo `https://github.com/andrewcooke/ParserCombinator.jl.git`
    Updating registry at `C:\Users\waltz\.julia\registries\General.toml`
   Resolving package versions...
    Updating `C:\Users\waltz\Documents\PolymorphicGNN\Project.toml`
  [fae87a5f] + ParserCombinator v2.1.1 `https://github.com/andrewcooke/ParserCombinator.jl.git#master`
    Updating `C:\Users\waltz\Documents\PolymorphicGNN\Manifest.toml`
  [15f4f7f2] + AutoHashEquals v0.2.0
  [fae87a5f] + ParserCombinator v2.1.1 `https://github.com/andrewcooke/ParserCombinator.jl.git#master`
Precompiling project...
  2 dependencies successfully precompiled in 4 seconds (142 already precompiled)

> using Graphs
> using GraphIO
> using ParserCombinator

> sample_graph() = loadgraph("path-to-awk", "code", GraphIO.DOT.DOTFormat())

> sample_graph()

Have a good one, and thanks everyone who is working on this project :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@KristofferC @etiennedeg @TorNATO-PRO