Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cider-debug-defn-at-point on a big defn results in "Method code too large!" exception #750

Open
andrewzhurov opened this issue Apr 11, 2022 · 7 comments
Labels

Comments

@andrewzhurov
Copy link

andrewzhurov commented Apr 11, 2022

Please note

As I've been filling this issue I have narrowed down the cause of the exception.
I'll provide the narrowed down version and the real-world version.

Narrowed-down version seems to have the root cause exception.
Real-world version seems to have an exception of not the root cause.

Narrowed-down expected behavior

In Emacs, cider-debug-defun-at-point on the following defn results in defn marked as being debugged.

(def a)
(defn a-fn []
  (let [a1 a a2 a a3 a a4 a a5 a a6 a a7 a a8 a a9 a a10 a
        a11 a a12 a a13 a a14 a a15 a a16 a a17 a a18 a a19 a a20
        a a21 a a22 a a23 a a24 a a25 a a26 a a27 a a28 a a29 a a30
        a a31 a a32 a a33 a a34 a a35 a a36 a a37 a a38 a a39 a a40
        a a41 a a42 a a43 a a44 a a45 a a46 a a47 a a48 a a49 a a50
        a a51 a a52 a a53 a a54 a a55 a a56 a a57 a a58 a a59 a a60
        a a61 a a62 a a63 a a64 a a65 a a66 a a67 a a68 a a69 a a70
        a a71 a a72 a a73 a a74 a a75 a a76 a a77 a a78 a a79 a a80
        a a81 a a82 a a83 a a84 a a85 a a86 a a87 a a88 a a89 a a90
        a a91 a a92 a a93 a a94 a a95 a a96 a a97 a a98 a]))

Narrowed-down actual behavior

Results in "Method code too large!" exception shown in *cider-error* buffer.
(whereas having a98 a binding removed results in no-error behavior)

*cider-error* buffer content:

  Show: Project-Only All 
  Hide: Clojure Java REPL Tooling Duplicates  (0 frames hidden)

2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling temp.clj at (3:1)
   #:clojure.error{:phase :compile-syntax-check,
                   :line 3,
                   :column 1,
                   :source "/home/user1/temp.clj",
                   :symbol fn*}
             Compiler.java: 7132  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java:   38  clojure.lang.Compiler/access$300
             Compiler.java:  596  clojure.lang.Compiler$DefExpr$Parser/parse
             Compiler.java: 7124  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 7112  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 6762  clojure.lang.Compiler/analyze
             Compiler.java: 6137  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 6453  clojure.lang.Compiler$LetExpr$Parser/parse
             Compiler.java: 7124  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 6762  clojure.lang.Compiler/analyze
             Compiler.java: 6137  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 5479  clojure.lang.Compiler$FnMethod/parse
             Compiler.java: 4041  clojure.lang.Compiler$FnExpr/parse
             Compiler.java: 7122  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 7191  clojure.lang.Compiler/eval
             Compiler.java: 7149  clojure.lang.Compiler/eval
                  core.clj: 3215  clojure.core/eval
                  core.clj: 3211  clojure.core/eval
                 debug.clj:  600  cider.nrepl.middleware.debug/instrument-and-eval
                 debug.clj:  595  cider.nrepl.middleware.debug/instrument-and-eval
                  Var.java:  384  clojure.lang.Var/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1990  clojure.core/with-bindings*
                  core.clj: 1990  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  748  java.lang.Thread/run

1. Caused by java.lang.IndexOutOfBoundsException
   Method code too large!

         MethodWriter.java: 2061  clojure.asm.MethodWriter/computeMethodInfoSize
          ClassWriter.java:  457  clojure.asm.ClassWriter/toByteArray
             Compiler.java: 4680  clojure.lang.Compiler$ObjExpr/compile
             Compiler.java: 4118  clojure.lang.Compiler$FnExpr/parse
             Compiler.java: 7122  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java:   38  clojure.lang.Compiler/access$300
             Compiler.java:  596  clojure.lang.Compiler$DefExpr$Parser/parse
             Compiler.java: 7124  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 7112  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 6762  clojure.lang.Compiler/analyze
             Compiler.java: 6137  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 6453  clojure.lang.Compiler$LetExpr$Parser/parse
             Compiler.java: 7124  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 6762  clojure.lang.Compiler/analyze
             Compiler.java: 6137  clojure.lang.Compiler$BodyExpr$Parser/parse
             Compiler.java: 5479  clojure.lang.Compiler$FnMethod/parse
             Compiler.java: 4041  clojure.lang.Compiler$FnExpr/parse
             Compiler.java: 7122  clojure.lang.Compiler/analyzeSeq
             Compiler.java: 6806  clojure.lang.Compiler/analyze
             Compiler.java: 7191  clojure.lang.Compiler/eval
             Compiler.java: 7149  clojure.lang.Compiler/eval
                  core.clj: 3215  clojure.core/eval
                  core.clj: 3211  clojure.core/eval
                 debug.clj:  600  cider.nrepl.middleware.debug/instrument-and-eval
                 debug.clj:  595  cider.nrepl.middleware.debug/instrument-and-eval
                  Var.java:  384  clojure.lang.Var/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  667  clojure.core/apply
                  core.clj: 1990  clojure.core/with-bindings*
                  core.clj: 1990  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  748  java.lang.Thread/run

Narrowed-down steps to reproduce

Create a local file with the code.
Launch an nrepl:

clj -A:test -Sdeps '{:deps {cider/cider-nrepl {:mvn/version "0.28.3"} org.clojure/clojure {:mvn/version "1.11.1"}}}' -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"

Open Emacs and connect to the nrepl.
E.g., via cider-connect-clj.
In my case, cider informs of such environment:

;; CIDER 1.3.0 (Ukraine), nREPL 0.9.0
;; Clojure 1.9.0, Java 1.8.0_161

Navigate to the defn.

Put cursor at the defn symbol, cider-debug-defun-at-point.
*cider-error* is expected to pop up.


Real-world expected behavior

In Emacs cider-debug-defun-at-point on this defn results in defn marked as being debugged.

Real-world actual behavior

Results in *cider-error* buffer with:

  Show: Project-Only All 
  Hide: Clojure Java REPL Tooling Duplicates  (0 frames hidden)

1. Unhandled java.lang.IllegalArgumentException
   No implementation of method: :send of protocol:
   #'nrepl.transport/Transport found for class: nil

          core_deftype.clj:  583  clojure.core/-cache-protocol-fn
          core_deftype.clj:  575  clojure.core/-cache-protocol-fn
             transport.clj:   23  nrepl.transport/eval631/fn/G
                 nrepl.clj:   18  cider.nrepl.middleware.util.nrepl/notify-client
                 nrepl.clj:    8  cider.nrepl.middleware.util.nrepl/notify-client
                 nrepl.clj:   15  cider.nrepl.middleware.util.nrepl/notify-client
                 nrepl.clj:    8  cider.nrepl.middleware.util.nrepl/notify-client
                 debug.clj:  603  cider.nrepl.middleware.debug/instrument-and-eval
                 debug.clj:  595  cider.nrepl.middleware.debug/instrument-and-eval
                  Var.java:  381  clojure.lang.Var/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  657  clojure.core/apply
                  core.clj: 1965  clojure.core/with-bindings*
                  core.clj: 1965  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  243  clojure.main/repl/read-eval-print/fn
                  main.clj:  243  clojure.main/repl/read-eval-print
                  main.clj:  261  clojure.main/repl/fn
                  main.clj:  261  clojure.main/repl
                  main.clj:  177  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  218  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  217  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  748  java.lang.Thread/run

Real-world steps to reproduce the problem

Have the same defn locally.
E.g., by having clj-kondo repo locally and checkouting to the exact commit (latest atm):

git clone https://github.com/clj-kondo/clj-kondo.git
cd clj-kondo
git checkout 9195cc0f0761a2b673816f5296a95380917ff05b

Launch cider-nrepl 0.28.3 (latest atm):

clj -A:test -Sdeps '{:deps {cider/cider-nrepl {:mvn/version "0.28.3"} }}' -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"

Open Emacs and connect to the nrepl.
E.g., via cider-connect-clj.
In my case, cider informs of such environment:

;; CIDER 1.3.0 (Ukraine), nREPL 0.9.0
;; Clojure 1.9.0, Java 1.8.0_161

Navigate to the defn.
E.g., via project-find-regexp RET defn lint-var-usage.
E.g., via find-file at clj-kondo/src/clj_kondo/impl/linters.clj, line 202.

Put cursor at the defn symbol, cider-debug-defun-at-point.
*cider-error* is expected to pop up.

Environment & Version information

cider-nrepl version

0.28.3

cider version

1.3.0 (Ukraine)

nrepl version

0.9.0

Clojure version

1.9.0

Java version

1.8.0_161

Operating system

Guix, linux kernel. Based on this config with some tweaks.

@andrewzhurov andrewzhurov changed the title cider-debug-defn-at-point on a defn with a big body results in exception cider-debug-defn-at-point on a big defn results in "Method code too large!" exception Apr 11, 2022
@vemv
Copy link
Member

vemv commented Apr 11, 2022

Does the debugger work for you with a variety of other defns in other projects?

@andrewzhurov
Copy link
Author

andrewzhurov commented Apr 11, 2022

Does the debugger work for you with a variety of other defns in other projects?

Yes, works as expected (regarding the subject behavior) with most of the defns I've used it with.
I have stumbled only on 2 defns, I think, in clj-kondo that caused that (may be that one of them used another internally).

@vemv
Copy link
Member

vemv commented Apr 11, 2022

Thanks! I now see in your updated issue that it boils down to a Method code too large!

IME hacking on various tools e.g. Eastwood and Cloverage, Method code too large! are often just part of life for dynamic tooling.

https://github.com/clj-kondo/clj-kondo/blob/9195cc0f0761a2b673816f5296a95380917ff05b/src/clj_kondo/impl/linters.clj#L202 is certainly one beast of a function. I also recall that clojure.core/doseq has a 'naive' implementation in terms of emitted bytecode, with no plans to change it.

99% of times, the defns that trigger these issues can be decomposed into smaller ones. I think everyone benefits from that, i.e. human readers too.

A fix might not be impossible (it wasn't for Eastwood) but this is the sort of edge case that is often not worth the effort. Of course, PRs welcome.

Cheers - V

@vemv
Copy link
Member

vemv commented Apr 11, 2022

...As a quick note, perhaps a ->> chain with filter, map, etc might be equivalent to that for but without the bytecode size bloat.

I do avoid for/doseq mainly for other reasons (often the aforementioned chains are neater, and translatable to transducers), but this one might be yet another reason for making for-avoidance a good habit to follow in your own codebases.

@andrewzhurov
Copy link
Author

andrewzhurov commented Apr 12, 2022

On my experience seeing debugger fail on the most juiciest pieces I would benefit having it the most was a major downer.
And seeing misleading error messages added frustration to it.
It worries me that it's not a rare occasion.

I expect from a debugger the exact opposite - it's a solid tool that gets me through tough times.

Having need to refactor code in order to be able to use debugger sounds not ideal, as:

  1. You may not want to have it in the first place. E.g., that giant doseq fn there for a reason and I'm fine with it.
  2. If you do want refactor, you better modify the source, otherwise, if refactorred locally, it'll be a lot of effort spend per each person that does the same. And modifying the source requires more effort, as it should be production-grade.
  3. Refactoring for an outside tooling to work - not project's responsibility in the first place.
    3.1 People won't keep it in mind, so such cases will keep on popping up.
  4. In order to refactor you need to understand code first.

All and all, I'd really hope for debugger to be robust.

@andrewzhurov
Copy link
Author

andrewzhurov commented Apr 12, 2022

I went through my code comments of where debugger resulted in an error - errors differ, but perhaps the root cause is the same.

https://github.com/clj-kondo/clj-kondo/blob/9195cc0f0761a2b673816f5296a95380917ff05b/test/clj_kondo/analysis_test.clj#L1557
(novel error)

Caused by java.lang.RuntimeException
   Unable to resolve symbol: deftest in this context

Same error is given by:
https://github.com/clj-kondo/clj-kondo/blob/9195cc0f0761a2b673816f5296a95380917ff05b/test/clj_kondo/analysis_test.clj#L1619
https://github.com/clj-kondo/clj-kondo/blob/9195cc0f0761a2b673816f5296a95380917ff05b/test/clj_kondo/analysis_test.clj#L1507


https://github.com/clj-kondo/clj-kondo/blob/9195cc0f0761a2b673816f5296a95380917ff05b/src/clj_kondo/impl/linters.clj#L594
(same error as in the issue)

1. Unhandled java.lang.IllegalArgumentException
   No implementation of method: :send of protocol:
   #'nrepl.transport/Transport found for class: nil

@vemv
Copy link
Member

vemv commented Apr 12, 2022

[...] All and all, I'd really hope for debugger to be robust.

That's a fair hope, I'm just informing you that technically is not easy / not a new issue at all (from what can be found in comparable projects) and that the chances of someone spontaneously fixing this are pretty low.

I agree that No implementation of method: :send of protocol: is misleading and might be low-hanging fruit to get fixed.

@bbatsov bbatsov added the bug label Oct 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants