Performance on specs with many values

[daveliepmann]

My emacs hangs when using vega-view to visualize sufficiently large source data. This is happening from Clojure source; I have not yet tested with elisp or JSON.

I ran into this with a datset of ~150,000 data points I wanted to view in a histogram. I replicated the behavior by extending the README example to 20,000 data points. It works but causes emacs to hang for 10 seconds until the SVG is ready:

{:data {:values (map hash-map
                     (repeat :a)
                     (range 1 20000)
                     (repeat :b)
                     (repeatedly #(* 100 (Math/random))))}
 :mark "bar",
 :width 800
 :height 600
 :encoding {:x {:field :a, :type "ordinal", :axis {"labelAngle" 0}},
            :y {:field :b, :type "quantitative"}}}

30k points takes about 20 seconds. The timings in the example match what I'm seeing with my actual dataset. For comparison, Oz renders all 150,000 in 3 seconds but doesn't hang emacs during that time.

I hope this isn't a fundamental limitation of emacs doing the work.

[jackrusher]

I made a small change that speeds up JSON and elisp plotting by ~6x and Clojure plotting by at least 2x. It looks like I'll have to look deeper into cider to further improve Clojure performance. Also, I might switch this to an async process so emacs doesn't freeze while the work is being done...

[jackrusher]

If you pull the latest version of set vega-view-prefer-png to something truth-y (see README for details), you should get better performance.

While SVG is generally smaller than PNG for charts, when there are tens of thousands of marks the file ends up being quite a bit larger than the equivalent PNG at the scales we're using here. That said, using a 20k value example it still takes ~8 seconds on my machine to get the results from cider and go through the two step process of convert them to JSON, so this isn't as snappy as I hope to make it in the future.

[jackrusher]

It turns out that part of the reason some of the SVGs are so big is because all x/y positions in paths are being written with a very long mantissa (like 13 digits). This, in turn, is coming from a long unaddressed bug in D3.

[volrath]

I also experienced this issue when I was plotting some stuff. I wanted to plot ~230k :values and emacs was not able to handle it. I tried reducing the size to 30k, 20k, 10k points until I was able to get results.

I did a small experiment that I thought it was worth sharing. I moved the Clojure->JSON conversion to the JVM, which improved the speed considerably. I can render ~30k points in ~2-3 seconds, and the original ~230k points in ~15-20 seconds (using SVGs)

Here's a patch for what I changed:

diff --git i/vega-view.el w/vega-view.el
index 8b5ed54..9602735 100644
--- i/vega-view.el
+++ w/vega-view.el
@@ -114,10 +114,12 @@ pass it to `vega-view--json' to display in `VEGA-BUFFER`."
     (setq cider-popup-output-marker (point-marker)))
   (cider-interactive-eval
    ;; in case local printer settings would truncate the output
-   (format "(do (set! *print-length* nil) %s)" clojure-form-string)
+   (format "(clojure.data.json/write-str %s)" clojure-form-string)
    (nrepl-make-response-handler vega-buffer
                                 (lambda (buffer value)
-                                  (vega-view--json (json-encode (parseedn-read-str value)) buffer))
+                                  (vega-view--json
+                                   (substring (replace-regexp-in-string "\\\\\\(.\\|\n\\)" "\\1" value) 1 -1)
+                                   buffer))
                                 (lambda (_buffer out)
                                   (cider-emit-interactive-eval-output out))
                                 (lambda (_buffer err)

Couple of comments:

• The whole substring + replace-regexp-in-string was my hacky way of 'unescaping' the JSON string returned by Clojure. I feel there might be a less horrible better way to do this, but I was trying to get it to work fast :D

• Unfortunately depending on clojure.data.json being present in the classpath isn't ideal. I just wanted to see whether the bottle neck was in the EDN parsing and/or JSON conversion. I didn't profile it further, though.

[jackrusher]

@volrath Yeah, I looked into this approach awhile ago but I'm not willing to depend on a non-default Clojure classpath. :/