Precompute all possible slider states #29

fonsp · 2021-10-26T15:01:49Z

This draft PR will find all possible combinations of sliders, run those, and save the results to a file. THIS MEANS that you will be able to get most of the coolness of PlutoSliderServer without having to run a server!

These files, and their filenames, exactly match the requests that the sliderserver would get if it was actually running. This means that you can statically serve the folder with generated files, and it respond to the GET requests just like a running slider server.

E.g., currently our sliderserver responds to requests like this:

GET ./staterequest/<notebook hash>/<bond values dict msgpack base64>

which exactly matches the request to access the generated file from our static server!

You need the matching changes to Pluto.jl by @Pangoraw fonsp/Pluto.jl#1600

TODO

Demo video of debugging GUI:

https://youtu.be/gDZ33Lpj8Hw

sethaxen · 2023-05-20T09:23:23Z

This seems to have problems with PlutoUI.MultiSelect.

Precomputing this notebook with

PlutoSliderServer.export_notebook("notebook.jl"; Precompute_enabled=true)

gives the warning

┌ Warning: Notebook cannot be (fully) precomputed
│   report =
│      Precomputed state summary
│      ≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡
│    
│      Total size estimate (based on a radom sample): 0 files, 0 bytes
│    
│      ❌ Group: x
│      ============
│    
│      This group could not be precomputed because:
│    
│        •  The set of possible values for x could not be determined because of an unknown reason:
│           Failed Distributed.RemoteException(1,
│           CapturedException(CompositeException(Any[CapturedException(ErrorException("Error
│           deserializing a remote exception from worker 2\nRemote(original) exception of type
│           MethodError\nRemote stacktrace : "), Any[(#33 at none:0, 1), (iterate at generator.jl:47
│           [inlined], 1), (collect at array.jl:782 [inlined], 1), (make_distributed_serializable at
│           PlutoRunner.jl:1795, 1), (#possible_bond_values#80 at PlutoRunner.jl:1788, 1), (top-level
│           scope at WorkspaceManager.jl:245, 1), (eval at boot.jl:370, 1), (#invokelatest#2 at
│           essentials.jl:816, 1), (invokelatest at essentials.jl:813, 1), (#110 at
│           process_messages.jl:285, 1), (run_work_thunk at process_messages.jl:70, 1), (macro expansion
│           at process_messages.jl:285 [inlined], 1), (#109 at task.jl:514, 1)]),
│           CapturedException(KeyError(PlutoUI [7f904dfe-b85e-4ff6-b463-dae2292396a8]), Any[(getindex at
│           dict.jl:484 [inlined], 1), (macro expansion at lock.jl:267 [inlined], 1),
│           (root_module(key::Base.PkgId) at loading.jl:1688, 1),
│           (deserialize_module(s::Distributed.ClusterSerializer{Sockets.TCPSocket}) at
│           Serialization.jl:996, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:898, 1), (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket})
│           at Serialization.jl:816, 1),
│           (deserialize_datatype(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, full::Bool) at
│           Serialization.jl:1384, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:869, 1), (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket})
│           at Serialization.jl:816, 1),
│           (deserialize_datatype(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, full::Bool) at
│           Serialization.jl:1409, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:869, 1), (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket})
│           at Serialization.jl:816, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:876, 1), (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket})
│           at Serialization.jl:816, 1),
│           ((::Serialization.var"#5#6"{Distributed.ClusterSerializer{Sockets.TCPSocket}})(i::Int64) at
│           Serialization.jl:975, 1),
│           (ntupleany(f::Serialization.var"#5#6"{Distributed.ClusterSerializer{Sockets.TCPSocket}},
│           n::Int64) at ntuple.jl:43, 1),
│           (deserialize_tuple(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, len::Int64) at
│           Serialization.jl:975, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:859, 1), (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket},
│           t::DataType) at Serialization.jl:1499, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:880, 1), (deserialize at Serialization.jl:816 [inlined], 1),
│           (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket},
│           t::Type{CapturedException}) at clusterserialize.jl:223, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:880, 1), (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket},
│           t::DataType) at Serialization.jl:1499, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:880, 1), (deserialize at Serialization.jl:816 [inlined], 1),
│           (deserialize_msg(s::Distributed.ClusterSerializer{Sockets.TCPSocket}) at messages.jl:87, 1),
│           (#invokelatest#2 at essentials.jl:816 [inlined], 1), (invokelatest at essentials.jl:813
│           [inlined], 1), (message_handler_loop(r_stream::Sockets.TCPSocket,
│           w_stream::Sockets.TCPSocket, incoming::Bool) at process_messages.jl:176, 1),
│           (process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket,
│           incoming::Bool) at process_messages.jl:133, 1),
│           ((::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() at task.jl:514,
│           1)])]), Any[(deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket},
│           t::Type{CapturedException}) at clusterserialize.jl:225, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:880, 1), (deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket},
│           t::DataType) at Serialization.jl:1499, 1),
│           (handle_deserialize(s::Distributed.ClusterSerializer{Sockets.TCPSocket}, b::Int32) at
│           Serialization.jl:880, 1), (deserialize at Serialization.jl:816 [inlined], 1),
│           (deserialize_msg(s::Distributed.ClusterSerializer{Sockets.TCPSocket}) at messages.jl:87, 1),
│           (#invokelatest#2 at essentials.jl:816 [inlined], 1), (invokelatest at essentials.jl:813
│           [inlined], 1), (message_handler_loop(r_stream::Sockets.TCPSocket,
│           w_stream::Sockets.TCPSocket, incoming::Bool) at process_messages.jl:176, 1),
│           (process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket,
│           incoming::Bool) at process_messages.jl:133, 1),
│           ((::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() at task.jl:514,
│           1)])).
└ @ PlutoSliderServer.var"../precomputed/index.jl" /home/sethaxen/.julia/packages/PlutoSliderServer/X2T6k/src/precomputed/index.jl:156

sethaxen · 2023-05-31T09:24:08Z

@fonsp, any chance the issue in #29 (comment) might be fixed within the next month? I'm hoping to use this feature to bring this static notebook to life for a workshop.

fonsp · 2023-05-31T19:50:29Z

Hey Seth! Are you happy with using a fork of PlutoUI.jl? In that case, you can change

https://github.com/JuliaPluto/PlutoUI.jl/blob/f74d58ca5f67ab9606a34b8a7ede3e73baf77344/src/MultiCheckBox.jl#L298

to

collect(subarrays((string(i) for i in 1:length(select.options))))

I can't just put this workaround in a PlutoUI release because of the large size of the array when you have lots of options.

Otherwise, this probably won't be fixed until autumn at the earliest :(

Pangoraw · 2023-05-31T20:03:37Z

@fonsp I think the issue is the input of subarrays being an iterator (which is not indexable) not the actual output of subarrays being an iterator. I was just looking at this too (JuliaPluto/PlutoUI.jl#260)!

sethaxen · 2023-06-14T17:07:02Z

@fonsp and @Pangoraw, I have tried both of your solutions but still get the same error.

Two remaining issues I'm seeing (from mlcolab/IntroML.jl#5 (comment)):

PlutoSliderServer still fails to precompute the group for the $g(x)$ multiselect here), even though we're using the solutions recommended in Precompute all possible slider states #29 (comment) and in Fix Bonds.possible_values(::MultiSelect) PlutoUI.jl#260. See the log message at https://github.com/mlcolab/IntroML.jl/actions/runs/5268763176/jobs/9525936481?pr=5#step:6:290.
The first and last values of some of the scrubbables seems to be dropped. i.e. we can scrub to the final value (e.g. $w_0=-2$ here, but that is not reflected in the plot. When the plot is dependent on 2 scrubbables (e.g. $w$ here), if we scrub one to the end and then scrub the other, the plot does not update.
Screencast from 06-14-2023 06:57:09 PM.webm

sethaxen · 2023-06-14T17:08:50Z

Here is the line from my branch of PlutoUI that is being used: https://github.com/sethaxen/PlutoUI.jl/blob/10e03038a9caa9159de5cc749561997e4c3b05d3/src/MultiCheckBox.jl#L298

lucaferranti · 2023-06-18T12:16:04Z

I was trying out this branch, but I am not getting it to work even for the following simple case.

after having generated the html with

julia> PlutoSliderServer.export_directory(; Precompute_enabled=true, Precompute_max_filesize_per_group=1e9)

I get

www_screencapture_com_2023-6-18_15_16.mp4

so it does update the value next to the slider, but the output of the other cell using n remains unchanged.

Am I doing something wrong?

For reference

julia> versioninfo()
Julia Version 1.9.1
Commit 147bdf428cd (2023-06-07 08:27 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × 12th Gen Intel(R) Core(TM) i7-1255U
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, alderlake)
  Threads: 1 on 12 virtual cores
Environment:
  JULIA_EDITOR = code
  JULIA_IMAGE_THREADS = 1

(pluto_sandbox) pkg> st
Status `~/work/teaching/pluto_sandbox/Project.toml`
  [c3e4b0f8] Pluto v0.19.26
  [2fc8631c] PlutoSliderServer v0.3.25 `https://github.com/JuliaPluto/PlutoSliderServer.jl.git#static-export-1`
  [7f904dfe] PlutoUI v0.7.51

sethaxen · 2023-06-18T13:11:34Z

@lucaferranti at least for me, when I loaded the generated notebook on my machine it was static, but when I deployed it with GitHub Pages, it became dynamic.

fonsp · 2023-06-20T13:35:26Z

@sethaxen Are you sure that you are using your PlutoUI fork? The manifest still refers to the latest version: https://github.com/mlcolab/IntroML.jl/pull/5/files#diff-de5aebd2f07be72d79aa0280756882a9be1d2711d41daeaa375da1e8792dd60fR2717

Use Pluto.activate_notebook_environment to fix this, and use AbstractPlutoDingetjes.possible_values inside your notebook to verify that it works.

fonsp · 2023-06-20T13:39:16Z

@lucaferranti It only works when hosted on a web server, like LiveServer.jl or deno run --allow-net --allow-read https://deno.land/std@0.192.0/http/file_server.ts or GitHub Pages (like Seth suggested), nginx, netlify, etc. Try to get familiar with the Chrome Dev Tools (Network tab) to be able to diagnose network issues! You should see that the requests are being blocked (because you are not allowed to request a file:// URL dynamically).

sethaxen · 2023-06-20T17:47:25Z

@sethaxen Are you sure that you are using your PlutoUI fork? The manifest still refers to the latest version: https://github.com/mlcolab/IntroML.jl/pull/5/files#diff-de5aebd2f07be72d79aa0280756882a9be1d2711d41daeaa375da1e8792dd60fR2717

Use Pluto.activate_notebook_environment to fix this, and use AbstractPlutoDingetjes.possible_values inside your notebook to verify that it works.

Thanks, that worked perfectly! I still see the other issue where the plots are not responsive to the first and last values in the scrubbable, see e.g. https://mlcolab.github.io/IntroML.jl/previews/PR6/supervised_learning.html#cd49e0a5-4120-481a-965e-72e7bdaf867c

lucaferranti · 2023-06-25T08:08:16Z

This currently errors if some notebooks are excluded for PlutoSliderServer.

Steps to reproduce

grab a simple pluto notebook, e.g. this one
add the following PlutoDeployment.toml

[SliderServer]
exclude = ["pluto_sandbox.jl"] # or whatever your notebook is called

Run

PlutoSliderServer.export_directory(; Precompute_enabled=true)

This will produce the error

┌ Info: 
│   ◐ [1/1] Launching...
└   s.path = "pluto_sandbox.jl"

┌ Info: Shutting down notebook process
└   s.path = "pluto_sandbox.jl"

ERROR: type FinishedNotebook has no field bond_connections
Stacktrace:
  [1] getproperty
    @ ./Base.jl:37 [inlined]
  [2] generate_precomputed_staterequests(notebook_session::PlutoSliderServer.var"../Types.jl".NotebookSession{String, String, PlutoSliderServer.var"../Types.jl".FinishedNotebook}; settings::PlutoSliderServer.var"../Configuration.jl".PlutoDeploySettings, pluto_session::Pluto.ServerSession, output_dir::String)
    @ PlutoSliderServer.var"../precomputed/index.jl" ~/.julia/packages/PlutoSliderServer/vYCzf/src/precomputed/index.jl:130
  [3] process(s::PlutoSliderServer.var"../Types.jl".NotebookSession{Nothing, String, Nothing}; server_session::Pluto.ServerSession, settings::PlutoSliderServer.var"../Configuration.jl".PlutoDeploySettings, output_dir::String, start_dir::String, progress::String)

lucaferranti · 2023-06-25T08:14:34Z

src/precomputed/index.jl

+
+    sesh = notebook_session
+    run = sesh.run
+    connections = run.bond_connections


this causes problems if I have some notebooks under PlutoSliderServer_exclude (see this comment), because in that case run will be FinishedNotebook.

Myabe one dirty fix could be to have before this

run isa FinishedNotebook && return

?

cleaner might be not to call this in the first place for notebooks under PlutoSliderServer_exclude.

gdalle · 2023-06-29T10:14:29Z

Hey Fons and friends! Not sure I can help, just wanted to cheer from the sidelines because this looks awesome

eguar11011 · 2024-10-24T20:55:12Z

I don’t quite understand how to precompute my sliders. What I did was run:

using PlutoSliderServer
PlutoSliderServer.export_notebook("path/to/notebook.jl";  Precompute_enabled=true)

And I had these errors:

ERROR: InvalidKeyError: invalid key Precompute_enabled, possible keys are: SliderServer_enabled, SliderServer_exclude, SliderServer_port, SliderServer_host, SliderServer_watch_dir, SliderServer_serve_static_export_folder, SliderServer_simulated_lag, SliderServer_cache_control... please check documentation for other valid keys

behinger · 2024-10-25T07:33:47Z

did you check out the correct branch?
e.g. via

]add PlutoSliderServer#static-export-1

eguar11011 · 2024-10-25T17:24:52Z

@behinger
I don't understand, would I have to download sliderserver but from another branch?
Meaning, not with using PlutoSliderServer?

behinger · 2024-10-25T20:31:32Z

using is fine, but you first need to install it from a different branch, e.g. https://stackoverflow.com/questions/62837941/julia-how-to-change-a-packages-branch

hope that helps!

eguar11011 · 2024-10-28T21:31:02Z

@behinger
Thank you very much; I now understand how to download it from a different branch and generate the precomputed files.

I already have the files, but when I upload them to the site, there’s still no interactivity. Is there anything else I should keep in mind?

hello

ba8a012

This was referenced Oct 26, 2021

API for widgets to return possible bind values fonsp/Pluto.jl#1600

Merged

slider and checkbox export possible values JuliaPluto/PlutoUI.jl#147

Closed

Merge branch 'main' into static-export-1

2e2ebe5

This was referenced Oct 28, 2021

🐣 New package AbstractPlutoDingetjes.jl for intial bond values, bond value transformation and more fonsp/Pluto.jl#1612

Merged

🐣 Transforming values: AbstractPlutoDingetjes integration JuliaPluto/PlutoUI.jl#148

Merged

fonsp changed the title ~~Statically export all possible slider states~~ Precompute all possible slider states Nov 4, 2021

fonsp mentioned this pull request Nov 27, 2021

Precompute all possible binder states #37

Open

fonsp linked an issue Nov 27, 2021 that may be closed by this pull request

Precompute all possible binder states #37

Open

fonsp mentioned this pull request Nov 29, 2021

Precomputed slider server support fonsp/Pluto.jl#1703

Merged

fonsp added 20 commits November 29, 2021 19:44

Merge branch 'main' into static-export-1

598ac19

Update Configuration.jl

53f72d6

merge part 2

31dcfba

merge part 3

4f98d91

Update Project.toml

df6881b

Update Actions.jl

1004343

tests

28908d3

silly rand

072e9a2

rng for ProductIterator

81af871

Merge branch 'main' into static-export-1

b9e8c6a

Terminal UI to debug combinations

8d4054c

Live debugging mode

984a4d7

fix tests

af8de51

Judgement day

001d96d

Match fonsp/Pluto.jl#1703

ff13848

whitespace

f72fcaa

Update Project.toml

3a61225

hopla

207a21c

Merge branch 'main' into static-export-1

b55ade9

Update Actions.jl

996b13a

ctrekker added 2 commits June 13, 2023 10:36

Merge branch 'main' into static-export-1

ce75bb1

Fix #117 by updating

d8f3a97

sethaxen mentioned this pull request Jun 14, 2023

Precompute notebook with PlutoSliderServer mlcolab/IntroML.jl#5

Merged

lucaferranti mentioned this pull request Jun 20, 2023

🌟 New layout, new system mitmath/computational-thinking#124

Merged

sethaxen mentioned this pull request Jun 20, 2023

Add custom PlutoUI as notebook dep mlcolab/IntroML.jl#6

Merged

lucaferranti reviewed Jun 25, 2023

View reviewed changes

fonsp assigned ctrekker and lucaferranti Jul 4, 2023

Option to precompute only if all groups can be

83194eb

fonsp mentioned this pull request Sep 18, 2023

Clock: repeat option JuliaPluto/PlutoUI.jl#271

Merged

fonsp mentioned this pull request Oct 30, 2023

Export HTML with baked bonds fonsp/Pluto.jl#16

Closed

fonsp mentioned this pull request Oct 25, 2024

Support for Pre-calculated Interactive Sliders on Static Pages #149

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Precompute all possible slider states #29

Precompute all possible slider states #29

fonsp commented Oct 26, 2021 •

edited

Loading

sethaxen commented May 20, 2023

sethaxen commented May 31, 2023

fonsp commented May 31, 2023

Pangoraw commented May 31, 2023

sethaxen commented Jun 14, 2023

sethaxen commented Jun 14, 2023

lucaferranti commented Jun 18, 2023 •

edited

Loading

sethaxen commented Jun 18, 2023

fonsp commented Jun 20, 2023

fonsp commented Jun 20, 2023

sethaxen commented Jun 20, 2023

lucaferranti commented Jun 25, 2023 •

edited

Loading

lucaferranti Jun 25, 2023 •

edited

Loading

gdalle commented Jun 29, 2023

eguar11011 commented Oct 24, 2024 •

edited

Loading

behinger commented Oct 25, 2024 •

edited

Loading

eguar11011 commented Oct 25, 2024

behinger commented Oct 25, 2024

eguar11011 commented Oct 28, 2024

Precompute all possible slider states #29

Are you sure you want to change the base?

Precompute all possible slider states #29

Conversation

fonsp commented Oct 26, 2021 • edited Loading

sethaxen commented May 20, 2023

sethaxen commented May 31, 2023

fonsp commented May 31, 2023

Pangoraw commented May 31, 2023

sethaxen commented Jun 14, 2023

sethaxen commented Jun 14, 2023

lucaferranti commented Jun 18, 2023 • edited Loading

sethaxen commented Jun 18, 2023

fonsp commented Jun 20, 2023

fonsp commented Jun 20, 2023

sethaxen commented Jun 20, 2023

lucaferranti commented Jun 25, 2023 • edited Loading

Steps to reproduce

lucaferranti Jun 25, 2023 • edited Loading

Choose a reason for hiding this comment

gdalle commented Jun 29, 2023

eguar11011 commented Oct 24, 2024 • edited Loading

behinger commented Oct 25, 2024 • edited Loading

eguar11011 commented Oct 25, 2024

behinger commented Oct 25, 2024

eguar11011 commented Oct 28, 2024

fonsp commented Oct 26, 2021 •

edited

Loading

lucaferranti commented Jun 18, 2023 •

edited

Loading

lucaferranti commented Jun 25, 2023 •

edited

Loading

lucaferranti Jun 25, 2023 •

edited

Loading

eguar11011 commented Oct 24, 2024 •

edited

Loading

behinger commented Oct 25, 2024 •

edited

Loading