No longer able to connect local Livebook to remote Fly.io node

[zachallaun]

Note: I posted about this on ElixirForum, but enough time has gone by that I think it makes sense to post here as well. (Perhaps it made sense at the beginning!) This is a lightly-edited version of that post.

---

Hi friends,

I wanted to solicit some advice on how to debug an attached node connection issue I’ve started to experience with Livebook.

Context & setup

This setup was previously working. In fact, I posted on ElixirForum to share how I got it set up and the script I was using to start the Livebook server with the correct flags. Sometime in the last couple of weeks, this setup has stopped working. I’ll describe exactly what I’m seeing a bit further down.

To get it out of the way, here are the versions of everything I’m using:

• Latest Livebook - 0.8.1 (though also tried with 0.7.2, 0.7.1, and 0.7.1)
• Latest Elixir/OTP - 1.14.3 / 25.2.1 (using same versions both locally and on server)

Other bits and bobs:

• Connection is through a wireguard tunnel between me and the server, as described in this article on fly.io.
• The node name and cookie is being set as expected.
• Livebook is being started with -proto_dist inet6_tcp.
• I am able to connect to the node through iex - more on that below.

Description of issue

After launching the Livebook server, opening a notebook, and attempting to connect to the attached node, Livebook hangs until the terminal eventually reports this error:

12:35:20.007 [error] GenServer #PID<0.592.0> terminating
** (CaseClauseError) no case clause matching: {:badrpc, :nodedown}
    (livebook 0.8.1) lib/livebook/runtime/erl_dist.ex:74: anonymous fn/3 in Livebook.Runtime.ErlDist.load_required_modules/1
    (elixir 1.14.3) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (livebook 0.8.1) lib/livebook/runtime/erl_dist.ex:71: Livebook.Runtime.ErlDist.load_required_modules/1
    (livebook 0.8.1) lib/livebook/runtime/erl_dist.ex:60: Livebook.Runtime.ErlDist.initialize/2
    (livebook 0.8.1) lib/livebook/runtime/attached.ex:45: Livebook.Runtime.Attached.connect/1
    (livebook 0.8.1) lib/livebook/session.ex:1519: Livebook.Session.handle_action/2
    (elixir 1.14.3) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (livebook 0.8.1) lib/livebook/session.ex:878: Livebook.Session.handle_cast/2
Last message: {:"$gen_cast", {:queue_cell_evaluation, #PID<0.594.0>, "setup"}}

Critically, I am able to connect to the node manually through IEx. I can start a local iex using the following command, connect to the remote node, and run some code through Node.spawn(server, ...).

# ERL_AFLAGS="-proto_dist inet6_tcp" iex --name local@127.0.0.1 --cookie my-shared-cookie
iex> server = :"my-server-node-name@some:ipv6:address"

iex> Node.connect(server)
true

# Test module that prints "hello"
iex> Node.spawn(server, Hello, :world, [])
hello
#PID<...>

I’m not really sure where to go from here. Any suggestions would be greatly appreciated!

[josevalim]

I guess the biggest question is if you are sure Livebook is being started with inet6. Can you connect to Livebook itself through IEx?

[josevalim]

I doubt the module is large enough too cause it to fail? Can you load this module into a regular iex session you connected remotely?

[josevalim]

Or what happens when you connect to the remote node with IEx and try to load the module? You just need to add Livebook ebin directory to your iex path to access the modules.

[zachallaun]

Or what happens when you connect to the remote node with IEx and try to load the module? You just need to add Livebook ebin directory to your iex path to access the modules.

Great idea, will give it a shot, one sec.

[zachallaun]

This replicates the issue and it doesn't seem to depend on file size. I can connect to the remote node, but when I run an :rpc.call(node, :code, :load_binary, ...) I get the same network backoff regardless of module size (tried loading a larger one and one that was only a couple of lines).

Interestingly, if I try a different call, like :rpc.call(node, :erlang, :system_info, [:otp_release]), I get the result back.

[zachallaun]

Okay, so once the data being sent is over a certain size, things start failing. This is officially not a Livebook problem, but a me- or Fly.io- problem. =)

[zachallaun]

Everything's working again!

This had nothing at all to do with Livebook, but instead was the result of a change in WSL2 that caused packets over a certain size to fail over Wireguard. Thanks for @josevalim's help, I was able to go "lower and lower" until I finally had a pretty easy-to-run test:

$ ping -s1168 -c1 -w1 $REMOTE_NODE_IP
PING [ipv6] 1168 data bytes
1176 bytes from [ipv6]: icmp_seq=1 ttl=62 time=41.7 ms

--- [ipv6] ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 41.734/41.734/41.734/0.000 ms

$ ping -s1169 -c1 -w1 $REMOTE_NODE_IP
PING [ipv6] 1169 data bytes

--- [ipv6] ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

The problem appeared to be that the MTU was set lower than the amount required by Wireguard. sudo ip link set dev eth0 mtu 1500 sets it to a sufficient size and everything starts to work.