Skip to content
This repository has been archived by the owner on Dec 27, 2022. It is now read-only.

Live reloading chokes when there are a lot of quick updates #603

Closed
rjcorwin opened this issue Jul 14, 2017 · 19 comments
Closed

Live reloading chokes when there are a lot of quick updates #603

rjcorwin opened this issue Jul 14, 2017 · 19 comments
Labels
bug
Milestone
0.7.4

Comments

@rjcorwin
Copy link

rjcorwin commented Jul 14, 2017 •
edited
Loading

Tonight @dwblair and I created a google docs like experience by creating an HTML doc with a contenteditable div with some Javascript that commits when changes are detected. A Google Doc like experience as in a Google Doc with permission of "Anyone with URL can read" granted this HTML doc is being published by a Beaker browser and someone else is on it's dat URL with live reloading turned on. See this gist for an example https://gist.github.com/rjsteinert/966525c209402320931dfd8b04dfd997. I would have published to hasbase or published the dat repo on an "always on raspberry pi" but I haven't gotten that far yet!

We some some interesting behavior. It worked ok if we cranked up debouncing but turn it down and we get some funkiness. The strangest behavior by far was not seeing an update on my friend @dwblair's browser until I hit the space bar on my end. As in... Type a bunch of things, wait for the change to propogate to friend's browser, nothing happens, hit space bar, instantly reloads for friend!

We thought it might be our code having the issue but we see the file changing correctly on the file system on the host machine. I realize now we should have captured the whole experiment on a screen recording but we didn't think of it at the time. Another day we will! Cheers.
@rjcorwin rjcorwin changed the title Live editing example with issues Live editing like Google docs between two Beaker Browsers example (with issues) Jul 14, 2017
@rjcorwin rjcorwin changed the title Live editing like Google docs between two Beaker Browsers example (with issues) Live editing like Google Docs between two Beaker Browsers example (with issues) Jul 14, 2017
@rjcorwin
Copy link
Author

Hashbase'd!

@pfrazee pfrazee changed the title Live editing like Google Docs between two Beaker Browsers example (with issues) Live reloading chokes when there are a lot of quick updates Jul 14, 2017
@pfrazee pfrazee added the bug label Jul 14, 2017
@pfrazee
Copy link
Member

pfrazee commented Jul 14, 2017

Yeah I just reproduced the issue with live reloading. I've added it to my todos!

@pfrazee pfrazee added this to the 0.7.4 milestone Jul 14, 2017
@rjcorwin
Copy link
Author

Thank for taking a look @pfrazee! For what it's worth, @dwblair and I wonder if it could be an issue on the dat level but have not had the time to test that.

@pfrazee
Copy link
Member

pfrazee commented Jul 14, 2017 via email

@dwblair
Copy link

dwblair commented Jul 14, 2017 •
edited
Loading

@pfrazee -- thank you so much for this, and for all the incredible work on Beaker. I'd been talking up Beaker and the decentralized web a lot -- but after @rjsteinert lead me last night through a 'decentralized live publishing' process', I'm 'mind blown' at the possibilities.

Do you have a sense for what the 'speed limit' on this sort of 'live updating' might be? Can dat handle fast data rates / streaming? Wondering now about 'live sensor data' on the p2p web ...

@pfrazee
Copy link
Member

pfrazee commented Jul 14, 2017

@dwblair thanks!

That's an interesting question to unpack. You have 3 main components that will affect performance:

  • Writing the updates, which (for dat) includes hashing and signing the data
  • Distribution over the network, which will include peer-connection time
  • Reading the updates, which includes hashing and checking the signatures

I dont have any hard numbers atm, nor have I experimented much with very large datasets, but anecdotally I can say it has handed live streaming of video (recording a screen) just fine. The crypto operations add a fixed overhead but that's probably not that different than SSL. The other most significant overhead is peer-discovery, since that requires a lookup.

@dwblair
Copy link

dwblair commented Jul 14, 2017

@pfrazee --Thanks for the thoughtful reply, and parsing out all of the contributions to performance. The ability to live stream video indicates that the kind of sensor data exchange we're considering should be well-supported -- great to hear. Still learning how peer-discovery works, but I'm really amazed at how fast it is already!

@rjsteinert and I are now on a (p2p, Web RTC) call right now talking about where to head with this 'decentralized google docs for dat' idea ... and wondering about what's possible inside dat archives regarding distributing write permissions / keys -- please join us if you like! https://talky.io/beakerbrowser

@pfrazee
Copy link
Member

pfrazee commented Jul 15, 2017

@rjsteinert @dwblair the more I think about it, the more I think your WYSIWYG Wiki demo is a pretty perfect demo for beaker. (WYSIWYWiki?) It shows the ability to write pages with really minimal effort, and those formatting controls you added were so nice

@dwblair
Copy link

dwblair commented Jul 16, 2017

@pfrazee @rjsteinert -- Saw this yesterday quickly, then found myself on a hike testing 'WYSIWYWiki' out-loud to myself. It's fun to say.

It does seem like using Beaker for wiki / collaborative document editing could be a nice, accessible way for newcomers to develop an intuition for how decentralized file systems work / can work. (It's already been performing that function for us!) And with dat's new mulit-writer protocol coming online, it could be a nice way to demonstrate various approaches to granting editing permissions, versioning, and resolving file conflicts.

I like the point that you've emphasized elsewhere, @pfrazee, that (paraphrasing) a blockchain isn't always necessary to handle decentralized collaborative file editing. One could almost imagine an ecosystem of Beaker wiki apps that take a range of approaches to handling decentralized editing -- from the simple, old-school "I will edit the doc on Monday. Then you edit it on Tuesday", all the way up to "blockchain".

  • Random idea re: write permissions -- I could imagine sharing a wiki document with folks in Beaker, and wanting to give 'granular permissions' to editing the wiki. E.g. There are some sentences / sections that I might want to lock down, to prevent others from editing them, while there would be other wiki sections that I would be happy to let the world edit. Maybe that would required having different wiki sections associated with their own dat archives, and combined via a script into one document?

  • It'd be really nice to visualize the 'graph' of document versions / forks in some way. Being able to understand the relationship of edits in a document that is editing by multiple agents offline / asynchronously feels like it is a challenge for most people -- and whatever helps one understand collaborative, decentralized document editing will help in understanding the decentralized web.

  • In addition to his wiki idea, @rjsteinert (who has been working a lot with Docker recently) was starting to wonder in conversation about using something like the dat / Beaker model to allow programs to run in secure 'containers' in a distributed, safe way -- which was fun to think about. But for now, on a basic level: with a dat multiwriter protocol, it might be cool to be able to constrain the size of a dat archive in advance, so that one could e.g. set up a Raspberry Pi and open it up the world and say "have at it -- edit to your heart's content" without worrying about the resultant dat archive eating all the available disk space.

Thanks again for all the awesome ideation, and accessible explanation!

@rjcorwin
Copy link
Author

Hi @dwblair @pfrazee - Just coming back to this thread now because I have an update but I haven't read your last two comments, I'm in the middle of building a PoC for Tangerine and Beaker with Polymer Elements + CKeditor for advanced survey building. I updated the beaker wiki concept to use CKEditor because Aloha is actively encouraging itself to not be used. Turns out CKEditor has nice inline functionality these days.

See the repo with links to live examples: https://github.com/rjsteinert/beaker-wiki

@pfrazee
Copy link
Member

pfrazee commented Jul 17, 2017 via email

@rjcorwin
Copy link
Author

@pfrazee We did some thinking tonight how one might approach publishing a form survey and then have folks fill it out in their own fork which causes the form data to write to disk, and then the person who created the form collect that data. Right now, in order for the person who created the form to collect data, folks who fill out the form in their own fork would have to send the dat url to the person who created the URL by way of email or some other method. Perhaps there could be a way for dat archives to know who forked them so the app in the root archive could open all children archives and collect the data. This reminds me of how you can see forks of a repo on Github, yet in many cases it would only be desirable for security reasons for the owner of the root archive to be able to know the children so people who fork and fill out the form app can't collect the data of others. We have lots of ideas on how to accomplish this but lack a foundational knowledge of dat to really know what we are talking/thinking about :P.

@pfrazee
Copy link
Member

pfrazee commented Jul 18, 2017 via email

@rjcorwin
Copy link
Author

rjcorwin commented Jul 19, 2017 •
edited
Loading

"outbox method" is a great way to describe that model. I'm also reminded of "pull request" where the owner of the fork requests to the parent they pull something from them (in there outbox). If dat had some permissive messaging protocol between Dats then you could load up a Dat in Beaker, see the requests, approve them, then the application deals with what is sent over wire. That might be saving that data directly into the current Dat or perhaps saving that data into another Dat as to keep the messages from other request senders. The application dealing with the messages is nice because it can decide how to deal with conflicts (as you have mentioned to me).

Use case I'm very much involved in:
Decentralized Forms where you want the Form to be public but the responses private to the owner of the Form.

Perhaps there is an existing decentralized messaging protocol that could help mediate these messages? WebRTC perhaps? As you know that requires a signaling server but perhaps Dat can provide those signals. That's easier to do when things are live, storing a request seems like a nonstarter due to DDoS potential of filling up hard drives (somewhere).

@rjcorwin
Copy link
Author

Coming back around to the root of this issue, if dat could handle signaling for connecting two BeakerBrowsers over WebRTC, then we may have this performance issue of quick updates nailed down.

@pfrazee
Copy link
Member

pfrazee commented Jul 19, 2017

If dat had some permissive messaging protocol between Dats then you could load up a Dat in Beaker, see the requests, approve them, then the application deals with what is sent over wire

The 'permissive' part is difficult because you have to handle spam.

Coming back around to the root of this issue, if dat could handle signaling for connecting two BeakerBrowsers over WebRTC, then we may have this performance issue of quick updates nailed down.

It's possible that we can solve WebRTC signaling with a DHT, but it's more likely that we'll use a decentralized service model, because you also need hole-punching, fallback proxies, and key-distribution. We'll probably introduce a set of specs and implement in hashbase.

All that said, the downside of using WebRTC to send survey responses is, it requires the device that created the survey to act like a server: it has to be online, and it has to accept the incoming connections. The 'being online' part is a non-starter IMO if we want this tech to "just work" for everybody.

So I think we have to look for some other solutions.

@pfrazee
Copy link
Member

pfrazee commented Jul 20, 2017

Ok I managed to solve the issue. You should see all changes cause reloads now

@dwblair
Copy link

dwblair commented Jul 20, 2017 via email

@rjcorwin
Copy link
Author

@pfrazee Don and I have built from master and are about to test out the live editing now. Join us if you like https://talky.io/beaker

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
0.7.4
Development

No branches or pull requests
3 participants
@rjcorwin @dwblair @pfrazee