Reload changed tiddlers from disc

[dufferzafar]

Is there a faster way to load changed tiddlers from disk into the browser without restarting the Node server?

Google Groups post didn't mention any solution but it is 2 years old. Has there been any development on this?

[j-xella]

+1 on this. Useful when TW is on dropbox, for example.

[sukima]

I have mused over this for some time. A few things need to be addressed:

1. You have to watch a directory is is always a troublesome task but there are Node.JS libs available to help.
2. Once a watching process realizes there is a change how do you sync this? Many apps assume you are editing in another browser so they can use livereload to force the browser to refresh the page on a change. But this is problematic in TiddlyWiki's design since you would loose you place on a full reload without permaview always on.
3. Since TiddlyWiki handles changes made in the app how would one synchronize unsaved changes in the browser with now saved changes on disk? The conflict would be quite significant and could lead to increased human error and lost data.

I think this is an excellent place to have some discussion around these problems and hopefully develop a possible solution or consensus on how to approach implementing such a feature.

[tobibeer]

@sukima

1. You have to watch a directory is is always a troublesome task but there are Node.JS libs available to help.

Any particular modules you have in mind / experience with? There appears to be chokidar and variants of it and the underlying fs.watch().

2. How do you sync this? Many apps assume you are editing in another browser so they can use livereload to force the browser to refresh the page on a change.

I think the key here is a "partial page refresh". I don't think reloading the full ui is necessary. The sync adapter is the one to ensure tiddler representations are refreshed in the story(s). So, the key is chaining up the syncer with the importer to come to a managable form of collision management.

3. How would one synchronize unsaved changes in the browser with now saved changes on disk?

1. conflict recognition, i.e. not brute-overwriting
2. conflict handling, i.e. merge tools
3. conflict resolution, i.e. commiting the eventual result of a merge

All of the above presupposes some form of change management and quite possibly a version history as well. So in that sense, a Git-/GithubPlugin (or even using Github for the very purpose of a tiddler store) might at first appear to yield too complex a process. On the other hand, it would provide exactly the type of conflict management that is needed based on a solid revision history. So, the task of a Github-syncer would in-fact be to relay the git status to the frontend... and require the user to be able to handle the interface, while simplifying the process to as little as necessary... and leaving out any of the complex merge scenarios.

[pmario]

Any particular modules you have in mind / experience with?

Some existing libs, only work well for 1 OS category. Those, that work well with unix, sometimes can't cope with windows and vice versa. .. IMO we should have a closer look to existing build frameworks, that allow you to activate a watch mechanism. ...

I'm using webpack -w a lot to create the file-backups AddOn. It works very well on windows and ubuntu. ... But I don't know macOS

[pmario]

it seems webpack uses: https://www.npmjs.com/package/watchpack

[inmysocks]

This is a bit old but the multiuser plugin has a part that does this. It will hopefully eventually be split into its own plugin. In case anyone is interested, the relevant parts in what I made are:

• A web socket server to handle the two-way communication between the browser and the file system
• A new sync adapter type that uses the websockets
• The file system monitor part which is based on nodes fs.watch and sends a message to the sync adapter when there are changes

This can probably be done without websockets, but using them makes it easy to update the page without doing a refresh. The way I have it set up tiddlers change the same way as if you had done the editing in tiddlywiki so you don't need any full page refreshes.

[sukima]

Yup, livereload uses websockets as well. Seems to me the solution that the multi-user plugin created is the correct solution here. Can we get that sync adapter split-out from the plugin? maybe we could bundle it into core in the same what the filesystem plugin is now.

[inmysocks]

I am planning on splitting it into separate plugins and trying to get them added to the core once I get it working well enough. Hopefully that will be soon.

[xeor]

Any progress on the plugin @inmysocks ? Having the files stored in git would mean a near perfect personal note/journal-taking solution.. Willing to donate some money to get this thing rolling.

[inmysocks]

Bob has come a long way since March but I have had very little success splitting it up into different plugins.

What changes from the current version would you want in order to make using git with it easier? I considered making a plugin to go with Bob that would let you access gits functions from within a wiki but I haven't gotten around to it yet.

[xeor]

If a change is happening via Tiddlywiki, it should commit (commit-message not that important) and optional push (to a configurable remote, default to origin)

The other part would be more tricky.. Maybe a git pull on a configurable interval would be enough..

Alternative, this git part could be a whole separate thing, then the only thing needed would be for TiddlyWiki to detect if a file have changed, and make it available in the UI.

To be honest, I think the latter would be the best way to do this. It would then be upto the user to do the sync, either with using tools like https://www.cis.upenn.edu/~bcpierce/unison/, or other syncing tools. Creating a job that does git pull, git push from time to time isn't that hard..

[inmysocks]

What you are describing is exactly what Bob does. If there is a change on the file system than it is immediately changed in the browser, or if there is a change in the browser it is immediately sent to the file system.

I am eventually going to make a git plugin that automatically does the git part but for now I have been doing it manually.

[xeor]

Thanks for the info.. I did not know what Bob was, so I assumed it was a person working on the problem... :) I found https://github.com/OokTech/TW5-Bob/, and it looks very promising.. I'll check it out. Thanks

[linonetwo]

But bob has a bug that prevents me from using it OokTech/TW5-Bob#130

I'm going to make its https://github.com/OokTech/TW5-Bob/blob/master/FileSystem/FileSystemMonitor.js a standalone plugin, so I can have this bidirectional sync before the fix #4630 is merged.

Or maybe I can change the Filesystemadaptor as said in #1617

[saqimtiaz]

@linonetwo I think writing a plugin or running a patched version of the core is your best option until #4630 is properly implemented.

[pmario]

@dufferzafar, @Jermolene ... I think this one can be closed. latest node server version can do it.

[linonetwo]

@pmario Really? I tried changed one of my tid file, but noting changed in the browser.

[pmario]

You need to wait up to 1 minute or so. There is a sync timer somewhere.

[linonetwo]

@pmario will this happen in 5.1.22? Or 5.1.23-pre release?

My 5.1.22 nodejs wiki didn't change. And if this is not programmatically controllable, it won't be a good solution for me, because I want to use this feature in my Electron App.

[saqimtiaz]

As far as I am aware, the node.js server currently does not reload changed tiddlers from disc.

What it does do now, is send changed tiddlers from the node.js server (that is in memory in the server) to clients. Therefore editing a tiddler file has no effect until the server is restarted.

[pmario]

@saqimtiaz ... seems you are right. I did have this example in the head: https://groups.google.com/d/msg/tiddlywiki/PW0_ZCyD5SM/CZZRbw_wAgAJ ... So if you use curl to create new content, it definitely gets synced. I did just try it.

curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New' --data '{
 "tags": "FirstTag [[Second Tag]]",
 "my-custom-field": "Field value",
 "text": "your text comes here!\n\nnext paragraph"
}' -H "X-Requested-With: TiddlyWiki"

The only problems users seem to have is, that they want to modify the title field, which is not possible. One needs to modify the URL eg:

curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New'
curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New 1'
curl -X PUT -i 'http://<yourIP>:8080/recipes/default/tiddlers/New 2'

and so on.

[linonetwo]

I tried to do this:

// on delete
      if (event == 'remove') {

        // can't use $tw.wiki.syncadaptor.deleteTiddler(tiddlerTitle);  because it will try to modify fs, and will failed:

        /* Sync error while processing delete of 'blabla': Error: ENOENT: no such file or directory, unlink '/Users//Desktop/repo/wiki/Meme-of-LinOnetwo/tiddlers/blabla.tid'
        syncer-server-filesystem: Dispatching 'delete' task: blabla 
        Sync error while processing delete of 'blabla': Error: ENOENT: no such file or directory, unlink '/Users//Desktop/repo/wiki/Meme-of-LinOnetwo/tiddlers/blabla.tid' */

        $tw.syncadaptor.wiki.deleteTiddler(tiddlerTitle);
      }

Seems

1. $tw.syncadaptor.wiki.deleteTiddler will try to modify disk, is there a method to only modify wiki, but not the disk?
2. Seems $tw.syncadaptor.wiki.addTiddler(tiddler) will not sync the change to the wiki, I still have to reload wiki in the Browser to see the change, why?

[linonetwo]

This is my current watch-fs plugin:

https://github.com/linonetwo/wiki/blob/6abb5904edfc77a39cfbdf2a8058f3a649be6f3f/Meme-of-LinOnetwo/plugins/linonetwo/watch-fs/FileSystemMonitor.js

It handles create and update properly, but when I call $tw.syncadaptor.wiki.deleteTiddler(tiddlerTitle), there are intervaled error:

syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'
 syncer-server-filesystem: Dispatching 'delete' task: aaa 
Sync error while processing delete of 'aaa': Error: ENOENT: no such file or directory, unlink '/Users/linonetwo/xxx/wiki/Meme-of-LinOnetwo/tiddlers/aaa.tid'

And change is stay in the server, I don't know why it doesn't sync to the wiki automatically.

[linonetwo]

I managed to create a plugin to reload file that changed in the disk:

https://github.com/linonetwo/tiddlywiki-plugins/tree/master/plugins/linonetwo/watch-fs

There are still some limitations, such as

1. can't handle rename in the disk, you can only rename from within the wiki (no such API to tell tw I've renamed a file)
2. I'm now refreshing the wiki using $:/config/SyncPollingInterval as said in Way to add/delete a tiddler without triggering the sync mechanism  #2698 (comment) (no such API to tell the browser to refresh)
3. I haven't tested this with $:/config/FileSystemPaths and Fix file info #4630 , but I use this feature every day, so I will definitely support it
4. Can't handle if git change the tiddler while you are open its Draft tiddler (might be fixed by [BUG] Deleting a draft tiddler should not also delete the original tiddler #4792 )

Here is the working screenshot about how it works with git:

[pmario]

@linonetwo Hi, Jeremy somewhere mentioned, that he would be interested in "Server sent events". The mechanism is much easier as web-sockets.

Since TW server doesn't use 3rd party libs, the most basic examples I could find was: http://html5doctor.com/server-sent-events/ from 2012. It also links to a github repo: https://github.com/remy/eventsource-h5d/blob/master/app.js

IMO it should be interesting to include this function.

[linonetwo]

@pmario I can see it is actually poll from the client, to see if there is an update in the server, if so, it triggers a pull from the client.

I implement this in my watch-fs plugin:

client-side poll for sync state, if server has something new, it triggers $tw.syncer.syncFromServer():
https://github.com/linonetwo/tiddlywiki-plugins/blob/c494c2edd3e1b3c066e2032dc816943a38e5a9ca/plugins/linonetwo/watch-fs/trigger-sync.html

fs watcher debounces, wait for the disc to be stable:
https://github.com/linonetwo/tiddlywiki-plugins/blob/c494c2edd3e1b3c066e2032dc816943a38e5a9ca/plugins/linonetwo/watch-fs/FileSystemMonitor.js#L117-L129

A simple server to handle the poll:
https://github.com/linonetwo/tiddlywiki-plugins/blob/c494c2edd3e1b3c066e2032dc816943a38e5a9ca/plugins/linonetwo/watch-fs/get-can-sync.js

[hoelzro]

If you are interested in Server-Sent Events, @Arlen22 has an open PR adding that functionality: #4609

[linonetwo]

I've refactored watch-fs plugin to work with server-send-event plugin, you can try it https://github.com/tiddly-gittly/watch-fs/releases/tag/v0.1.0 with https://github.com/twcloud/tiddlyweb-sse/releases

Seem to work properly on simple cases, still need some testing.

And even it uses the chokidar package, the size is 137 KB, relatively small I think.

sourcecode: https://github.com/tiddly-gittly/watch-fs

[linonetwo]

#6047 is wrong.

Currently,

Disk change -> watcher emit 'change' -> calls $tw.wiki.addTiddler/deleteTiddler -> trigger 'change' on $tw.wiki -> filesystemadaptor & SSE listen on 'change', and modify disk and client.

The "modify disk" part is causing bug. So I should move my logic to filesystemadaptor, and let it don't react for change when it is emit from watcher.

And if MWS also listen on & triggering $tw.wiki 'change' event, then MWS can also reload changed tiddlers from disc.

The latest chokidar package works on node v14+ so its size is shrink to less than 40kB, I will prepare a PR to add it to the core filesystem plugin.

[pmario]

The latest chokidar package works on node v14+ so its size is shrink to less than 40kB, I will prepare a PR to add it to the core filesystem plugin.

I am not sure, what the PR should be about. Do you want to add a 3rd party dependency to the core?

The whole thread is 4 years old. So the used tools that have been up to date 4 years ago, are probably out of date now. So from my point of view, experiments using native functions should be discussed first.

[linonetwo]

Do you want to add a 3rd party dependency to the core?

Yes, or I can copy the code as JS and remove useless part, it is only 1.4k loc.

It solves some edge cases like

changes are reported as add / change / unlink instead of useless rename

which is still not solved in NodeJS native fs.watch method (as described in its readme.)

Anyway it is not going into the core, it goes to tiddlywiki/filesystem plugin.

---

But I'm not very sure about this, after reviewing twcloud/tiddlyweb-sse#1 (comment) , as coded in tiddlyweb-sse, it still triggers Syncer.syncFromServer, which calls getSkinnyTiddlers.

So if you don't like including chokidar, I can try implement getSkinnyTiddlers for filesystem syncadaptor, which will scan the disk again like booting. And syncer just need to poll filesystem every a few minutes, like it poll tiddlyweb.

If including chokidar is good, we can still implement getUpdatedTiddlers, so when it is polled, it reports cached updates every few minutes, instead of report each change immediately, which cause many conor cases.

[pmario]

Watching the filesystem, in an OS agnostic way, is very complex and prone to OS level bugs. IMO it is OK to watch the file system for changes to restart a development server, because it does not really matter if it breaks.

For production in an unknown OS from my point of view it is a no-go. It will always have to be resolved depending on the exact usecase for an exact OS used in production. I doubt, that there will ever be a "one code base solves all the problems" solution.

---

But I'm not very sure about this, after reviewing twcloud/tiddlyweb-sse#1 (comment) , as coded in tiddlyweb-sse, it still triggers Syncer.syncFromServer, which calls getSkinnyTiddlers.

Those 2 usecases can not really be compared because the underlaying mechanisms are fundamentally different.

I think the mechanism used in tiddlyweb-sse which calls getSkinnyTidddlers client-side is a missed opportunity. tiddlyweb-sse knows, which tiddler exactly has been changed, but it triggers a "brute force" read everything, to avoid bigger code changes at the client. I think, the better way would have been to send the changes, instead of triggering getSkinnyTiddlers. -- But that's just my opinion.