fix: update orgmode db on roam node creation #64

seflue · 2024-09-19T22:47:47Z

To make a new roam node accessible for refiling, searching and agenda the org files need to be loaded again.

chipsenkbeil · 2024-09-20T03:25:26Z

@seflue help me understand. Is this because we have to maintain a separate OrgFiles instance than the orgmode plugin, and therefore when adding a new node we need to do this to make it accessible to the broader orgmode plugin?

If so, can you add a comment above the line to explain that? Pretty sure I'll forget why we're doing this.

kristijanhusak · 2024-09-20T05:06:49Z

lua/org-roam/database/loader.lua

@@ -348,6 +348,8 @@ function M:load_file(opts)
                        insert_new_file_into_database(db, file, {
                            force = opts.force or file.metadata.changedtick ~= changedtick,
                        })
+
+                        require("orgmode").files:load(true)


@seflue line 329 should already add this file.
This might be an issue in orgmode, because we do early return here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/files/init.lua#L62-L64.

Can you try comment out that early return, remove this line, and see if it works then?

Edit: Ok, I think line 329 points to a different file loader. If that's the case, we need to do this, but this is not the most optimized way. Let me think a bit about this.

@kristijanhusak Thanks for considering a more optimal solution. I also have the feeling, that something can be improved in loading org-files.

While you are already reflecting on this topic I want to bring in another thought: For Telescope-orgmode I always wished for incremental loading capabilities of orgfiles. Like a stream, so user get's immediate feedback as soon as the first org file has been parsed and the results can be updated with each file parsed. Telescope itself is built to handle such cases, but orgmode currently does not support this.

Is this something you would need to factor into your considerations of optimizing file loading or should I create another issue for this?

While you are already reflecting on this topic I want to bring in another thought: For Telescope-orgmode I always wished for incremental loading capabilities of orgfiles. Like a stream, so user get's immediate feedback as soon as the first org file has been parsed and the results can be updated with each file parsed. Telescope itself is built to handle such cases, but orgmode currently does not support this.

I don't plan to add a stream like loading, because then I would have to worry about the pending files when showing the agenda and similar. There is caching in place so loading is slow only once.
Even this load(true) function should be fairly fast in this scenario because only files that are not already loaded will be loaded, but I'll still see to add a method that will load the file and add it to path, but only if it belongs to path.

If you add this method to load a single file, I think, I could add this stream-based approach myself in Telescope-orgmode. For me it's just a matter of a flexible API to provide plugins the possibility to do, what they need. That you don't want to implement a stream-based loading directly in orgmode makes sense.

@chipsenkbeil do you recall if this add_to_paths is needed to just load the file to files table, or we also need it to append it to path?

Currently, add_to_paths will load the file, put it in OrgFiles.files table, and also append the filename to OrgFiles.paths if the file does not belong to the defined path wildcard.

So for example, if my org-roam config is ~/roam/**/*, orgmode org_agenda_files is ~/org, and I do :add_to_paths('~/some/other/path/file.org') in any of those file loaders, this will add the file to the paths for the session, even though that might not be desired.
If a file that is already part of the path wildcard, for example :add_to_paths('~/roam/myfile.org') with a roam file loader, we are not adding it to paths, since it will be loaded anyway.

My question is: Is it ok if we do not append the filename to OrgFiles.paths in add_to_paths? Does org-roam always creates files that are defined in the path wildcard (in my example, ~/roam/**/*)?

Org roam takes a directory path, not a glob, and always creates node files within that directory. Nowhere else.

To be compatible with your plugin, I convert the directory to a glob doing a simple path join: join_path("path/to/roam", "**", "*").

I'd have to go back to read why the function was added. I've got time today to do that.

@kristijanhusak I tried to skim through the code to refresh myself.

With OrgFiles, I directly access the all_files field, and make use of these functions:

filenames

load_file

add_to_paths

load

I pass OrgFiles to things like OrgCapture. I don't think I access paths directly.

and always creates node files within that directory

This answers my question, thanks!

@seflue let me do some updates on the orgmode side and I'll let you know what to change here.

@chipsenkbeil If you need some helper function to avoid accessing properties directly let me know, we can add something.

seflue · 2024-09-21T12:28:22Z

@seflue help me understand. Is this because we have to maintain a separate OrgFiles instance than the orgmode plugin, and therefore when adding a new node we need to do this to make it accessible to the broader orgmode plugin?

If so, can you add a comment above the line to explain that? Pretty sure I'll forget why we're doing this.

I added the comment you requested. @chipsenkbeil Are you ok with merging that, as long as nvim-orgmode does not provide this directly in files:add_to_path?
@kristijanhusak If you adjust this behavior in orgmode, do you agree, that we should increase the version number, so plugins can be clear in it's dependency?

chipsenkbeil · 2024-09-21T15:28:55Z

@seflue help me understand. Is this because we have to maintain a separate OrgFiles instance than the orgmode plugin, and therefore when adding a new node we need to do this to make it accessible to the broader orgmode plugin?

If so, can you add a comment above the line to explain that? Pretty sure I'll forget why we're doing this.

I added the comment you requested. @chipsenkbeil Are you ok with merging that, as long as nvim-orgmode does not provide this directly in files:add_to_path?

I'm fine with that. Give me today to try to fully answer @kristijanhusak question in the review in case that changes what we need to do.

kristijanhusak · 2024-09-21T17:23:48Z

@kristijanhusak If you adjust this behavior in orgmode, do you agree, that we should increase the version number, so plugins can be clear in it's dependency?

Yes, we can do that. @chipsenkbeil just keep in mind recent changes to hyperlinks before we decide to bump the version.

kristijanhusak · 2024-09-23T07:36:14Z

@seflue @chipsenkbeil I pushed nvim-orgmode/orgmode@2a10172 that adds an optional opts argument to load_file to persist the file in the session, if it belongs to the defined path, and then made add_to_paths use that in nvim-orgmode/orgmode@fd95708. It should not break anything since org-roam adds files that are already defined in the paths on the file loader.
Also made a new release https://github.com/nvim-orgmode/orgmode/releases/tag/0.3.6.

Since we need to keep things backward compatible, let's do this in this PR:

Do require('orgmode').files:add_to_paths(filename) instead of :load(true). This should make it work even with older orgmode versions, but it will append the files to the path. Not a big deal.

@chipsenkbeil once you update the minimum orgmode version to 0.3.6, make sure to update all add_to_paths(filename) to load_file(filename, { persist = true }), since I deprecated add_to_paths.

seflue · 2024-09-23T17:56:59Z

Do require('orgmode').files:add_to_paths(filename) instead of :load(true). This should make it work even with older orgmode versions, but it will append the files to the path. Not a big deal.

@kristijanhusak We already call :add_to_paths in line 329, so I am not sure, how (with older orgmode versions) the exchange of :load(true) with another call to :add_to_path(filename) make this work. Can you elaborate on that or do I have a misunderstanding?

@chipsenkbeil once you update the minimum orgmode version to 0.3.6, make sure to update all add_to_paths(filename) to load_file(filename, { persist = true }), since I deprecated add_to_paths.

@chipsenkbeil Should I create a new PR to do the adjustments including the version bump? Depending on what @kristijanhusak says, we can merge this PR before the second one or close it, if we don't care about the issue in older versions. Or we reuse it and I do the adjustments here right away.

seflue · 2024-09-23T18:44:10Z

@kristijanhusak I played around with your suggestions but in the end, only the call to :load(true) makes a new node available for refiling and search after creation. Even with the 0.3.6 version of orgmode.

kristijanhusak · 2024-09-23T18:50:34Z

@seflue just replace the line that you added with this:

require('orgmode').files:add_to_paths(file.filename)

And give it a test. It should also work fine.

seflue · 2024-09-23T18:52:47Z

@seflue just replace the line that you added with this:
require('orgmode').files:add_to_paths(file.filename)
And give it a test. It should also work fine.

@kristijanhusak As I said, I tested all possible variations of what you suggested and none of them worked. But bringing back my old line require('orgmode').files:load(true) works.

kristijanhusak · 2024-09-23T18:57:25Z

@seflue I'll give it a test. Are you using the same path for org roam and org agenda files?

seflue · 2024-09-23T18:58:30Z

@seflue I'll give it a test. Are you using the same path for org roam and org agenda files?

My paths for roam files is a subset of my agenda paths in orgmode. Or to be more precise: It is included by the globbing of one of my agenda file paths.

For orgmode I have something like this:

opts = {
-- ...
  org_agenda_files = {
    "~/my/agenda/path1/**/*.org",
    "~/my/agenda/path2/**/*.org",
    "~/my/agenda/path3/**/*.org",
  },
-- ...
},

And for org-roam:

opts = {
-- ...
  directory = "~/my/agenda/path3/org-roam-files"
-- ...
},

kristijanhusak · 2024-09-23T19:48:20Z

@seflue what action are you doing in org roam? This is what I tried:

Setup:

  require('orgmode').setup({
    org_agenda_files = { '~/orgtest/**/*.org' },
    org_default_notes_file = '~/orgtest/refile.org',
  })
  require('org-roam').setup({
    directory = '~/orgtest/roam',
  })

Open up neovim, do <leader>nc to capture a new node, name it my test node
The content of the capture buffer looks something like this:

:PROPERTIES:
:ID: 18a8b8f9-8ac3-4c71-ba23-328101903016
:END:
#+TITLE: my test node

* My test headline

I finish capturing
do :edit ~/orgtest/todos.org, go to a headline, do <leader>or to refile a headline to a destination
In prompt, press 2 and <tab>, node appears

This works when I add require('orgmode').files:add_to_paths(file.filename) as suggested.

Are you doing something else?

seflue · 2024-09-23T20:10:27Z

@kristijanhusak: Yes, I do. My particular use case is, that I want to use Telescope-orgmode.nvim for refiling and searching.

Here is, what I do, because this is exactly my use case:

In an existing org agenda file I write the words "Test Node", mark them in visual mode and press <leader>nm to create a new node and a link to it.
Either I create a new Headline next to it, and open the refiling option of Telescope-orgmode.nvim. Here I switch to "files" and expect the new node to appear. Alternatively I can just search for "Test Node" using Telescope-orgmode.nvims "Search orgfile" feature. Here I expect the new node to appear directly. With :load(true) it does, with :add_to_path it does not.

So :load(true) seems to update a cache, which is not updated by add_to_path or load_file. My goal is to implicitly update this cache, when I create a new node in org-roam. But I didn't quite understand the difference between these two functions, I only see the difference in the result.

kristijanhusak · 2024-09-23T20:49:24Z

@seflue Ok, I found it. I mistyped the argument. Fixed in nvim-orgmode/orgmode@08d763d.

It should work with the suggested orgmode version from org-roam (0.3.4). This was issue just on master.

To make a new roam node accessible for refiling, searching and agenda the org files need to be loaded again.

seflue · 2024-09-23T21:31:22Z

@kristijanhusak I can confirm now, that with orgmode on master it now works for me, if I use :load_file instead of add_to_path on line 329 and remove the additional :load(true) in line 358.
And it also works, if I replace :load(true) with :add_to_path whith orgmode on 0.3.4. Thanks for your effort! I think we should bump the orgmode version another time to allow to pin org-roam to a working version of orgmode.

@chipsenkbeil: We can merge this PR now, if that's ok for you. I already have an adjusted version, which is compatible with orgmode master ready on another branch. I would rebase it after we merged this and create a new PR.

seflue · 2024-09-24T08:21:25Z

@kristijanhusak I saw that you created a tag 0.3.61 and a release 0.3.7 pointing to this tag. Is there a reason for this inconsistency? I would prefer, if we just count up the last number for bugfix releases - no matter if a release includes just one commit or dozens. And bound release numbers strictly to tags.

kristijanhusak · 2024-09-24T08:35:13Z

@seflue that was a typo. I just released a regular 0.3.7 version with another bug fix, that also includes this one.

chipsenkbeil · 2024-09-24T13:12:10Z

@seflue i want to release one more minor version before we bump the required orgmode version.

If we merge this, we're still compatible with 0.3.4?

seflue · 2024-09-24T13:17:46Z

If we merge this, we're still compatible with 0.3.4?

Yes.

seflue mentioned this pull request Sep 19, 2024

Trigger org database update at runtime nvim-orgmode/orgmode#809

Closed

kristijanhusak reviewed Sep 20, 2024

View reviewed changes

seflue force-pushed the fix/update_org_db branch 2 times, most recently from 0d4666f to 73b8474 Compare September 21, 2024 12:25

seflue mentioned this pull request Sep 21, 2024

No refile option presented when creating node #61

Closed

fix: update orgmode db on roam node creation

7efcffd

To make a new roam node accessible for refiling, searching and agenda the org files need to be loaded again.

seflue force-pushed the fix/update_org_db branch from 73b8474 to 7efcffd Compare September 23, 2024 21:15

chipsenkbeil merged commit 17f85ab into main Sep 24, 2024
6 checks passed

chipsenkbeil deleted the fix/update_org_db branch September 24, 2024 13:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: update orgmode db on roam node creation #64

fix: update orgmode db on roam node creation #64

seflue commented Sep 19, 2024

chipsenkbeil commented Sep 20, 2024

kristijanhusak Sep 20, 2024 •

edited

Loading

seflue Sep 20, 2024

kristijanhusak Sep 20, 2024

seflue Sep 20, 2024

kristijanhusak Sep 20, 2024

chipsenkbeil Sep 21, 2024 •

edited

Loading

chipsenkbeil Sep 22, 2024

kristijanhusak Sep 22, 2024

seflue commented Sep 21, 2024

chipsenkbeil commented Sep 21, 2024

kristijanhusak commented Sep 21, 2024

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 •

edited

Loading

seflue commented Sep 23, 2024

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 •

edited

Loading

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 •

edited

Loading

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 •

edited

Loading

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024

seflue commented Sep 24, 2024

kristijanhusak commented Sep 24, 2024

chipsenkbeil commented Sep 24, 2024

seflue commented Sep 24, 2024

fix: update orgmode db on roam node creation #64

fix: update orgmode db on roam node creation #64

Conversation

seflue commented Sep 19, 2024

chipsenkbeil commented Sep 20, 2024

kristijanhusak Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

seflue Sep 20, 2024

Choose a reason for hiding this comment

kristijanhusak Sep 20, 2024

Choose a reason for hiding this comment

seflue Sep 20, 2024

Choose a reason for hiding this comment

kristijanhusak Sep 20, 2024

Choose a reason for hiding this comment

chipsenkbeil Sep 21, 2024 • edited Loading

Choose a reason for hiding this comment

chipsenkbeil Sep 22, 2024

Choose a reason for hiding this comment

kristijanhusak Sep 22, 2024

Choose a reason for hiding this comment

seflue commented Sep 21, 2024

chipsenkbeil commented Sep 21, 2024

kristijanhusak commented Sep 21, 2024

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 • edited Loading

seflue commented Sep 23, 2024

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 • edited Loading

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 • edited Loading

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024 • edited Loading

kristijanhusak commented Sep 23, 2024

seflue commented Sep 23, 2024

seflue commented Sep 24, 2024

kristijanhusak commented Sep 24, 2024

chipsenkbeil commented Sep 24, 2024

seflue commented Sep 24, 2024

kristijanhusak Sep 20, 2024 •

edited

Loading

chipsenkbeil Sep 21, 2024 •

edited

Loading

seflue commented Sep 23, 2024 •

edited

Loading

seflue commented Sep 23, 2024 •

edited

Loading

seflue commented Sep 23, 2024 •

edited

Loading

seflue commented Sep 23, 2024 •

edited

Loading