Support OneDrive by properly handling symlinks

[ssylvan]

OneDrive (and presumably other storage providers) uses symlinks to handle "placeholder" files. Hugo does not properly handle these (e.g. IsDir() will not return true for a symlink to a directory, but must be checked explicitly). There are probably other use cases for having working symlink support too, but OneDrive is the main repro case right now (when in "placeholder mode").

See this discussion for more info: https://discourse.gohugo.io/t/error-error-copying-static-files-when-run-from-a-onedrive-folder-on-windows/8872

[bep]

This will, most likely, be improved as part of #3651 -- In my long-running PR for that, I have rewritten the "file walking" for content files. But, there will probably be others -- the static files, maybe.

What do you mean by "placeholder" files?

[ssylvan]

"Placholders" is the OneDrive name for files that show up in the file system as if they were really there, but are actually downloaded off of the cloud storage on-demand, as they are used. It appears they are implemented using symlinks, so as long as all the IO operations Hugo does work for symlinks, it should work for placeholders.

[ssylvan]

This might actually be a legitimate Go bug, not hugo. See recent discussion at
arduino/arduino-builder#254 and golang/go#22579

[matthijskooijman]

For completeness: I think these are not actually "symlinks", but instead "rebranch points", a Windows filesystem mechanism that is used to implement, among other things, symlinks. See the issues linked above for some more details.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[ssylvan]

My understanding is that this issue will be fixed once this pull request is merged (and Hugo releases are subsequently recompiled with the new version of Go): rust-lang/rust#47956

[moorereason]

@ssylvan: except that PR is for Rust.

[ssylvan]

Doh. This is the issue for Go, I believe golang/go#22579 (comment)

On Fri, Feb 16, 2018 at 3:24 PM Cameron Moore ***@***.***> wrote: @ssylvan <https://github.com/ssylvan>: except that PR is for Rust. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4079 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABTVp8KIModJaQrc1pmoBhpW8p5g21LVks5tVg4YgaJpZM4QbWS8> .

-- Sebastian Sylvan

[psychelzh]

I found the same problem for blogdown package of r, which depends on hugo as the site building tool. And I filed a StackOverflow question previously.

[psychelzh]

golang/go#22579 is fixed now. Will hugo work now? @ssylvan @bep

[psychelzh]

FYI, this issue should have been fixed in the newest release of v0.40.3.

[anatomicunderground]

This issue does not seem to be fixed with the v0.40.3 Windows binaries provided in either 32 or 64 bit. I also tried recompiling Hugo from source with the latest Go (go1.10.2 windows/amd64) with no luck.

[moorereason]

The fix will be released in Go 1.11.

[mjmurphy]

Has anyone managed to get this working with a custom build of Hugo and Go 1.11beta to confirm it works?

When the official release of Go 1.11 is done, will Hugo start using this for all future builds or is there a wait period before adopting new versions?

[bep]

will Hugo start using this for all future builds or is there a wait period before adopting new versions?

We will start using it right away. I'm so excited about Go 1.11 (mostly because of the Go template variable re-assign fix) that I will probably do a Go release on the same day (but I suspect I will have to wait for the day after for the Go Docker images to be ready ...)

[mjmurphy]

Been testing the latest 0.48 build to see if it resolves issues when working with OneDrive. In summary it kind of works, but you need to make sure the build destination folder does not exist before building. Below are my results:

• Windows 10 (version 1709)
• OneDrive for Business
• Hugo v0.48 windows/amd64

All tests done with the whole Hugo project folder on OneDrive:

1. The standard Hugo server function - Success

hugo server
works as normal

2. Default Hugo build - with no /public/ folder in the project root - Success

Structured like so:
├── content
├── data
├── layouts
├── static
└── themes
Running:
hugo
Hugo build creates the public folder and the whole site builds runs fine

3. Default Hugo build - with an existing /public/ folder in root - Fail

Structured like so:
├── content
├── data
├── layouts
├── public
├── static
└── themes
Running:
hugo
Build fails with error:
Error: Error copying static files: mkdir "path to OneDrive project folder\public\": Cannot create a file when that file already exists.

4. Hugo build with destination set to another OneDrive folder - Fail

Running:
hugo --destination "path to another OneDrive folder"
Build fails with error:
Error: Error copying static files: mkdir "path to another OneDrive folder": Cannot create a file when that file already exists.

5. Hugo build with destination set to a standard windows folder (e.g. C:\project-name) - Success

hugo --destination "path to a standard windows folder"
Build and copy works fine.

[mjmurphy]

Update - I did not fully test the server command properly:

Running hugo server command works, but the live-reload does not detect any changes in the content folder. However, Live-reload does detect template changes in the themes and data directory.

It says Watching for changes in C:\Users\user\OneDrive\Projects\ProjectSiteName\{data,static,themes} so it looks like it's completely ignoring the content folder when watching

I just tested again with v0.49 windows/amd64 and all the build tests I mentioned in my previous comment are all the same.

Has anyone else on this thread managed to get Hugo working smoothly (server live-reload and without the build workarounds) on OneDrive since 0.48?

[mjmurphy]

I just tested some of the Hugo Pipes features using v0.49/extended windows/amd64

The following features I tested all worked flawlessly from a OneDrive folder:

• resources.Get
• resources.ToCSS
• resources.Concat
• resources.Minify

In summary, the main issues I've discovered using the latest version of Hugo with OneDrive are:

1. Hugo server does not detect changes in the content folder (it does detect changes in all other folders data, themes etc.)
2. Hugo build will not copy to another location on OneDrive if the folder already exists. With the default build this can be solved by deleting the public folder before running the build. I'm guessing this is some kind of permission issue as other features like Hugo pipes seems to be working fine to create and update files / folders elsewhere.

[bep]

@mjmurphy I guess both of those issues are fixable. For 1) I do remember that we have some special handling when walking the content directories. The fsnotify watcher we use isn't recursive, so we need to walk the tree and add the directories we find to watch.

[mjmurphy]

thanks @bep
I can setup a Windows 10 VM with OneDrive for testing if it helps. Just let me know and I will send you the login details.

[mjmurphy]

Good news! We just upgraded our Windows 10 build from 1709 to 1803 and all the issues I described in this earlier comment have now been resolved! 🎉🎈

The company I work for has quite a delayed release cycle for windows updates - Windows 10 Build 1803 was actually released on April 30, 2018 so maybe most people on this thread had already upgraded, hence why I was the only one reporting any problems!

If no one else reports anything, I wonder if this issue can now be closed?

It's great that we can use Hugo + OneDrive again without any workarounds, thanks again for your support @bep

[jmooring]

Closing. Comments from @mjmurphy indicate resolution a long time ago.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.