Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WARNING: fsync is not supported by the data storage #230

Closed
r0l1 opened this issue May 8, 2023 · 7 comments · Fixed by #231
Closed

WARNING: fsync is not supported by the data storage #230

r0l1 opened this issue May 8, 2023 · 7 comments · Fixed by #231

Comments

@r0l1
Copy link

r0l1 commented May 8, 2023 •
edited
Loading

Output of rest-server --version

docker image: f41f6db080c5
Digest: sha256:8b806c911815e78db8a95e7d41f34780d0690eae0d75cf8a49c35c8ec3c831e3

How did you run rest-server exactly?

docker run --name "restic-server" -p 8000:8000 -v /data:/data --env "OPTIONS=--private-repos --append-only" restic/rest-server

What backend/server/service did you use to store the repository?

Expected behavior

fsync working. This is an ext4 drive (raid1 -> luks -> ext4).

Actual behavior

I get hundreds of following messages during a backup process:

WARNING: fsync is not supported by the data storage. This can lead to data loss, if the system crashes or the storage is unexpectedly disconnected.

Steps to reproduce the behavior

Don't know. I compiled a simple go program (via docker golang:alpine) and tested for f.Sync support. No error is thrown and it also runs within the same docker container without any problems.

Do you have any idea what may have caused this?

Maybe the compiled binary within the official docker image differs from my test binary? Different go versions?

Do you have an idea how to solve the issue?

Currently not. The filesystem supports fsync.

Did rest-server help you today? Did it make you happy in any way?

Restic is just awesome and I really appreciate this open source project.
@r0l1
Copy link
Author

r0l1 commented May 8, 2023

And why are there so many log messages? Looking at the code:

fsyncWarning sync.Once

should actually result into one single log message. Any hints?

@MichaelEischer MichaelEischer mentioned this issue May 8, 2023
Merged
7 tasks
@MichaelEischer
Copy link
Member

MichaelEischer commented May 8, 2023 •
edited
Loading

Arghh, looks like I've messed up the check for fsync support in #199. #231 fixes the fsync check. But I'm not yet sure why the warning is printed multiple times.

@wojas
Copy link
Contributor

wojas commented May 9, 2023

But I'm not yet sure why the warning is printed multiple times.

A new Handler is created for every request, so it will current be printed at most once per request, which is not very helpful.

Perhaps create a global var for the sync.Once?

@MichaelEischer
Copy link
Member

@wojas I've moved the sync.Once instance into the server struct.

@DanielGibson
Copy link

Good to see this fixed, I was just gonna report this :)

My setup is similar (harddisk partition -> LUKS encryption -> ext4), and I wasn't 100% sure if the warning maybe is true and due to the LUKS in the middle maybe fsync() on ext4 doesn't go through to the physical disk, but according to https://stackoverflow.com/questions/32225517/fsync-on-mapped-crypted-device-with-dm-crypt it does?

(either way the warning should of course only be logged once, so I'm glad that's fixed)

@kittkott
Copy link

kittkott commented Jul 4, 2023

Windows Server 2019
rest-server version rest-server 0.12.0 compiled with go1.20.3 on windows/amd64

I get hundreds of following messages during a backup process:

"WARNING: fsync is not supported by the data storage. This can lead to data loss, if the system crashes or the storage is unexpectedly disconnected."

@DanielGibson
Copy link

The fix has been implemented on May 18th, but the last release (0.12.0, the one you're using) is from April.

I think a new release would make sense, as those messages are both noisy and (apparently wrongly) alarming.
(Says the guy who hasn't done a release of his open source project in over a year 🙃)

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Oct 19, 2023
@0-wiz-0
restic-rest-server: update to 0.12.1.
6f5bcca 
Fix PKGNAME while here.

Changelog for rest-server 0.12.1 (2023-07-09)
============================================

The following sections list the changes in rest-server 0.12.1 relevant
to users. The changes are ordered by importance.

Summary
-------

 * Fix #230: Fix erroneous warnings about unsupported fsync
 * Fix #238: API: Return empty array when listing empty folders
 * Enh #217: Log to stdout using the `--log -` option

Details
-------

 * Bugfix #230: Fix erroneous warnings about unsupported fsync

   Due to a regression in rest-server 0.12.0, it continuously printed `WARNING: fsync is not
   supported by the data storage. This can lead to data loss, if the system crashes or the storage is
   unexpectedly disconnected.` for systems that support fsync. We have fixed the warning.

   restic/rest-server#230
   restic/rest-server#231

 * Bugfix #238: API: Return empty array when listing empty folders

   Rest-server returned `null` when listing an empty folder. This has been changed to returning
   an empty array in accordance with the REST protocol specification. This change has no impact on
   restic users.

   restic/rest-server#238
   restic/rest-server#239

 * Enhancement #217: Log to stdout using the `--log -` option

   Logging to stdout was possible using `--log /dev/stdout`. However, when the rest server is run
   as a different user, for example, using

   `sudo -u restic rest-server [...] --log /dev/stdout`

   This did not work due to permission issues.

   For logging to stdout, the `--log` option now supports the special filename `-` which also
   works in these cases.

   restic/rest-server#217
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Aug 6, 2024
@0-wiz-0
restic-rest-server: update to 0.13.0.
c4e0217 
Changelog for rest-server 0.13.0 (2024-07-26)
============================================

The following sections list the changes in rest-server 0.13.0 relevant
to users. The changes are ordered by importance.

Summary
-------

 * Chg #267: Update dependencies and require Go 1.18 or newer
 * Chg #273: Shut down cleanly on TERM and INT signals
 * Enh #271: Print listening address after start-up
 * Enh #272: Support listening on a unix socket

Details
-------

 * Change #267: Update dependencies and require Go 1.18 or newer

   Most dependencies have been updated. Since some libraries require newer language
   features, support for Go 1.17 has been dropped, which means that rest-server now
   requires at least Go 1.18 to build.

   restic/rest-server#267

 * Change #273: Shut down cleanly on TERM and INT signals

   Rest-server now listens for TERM and INT signals and cleanly closes down the
   http.Server and listener when receiving either of them.

   This is particularly useful when listening on a unix socket, as the server will
   now remove the socket file when it shuts down.

   restic/rest-server#273

 * Enhancement #271: Print listening address after start-up

   When started with `--listen :0`, rest-server would print `start server on :0`

   The message now also includes the actual address listened on, for example `start
   server on 0.0.0.0:37333`. This is useful when starting a server with an
   auto-allocated free port number (port 0).

   restic/rest-server#271

 * Enhancement #272: Support listening on a unix socket

   It is now possible to make rest-server listen on a unix socket by prefixing the
   socket filename with `unix:` and passing it to the `--listen` option, for
   example `--listen unix:/tmp/foo`.

   This is useful in combination with remote port forwarding to enable a remote
   server to backup locally, e.g.:

   ```
   rest-server --listen unix:/tmp/foo &
   ssh -R /tmp/foo:/tmp/foo user@host restic -r rest:http+unix:///tmp/foo:/repo backup
   ```

   restic/rest-server#272


Changelog for rest-server 0.12.1 (2023-07-09)
============================================

The following sections list the changes in rest-server 0.12.1 relevant
to users. The changes are ordered by importance.

Summary
-------

 * Fix #230: Fix erroneous warnings about unsupported fsync
 * Fix #238: API: Return empty array when listing empty folders
 * Enh #217: Log to stdout using the `--log -` option

Details
-------

 * Bugfix #230: Fix erroneous warnings about unsupported fsync

   Due to a regression in rest-server 0.12.0, it continuously printed `WARNING:
   fsync is not supported by the data storage. This can lead to data loss, if the
   system crashes or the storage is unexpectedly disconnected.` for systems that
   support fsync. We have fixed the warning.

   restic/rest-server#230
   restic/rest-server#231

 * Bugfix #238: API: Return empty array when listing empty folders

   Rest-server returned `null` when listing an empty folder. This has been changed
   to returning an empty array in accordance with the REST protocol specification.
   This change has no impact on restic users.

   restic/rest-server#238
   restic/rest-server#239

 * Enhancement #217: Log to stdout using the `--log -` option

   Logging to stdout was possible using `--log /dev/stdout`. However, when the rest
   server is run as a different user, for example, using

   `sudo -u restic rest-server [...] --log /dev/stdout`

   This did not work due to permission issues.

   For logging to stdout, the `--log` option now supports the special filename `-`
   which also works in these cases.

   restic/rest-server#217
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix inverted condition for fsync warning MichaelEischer/rest-server
5 participants
@wojas @DanielGibson @MichaelEischer @r0l1 @kittkott