Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write buffer metadata into file ASAP #1628

Merged
merged 2 commits into from
Jul 21, 2017
Merged

Write buffer metadata into file ASAP #1628

merged 2 commits into from
Jul 21, 2017

Conversation

tagomoris
Copy link
Member

As shown in #1589, file buffer chunks possibly make broken/empty .meta files for its metadata.
I found that some points to be improved to prevent such cases:

  • write metadata into files just after initializing buffer chunk object
    • before: .meta content is written at the time of #commit or enqueue! - it's too late
  • write metadata msgpack binary, then truncate .meta file into the size of metadata
    • before: .meta was truncated into 0 at first, then msgpack binary was written

@tagomoris
Write metadata from head of file at first, then truncate the file to …
34ef000 
…the size of metadata binary.

It's to minimize the possibility to make empty metadata file (by failing write metadata after truncating the file), and
it should work well with `#restore_metadata` method because MessagePack unpacker reads just one complete msgpack object
from the head of file (and ignore rest).
@tagomoris
write the metadata into file as soon as possible after initializing i…
ea86fc0 
…nternal data
@tagomoris
Copy link
Member Author

tagomoris commented Jul 13, 2017 •
edited
Loading

The behavior of msgpack unpacker for non-truncated files is proven here: https://gist.github.com/tagomoris/4b22651072aadc74b0ab9ecdfcf658a2

[a loooooooooooooooooooooooong msgpack binary]
[overwriting short binary]oong msgpack binary] # => unpacker unpacks shorter one only

@repeatedly repeatedly added bug Something isn't working v0.14 labels Jul 13, 2017
@repeatedly repeatedly mentioned this pull request Jul 13, 2017
Closed
@repeatedly repeatedly merged commit 3492a75 into master Jul 21, 2017
@repeatedly
Copy link
Member

repeatedly commented Jul 21, 2017 •
edited
Loading

I tested this patch with several days and no problem on my mac.
So merged and will release rc version for testing at other environment.

@repeatedly repeatedly deleted the truncate-after-write-on-file-metadata branch March 9, 2018 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@repeatedly repeatedly Awaiting requested review from repeatedly

Assignees
No one assigned
Labels
bug Something isn't working v0.14
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tagomoris @repeatedly