-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shuffle ParseSink
code in preparation for git hashing support
#9024
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
39ba81a
Improve internal API docs for two file hashing functions
Ericson2314 9d61143
Move `ParseSink` to its own header
Ericson2314 8a416e8
Move `RestoreSink` to `fs-sink.cc`
Ericson2314 f2e201f
Expose `RestoreSink` in header (`fs-sink.hh`)
Ericson2314 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
#include <fcntl.h> | ||
|
||
#include "config.hh" | ||
#include "fs-sink.hh" | ||
|
||
namespace nix { | ||
|
||
struct RestoreSinkSettings : Config | ||
{ | ||
Setting<bool> preallocateContents{this, false, "preallocate-contents", | ||
"Whether to preallocate files when writing objects with known size."}; | ||
}; | ||
|
||
static RestoreSinkSettings restoreSinkSettings; | ||
|
||
static GlobalConfig::Register r1(&restoreSinkSettings); | ||
|
||
|
||
void RestoreSink::createDirectory(const Path & path) | ||
{ | ||
Path p = dstPath + path; | ||
if (mkdir(p.c_str(), 0777) == -1) | ||
throw SysError("creating directory '%1%'", p); | ||
}; | ||
|
||
void RestoreSink::createRegularFile(const Path & path) | ||
{ | ||
Path p = dstPath + path; | ||
fd = open(p.c_str(), O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC, 0666); | ||
if (!fd) throw SysError("creating file '%1%'", p); | ||
} | ||
|
||
void RestoreSink::closeRegularFile() | ||
{ | ||
/* Call close explicitly to make sure the error is checked */ | ||
fd.close(); | ||
} | ||
|
||
void RestoreSink::isExecutable() | ||
{ | ||
struct stat st; | ||
if (fstat(fd.get(), &st) == -1) | ||
throw SysError("fstat"); | ||
if (fchmod(fd.get(), st.st_mode | (S_IXUSR | S_IXGRP | S_IXOTH)) == -1) | ||
throw SysError("fchmod"); | ||
} | ||
|
||
void RestoreSink::preallocateContents(uint64_t len) | ||
{ | ||
if (!restoreSinkSettings.preallocateContents) | ||
return; | ||
|
||
#if HAVE_POSIX_FALLOCATE | ||
if (len) { | ||
errno = posix_fallocate(fd.get(), 0, len); | ||
/* Note that EINVAL may indicate that the underlying | ||
filesystem doesn't support preallocation (e.g. on | ||
OpenSolaris). Since preallocation is just an | ||
optimisation, ignore it. */ | ||
if (errno && errno != EINVAL && errno != EOPNOTSUPP && errno != ENOSYS) | ||
throw SysError("preallocating file of %1% bytes", len); | ||
} | ||
#endif | ||
} | ||
|
||
void RestoreSink::receiveContents(std::string_view data) | ||
{ | ||
writeFull(fd.get(), data); | ||
} | ||
|
||
void RestoreSink::createSymlink(const Path & path, const std::string & target) | ||
{ | ||
Path p = dstPath + path; | ||
nix::createSymlink(target, p); | ||
} | ||
|
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
#pragma once | ||
///@file | ||
|
||
#include "types.hh" | ||
#include "serialise.hh" | ||
|
||
namespace nix { | ||
|
||
/** | ||
* \todo Fix this API, it sucks. | ||
*/ | ||
struct ParseSink | ||
{ | ||
virtual void createDirectory(const Path & path) { }; | ||
|
||
virtual void createRegularFile(const Path & path) { }; | ||
virtual void closeRegularFile() { }; | ||
virtual void isExecutable() { }; | ||
virtual void preallocateContents(uint64_t size) { }; | ||
virtual void receiveContents(std::string_view data) { }; | ||
|
||
virtual void createSymlink(const Path & path, const std::string & target) { }; | ||
}; | ||
|
||
struct RestoreSink : ParseSink | ||
{ | ||
Path dstPath; | ||
AutoCloseFD fd; | ||
|
||
|
||
void createDirectory(const Path & path) override; | ||
|
||
void createRegularFile(const Path & path) override; | ||
void closeRegularFile() override; | ||
void isExecutable() override; | ||
void preallocateContents(uint64_t size) override; | ||
void receiveContents(std::string_view data) override; | ||
|
||
void createSymlink(const Path & path, const std::string & target) override; | ||
}; | ||
|
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps this should be
filesystem.cc
, together with other fs utils (to be moved later), considering per-compilation-unit overhead as mentioned in the meeting today.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually I forgot before during the meeting, I really disagree with combining files because it is worse for incremental builds. Latency is most important during development when we are doing incremental builds.
Optimizing non-incremental building is IMO quite a smell. (And we can still have the automatic way of combining files based on number of CPUs if we like.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is that something Meson can do?
Couldn't find details about how that is achieved. Maybe you know the right search term?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No problem, https://mesonbuild.com/Unity-builds.html