Add git add-to-store support for Nix #3

matthewbauer · 2020-05-28T22:25:21Z

sets up addToStore to work with both remote and local daemons.

The base branch was changed.

src/libstore/daemon.cc

src/libstore/local-store.cc

src/libstore/remote-store.cc

src/libstore/daemon.cc

src/nix-store/nix-store.cc

src/nix/add-to-store.cc

This matches what we want for blobs. Trees are still in progress - we need a way to symlink to other objects, using that to determine ca.

This is really bad and dangerous! But Git migration to sha256 is still a ways away: https://lwn.net/Articles/811068/ So we need to allow it for the time being.

We need access to other things in the store. This is kind of dangerous though if things are added in the wrong order�.

This is needed to create files based on git permissions

This updates the remote protocol to try to handle sha1 hashes.

This is more direct.

We need both to properly mess with the file system. storeDir goes into the hash while realStoreDir is what we read & write to.

Need to use the htSHA1 we were given, don’t recompute.

this makes maintanence easier

This reverts commit 7b6186d.

narHash is the hash of the nar, not the git objects.

We need to dump path and dump git here so that the hashes match what is expected elsewhere.

This should always use sha1 hash type, but we want to make sure the caller knows that. So just throw an error instead of ignoring hashAlgo on Git ingestion.

symlinks should be relative so that they look like: ../s5c0hnz9qfnpnn1bszfxicgz21d1fam3-dummy3 instead of /build/nix-test/store/s5c0hnz9qfnpnn1bszfxicgz21d1fam3-dummy3 This way our hashes will work with any real store dir. Note that /nix/store is still embedded in the store entry function.

this shouldn’t be needed - FdSink handles this for us.

Symlinks are resolved in the nar format so we end up with references to the real store dir. Copying avoids this and gives us a stable hash. Also update the ParseSink api with two methods: - copyFile - copyDirectory both are needed to properly implement git parsing.

libc++fs isn’t alway available

This deadlocks ProgressBar, e.g. # nix run --impure --no-substitute --store '/tmp/nix2?store=/foo' --expr 'derivation { builder = /nix/store/zi90rxslsm4mlr46l2xws1rm94g7pk8p-busybox-1.31.1-x86_64-unknown-linux-musl/bin/busybox; }' leads to Thread 1 (Thread 0x7ffff6126e80 (LWP 12250)): #0 0x00007ffff7215d62 in __lll_lock_wait () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0 #1 0x00007ffff720e721 in pthread_mutex_lock () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0 #2 0x00007ffff7ad17fa in __gthread_mutex_lock (__mutex=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:749 #3 std::mutex::lock (this=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_mutex.h:100 #4 std::unique_lock<std::mutex>::lock (this=0x7fffffff09a8, this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:141 #5 std::unique_lock<std::mutex>::unique_lock (__m=..., this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:71 #6 nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::Lock (s=0x6c5448, this=0x7fffffff09a0) at src/libutil/sync.hh:45 #7 nix::Sync<nix::ProgressBar::State, std::mutex>::lock (this=0x6c5448) at src/libutil/sync.hh:85 #8 nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:131 #9 0x00007ffff7608cfd in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88 #10 nix::getCodeLines (errPos=...) at src/libutil/error.cc:66 #11 0x00007ffff76073f2 in nix::showErrorInfo (out=..., einfo=..., showTrace=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/optional:897 #12 0x00007ffff7ad19e7 in nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:134 #13 0x00007ffff7ab9d10 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88 #14 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName="/home/eelco/Dev/nix/outputs/out/bin/nix", fun=...) at src/libmain/shared.cc:328 #15 0x000000000046226b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/ext/new_allocator.h:80

Ericson2314 previously approved these changes May 29, 2020

View reviewed changes

Ericson2314 changed the base branch from ipfs-develop to git-objects-develop May 29, 2020 03:04

Ericson2314 changed the base branch from git-objects-develop to ipfs-develop May 29, 2020 03:05